# Exporting Billing Metrics
Source: https://docs.fireworks.ai/accounts/exporting-billing-metrics

Export billing and usage metrics for all Fireworks services

## Overview

Fireworks provides a CLI tool to export comprehensive billing metrics for all usage types including serverless inference, on-demand deployments, and fine-tuning jobs. The exported data can be used for cost analysis, internal billing, and usage tracking.

## Exporting billing metrics

Use the Fireworks CLI to export a billing CSV that includes all usage:

```bash theme={null}
# Authenticate (once)
firectl login

# Export billing metrics to CSV
firectl billing export-metrics
```

## Examples

Export all billing metrics for an account:

```bash theme={null}
firectl billing export-metrics
```

Export metrics for a specific date range and filename:

```bash theme={null}
firectl billing export-metrics \
  --start-time "2025-01-01" \
  --end-time "2025-01-31" \
  --filename january_metrics.csv
```

## Output format

The exported CSV includes the following columns:

* **email**: Account email
* **start\_time**: Request start timestamp
* **end\_time**: Request end timestamp
* **usage\_type**: Type of usage (e.g., TEXT\_COMPLETION\_INFERENCE\_USAGE)
* **accelerator\_type**: GPU/hardware type used
* **accelerator\_seconds**: Compute time in seconds
* **base\_model\_name**: The model used
* **model\_bucket**: Model category
* **parameter\_count**: Model size
* **prompt\_tokens**: Input tokens
* **completion\_tokens**: Output tokens
* **cached\_prompt\_tokens**: Prompt tokens served from cache (text inference only). Subset of `prompt_tokens`.
* **uncached\_prompt\_tokens**: Prompt tokens not served from cache (text inference only). `prompt_tokens - cached_prompt_tokens`.

<Note>
  Older usage records and non-text usage types may not have a cached/uncached split
  in the underlying data. Exports normalize these rows to `cached_prompt_tokens=0`
  and `uncached_prompt_tokens=prompt_tokens`, so `prompt_tokens =
      cached_prompt_tokens + uncached_prompt_tokens` always holds.
</Note>

### Sample row

```csv theme={null}
email,start_time,end_time,usage_type,accelerator_type,accelerator_seconds,base_model_name,model_bucket,parameter_count,prompt_tokens,completion_tokens,cached_prompt_tokens,uncached_prompt_tokens
user@example.com,2025-10-20 17:16:48 UTC,2025-10-20 17:16:48 UTC,TEXT_COMPLETION_INFERENCE_USAGE,,,accounts/fireworks/models/llama4-maverick-instruct-basic,Llama 4 Maverick Basic,401583781376,803,109,200,603
```

## Automation

Each `firectl billing export-metrics` call supports a maximum 31-day time range.
To export longer historical ranges, run the command in multiple 31-day chunks and
combine the CSV files in your downstream pipeline.

You can automate exports in cron jobs and load the CSV into your internal systems:

```bash theme={null}
# Example: Daily export with dated filename
firectl billing export-metrics \
  --start-time "$(date -v-1d '+%Y-%m-%d')" \
  --end-time "$(date '+%Y-%m-%d')" \
  --filename "billing_$(date '+%Y%m%d').csv"
```

```bash theme={null}
# Example: Backfill 6 months in 31-day chunks
start_date="2025-01-01"
end_date="2025-07-01"
current_start="$start_date"

while [ "$(date -j -f "%Y-%m-%d" "$current_start" "+%s")" -lt "$(date -j -f "%Y-%m-%d" "$end_date" "+%s")" ]; do
  current_end="$(date -j -v+31d -f "%Y-%m-%d" "$current_start" "+%Y-%m-%d")"

  # Clamp the chunk end to the requested end_date
  if [ "$(date -j -f "%Y-%m-%d" "$current_end" "+%s")" -gt "$(date -j -f "%Y-%m-%d" "$end_date" "+%s")" ]; then
    current_end="$end_date"
  fi

  firectl billing export-metrics \
    --start-time "$current_start" \
    --end-time "$current_end" \
    --filename "billing_${current_start}_to_${current_end}.csv"

  current_start="$current_end"
done
```

<Tip>
  Run `firectl billing export-metrics --help` to see all available flags and
  options.
</Tip>

## Coverage

This export includes:

* **Serverless inference**: All serverless API usage
* **On-demand deployments**: Deployment usage (see also [Exporting deployment metrics](/deployments/exporting-metrics) for real-time Prometheus metrics)
* **Fine-tuning jobs**: Fine-tuning compute usage
* **Other services**: All billable Fireworks services

<Note>
  For real-time monitoring of on-demand deployment performance metrics (latency,
  throughput, etc.), use the [Prometheus metrics
  endpoint](/deployments/exporting-metrics) instead.
</Note>

## See also

* [firectl CLI overview](/tools-sdks/firectl/firectl)
* [Exporting deployment metrics](/deployments/exporting-metrics) - Real-time Prometheus metrics for on-demand deployments
* [Account quotas](/guides/quotas_usage/account-quotas) - Spending tiers, monthly spend limits, and account-wide request limits
* [Serverless rate limits](/serverless/rate-limits) - Adaptive serverless TPM bounds


# Usage & Cost Breakdown
Source: https://docs.fireworks.ai/accounts/exporting-usage-and-costs

Break down usage by deployment, model, API key, or custom tags, and read account-level rated costs — via firectl or the billingUsage API

## Overview

Fireworks reports billing along two dimensions:

* **Usage** — metered quantities such as tokens, accelerator-seconds, and audio input seconds. You can break usage down by deployment, model, API key, or custom tags.
* **Cost** — rated dollar amounts. Costs are reported at the account level: a range-wide total, or line items grouped by billing category (serverless, dedicated, training). They aren't broken down by the same dimensions as usage, so per-API-key or per-deployment dollar figures aren't returned today — to approximate them, multiply usage by the published [serverless prices](/serverless/pricing).

For Enterprise accounts, these rated costs match monthly spend alerts: they include usage paid for with credits. Credit grants and purchases are reported separately.

Two tools expose this data:

* **CLI** — [`firectl billing get-usage`](/tools-sdks/firectl/commands/billing-get-usage) shows the account cost total alongside the usage breakdown. Best for ad-hoc queries and shell scripting.
* **HTTP API** — [`GET /v1/accounts/{account_id}/billingUsage`](/api-reference/get-billing-usage) returns the usage breakdown, and its companion [`GET /v1/accounts/{account_id}/billing/summary`](/api-reference/get-billing-summary) returns rated costs. Best for cron jobs, dashboards, and reporting pipelines.

The CLI and `billingUsage` share the same usage response shape and dimensions. Most examples below show the CLI form and the equivalent cURL side by side. Over HTTP, grouping and time-range parameters go on `GET /billingUsage`; **filtering** uses the companion **`POST /billingUsage:query`** endpoint, which carries the filter in a JSON body.

<Note title="Three billing endpoints, what each returns">
  * **`GET /billingUsage`** — metered *quantities* (tokens, accelerator-seconds) grouped by deployment/model/API key/custom tags. No dollars. To **filter** (not just group) over HTTP, use **`POST /billingUsage:query`** with the same request shape in a JSON body (see the filter examples below).
  * **`GET /billing/summary`** — rated *dollar line items* by billing category (serverless, dedicated, training), grouped by your billing config. Optional daily buckets. No per-model/per-key breakdown.
  * **`POST /usageCosts:query`** — rated *dollar subtotals* grouped by caller-supplied dimensions (`HOUR`, `DAY`, `MODEL`, `USER`, `API_KEY`), with pagination and an account-wide `subtotal`. This is the endpoint to use when you need *costs* (not just quantities) broken down by model, user, or API key. Requires account administrator access for `ACCOUNT` scope; `SELF` scope returns only the authenticated user's costs. See [Query usage costs](/api-reference/query-usage-costs).
</Note>

This page complements [Exporting Billing Metrics](/accounts/exporting-billing-metrics): use `export-metrics` for a raw per-event CSV dump, and the workflows here for grouped usage and rated views.

<Note>
  CLI examples require `firectl` 1.7.21 or later. Run `firectl version`, then `firectl upgrade` if needed.
</Note>

## Authentication

For the API, send your Fireworks API key as a bearer token. Any key on the target account works.

```bash theme={null}
export ACCOUNT_ID="<your-account-slug>"
export FIREWORKS_API_KEY="fw_..."
```

For the CLI, run `firectl login` once and `firectl` reads credentials from `~/.fireworks/auth.ini`.

## Basic usage

Get a 30-day account-wide breakdown (defaults to all usage types, grouped by model for serverless and by deployment + accelerator for dedicated):

<Tabs>
  <Tab title="firectl">
    ```bash theme={null}
    firectl billing get-usage \
      --start-time 2026-05-01 \
      --end-time   2026-06-01
    ```

    Add `-o json` for machine-readable output.
  </Tab>

  <Tab title="cURL">
    ```bash theme={null}
    curl -sG "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/billingUsage" \
      -H "Authorization: Bearer ${FIREWORKS_API_KEY}" \
      --data-urlencode "startTime=2026-05-01T00:00:00Z" \
      --data-urlencode "endTime=2026-06-01T00:00:00Z"
    ```
  </Tab>
</Tabs>

## Examples

### Serverless usage by model

<Tabs>
  <Tab title="firectl">
    ```bash theme={null}
    firectl billing get-usage \
      --start-time 2026-05-01 --end-time 2026-06-01 \
      --usage-type serverless \
      --group-by model_name
    ```
  </Tab>

  <Tab title="cURL">
    ```bash theme={null}
    curl -sG "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/billingUsage" \
      -H "Authorization: Bearer ${FIREWORKS_API_KEY}" \
      --data-urlencode "startTime=2026-05-01T00:00:00Z" \
      --data-urlencode "endTime=2026-06-01T00:00:00Z" \
      --data-urlencode "usageType=SERVERLESS" \
      --data-urlencode "groupBy=model_name"
    ```
  </Tab>
</Tabs>

### Serverless usage by API key

Breaks out serverless token consumption per API key. Pass both `api_key_id` (stable internal ID) and `api_key_name` (human-readable label from the console / `firectl api-key create --name`) so the response carries both.

<Tabs>
  <Tab title="firectl">
    ```bash theme={null}
    firectl billing get-usage \
      --start-time 2026-05-01 --end-time 2026-06-01 \
      --usage-type serverless \
      --group-by api_key_id \
      --group-by api_key_name \
      --group-by model_name
    ```
  </Tab>

  <Tab title="cURL">
    ```bash theme={null}
    curl -sG "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/billingUsage" \
      -H "Authorization: Bearer ${FIREWORKS_API_KEY}" \
      --data-urlencode "startTime=2026-05-01T00:00:00Z" \
      --data-urlencode "endTime=2026-06-01T00:00:00Z" \
      --data-urlencode "usageType=SERVERLESS" \
      --data-urlencode "groupBy=api_key_id" \
      --data-urlencode "groupBy=api_key_name" \
      --data-urlencode "groupBy=model_name"
    ```
  </Tab>
</Tabs>

Sample row from the API response:

```json theme={null}
{
  "startTime": "2026-05-28T00:00:00Z",
  "endTime":   "2026-05-29T00:00:00Z",
  "promptTokens":     "1842301",
  "completionTokens": "412980",
  "audioInputSeconds": 0,
  "usageType": "TEXT_COMPLETION_INFERENCE_USAGE",
  "group": {
    "api_key_id":   "key_4nMFyHCSZP4CRKqa",
    "api_key_name": "prod-eng",
    "model_name":   "accounts/fireworks/models/kimi-k2.6"
  }
}
```

<Note>
  Token counts come back as JSON **strings** (int64 over JSON). Cast them with `tonumber` in `jq` or the equivalent in your client before doing arithmetic. The deprecated top-level `apiKeyId` field is only populated when `groupBy=api_key_id` is requested — always read API-key values from the `group` map.
</Note>

### Filter to a specific API key

Multiple values for the same dimension are OR'ed; different dimensions are AND'ed. In `firectl`, repeat `--filter` to OR values; over the API, list them in the dimension's `values` array.

Over HTTP, filter with **`POST /billingUsage:query`**, which takes the same request in a JSON body, where `filter` is a map of dimension → `{ "values": [...] }`.

<Tabs>
  <Tab title="firectl">
    ```bash theme={null}
    firectl billing get-usage \
      --start-time 2026-05-01 --end-time 2026-06-01 \
      --usage-type serverless \
      --group-by model_name \
      --filter api_key_name=prod-eng
    ```
  </Tab>

  <Tab title="cURL">
    ```bash theme={null}
    curl -sS -X POST "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/billingUsage:query" \
      -H "Authorization: Bearer ${FIREWORKS_API_KEY}" \
      -H "Content-Type: application/json" \
      -d '{
        "startTime": "2026-05-01T00:00:00Z",
        "endTime":   "2026-06-01T00:00:00Z",
        "usageType": "SERVERLESS",
        "groupBy":   ["model_name"],
        "filter":    { "api_key_name": { "values": ["prod-eng"] } }
      }'
    ```
  </Tab>
</Tabs>

### Dedicated deployment usage by deployment and GPU type

<Tabs>
  <Tab title="firectl">
    ```bash theme={null}
    firectl billing get-usage \
      --start-time 2026-05-01 --end-time 2026-06-01 \
      --usage-type dedicated-deployment \
      --group-by deployment_name \
      --group-by accelerator_type
    ```
  </Tab>

  <Tab title="cURL">
    ```bash theme={null}
    curl -sG "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/billingUsage" \
      -H "Authorization: Bearer ${FIREWORKS_API_KEY}" \
      --data-urlencode "startTime=2026-05-01T00:00:00Z" \
      --data-urlencode "endTime=2026-06-01T00:00:00Z" \
      --data-urlencode "usageType=DEDICATED_DEPLOYMENT" \
      --data-urlencode "groupBy=deployment_name" \
      --data-urlencode "groupBy=accelerator_type"
    ```
  </Tab>
</Tabs>

### Filter to a single deployment

<Tabs>
  <Tab title="firectl">
    ```bash theme={null}
    firectl billing get-usage \
      --start-time 2026-05-01 --end-time 2026-06-01 \
      --filter deployment_name=accounts/my-account/deployments/my-deployment
    ```
  </Tab>

  <Tab title="cURL">
    ```bash theme={null}
    curl -sS -X POST "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/billingUsage:query" \
      -H "Authorization: Bearer ${FIREWORKS_API_KEY}" \
      -H "Content-Type: application/json" \
      -d '{
        "startTime": "2026-05-01T00:00:00Z",
        "endTime":   "2026-06-01T00:00:00Z",
        "filter":    { "deployment_name": { "values": ["accounts/my-account/deployments/my-deployment"] } }
      }'
    ```
  </Tab>
</Tabs>

### Account-level cost totals only

Get just the rated costs, without the usage rows:

<Tabs>
  <Tab title="firectl">
    ```bash theme={null}
    firectl billing get-usage \
      --start-time 2026-05-01 --end-time 2026-06-01 \
      --account-costs-only
    ```
  </Tab>

  <Tab title="cURL">
    ```bash theme={null}
    curl -sG "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/billing/summary" \
      -H "Authorization: Bearer ${FIREWORKS_API_KEY}" \
      --data-urlencode "startTime=2026-05-01T00:00:00Z" \
      --data-urlencode "endTime=2026-06-01T00:00:00Z" \
      --data-urlencode "granularity=DAILY"
    ```

    The companion [`GET /v1/accounts/{account_id}/billing/summary`](/api-reference/get-billing-summary) endpoint returns rated billing line items. Grouping comes from your billing configuration rather than a caller-supplied `groupBy` or `filter`, so line items follow billing categories (serverless, dedicated, training) instead of arbitrary dimensions. Each `lineItem` carries a `series` and its rated `totalCost`, and `granularity=DAILY` adds a per-day `usageBuckets` breakdown. These are rated line items, so they may differ from the final invoice once credits or adjustments are applied.
  </Tab>
</Tabs>

## Reference

### CLI flags

| Flag                   | Description                                                                        |
| ---------------------- | ---------------------------------------------------------------------------------- |
| `--start-time`         | Start time (inclusive), as `YYYY-MM-DD` or `'YYYY-MM-DD hh:mm:ss'`.                |
| `--end-time`           | End time (exclusive), same formats.                                                |
| `--usage-type`         | `all`, `serverless`, or `dedicated-deployment`. Defaults to all.                   |
| `--group-by`           | Dimension to group by. Repeatable.                                                 |
| `--filter`             | `key=value` filter. Repeatable; repeated values for the same key are OR'ed.        |
| `--timezone`           | IANA timezone for daily aggregation (e.g. `America/Los_Angeles`). Defaults to UTC. |
| `--account-costs-only` | Print only account-level cumulative costs for the range.                           |
| `-o, --output`         | `text` (default) or `json`.                                                        |

Run `firectl billing get-usage --help` for the full list.

### API parameters

Over HTTP, pass dimensions as `groupBy=<dim>` (repeat for multiple). `usageType` takes `SERVERLESS`, `DEDICATED_DEPLOYMENT`, or omitted for all. `timezone` and `startTime`/`endTime` mirror the CLI flags. See [the full API reference](/api-reference/get-billing-usage) for parameter schemas and response types.

To filter over HTTP, send **`POST /v1/accounts/{account_id}/billingUsage:query`** with the same request in a JSON body, where `filter` is a map of dimension → `{ "values": [...] }` (repeated values OR within a dimension; distinct dimensions are AND'ed).

### Grouping dimensions

Valid `--group-by` / `groupBy` and `--filter` / `filter` dimensions depend on the usage type, and both are available over CLI and HTTP (grouping on `GET /billingUsage`, filtering on `POST /billingUsage:query`):

* **Serverless**: `model_name`, `api_key_id`, `api_key_name`, `annotations.team`, `annotations.project`, `annotations.environment`
* **Dedicated deployment**: `deployment_name`, `accelerator_type`, `annotations.team`, `annotations.project`, `annotations.environment`

The `annotations.*` dimensions require an **Enterprise** plan (see [Custom tags](#custom-tags-team--project--environment)); the other dimensions are available to all accounts.

Dedicated-deployment rows also include the deployment's region (`placement`, e.g. `US`, `EUROPE`, `GLOBAL`) and metered `accelerator_seconds`.

## Custom tags (team / project / environment)

<Note>
  Breaking usage down by annotations (`annotations.team` / `annotations.project` / `annotations.environment`) requires an **Enterprise** plan. Grouping or filtering by an annotation dimension without it returns HTTP `400` (`FAILED_PRECONDITION`); breakdowns by model, API key, or deployment remain available to all accounts. Annotation tags are still recorded on your usage regardless of plan, so past usage is immediately available for these breakdowns once an account is on Enterprise.
</Note>

Group by `annotations.team`, `annotations.project`, or `annotations.environment` to split usage by your own labels. The tag source depends on usage type:

* **Dedicated deployments**: set an `annotations` map on the deployment, e.g. `{"team": "search", "project": "x", "environment": "prod"}`.
* **Serverless**: send a per-request header on inference calls:

  ```http theme={null}
  POST /inference/v1/chat/completions HTTP/1.1
  Host: api.fireworks.ai
  Authorization: Bearer fw_...
  Fireworks-Annotations: team=search,project=ranker,environment=prod
  Content-Type: application/json
  ```

  Annotation values are validated server-side. Only the recognized keys — `team`, `project`, `environment` — are stored; any other segment (an unknown key, an empty value like `project=`, or a bare token) is dropped on its own, and the recognized keys in the same header are always preserved.

  In responses these tags appear under the short keys `team` / `project` / `environment` in the `group` map: you group and filter by the `annotations.`-prefixed names, but the response omits the prefix.

## Cookbook: per-API-key reporting recipes

These recipes target the HTTP API, where downstream aggregation in `jq` (or any client) is easiest.

### Aggregate per key, across models

Sums prompt and completion tokens for each API key across every model it called, sorted by prompt volume.

```bash theme={null}
curl -sG "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/billingUsage" \
  -H "Authorization: Bearer ${FIREWORKS_API_KEY}" \
  --data-urlencode "startTime=2026-05-01T00:00:00Z" \
  --data-urlencode "endTime=2026-06-01T00:00:00Z" \
  --data-urlencode "usageType=SERVERLESS" \
  --data-urlencode "groupBy=api_key_id" \
  --data-urlencode "groupBy=api_key_name" \
  --data-urlencode "groupBy=model_name" \
  | jq '.serverlessCosts
        | group_by(.group.api_key_id)
        | map({
            api_key_id:        .[0].group.api_key_id,
            api_key_name:      .[0].group.api_key_name,
            models:            (map(.group.model_name) | unique),
            prompt_tokens:     ([.[].promptTokens     | tonumber] | add),
            completion_tokens: ([.[].completionTokens | tonumber] | add)
          })
        | sort_by(-.prompt_tokens)'
```

### Group by model, then by key (cost-by-tool view)

If reporting starts from "how much did each model cost me, and which keys drove that", flip the nesting:

```bash theme={null}
curl -sG "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/billingUsage" \
  -H "Authorization: Bearer ${FIREWORKS_API_KEY}" \
  --data-urlencode "startTime=2026-05-01T00:00:00Z" \
  --data-urlencode "endTime=2026-06-01T00:00:00Z" \
  --data-urlencode "usageType=SERVERLESS" \
  --data-urlencode "groupBy=api_key_id" \
  --data-urlencode "groupBy=api_key_name" \
  --data-urlencode "groupBy=model_name" \
  | jq '.serverlessCosts
        | group_by(.group.model_name)
        | map({
            model: .[0].group.model_name,
            api_keys: (
              group_by(.group.api_key_id)
              | map({
                  api_key_id:        .[0].group.api_key_id,
                  api_key_name:      .[0].group.api_key_name,
                  prompt_tokens:     ([.[].promptTokens     | tonumber] | add),
                  completion_tokens: ([.[].completionTokens | tonumber] | add)
                })
              | sort_by(-.prompt_tokens)
            )
          })
        | sort_by(.model)'
```

Multiply the token totals by the published [serverless prices](/serverless/pricing) to convert to dollars for chargeback.

### Backfill more than 31 days

The endpoint caps each request at a 31-day window. To pull a longer history, loop month-by-month:

```bash theme={null}
start_date="2026-01-01"
end_date="2026-06-01"
current="$start_date"

while [ "$(date -u -d "$current" '+%s')" -lt "$(date -u -d "$end_date" '+%s')" ]; do
  next="$(date -u -d "$current +30 days" '+%Y-%m-%d')"
  if [ "$(date -u -d "$next" '+%s')" -gt "$(date -u -d "$end_date" '+%s')" ]; then
    next="$end_date"
  fi

  curl -sG "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/billingUsage" \
    -H "Authorization: Bearer ${FIREWORKS_API_KEY}" \
    --data-urlencode "startTime=${current}T00:00:00Z" \
    --data-urlencode "endTime=${next}T00:00:00Z" \
    --data-urlencode "usageType=SERVERLESS" \
    --data-urlencode "groupBy=api_key_id" \
    --data-urlencode "groupBy=api_key_name" \
    > "usage_${current}_to_${next}.json"

  current="$next"
done
```

## Granularity and freshness

* Usage is aggregated into **daily** buckets (`--timezone` / `timezone=` sets the day boundary). There are no sub-daily buckets.
* Responses are cached for several minutes — fine for cron jobs and dashboards, not for real-time.

## Coverage caveats

* **Tokens, not dollars.** The endpoint returns metered quantities (`promptTokens`, `completionTokens`, `accelerator_seconds`, `audioInputSeconds`). Multiply by the [serverless prices](/serverless/pricing) for cost, or use `--account-costs-only` for account-level dollar totals.
* **Inference types covered today**: text completion / chat completion and audio inference. Embeddings and image generation aren't yet reflected in `billingUsage` responses; coverage will expand in subsequent releases.
* **Dedicated deployments** are attributed at the deployment level, not by API key. Use `usageType=DEDICATED_DEPLOYMENT` with `groupBy=deployment_name` for that breakdown.

<Tip>
  Run `firectl billing get-usage --help` to see all available CLI flags and options.
</Tip>

## See also

* [`firectl billing get-usage`](/tools-sdks/firectl/commands/billing-get-usage) - CLI command reference
* [`GET /v1/accounts/{account_id}/billingUsage`](/api-reference/get-billing-usage) - HTTP API reference
* [`GET /v1/accounts/{account_id}/billing/summary`](/api-reference/get-billing-summary) - Rated dollar costs by billing category, with optional daily breakdown
* [`POST /v1/accounts/{account_id}/usageCosts:query`](/api-reference/query-usage-costs) - Rated dollar subtotals grouped by hour/day/model/user/API key
* [Exporting Billing Metrics](/accounts/exporting-billing-metrics) - Raw per-event billing CSV export
* [Account quotas](/guides/quotas_usage/account-quotas) - Spending tiers and monthly spend limits


# Service Accounts
Source: https://docs.fireworks.ai/accounts/service-accounts

How to manage and use service accounts in Fireworks

Service accounts in Fireworks allow applications, scripts, and automated systems to authenticate and perform actions securely—without relying on human credentials. They are ideal for CI/CD pipelines, backend services, and automated workflows. Service Accounts let you avoid shared credentials and easily distinguish between what automated systems did vs humans in audit logs.

Service accounts can take actions using an API key, like creating deployments, running models or creating datasets (see [API reference](https://fireworks.ai/docs/api-reference/introduction)). Service accounts cannot login through the web interface or use OIDC tokens.

To manage service accounts via the Fireworks web UI visit [app.fireworks.ai/account/users](https://app.fireworks.ai/account/users).

## Creating a Service Account

Using our firectl you can create service accounts

```bash theme={null}
firectl user create --user-id "my-service-account" --service-account
```

## Creating an API Key for a Service Account

Using firectl you can create an API key on behalf of a service account:

```bash theme={null}
firectl api-key create --service-account "my-service-account"
```

## Roles

You can assign a role when creating a service account using the `--role` flag:

```bash theme={null}
firectl user create --user-id "my-service-account" --service-account --role=contributor
```

If not specified, the default service account role is `user`.

To change the role of an existing service account, use the update command:

```bash theme={null}
firectl user update my-service-account --role=inference-user
```

See [Managing users](/accounts/users) for available roles.

## Listing Service Accounts

To list all service accounts in your account:

```bash theme={null}
firectl user list --filter 'service_account=true'
```

## Billing

* Service accounts count toward the same account quotas and limits assigned to the account
* Usage is tracked by the account, not individual user vs service account

## Auditing

In audit logs users are referenced by their email id's. Service accounts are referenced by `my-service-account@my-account.sa.fireworks.ai`.


# Custom SSO
Source: https://docs.fireworks.ai/accounts/sso

Set up custom Single Sign-On (SSO) authentication and SCIM user provisioning for Fireworks AI

Fireworks uses single sign-on (SSO) as the primary mechanism to authenticate with the platform.
By default, Fireworks supports Google SSO.

If you have an enterprise account, Fireworks supports bringing your own identity provider using:

* OpenID Connect (OIDC) provider
* SAML 2.0 provider
* Optionally, the work-email domain your users sign in with, so they can start SSO by entering their email instead of your account ID

<Info>
  Coordinate with your Fireworks AI representative to enable the integration.
</Info>

## OpenID Connect (OIDC) provider

<Steps>
  <Step title="Create OIDC client application">
    Create an OIDC client application in your identity provider, e.g. Okta.
  </Step>

  <Step title="Configure client">
    Ensure the client is configured for "code authorization" of the "web" type (i.e. with a client\_secret).
  </Step>

  <Step title="Set redirect URL">
    Set the client's "allowed redirect URL" to the URL provided by Fireworks. It looks like:

    ```
    https://fireworks-<your-company-name>.auth.us-west-2.amazoncognito.com/oauth2/idpresponse
    ```
  </Step>

  <Step title="Note down client details">
    Note down the `issuer`, `client_id`, and `client_secret` for the newly created client. You will need to provide this to your Fireworks.ai representative to complete your account set up.
  </Step>
</Steps>

## SAML 2.0 provider

<Steps>
  <Step title="Create SAML 2.0 application">
    Create a SAML 2.0 application in your identity provider, e.g. [Okta](https://help.okta.com/en-us/Content/Topics/Apps/Apps_App_Integration_Wizard_SAML.htm).
  </Step>

  <Step title="Set SSO URL">
    Set the SSO URL to the URL provided by Fireworks. It looks like:

    ```
    https://fireworks-<your-company-name>.auth.us-west-2.amazoncognito.com/saml2/idpresponse
    ```
  </Step>

  <Step title="Configure Audience URI">
    Configure the Audience URI (SP Entity ID) as provided by Fireworks. It looks like:

    ```
    urn:amazon:cognito:sp:<some-unique-identifier>
    ```
  </Step>

  <Step title="Create Attribute Statement">
    Create an Attribute Statement with the name:

    ```
    http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress
    ```

    and the value `user.email`

    <Note>
      **Okta:** After saving the app, open **Sign On** → **Attribute Statements (SAML)** → expand **Show legacy configuration** → add the attribute statement there. Okta no longer configures this during app creation.
    </Note>
  </Step>

  <Step title="Keep default settings">
    Leave the rest of the settings as defaults.
  </Step>

  <Step title="Note down metadata URL">
    Note down the "metadata url" for your newly created application. You will need to provide this to your Fireworks AI representative to complete your account set up.
  </Step>
</Steps>

## Just-In-Time (JIT) user provisioning

JIT user provisioning automatically creates user accounts when they sign in through SSO for the first time. When enabled, users who authenticate through your identity provider are automatically added to your Fireworks account without requiring manual user creation.

To enable JIT user provisioning, use the [`--enable-jit-user-provisioning`](/tools-sdks/firectl/commands/identity-provider-create) flag when creating your identity provider with firectl.

## SCIM user provisioning

System for Cross-domain Identity Management (SCIM) provisioning synchronizes the user lifecycle between your identity provider and Fireworks. Users assigned to Fireworks in your directory are added to your Fireworks account, and users are removed when they are deactivated or unassigned in the directory.

SCIM provisioning is available for enterprise accounts and works with supported directory providers, including Okta, Microsoft Entra ID, and Google Workspace. Fireworks uses [WorkOS Directory Sync](https://workos.com/docs/directory-sync) to connect to your directory.

<Note>
  SCIM manages provisioning only. Users continue to authenticate through your
  existing OIDC or SAML SSO integration.
</Note>

### Set up SCIM provisioning

<Steps>
  <Step title="Configure custom SSO">
    Complete the OIDC or SAML setup above. Custom SSO must be configured before
    you can enable SCIM provisioning.
  </Step>

  <Step title="Request SCIM enablement">
    Contact your Fireworks AI representative. Fireworks will enable Directory
    Sync for your account and provide a secure setup link.
  </Step>

  <Step title="Connect your directory">
    Open the setup link, select your directory provider, and follow the
    provider-specific instructions to authorize the connection.
  </Step>

  <Step title="Assign users">
    In your identity provider, assign the users who should have access to
    Fireworks. Confirm that they appear in the [Users
    page](https://app.fireworks.ai/account/users).
  </Step>
</Steps>

<Warning>
  SCIM group synchronization and group-to-role mappings are not currently
  supported. Provisioned users receive the `User` role by default.
</Warning>

We recommend disabling JIT provisioning when SCIM is enabled so that your directory remains the source of truth for account membership. SSO enforcement is also recommended to prevent access outside your configured identity provider.

## Enforce SSO

When SSO enforcement is enabled, account access is restricted to users with approved tenant domains only. Users with matching domains must authenticate via the identity provider, and users with other domains are blocked.

To enforce SSO, use the [`--enforce-sso`](/tools-sdks/firectl/commands/identity-provider-create) flag when creating your identity provider with firectl, or toggle "Enforce SSO for all users" in the Fireworks console.

## Troubleshooting

### Invalid samlResponse or relayState from identity provider

This error occurs if you are trying to use identity provider (IdP) initiated login. Fireworks currently only supports
service provider (SP) initiated login.

See [Understanding SAML](https://developer.okta.com/docs/concepts/saml/#understand-sp-initiated-sign-in-flow) for an
in-depth explanation.

### Required String parameter 'RelayState' is not present

See above.


# Managing users
Source: https://docs.fireworks.ai/accounts/users

Add, delete, and manage roles for users in your Fireworks account

See the concepts [page](/getting-started/concepts#account) for definitions of accounts and users.

## User roles

Each user in an account is assigned a role that determines their level of access:

| Role               | Description                                                                                                             |
| :----------------- | :---------------------------------------------------------------------------------------------------------------------- |
| **Admin**          | Full administrative control over resources, users, and access. Can manage all account settings and add or remove users. |
| **User** (default) | Can manage all resources, including those owned by others, but cannot manage users or access settings.                  |
| **Contributor**    | Can run inference on any resource and create and manage their own resources. Cannot modify resources owned by others.   |
| **Inference User** | Can view all resources and run inference, but cannot create or modify resources.                                        |

<Note>
  The `contributor` and `inference-user` roles are newer roles that provide more granular access control. Contact Fireworks support if you need these roles enabled for your account.
</Note>

#### Resource management

| Permission                                                             | Inference User | Contributor | User | Admin |
| :--------------------------------------------------------------------- | :------------: | :---------: | :--: | :---: |
| Execute inference on any deployment                                    |        ✅       |      ✅      |   ✅  |   ✅   |
| View all resources (deployments, models, fine tuning jobs, datasets)   |        ✅       |      ✅      |   ✅  |   ✅   |
| Create new resources (deployments, models, fine tuning jobs, datasets) |        ❌       |      ✅      |   ✅  |   ✅   |
| Manage their own resources (edit/delete)                               |        ❌       |      ✅      |   ✅  |   ✅   |
| Manage resources owned by others (edit/delete)                         |        ❌       |      ❌      |   ✅  |   ✅   |

#### API key & account management

| Permission                                       | Inference User | Contributor | User | Admin |
| :----------------------------------------------- | :------------: | :---------: | :--: | :---: |
| Manage self-owned API keys (create/delete)       |        ✅       |      ✅      |   ✅  |   ✅   |
| View all users and service accounts              |        ✅       |      ✅      |   ✅  |   ✅   |
| Create service account API keys                  |        ❌       |      ❌      |   ❌  |   ✅   |
| Delete other users and service accounts API keys |        ❌       |      ❌      |   ❌  |   ✅   |
| Add/modify/delete users and their access         |        ❌       |      ❌      |   ❌  |   ✅   |

## Adding users

To add a new user to your Fireworks account, run the following command. If the email for the new user is already associated with a Fireworks account, they will have the option to freely switch between your account and their existing account(s). You can also add users in the Fireworks web UI at [https://app.fireworks.ai/account/users](https://app.fireworks.ai/account/users).

```bash theme={null}
firectl user create --email="alice@example.com"
```

To create another admin user, pass the `--role=admin` flag:

```bash theme={null}
firectl user create --email="alice@example.com" --role=admin
```

## Updating a user's role

To update a user's role, run

```bash theme={null}
firectl user update <USER_ID> --role=<ROLE>
```

Where `<ROLE>` is one of: `admin`, `user`, `contributor`, or `inference-user`.

## Deleting users

You can remove a user from your account by running:

```bash theme={null}
firectl user delete <USER_ID>
```


# Create a Message
Source: https://docs.fireworks.ai/api-reference/anthropic-messages

post /v1/messages
**Anthropic-compatible endpoint.**

Send a structured list of input messages with text and/or image content, and the model will generate the next message in the conversation.

The Messages API can be used for either single queries or stateless multi-turn conversations.

**Fireworks Quickstarts:**
- [Serverless Quickstart](/getting-started/quickstart)
- [Deployments Quickstart](/getting-started/ondemand-quickstart)

<Note>
  This endpoint provides an Anthropic-compatible Messages API surface on Fireworks.

  For setup, supported features, and known differences, see [Anthropic compatibility](/tools-sdks/anthropic-compatibility).
</Note>


# Cancel Reinforcement Fine-tuning Job
Source: https://docs.fireworks.ai/api-reference/cancel-reinforcement-fine-tuning-job

post /v1/accounts/{account_id}/reinforcementFineTuningJobs/{reinforcement_fine_tuning_job_id}:cancel



# Create API Key
Source: https://docs.fireworks.ai/api-reference/create-api-key

post /v1/accounts/{account_id}/users/{user_id}/apiKeys



# Create Batch Inference Job
Source: https://docs.fireworks.ai/api-reference/create-batch-inference-job

post /v1/accounts/{account_id}/batchInferenceJobs



# Create Dataset
Source: https://docs.fireworks.ai/api-reference/create-dataset

post /v1/accounts/{account_id}/datasets



# Load LoRA
Source: https://docs.fireworks.ai/api-reference/create-deployed-model

post /v1/accounts/{account_id}/deployedModels



# Create Deployment
Source: https://docs.fireworks.ai/api-reference/create-deployment

post /v1/accounts/{account_id}/deployments

## Creating a deployment with a deployment shape

[Deployment shapes](/guides/ondemand-deployments#deployment-shapes) are pre-configured templates optimized for speed, cost, or efficiency. To create a deployment with a specific shape, pass the `deploymentShape` field in the request body along with `baseModel`.

Use the [List Deployment Shape Versions](/api-reference/list-deployment-shape-versions) endpoint to find available shapes for your model.

```bash theme={null}
curl -X POST "https://api.fireworks.ai/v1/accounts/YOUR_ACCOUNT_ID/deployments" \
  -H "Authorization: Bearer $FIREWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "baseModel": "accounts/fireworks/models/gpt-oss-120b",
    "deploymentShape": "accounts/fireworks/deploymentShapes/gpt-oss-120b-minimal",
    "minReplicaCount": 0,
    "maxReplicaCount": 1
  }'
```

<Note>
  When using a deployment shape, you do not need to specify `activeModelVersion` or `targetModelVersion` — the shape provides the necessary configuration.
</Note>


# Create dpo job
Source: https://docs.fireworks.ai/api-reference/create-dpo-job

post /v1/accounts/{account_id}/dpoJobs



# Create Evaluation Job
Source: https://docs.fireworks.ai/api-reference/create-evaluation-job

post /v1/accounts/{account_id}/evaluationJobs



# Create Evaluator
Source: https://docs.fireworks.ai/api-reference/create-evaluator

post /v1/accounts/{account_id}/evaluatorsV2
Creates a custom evaluator for scoring model outputs. Evaluators use the
[Eval Protocol](https://evalprotocol.io) to define test cases, run model
inference, and score responses. They are used with evaluation jobs and
Reinforcement Fine-Tuning (RFT).

## Source Code Requirements

Your project should contain:
- `requirements.txt` - Python dependencies for your evaluator
- `test_*.py` - Pytest test file(s) with
  [`@evaluation_test`](https://evalprotocol.io/reference/evaluation-test)
  decorated functions
- Any additional code/modules your evaluator needs

## Workflow

**Recommended:** Use the [`ep upload`](https://evalprotocol.io/reference/cli#ep-upload)
CLI command to handle all these steps automatically.

If using the API directly:

1. Call this endpoint to create the evaluator resource
2. Package your source directory as a `.tar.gz` (respecting `.gitignore`)
3. Call [Get Evaluator Upload Endpoint](/api-reference/get-evaluator-upload-endpoint) to get a signed upload URL
4. `PUT` the tar.gz file to the signed URL
5. Call [Validate Evaluator Upload](/api-reference/validate-evaluator-upload) to trigger server-side validation
6. Poll [Get Evaluator](/api-reference/get-evaluator) until ready

Once active, reference the evaluator in [Create Evaluation Job](/api-reference/create-evaluation-job) or [Create Reinforcement Fine-tuning Job](/api-reference/create-reinforcement-fine-tuning-job).



# Create Model
Source: https://docs.fireworks.ai/api-reference/create-model

post /v1/accounts/{account_id}/models



# Create Reinforcement Fine-tuning Job
Source: https://docs.fireworks.ai/api-reference/create-reinforcement-fine-tuning-job

post /v1/accounts/{account_id}/reinforcementFineTuningJobs



# Create Reinforcement Fine-tuning Step
Source: https://docs.fireworks.ai/api-reference/create-reinforcement-fine-tuning-step

post /v1/accounts/{account_id}/rlorTrainerJobs



# Create Router
Source: https://docs.fireworks.ai/api-reference/create-router

post /v1/accounts/{account_id}/routers



# Create secret
Source: https://docs.fireworks.ai/api-reference/create-secret

post /v1/accounts/{account_id}/secrets



# Create Supervised Fine-tuning Job
Source: https://docs.fireworks.ai/api-reference/create-supervised-fine-tuning-job

post /v1/accounts/{account_id}/supervisedFineTuningJobs

## Learning rate scheduler

Supervised fine-tuning jobs accept an optional `lrScheduler` object on the request body. Set **exactly one** of `constant`, `linear`, or `cosine`. When omitted, the trainer uses a constant learning rate after warmup.

Configure warmup separately with `learningRateWarmupSteps` (not inside `lrScheduler`).

| Schedule   | Object shape                                             | Notes                                               |
| ---------- | -------------------------------------------------------- | --------------------------------------------------- |
| `constant` | `{ "constant": {} }`                                     | Flat LR after warmup                                |
| `linear`   | `{ "linear": { "minLrRatio": 0.1, "decayRatio": 0.8 } }` | Linear decay toward `learningRate * minLrRatio`     |
| `cosine`   | `{ "cosine": { "minLrRatio": 0.1, "decayRatio": 0.8 } }` | Cosine annealing toward `learningRate * minLrRatio` |

For `linear` and `cosine`:

* `minLrRatio` — floor LR as a fraction of `learningRate` (0.0–1.0).
* `decayRatio` — fraction of total training steps over which to decay. Omit or set `0` to decay over the full run.

`linear` and `cosine` require a base model on the **Training V2** path. V1-routed models only support a constant schedule.

### Example: cosine schedule

```json theme={null}
{
  "supervisedFineTuningJob": {
    "baseModel": "accounts/my-account/models/qwen3-8b",
    "dataset": "accounts/my-account/datasets/my-data",
    "outputModel": "accounts/my-account/models/my-tuned-model",
    "learningRate": 0.0001,
    "learningRateWarmupSteps": 10,
    "lrScheduler": {
      "cosine": {
        "minLrRatio": 0.1,
        "decayRatio": 0.8
      }
    }
  }
}
```

See also [Fine-tuning models](/fine-tuning/fine-tuning-models) for CLI equivalents (`--learning-rate-scheduler`, `--learning-rate-min-lr-ratio`, `--learning-rate-decay-ratio`).


# Create User
Source: https://docs.fireworks.ai/api-reference/create-user

post /v1/accounts/{account_id}/users



# Create embeddings
Source: https://docs.fireworks.ai/api-reference/creates-an-embedding-vector-representing-the-input-text

post /embeddings



# Delete API Key
Source: https://docs.fireworks.ai/api-reference/delete-api-key

post /v1/accounts/{account_id}/users/{user_id}/apiKeys:delete



# Delete Batch Inference Job
Source: https://docs.fireworks.ai/api-reference/delete-batch-inference-job

delete /v1/accounts/{account_id}/batchInferenceJobs/{batch_inference_job_id}



# Delete Dataset
Source: https://docs.fireworks.ai/api-reference/delete-dataset

delete /v1/accounts/{account_id}/datasets/{dataset_id}



# Unload LoRA
Source: https://docs.fireworks.ai/api-reference/delete-deployed-model

delete /v1/accounts/{account_id}/deployedModels/{deployed_model_id}



# Delete Deployment
Source: https://docs.fireworks.ai/api-reference/delete-deployment

delete /v1/accounts/{account_id}/deployments/{deployment_id}



# Delete dpo job
Source: https://docs.fireworks.ai/api-reference/delete-dpo-job

delete /v1/accounts/{account_id}/dpoJobs/{dpo_job_id}



# Delete Evaluation Job
Source: https://docs.fireworks.ai/api-reference/delete-evaluation-job

delete /v1/accounts/{account_id}/evaluationJobs/{evaluation_job_id}



# Delete Evaluator
Source: https://docs.fireworks.ai/api-reference/delete-evaluator

delete /v1/accounts/{account_id}/evaluators/{evaluator_id}
Deletes an evaluator and its associated versions and build artifacts.



# Delete Model
Source: https://docs.fireworks.ai/api-reference/delete-model

delete /v1/accounts/{account_id}/models/{model_id}



# Delete Reinforcement Fine-tuning Job
Source: https://docs.fireworks.ai/api-reference/delete-reinforcement-fine-tuning-job

delete /v1/accounts/{account_id}/reinforcementFineTuningJobs/{reinforcement_fine_tuning_job_id}



# Delete Reinforcement Fine-tuning Step
Source: https://docs.fireworks.ai/api-reference/delete-reinforcement-fine-tuning-step

delete /v1/accounts/{account_id}/rlorTrainerJobs/{rlor_trainer_job_id}



# Delete Response
Source: https://docs.fireworks.ai/api-reference/delete-response

delete /v1/responses/{response_id}
Deletes a model response by its ID. Once deleted, the response data will be gone immediately and permanently.

The response cannot be recovered and any conversations that reference this response ID will no longer be able to access it.



# Delete Router
Source: https://docs.fireworks.ai/api-reference/delete-router

delete /v1/accounts/{account_id}/routers/{router_id}



# Delete secret
Source: https://docs.fireworks.ai/api-reference/delete-secret

delete /v1/accounts/{account_id}/secrets/{secret_id}



# Delete Supervised Fine-tuning Job
Source: https://docs.fireworks.ai/api-reference/delete-supervised-fine-tuning-job

delete /v1/accounts/{account_id}/supervisedFineTuningJobs/{supervised_fine_tuning_job_id}



# Execute reinforcement fine tuning step
Source: https://docs.fireworks.ai/api-reference/execute-reinforcement-fine-tuning-step

post /v1/accounts/{account_id}/rlorTrainerJobs/{rlor_trainer_job_id}:executeTrainStep



# Get Account
Source: https://docs.fireworks.ai/api-reference/get-account

get /v1/accounts/{account_id}



# Get Batch Inference Job
Source: https://docs.fireworks.ai/api-reference/get-batch-inference-job

get /v1/accounts/{account_id}/batchInferenceJobs/{batch_inference_job_id}



# Get billing summary information for an account
Source: https://docs.fireworks.ai/api-reference/get-billing-summary

get /v1/accounts/{account_id}/billing/summary



# Get Account Usage
Source: https://docs.fireworks.ai/api-reference/get-billing-usage

get /v1/accounts/{account_id}/billingUsage



# Get Dataset
Source: https://docs.fireworks.ai/api-reference/get-dataset

get /v1/accounts/{account_id}/datasets/{dataset_id}



# Get Dataset Download Endpoint
Source: https://docs.fireworks.ai/api-reference/get-dataset-download-endpoint

get /v1/accounts/{account_id}/datasets/{dataset_id}:getDownloadEndpoint



# Get Dataset Upload Endpoint
Source: https://docs.fireworks.ai/api-reference/get-dataset-upload-endpoint

post /v1/accounts/{account_id}/datasets/{dataset_id}:getUploadEndpoint



# Get LoRA
Source: https://docs.fireworks.ai/api-reference/get-deployed-model

get /v1/accounts/{account_id}/deployedModels/{deployed_model_id}



# Get Deployment
Source: https://docs.fireworks.ai/api-reference/get-deployment

get /v1/accounts/{account_id}/deployments/{deployment_id}



# Get Deployment Shape
Source: https://docs.fireworks.ai/api-reference/get-deployment-shape

get /v1/accounts/{account_id}/deploymentShapes/{deployment_shape_id}



# Get Deployment Shape Version
Source: https://docs.fireworks.ai/api-reference/get-deployment-shape-version

get /v1/accounts/{account_id}/deploymentShapes/{deployment_shape_id}/versions/{version_id}



# Get dpo job
Source: https://docs.fireworks.ai/api-reference/get-dpo-job

get /v1/accounts/{account_id}/dpoJobs/{dpo_job_id}



# Get dpo job metrics file endpoint
Source: https://docs.fireworks.ai/api-reference/get-dpo-job-metrics-file-endpoint

get /v1/accounts/{account_id}/dpoJobs/{dpo_job_id}:getMetricsFileEndpoint



# Get Evaluation Job
Source: https://docs.fireworks.ai/api-reference/get-evaluation-job

get /v1/accounts/{account_id}/evaluationJobs/{evaluation_job_id}



# Get Evaluation Job execution logs (stream log endpoint + tracing IDs).
Source: https://docs.fireworks.ai/api-reference/get-evaluation-job-log-endpoint

get /v1/accounts/{account_id}/evaluationJobs/{evaluation_job_id}:getExecutionLogEndpoint



# Get Evaluator
Source: https://docs.fireworks.ai/api-reference/get-evaluator

get /v1/accounts/{account_id}/evaluators/{evaluator_id}
Retrieves an evaluator by name. Use this to monitor build progress after
creation (**step 6** in the [Create Evaluator](/api-reference/create-evaluator) workflow).

Possible states:

- `BUILDING` - Environment is being prepared
- `ACTIVE` - Evaluator is ready to use
- `BUILD_FAILED` - Check build logs via [Get Evaluator Build Log Endpoint](/api-reference/get-evaluator-build-log-endpoint)



# Get Evaluator Build Log Endpoint
Source: https://docs.fireworks.ai/api-reference/get-evaluator-build-log-endpoint

get /v1/accounts/{account_id}/evaluators/{evaluator_id}:getBuildLogEndpoint
Returns a signed URL to download the evaluator's build logs. Useful for
debugging `BUILD_FAILED` state.



# Get Evaluator Source Code Endpoint
Source: https://docs.fireworks.ai/api-reference/get-evaluator-source-code-endpoint

get /v1/accounts/{account_id}/evaluators/{evaluator_id}:getSourceCodeSignedUrl
Returns a signed URL to download the evaluator's source code archive.
Useful for debugging or reviewing the uploaded code.



# Get Evaluator Upload Endpoint
Source: https://docs.fireworks.ai/api-reference/get-evaluator-upload-endpoint

post /v1/accounts/{account_id}/evaluators/{evaluator_id}:getUploadEndpoint
Returns signed URLs for uploading evaluator source code (**step 3** in the
[Create Evaluator](/api-reference/create-evaluator) workflow). After receiving
the signed URL, upload your `.tar.gz` archive using HTTP `PUT` with
`Content-Type: application/octet-stream` header.



# Get Model
Source: https://docs.fireworks.ai/api-reference/get-model

get /v1/accounts/{account_id}/models/{model_id}



# Get Model Download Endpoint
Source: https://docs.fireworks.ai/api-reference/get-model-download-endpoint

get /v1/accounts/{account_id}/models/{model_id}:getDownloadEndpoint



# Get Model Upload Endpoint
Source: https://docs.fireworks.ai/api-reference/get-model-upload-endpoint

post /v1/accounts/{account_id}/models/{model_id}:getUploadEndpoint



# Get Quota
Source: https://docs.fireworks.ai/api-reference/get-quota

get /v1/accounts/{account_id}/quotas/{quota_id}
Gets a single quota by resource name.



# Get Reinforcement Fine-tuning Job
Source: https://docs.fireworks.ai/api-reference/get-reinforcement-fine-tuning-job

get /v1/accounts/{account_id}/reinforcementFineTuningJobs/{reinforcement_fine_tuning_job_id}



# Get Reinforcement Fine-tuning Step
Source: https://docs.fireworks.ai/api-reference/get-reinforcement-fine-tuning-step

get /v1/accounts/{account_id}/rlorTrainerJobs/{rlor_trainer_job_id}



# Get Response
Source: https://docs.fireworks.ai/api-reference/get-response

get /v1/responses/{response_id}



# CRUD APIs for routers.
Get Router
Source: https://docs.fireworks.ai/api-reference/get-router

get /v1/accounts/{account_id}/routers/{router_id}



# Get Secret
Source: https://docs.fireworks.ai/api-reference/get-secret

get /v1/accounts/{account_id}/secrets/{secret_id}
Retrieves a secret by name. Note that the `value` field is not returned in the response for security reasons. Only the `name` and `key_name` fields are included.



# Get Supervised Fine-tuning Job
Source: https://docs.fireworks.ai/api-reference/get-supervised-fine-tuning-job

get /v1/accounts/{account_id}/supervisedFineTuningJobs/{supervised_fine_tuning_job_id}



# Get User
Source: https://docs.fireworks.ai/api-reference/get-user

get /v1/accounts/{account_id}/users/{user_id}



# Introduction
Source: https://docs.fireworks.ai/api-reference/introduction



Fireworks AI REST API enables you to interact with various language, image and embedding models using an API Key. It also lets you automate management of models, deployments, datasets, and more.

## Authentication

All requests made to the Fireworks AI REST API must include an `Authorization` header with a valid `Bearer` token using your API key, along with the `Content-Type: application/json` header.

### Getting your API key

You can obtain an API key by:

* Using the [`firectl api-key create`](/tools-sdks/firectl/commands/api-key-create) command
* Generating one through the [Fireworks AI dashboard](https://app.fireworks.ai/settings/users/api-keys)

### Request headers

Include the following headers in your REST API requests:

```json theme={null}
authorization: Bearer <API_KEY>
content-type: application/json
```

## Account management APIs

In addition to inference and deployment APIs, Fireworks exposes account-scoped
quota endpoints.

* [List Quotas](/api-reference/list-quotas)
* [Get Quota](/api-reference/get-quota)
* [Update Quota](/api-reference/update-quota)


# List Accounts
Source: https://docs.fireworks.ai/api-reference/list-accounts

get /v1/accounts



# List API Keys
Source: https://docs.fireworks.ai/api-reference/list-api-keys

get /v1/accounts/{account_id}/users/{user_id}/apiKeys

List API keys for the user named in `user_id`. Admins can pass `-` as `user_id` to list keys for all users and service accounts in the account (equivalent to [`firectl api-key list --all-users`](/tools-sdks/firectl/commands/api-key-list)).

```bash theme={null}
curl -s "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/users/-/apiKeys" \
  -H "Authorization: Bearer ${FIREWORKS_API_KEY}"
```


# List Batch Inference Jobs
Source: https://docs.fireworks.ai/api-reference/list-batch-inference-jobs

get /v1/accounts/{account_id}/batchInferenceJobs



# List Datasets
Source: https://docs.fireworks.ai/api-reference/list-datasets

get /v1/accounts/{account_id}/datasets



# List LoRAs
Source: https://docs.fireworks.ai/api-reference/list-deployed-models

get /v1/accounts/{account_id}/deployedModels



# List Deployment Shapes Versions
Source: https://docs.fireworks.ai/api-reference/list-deployment-shape-versions

get /v1/accounts/{account_id}/deploymentShapes/{deployment_shape_id}/versions

Use this endpoint to query available deployment shape versions for a given model. Use `-` as a wildcard for both `account_id` and `deployment_shape_id` to search across all accounts and shapes.

## Example: List shapes for a model

To list validated deployment shapes for a specific model, use the `filter` parameter with `snapshot.base_model` and `latest_validated=true`:

```bash theme={null}
curl -s "https://api.fireworks.ai/v1/accounts/-/deploymentShapes/-/versions?filter=snapshot.base_model%3D%22accounts%2Ffireworks%2Fmodels%2Fgpt-oss-120b%22%20AND%20latest_validated%3Dtrue&order_by=create_time%20desc" \
  -H "Authorization: Bearer $FIREWORKS_API_KEY" | jq .
```

### Filter syntax

The `filter` parameter uses [AIP-160 filtering](https://google.aip.dev/160). Common patterns:

| Filter                                                       | Description                                            |
| ------------------------------------------------------------ | ------------------------------------------------------ |
| `snapshot.base_model="accounts/fireworks/models/MODEL_NAME"` | Filter by base model                                   |
| `latest_validated=true`                                      | Only return the latest validated version of each shape |

Combine multiple conditions with `AND`:

```
snapshot.base_model="accounts/fireworks/models/MODEL_NAME" AND latest_validated=true
```

<Note>
  Remember to URL-encode the filter value when using curl directly. `=` becomes `%3D`, `"` becomes `%22`, and `/` becomes `%2F`.
</Note>


# List Deployments
Source: https://docs.fireworks.ai/api-reference/list-deployments

get /v1/accounts/{account_id}/deployments



# List dpo jobs
Source: https://docs.fireworks.ai/api-reference/list-dpo-jobs

get /v1/accounts/{account_id}/dpoJobs



# List Evaluation Jobs
Source: https://docs.fireworks.ai/api-reference/list-evaluation-jobs

get /v1/accounts/{account_id}/evaluationJobs



# List Evaluators
Source: https://docs.fireworks.ai/api-reference/list-evaluators

get /v1/accounts/{account_id}/evaluators
Lists all evaluators for an account with pagination support.



# List Models
Source: https://docs.fireworks.ai/api-reference/list-models

get /v1/accounts/{account_id}/models



# List Quotas
Source: https://docs.fireworks.ai/api-reference/list-quotas

get /v1/accounts/{account_id}/quotas
Lists all quotas for an account.



# List Reinforcement Fine-tuning Jobs
Source: https://docs.fireworks.ai/api-reference/list-reinforcement-fine-tuning-jobs

get /v1/accounts/{account_id}/reinforcementFineTuningJobs



# List Reinforcement Fine-tuning Steps
Source: https://docs.fireworks.ai/api-reference/list-reinforcement-fine-tuning-steps

get /v1/accounts/{account_id}/rlorTrainerJobs



# List Responses
Source: https://docs.fireworks.ai/api-reference/list-responses

get /v1/responses
Get a list of all responses for the authenticated account.

Args:
    limit: Maximum number of responses to return (default: 20, max: 100)
    after: Cursor for pagination - return responses after this ID
    before: Cursor for pagination - return responses before this ID



# List Routers
Source: https://docs.fireworks.ai/api-reference/list-routers

get /v1/accounts/{account_id}/routers



# List Secrets
Source: https://docs.fireworks.ai/api-reference/list-secrets

get /v1/accounts/{account_id}/secrets
Lists all secrets for an account. Note that the `value` field is not returned in the response for security reasons. Only the `name` and `key_name` fields are included for each secret.



# List Supervised Fine-tuning Jobs
Source: https://docs.fireworks.ai/api-reference/list-supervised-fine-tuning-jobs

get /v1/accounts/{account_id}/supervisedFineTuningJobs



# List Users
Source: https://docs.fireworks.ai/api-reference/list-users

get /v1/accounts/{account_id}/users



# Create Chat Completion
Source: https://docs.fireworks.ai/api-reference/post-chatcompletions

post /v1/chat/completions
Create a completion for the provided prompt and parameters.

For RL / agent rollouts, Fireworks inference exposes additional
rollout-specific features:
[`x-session-affinity` and `x-multi-turn-session-id`](https://docs.fireworks.ai/guides/rollout-inference#session-affinity)
for multi-turn trajectories, and
[MoE Router Replay (R3)](https://docs.fireworks.ai/guides/rollout-inference#moe-router-replay)
for MoE expert tracing during rollouts.



# Create Completion
Source: https://docs.fireworks.ai/api-reference/post-completions

post /v1/completions
Create a completion for the provided prompt and parameters.

For RL / agent rollouts, Fireworks inference exposes additional
rollout-specific features:
[`x-session-affinity` and `x-multi-turn-session-id`](https://docs.fireworks.ai/guides/rollout-inference#session-affinity)
for multi-turn trajectories, and
[MoE Router Replay (R3)](https://docs.fireworks.ai/guides/rollout-inference#moe-router-replay)
for MoE expert tracing during rollouts.



# Create Response
Source: https://docs.fireworks.ai/api-reference/post-responses

post /v1/responses
Creates a model response, optionally interacting with custom tools via the Model Context Protocol (MCP). This endpoint supports conversational continuation and streaming.

Explore our cookbooks for detailed examples:

- [Basic MCP Usage](https://github.com/fw-ai/cookbook/blob/main/archived/learn/response-api/fireworks_mcp_examples.ipynb)
- [Streaming with MCP](https://github.com/fw-ai/cookbook/blob/main/archived/learn/response-api/fireworks_mcp_with_streaming.ipynb)
- [Conversational History with `previous_response_id`](https://github.com/fw-ai/cookbook/blob/main/archived/learn/response-api/fireworks_previous_response_cookbook.ipynb)
- [Basic Streaming](https://github.com/fw-ai/cookbook/blob/main/archived/learn/response-api/fireworks_streaming_example.ipynb)
- [Controlling Response Storage](https://github.com/fw-ai/cookbook/blob/main/archived/learn/response-api/mcp_server_with_store_false_argument.ipynb)



# Prepare Model for different precisions
Source: https://docs.fireworks.ai/api-reference/prepare-model

post /v1/accounts/{account_id}/models/{model_id}:prepare



# Query grouped usage cost subtotals for an account.
Source: https://docs.fireworks.ai/api-reference/query-usage-costs

post /v1/accounts/{account_id}/usageCosts:query
Returns rated dollar subtotals for usage over a time range, grouped by up to two of: HOUR, DAY, MODEL, USER, or API_KEY. Unlike `GET /billingUsage` (metered quantities) and `GET /billing/summary` (line items by billing category), this endpoint returns *rated costs* broken down by caller-supplied dimensions, with pagination and an account-wide `subtotal`. Requires account administrator access for `ACCOUNT` scope; `SELF` scope returns costs for the authenticated principal only.



# Rerank documents
Source: https://docs.fireworks.ai/api-reference/rerank-documents

post /rerank
Rerank documents for a query using relevance scoring



# Resume Dpo Job
Source: https://docs.fireworks.ai/api-reference/resume-dpo-job

post /v1/accounts/{account_id}/dpoJobs/{dpo_job_id}:resume



# Resume Reinforcement Fine-tuning Job
Source: https://docs.fireworks.ai/api-reference/resume-reinforcement-fine-tuning-job

post /v1/accounts/{account_id}/reinforcementFineTuningJobs/{reinforcement_fine_tuning_job_id}:resume



# Resume Rlor Trainer Job
Source: https://docs.fireworks.ai/api-reference/resume-reinforcement-fine-tuning-step

post /v1/accounts/{account_id}/rlorTrainerJobs/{rlor_trainer_job_id}:resume



# Resume Supervised Fine-tuning Job
Source: https://docs.fireworks.ai/api-reference/resume-supervised-fine-tuning-job

post /v1/accounts/{account_id}/supervisedFineTuningJobs/{supervised_fine_tuning_job_id}:resume



# Scale Deployment to a specific number of replicas or to zero
Source: https://docs.fireworks.ai/api-reference/scale-deployment

patch /v1/accounts/{account_id}/deployments/{deployment_id}:scale



# Undelete Deployment
Source: https://docs.fireworks.ai/api-reference/undelete-deployment

post /v1/accounts/{account_id}/deployments/{deployment_id}:undelete



# Update Dataset
Source: https://docs.fireworks.ai/api-reference/update-dataset

patch /v1/accounts/{account_id}/datasets/{dataset_id}



# Update LoRA
Source: https://docs.fireworks.ai/api-reference/update-deployed-model

patch /v1/accounts/{account_id}/deployedModels/{deployed_model_id}



# Update Deployment
Source: https://docs.fireworks.ai/api-reference/update-deployment

patch /v1/accounts/{account_id}/deployments/{deployment_id}



# Update Evaluator
Source: https://docs.fireworks.ai/api-reference/update-evaluator

patch /v1/accounts/{account_id}/evaluators/{evaluator_id}
Updates evaluator metadata (display_name, description, default_dataset).
Changing `requirements` or `entry_point` triggers a rebuild. To upload new
source code, set `prepare_code_upload: true` then follow the upload flow.



# Update Model
Source: https://docs.fireworks.ai/api-reference/update-model

patch /v1/accounts/{account_id}/models/{model_id}



# Update Quota
Source: https://docs.fireworks.ai/api-reference/update-quota

patch /v1/accounts/{account_id}/quotas/{quota_id}
Updates a quota.



# Update Router
Source: https://docs.fireworks.ai/api-reference/update-router

patch /v1/accounts/{account_id}/routers/{router_id}



# Update secret
Source: https://docs.fireworks.ai/api-reference/update-secret

patch /v1/accounts/{account_id}/secrets/{secret_id}



# Update User
Source: https://docs.fireworks.ai/api-reference/update-user

patch /v1/accounts/{account_id}/users/{user_id}



# Upload Dataset Files
Source: https://docs.fireworks.ai/api-reference/upload-dataset-files

post /v1/accounts/{account_id}/datasets/{dataset_id}:upload
Provides a streamlined way to upload a dataset file in a single API request. This path can handle file sizes up to 150Mb. For larger file sizes use [Get Dataset Upload Endpoint](get-dataset-upload-endpoint).




# Validate Dataset Upload
Source: https://docs.fireworks.ai/api-reference/validate-dataset-upload

post /v1/accounts/{account_id}/datasets/{dataset_id}:validateUpload



# Validate Evaluator Upload
Source: https://docs.fireworks.ai/api-reference/validate-evaluator-upload

post /v1/accounts/{account_id}/evaluators/{evaluator_id}:validateUpload
Triggers server-side validation of the uploaded source code (**step 5** in
the [Create Evaluator](/api-reference/create-evaluator) workflow). The server
extracts and processes the archive, then builds the evaluator environment.
Poll [Get Evaluator](/api-reference/get-evaluator) to monitor progress.



# Validate Model Upload
Source: https://docs.fireworks.ai/api-reference/validate-model-upload

get /v1/accounts/{account_id}/models/{model_id}:validateUpload



# Autoscaling
Source: https://docs.fireworks.ai/deployments/autoscaling

Configure how your deployment scales based on traffic

Control how your deployment scales based on traffic and load.

## Configuration options

| Flag                     | Type      | Default       | Description                                            |
| ------------------------ | --------- | ------------- | ------------------------------------------------------ |
| `--min-replica-count`    | Integer   | 0             | Minimum number of replicas. Set to 0 for scale-to-zero |
| `--max-replica-count`    | Integer   | 1             | Maximum number of replicas                             |
| `--scale-up-window`      | Duration  | 30s           | Wait time before scaling up                            |
| `--scale-down-window`    | Duration  | 10m           | Wait time before scaling down                          |
| `--scale-to-zero-window` | Duration  | 1h            | Idle time before scaling to zero (min: 5m)             |
| `--load-targets`         | Key-value | `default=0.8` | Scaling thresholds. See options below                  |

**Load target options** (use as `--load-targets <key>=<value>[,<key>=<value>...]`):

* `default=<Fraction>` - General load target from 0 to 1
* `tokens_generated_per_second=<Integer>` - Desired tokens per second per replica
* `prompt_tokens_per_second=<Integer>` - Desired prompt tokens per second per replica
* `requests_per_second=<Number>` - Desired requests per second per replica
* `concurrent_requests=<Number>` - Desired concurrent requests per replica

When multiple targets are specified, the maximum replica count across all is used.

## Common patterns

<Tabs>
  <Tab title="Cost optimization">
    Scale to zero when idle to minimize costs:

    ```bash theme={null}
    firectl deployment create <MODEL_NAME> \
      --min-replica-count 0 \
      --max-replica-count 3 \
      --scale-to-zero-window 1h
    ```

    Best for: Development, testing, or intermittent production workloads.
  </Tab>

  <Tab title="Performance-focused">
    Keep replicas running for instant response:

    ```bash theme={null}
    firectl deployment create <MODEL_NAME> \
      --min-replica-count 2 \
      --max-replica-count 10 \
      --scale-up-window 15s \
      --load-targets concurrent_requests=5
    ```

    Best for: Low-latency requirements, avoiding cold starts, high-traffic applications.
  </Tab>

  <Tab title="Predictable traffic">
    Match known traffic patterns:

    ```bash theme={null}
    firectl deployment create <MODEL_NAME> \
      --min-replica-count 3 \
      --max-replica-count 5 \
      --scale-down-window 30m \
      --load-targets tokens_generated_per_second=150
    ```

    Best for: Steady workloads where you know typical load ranges.
  </Tab>
</Tabs>

## Scaling from zero behavior

When a deployment is scaled to zero and receives a request, the system immediately returns a `503` error with the `DEPLOYMENT_SCALING_UP` error code while initiating the scale-up process:

```json theme={null}
{
  "error": {
    "message": "Deployment is currently scaled to zero and is scaling up. Please retry your request in a few minutes.",
    "code": "DEPLOYMENT_SCALING_UP",
    "type": "error"
  }
}
```

<Warning>
  Requests to a scaled-to-zero deployment are **not queued**. Your application must implement retry logic to handle `503` responses while the deployment scales up.
</Warning>

### Handling scale-from-zero responses

Implement retry logic with exponential backoff to gracefully handle scale-up delays:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    import time
    import requests

    def query_deployment_with_retry(url, payload, max_retries=30, initial_delay=5):
        """Query a deployment with retry logic for scale-from-zero scenarios."""
        delay = initial_delay
        
        for attempt in range(max_retries):
            response = requests.post(url, json=payload, headers=headers)
            
            # Only retry if deployment is scaling up
            if response.status_code == 503:
                error_code = response.json().get("error", {}).get("code")
                if error_code == "DEPLOYMENT_SCALING_UP":
                    print(f"Deployment scaling up, retrying in {delay}s...")
                    time.sleep(delay)
                    delay = min(delay * 1.5, 60)  # Cap at 60 seconds
                    continue
                
            response.raise_for_status()
            return response.json()
        
        raise Exception("Deployment did not scale up in time")
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    async function queryDeploymentWithRetry(url, payload, maxRetries = 30, initialDelay = 5000) {
      let delay = initialDelay;
      
      for (let attempt = 0; attempt < maxRetries; attempt++) {
        const response = await fetch(url, {
          method: 'POST',
          headers: { 'Content-Type': 'application/json', ...headers },
          body: JSON.stringify(payload)
        });
        
        // Only retry if deployment is scaling up
        if (response.status === 503) {
          const body = await response.json();
          if (body.error?.code === 'DEPLOYMENT_SCALING_UP') {
            console.log(`Deployment scaling up, retrying in ${delay/1000}s...`);
            await new Promise(resolve => setTimeout(resolve, delay));
            delay = Math.min(delay * 1.5, 60000); // Cap at 60 seconds
            continue;
          }
        }
        
        if (!response.ok) throw new Error(`HTTP ${response.status}`);
        return response.json();
      }
      
      throw new Error('Deployment did not scale up in time');
    }
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    # Simple retry loop for scale-from-zero
    MAX_RETRIES=30
    RETRY_DELAY=5

    for i in $(seq 1 $MAX_RETRIES); do
      response=$(curl -s -w "\n%{http_code}" \
        https://api.fireworks.ai/inference/v1/chat/completions \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $FIREWORKS_API_KEY" \
        -d '{"model": "accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>", ...}')
      
      http_code=$(echo "$response" | tail -n1)
      body=$(echo "$response" | head -n -1)
      
      # Only retry if deployment is scaling up
      if [ "$http_code" -eq 503 ]; then
        error_code=$(echo "$body" | jq -r '.error.code // empty')
        if [ "$error_code" = "DEPLOYMENT_SCALING_UP" ]; then
          echo "Deployment scaling up, retrying in ${RETRY_DELAY}s..."
          sleep $RETRY_DELAY
          RETRY_DELAY=$((RETRY_DELAY * 2))
          continue
        fi

        echo "$body"
        exit 1
      fi
      
      # Check for success (2xx status codes)
      if [ "$http_code" -ge 200 ] && [ "$http_code" -lt 300 ]; then
        echo "$body"
        exit 0
      fi

      echo "$body"
      exit 1
    done

    echo "Deployment did not scale up in time"
    exit 1
    ```
  </Tab>
</Tabs>

<Tip>
  Cold start times vary depending on model size—larger models may take longer to download and initialize. If you need instant responses without cold starts, set `--min-replica-count 1` or higher to keep replicas always running.
</Tip>

<Note>
  Deployments with min replicas = 0 are auto-deleted after 7 days of no traffic. [Reserved capacity](/deployments/reservations) guarantees availability during scale-up.
</Note>


# Performance benchmarking
Source: https://docs.fireworks.ai/deployments/benchmarking

Measure and optimize your deployment's performance with load testing

Understanding your deployment's performance under various load conditions is essential for production readiness. Fireworks provides tools and best practices for benchmarking throughput, latency, and identifying bottlenecks.

## Fireworks Benchmark Tool

Use our open-source benchmarking tool to measure and optimize your deployment's performance:

**[Fireworks Benchmark Tool](https://github.com/fw-ai/benchmark)**

This tool allows you to:

* Test throughput and latency under various load conditions
* Simulate production traffic patterns
* Identify performance bottlenecks
* Compare different deployment configurations

### Installation

```bash theme={null}
git clone https://github.com/fw-ai/benchmark.git
cd benchmark
pip install -r requirements.txt
```

### Basic usage

Run a basic benchmark test:

```bash theme={null}
python benchmark.py \
  --model "accounts/fireworks/models/llama-v3p1-8b-instruct" \
  --deployment "your-deployment-id" \
  --num-requests 1000 \
  --concurrency 10
```

### Key metrics to monitor

When benchmarking your deployment, focus on these key metrics:

* **Throughput**: Requests per second (RPS) your deployment can handle
* **Latency**: Time to first token (TTFT) and end-to-end response time
* **Token generation rate**: Tokens per second during generation
* **Error rate**: Failed requests under load

## Custom benchmarking

You can also develop custom performance testing scripts or integrate with monitoring tools to track metrics over time. Consider:

* Using production-like request patterns and payloads
* Testing with various concurrency levels
* Monitoring resource utilization (GPU, memory, network)
* Testing autoscaling behavior under load

## Best practices

1. **Warm up your deployment**: Run a few requests before benchmarking to ensure models are loaded
2. **Test realistic scenarios**: Use request patterns and payloads similar to your production workload
3. **Gradually increase load**: Start with low concurrency and gradually increase to find your deployment's limits
4. **Monitor for errors**: Track error rates and response codes to identify issues under load
5. **Compare configurations**: Test different deployment shapes, quantization levels, and hardware to optimize cost and performance

## Next steps

<CardGroup>
  <Card title="Autoscaling" href="/deployments/autoscaling" icon="arrows-up-down">
    Configure autoscaling to handle variable load
  </Card>

  <Card title="Client-side optimization" href="/deployments/client-side-performance-optimization" icon="bolt">
    Optimize your client code for maximum throughput
  </Card>
</CardGroup>


# Client-side performance optimization
Source: https://docs.fireworks.ai/deployments/client-side-performance-optimization

Optimize your client code for maximum performance with dedicated deployments

When using a dedicated deployment, it is important to optimize the client-side HTTP connection pooling for maximum performance. We recommend using our [Python SDK](/tools-sdks/python-sdk) as it has good defaults for connection pooling and utilizes [httpx](https://www.python-httpx.org/) for optimal performance with Python's `asyncio` library. It also includes retry logic for handling `429` errors that Fireworks returns when the server is overloaded.

## General optimization recommendations

Based on our benchmarks, we recommend the following:

1. Use a client library optimized for high concurrency, such as [httpx](https://www.python-httpx.org/) in Python or [http.Agent](https://nodejs.org/api/http.html#class-httpagent) in Node.js.
2. Use the `AsyncFireworks` client for high-concurrency workloads.
3. Increase concurrency until performance stops improving or you observe too many `429` errors.

## Code example: Optimal concurrent requests (Python)

Install the [Fireworks Python SDK](/tools-sdks/python-sdk):

<Note>
  The SDK is currently in alpha. Use the `--pre` flag when installing to get the latest version.
</Note>

<CodeGroup>
  ```bash pip theme={null}
  pip install --pre fireworks-ai
  ```

  ```bash poetry theme={null}
  poetry add --pre fireworks-ai
  ```

  ```bash uv theme={null}
  uv add --pre fireworks-ai
  ```
</CodeGroup>

Here's how to implement optimal concurrent requests using `asyncio` and the `AsyncFireworks` client:

```python main.py theme={null}
import asyncio
import time
import statistics
from fireworks import AsyncFireworks


async def make_concurrent_requests(
    messages: list[str],
    model: str,
    max_workers: int = 1000,
):
    """Make concurrent requests with optimized connection pooling"""

    client = AsyncFireworks(
        max_retries=5,
    )

    # Semaphore to limit concurrent requests
    semaphore = asyncio.Semaphore(max_workers)
    latencies = []

    async def single_request(message: str):
        """Make a single request with semaphore control"""
        async with semaphore:
            start_time = time.perf_counter()
            response = await client.chat.completions.create(
                model=model,
                messages=[{"role": "user", "content": message}],
                max_tokens=100,
            )
            latency = time.perf_counter() - start_time
            latencies.append(latency)
            return response.choices[0].message.content

    # Create all request tasks
    tasks = [single_request(message) for message in messages]

    # Execute all requests concurrently
    results = await asyncio.gather(*tasks)
    return results, latencies


# Usage example
async def main():
    messages = ["Hello!"] * 1000  # 1000 requests

    model = "accounts/fireworks/models/qwen3-0p6b"

    start_time = time.perf_counter()
    results, latencies = await make_concurrent_requests(
        messages=messages,
        model=model,
    )
    total_time = time.perf_counter() - start_time

    # Calculate performance metrics
    num_requests = len(results)
    requests_per_second = num_requests / total_time

    # Latency statistics (in milliseconds)
    latencies_ms = [lat * 1000 for lat in latencies]
    avg_latency = statistics.mean(latencies_ms)
    min_latency = min(latencies_ms)
    max_latency = max(latencies_ms)
    p50_latency = statistics.median(latencies_ms)
    p95_latency = statistics.quantiles(latencies_ms, n=20)[18]  # 95th percentile
    p99_latency = statistics.quantiles(latencies_ms, n=100)[98]  # 99th percentile

    print("\n" + "=" * 50)
    print("Performance Results")
    print("=" * 50)
    print(f"Total requests:      {num_requests}")
    print(f"Total time:          {total_time:.2f} seconds")
    print(f"Throughput:          {requests_per_second:.2f} requests/second")
    print("\nLatency Statistics (ms):")
    print(f"  Min:               {min_latency:.2f}")
    print(f"  Max:               {max_latency:.2f}")
    print(f"  Avg:               {avg_latency:.2f}")
    print(f"  P50 (median):      {p50_latency:.2f}")
    print(f"  P95:               {p95_latency:.2f}")
    print(f"  P99:               {p99_latency:.2f}")
    print("=" * 50)


if __name__ == "__main__":
    asyncio.run(main())
```

This implementation:

* Uses `AsyncFireworks` for non-blocking async requests with optimized connection pooling
* Uses `asyncio.Semaphore` to control concurrency to avoid overwhelming the server


# Exporting Metrics
Source: https://docs.fireworks.ai/deployments/exporting-metrics

Export metrics from your dedicated deployments to your observability stack

## Overview

Fireworks provides a metrics endpoint in Prometheus format, enabling integration with popular observability tools like Prometheus, OpenTelemetry (OTel) Collector, Datadog Agent, and Vector.

<Note>
  This page covers real-time performance metrics (latency, throughput, etc.) for on-demand deployments. For billing and usage data across all Fireworks services, see [Exporting Billing Metrics](/accounts/exporting-billing-metrics).
</Note>

## Setting Up Metrics Collection

### Endpoint

The metrics endpoint is as follows. This URL and authorization header can be directly used by services like Grafana Cloud to ingest Fireworks metrics.

```
https://api.fireworks.ai/v1/accounts/<account-id>/metrics
```

### Authentication

Use the Authorization header with your Fireworks API key:

```json theme={null}
{
  "Authorization": "Bearer YOUR_API_KEY"
}
```

### Scrape Interval

We recommend using a 1-minute scrape interval as metrics are updated every 30s.

### Rate Limits

To ensure service stability and fair usage:

* Maximum of 6 requests per minute per account
* Exceeding this limit results in HTTP 429 (Too Many Requests) responses
* Use a 1-minute scrape interval to stay within limits

## Integration Options

Fireworks metrics can be integrated with various observability platforms through multiple approaches:

### OpenTelemetry Collector Integration

The Fireworks metrics endpoint can be integrated with OpenTelemetry Collector by configuring a Prometheus receiver that scrapes the endpoint. This allows Fireworks metrics to be pushed to a variety of popular exporters—see the [OpenTelemetry registry](https://opentelemetry.io/ecosystem/registry/) for a full list.

### Direct Prometheus Integration

To integrate directly with Prometheus, specify the Fireworks metrics endpoint in your scrape config:

```yaml theme={null}
global:
  scrape_interval: 60s
scrape_configs:
  - job_name: 'fireworks'
    metrics_path: 'v1/accounts/<account-id>/metrics'
    authorization:
      type: "Bearer"
      credentials: "YOUR_API_KEY"
    static_configs:
      - targets: ['api.fireworks.ai']
    scheme: https
```

For more details on Prometheus configuration, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/).

### Supported Platforms

Fireworks metrics can be exported to various observability platforms including:

* Prometheus
* Datadog
* Grafana
* New Relic

## Available Metrics

### Common Labels

All metrics include the following common labels:

* `base_model`: The base model identifier (e.g., "accounts/fireworks/models/deepseek-v3")
* `deployment`: Full deployment path (e.g., "accounts/account-name/deployments/deployment-id")
* `deployment_account`: The account name
* `deployment_id`: The deployment identifier

### Rate Metrics (per second)

These metrics show activity rates calculated using 1-minute windows:

#### Request Rate

* `request_counter_total:sum_by_deployment`: Request rate per deployment

#### Error Rate

* `requests_error_total:sum_by_deployment`: Error rate per deployment, broken down by HTTP status code (includes additional `http_code` label)

#### Token Processing Rates

* `tokens_cached_prompt_total:sum_by_deployment`: Rate of cached prompt tokens per deployment
* `tokens_prompt_total:sum_by_deployment`: Rate of total prompt tokens processed per deployment

### Latency Histogram Metrics

These metrics provide latency distribution data with histogram buckets, calculated using 1-minute windows:

#### Generation Latency

* `latency_generation_per_token_ms_bucket:sum_by_deployment`: Per-token generation time distribution
* `latency_generation_queue_ms_bucket:sum_by_deployment`: Time spent waiting in generation queue

#### Request Latency

* `latency_overall_ms_bucket:sum_by_deployment`: End-to-end request latency distribution
* `latency_to_first_token_ms_bucket:sum_by_deployment`: Time to first token distribution

#### Prefill Latency

* `latency_prefill_ms_bucket:sum_by_deployment`: Prefill processing time distribution
* `latency_prefill_queue_ms_bucket:sum_by_deployment`: Time spent waiting in prefill queue

### Token Distribution Metrics

These histogram metrics show token count distributions per request, calculated using 1-minute windows:

* `tokens_generated_per_request_bucket:sum_by_deployment`: Distribution of generated tokens per request
* `tokens_prompt_per_request_bucket:sum_by_deployment`: Distribution of prompt tokens per request

### Resource Utilization Metrics

These gauge metrics show average resource usage:

* `generator_kv_blocks_fraction:avg_by_deployment`: Average fraction of KV cache blocks in use
* `generator_kv_slots_fraction:avg_by_deployment`: Average fraction of KV cache slots in use
* `generator_model_forward_time:avg_by_deployment`: Average time spent in model forward pass
* `requests_coordinator_concurrent_count:avg_by_deployment`: Average number of concurrent requests
* `prefiller_prompt_cache_ttl:avg_by_deployment`: Average prompt cache time-to-live


# Regions
Source: https://docs.fireworks.ai/deployments/regions

Fireworks runs a global fleet of hardware on which you can deploy your models.

Fireworks runs a global fleet so you can deploy models close to users, meet data-residency needs, and scale across clouds. This page covers **multi-region** (default behavior and quota groupings), **single-region** availability and hardware, how to **use and change** regions, and **quotas**.

## Multi-region (recommended)

By default, deployments are multi-region: Fireworks can move and spread them across regions as needed. Multi-regions (**GLOBAL**, **US**, **EUROPE**, **APAC**) are high-level groupings of single regions. Your deployment may run in any single region(s) within that multi-region.

<CardGroup>
  <Card title="Flexible, elastic scaling" icon="gauge">
    Utilizing multiple clouds and locations maximizes the odds that there's capacity to scale.
  </Card>

  <Card title="Higher reliability" icon="shield-check">
    Multi-region deployments enable resilience to localized outages, maintaining application availability as workloads scale across regions.
  </Card>
</CardGroup>

### Supported multi-regions

Supported multi-regions: `GLOBAL`, `US`, `EUROPE`, `APAC`.

## Single region availability

Single regions are concrete locations (e.g. `US_IOWA_1`, `EU_FRANKFURT_1`) where your deployment can run. We have the single regions listed below available; we recommend multi-region for most users because of its advantages (elastic scaling, higher reliability). If you have a specific need for a single region, contact [Fireworks](mailto:inquiries@fireworks.ai) to request it. The table below shows which single regions are available and what hardware is offered in each.

| **Region**        | **Accelerator Type(s)**                  |   |
| ----------------- | ---------------------------------------- | - |
| `US_ARIZONA_1`    | `NVIDIA_H100_80GB`                       |   |
| `US_CALIFORNIA_1` | `NVIDIA_H200_141GB`                      |   |
| `US_GEORGIA_2`    | `NVIDIA_B200_180GB`                      |   |
| `US_GEORGIA_3`    | `NVIDIA_H200_141GB`                      |   |
| `US_ILLINOIS_1`   | `NVIDIA_H100_80GB`                       |   |
| `US_ILLINOIS_2`   | `NVIDIA_A100_80GB`                       |   |
| `US_IOWA_1`       | `NVIDIA_H100_80GB`                       |   |
| `US_OHIO_1`       | `NVIDIA_B200_180GB`                      |   |
| `US_TEXAS_2`      | `NVIDIA_H100_80GB`                       |   |
| `US_UTAH_1`       | `NVIDIA_B200_180GB`                      |   |
| `US_VIRGINIA_1`   | `NVIDIA_H100_80GB`, `NVIDIA_H200_141GB`  |   |
| `US_WASHINGTON_2` | `NVIDIA_H100_80GB`                       |   |
| `US_WASHINGTON_3` | `NVIDIA_B200_180GB`                      |   |
| `US_WASHINGTON_4` | `NVIDIA_B200_180GB`                      |   |
| `EU_FRANKFURT_1`  | `NVIDIA_H100_80GB`                       |   |
| `EU_ICELAND_1`    | `NVIDIA_H200_141GB`                      |   |
| `EU_ICELAND_2`    | `NVIDIA_B200_180GB`, `NVIDIA_H200_141GB` |   |
| `AP_TOKYO_1`      | `NVIDIA_H100_80GB`                       |   |
| `AP_TOKYO_2`      | `NVIDIA_H200_141GB`                      |   |

## Using a region

When creating a deployment, you can pass the `--region` flag to pin it to a single region:

```
firectl deployment create accounts/fireworks/models/llama-v3p1-8b-instruct \
    --region GLOBAL
```

## Changing regions

Updating the single region for a deployment in-place is not supported. To move a deployment to a different single region, create a new deployment in the desired region, then delete the old deployment.

## Quotas

Quota is granted at the **multi-region** level for new users. By default, all users receive quota for **GLOBAL** multi-region. For specific single region quota, please contact Fireworks. To view your current quotas, run:

```
firectl quota list
```

To use single regions that are not generally available (see the table above), or to request additional multi-region quota, contact [inquiries@fireworks.ai](mailto:inquiries@fireworks.ai).


# Reserved capacity
Source: https://docs.fireworks.ai/deployments/reservations



Enterprise accounts can purchase reserved capacity, typically with 1 year commitments. Reserved capacity has the following advantages over ordinary [on-demand deployments](/guides/ondemand-deployments):

* Guaranteed capacity
* Higher quotas
* Lower GPU-hour prices
* Pre-GA access to newer regions
* Pre-GA access to newest hardware

## Usage and billing

Consuming a reservation is done by creating a deployment that meets the reservation parameters. For example, suppose you have a reservation for 12 H100 GPUs and create two deployments, each using 8 H100 GPUs. While both deployments are running, 12 of the H100s will count towards using your reservation, while the excess 4 H100s will be metered and billed at the on-demand rate. Follow [deploying models on-demand](/guides/ondemand-deployments) to create a deployment.

When a reservation approaches its end time, ensure that you either renew your reservation or turn down a corresponding number of deployments, otherwise you may be billed at for your usage at on-demand rates.

Reservations are invoiced separately from your on-demand usage, at a frequency determined by your reservation contract
(e.g. monthly, quarterly, or yearly).

<Note>Reserved capacity will always be billed until the reservation ends, regardless of whether the reservation is
actively used.</Note>

## Purchasing or renewing a reservation

To purchase a reservation or increase the size or duration of an existing reservation, contact your Fireworks account
manager. If you are a new, prospective customer, please reach out to our [sales team](https://fireworks.ai/company/contact-us).

## Viewing your reservations

To view your existing reservations, run:

```
firectl reservation list
```


# Routers
Source: https://docs.fireworks.ai/deployments/routers

Distribute traffic across multiple deployments for A/B testing, traffic migration, and load distribution.

A **Router** is a resource that controls how inference traffic is routed to one or more deployments. Instead of sending all requests to a single deployment, a router lets you split traffic across multiple deployments — useful for A/B testing model variants, gradually migrating traffic to a new deployment, or distributing load.

Traffic is split proportionally based on the number of replicas in each deployment. For example, if a router covers two deployments — one with 3 replicas and another with 2 — the first receives 60% of traffic and the second receives 40%.

<Warning>
  Routers only work with multi-region deployments.
</Warning>

## When to use a router

### Stable alias for deployment replacement

If you plan to replace a deployment later (e.g., changing to a new model later), give your application the **router name** instead of the deployment name. You can then swap the underlying deployment without your application changing anything.

```
Your app calls: accounts/<ACCOUNT_ID>/routers/my-router
  └── Initially routes to: accounts/<ACCOUNT_ID>/deployments/v1
  └── Later updated to:    accounts/<ACCOUNT_ID>/deployments/v2
```

### A/B testing between deployments

Place multiple deployments under a single router. Traffic is automatically split by replica count, so you can control the ratio by adjusting replicas on each deployment.

```bash theme={null}
firectl router create \
    --router-id=ab-test \
    --deployments=model-a,model-b
```

### Gradual traffic migration

Shift traffic from an old deployment to a new one with zero downtime by scaling replicas up on the new deployment and down on the old. See the [worked example](#example-traffic-migration) below.

## How traffic routing works

Traffic is distributed based on **replica count**. Each replica across all deployments in the router receives an equal share of traffic.

| Deployment     | Replicas | Traffic share |
| -------------- | -------- | ------------- |
| `deployment-a` | 3        | 60%           |
| `deployment-b` | 2        | 40%           |
| **Total**      | **5**    | **100%**      |

To shift traffic, scale the replica counts on the underlying deployments. The router automatically adjusts the distribution.

### Sending traffic to a router

Use the router's name in the `model` field of your API request, just like you would use a deployment name:

```bash theme={null}
curl -s -X POST https://api.fireworks.ai/inference/v1/chat/completions \
  -H "Authorization: Bearer $FIREWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "accounts/<ACCOUNT_ID>/routers/<ROUTER_ID>",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
```

### Routing strategy

Traffic is routed using **weighted replica** selection: each request is randomly assigned to a deployment, weighted by its replica count. A deployment with more replicas receives proportionally more traffic.

## Managing routers

### Creating a router

A router requires at least one deployment.

```bash theme={null}
firectl router create \
    --deployments=<DEPLOYMENT_1>,<DEPLOYMENT_2>
```

Optional flags:

| Flag             | Description                                                    |
| ---------------- | -------------------------------------------------------------- |
| `--router-id`    | Set a specific router ID. If omitted, a random ID is generated |
| `--display-name` | Human-readable name for the router                             |
| `--model`        | The model to route traffic to                                  |
| `--strategy`     | Routing strategy. Default: `weighted-random`                   |
| `--public`       | Make the router accessible to other accounts                   |

### Listing routers

```bash theme={null}
firectl router list
```

### Getting router details

```bash theme={null}
firectl router get <ROUTER_ID>
```

You can also use the full resource name:

```bash theme={null}
firectl router get accounts/<ACCOUNT_ID>/routers/<ROUTER_ID>
```

### Updating a router

Update the deployments, strategy, or other properties of an existing router:

```bash theme={null}
firectl router update <ROUTER_ID> \
    --deployments=<DEPLOYMENT_1>,<DEPLOYMENT_2>,<DEPLOYMENT_3>
```

### Deleting a router

```bash theme={null}
firectl router delete <ROUTER_ID>
```

<Warning>
  Deleting a router takes effect immediately. Any traffic sent to the router's alias will fail. Make sure all clients have switched to a different route before deleting.
</Warning>

## Example: traffic migration

This example walks through migrating traffic from an existing deployment to a new one with zero downtime.

**Step 1** — Create a router for your existing deployment and point your application at the router alias:

```bash theme={null}
firectl router create \
    --router-id=my-router \
    --deployments=current-deployment
```

Your application sends traffic to `accounts/<ACCOUNT_ID>/routers/my-router`. All traffic goes to `current-deployment`.

**Step 2** — Create the new deployment and add it to the router:

```bash theme={null}
firectl deployment create accounts/<ACCOUNT_ID>/models/<MODEL_ID> \
    --deployment-id=new-deployment
```

```bash theme={null}
firectl router update my-router \
    --deployments=current-deployment,new-deployment
```

A new deployment starts with 1 replica by default, so if `current-deployment` has 4 replicas, the split is immediately 80%/20%.

**Step 3** — Shift more traffic by increasing replicas on the new deployment and decreasing the old:

```bash theme={null}
firectl deployment update new-deployment \
    --min-replica-count=4 \
    --max-replica-count=4

firectl deployment update current-deployment \
    --min-replica-count=1 \
    --max-replica-count=1
```

Traffic split is now 20% old / 80% new.

**Step 4** — Complete the migration by scaling the old deployment to zero:

```bash theme={null}
firectl deployment update current-deployment \
    --min-replica-count=0 \
    --max-replica-count=0
```

All traffic now flows to `new-deployment`. Clean up by removing the old deployment from the router:

```bash theme={null}
firectl router update my-router --deployments=new-deployment
```

<Tip>
  Monitor your new deployment's latency and error rates at each step before shifting more traffic. This lets you catch issues early and roll back by increasing replicas on the old deployment.
</Tip>


# Speculative Decoding
Source: https://docs.fireworks.ai/deployments/speculative-decoding

Speed up generation with draft models and n-gram speculation

Speed up text generation by using a smaller "draft" model to assist the main model, or using n-gram based speculation.

## Default drafters

<Tip>
  **For most deployments, a default drafter is already attached automatically**, so you don't need to configure a draft model yourself to benefit from speculative decoding.

  For further optimized throughput, [reach out to us](https://fireworks.ai/company/contact-us) about a custom speculative decoding model adapted to your traffic pattern.
</Tip>

<Note>
  Speculative decoding may slow down output generation if the draft model is not a good speculator, or if token count/speculation length is too high or too low. It may also reduce max throughput. Test different models and speculation lengths for your use case.
</Note>

## Configuration options

| Flag                         | Type   | Description                                                                                 |
| ---------------------------- | ------ | ------------------------------------------------------------------------------------------- |
| `--draft-model`              | string | Draft model name. Can be a Fireworks model or custom model. See recommendations below.      |
| `--draft-token-count`        | int32  | Tokens to generate per step. Required when using draft model or n-gram. Typically set to 4. |
| `--ngram-speculation-length` | int32  | Alternative to draft model: uses N-gram based speculation from previous input.              |

<Note>
  `--draft-model` and `--ngram-speculation-length` cannot be used together.
</Note>

## Custom draft models

While you can specify a custom draft model with `--draft-model`, we recommend only setting a **small base model** as the drafter when using this setting. Other speculative decoding methods such as DFlash, EAGLE, and Medusa require specific configurations. If you need to set one of those up, [reach out to us](https://fireworks.ai/company/contact-us) and we'll help you configure it.

### Fallback draft models

If a default drafter isn't attached and you'd like a starting point, these small base models can be used as fallback drafters. Note that using a base model as a drafter is generally not ideal.

| Draft model                                        | Use with              |
| -------------------------------------------------- | --------------------- |
| `accounts/fireworks/models/llama-v3p2-1b-instruct` | All Llama models > 3B |
| `accounts/fireworks/models/qwen2p5-0p5b-instruct`  | All Qwen models > 3B  |

## Examples

<Tabs>
  <Tab title="Draft model">
    Use a smaller model to speed up generation:

    ```bash theme={null}
    firectl deployment create accounts/fireworks/models/llama-v3p3-70b-instruct \
      --draft-model="accounts/fireworks/models/llama-v3p2-1b-instruct" \
      --draft-token-count=4
    ```
  </Tab>

  <Tab title="N-gram speculation">
    Use input history for speculation (no draft model needed):

    ```bash theme={null}
    firectl deployment create accounts/fireworks/models/llama-v3p3-70b-instruct \
      --ngram-speculation-length=3 \
      --draft-token-count=4
    ```
  </Tab>
</Tabs>

<Tip>
  Fireworks also supports [Predicted Outputs](/guides/predicted-outputs) which works in addition to model-based speculative decoding.
</Tip>


# Claude Code
Source: https://docs.fireworks.ai/ecosystem/fireconnect/claude-code

Use Fireworks AI models in Claude Code with the FireConnect CLI

[FireConnect](https://github.com/fw-ai/fireconnect) routes [Claude Code](https://claude.ai/code) through Fireworks AI models. See the [FireConnect overview](/ecosystem/fireconnect/overview) for install and CLI basics.

<Tip>
  **Change models:** `fireconnect claude on --model <id>` (or `--opus` / `--sonnet` / …). See [Models](/ecosystem/fireconnect/models).
</Tip>

## Prerequisites

* [Claude Code](https://claude.ai/code) installed
* A [Fireworks API key](https://app.fireworks.ai/settings/users/api-keys) (`fw_...`) or a [Fire Pass](/firepass) key (`fpk_...`)
* The FireConnect CLI v0.9.2+ (see [Install](/ecosystem/fireconnect/overview#install))

<Note>
  **Azure routing not implemented yet for Claude Code.** `fireconnect claude on` always configures direct Fireworks, even when global config has `--provider azure` or you pass `--azure`. See [Microsoft Foundry in FireConnect](/ecosystem/fireconnect/microsoft-foundry#supported-harnesses).
</Note>

## Enable Fireworks routing

```bash theme={null}
fireconnect login
fireconnect claude on
```

Or pass the key once:

```bash theme={null}
fireconnect claude on --api-key fw_...
```

Restart Claude Code after enabling, then test with:

```text theme={null}
hi
```

After `fireconnect claude on`, start a new Claude Code session or run `/model` to pick up model changes. To use a new model in the same session, start a new session or `/exit` and resume with `claude --resume <id>`.

## Change models

```bash theme={null}
fireconnect model list --search glm
fireconnect claude on --model firerouter               # main only
fireconnect claude on --interactive                    # wizard for all six slots
fireconnect claude on --opus glm-fast-latest --sonnet glm-fast-latest
fireconnect claude status
```

Slot flags are independent. Re-running `on` without flags keeps your current mapping.

## Using Fire Pass

Use your `fpk_...` key during `login` or with `--api-key`:

```bash theme={null}
fireconnect claude on --api-key fpk_...
```

FireConnect detects Fire Pass keys and routes all model aliases to `kimi-fast-latest`.

## Example model mapping

When you run `fireconnect claude on` without model flags, FireConnect applies the mapping below. First-time setup opens an interactive model picker unless you pass `--non-interactive`. Override any slot with `--model`, `--opus`, `--sonnet`, `--haiku`, `--fable`, or `--subagent`.

| Alias    | Standard key (`fw_...`)                                                                                                                   | Fire Pass key (`fpk_...`) |
| -------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| main     | `firerouter` on first connect when Claude already has Anthropic auth (OAuth / `sk-ant-…`) or workspace BYOK; otherwise `kimi-fast-latest` | `kimi-fast-latest`        |
| opus     | `glm-fast-latest`                                                                                                                         | `kimi-fast-latest`        |
| sonnet   | `glm-fast-latest`                                                                                                                         | `kimi-fast-latest`        |
| haiku    | `deepseek-v4-flash`                                                                                                                       | `kimi-fast-latest`        |
| fable    | `kimi-fast-latest`                                                                                                                        | `kimi-fast-latest`        |
| subagent | `deepseek-v4-flash`                                                                                                                       | `kimi-fast-latest`        |

Short model IDs expand automatically. Claude Code adds `[1m]` on 1M-context models (not `subagent`).

FireConnect saves your chosen mapping per key type. Reopen the wizard anytime:

```bash theme={null}
fireconnect claude on --interactive
```

Use `--non-interactive` to skip the wizard and apply saved preferences or the example mapping above. `--interactive` cannot be combined with model flags like `--model` or `--opus`.

In the wizard, toggle between **fast models** (routers on the high-speed path) and **non-fast models** (pinned model IDs that stay stable across catalog updates).

## What gets written

FireConnect writes these settings to `~/.claude/settings.json`. Claude Code authenticates via the `X-Fireworks-Api-Key` custom header (not `apiKeyHelper`). The Fireworks key is written to the file with mode `0600`:

```json theme={null}
{
  "model": "kimi-fast-latest",
  "env": {
    "ANTHROPIC_BASE_URL": "https://api.fireworks.ai/inference",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "glm-fast-latest[1m]",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "glm-fast-latest[1m]",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "deepseek-v4-flash",
    "ANTHROPIC_DEFAULT_FABLE_MODEL": "kimi-fast-latest",
    "CLAUDE_CODE_SUBAGENT_MODEL": "deepseek-v4-flash",
    "ANTHROPIC_CUSTOM_HEADERS": "X-Fireworks-Api-Key: YOUR_FIREWORKS_API_KEY"
  }
}
```

FireConnect saves a backup of your previous provider settings to `~/.fireconnect/claude/` so `fireconnect claude off` can restore them.

## Web search MCP

When you run `fireconnect claude on`, FireConnect can install the Fireworks [WebSearch MCP](/ecosystem/fireconnect/websearch-mcp) for eligible accounts. FireConnect WebSearch integration is **Claude Code only today**; other harnesses are coming soon.

The MCP itself works from any harness that supports HTTP MCP. See the [WebSearch MCP guide](/ecosystem/fireconnect/websearch-mcp) for the endpoint URL, bearer token auth, and manual setup on Claude Code or other tools.

When installation succeeds, `fireconnect claude on` prints a confirmation such as `Web search → fireworks-websearch (installed)`. Restart Claude Code, then run `/mcp` and connect to `fireworks-websearch`.

`fireconnect claude off` removes the managed MCP entry and restores your previous `~/.claude/settings.json` (including any `permissions` rules).

## Browsing and picking models

```bash theme={null}
fireconnect model list --search glm
fireconnect claude on --sonnet kimi-latest
```

`fireconnect model list` shows serverless endpoints and pricing. `fireconnect claude status` shows your current mapping and rates.

Fire Pass keys only list Fire Pass routers. FireConnect rejects `--model firerouter` with Fire Pass (`fpk_...`) on every harness; use an `fw_...` key.

## FireRouter

Route requests through [FireRouter](/ecosystem/firerouter/overview). FireConnect rejects `--model firerouter` with Fire Pass (`fpk_...`) on every harness; use an `fw_...` key.

```bash theme={null}
fireconnect claude on --model firerouter
fireconnect claude on --opus firerouter --anthropic-api-key sk-ant-...
```

Use `--model firerouter` for main, or slot flags like `--opus firerouter`. Pass `--anthropic-api-key sk-ant-...` on `on`, or store it once with `fireconnect configure --anthropic-api-key sk-ant-...`.

Bias savings vs quality with `--routing-preference` (`1`-`5`; default `3`):

```bash theme={null}
fireconnect claude on --model firerouter --routing-preference 4
```

See [Routing preferences](/ecosystem/firerouter/routing-preferences).

## Usage and live meter

Claude Code's `/model` picker shows Anthropic list prices, not Fireworks rates. Use `fireconnect claude usage` to estimate real Fireworks spend from session logs.

On a TTY, `usage` opens a session picker (last 3 days by default), then a live cost meter. Tab: agents pane. Esc: session list. q: quit.

```bash theme={null}
fireconnect claude usage                 # picker → live meter
fireconnect claude usage --days 7        # picker lookback only (1–365)
fireconnect claude usage --session <id>  # start on one session
fireconnect claude usage --last-n 5      # snapshot, no picker (--days ignored)
fireconnect claude usage --plain         # plain text
fireconnect claude usage --json          # JSON
```

For a tmux split (Claude Code left, meter right):

```bash theme={null}
fireconnect claude live
fireconnect claude live --session <id>
```

Requires `tmux`. Neither command changes harness settings.

## Troubleshooting

### Text-only models and images

Claude Code cannot mark a model as non-vision. Pasting an image on a **text-only** slot (for example `glm-fast-latest` or `deepseek-v4-flash`) can break the session.

**Recover with `/rewind`**, then avoid images on that slot or map it to a vision model (for example `kimi-fast-latest`):

```bash theme={null}
fireconnect claude on --sonnet kimi-fast-latest
```

`fireconnect claude on` warns when your mapping includes text-only models. `fireconnect claude status` labels each slot `vision` or `text-only`.

## CLI reference

```bash theme={null}
fireconnect claude on         # Route Claude Code through Fireworks
fireconnect claude off        # Restore your previous provider
fireconnect claude status     # Provider, auth, and model mapping
fireconnect claude usage      # Session cost meter
fireconnect claude live       # tmux split with live meter
fireconnect claude help       # Harness-specific help
```

Run `fireconnect claude help` for all options.

### Turn off Fireworks routing

```bash theme={null}
fireconnect claude off
```

This restores your previous `~/.claude/settings.json` from the backup saved in `~/.fireconnect/claude/`.

## Uninstall

To remove FireConnect from your machine entirely (all harnesses):

```bash theme={null}
fireconnect uninstall
```

## Source

FireConnect is open source: [github.com/fw-ai/fireconnect](https://github.com/fw-ai/fireconnect)


# CLI reference
Source: https://docs.fireworks.ai/ecosystem/fireconnect/cli-reference

FireConnect global commands, providers, authentication, and migration

FireConnect uses **harness-first** syntax: `fireconnect <harness> <command>`. Bare harness names run `on` (for example, `fireconnect claude` is the same as `fireconnect claude on`).

To pick or switch models, see **[Models](/ecosystem/fireconnect/models)**. This page is the command and auth reference.

## Global commands

```bash theme={null}
fireconnect login        # Sign in: browser (creates a key) or paste a key you have
fireconnect logout       # Clear the stored key (keychain entry + config ref)
fireconnect status       # Show sign-in state and where the key is stored
fireconnect configure    # Set the provider (Azure/Foundry) and Anthropic key for FireRouter
fireconnect model list   # Browse the global Fireworks coding model catalog
fireconnect demo         # Race your provider vs Fireworks on the same prompt
fireconnect upgrade      # Update FireConnect (curl/git install only)
fireconnect uninstall    # Disable all harnesses, restore configs, remove CLI
fireconnect help         # Show help
fireconnect --version    # Print the installed CLI version (-V also works; --json for machine-readable)
```

Run `fireconnect help` for the overview, or `fireconnect claude help` (and similarly for other harnesses) for harness-level options.

## Sign in options

| Flag               | Use when                                                                                           |
| ------------------ | -------------------------------------------------------------------------------------------------- |
| `--paste`          | Skip the browser chooser and paste a key at the prompt                                             |
| `--api-key fw_...` | Sign in with a key directly (no prompt)                                                            |
| `--with-token`     | Read a key from stdin (handy in CI): `echo "$FIREWORKS_API_KEY" \| fireconnect login --with-token` |
| `--account <id>`   | Enterprise SSO sign-in (same account id as `firectl signin`)                                       |
| `--force`          | Replace an existing stored key without a confirmation prompt                                       |
| `logout --revoke`  | Clear local credentials **and** revoke the machine key on Fireworks                                |

```bash theme={null}
fireconnect status --json   # machine-readable sign-in and key-storage details
```

`~/.fireconnect/config.json` stores a reference such as `{keychain:fireworks-api-key}` or `{env:FIREWORKS_API_KEY}`. It never stores a literal key in global config.

## Global configuration

`fireconnect configure` sets provider defaults and shared keys. It does **not** sign you in. Use `login` for your Fireworks API key.

```bash theme={null}
fireconnect configure \
  --provider azure \
  --base-url https://<resource>.services.ai.azure.com \
  --api-key $AZURE_API_KEY

fireconnect configure --anthropic-api-key sk-ant-...
fireconnect configure --provider fireworks
```

In `configure`, `--api-key` is the **Azure** endpoint key and requires `--provider azure`. For Fireworks keys, use `fireconnect login`.

## Providers

| Provider flag         | Where inference runs           | API key                                           | Supported harnesses                |
| --------------------- | ------------------------------ | ------------------------------------------------- | ---------------------------------- |
| `fireworks` (default) | Fireworks gateway              | `fw_...` (all harnesses) or `fpk_...` (not Codex) | All harnesses with a supported key |
| `azure`               | Fireworks on Microsoft Foundry | Azure API key                                     | All harnesses except Claude Code   |

Set the default with `fireconnect configure --provider fireworks` or `--provider azure`. Harness `on` commands use the configured provider unless you pass `--azure` or per-command `--base-url` / `--api-key` overrides.

## Per-harness commands

```bash theme={null}
fireconnect model list --search glm
fireconnect claude on --model glm-fast-latest --sonnet kimi-latest
fireconnect opencode on --model glm-fast-latest
```

Each CLI harness (`claude`, `opencode`, `codex`, `pi`, `deepagents`) supports:

* `fireconnect <harness> on`: route through the configured provider
* `fireconnect <harness> off`: restore your previous config
* `fireconnect <harness> status`: show provider, auth, and models
* `fireconnect <harness> help`: harness-specific help

Claude Code also has `usage` and `live`. See [Usage and live meter](/ecosystem/fireconnect/claude-code#usage-and-live-meter).

Each IDE harness (`cursor`, `vscode`) supports `on`, `off`, `status`, and `help`. Commands that write settings require quitting the IDE first; `status` is read-only.

### Model flags

Use `--model <id>` for the primary model. Claude Code also supports slot flags: `--opus`, `--sonnet`, `--haiku`, `--fable`, `--subagent`.

See **[Models](/ecosystem/fireconnect/models)** for a cross-harness quick reference, restart requirements, and common mistakes.

Claude Code-only flags:

* `--interactive`: open the model mapping wizard (cannot combine with model flags)
* `--non-interactive`: skip first-run onboarding; use saved preferences or example defaults

On the Foundry path, pass your model with `--model` (for example, `--model FW-GLM-5.2`).

<Tip>
  `--main` is a retired alias for `--model` in v0.9.0+. Prefer `--model` in new scripts.
</Tip>

FireRouter flags (when using `firerouter`):

* `--anthropic-api-key sk-ant-...`: BYOK for Claude Opus 5 pass-through (where the harness supports it)
* `--routing-preference <1-5>`: savings vs. quality (Claude Code, OpenCode, Pi, VS Code). See [Routing preferences](/ecosystem/firerouter/routing-preferences)

## API key resolution

**Direct Fireworks routing** (`--provider fireworks`)

1. Explicit `--api-key`
2. OS keychain (via `fireconnect login`)
3. Global `~/.fireconnect/config.json` reference
4. `FIREWORKS_API_KEY` environment variable

When `FIREWORKS_API_KEY` is set, `login` uses it without storing a copy. Unset it before `login --api-key`, `--with-token`, or browser sign-in.

Claude Code additionally reads harness-local keys from `~/.claude/settings.json` when FireConnect is already enabled there.

**Fireworks on Microsoft Foundry** (`--provider azure`)

1. Explicit `--api-key`
2. Global `~/.fireconnect/config.json`
3. `AZURE_API_KEY` environment variable

## Migration from earlier syntax

Only needed if you still have old scripts or muscle memory. Day-to-day use is `fireconnect <harness> on --model <id>`.

<AccordionGroup>
  <Accordion title="Pre-0.5.0 syntax">
    | Before                              | After                                         |
    | ----------------------------------- | --------------------------------------------- |
    | `fireconnect on`                    | `fireconnect claude on`                       |
    | `fireconnect off`                   | `fireconnect claude off`                      |
    | `fireconnect status`                | `fireconnect claude status`                   |
    | `fireconnect list`                  | `fireconnect claude status`                   |
    | `fireconnect set --main <id>`       | `fireconnect claude on --model <id>`          |
    | `fireconnect reset`                 | `fireconnect claude on` (re-applies defaults) |
    | `fireconnect on --harness opencode` | `fireconnect opencode on`                     |
  </Accordion>

  <Accordion title="v0.9.2 changes">
    | Feature               | Details                                                  |
    | --------------------- | -------------------------------------------------------- |
    | `claude live`         | tmux split with live usage meter                         |
    | `claude usage --days` | Wider session picker lookback (interactive mode only)    |
    | VS Code               | Uses `chat-completions` API (auto-migrated)              |
    | Cursor                | Hides built-in models; preserves native IDs on re-enable |
  </Accordion>

  <Accordion title="v0.9.1 changes">
    | Feature                   | Details                                                                                                                                                                 |
    | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | `fireconnect upgrade`     | In-place upgrade for curl/git installs; interactive terminals may prompt **Upgrade now?**                                                                               |
    | Seamless upgrade          | From 0.9.0 on, harness settings (including Claude Code) are preserved across upgrade/reinstall                                                                          |
    | Upgrade from before 0.9.0 | With Claude connected, FireConnect restores original Claude settings before updating; run `fireconnect claude on` afterward. In CI, set `FIRECONNECT_AUTO_OFF_CLAUDE=1` |
    | Claude `--interactive`    | Model mapping wizard with fast/non-fast profile toggle; preferences persist per key type                                                                                |
  </Accordion>

  <Accordion title="v0.9.0 changes">
    | Before                               | After                                                  |
    | ------------------------------------ | ------------------------------------------------------ |
    | `fireconnect <harness> model list`   | `fireconnect model list`                               |
    | `fireconnect <harness> model select` | `fireconnect <harness> on --model <id>` or slot flags  |
    | `fireconnect <harness> model reset`  | `fireconnect <harness> on` (re-applies defaults)       |
    | `--main <id>` on `on`                | `--model <id>`                                         |
    | Claude `apiKeyHelper` auth           | `X-Fireworks-Api-Key` custom header in `settings.json` |
  </Accordion>
</AccordionGroup>

## See also

* [FireConnect overview](/ecosystem/fireconnect/overview)
* [Models](/ecosystem/fireconnect/models)
* [Upgrade FireConnect](/ecosystem/fireconnect/overview#upgrade-fireconnect)


# Codex
Source: https://docs.fireworks.ai/ecosystem/fireconnect/codex

Use Fireworks AI models in OpenAI Codex CLI with the FireConnect CLI

[FireConnect](https://github.com/fw-ai/fireconnect) routes the [OpenAI Codex CLI](https://developers.openai.com/codex) through Fireworks AI models via the Responses API. See the [FireConnect overview](/ecosystem/fireconnect/overview) for install and CLI basics.

<Tip>
  **Change models:** `fireconnect codex on --model <id>`. See [Models](/ecosystem/fireconnect/models).
</Tip>

## Prerequisites

* [OpenAI Codex CLI](https://developers.openai.com/codex) installed (0.134+)
* A standard [Fireworks API key](https://app.fireworks.ai/settings/users/api-keys) (`fw_...`)
* FireConnect **v0.9.1+** (see [Install](/ecosystem/fireconnect/overview#install))

<Warning>
  Fire Pass keys (`fpk_...`) are not supported for Codex yet. The `/responses` endpoint requires a standard Fireworks API key.
</Warning>

## Enable Fireworks routing

```bash theme={null}
fireconnect login
fireconnect codex on
```

Or pass the key once:

```bash theme={null}
fireconnect codex on --api-key fw_...
```

After `fireconnect codex on` or `off`, `config.toml` is updated immediately. To use updated routing, exit Codex and resume with `codex resume <id>`, or start a new session.

## Default model

Codex routes a single default model. The default is `kimi-fast-latest`.

```bash theme={null}
fireconnect codex status
```

## What gets written

FireConnect edits `~/.codex/config.toml`:

* Sets root `model_provider` / `model` for Codex 0.134+ (stored as a short slug)
* Adds a `[model_providers.fireworks-ai]` block with `wire_api = "responses"` and a **baked** `experimental_bearer_token` literal (file mode `0600`). Codex reads the key from config; no shell hook is required.

FireConnect snapshots your original `~/.codex/config.toml` before the first change. The snapshot lives in `~/.fireconnect/codex/`. Running `fireconnect codex off` restores it byte-for-byte. Unrelated Codex settings (for example `[[mcp_servers]]`) are preserved via surgical TOML edits.

## Codex model catalog

When you run `fireconnect codex on`, FireConnect fetches your account's serverless catalog and writes Codex-compatible model metadata to `~/.codex/fireworks-model-catalog.json`. It links that file from `config.toml` via `model_catalog_json` so Codex knows display names, context windows, reasoning levels, and tool-calling support for each model.

```bash theme={null}
fireconnect codex status   # shows whether the catalog is linked and on disk
```

The catalog includes serverless models that support tool calling with a non-zero context window, plus curated aliases such as `glm-fast-latest`. Deprecated models are excluded. If catalog generation fails (for example, due to an invalid API key), routing still works but Codex may show limited model metadata until you re-run `fireconnect codex on`.

Browse available models globally:

```bash theme={null}
fireconnect model list --search glm
fireconnect model list --json   # includes IN / OUT pricing where known
```

## FireRouter

Route requests through [FireRouter](/ecosystem/firerouter/overview):

```bash theme={null}
fireconnect codex on --model firerouter
export ANTHROPIC_API_KEY=sk-ant-...   # optional BYOK for pass-through to Claude Opus 5
fireconnect codex on --model firerouter --anthropic-api-key sk-ant-...
```

FireConnect rejects `--model firerouter` with Fire Pass (`fpk_...`) on every harness; use an `fw_...` key. Codex also does not support Fire Pass for direct routing. FireRouter BYOK reads `ANTHROPIC_API_KEY` from your shell (Codex does not support `--routing-preference`).

<Warning>
  **MiniMax models are not supported in Codex.** Codex uses the Fireworks Responses API and may insert assistant messages between `tool_calls` and `tool_results`. MiniMax chat templates require `tool_results` to follow `tool_calls` directly. Use Chat Completions harnesses (for example Claude Code or OpenCode) for MiniMax.
</Warning>

## CLI reference

```bash theme={null}
fireconnect codex on              # Enable Fireworks routing
fireconnect codex off             # Restore original config
fireconnect codex status          # Check current provider and model
fireconnect codex help            # Show harness-specific help
```

Run `fireconnect codex help` for all options.

### Switch models

```bash theme={null}
fireconnect codex on --model glm-5p2
fireconnect codex on --model deepseek-v4-flash
```

Some models expose multiple reasoning levels in the Codex catalog (for example, `glm-5p2` supports `high` and `max`). Pick the model in Codex with `/model` after switching.

### Turn off Fireworks routing

```bash theme={null}
fireconnect codex off
```

This restores your previous `config.toml` from the backup in `~/.fireconnect/codex/`.

### Use a non-default config file

```bash theme={null}
fireconnect codex on --config-path /path/to/config.toml
```

## Fireworks on Microsoft Foundry

Codex supports **Fireworks on Microsoft Foundry** (CLI: `--provider azure` or `on --azure`). See the [FireConnect overview](/ecosystem/fireconnect/microsoft-foundry) and [Microsoft Foundry integration guide](/ecosystem/integrations/azure-foundry) for portal setup.

<Warning>
  Foundry routing requires a standard Azure API key. Fire Pass keys (`fpk_...`) are not supported. FireConnect does not write a Fireworks model catalog on the Foundry path. Set your deployment with `--model`.
</Warning>

### Configure and enable

```bash theme={null}
export AZURE_API_KEY=<your-azure-api-key>

fireconnect configure \
  --provider azure \
  --base-url https://<resource>.services.ai.azure.com \
  --api-key $AZURE_API_KEY

fireconnect codex on --model FW-GLM-5.2
```

One-off routing:

```bash theme={null}
fireconnect codex on --azure --base-url https://<resource>.services.ai.azure.com --model FW-MiniMax-M2.5
```

### What gets written

FireConnect sets `model_provider = "fireworks-azure"` in `config.toml` with a **Fireworks on Microsoft Foundry** provider block pointed at your Foundry endpoint. The Azure API key is baked as a literal when passed with `--api-key`, or referenced via `env_key = "AZURE_API_KEY"` when resolved from the environment.

Pass your Foundry model with `--model` (for example, `FW-GLM-5.2`). Use `fireconnect model list` only for browsing Fireworks serverless models on the direct gateway path.

### Turn off Foundry routing

To switch back to the Fireworks gateway:

```bash theme={null}
fireconnect configure --provider fireworks
fireconnect codex on
```

To remove FireConnect entirely and restore your original `config.toml`:

```bash theme={null}
fireconnect codex off
```

See [Turn off Foundry routing](/ecosystem/fireconnect/microsoft-foundry#turn-off-foundry-routing) for details on global config behavior and `uninstall`.

## Source

FireConnect is open source: [github.com/fw-ai/fireconnect](https://github.com/fw-ai/fireconnect)


# Cursor
Source: https://docs.fireworks.ai/ecosystem/fireconnect/cursor

Use Fireworks AI models in Cursor IDE with the FireConnect CLI

[FireConnect](https://github.com/fw-ai/fireconnect) routes [Cursor](https://cursor.com) through Fireworks AI models via Cursor's OpenAI-compatible BYOK path. See the [FireConnect overview](/ecosystem/fireconnect/overview) for install and CLI basics.

<Tip>
  **Change models:** quit Cursor → `fireconnect cursor on --model <id>`. See [Models](/ecosystem/fireconnect/models).
</Tip>

## Prerequisites

* [Cursor](https://cursor.com) installed
* A [Fireworks API key](https://app.fireworks.ai/settings/users/api-keys) (`fw_...`) or a [Fire Pass](/firepass) key (`fpk_...`)
* FireConnect **v0.9.1+** (see [Install](/ecosystem/fireconnect/overview#install))

<Note>
  **Quit Cursor before `on` or `off`.** FireConnect writes Cursor's SQLite settings database; the IDE must be fully closed. `status` is read-only while Cursor is running.
</Note>

<Warning>
  **FireRouter on Cursor needs workspace BYOK** (no local Anthropic key). FireConnect rejects `--model firerouter` with Fire Pass (`fpk_...`) on every harness; use an `fw_...` key, then `fireconnect cursor on --model firerouter`. See [FireRouter](/ecosystem/firerouter/overview).
</Warning>

## Enable Fireworks routing

Cursor stores AI settings in a SQLite database (`state.vscdb`). **Fully quit Cursor** before running commands that write to it (for example, **Cmd+Q** on macOS or close all Cursor windows on Linux). Otherwise Cursor's in-memory state can overwrite FireConnect's changes.

In an interactive terminal, if Cursor is still running FireConnect asks you to quit it and press Enter to continue. Pass `--force` to write anyway without waiting.

```bash theme={null}
fireconnect login
fireconnect cursor on
```

Or pass the key once:

```bash theme={null}
fireconnect cursor on --api-key fw_...
```

`cursor on` sets **every mode that already exists** in `modelConfig` to the default Fireworks model (non-destructive: it won't create mode entries that aren't already there) and registers the preferred serverless catalog in the picker.

Quit and reopen Cursor for the change to take effect, then open the model picker and choose a Fireworks model.

```bash theme={null}
fireconnect cursor status   # read-only; works while Cursor is running
```

## Browse and pick models

Browse the global catalog, then switch models with `on`:

```bash theme={null}
fireconnect model list --search glm
fireconnect cursor on --model glm-fast-latest
```

`fireconnect model list` and `status` are read-only and work while Cursor is running. Commands that write to `state.vscdb` (`on`, `off`) require Cursor to be quit first.

Short model IDs are expanded to full Fireworks paths automatically.

Cursor modes include `composer` (default), `cmd-k`, `background-composer`, `composer-ensemble`, `plan-execution`, `spec`, `deep-search`, and `quick-agent`. Run `fireconnect cursor status` to see the current model for each mode.

<Warning>
  Cursor enforces an allowlist on the server side. Not every Fireworks model appears in the picker even after you add it. Models such as GLM 5.2 and Kimi K2.6 are known to work; if a model is blocked, Cursor shows an error when you select it.
</Warning>

<Warning>
  **While FireConnect is on, only Fireworks models work.** Built-in Cursor models are hidden. `fireconnect cursor off` restores them. v0.9.2+ preserves your native model IDs (for example `auto-smart`) on re-enable.
</Warning>

## What gets written

FireConnect writes Cursor's BYOK OpenAI settings in the local SQLite state database at `state.vscdb`:

| Setting          | Location                                                                                     |
| ---------------- | -------------------------------------------------------------------------------------------- |
| API key          | `cursorAuth/openAIKey` (plaintext)                                                           |
| Base URL         | `openAIBaseUrl` on the `applicationUser` blob: `https://api.fireworks.ai/inference/v1`       |
| Custom models    | `aiSettings.userAddedModels` + `aiSettings.fireconnectAddedModels` (tracked for clean `off`) |
| Hidden built-ins | `aiSettings.modelOverrideDisabled` (Cursor subscription and built-in models)                 |
| Per-mode model   | `aiSettings.modelConfig[<mode>]`                                                             |

Platform paths for `state.vscdb`:

| Platform | Path                                                                  |
| -------- | --------------------------------------------------------------------- |
| Linux    | `~/.config/Cursor/User/globalStorage/state.vscdb`                     |
| macOS    | `~/Library/Application Support/Cursor/User/globalStorage/state.vscdb` |
| Windows  | `%APPDATA%\Cursor\User\globalStorage\state.vscdb`                     |

FireConnect snapshots your previous Cursor auth state under `~/.fireconnect/cursor/` before the first change. Running `fireconnect cursor off` restores it. `off` only removes models FireConnect registered; your own custom models are preserved.

## Cursor feature coverage

FireConnect configures Cursor's OpenAI BYOK path. Features that route through that path can use Fireworks models.

Some Cursor features (for example, Composer, inline edit, and autocomplete) may still use Cursor's own backend depending on your plan and Cursor version. Test the workflows you care about after enabling.

## Using Fire Pass

Use your `fpk_...` key during `login` or with `--api-key`:

```bash theme={null}
fireconnect cursor on --api-key fpk_...
```

Fire Pass keys default to `kimi-fast-latest`.

## CLI reference

```bash theme={null}
fireconnect cursor on                    # Enable Fireworks routing (quit Cursor first)
fireconnect cursor off                   # Restore your previous Cursor auth state
fireconnect cursor status                # Show provider, auth, modes, and per-mode models
fireconnect cursor help                  # Show harness-specific help
```

Run `fireconnect cursor help` for all options, including `--db-path` (explicit `state.vscdb` path) and `--force` (write even if Cursor appears to be running; not recommended).

### Turn off Fireworks routing

Quit Cursor, then run:

```bash theme={null}
fireconnect cursor off
```

This restores your previous Cursor auth state from the backup in `~/.fireconnect/cursor/`. Quit and reopen Cursor for full effect.

## Manual setup

You can also configure Cursor without FireConnect:

1. In Cursor settings, add a **Custom Model** with a Fireworks model ID (for example, `accounts/fireworks/models/glm-5p2` or a short alias like `glm-fast-latest`).
2. Set **Override OpenAI Base URL** to `https://api.fireworks.ai/inference/v1`.
3. Paste your Fireworks or Fire Pass API key.

FireConnect automates these steps and makes it easy to swap models from the terminal.

## Fireworks on Microsoft Foundry

Cursor supports **Fireworks on Microsoft Foundry** (CLI: `--provider azure` or `on --azure`). See [Microsoft Foundry in FireConnect](/ecosystem/fireconnect/microsoft-foundry) and the [portal setup guide](/ecosystem/integrations/azure-foundry).

<Warning>
  Foundry routing requires a standard Azure API key. Fire Pass keys (`fpk_...`) are not supported. **Quit Cursor** before `on` or `off`.
</Warning>

```bash theme={null}
export AZURE_API_KEY=<your-azure-api-key>

fireconnect configure \
  --provider azure \
  --base-url https://<resource>.services.ai.azure.com \
  --api-key $AZURE_API_KEY

fireconnect cursor on --model FW-GLM-5.2
```

One-off routing without changing global config:

```bash theme={null}
fireconnect cursor on \
  --azure \
  --base-url https://<resource>.services.ai.azure.com \
  --model FW-MiniMax-M2.5
```

Pass your Foundry model with `--model` (for example, `FW-GLM-5.2`), not a Fireworks serverless short ID.

FireConnect points Cursor's OpenAI BYOK override at your Foundry OpenAI-compatible endpoint and registers the deployment in the model picker. `fireconnect model list` only browses the Fireworks serverless catalog on the direct gateway path.

To switch back to the Fireworks gateway:

```bash theme={null}
fireconnect configure --provider fireworks
fireconnect cursor on
```

See [Turn off Foundry routing](/ecosystem/fireconnect/microsoft-foundry#turn-off-foundry-routing) for `off` and global config behavior.

## Source

FireConnect is open source: [github.com/fw-ai/fireconnect](https://github.com/fw-ai/fireconnect)


# Deep Agents
Source: https://docs.fireworks.ai/ecosystem/fireconnect/deepagents

Use Fireworks AI models in LangChain Deep Agents Code with the FireConnect CLI

[FireConnect](https://github.com/fw-ai/fireconnect) routes [LangChain Deep Agents Code](https://docs.langchain.com/oss/python/deepagents/cli) (`dcode`) through Fireworks AI models. See the [FireConnect overview](/ecosystem/fireconnect/overview) for install and CLI basics.

<Tip>
  **Change models:** `fireconnect deepagents on --model <id>`. See [Models](/ecosystem/fireconnect/models).
</Tip>

## Prerequisites

* [Deep Agents Code](https://docs.langchain.com/oss/python/deepagents/cli) (`dcode`) installed
* A [Fireworks API key](https://app.fireworks.ai/settings/users/api-keys) (`fw_...`) or a [Fire Pass](/firepass) key (`fpk_...`)
* FireConnect **v0.9.1+** (see [Install](/ecosystem/fireconnect/overview#install))

## Enable Fireworks routing

```bash theme={null}
fireconnect login
fireconnect deepagents on
```

Or pass the key once:

```bash theme={null}
fireconnect deepagents on --api-key fw_...
```

Restart `dcode` after `on` or `off` if it is already running.

```bash theme={null}
fireconnect deepagents status
```

## Default model

Deep Agents routes a single default model. The default is `kimi-fast-latest`, written to config as `fireworks:<model-id>`.

## What gets written

FireConnect edits `~/.deepagents/config.toml`:

* Sets `[models].default` to `fireworks:<model>` and configures `[models.providers.fireworks]` with the Fireworks OpenAI-compatible base URL (`https://api.fireworks.ai/inference`) and a **baked** `api_key` literal (file mode `0600`)
* Stores your Fireworks API key in the FireConnect keychain via `login` or `deepagents on --api-key`, then bakes it into `config.toml`
* FireConnect does **not** write `~/.deepagents/.state/auth.json`. Use dcode's `/auth` for credentials stored in that file

FireConnect snapshots `config.toml` under `~/.fireconnect/deepagents/` before the first change. Running `fireconnect deepagents off` restores it byte-for-byte.

## Browsing and picking models

```bash theme={null}
fireconnect model list --search glm
fireconnect deepagents on --model glm-5p2
```

Fire Pass keys only list Fire Pass routers. FireConnect rejects `--model firerouter` with Fire Pass (`fpk_...`) on every harness; use an `fw_...` key.

## FireRouter

```bash theme={null}
fireconnect deepagents on --model firerouter
```

Needs workspace BYOK (same as Cursor).

## CLI reference

```bash theme={null}
fireconnect deepagents on              # Enable Fireworks routing
fireconnect deepagents off             # Restore original config
fireconnect deepagents status          # Check current provider and model
fireconnect deepagents help            # Show harness-specific help
```

Run `fireconnect deepagents help` for all options.

### Switch models

```bash theme={null}
fireconnect deepagents on --model glm-5p2
```

### Turn off Fireworks routing

```bash theme={null}
fireconnect deepagents off
```

This restores your previous `config.toml` from the backup in `~/.fireconnect/deepagents/`.

### Use a non-default config file

```bash theme={null}
fireconnect deepagents on --config-path /path/to/config.toml
```

## Fireworks on Microsoft Foundry

Deep Agents supports **Fireworks on Microsoft Foundry** (CLI: `--provider azure` or `on --azure`). See [Microsoft Foundry in FireConnect](/ecosystem/fireconnect/microsoft-foundry) and the [portal setup guide](/ecosystem/integrations/azure-foundry).

<Warning>
  Foundry routing requires a standard Azure API key. Fire Pass keys (`fpk_...`) are not supported.
</Warning>

```bash theme={null}
export AZURE_API_KEY=<your-azure-api-key>

fireconnect configure \
  --provider azure \
  --base-url https://<resource>.services.ai.azure.com \
  --api-key $AZURE_API_KEY

fireconnect deepagents on --model FW-GLM-5.2
```

Pass your Foundry model with `--model` (for example, `FW-GLM-5.2`), not a Fireworks serverless short ID.

FireConnect writes a `fireworks-azure` provider in `config.toml` with the Foundry base URL and deployment name. `fireconnect model list` only browses the Fireworks serverless catalog on the direct gateway path.

To switch back to the Fireworks gateway:

```bash theme={null}
fireconnect configure --provider fireworks
fireconnect deepagents on
```

See [Turn off Foundry routing](/ecosystem/fireconnect/microsoft-foundry#turn-off-foundry-routing) for `off` and global config behavior.

## Source

FireConnect is open source: [github.com/fw-ai/fireconnect](https://github.com/fw-ai/fireconnect)


# Side-by-side demo
Source: https://docs.fireworks.ai/ecosystem/fireconnect/demo

Race Claude Code on Anthropic against Fireworks on the same prompt with fireconnect demo

Not ready to rewire your whole setup? **`fireconnect demo`** runs the same code-generation prompt through both sides: your current Claude Code provider on Anthropic, and a Fireworks model on the challenger side. It then puts the results next to each other so you can judge speed, cost, and output quality yourself.

The demo is read-only against your environment. It uses throwaway config dirs, so your `~/.claude/settings.json` is never touched.

## What you get

1. **Live terminal race**: both models stream code in a split-pane TUI with real token counts and wall-clock timers.
2. **Browser comparison**: both generated apps run side by side, with measured speed and cost at the bottom.
3. **Shareable artifacts**: outputs land in `./fireconnect-demo/` (`compare.html`, `result.json`, stream logs).

### Browser comparison

Side-by-side runnable apps with measured speed and cost:

<Frame>
  <img alt="FireConnect demo browser comparison page with Tetris apps from Anthropic and Fireworks side by side and speed and cost metrics" />
</Frame>

### Terminal race

Split-pane TUI while both models stream the same prompt:

<Frame>
  <img alt="Animated capture of a real FireConnect demo split-pane terminal race: Anthropic Claude Sonnet vs Fireworks GLM 5.2 Fast on a Tetris prompt" />
</Frame>

## Prerequisites

* FireConnect installed ([overview](/ecosystem/fireconnect/overview#install))
* [Claude Code](https://claude.ai/code) CLI (`claude`) on your `PATH`
* An **Anthropic API key** for the incumbent side (`ANTHROPIC_API_KEY`, or pass `--anthropic-key`)
* A **Fireworks API key** for the challenger side (`fireconnect login` or `FIREWORKS_API_KEY`)

<Tip>
  Run `fireconnect login` first so the Fireworks side resolves your key automatically. The demo only supports **Claude Code** today.
</Tip>

## Run the demo

```bash theme={null}
fireconnect demo
```

That runs the default **Tetris** preset: both sides get the same prompt to build a playable game in a single HTML file.

### Try other presets

```bash theme={null}
fireconnect demo --prompt snake
fireconnect demo --prompt tictactoe
fireconnect demo --prompt clock
```

Presets: `tetris` (default), `snake`, `tictactoe`, `clock`, or `custom` with your own task text.

### Pick the Fireworks model

```bash theme={null}
fireconnect demo --challenger glm-5p2-fast
```

Default challenger is `glm-5p2-fast`. Pass any serverless ID from [Models](/ecosystem/fireconnect/models) (same IDs as `fireconnect claude on --model`).

### Non-interactive / CI-friendly

```bash theme={null}
fireconnect demo --yes --no-open --json
```

| Flag                        | What it does                                              |
| --------------------------- | --------------------------------------------------------- |
| `--yes`                     | Skip the setup form                                       |
| `--no-open`                 | Do not open a browser; write outputs to disk only         |
| `--json`                    | Print a machine-readable result to stdout (skips the TUI) |
| `--out <dir>`               | Output directory (default: `./fireconnect-demo/`)         |
| `--anthropic-model <alias>` | Incumbent model: `opus` (default), `sonnet`, or `haiku`   |
| `--anthropic-key <key>`     | Anthropic API key for the incumbent side                  |
| `--api-key <key>`           | Fireworks API key for the challenger side                 |

## Clean up

```bash theme={null}
fireconnect demo clean          # prompts before deleting ./fireconnect-demo/
fireconnect demo clean --yes    # delete without prompting
fireconnect demo clean --out /path/to/output
```

`demo clean` only removes directories that contain demo markers (`result.json`, `compare.html`, etc.), so it will not delete an unrelated folder you pointed `--out` at by mistake.

## After the demo

Liked what you saw on the Fireworks side? Wire it into your daily driver:

```bash theme={null}
fireconnect claude on --model glm-5p2-fast
```

More IDs and latest vs fast guidance: [Models](/ecosystem/fireconnect/models).

Want automatic cost routing instead of a fixed model? Try [FireRouter](/ecosystem/firerouter/overview):

```bash theme={null}
fireconnect claude on --model firerouter
```

## How it works

* Each side runs real `claude -p` in an isolated temporary config directory.
* Numbers in the comparison strip are **measured from the run**, not list-price estimates.
* If one side fails to finish, the page says so instead of fabricating a winner.
* Open `compare.html` from the output folder anytime. It inlines both apps and works offline.

## Source

The demo ships with FireConnect: [github.com/fw-ai/fireconnect](https://github.com/fw-ai/fireconnect)

## See also

* [FireConnect overview](/ecosystem/fireconnect/overview)
* [Models](/ecosystem/fireconnect/models)
* [CLI reference](/ecosystem/fireconnect/cli-reference)


# Microsoft Foundry
Source: https://docs.fireworks.ai/ecosystem/fireconnect/microsoft-foundry

Route FireConnect harnesses through Fireworks models deployed in your Azure subscription

FireConnect can route supported harnesses through [Fireworks on Microsoft Foundry](/ecosystem/integrations/azure-foundry) instead of the Fireworks gateway. Usage is billed through Azure and counts toward your Microsoft Azure Consumption Commitment (MACC) where applicable.

<Card title="Microsoft Foundry portal setup" icon="microsoft" href="/ecosystem/integrations/azure-foundry">
  Enable Fireworks on Foundry, create a deployment, and find your project endpoint. Start here if you have not set up a Foundry resource yet.
</Card>

<Note>
  **CLI terminology:** The Foundry provider is `--provider azure` (or `on --azure`). Harness configs display the label **Fireworks on Microsoft Foundry**. With Foundry, `--model` is the model you deployed in Azure (for example, `FW-GLM-5.2`), not a Fireworks serverless short ID like `glm-fast-latest`.
</Note>

## Supported harnesses

| Harness     | Azure in FireConnect | Notes                                                         |
| ----------- | -------------------- | ------------------------------------------------------------- |
| OpenCode    | Yes                  | `fireworks-azure` provider in `opencode.json`                 |
| Codex       | Yes                  | `fireworks-azure` block in `config.toml`                      |
| Pi          | Yes                  | `fireworks-azure` provider in `models.json`                   |
| Cursor      | Yes                  | OpenAI BYOK override pointed at Foundry                       |
| VS Code     | Yes                  | Custom chat-completions endpoint in `chatLanguageModels.json` |
| Deep Agents | Yes                  | `fireworks-azure` provider in `config.toml`                   |
| Claude Code | Not yet              | `claude on` always wires direct Fireworks today               |

Claude Code is the remaining gap. Every other harness in the matrix supports Foundry routing in FireConnect v0.9.0+.

## Prerequisites

* A Microsoft Foundry resource with at least one Fireworks model deployment (for example, `FW-GLM-5.2` or `FW-MiniMax-M2.5`)
* Your Foundry resource endpoint and Azure API key from the [Microsoft Foundry portal](https://ai.azure.com/)
* A supported harness installed locally
* FireConnect v0.9.0+ (see [Overview: Install](/ecosystem/fireconnect/overview#install))

<Warning>
  Use an **Azure API key** from Foundry, not a Fireworks key (`fw_...`) or Fire Pass key (`fpk_...`). Fire Pass is not supported on the Foundry path.
</Warning>

## Configure once, then enable harnesses

`fireconnect configure` sets the **Foundry provider and endpoint**. It does **not** set your Fireworks API key. Use `fireconnect login` for that when routing through the Fireworks gateway.

```bash theme={null}
export AZURE_API_KEY=<your-azure-api-key>

fireconnect configure \
  --provider azure \
  --base-url https://<resource>.services.ai.azure.com \
  --api-key $AZURE_API_KEY
```

FireConnect stores a top-level `provider` and `azure` block in `~/.fireconnect/config.json`. After configuring, enable harnesses normally:

```bash theme={null}
fireconnect opencode on --model FW-GLM-5.2
fireconnect codex on --model FW-GLM-5.2
fireconnect pi on --model FW-GLM-5.2
fireconnect cursor on --model FW-GLM-5.2
fireconnect vscode on --model FW-GLM-5.2
fireconnect deepagents on --model FW-GLM-5.2
```

If you omit `--model`, FireConnect defaults to `FW-GLM-5.2`.

<Warning>
  **Cursor and VS Code:** fully quit the IDE before `on` or `off`. FireConnect writes SQLite state. In an interactive terminal it waits for you to quit; pass `--force` to write anyway.
</Warning>

### Endpoint normalization

Pass your Foundry endpoint to `--base-url`. FireConnect normalizes whatever you paste to the correct OpenAI-compatible base at `https://<resource>.services.ai.azure.com/openai/v1`:

* Bare resource root (`https://<resource>.services.ai.azure.com`)
* Portal **project endpoint** (`.../api/projects/<name>`)
* Foundry **Models** route (`.../models`)
* An already-correct base (`.../openai/v1`)

Find the endpoint in the Microsoft Foundry portal under **Project settings**.

### API key storage

* Pass `--api-key` to write the Azure key into `~/.fireconnect/config.json`
* Or export `AZURE_API_KEY` before `configure`. FireConnect stores an environment reference instead.

## One-off Foundry routing

Route a single harness through Foundry without changing global config:

```bash theme={null}
fireconnect opencode on \
  --azure \
  --base-url https://<resource>.services.ai.azure.com \
  --api-key $AZURE_API_KEY \
  --model FW-MiniMax-M2.5
```

If global config already has a Foundry endpoint, `--azure` alone reuses it:

```bash theme={null}
fireconnect cursor on --azure --model FW-GLM-5.2
```

## What each harness writes

Each harness writes a dedicated Foundry config distinct from the Fireworks gateway. `off` restores your original config byte-for-byte.

| Harness     | Config file                                 | Provider ID                  | Notes                                                                            |
| ----------- | ------------------------------------------- | ---------------------------- | -------------------------------------------------------------------------------- |
| OpenCode    | `~/.config/opencode/opencode.json`          | `fireworks-azure/FW-GLM-5.2` | `@ai-sdk/openai-compatible` adapter; `options.baseURL` + `options.apiKey`        |
| Codex       | `~/.codex/config.toml`                      | `fireworks-azure`            | `wire_api = "chat"`; bearer or `env_key = "AZURE_API_KEY"`                       |
| Pi          | `~/.pi/agent/models.json` + `settings.json` | `fireworks-azure`            | `openai-completions` provider; key as literal or `$AZURE_API_KEY` in `auth.json` |
| Cursor      | `state.vscdb`                               | Foundry deployment name      | OpenAI BYOK base URL + Azure key; one deployment in the picker                   |
| VS Code     | `chatLanguageModels.json` + `state.vscdb`   | Foundry deployment name      | Chat-completions endpoint; Azure key in secret storage                           |
| Deep Agents | `~/.deepagents/config.toml`                 | `fireworks-azure:FW-GLM-5.2` | `openai-completions` provider with Foundry base URL                              |

`fireconnect <harness> status` reports `azure` as the provider along with the endpoint and model.

<Warning>
  `fireconnect model list` browses the **Fireworks serverless catalog** only. It does not list Foundry deployments. With `--provider azure`, set your model with `--model` on `on`.
</Warning>

## Turn off Foundry routing

There are two ways to stop using Microsoft Foundry, depending on what you want next.

### Switch back to direct Fireworks

Use this when you want to keep FireConnect enabled but route through the Fireworks gateway again instead of your Foundry deployment.

<Warning>
  While `~/.fireconnect/config.json` has `provider: azure`, running `fireconnect <harness> on` **without** `--azure` still routes through Foundry. Change the global provider first.
</Warning>

```bash theme={null}
fireconnect configure --provider fireworks

fireconnect login                    # if you have not signed in yet
fireconnect opencode on
fireconnect codex on
fireconnect pi on
fireconnect cursor on
fireconnect vscode on
fireconnect deepagents on
```

Re-running `on` replaces the Foundry config with the normal Fireworks gateway config. You do **not** need to run `off` first.

The Azure endpoint and key remain stored in `~/.fireconnect/config.json` but are unused while `provider` is `fireworks`. They are used again if you run `configure --provider azure` later.

### Remove FireConnect from a harness entirely

Use `off` to restore the config snapshot from **before FireConnect was first enabled** for that harness. Your original provider settings are restored byte-for-byte:

```bash theme={null}
fireconnect pi off
fireconnect opencode off
fireconnect codex off
fireconnect cursor off
fireconnect vscode off
fireconnect deepagents off
```

`off` removes Foundry wiring from harness config files. It does **not** change the global `provider` field in `~/.fireconnect/config.json`. If `provider` is still `azure`, the next `on` will route through Foundry again unless you run `configure --provider fireworks` first.

Restart the harness after `off` if it is already running. For Cursor and VS Code, quit the IDE before `off`.

### Remove FireConnect everywhere

```bash theme={null}
fireconnect uninstall
```

Disables all harnesses, restores every backup, and removes the CLI.

| Goal                                        | Commands                                                                     |
| ------------------------------------------- | ---------------------------------------------------------------------------- |
| Stop Foundry, keep FireConnect on Fireworks | `fireconnect configure --provider fireworks` then `fireconnect <harness> on` |
| Undo FireConnect for one harness            | `fireconnect <harness> off`                                                  |
| Undo FireConnect on all harnesses           | `fireconnect uninstall`                                                      |

## Verify routing

```bash theme={null}
fireconnect opencode status   # provider=azure, base URL, model
fireconnect codex status
fireconnect pi status
fireconnect cursor status
fireconnect vscode status
fireconnect deepagents status
```

## Per-harness guides

* [OpenCode](/ecosystem/fireconnect/opencode#fireworks-on-microsoft-foundry)
* [Codex](/ecosystem/fireconnect/codex#fireworks-on-microsoft-foundry)
* [Pi](/ecosystem/fireconnect/pi#fireworks-on-microsoft-foundry)
* [Cursor](/ecosystem/fireconnect/cursor#fireworks-on-microsoft-foundry)
* [VS Code](/ecosystem/fireconnect/vscode#fireworks-on-microsoft-foundry)
* [Deep Agents](/ecosystem/fireconnect/deepagents#fireworks-on-microsoft-foundry)

## Source

FireConnect is open source: [github.com/fw-ai/fireconnect](https://github.com/fw-ai/fireconnect)


# Models
Source: https://docs.fireworks.ai/ecosystem/fireconnect/models

Switch models in FireConnect harnesses: latest vs fast, when to pick each ID, and US-only routers

```bash theme={null}
fireconnect model list --search kimi
fireconnect <harness> on --model <id>
# Cursor / VS Code: quit the IDE before on, then reopen
# Other harnesses: restart after on
```

`<harness>` is one of: `claude`, `opencode`, `codex`, `pi`, `cursor`, `vscode`, `deepagents`.

There is no `model select` command. Always use `on --model`.

```bash theme={null}
fireconnect opencode on --model kimi-fast-latest
fireconnect claude on --model firerouter
fireconnect claude on --interactive              # Claude: map all six slots
fireconnect claude on --opus glm-fast-latest --sonnet glm-fast-latest
```

Claude Code: `--model` sets **main** only; use slot flags or `--interactive` for the rest. Claude adds `[1m]` on 1M-context models (not `subagent`). Re-running `on` without flags keeps your current mapping.

| Harness                          | Apply the change                                    |
| -------------------------------- | --------------------------------------------------- |
| Cursor, VS Code                  | **Quit the IDE**, run `on --model`, then reopen     |
| Claude Code                      | New session, or `/exit` then `claude --resume <id>` |
| OpenCode, Codex, Pi, Deep Agents | Restart the CLI after `on`                          |

## Latest vs Fast

FireConnect short IDs follow the same [Serverless serving paths](/serverless/serving-paths) as the API:

| Kind                       | Example IDs                           | When to use                                                                                                           |
| -------------------------- | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| **Latest** (standard path) | `kimi-latest`, `glm-latest`           | Best price/quality; tracks current model versions                                                                     |
| **Fast**                   | `kimi-fast-latest`, `glm-fast-latest` | Interactive coding where token speed matters. Same model quality as latest, higher \$/token, aims for **100+ tok/s**. |
| **Pinned**                 | `kimi-k3`, `glm-5p2`, `kimi-k3-fast`  | Stable ID that does not track new versions.                                                                           |

Prefer `*-latest` / `*-fast-latest` unless you need a pin. Browse live IDs and prices with `fireconnect model list`.

## Which model when

| ID                  | Use when                                                        | Notes                                                  |
| ------------------- | --------------------------------------------------------------- | ------------------------------------------------------ |
| `kimi-fast-latest`  | Default interactive coding; screenshots / UI                    | Vision. Default for most harnesses and Claude `fable`. |
| `kimi-latest`       | Strong agentic coding with vision, lower \$/token than Fast     | Vision. Standard path.                                 |
| `glm-fast-latest`   | Fast text-only agent loops (Claude `opus` / `sonnet` defaults)  | Text-only. 1M context.                                 |
| `glm-latest`        | Cheaper text-only coding / long context                         | Text-only. Same family as Fast at standard price.      |
| `deepseek-v4-flash` | Haiku / subagent / high-volume background work                  | Text-only. Lowest cost in the coding catalog.          |
| `firerouter`        | Auto-route easy work to open models, hard work to Claude Opus 5 | See [FireRouter](/ecosystem/firerouter/overview).      |

Also useful from `fireconnect model list`: `minimax-latest` / `qwen-plus-latest` (cheaper vision), `deepseek-v4-pro` (stronger text coding than Flash), `kimi-k2p7-code` / `kimi-k2p7-code-fast` (code-focused Kimi).

<Warning>
  GLM and DeepSeek Flash/Pro are **text-only**. Pasting images on those slots in Claude Code can break the session. Recover with `/rewind`, or use a Kimi ID. See [Claude Code troubleshooting](/ecosystem/fireconnect/claude-code#troubleshooting).
</Warning>

## US-only

For US-only inference (compliance), pass the **full router path** with `on --model`. FireConnect does not yet expand the short US slugs to routers (unknown short IDs map to `accounts/fireworks/models/...`, which is wrong for these endpoints).

| Model             | Full ID                                      |
| ----------------- | -------------------------------------------- |
| Kimi K3 (US)      | `accounts/fireworks/routers/kimi-k3-us`      |
| GLM 5.2 Fast (US) | `accounts/fireworks/routers/glm-5p2-fast-us` |

```bash theme={null}
fireconnect claude on --model accounts/fireworks/routers/kimi-k3-us
fireconnect opencode on --model accounts/fireworks/routers/glm-5p2-fast-us
```

US-only endpoints are a **10% premium** except GLM 5.2 Fast US (same price as global Fast). Details: [US-only Serverless](/serverless/us-only-serverless).

## Limits

* **Fire Pass** (`fpk_...`): Claude defaults to `kimi-fast-latest`. Catalog is limited. Not on Codex. FireConnect rejects `--model firerouter` with a Fire Pass key on **every** harness; use an `fw_...` account key for FireRouter.
* **Foundry**: pass the Azure deployment name (`FW-GLM-5.2`), not a short ID. Claude Code does not support Foundry.
* **FireRouter**: standard key only (`fw_...`), not Fire Pass. Cursor / Deep Agents need workspace BYOK.

## Troubleshooting

<AccordionGroup>
  <Accordion title="Nothing changed after on --model">
    Restart the harness. For Cursor and VS Code, fully quit the IDE **before** running `on`.
  </Accordion>

  <Accordion title="Stuck on FireRouter / Fire Pass">
    FireConnect rejects `--model firerouter` when your stored key is Fire Pass (`fpk_...`), on every harness. Switch to an `fw_...` account key, then run `fireconnect <harness> on --model firerouter` and restart the harness.
  </Accordion>

  <Accordion title="Foundry says the model was not found">
    Pass the Azure **deployment name** (for example `FW-GLM-5.2`), not a serverless short ID.
  </Accordion>
</AccordionGroup>

## See also

* [FireConnect overview](/ecosystem/fireconnect/overview)
* [CLI reference](/ecosystem/fireconnect/cli-reference)
* [Harness guides](/ecosystem/fireconnect/claude-code)


# OpenCode
Source: https://docs.fireworks.ai/ecosystem/fireconnect/opencode

Use Fireworks AI models in OpenCode with the FireConnect CLI

[FireConnect](https://github.com/fw-ai/fireconnect) routes [OpenCode](https://opencode.ai) through Fireworks AI models. See the [FireConnect overview](/ecosystem/fireconnect/overview) for install and CLI basics.

<Tip>
  **Change models:** `fireconnect opencode on --model <id>`. See [Models](/ecosystem/fireconnect/models).
</Tip>

## Prerequisites

* [OpenCode](https://opencode.ai) installed
* A [Fireworks API key](https://app.fireworks.ai/settings/users/api-keys) (`fw_...`) or a [Fire Pass](/firepass) key (`fpk_...`)
* FireConnect **v0.9.1+** (see [Install](/ecosystem/fireconnect/overview#install))

## Enable Fireworks routing

```bash theme={null}
fireconnect login
fireconnect opencode on
```

Restart OpenCode after enabling, then confirm routing:

```bash theme={null}
fireconnect opencode status
```

## Using Fire Pass

Use your `fpk_...` key during `login` or with `--api-key`:

```bash theme={null}
fireconnect opencode on --api-key fpk_...
```

FireConnect detects Fire Pass keys and defaults OpenCode to `kimi-fast-latest`.

## Default model

OpenCode routes a single default model (no opus/sonnet/haiku alias slots). The default is `kimi-fast-latest`, written to config as `fireworks-ai/kimi-fast-latest`.

Short model IDs like `glm-5p2` are expanded to full Fireworks paths (for example, `accounts/fireworks/models/glm-5p2`).

## What gets written

FireConnect merges a `fireworks-ai` provider block into `~/.config/opencode/opencode.json`:

* An OpenAI-compatible adapter pointed at `https://api.fireworks.ai/inference/v1`
* A default `model` set to `fireworks-ai/<model-id>` (for example, `fireworks-ai/glm-fast-latest`)
* `options.apiKey` as a **baked plaintext literal** (file mode `0600`)
* The **preferred serverless catalog** registered in the provider's `models` for OpenCode's `/model` picker

FireConnect snapshots your original `opencode.json` before the first change. The snapshot lives in `~/.fireconnect/opencode/`. Running `fireconnect opencode off` restores the file byte-for-byte. OpenCode's `auth.json` is never touched.

## Browsing and picking models

```bash theme={null}
fireconnect model list --search glm
fireconnect opencode on --model glm-5p2
```

`fireconnect model list` uses your stored key (keychain, config, or `FIREWORKS_API_KEY`). Fire Pass keys only list Fire Pass routers. FireConnect rejects `--model firerouter` with Fire Pass (`fpk_...`) on every harness; use an `fw_...` key.

## FireRouter

```bash theme={null}
fireconnect opencode on --model firerouter
```

Pass `--anthropic-api-key sk-ant-...` when your workspace does not have Anthropic BYOK provisioned server-side.

## CLI reference

```bash theme={null}
fireconnect opencode on              # Enable Fireworks routing
fireconnect opencode off             # Restore original config
fireconnect opencode status          # Check current provider and model
fireconnect opencode help            # Show harness-specific help
```

Run `fireconnect opencode help` for all options.

### Switch models

```bash theme={null}
fireconnect opencode on --model glm-5p2
```

### Turn off Fireworks routing

```bash theme={null}
fireconnect opencode off
```

This restores your previous `opencode.json` from the backup in `~/.fireconnect/opencode/`.

### Use a non-default config file

```bash theme={null}
fireconnect opencode on --config-path /path/to/opencode.json
```

## Fireworks on Microsoft Foundry

OpenCode supports **Fireworks on Microsoft Foundry** (CLI: `--provider azure` or `on --azure`). See the [FireConnect overview](/ecosystem/fireconnect/microsoft-foundry) and [Microsoft Foundry integration guide](/ecosystem/integrations/azure-foundry) for portal setup.

### Configure and enable

```bash theme={null}
export AZURE_API_KEY=<your-azure-api-key>

fireconnect configure \
  --provider azure \
  --base-url https://<resource>.services.ai.azure.com \
  --api-key $AZURE_API_KEY

fireconnect opencode on --model FW-GLM-5.2
```

One-off routing without changing global config:

```bash theme={null}
fireconnect opencode on \
  --azure \
  --base-url https://<resource>.services.ai.azure.com \
  --model FW-MiniMax-M2.5
```

Pass your Foundry model with `--model` (for example, `FW-GLM-5.2`), not a Fireworks serverless short ID like `glm-latest`.

### What gets written

FireConnect adds a `fireworks-azure` provider labeled **Fireworks on Microsoft Foundry** to `opencode.json`, pointed at your Foundry OpenAI-compatible endpoint (`.../openai/v1`). The default model reference becomes `fireworks-azure/FW-GLM-5.2`.

Use `fireconnect model list` only for browsing Fireworks serverless models on the direct gateway path. With Foundry, switch models with `on --model FW-GLM-5.2`.

### Turn off Foundry routing

To switch back to the Fireworks gateway, change the global provider and re-enable:

```bash theme={null}
fireconnect configure --provider fireworks
fireconnect opencode on
```

To remove FireConnect entirely and restore your original `opencode.json`:

```bash theme={null}
fireconnect opencode off
```

See [Turn off Foundry routing](/ecosystem/fireconnect/microsoft-foundry#turn-off-foundry-routing) for details on global config behavior and `uninstall`.

## Built-in provider connection

OpenCode also supports connecting to Fireworks directly without FireConnect:

1. Type `/connect` in OpenCode and search for **fireworks.ai**
2. Paste your Fireworks API key and press Enter
3. Type `/models` and select a model (for Fire Pass, choose a supported model such as **GLM Fast Latest**)

## Source

FireConnect is open source: [github.com/fw-ai/fireconnect](https://github.com/fw-ai/fireconnect)


# Overview
Source: https://docs.fireworks.ai/ecosystem/fireconnect/overview

Route Claude Code, OpenCode, Codex, Pi, Cursor, VS Code, and Deep Agents through Fireworks AI or Microsoft Foundry models

[FireConnect](https://github.com/fw-ai/fireconnect) is an open-source CLI that routes agentic coding harnesses through Fireworks models. Install once, sign in once, then flip any supported harness on or off without hand-editing config files.

Choose where inference runs:

* **Direct Fireworks routing** (default): the [Fireworks gateway](https://fireworks.ai). Sign in with `fireconnect login` or use a Fireworks API key (`fw_...`) or [Fire Pass](/firepass) key (`fpk_...`).
* **Fireworks on Microsoft Foundry**: models in your Azure subscription, billed through Azure. See [Microsoft Foundry](/ecosystem/fireconnect/microsoft-foundry).

<Tip>
  New here? Install and sign in below, then open the [harness guide](#choose-your-harness) for the tool you use. Try the [side-by-side demo](/ecosystem/fireconnect/demo) if you want to compare before switching.
</Tip>

## Prerequisites

* A [Fireworks API key](https://app.fireworks.ai/settings/users/api-keys) (`fw_...`) or [Fire Pass](/firepass) key (`fpk_...`) for direct routing
* For Foundry: Azure resource, API key, and deployment. See [portal setup](/ecosystem/integrations/azure-foundry).
* Node.js 18+
* At least one supported harness installed locally

## Install

```bash theme={null}
curl -fsSL https://raw.githubusercontent.com/fw-ai/fireconnect/main/install.sh | bash
fireconnect login
```

The installer is a **bash** script. It clones the CLI to `~/.fireconnect/cli`, installs dependencies, and adds `~/.local/bin/fireconnect`. It does **not** sign you in or write harness settings.

After `login`, open your [harness guide](#choose-your-harness) and run `fireconnect <harness> on`. You can also pass `--api-key` or set `FIREWORKS_API_KEY` instead of `login`.

<Note>
  **Windows:** run from Git Bash with the same command. Piping through PowerShell corrupts line endings (`set: pipefail\r: invalid option name`).
</Note>

### Upgrade FireConnect

```bash theme={null}
fireconnect upgrade
```

Or re-run the install curl above. On v0.9.0+, harness settings stay connected across upgrade.

Check version with `fireconnect --version`. See [CLI reference: Migration](/ecosystem/fireconnect/cli-reference#migration-from-earlier-syntax) for renames.

## Sign in

```bash theme={null}
fireconnect login     # browser sign-in or paste a key
fireconnect logout    # clear stored credentials
fireconnect status    # sign-in state and key storage
```

Fire Pass keys (`fpk_...`) work during `login` or `on`. FireConnect detects the key type and applies the correct defaults.

See [CLI reference: Sign in options](/ecosystem/fireconnect/cli-reference#sign-in-options) for `--with-token`, `--account`, `logout --revoke`, and `configure`.

## Choose your harness

After install and sign-in, open the guide for the tool you use. Each page covers `on` / `off`, models, and harness-specific notes.

<CardGroup>
  <Card title="Claude Code" icon="terminal" href="/ecosystem/fireconnect/claude-code">
    Six model slots + usage meter
  </Card>

  <Card title="OpenCode" icon="terminal" href="/ecosystem/fireconnect/opencode">
    OpenAI-compatible adapter in `opencode.json`
  </Card>

  <Card title="Codex" icon="terminal" href="/ecosystem/fireconnect/codex">
    OpenAI Codex CLI via the Responses API
  </Card>

  <Card title="Pi" icon="terminal" href="/ecosystem/fireconnect/pi">
    Pi agent settings and auth
  </Card>

  <Card title="Cursor" icon="laptop-code" href="/ecosystem/fireconnect/cursor">
    OpenAI BYOK settings for Cursor IDE
  </Card>

  <Card title="VS Code" icon="code" href="/ecosystem/fireconnect/vscode">
    GitHub Copilot Chat custom endpoint
  </Card>

  <Card title="Deep Agents" icon="robot" href="/ecosystem/fireconnect/deepagents">
    LangChain Deep Agents Code (`dcode`)
  </Card>
</CardGroup>

## Models

```bash theme={null}
fireconnect model list --search glm
fireconnect <harness> on --model glm-fast-latest
# Cursor / VS Code: quit before on; others: restart after
```

`<harness>` is one of: `claude`, `opencode`, `codex`, `pi`, `cursor`, `vscode`, `deepagents`. Full walkthrough: **[Models](/ecosystem/fireconnect/models)**.

## Harness support

| Harness     | Fireworks gateway | Fire Pass | Microsoft Foundry | [FireRouter](/ecosystem/firerouter/overview) | Guide                                             |
| ----------- | :---------------: | :-------: | :---------------: | :------------------------------------------: | ------------------------------------------------- |
| Claude Code |        Yes        |    Yes    |         No        |                      Yes                     | [Claude Code](/ecosystem/fireconnect/claude-code) |
| OpenCode    |        Yes        |    Yes    |        Yes        |                      Yes                     | [OpenCode](/ecosystem/fireconnect/opencode)       |
| Codex       |        Yes        |     No    |        Yes        |                      Yes                     | [Codex](/ecosystem/fireconnect/codex)             |
| Pi          |        Yes        |    Yes    |        Yes        |                      Yes                     | [Pi](/ecosystem/fireconnect/pi)                   |
| Cursor      |        Yes        |    Yes    |        Yes        |                Workspace BYOK                | [Cursor](/ecosystem/fireconnect/cursor)           |
| VS Code     |        Yes        |    Yes    |        Yes        |                      Yes                     | [VS Code](/ecosystem/fireconnect/vscode)          |
| Deep Agents |        Yes        |    Yes    |        Yes        |                Workspace BYOK                | [Deep Agents](/ecosystem/fireconnect/deepagents)  |

**Notes**

* **Foundry**: not on Claude Code yet.
* **Fire Pass**: not on Codex or Foundry. FireConnect rejects `--model firerouter` with a Fire Pass (`fpk_...`) key on **every** harness; use an `fw_...` account key for FireRouter.
* **FireRouter**: Cursor and Deep Agents need workspace BYOK.
* **Web search MCP**: Claude Code auto-install only; any harness can add the HTTP MCP manually. See [WebSearch MCP](/ecosystem/fireconnect/websearch-mcp).
* **Cursor / VS Code**: quit the IDE before `on` or `off`. `status` is read-only.

## FireRouter

[FireRouter](/ecosystem/firerouter/overview) automatically routes simple requests to cheaper open models and hard requests to Claude Opus 5. In FireConnect, select it like any other model:

```bash theme={null}
fireconnect <harness> on --model firerouter
```

Requires a standard Fireworks key (`fw_...`), not Fire Pass. Cursor and Deep Agents also need workspace BYOK. See the [FireRouter overview](/ecosystem/firerouter/overview#fireconnect) and [Harness support](#harness-support).

## Source

FireConnect is open source: [github.com/fw-ai/fireconnect](https://github.com/fw-ai/fireconnect)

## See also

* [CLI reference](/ecosystem/fireconnect/cli-reference)
* [Microsoft Foundry](/ecosystem/fireconnect/microsoft-foundry)
* [WebSearch MCP](/ecosystem/fireconnect/websearch-mcp)
* [Side-by-side demo](/ecosystem/fireconnect/demo)


# Pi
Source: https://docs.fireworks.ai/ecosystem/fireconnect/pi

Use Fireworks AI models in Pi with the FireConnect CLI

[FireConnect](https://github.com/fw-ai/fireconnect) routes [Pi](https://pi.dev) through Fireworks AI models. See the [FireConnect overview](/ecosystem/fireconnect/overview) for install and CLI basics.

<Tip>
  **Change models:** `fireconnect pi on --model <id>`. See [Models](/ecosystem/fireconnect/models).
</Tip>

## Prerequisites

* [Pi](https://pi.dev) installed
* A [Fireworks API key](https://app.fireworks.ai/settings/users/api-keys) (`fw_...`) or a [Fire Pass](/firepass) key (`fpk_...`)
* FireConnect **v0.9.1+** (see [Install](/ecosystem/fireconnect/overview#install))

## Enable Fireworks routing

```bash theme={null}
fireconnect login
fireconnect pi on
```

Restart Pi after enabling if it is already running.

```bash theme={null}
fireconnect pi status
```

## Using Fire Pass

Use your `fpk_...` key during `login` or with `--api-key`:

```bash theme={null}
fireconnect pi on --api-key fpk_...
```

FireConnect detects Fire Pass keys and defaults Pi to `kimi-fast-latest`.

## Default model

Pi routes a single default model. The default is `kimi-fast-latest`.

## What gets written

FireConnect:

* Sets `defaultProvider` / `defaultModel` in `~/.pi/agent/settings.json`
* Stores a **baked plaintext literal** in `fireworks.key` in `~/.pi/agent/auth.json` (mode `0600`)
* Registers the **preferred serverless catalog** in `~/.pi/agent/models.json` for Pi's `/model` picker

FireConnect snapshots `settings.json`, `auth.json`, and `models.json` under `~/.fireconnect/pi/` before the first change. Running `fireconnect pi off` restores them.

## Browsing and picking models

```bash theme={null}
fireconnect model list --search glm
fireconnect pi on --model glm-5p2
```

Fire Pass keys only list Fire Pass routers. FireConnect rejects `--model firerouter` with Fire Pass (`fpk_...`) on every harness; use an `fw_...` key.

## FireRouter

```bash theme={null}
fireconnect pi on --model firerouter
fireconnect pi on --model firerouter --anthropic-api-key sk-ant-...
```

## CLI reference

```bash theme={null}
fireconnect pi on              # Enable Fireworks routing
fireconnect pi off             # Restore original settings and auth
fireconnect pi status          # Check current provider and model
fireconnect pi help            # Show harness-specific help
```

Run `fireconnect pi help` for all options.

### Switch models

```bash theme={null}
fireconnect pi on --model glm-5p2
```

### Turn off Fireworks routing

```bash theme={null}
fireconnect pi off
```

This restores your previous `settings.json` and `auth.json` from the backup in `~/.fireconnect/pi/`.

### Use a non-default settings file

```bash theme={null}
fireconnect pi on --settings-path /path/to/settings.json
```

## Fireworks on Microsoft Foundry

Pi supports **Fireworks on Microsoft Foundry** (CLI: `--provider azure` or `on --azure`). See the [FireConnect overview](/ecosystem/fireconnect/microsoft-foundry) and [Microsoft Foundry integration guide](/ecosystem/integrations/azure-foundry) for portal setup.

### Configure and enable

```bash theme={null}
export AZURE_API_KEY=<your-azure-api-key>

fireconnect configure \
  --provider azure \
  --base-url https://<resource>.services.ai.azure.com \
  --api-key $AZURE_API_KEY

fireconnect pi on --model FW-GLM-5.2
```

One-off routing:

```bash theme={null}
fireconnect pi on --azure --base-url https://<resource>.services.ai.azure.com --model FW-MiniMax-M2.5
```

### What gets written

FireConnect registers a `fireworks-azure` **openai-completions** provider in Pi's `models.json` (labeled **Fireworks on Microsoft Foundry**) and sets `defaultProvider` / `defaultModel` in `settings.json`. The Azure API key is stored in `auth.json` as a literal when passed with `--api-key`, or as `$AZURE_API_KEY` when resolved from the environment.

Pass your Foundry model with `--model` (for example, `FW-GLM-5.2`). Use `fireconnect model list` only for browsing Fireworks serverless models on the direct gateway path.

### Turn off Foundry routing

To switch back to the Fireworks gateway:

```bash theme={null}
fireconnect configure --provider fireworks
fireconnect pi on
```

To remove FireConnect entirely and restore your original `settings.json`, `auth.json`, and `models.json`:

```bash theme={null}
fireconnect pi off
```

Restart Pi after switching or running `off`. See [Turn off Foundry routing](/ecosystem/fireconnect/microsoft-foundry#turn-off-foundry-routing) for details on global config behavior and `uninstall`.

## Source

FireConnect is open source: [github.com/fw-ai/fireconnect](https://github.com/fw-ai/fireconnect)


# VS Code
Source: https://docs.fireworks.ai/ecosystem/fireconnect/vscode

Use Fireworks AI models in GitHub Copilot Chat with the FireConnect CLI

[FireConnect](https://github.com/fw-ai/fireconnect) adds Fireworks AI models to **[GitHub Copilot](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot) Chat** in [Visual Studio Code](https://code.visualstudio.com) by writing a custom language-model endpoint. See the [FireConnect overview](/ecosystem/fireconnect/overview) for install and CLI basics.

<Tip>
  **Change models:** quit VS Code → `fireconnect vscode on --model <id>`. See [Models](/ecosystem/fireconnect/models).
</Tip>

<Card title="Manual setup (GitHub Copilot)" icon="github" href="/ecosystem/integrations/github-copilot">
  Step-by-step UI walkthrough with screenshots. Add a Fireworks custom endpoint without the FireConnect CLI
</Card>

## Prerequisites

* [Visual Studio Code](https://code.visualstudio.com) with the [GitHub Copilot](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot) extension
* GitHub Copilot **Pro** or **Enterprise** (the free tier only supports the Auto model)
* A [Fireworks API key](https://app.fireworks.ai/settings/users/api-keys) (`fw_...`) or a [Fire Pass](/firepass) key (`fpk_...`)
* FireConnect **v0.9.2+** (see [Install](/ecosystem/fireconnect/overview#install))

## Enable Fireworks routing

VS Code stores custom-endpoint API keys (encrypted) in `state.vscdb`. **Quit VS Code** before running `on` or `off`. In an interactive terminal, FireConnect waits for you to quit (or pass `--force` to write anyway).

```bash theme={null}
fireconnect login
fireconnect vscode on
```

Or pass the key once:

```bash theme={null}
fireconnect vscode on --api-key fw_...
```

Start or restart VS Code, then open Copilot Chat and pick a Fireworks model from the model picker.

```bash theme={null}
fireconnect vscode status   # read-only; works while VS Code is running
```

## Browse and switch models

Direct Fireworks routing uses the **chat-completions** API. Kimi models support vision; GLM models are text-only.

```bash theme={null}
fireconnect model list --search glm
fireconnect vscode on --model deepseek-v4-flash
fireconnect vscode on --model firerouter
```

`on` registers the preferred serverless catalog in the VS Code Chat model picker. Pick models under **Other Models → Fireworks** in Copilot Chat.

## FireRouter

```bash theme={null}
fireconnect vscode on --model firerouter --anthropic-api-key sk-ant-...
```

The Fireworks key stays encrypted in `state.vscdb`. An Anthropic BYOK key is optional for pass-through to Claude Opus 5.

## What gets written

FireConnect merges a **Fireworks** custom endpoint into VS Code's language-model config and stores the API key in VS Code's secret storage:

| What              | Where                                                             |
| ----------------- | ----------------------------------------------------------------- |
| Provider + models | `chatLanguageModels.json`                                         |
| Encrypted API key | `state.vscdb` (`ItemTable`, key `secret://chat.lm.secret.fw-...`) |

Platform paths:

| Platform | `chatLanguageModels.json`                                         | `state.vscdb`                                                       |
| -------- | ----------------------------------------------------------------- | ------------------------------------------------------------------- |
| Linux    | `~/.config/Code/User/chatLanguageModels.json`                     | `~/.config/Code/User/globalStorage/state.vscdb`                     |
| macOS    | `~/Library/Application Support/Code/User/chatLanguageModels.json` | `~/Library/Application Support/Code/User/globalStorage/state.vscdb` |
| Windows  | `%APPDATA%\Code\User\chatLanguageModels.json`                     | `%APPDATA%\Code\User\globalStorage\state.vscdb`                     |

The endpoint URL is `https://api.fireworks.ai/inference` (VS Code appends `/v1/chat/completions`). The API key lives in `state.vscdb`, not the JSON.

On macOS, `safeStorage` encrypts with a master key VS Code stores in the login Keychain. On Linux, `safeStorage` needs `libsecret` (`secret-tool`) for real encryption. Without it, Chromium falls back to a hardcoded password (obfuscated, not encrypted), which FireConnect still writes but warns about.

FireConnect snapshots the original `chatLanguageModels.json` under `~/.fireconnect/vscode/` before the first change. Running `fireconnect vscode off` restores it byte-for-byte and deletes the `chat.lm.secret.fw-*` secret row from `state.vscdb`.

## CLI reference

```bash theme={null}
fireconnect vscode on              # Add the Fireworks provider (quit VS Code first)
fireconnect vscode off             # Restore config and remove the stored key
fireconnect vscode status          # Show provider, auth, and registered models
fireconnect vscode help            # Show harness-specific help
```

Run `fireconnect vscode help` for all options, including `--vscode-path` (explicit `chatLanguageModels.json` path) and `--force`.

### Turn off Fireworks routing

Quit VS Code, then run:

```bash theme={null}
fireconnect vscode off
```

This restores your previous `chatLanguageModels.json` from the backup in `~/.fireconnect/vscode/` and removes the FireConnect secret from `state.vscdb`. Restart VS Code for the change to take effect.

## Fireworks on Microsoft Foundry

VS Code supports **Fireworks on Microsoft Foundry** (CLI: `--provider azure` or `on --azure`). See [Microsoft Foundry in FireConnect](/ecosystem/fireconnect/microsoft-foundry) and the [portal setup guide](/ecosystem/integrations/azure-foundry).

<Warning>
  Foundry routing requires a standard Azure API key. Fire Pass keys (`fpk_...`) are not supported. **Quit VS Code** before `on` or `off`.
</Warning>

```bash theme={null}
export AZURE_API_KEY=<your-azure-api-key>

fireconnect configure \
  --provider azure \
  --base-url https://<resource>.services.ai.azure.com \
  --api-key $AZURE_API_KEY

fireconnect vscode on --model FW-GLM-5.2
```

One-off routing without changing global config:

```bash theme={null}
fireconnect vscode on \
  --azure \
  --base-url https://<resource>.services.ai.azure.com \
  --model FW-MiniMax-M2.5
```

Pass your Foundry model with `--model` (for example, `FW-GLM-5.2`), not a Fireworks serverless short ID.

FireConnect adds a chat-completions custom endpoint pointed at Foundry and stores the Azure key in VS Code secret storage. `fireconnect model list` only browses the Fireworks serverless catalog on the direct gateway path.

To switch back to the Fireworks gateway:

```bash theme={null}
fireconnect configure --provider fireworks
fireconnect vscode on
```

See [Turn off Foundry routing](/ecosystem/fireconnect/microsoft-foundry#turn-off-foundry-routing) for `off` and global config behavior.

## Related

* [GitHub Copilot integration guide](/ecosystem/integrations/github-copilot): manual custom-endpoint setup with screenshots
* [Cursor](/ecosystem/fireconnect/cursor): use Fireworks models in Cursor IDE
* [FireRouter](/ecosystem/firerouter/overview): automatic cost routing via `--model firerouter`

## Source

FireConnect is open source: [github.com/fw-ai/fireconnect](https://github.com/fw-ai/fireconnect)


# WebSearch MCP
Source: https://docs.fireworks.ai/ecosystem/fireconnect/websearch-mcp

Fireworks-hosted live web search for coding harnesses via HTTP MCP

WebSearch MCP provides live internet search through Fireworks. It works from **any harness that supports HTTP MCP** once you add the server URL and authenticate with a Fireworks API key.

This is separate from the [Fireworks Docs MCP](/ecosystem/integrations/development-setup), which searches Fireworks documentation only.

## Access

Web search MCP is not enabled on every account. [Contact the Fireworks team](https://fireworks.ai/contact) to request access, then use a standard [Fireworks API key](https://app.fireworks.ai/settings/users/api-keys) (`fw_...`) as the bearer token below.

### Check eligibility

There is no self-serve toggle in the Fireworks dashboard yet. After access is granted:

| Method                                 | Eligible                                              | Not eligible                                   |
| -------------------------------------- | ----------------------------------------------------- | ---------------------------------------------- |
| Connect the MCP server in your harness | Server connects; search tools respond                 | Auth or connection errors                      |
| `fireconnect claude on`                | Prints `Web search → fireworks-websearch (installed)` | Skips MCP setup; Fireworks routing still works |

If you see auth errors, [contact the Fireworks team](https://fireworks.ai/contact) to confirm web search is enabled on your account.

## MCP endpoint

Use this HTTP MCP server in any harness that supports remote MCP:

| Field    | Value                                          |
| -------- | ---------------------------------------------- |
| **URL**  | `https://mcp.fireworks.ai/work/mcp`            |
| **Auth** | `Authorization: Bearer YOUR_FIREWORKS_API_KEY` |

Example JSON (exact file and shape depend on your harness):

```json theme={null}
{
  "mcpServers": {
    "fireworks-websearch": {
      "type": "http",
      "url": "https://mcp.fireworks.ai/work/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_FIREWORKS_API_KEY"
      }
    }
  }
}
```

Replace `YOUR_FIREWORKS_API_KEY` with your Fireworks API key. Wire this into your harness MCP config the same way you would any other HTTP MCP server.

## FireConnect integration

FireConnect can install and wire WebSearch MCP automatically. **Today this is integrated for Claude Code only.** OpenCode, Codex, Pi, Cursor, VS Code, and other harnesses are coming soon.

For Claude Code with FireConnect:

```bash theme={null}
fireconnect claude on
```

For eligible accounts, FireConnect installs `fireworks-websearch`, adds Claude Code-specific `permissions.deny` entries for built-in web tools, and writes a literal Bearer token into the MCP config (refreshed on `fireconnect upgrade` or `fireconnect login`). Look for `Web search → fireworks-websearch (installed)` in the CLI output.

On other harnesses, add the [MCP endpoint](#mcp-endpoint) manually until FireConnect support ships.

## Claude Code manual setup

Claude Code is the most common path today. You can configure the MCP yourself or use FireConnect above.

**Prerequisites**

* [Claude Code](https://claude.ai/code) installed
* A Fireworks API key with web search access
* Claude Code routed through Fireworks ([FireConnect](/ecosystem/fireconnect/claude-code) or [manual settings](/ecosystem/firerouter/claude-code))

### 1. Add the MCP server

Add the [MCP endpoint](#mcp-endpoint) to `~/.claude.json` (merge with any existing `mcpServers` entries).

Or add it from the CLI:

```bash theme={null}
claude mcp add --transport http fireworks-websearch https://mcp.fireworks.ai/work/mcp \
  --header "Authorization: Bearer YOUR_FIREWORKS_API_KEY"
```

### 2. Deny Claude Code's built-in web tools

Claude Code's built-in `WebSearch` and `WebFetch` tools are Anthropic server-side tools. They do not run when inference is routed through Fireworks. Deny them in `~/.claude/settings.json`:

```json theme={null}
{
  "permissions": {
    "deny": ["WebSearch", "WebFetch"]
  }
}
```

Merge this with your existing `permissions` rules if you already have them.

### 3. Connect in Claude Code

1. Restart Claude Code.
2. Run `/mcp` and connect to `fireworks-websearch`.

`fireconnect claude off` removes the FireConnect-managed MCP entry and restores your previous settings.

## See also

* [FireConnect overview](/ecosystem/fireconnect/overview)
* [Claude Code](/ecosystem/fireconnect/claude-code)


# Authentication
Source: https://docs.fireworks.ai/ecosystem/firerouter/authentication

BYOK headers and API keys for FireRouter

FireRouter uses a **bring-your-own-key** contract. The service validates your Fireworks key on every request and forwards your provider key only on pass-through legs to closed-source models. FireRouter does not store provider keys server-side.

## Required keys

| Key                              | Header or env                                    | Used for                                           |
| -------------------------------- | ------------------------------------------------ | -------------------------------------------------- |
| Fireworks API key (`fw_...`)     | `Authorization: Bearer` or `X-Fireworks-Api-Key` | FireRouter auth and redirected Fireworks inference |
| Anthropic API key (`sk-ant-...`) | `x-anthropic-api-key`                            | Calls to Claude models                             |
| OpenAI API key (`sk-...`)        | `x-openai-api-key`                               | Calls to OpenAI models                             |

The default `firerouter` model uses **Claude Opus 5** as its primary model, so every request requires an Anthropic key. FireRouter fails closed when the primary model's credential is missing; it does not silently restrict the route to Fireworks models.

For a model-specific [FireRouter slug](/ecosystem/firerouter/overview#choose-different-models), the first model's credential is required. Later models are eligible only when their credentials are present. Slugs that contain only Fireworks models need no additional provider key beyond the Fireworks API key.

## Fireworks key header

Send your Fireworks API key with either header:

```bash theme={null}
-H "Authorization: Bearer $FIREWORKS_API_KEY"
# or
-H "X-Fireworks-Api-Key: $FIREWORKS_API_KEY"
```

`X-FireRouter-Fireworks-Key` is also accepted. New integrations should prefer `X-Fireworks-Api-Key`.

## Anthropic provider key

The canonical header for Anthropic pass-through is:

```bash theme={null}
-H "x-anthropic-api-key: $ANTHROPIC_API_KEY"
```

`x-api-key` and `Authorization: Bearer` are also accepted for Anthropic clients that send credentials that way. For new integrations, prefer `x-anthropic-api-key`.

Example:

```bash theme={null}
curl https://api.fireworks.ai/inference/v1/chat/completions \
  -H "Authorization: Bearer $FIREWORKS_API_KEY" \
  -H "x-anthropic-api-key: $ANTHROPIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "firerouter", "messages": [...]}'
```

## OpenAI provider key

When the selected FireRouter slug includes an OpenAI model, send your OpenAI API key as:

```bash theme={null}
-H "x-openai-api-key: $OPENAI_API_KEY"
```

Do not put the OpenAI provider key in the OpenAI client's `api_key` field. That field supplies the `Authorization` header used to authenticate to the Fireworks gateway, so it must contain your Fireworks API key. Pass the OpenAI provider key as a custom header:

```python theme={null}
import os

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["FIREWORKS_API_KEY"],
    base_url="https://api.fireworks.ai/inference/v1",
    default_headers={"x-openai-api-key": os.environ["OPENAI_API_KEY"]},
)
```

## Common errors

| Response                        | Cause                                                        | Fix                                                                                           |
| ------------------------------- | ------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| `401 Missing Fireworks API key` | Fireworks key header missing or empty                        | Set `X-Fireworks-Api-Key` to your `fw_...` key                                                |
| `401 invalid Fireworks API key` | Key rejected                                                 | Confirm the key is valid in the [dashboard](https://app.fireworks.ai/settings/users/api-keys) |
| `400` with `no_credential`      | Required provider key missing or sent under the wrong header | Send the key under the provider-specific header                                               |
| Provider `401`                  | Provider key is invalid                                      | Check the key sent as `x-anthropic-api-key` or `x-openai-api-key`                             |

## Related

* [Quickstart](/ecosystem/firerouter/quickstart): API call examples
* [Claude Code (manual setup)](/ecosystem/firerouter/claude-code): `settings.json` setup
* [Overview](/ecosystem/firerouter/overview): routing model and model ID


# Claude Code (manual setup)
Source: https://docs.fireworks.ai/ecosystem/firerouter/claude-code

Configure FireRouter in Claude Code by editing settings.json

<Tip>
  **Prefer FireConnect.** Run `fireconnect claude on --model firerouter` (v0.9.0+) instead of editing JSON by hand. See [Claude Code](/ecosystem/fireconnect/claude-code#firerouter) and [Models](/ecosystem/fireconnect/models).
</Tip>

This page is only for manual `~/.claude/settings.json` setup when you cannot use FireConnect.

## Prerequisites

* [Claude Code](https://claude.ai/code) installed
* A [Fireworks API key](https://app.fireworks.ai/settings/users/api-keys) (`fw_...`)
* An Anthropic API key if you need pass-through to Claude Opus 5. See [Authentication](/ecosystem/firerouter/authentication).

Add one of the configs below to `~/.claude/settings.json` (on Windows: `%USERPROFILE%\.claude\settings.json`), then restart Claude Code.

## With a Claude subscription

You do not need to pass any access or API token. Use `ANTHROPIC_CUSTOM_HEADERS` to pass your Fireworks API key.

```json theme={null}
{
  "env": {
    "ANTHROPIC_BASE_URL": "https://api.fireworks.ai/inference",
    "ANTHROPIC_MODEL": "firerouter[1m]",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "firerouter[1m]",
    "ANTHROPIC_DEFAULT_FABLE_MODEL": "glm-fast-latest[1m]",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "kimi-latest",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "deepseek-v4-flash",
    "CLAUDE_CODE_SUBAGENT_MODEL": "deepseek-v4-flash",
    "ANTHROPIC_CUSTOM_HEADERS": "x-fireworks-api-key: YOUR_FIREWORKS_API_KEY"
  }
}
```

## With an Anthropic API key

Add `ANTHROPIC_API_KEY` for pass-through. You still need `ANTHROPIC_CUSTOM_HEADERS` for your Fireworks API key.

```json theme={null}
{
  "env": {
    "ANTHROPIC_BASE_URL": "https://api.fireworks.ai/inference",
    "ANTHROPIC_MODEL": "firerouter[1m]",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "firerouter[1m]",
    "ANTHROPIC_DEFAULT_FABLE_MODEL": "glm-fast-latest[1m]",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "kimi-latest",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "deepseek-v4-flash",
    "CLAUDE_CODE_SUBAGENT_MODEL": "deepseek-v4-flash",
    "ANTHROPIC_CUSTOM_HEADERS": "x-fireworks-api-key: YOUR_FIREWORKS_API_KEY",
    "ANTHROPIC_API_KEY": "YOUR_ANTHROPIC_API_KEY"
  }
}
```


# LiteLLM
Source: https://docs.fireworks.ai/ecosystem/firerouter/litellm

Add FireRouter to a LiteLLM Proxy deployment

Add FireRouter to an existing [LiteLLM Proxy](/ecosystem/integrations/litellm) deployment so developers request one model instead of choosing between open and closed-source models on every call.

FireRouter uses bring-your-own-key (BYOK): your Fireworks API key pays for redirected calls, and an Anthropic API key pays for pass-through to Claude Opus 5. See [Authentication](/ecosystem/firerouter/authentication). FireRouter does not store provider keys.

## Prerequisites

* A running LiteLLM Proxy on **v1.95.0** configured for Fireworks (see [LiteLLM integration](/ecosystem/integrations/litellm))
* A [Fireworks API key](https://app.fireworks.ai/settings/users/api-keys) (`fw_...`)
* An **Anthropic API key** (`sk-ant-...`) for pass-through

## Add FireRouter to `config.yaml`

```yaml theme={null}
model_list:
  - model_name: accounts/fireworks/routers/firerouter
    litellm_params:
      model: fireworks_ai/accounts/fireworks/routers/firerouter
      api_key: os.environ/FIREWORKS_AI_API_KEY
      extra_headers:
        x-anthropic-api-key: os.environ/ANTHROPIC_API_KEY
```

Always use the full router resource ID in `litellm_params.model`. LiteLLM rewrites short names like `fireworks_ai/firerouter` to `accounts/fireworks/models/firerouter`, which FireRouter does not recognize.

Set `ANTHROPIC_API_KEY` on the LiteLLM server so pass-through requests include `x-anthropic-api-key` automatically. Developers do not need to send the Anthropic key on each request when it is configured here.

<Note>
  A server-side Anthropic key is shared across all callers of that proxy deployment. If each developer should bring their own Anthropic key, omit `extra_headers` and have clients send `x-anthropic-api-key` on each request instead (see below).
</Note>

Start or restart the proxy with both keys exported:

```bash theme={null}
export FIREWORKS_AI_API_KEY="fw_..."
export ANTHROPIC_API_KEY="sk-ant-..."
litellm --config config.yaml
```

## Call FireRouter

Clients authenticate to LiteLLM with a virtual key:

```bash theme={null}
curl http://localhost:4000/chat/completions \
  -H "Authorization: Bearer $LITELLM_VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "accounts/fireworks/routers/firerouter",
    "messages": [{"role": "user", "content": "Say pong in one word."}]
  }'
```

Optional routing preference:

```bash theme={null}
-H "x-routing-preference: 4"
```

See [Routing preferences](/ecosystem/firerouter/routing-preferences) for values `1`–`5`.

## Per-developer Anthropic keys

If each caller should use their own Anthropic key, omit `extra_headers` from the model config and have clients send the header on every request:

```bash theme={null}
curl http://localhost:4000/chat/completions \
  -H "Authorization: Bearer $LITELLM_VIRTUAL_KEY" \
  -H "x-anthropic-api-key: $ANTHROPIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "accounts/fireworks/routers/firerouter",
    "messages": [{"role": "user", "content": "Say pong in one word."}]
  }'
```

Enable [client header forwarding](https://docs.litellm.ai/docs/proxy/forward_client_headers) for the `accounts/fireworks/routers/firerouter` model group if your LiteLLM version does not forward `x-` headers by default.

## API key layout

| Key                              | Who holds it                                                     | Used for                                 |
| -------------------------------- | ---------------------------------------------------------------- | ---------------------------------------- |
| Fireworks API key (`fw_...`)     | LiteLLM server                                                   | FireRouter auth and redirected inference |
| LiteLLM virtual key              | Each developer or service                                        | Proxy authentication and spend tracking  |
| Anthropic API key (`sk-ant-...`) | LiteLLM server (`extra_headers`) or each caller (request header) | Claude pass-through billing              |

## Related

* [LiteLLM integration](/ecosystem/integrations/litellm): configure Fireworks models in LiteLLM Proxy
* [Quickstart](/ecosystem/firerouter/quickstart): direct API call examples
* [Authentication](/ecosystem/firerouter/authentication): header reference
* [Routing preferences](/ecosystem/firerouter/routing-preferences): tune cost vs. quality


# Overview
Source: https://docs.fireworks.ai/ecosystem/firerouter/overview

Route LLM requests between closed-source and open models with FireRouter

FireRouter is a managed routing service for **any LLM workload**. Request a FireRouter model through the [Fireworks inference API](/tools-sdks/openai-compatibility), and FireRouter scores each turn to select a model from the configured set. By default, it routes between a Fireworks open model and a closed-source model.

The result is lower cost on simpler requests without giving up closed-source quality on harder ones.

<Note>
  FireRouter is in **research preview**. APIs, routing behavior, and the models in the routing pair may change. This documentation is updated to reflect the current configuration.
</Note>

## When to use FireRouter

Use FireRouter when you want **automatic cost optimization** without picking a different model per request:

* You want closed-source quality (for example Claude Opus) on hard prompts but do not need it on every call.
* Many of your requests are straightforward (summaries, formatting, simple Q\&A) and can be served by a Fireworks open model.
* You want to select the models available to the router without choosing a target for every request.
* For the default configuration, you are able to send both a Fireworks API key and a provider API key on each request.

## How it works

FireRouter evaluates each new user request. The default `firerouter` configuration picks one of two paths:

| Path             | When                                      | What runs                                         | Billing                           |
| ---------------- | ----------------------------------------- | ------------------------------------------------- | --------------------------------- |
| **Redirect**     | Simple or low-complexity work             | A Fireworks open model (for example GLM 5.2 Fast) | Your Fireworks API key            |
| **Pass-through** | Hard reasoning, judgment, or long context | A closed-source model (by default Claude Opus 5)  | Your provider API key (Anthropic) |

FireRouter uses a bring-your-own-key (BYOK) model:

* Your **Fireworks API key** authenticates to FireRouter and pays for calls to Fireworks models.
* A third-party **provider API key** pays for calls to that provider's models.

FireRouter never stores your provider keys server-side. You send them on each request.

## FireConnect

The easiest way to use FireRouter in coding harnesses is [FireConnect](/ecosystem/fireconnect/overview). As of FireConnect **v0.9.0**, select FireRouter like any other model:

```bash theme={null}
fireconnect login
fireconnect claude on --model firerouter
```

See [FireConnect Models](/ecosystem/fireconnect/models) for how `firerouter` fits next to other short IDs, and [Harness support](/ecosystem/fireconnect/overview#harness-support) for which harnesses support FireRouter. Per-harness details (Claude slot flags, Codex Anthropic env, routing preference) live on each harness page.

Upgrade FireConnect before enabling FireRouter on an older install. See [Upgrade FireConnect](/ecosystem/fireconnect/overview#upgrade-fireconnect).

Pass `--anthropic-api-key sk-ant-...` on `on`, or store a key once with `fireconnect configure --anthropic-api-key sk-ant-...`. The default `firerouter` requires an Anthropic key because Claude Opus 5 is its primary model; FireRouter fails closed instead of silently restricting the route to GLM. See [Authentication](/ecosystem/firerouter/authentication).

Cursor and Deep Agents need workspace BYOK for FireRouter (no local Anthropic key).

## Endpoint

FireRouter is available through the Fireworks inference API:

```text theme={null}
https://api.fireworks.ai/inference/v1
```

Common API paths:

| Wire format        | Path                                                     |
| ------------------ | -------------------------------------------------------- |
| Chat Completions   | `https://api.fireworks.ai/inference/v1/chat/completions` |
| Anthropic Messages | `https://api.fireworks.ai/inference/v1/messages`         |

## Model ID

FireRouter is a first-party model on the Fireworks provider. Use `firerouter` in the `model` field:

```text theme={null}
firerouter
```

These longer forms are also accepted:

```text theme={null}
fireworks/firerouter
accounts/fireworks/routers/firerouter
```

For [LiteLLM](/ecosystem/firerouter/litellm), use the full router path in `litellm_params.model`.

FireRouter decides the target model for each request based on request complexity. The model ID selects the set of models FireRouter can route to.

### Current routing pair

The short `firerouter` model ID currently routes between:

| Role                         | Model                             |
| ---------------------------- | --------------------------------- |
| Pass-through (closed-source) | **Claude Opus 5**                 |
| Redirect (open)              | **GLM 5.2 Fast** (`glm-5p2-fast`) |

These models are subject to change as FireRouter is updated. This page reflects the current configuration.

Because the default pass-through target is Claude Opus 5, supply an Anthropic API key to use that path. See [Authentication](/ecosystem/firerouter/authentication).

### Choose different models

Use a slash-delimited FireRouter slug to change the models available to the router. The first model is the primary; the remaining models are alternatives that FireRouter ranks for each request. Each member must be an exact deployed model ID or a unique model alias, and a slug can contain up to eight distinct models.

Examples:

| Model ID                                                 | Models                                              |
| -------------------------------------------------------- | --------------------------------------------------- |
| `firerouter/kimi-k3/glm-5p2-fast`                        | Kimi K3 and GLM 5.2 Fast                            |
| `firerouter/kimi-k3/deepseek-v4-flash-0731`              | Kimi K3 and DeepSeek V4 Flash (0731)                |
| `firerouter/kimi-k3/glm-5p2-fast/deepseek-v4-flash-0731` | Kimi K3, GLM 5.2 Fast, and DeepSeek V4 Flash (0731) |
| `firerouter/claude-opus-5/kimi-k3/glm-5p2-fast`          | Claude Opus 5, Kimi K3, and GLM 5.2 Fast            |
| `firerouter/gpt-5.6-sol/glm-5p2-fast`                    | GPT 5.6 Sol and GLM 5.2 Fast                        |

Set the selected slug in the `model` field just as you would use `firerouter`. FireRouter still chooses the target for each request.

The primary model's credential is required. A later model is eligible only when its credential is present. Use `x-anthropic-api-key` for Claude models and `x-openai-api-key` for OpenAI models. See [Authentication](/ecosystem/firerouter/authentication).

## Client integrations

| Integration                                                     | When to use                                                                             |
| --------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| [FireConnect](/ecosystem/fireconnect/overview)                  | One-command setup for coding harnesses (recommended). Select with `--model firerouter`. |
| [Quickstart](/ecosystem/firerouter/quickstart)                  | Direct HTTP calls (curl, OpenAI SDK, any OpenAI-compatible client)                      |
| [Claude Code (manual setup)](/ecosystem/firerouter/claude-code) | Manual `settings.json` setup without FireConnect                                        |
| [LiteLLM](/ecosystem/firerouter/litellm)                        | Add FireRouter to a LiteLLM Proxy deployment                                            |

## What FireRouter is not

* **Not a deployment router.** FireRouter is different from [deployment routers](/deployments/routers), which load-balance traffic across your own Fireworks deployments.

## Next steps

<CardGroup>
  <Card title="FireConnect" icon="plug" href="/ecosystem/fireconnect/overview">
    Enable FireRouter with one command
  </Card>

  <Card title="Quickstart" icon="rocket" href="/ecosystem/firerouter/quickstart">
    Make your first API call
  </Card>

  <Card title="Claude Code (manual)" icon="terminal" href="/ecosystem/firerouter/claude-code">
    Edit settings.json for FireRouter in Claude Code
  </Card>

  <Card title="Authentication" icon="key" href="/ecosystem/firerouter/authentication">
    BYOK headers and API key requirements
  </Card>

  <Card title="Routing preferences" icon="sliders" href="/ecosystem/firerouter/routing-preferences">
    Tune the quality vs. savings dial
  </Card>

  <Card title="LiteLLM" icon="server" href="/ecosystem/firerouter/litellm">
    Add FireRouter to LiteLLM Proxy
  </Card>
</CardGroup>


# Quickstart
Source: https://docs.fireworks.ai/ecosystem/firerouter/quickstart

Make your first FireRouter API call

This guide shows how to call FireRouter directly through the Fireworks inference API. See the [overview](/ecosystem/firerouter/overview) for how routing works and the [authentication](/ecosystem/firerouter/authentication) page for header details.

For coding harnesses, use [FireConnect](/ecosystem/fireconnect/overview): `fireconnect <harness> on --model firerouter` (v0.9.0+). For manual Claude Code setup, see [Claude Code (manual setup)](/ecosystem/firerouter/claude-code).

## Prerequisites

* A [Fireworks API key](https://app.fireworks.ai/settings/users/api-keys) (`fw_...`)
* An **Anthropic API key** (`sk-ant-...`) for the default pass-through to Claude Opus 5

## Chat Completions

Send a request to the [Chat Completions](/tools-sdks/openai-compatibility) endpoint with the FireRouter model ID:

```bash theme={null}
curl https://api.fireworks.ai/inference/v1/chat/completions \
  -H "Authorization: Bearer $FIREWORKS_API_KEY" \
  -H "x-anthropic-api-key: $ANTHROPIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "firerouter",
    "messages": [{"role": "user", "content": "Say pong in one word."}]
  }'
```

With the default `firerouter` model ID, simple prompts are usually redirected to GLM 5.2 Fast on Fireworks. Harder prompts pass through to Claude Opus 5 using your Anthropic key.

To use a different model combination, replace `firerouter` with one of the [model-specific FireRouter slugs](/ecosystem/firerouter/overview#choose-different-models).

If the selected slug includes an OpenAI model, send its key as `-H "x-openai-api-key: $OPENAI_API_KEY"`. See [Authentication](/ecosystem/firerouter/authentication).

## Anthropic Messages

For clients that speak the Anthropic Messages API:

```bash theme={null}
curl https://api.fireworks.ai/inference/v1/messages \
  -H "Authorization: Bearer $FIREWORKS_API_KEY" \
  -H "x-anthropic-api-key: $ANTHROPIC_API_KEY" \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "firerouter",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Say pong in one word."}]
  }'
```

## Verify routing

1. Send a trivial prompt (for example "rename foo to bar"). Expect a fast response routed to the open model.
2. Send a hard reasoning prompt. With the default `firerouter` model ID, expect pass-through to Claude Opus 5.
3. Optionally set `x-routing-preference` to bias routing. See [Routing preferences](/ecosystem/firerouter/routing-preferences).

You can also send the Fireworks key as `X-Fireworks-Api-Key` instead of `Authorization: Bearer`. See [Authentication](/ecosystem/firerouter/authentication) for the full header reference.

## Related

* [Overview](/ecosystem/firerouter/overview): model ID and routing pair
* [Authentication](/ecosystem/firerouter/authentication): header reference
* [Routing preferences](/ecosystem/firerouter/routing-preferences): tune cost vs. quality
* [LiteLLM](/ecosystem/firerouter/litellm): add FireRouter to LiteLLM Proxy
* [Claude Code (manual setup)](/ecosystem/firerouter/claude-code): edit `settings.json` directly


# Routing preferences
Source: https://docs.fireworks.ai/ecosystem/firerouter/routing-preferences

Tune FireRouter between closed-source quality and open-model savings

FireRouter scores each request to `firerouter` and routes to either a closed-source model (pass-through) or a Fireworks open model (redirect). The **`x-routing-preference`** request header lets you bias that decision toward quality or savings.

## Preference levels

Send an integer from **1** (most quality-protective) to **5** (most savings-focused):

| Value | Name                | Behavior                                                           |
| ----- | ------------------- | ------------------------------------------------------------------ |
| `1`   | `max-intelligence`  | Strongest bias toward pass-through; use when quality is critical   |
| `2`   | `more-intelligence` | Favor closed-source models on borderline requests                  |
| `3`   | `balanced`          | Default when the header is omitted. FireRouter's standard tradeoff |
| `4`   | `more-savings`      | Favor open-model redirects on borderline requests                  |
| `5`   | `max-savings`       | Strongest bias toward redirect; maximize cost savings              |

## HTTP header

Set the header on each request:

```bash theme={null}
-H "x-routing-preference: 4"
```

Example with curl:

```bash theme={null}
curl https://api.fireworks.ai/inference/v1/chat/completions \
  -H "Authorization: Bearer $FIREWORKS_API_KEY" \
  -H "x-anthropic-api-key: $ANTHROPIC_API_KEY" \
  -H "x-routing-preference: 4" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "firerouter",
    "messages": [{"role": "user", "content": "Add a docstring to this function."}]
  }'
```

If the header is missing, invalid, or out of range, FireRouter uses the balanced default (`3`).

## FireConnect

When you enable FireRouter through FireConnect, pass `--routing-preference` on `on` instead of setting a header yourself:

```bash theme={null}
fireconnect claude on --model firerouter --routing-preference 4
fireconnect opencode on --model firerouter --routing-preference 2
fireconnect pi on --model firerouter --routing-preference 5
```

Supported on Claude Code, OpenCode, Pi, and VS Code. Values are `1`–`5` or the level names (`max-intelligence`, `balanced`, `max-savings`, etc.). The flag applies when at least one configured slot uses `firerouter`. Codex, Cursor, and Deep Agents do not support `--routing-preference`.

You can also store a global Anthropic BYOK key once:

```bash theme={null}
fireconnect configure --anthropic-api-key sk-ant-...
```

## When to adjust

* **High-volume workloads with many simple requests**: try `4` or `5` to redirect summaries, formatting, and straightforward Q\&A.
* **Tasks where quality is critical** (security review, complex reasoning, nuanced writing): try `1` or `2` to keep closed-source models on harder prompts.
* **Evaluating routing**: start at `3` (balanced), then move one step at a time and compare cost and output quality.

FireRouter caches routing decision within a conversation, so preference changes can take a few turns to take effect without restarting your client.

## Related

* [Overview](/ecosystem/firerouter/overview): how redirect vs. pass-through works
* [FireConnect overview](/ecosystem/fireconnect/overview#firerouter): enable FireRouter in coding harnesses
* [Quickstart](/ecosystem/firerouter/quickstart): API call examples
* [LiteLLM](/ecosystem/firerouter/litellm): add FireRouter to LiteLLM Proxy


# Agent Frameworks
Source: https://docs.fireworks.ai/ecosystem/integrations/agent-frameworks

Build production-ready AI agents with Fireworks and leading open-source frameworks

Fireworks AI seamlessly integrates with the best open-source agent frameworks, enabling you to build magical, production-ready applications powered by state-of-the-art language models.

## Supported Frameworks

<CardGroup>
  <Card title="LangChain" icon="link" href="https://docs.langchain.com/oss/python/integrations/providers/fireworks">
    Build LLM applications with powerful orchestration and tool integration
  </Card>

  <Card title="LlamaIndex" icon="link" href="https://developers.llamaindex.ai/python/examples/llm/fireworks/">
    Efficient data retrieval and document indexing for LLM-based agents
  </Card>

  <Card title="CrewAI" icon="link" href="https://docs.crewai.com/en/concepts/llms#fireworks-ai">
    Orchestrate collaborative multi-agent systems for complex tasks
  </Card>

  <Card title="PydanticAI" icon="link" href="https://ai.pydantic.dev/models/openai/#fireworks-ai">
    Type-safe AI agent development with Pydantic validation
  </Card>

  <Card title="Strands Agents" icon="link" href="https://strandsagents.com/latest/documentation/docs/community/model-providers/fireworksai/">
    Modern agent orchestration with seamless OpenAI-compatible integration
  </Card>

  <Card title="AWS AgentCore" icon="aws" href="/ecosystem/integrations/agentcore">
    Build and deploy production AI agents with Fireworks models on AgentCore Runtime
  </Card>
</CardGroup>

## Need Help?

For assistance with agent framework integrations, [contact our team](https://fireworks.ai/contact) or join our [Discord community](https://discord.gg/fireworks-ai).


# Microsoft Foundry
Source: https://docs.fireworks.ai/ecosystem/integrations/azure-foundry

Deploy frontier open models inside your Azure subscription, billed through Azure.

Fireworks AI is a first-party inference provider inside Microsoft Foundry. You can access frontier open models through your existing Azure account, with usage billed through Azure and counting toward your Microsoft Azure Consumption Commitment (MACC).

This page covers the Fireworks side of the integration. For Azure portal setup steps, see the [Microsoft Learn guide](https://learn.microsoft.com/en-us/azure/foundry/how-to/fireworks/enable-fireworks-models).

<Info>
  **New to Fireworks?** Foundry users get the same OpenAI-compatible API and model catalog as direct Fireworks customers. Start with the [PayGo quickstart](#paygo-quickstart) below. You can be making requests in about 10 minutes.
</Info>

## Prerequisites

* An active Azure subscription
* The Fireworks integration enabled at the subscription level (see below)
* A Microsoft Foundry project with the **Azure AI Developer** role assigned

### Opt-in

Fireworks on Foundry requires a one-time opt-in per Azure subscription before you can create deployments. Follow the steps in the [Microsoft Learn guide](https://learn.microsoft.com/en-us/azure/foundry/how-to/fireworks/enable-fireworks-models#enable-fireworks-on-foundry).

## Deployment modes

Fireworks on Foundry supports three deployment modes.

| Mode              | Also called                    | Pricing                           | Regions              | Right for                                    |
| ----------------- | ------------------------------ | --------------------------------- | -------------------- | -------------------------------------------- |
| **PayGo**         | Serverless, Data Zone Standard | Per token, MACC-eligible          | US Data Zone only    | Prototyping, low-volume workloads            |
| **PTU**           | Provisioned Throughput         | Per PTU-hour, ACD + MACC eligible | Global               | Production workloads with consistent traffic |
| **Custom Models** | Bring Your Own Model           | PTU pricing                       | Global (PTU regions) | Fine-tuned model deployment                  |

PTU deployments can be created directly in the Azure portal. For help with PTU sizing on Fireworks models, contact [sales@fireworks.ai](mailto:sales@fireworks.ai).

## Available models

All models use the OpenAI-compatible chat completions API and are added to the catalog on a rolling basis. For the current list of available models, see the [Microsoft Learn catalog](https://learn.microsoft.com/en-us/azure/foundry/how-to/fireworks/enable-fireworks-models#available-catalog-models).

Chat completions only. Embeddings, image generation, and audio modalities are not available through Foundry.

## PayGo quickstart

PayGo (Data Zone Standard) is available in: East US, East US 2, Central US, North Central US, West US, West US 3.

The throughput limit for PayGo deployments is **500,000 tokens per minute (TPM)**. For higher limits, submit a limit increase request on [aka.ms/fireworks-quota](https://aka.ms/fireworks-quota) and contact [sales@fireworks.ai](mailto:sales@fireworks.ai).

### Make your first request

Foundry deployments use an OpenAI-compatible endpoint. Use your Foundry project endpoint and Azure API key.

```python theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="https://<your-project>.services.ai.azure.com/models",
    api_key="<your-azure-api-key>",
)

response = client.chat.completions.create(
    model="fireworks-ai/FW-GLM-5.2",
    messages=[{"role": "user", "content": "Hello"}],
)

print(response.choices[0].message.content)
```

Find your project endpoint in the Microsoft Foundry portal under **Project settings**.

## PTU (Provisioned Throughput)

PTU deployments provide dedicated GPU capacity reserved for your workload, with consistent throughput and global region availability.

* Dedicated capacity, not shared with other tenants
* Available globally, not limited to US Data Zone
* ACD-eligible and MACC-eligible

You can create a PTU deployment directly in the Azure portal. For more on provisioned throughput, see the [Microsoft Learn guide](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/provisioned-throughput).

For help with PTU sizing on Fireworks models, contact [sales@fireworks.ai](mailto:sales@fireworks.ai).

## Custom Models

Fine-tune on Fireworks and deploy on Foundry, or bring your own weights from wherever you post-train to deploy on Foundry. Your model is served on Fireworks infrastructure within Azure, billed through your Azure account.

### Supported base architectures

For the list of supported custom model architectures, see the [Microsoft Learn guide](https://learn.microsoft.com/en-us/azure/foundry/how-to/fireworks/enable-fireworks-models#supported-model-architectures).

### Deployment

To import and deploy a custom model, follow the [Import custom models into Foundry guide](https://learn.microsoft.com/en-us/azure/foundry/how-to/fireworks/import-custom-models?tabs=rest-api).

## Billing

All Fireworks on Foundry usage is billed through Azure. You do not need a separate Fireworks billing account or contract.

* PayGo and PTU usage is MACC-eligible
* PTU deployments are ACD-eligible and qualify for quota retirement
* Direct Fireworks usage at [fireworks.ai](https://fireworks.ai) is billed separately and does not count toward MACC

## Troubleshooting

| Issue                           | Resolution                                                                                                                                                                                                                  |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Quota exceeded error            | Request a limit increase at [aka.ms/fireworks-quota](https://aka.ms/fireworks-quota)                                                                                                                                        |
| Access denied on deployment     | Verify you have the **Azure AI Developer** role on the project                                                                                                                                                              |
| Opt-in not propagating          | Allow up to 30 minutes after registering `Fireworks.EnableDeploy`                                                                                                                                                           |
| Custom Model deployment failing | Confirm weights are full-weight (not LoRA adapters) and the architecture is in the [supported list](https://learn.microsoft.com/en-us/azure/foundry/how-to/fireworks/enable-fireworks-models#supported-model-architectures) |
| PTU provisioning questions      | Contact [sales@fireworks.ai](mailto:sales@fireworks.ai)                                                                                                                                                                     |

## FireConnect

Use FireConnect to route local coding harnesses through your Foundry deployments without hand-editing config files.

<Card title="FireConnect + Microsoft Foundry" icon="terminal" href="/ecosystem/fireconnect/microsoft-foundry">
  Configure `--provider azure`, then run `fireconnect opencode on`, `fireconnect codex on`, `fireconnect cursor on`, `fireconnect vscode on`, `fireconnect pi on`, or `fireconnect deepagents on --model FW-GLM-5.2`
</Card>

FireConnect implements Azure routing for **OpenCode**, **Codex**, **Pi**, **Cursor**, **VS Code**, and **Deep Agents** in v0.9.0+. **Claude Code** does not yet. Running `fireconnect claude on` always wires direct Fireworks for now, regardless of global `--provider azure`.

```bash theme={null}
export AZURE_API_KEY=<your-azure-api-key>

fireconnect configure \
  --provider azure \
  --base-url https://<resource>.services.ai.azure.com \
  --api-key $AZURE_API_KEY

fireconnect opencode on --model FW-GLM-5.2
```

See [FireConnect + Microsoft Foundry](/ecosystem/fireconnect/microsoft-foundry) for Foundry models, per-harness config details, one-off `--azure` routing, [turning Foundry off](/ecosystem/fireconnect/microsoft-foundry#turn-off-foundry-routing), and switching back to direct Fireworks.

## Additional resources

* [Enable Fireworks on Foundry (Microsoft Learn)](https://learn.microsoft.com/en-us/azure/foundry/how-to/fireworks/enable-fireworks-models)
* [Microsoft Foundry portal](https://ai.azure.com/)
* [Fireworks fine-tuning docs](/fine-tuning/finetuning-intro)
* [Fireworks Trust Center](https://fireworks.ai/trust)
* [sales@fireworks.ai](mailto:sales@fireworks.ai) for PTU provisioning and Custom Model support


# How Setup Works
Source: https://docs.fireworks.ai/ecosystem/integrations/byoc/how-setup-works

Understand the high-level onboarding flow for Bring Your Own Cluster.

This page describes the high-level setup flow for Enterprise customers onboarding to Bring Your Own Cluster (BYOC).

<Note>
  Bring Your Own Cluster is in Private Preview for Enterprise customers. Contact [sales@fireworks.ai](mailto:sales@fireworks.ai) to participate in the preview and confirm the onboarding path for your environment.
</Note>

## Prerequisites

Before installation, Fireworks works with your team to confirm that the target environment can support BYOC.

At a high level, you need:

* A Kubernetes cluster with NVIDIA GPU nodes
* Network configuration that allows Fireworks to manage the cluster
* A customer-approved endpoint and DNS plan
* An administrative credential for the setup phase

Fireworks supports BYOC on major cloud providers and their managed Kubernetes offerings, select GPU cloud providers, and on-premises environments that provide a reachable Kubernetes endpoint, supported NVIDIA GPU nodes, and the required network setup. Fireworks confirms provider, environment, GPU capacity, and networking support during preview onboarding.

<Warning>
  Fine-tuning is not supported in BYOC during Private Preview.
</Warning>

## Setup flow

<Steps>
  <Step title="Provision the cluster">
    Your team provisions the Kubernetes cluster with NVIDIA GPU nodes, or provisions it jointly with Fireworks during onboarding. You retain ownership of the cloud account or data center environment, cluster, GPU nodes, and networking.
  </Step>

  <Step title="Grant setup access">
    You grant Fireworks an administrative credential to the cluster for setup. Fireworks recommends a dedicated, clearly named, revocable credential so your team can audit and manage access through your normal governance process.
  </Step>

  <Step title="Install the serving stack">
    Fireworks installs the serving stack using managed deployment tooling such as Helm or GitOps. During installation, Fireworks creates scoped-down Kubernetes identities and roles for the components that run in the cluster.
  </Step>

  <Step title="Configure networking and DNS">
    Fireworks works with your team to configure networking and DNS so your customer endpoint is reachable through the Fireworks API experience while inference request and response handling runs in your environment.
  </Step>

  <Step title="Validate and deploy models">
    Fireworks validates the cluster, then deploys inference workloads into the environment. After validation, your applications use the same Fireworks APIs and SDKs used across the rest of the platform.
  </Step>
</Steps>

## Access model during setup

Installation requires an administrative Kubernetes credential because the platform must create cluster-scoped resources, namespaces, standard dependency resources, and node-level scheduling configuration. This is a Kubernetes permission requirement for installing and operating a platform inside a cluster.

After installation, day-to-day workloads run under scoped service identities created for each component.

## Validation

Before the cluster is used for production traffic, Fireworks validates the environment with your team. Validation typically covers:

* GPU node readiness and scheduling behavior
* Endpoint reachability
* Routing and load balancing behavior
* Autoscaling behavior within the available GPU capacity
* Observability signals used for ongoing operations

## After onboarding

Once validation is complete, Fireworks operates the serving stack and model lifecycle in the cluster. Your team continues to own the cloud account or data center environment, network policy, GPU capacity, and any customer-side approval processes.

For ongoing responsibilities, see [Operational Model](/ecosystem/integrations/byoc/operational-model).


# Operational Model
Source: https://docs.fireworks.ai/ecosystem/integrations/byoc/operational-model

Learn how Fireworks operates Bring Your Own Cluster environments day to day.

Bring Your Own Cluster (BYOC) is designed so your team keeps ownership of the cloud or hardware environment while Fireworks operates the model serving stack inside it.

<Note>
  Bring Your Own Cluster is in Private Preview for Enterprise customers. Contact [sales@fireworks.ai](mailto:sales@fireworks.ai) to participate in the preview and confirm operational support terms for your deployment.
</Note>

## Responsibilities

In steady state, your team owns the cloud account or data center environment, Kubernetes cluster, GPU nodes, networking, and customer-side governance. Fireworks owns the serving software stack, model deployment lifecycle, scaling behavior, performance optimization, observability, upgrades, and operational support for the Fireworks-managed components.

## Shared responsibility model

BYOC is a shared operational model.

| Area                         | Customer owns                                                                           | Fireworks owns                                                                                          |
| ---------------------------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| Cloud / hardware environment | Cloud account or data center environment, Kubernetes cluster, GPU nodes, and networking | Guidance during onboarding and validation                                                               |
| Setup access                 | A dedicated, clearly named, revocable administrative credential during setup            | Installation of the serving stack and creation of scoped runtime identities                             |
| Model serving                | Available GPU capacity and required customer-side dependencies                          | Model deployment lifecycle, routing, autoscaling, performance optimization, and upgrades                |
| Reliability                  | Underlying hardware capacity                                                            | GPU and node health monitoring, automated remediation workflows, observability, and operational support |

## Model lifecycle

Fireworks deploys, updates, and version-manages inference deployments in the cluster. Customers request model or deployment changes through Fireworks, and Fireworks applies those changes through the managed BYOC operating process.

<Warning>
  Fine-tuning is not supported in BYOC during Private Preview.
</Warning>

## Autoscaling

Serving capacity scales with demand within the GPU capacity available in your cluster. Where appropriate for the workload, Fireworks can scale deployments down when idle and scale them back up when traffic returns.

Autoscaling behavior depends on model size, traffic shape, GPU availability, and customer-defined capacity constraints.

## Hybrid capacity

For customers who elect a hybrid BYOC model, Fireworks can help coordinate overflow scheduling onto Fireworks-managed capacity for eligible workloads. This is useful when traffic exceeds available BYOC capacity, customer-owned hardware is temporarily constrained, or some workloads do not need to remain in the BYOC environment.

Hybrid capacity is configured during onboarding. Fireworks works with your team to define which workloads are eligible, when overflow may be used, how routing is handled, and what data handling or compliance constraints apply.

## GPU fleet reliability

Fireworks continuously monitors GPU and node health for the serving stack. When a node or GPU becomes unhealthy, Fireworks automation detects the condition and safely remediates it, such as by removing traffic from the affected node and replacing capacity when available.

This minimizes the operational burden on your team while preserving your ownership of the underlying environment and GPU capacity.

## Capacity efficiency

Fireworks optimizes placement of workloads across the available GPUs to improve utilization. This includes consolidating compatible workloads where appropriate and balancing efficiency with reliability and performance.

## Observability and incident response

Fireworks operates the stack using metrics, logs, and dashboards for the managed serving components. Fireworks monitors the environment, investigates service-impacting issues, and coordinates with your team when an incident requires customer-side action, such as hardware capacity, cloud account or data center, networking, or policy changes.

<Note>
  Detailed Enterprise BYOC support terms are coming soon. During Private Preview, Fireworks confirms support channels, escalation paths, and any deployment-specific operational expectations during onboarding.
</Note>

## Upgrades

Fireworks updates the serving stack using managed rolling updates. Keeping the operator credential in place after setup helps Fireworks deploy upgrades, roll out model changes, and respond to incidents without requiring a new approval for every routine operational action.

Customers with strict access policies can discuss a hardened onboarding variant with Fireworks, where customer teams pre-create required cluster-scoped access structures.

## Customer coordination

BYOC operations work best when Fireworks and the customer agree on:

* Approved GPU capacity and scaling boundaries
* Whether any workloads are eligible for hybrid overflow onto Fireworks-managed capacity
* Customer-side change windows or approval requirements
* Contacts for networking, cloud account or data center, and capacity issues
* Support and escalation expectations
* Any environment-specific compliance requirements


# Bring Your Own Cluster
Source: https://docs.fireworks.ai/ecosystem/integrations/byoc/overview

Run Fireworks inference in your own Kubernetes cluster, cloud account or data center, and network boundary.

Bring Your Own Cluster (BYOC) lets Enterprise customers run Fireworks inference inside their own Kubernetes cluster. Your inference compute runs in your cloud account or data center boundary, while Fireworks installs and operates the serving stack for you.

<Warning>
  Bring Your Own Cluster is in Private Preview for Enterprise customers. Contact [sales@fireworks.ai](mailto:sales@fireworks.ai) to discuss whether BYOC is a fit and to participate in the preview.
</Warning>

## What BYOC provides

With BYOC, Fireworks deploys the managed serving software stack into Kubernetes infrastructure that you own. Fireworks operates model deployment, performance optimization, autoscaling, GPU node health and reliability, load balancing and routing, and observability for the cluster.

You continue using the Fireworks product surface: the same APIs, SDKs, model deployment workflows, and performance work available across the Fireworks platform. The main difference is where inference runs: the model serving workload runs in your environment instead of Fireworks-managed cloud infrastructure.

<Note>
  Architecture diagram coming soon. During Private Preview, Fireworks reviews the exact deployment architecture, networking boundaries, and request flow with each customer during onboarding.
</Note>

## Why choose BYOC

BYOC is designed for organizations that need more control over where inference runs without taking on the operational burden of self-hosting raw open-source serving infrastructure.

Common reasons to consider BYOC include:

* **Data residency and compliance:** Inference request and response handling runs within your Kubernetes environment, aligned to your cloud account, data center, and network requirements.
* **Existing GPU capacity:** Use GPU capacity you already own or procure in your preferred cloud or data center environment.
* **Network boundary control:** Keep inference workloads inside your cloud account or data center network architecture.
* **Managed Fireworks experience:** Fireworks runs the serving stack, applies performance optimizations, manages model deployment, and operates the cluster day to day.
* **Consistent developer interface:** Use Fireworks APIs and SDKs across serverless, dedicated Fireworks-hosted deployments, and BYOC deployments.

## When BYOC fits

BYOC is usually a fit when you need Fireworks-managed inference but have requirements that place compute or data in your own environment:

* You have data residency, compliance, or internal policy requirements for inference traffic.
* You want Fireworks to operate model serving on GPU capacity in your cloud account or data center.
* You need the same Fireworks API and managed operations model across multiple deployment environments.
* You are an Enterprise customer planning a production deployment with Fireworks support.

For workloads where Fireworks-managed infrastructure already satisfies your compliance and operational needs, serverless or dedicated deployments on Fireworks cloud may be simpler to start with.

## Benefits of Fireworks-managed operations

BYOC is not a raw self-hosting kit. Fireworks brings the managed serving experience to infrastructure you own, so your team can focus on applications instead of rebuilding model serving operations.

<CardGroup>
  <Card title="Inference performance focus" icon="gauge-high">
    Fireworks continuously optimizes serving performance across model configuration, deployment shape, scheduling, quantization, speculative decoding, and workload-specific tuning such as FireOptimizer where enabled.
  </Card>

  <Card title="Managed operations and upgrades" icon="gears">
    Fireworks handles serving-stack installation, rollout, maintenance, upgrades, and day-to-day operations so your team does not have to rebuild a self-hosted inference platform.
  </Card>

  <Card title="GPU reliability operations" icon="heart-pulse">
    Fireworks monitors GPU and node health, detects unhealthy capacity, and safely remediates issues to reduce the operational burden of running inference on large GPU fleets.
  </Card>

  <Card title="Current models and GPU generations" icon="microchip">
    Fireworks tracks new model releases and new GPU generations, including day-0 enablement for supported models and kernel-level optimizations for supported hardware, so your BYOC environment can adopt supported updates without your team rebuilding the serving stack.
  </Card>
</CardGroup>

## Hybrid BYOC and Fireworks-managed capacity

Some Enterprise customers want BYOC for their primary environment, but still want the option to use Fireworks-managed capacity for specific workloads or traffic spikes. During Private Preview, Fireworks can review hybrid BYOC patterns with your team, including overflow scheduling onto Fireworks-managed capacity when elected during onboarding and supported for the workload.

Hybrid operation can help when:

* Traffic occasionally exceeds the GPU capacity available in your cluster
* You want a fallback path while customer-owned GPU capacity is being expanded or remediated
* Some workloads can run outside the BYOC environment while others must remain within your network boundary
* Your team wants a consistent Fireworks API surface across BYOC and Fireworks-managed deployments

<Note>
  Hybrid BYOC is optional and must be reviewed during onboarding. Fireworks works with your team to confirm routing behavior, workload eligibility, data handling expectations, and any compliance constraints before enabling overflow onto Fireworks-managed capacity.
</Note>

## Known gaps during Private Preview

Fine-tuning is not supported in BYOC during Private Preview. If you need fine-tuning and BYOC together, contact [sales@fireworks.ai](mailto:sales@fireworks.ai) so the team can review your requirements and roadmap fit.

## Supported environments

Fireworks supports BYOC on major cloud providers and their managed Kubernetes offerings, select GPU cloud providers, and on-premises environments that provide a reachable Kubernetes endpoint, supported NVIDIA GPU nodes, and the required network setup. During preview onboarding, Fireworks confirms whether your target environment, GPU capacity, and networking model are supported.

At a high level, BYOC requires:

* A Kubernetes cluster with NVIDIA GPU nodes
* Outbound network access that allows Fireworks to manage the cluster
* A deployment and support plan agreed with Fireworks during Enterprise onboarding

## Next steps

If BYOC may be a fit, contact [sales@fireworks.ai](mailto:sales@fireworks.ai) to review your requirements and preview eligibility.

<CardGroup>
  <Card title="How setup works" icon="diagram-project" href="/ecosystem/integrations/byoc/how-setup-works">
    Understand prerequisites, setup access, installation, and validation.
  </Card>

  <Card title="Operational model" icon="gears" href="/ecosystem/integrations/byoc/operational-model">
    Learn how Fireworks operates BYOC clusters after onboarding.
  </Card>
</CardGroup>


# Development Setup with Fireworks Docs MCP
Source: https://docs.fireworks.ai/ecosystem/integrations/development-setup

Configure the Fireworks AI Docs MCP server for Claude Code and Cursor

## Claude Code

Add the MCP server via the CLI:

```bash theme={null}
claude mcp add --transport http fireworks-docs https://docs.fireworks.ai/mcp
```

Or add it to your project's `mcp.json`:

```json theme={null}
{
  "mcpServers": {
    "fireworks-docs": {
      "url": "https://docs.fireworks.ai/mcp"
    }
  }
}
```

## Cursor

One-click install:

[Install Fireworks Docs MCP](https://cursor.com/en/install-mcp?name=fireworks-docs\&config=eyJ1cmwiOiJodHRwczovL2RvY3MuZmlyZXdvcmtzLmFpL21jcCJ9)

Or manually add to your workspace's `mcp.json`:

```json theme={null}
{
  "mcpServers": {
    "fireworks-docs": {
      "url": "https://docs.fireworks.ai/mcp"
    }
  }
}
```

## Using the MCP Server

Once configured, your AI coding agent can search the full Fireworks AI documentation. Example queries:

* "How do I configure autoscaling for deployments?"
* "What parameters does the chat completions endpoint accept?"
* "Show me examples of function calling with Fireworks models"
* "Find the API reference for batch inference"


# LiteLLM
Source: https://docs.fireworks.ai/ecosystem/integrations/litellm

Configure Fireworks models in LiteLLM Proxy

Use [LiteLLM Proxy](https://docs.litellm.ai/docs/proxy/quick_start) as a shared gateway for Fireworks serverless models, router endpoints, and deployments. LiteLLM exposes a single OpenAI-compatible API to your developers while you manage provider credentials and access control on the server.

## Prerequisites

* A [Fireworks API key](https://app.fireworks.ai/settings/users/api-keys) (`fw_...`)
* LiteLLM v1.95.0 installed (`pip install "litellm[proxy]==1.95.0"`)

## Configure Fireworks models

Create a `config.yaml` with one entry per model you want to expose. Use the `fireworks_ai/` provider prefix and the **full** Fireworks model or router ID:

<Note>
  Always use full resource paths (for example `accounts/fireworks/models/glm-5p2` or `accounts/fireworks/routers/firerouter`). LiteLLM rewrites short names to `accounts/fireworks/models/...`, which breaks router endpoints like FireRouter.
</Note>

```yaml theme={null}
model_list:
  - model_name: accounts/fireworks/models/glm-5p2
    litellm_params:
      model: fireworks_ai/accounts/fireworks/models/glm-5p2
      api_key: os.environ/FIREWORKS_AI_API_KEY

  - model_name: accounts/fireworks/routers/kimi-k2p6-turbo
    litellm_params:
      model: fireworks_ai/accounts/fireworks/routers/kimi-k2p6-turbo
      api_key: os.environ/FIREWORKS_AI_API_KEY
```

LiteLLM sends requests to `https://api.fireworks.ai/inference/v1` by default. See the [LiteLLM Fireworks AI provider docs](https://docs.litellm.ai/docs/providers/fireworks_ai) for direct-route deployments and other options.

## Start the proxy

```bash theme={null}
export FIREWORKS_AI_API_KEY="fw_..."
litellm --config config.yaml
```

## Call a model

Clients authenticate to LiteLLM with a [virtual key](https://docs.litellm.ai/docs/proxy/virtual_keys):

```bash theme={null}
curl http://localhost:4000/chat/completions \
  -H "Authorization: Bearer $LITELLM_VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "accounts/fireworks/models/glm-5p2",
    "messages": [{"role": "user", "content": "Say pong in one word."}]
  }'
```

## API key layout

| Key                          | Who holds it                         | Used for                                |
| ---------------------------- | ------------------------------------ | --------------------------------------- |
| Fireworks API key (`fw_...`) | LiteLLM server (env or secret store) | Upstream Fireworks inference            |
| LiteLLM virtual key          | Each developer or service            | Proxy authentication and spend tracking |

A common pattern is one Fireworks service-account API key on the LiteLLM server, with per-developer virtual keys for access control and attribution.

## FireRouter

To add automatic routing between closed-source and open models, register [FireRouter](/ecosystem/firerouter/overview) in the same `model_list`. FireRouter requires an Anthropic API key for pass-through. See [FireRouter with LiteLLM](/ecosystem/firerouter/litellm).

## Related

* [LiteLLM Proxy quick start](https://docs.litellm.ai/docs/proxy/quick_start)
* [LiteLLM Fireworks AI provider](https://docs.litellm.ai/docs/providers/fireworks_ai)
* [FireRouter](/ecosystem/firerouter/overview): managed routing between open and closed-source models
* [OpenAI compatibility](/tools-sdks/openai-compatibility): Fireworks inference API reference


# MLOps & Observability
Source: https://docs.fireworks.ai/ecosystem/integrations/mlops-observability

Track and monitor your Fireworks AI deployments with leading MLOps and observability platforms

Fireworks AI integrates with industry-leading MLOps and observability platforms to help you monitor, track, and optimize your AI applications in production.

## Supported Platforms

<CardGroup>
  <Card title="Weights & Biases" icon="chart-line" href="/ecosystem/integrations/wandb">
    Track fine-tuning experiments and visualize training metrics with W\&B
  </Card>

  <Card title="MLflow" icon="chart-mixed" href="https://mlflow.org/docs/latest/genai/tracing/integrations/listing/fireworksai/">
    Mlflow Tracing to track prompts, outputs, latency etc as your build AI applications with FireworksAI
  </Card>
</CardGroup>

## Need Help?

For assistance with MLOps and observability integrations, [contact our team](https://fireworks.ai/contact) or join our [Discord community](https://discord.gg/fireworks-ai).


# Cookbooks
Source: https://docs.fireworks.ai/examples/cookbooks

Interactive Jupyter notebooks demonstrating advanced use cases and best practices with Fireworks AI

Explore our collection of notebooks that showcase real-world applications, best practices, and advanced techniques for building with Fireworks AI.

## Fine-Tuning & Training

<CardGroup>
  <Card title="Knowledge Distillation" href="https://colab.research.google.com/github/fw-ai/cookbook/blob/main/archived/learn/finetuning/knowledge_distillation.ipynb" icon="graduation-cap">
    Transfer large model capabilities to efficient models using a two-stage SFT + RFT approach.

    **Techniques:** Supervised Fine-Tuning (SFT) + Reinforcement Fine-Tuning (RFT)

    **Results:** 52% → 70% accuracy on GSM8K mathematical reasoning
  </Card>

  <Card title="VLM Fine-tuning + Evals" href="https://huggingface.co/spaces/fireworks-ai/catalog-extract/blob/main/notebooks/01-eda-and-fine-tuning.ipynb" icon="image">
    Beat frontier closed-source models for product catalog cleansing with vision-language model fine-tuning.

    **Techniques:** Supervised Fine-Tuning (SFT)

    **Results:** 48% increase in quality from base model
  </Card>
</CardGroup>

## Multimodal AI

<CardGroup>
  <Card title="NVIDIA Nemotron VL for Document Intelligence" href="https://colab.research.google.com/github/fw-ai/cookbook/blob/main/archived/multimedia/vlm/nvidia-nemotron-vl/NVIDIA-Nemotron-v2-VL-cookbook.ipynb" icon="file-invoice">
    Extract structured data from invoices, forms, and financial documents using state-of-the-art OCR and document understanding.

    **Use Cases:** Forms, invoices, financial documents, product catalogs

    **Results:** 90.8% accuracy on invoice extraction (100% on invoice numbers and dates)
  </Card>

  <Card title="Audio Streaming Speech-to-Text" href="https://colab.research.google.com/github/fw-ai/cookbook/blob/main/archived/learn/audio/audio_streaming_speech_to_text/audio_streaming_speech_to_text.ipynb" icon="microphone">
    Real-time audio transcription with streaming support and low latency.

    **Features:** Streaming support, low-latency transcription, production-ready
  </Card>

  <Card title="Chat with Video using Qwen3 Omni" href="https://colab.research.google.com/github/fw-ai/cookbook/blob/main/archived/multimedia/video/Qwen3-Omni-Chat-With-Video-Cookbook.ipynb" icon="video">
    Analyze video and audio content with Qwen3 Omni, a multimodal model supporting video, audio, and text inputs.

    **Features:** Video captioning, scene analysis, content understanding, multimodal Q\&A
  </Card>
</CardGroup>

## Embeddings & RAG

<CardGroup>
  <Card title="Two-Stage Retrieval with Voyage Embeddings and Reranking" href="https://colab.research.google.com/github/fw-ai/cookbook/blob/main/partners/voyage-ai/voyage_two_stage_retrieval.ipynb" icon="magnifying-glass">
    Build a retrieval pipeline that recalls candidates with vector search, then reorders them with a reranker for precision.

    **Techniques:** Voyage embeddings, MongoDB `$vectorSearch`, Voyage ReRank 2.5

    **Models:** Voyage AI embedders and rerankers on dedicated deployments
  </Card>
</CardGroup>

## API Features

<CardGroup>
  <Card title="Fireworks MCP Examples" href="https://colab.research.google.com/github/fw-ai/cookbook/blob/main/archived/learn/response-api/fireworks_mcp_examples.ipynb" icon="code">
    Leverage Model Context Protocol (MCP) for GitHub repository analysis, code search, and documentation Q\&A.

    **Features:** Repository analysis, code search, documentation Q\&A, GitMCP integration

    **Models:** Qwen 3 235B with external tool support
  </Card>
</CardGroup>


# Courses
Source: https://docs.fireworks.ai/examples/introduction

Standalone end-to-end examples showing how to use Fireworks to solve real-world use cases

<CardGroup>
  <Card title="Text to SQL" href="/examples/text-to-sql">
    Learn how to use Fireworks to fine-tune a model to convert natural language to SQL queries.
  </Card>

  <Card title="Reward Hacking" href="/examples/reward-hacking">
    Learn how to build reinforcement learning systems that avoid reward hacking.
  </Card>

  <Card title="Knowledge Distillation" href="/examples/knowledge-distillation">
    Learn to distill the knowledge of large AI models into efficient, deployable alternatives.
  </Card>
</CardGroup>


# How do I close my Fireworks.ai account?
Source: https://docs.fireworks.ai/faq-new/account-access/how-do-i-close-my-fireworksai-account



To close your account:

1. Email [inquiries@fireworks.ai](mailto:inquiries@fireworks.ai)
2. Include in your request:
   * Your account ID
   * A clear request for account deletion

Before closing your account, please ensure:

* All outstanding invoices are paid, and any payment issues on prepaid accounts are resolved
* Any active deployments are terminated
* Important data is backed up if needed


# I have multiple Fireworks accounts. When I try to login with Google on Fireworks' web UI, I'm getting signed into the wrong account. How do I fix this?
Source: https://docs.fireworks.ai/faq-new/account-access/i-have-multiple-fireworks-accounts-when-i-try-to-login-with-google-on-fireworks



If you log in with Google, account management is controlled by Google. You can log in through an incognito mode or create separate Chrome/browser profiles to log in with different Google accounts. You could also follow the steps in this [guide](https://support.google.com/accounts/answer/13533235?hl=en#zippy=%2Csign-in-with-google) to disassociate Fireworks.ai with a particular Google account sign-in. If you have more complex issues please contact us on Discord.


# What email does GitHub authentication use?
Source: https://docs.fireworks.ai/faq-new/account-access/what-email-does-github-authentication-use



When you authenticate with Fireworks using GitHub, we use the **primary email address** associated with your GitHub account for identification and account management.

## How it works

Fireworks automatically retrieves your primary email address from your GitHub profile during the authentication process. This email address becomes your Fireworks account identifier.

## Managing your primary email

To change your primary email address on GitHub:

1. Go to your [GitHub email settings](https://github.com/settings/emails)
2. Select the email address you want to set as primary in the "Primary email address" section

<Info>
  You can also follow the [GitHub documentation](https://docs.github.com/en/enterprise-cloud@latest/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-email-preferences/changing-your-primary-email-address) for detailed instructions on managing email preferences.
</Info>

## Switching between accounts

You can easily switch which Fireworks account your GitHub authentication logs into by changing your primary email address on GitHub before logging in. This allows you to:

* Log into different Fireworks accounts using the same GitHub account
* Switch between personal and work accounts by updating your GitHub primary email
* Maintain separate billing and usage tracking for different email addresses

The authentication will use whatever email is set as primary at the time of login, so you can switch accounts by simply updating your GitHub primary email before authenticating.


# What email does LinkedIn authentication use?
Source: https://docs.fireworks.ai/faq-new/account-access/what-email-does-linkedin-authentication-use



When you authenticate with Fireworks using LinkedIn, we use the **primary email address** associated with your LinkedIn account for identification and account management.

## How it works

Fireworks automatically retrieves your primary email address from your LinkedIn profile during the authentication process. This email address becomes your Fireworks account identifier.

## Managing your primary email

To change your primary email address on LinkedIn:

1. Go to your [LinkedIn email settings](https://www.linkedin.com/mypreferences/d/manage-email-addresses)
2. From there, you can add new email addresses or change your primary email
3. Click **Add email address** to add a new email or select an existing one to make primary

<Info>
  You can also follow the [LinkedIn documentation](https://www.linkedin.com/help/linkedin/answer/a519904) for detailed instructions on managing email preferences.
</Info>

## Switching between accounts

You can easily switch which Fireworks account your LinkedIn authentication logs into by changing your primary email address on LinkedIn before logging in. This allows you to:

* Log into different Fireworks accounts using the same LinkedIn account
* Switch between personal and work accounts by updating your LinkedIn primary email
* Maintain separate billing and usage tracking for different email addresses

The authentication will use whatever email is set as primary at the time of login, so you can switch accounts by simply updating your LinkedIn primary email before authenticating.


# What should I do if I can't access my company account after being invited when I already have a personal account?
Source: https://docs.fireworks.ai/faq-new/account-access/what-should-i-do-if-i-cant-access-my-company-account-after-being-invited-when-i



This issue can occur when you have multiple accounts associated with the same email address (e.g., a personal account created with Google login and a company account you've been invited to).

To resolve this:

1. Email [inquiries@fireworks.ai](mailto:inquiries@fireworks.ai) from the email address associated with both accounts
2. Include in your email:
   * The account ID you created personally (e.g., username-44ace8)
   * The company account ID you need access to (e.g., company-a57b2a)
   * Mention that you're having trouble accessing your company account

Note: This is a known scenario that support can resolve once they verify your email ownership.


# Are there discounts for bulk usage?
Source: https://docs.fireworks.ai/faq-new/billing-pricing/are-there-discounts-for-bulk-usage



We offer discounts for bulk or pre-paid purchases. Contact [inquiries@fireworks.ai](mailto:inquiries@fireworks.ai) to discuss volume pricing.


# Are there extra fees for serving fine-tuned models?
Source: https://docs.fireworks.ai/faq-new/billing-pricing/are-there-extra-fees-for-serving-fine-tuned-models



Fine-tuned (LoRA) models require a dedicated deployment to serve. Here's what you need to know:

**What you pay for**:

* **Deployment costs** on a per-GPU-second basis for hosting the model
* **The fine-tuning process** itself, if applicable

**Deployment options**:

* **Live-merge deployment**: Deploy your LoRA model with weights merged into the base model for optimal performance
* **Multi-LoRA deployment**: Deploy up to 100 LoRA models as addons on a single base model deployment

<Tip>
  For more details on deploying fine-tuned models, see the [Deploying Fine Tuned Models guide](/fine-tuning/deploying-loras).
</Tip>


# How does billing and credit usage work?
Source: https://docs.fireworks.ai/faq-new/billing-pricing/how-does-billing-and-credit-usage-work



<Note>
  Contracted customers may have the option to move to post-paid billing. [Contact our sales team](https://fireworks.ai/company/contact-us) to discuss your options.
</Note>

Fireworks operates on a **pre-paid credits** billing system. You purchase credits to use the platform:

* Add a valid payment method and billing address, then purchase credits.
* Usage across serverless, on-demand deployments, and fine-tuning deducts from your credit balance.
* If your balance reaches zero and Auto Reload is not enabled, usage pauses until you add credits.
* You can configure Auto Reload to purchase credits when your balance is low.
* Separately, you can set a monthly spend limit for your usage. Adding credits does not raise this limit.

<Note>
  Enterprise accounts do not have the same self-serve limits. Their monthly spend alerts track the cost of Fireworks usage, including usage paid for with credits. Adding credits itself does not count as spend. See [Enterprise quotas](/faq/enterprise/service/quotas) for more information.
</Note>

<Tip>
  For details on spend limits and quota controls, see our [Account quotas guide](/guides/quotas_usage/account-quotas#view-and-adjust-your-spend-limit).
</Tip>


# How many tokens per image?
Source: https://docs.fireworks.ai/faq-new/billing-pricing/how-many-tokens-per-image

Learn how to calculate token usage for images in vision models and understand pricing implications

Image token consumption varies by model and resolution, typically ranging from 1,000 to 2,500 tokens per image for most common resolutions.

## Common resolution token counts

The following table shows the token counts for a single image for Qwen2.5 VL at different image resolutions:

| Resolution | Token Count |
| ---------- | ----------- |
| 336×336    | 144         |
| 672×672    | 576         |
| 1024×1024  | 1,369       |
| 1280×720   | 1,196       |
| 1920×1080  | 2,769       |
| 2560×1440  | 4,641       |
| 3840×2160  | 10,549      |

## Calculating exact token count for your images

You can determine exact token usage by processing your images through the model's tokenizer.
For instance, for Qwen2.5 VL, you can use the following code:

<Steps>
  <Step title="Install dependencies">
    ```bash theme={null}
    pip install torch torchvision transformers pillow
    ```
  </Step>

  <Step title="Tokenize your image">
    ```python Tokenizing your image theme={null}
    import requests
    from PIL import Image
    from transformers import AutoProcessor
    import os

    # Your image source - can be URL or local path
    IMAGE_URL_OR_PATH = "https://images.unsplash.com/photo-1519125323398-675f0ddb6308"

    def load_image(source):
        """Load image from URL or local file path"""
        if source.startswith(('http://', 'https://')):
            print(f"Downloading image from URL: {source}")
            response = requests.get(source)
            response.raise_for_status()
            return Image.open(requests.get(source, stream=True).raw)
        else:
            print(f"Loading image from path: {source}")
            if not os.path.exists(source):
                raise FileNotFoundError(f"Image file not found: {source}")
            return Image.open(source)

    def count_image_tokens(image):
        """Count how many tokens an image takes using Qwen 2.5 VL processor"""
        processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")
        
        messages = [
            {
                "role": "user",
                "content": [
                    {"type": "image", "image": image},
                    {"type": "text", "text": "What's in this image?"},
                ],
            }
        ]
        
        text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
        inputs = processor(text=text, images=[image], return_tensors="pt")
        input_ids = inputs["input_ids"][0]
        
        # Count the image pad tokens (151655 is Qwen2.5 VL's image token ID)
        image_tokens = (input_ids == 151655).sum().item()
        
        return image_tokens, input_ids

    def main():
        import sys
        
        image_source = sys.argv[1] if len(sys.argv) > 1 else IMAGE_URL_OR_PATH
        
        print(f"Processing image: {image_source}")
        image = load_image(image_source)
        print(f"Image size: {image.size}")
        print(f"Image mode: {image.mode}")
        
        print("\nCalculating tokens...")
        image_tokens, input_ids = count_image_tokens(image)
        
        print(f"Total tokens: {len(input_ids)}")
        print(f"Image tokens: {image_tokens}")
        print(f"Text tokens: {len(input_ids) - image_tokens}")
        
    if __name__ == "__main__":
        main()
    ```

    ```bash Usage theme={null}
    # Calculate tokens for an image URL
    python token_calculator.py "https://example.com/image.jpg"

    # Calculate tokens for a local image
    python token_calculator.py "path/to/your/image.png"
    ```
  </Step>
</Steps>


# How much does Fireworks cost?
Source: https://docs.fireworks.ai/faq-new/billing-pricing/how-much-does-fireworks-cost



<Note>
  Contracted customers may have the option to move to post-paid billing. [Contact our sales team](https://fireworks.ai/company/contact-us) to discuss your options.
</Note>

Fireworks AI uses a **usage-based pre-paid** billing system. You purchase credits, then usage is deducted based on:

* **Per token** for serverless inference
* **Per GPU usage time** for on-demand deployments
* **Per token of training data** for fine-tuning

For customers needing **enterprise-grade security and reliability**, please reach out to us at [inquiries@fireworks.ai](mailto:inquiries@fireworks.ai) to discuss options.

Find out more about our current pricing on our [Pricing page](https://fireworks.ai/pricing).


# Is prompt caching billed differently for serverless models?
Source: https://docs.fireworks.ai/faq-new/billing-pricing/is-prompt-caching-billed-differently



Yes, **cached prompt tokens are discounted compared to uncached tokens for serverless models**. The default discount is 50%, but the exact discount varies by model. Check the [Model Library](https://fireworks.ai/models) for model-specific cached and uncached input token pricing.


# How do credits work?
Source: https://docs.fireworks.ai/faq-new/billing-pricing/what-happens-when-i-finish-my-1-dollar-credit



<Note>
  Contracted customers may have the option to move to post-paid billing. [Contact our sales team](https://fireworks.ai/company/contact-us) to discuss your options.
</Note>

## How credits are applied

Fireworks operates on a **pre-paid credits** billing system. You purchase credits to use the platform:

* Credits are used first for all usage.
* If credits are exhausted and Auto Reload is disabled, usage pauses until you add credits.
* If Auto Reload is enabled, credits are purchased automatically when your balance reaches your configured minimum.
* You can separately set a monthly spend limit for your usage. Adding credits does not raise this limit.

## Missing credits after purchase?

If you don't see your credits reflected immediately:

1. Visit your **billing dashboard**
2. Review the **"Credits"** section
3. Check your **credit balance** and **Auto Reload settings**

**Important**: Usage consumes available credits. If your balance is low, enable Auto Reload to avoid interruptions.

## Why did I receive an invoice after depositing credits?

Most accounts on pre-paid billing should not see month-end overage invoices. If you received an invoice, your account may be on a post-paid contract. Contact [community\_billing@fireworks.ai](mailto:community_billing@fireworks.ai) so we can confirm your billing configuration.

## What happens when I finish my \$1 credit?

When you finish your \$1 credit, the following occurs:

## Account Status

* **Without payment method**: Your account will be **suspended** until you add a payment method. For request-rate behavior, see [Account quotas](/guides/quotas_usage/account-quotas#account-wide-request-limits); for serverless TPM upper bounds, see [Serverless rate limits](/serverless/rate-limits).
* **With payment method**: Add credits to continue usage. [Account-wide request limits](/guides/quotas_usage/account-quotas#account-wide-request-limits) increase, and [serverless TPM upper bounds](/serverless/rate-limits) grow as your account spend tier rises.

**Payment Method Requirements:**

* Adding a payment method is required to continue service after credit depletion
* Add credits (or enable Auto Reload) to continue service after credit depletion
* As you spend more with Fireworks, your adaptive usage limits and serverless TPM upper bounds can increase

## Where's my receipt for purchased credits?

Receipts for purchased credits are sent via Stripe upon purchase. Check your email for receipts from Stripe (not Fireworks). If you can't find your receipt, contact [community\_billing@fireworks.ai](mailto:community_billing@fireworks.ai).

<Note>
  For spend limits, tiers, and account-wide request limits, see [Account quotas](/guides/quotas_usage/account-quotas). For adaptive serverless TPM upper bounds, see [Serverless rate limits](/serverless/rate-limits).
</Note>


# Why might my account be suspended even with remaining credits?
Source: https://docs.fireworks.ai/faq-new/billing-pricing/why-might-my-account-be-suspended-even-with-remaining-credits



Your account may be suspended due to several factors:

1. **Monthly spend limit reached**:
   * Your monthly spend limit can pause usage even if you still have a credit balance.
   * Increase your spend limit in Billing to resume usage. Adding credits alone does not raise the limit.

2. **Payment or risk checks**:
   * Accounts may be temporarily paused if payment verification fails.
   * In some cases, manual review can temporarily limit usage.

3. **Post-paid contract terms**:
   * Contracted customers on post-paid billing may have different suspension rules under their agreement.
   * Enterprise monthly spend alerts are informational and do not pause service.

<Note>
  If you're experiencing account suspension issues or need assistance with your spend limit or billing settings, please contact [inquiries@fireworks.ai](mailto:inquiries@fireworks.ai).
</Note>


# Are there any quotas for serverless?
Source: https://docs.fireworks.ai/faq-new/deployment-infrastructure/are-there-any-quotas-for-serverless



Yes. Standard serverless, Priority tier, and Fast all have serverless rate limits and quotas.

For the detailed serverless policy, see our [Serverless rate limits guide](/serverless/rate-limits).


# Do you provide notice before removing model availability?
Source: https://docs.fireworks.ai/faq-new/deployment-infrastructure/do-you-provide-notice-before-removing-model-availability



Yes, we provide advance notice before removing models from the serverless infrastructure:

* **Minimum 2 weeks’ notice** before model removal
* Longer notice periods may be provided for **popular models**, depending on usage
* Higher-usage models may have extended deprecation timelines

**Best Practices**:

1. Monitor announcements regularly.
2. Prepare a migration plan in advance.
3. Test alternative models to ensure continuity.
4. Keep your contact information updated for timely notifications.


# Do you support Auto Scaling?
Source: https://docs.fireworks.ai/faq-new/deployment-infrastructure/do-you-support-auto-scaling



Yes, our system supports **auto scaling** with the following features:

* **Scaling down to zero** capability for resource efficiency
* Controllable **scale-up and scale-down velocity**
* **Custom scaling rules and thresholds** to match your specific needs


# How does autoscaling affect my costs?
Source: https://docs.fireworks.ai/faq-new/deployment-infrastructure/how-does-autoscaling-affect-my-costs



* **Scaling from 0**: No minimum cost when scaled to zero
* **Scaling up**: Each new replica adds to your total cost proportionally. For example:
  * Scaling from 1 to 2 replicas doubles your GPU costs
  * If each replica uses multiple GPUs, costs scale accordingly (e.g., scaling from 1 to 2 replicas with 2 GPUs each means paying for 4 GPUs total)

For current pricing details, please visit our [pricing page](https://fireworks.ai/pricing).


# How does billing and scaling work for on-demand GPU deployments?
Source: https://docs.fireworks.ai/faq-new/deployment-infrastructure/how-does-billing-and-scaling-work-for-on-demand-gpu-deployments



On-demand GPU deployments have unique billing and scaling characteristics compared to serverless deployments:

**Billing**:

* Charges start when the server begins accepting requests
* **Billed by GPU-second** for each active instance
* Costs accumulate even if there are no active API calls

**Scaling options**:

* Supports **autoscaling** from 0 to multiple GPUs
* Each additional GPU **adds to the billing rate**
* Can handle unlimited requests within the GPU’s capacity

**Management requirements**:

* Not fully serverless; requires some manual management
* **Manually delete deployments** when no longer needed
* Or configure autoscaling to **scale down to 0** during inactive periods

**Cost control tips**:

* Regularly **monitor active deployments**
* **Delete unused deployments** to avoid unnecessary costs
* Consider **serverless options** for intermittent usage
* Use **autoscaling to 0** to optimize costs during low-demand times


# How does billing work for on-demand deployments?
Source: https://docs.fireworks.ai/faq-new/deployment-infrastructure/how-does-billing-work-for-on-demand-deployments



On-demand deployments come with automatic cost optimization features:

* **Default autoscaling**: Automatically scales to 0 replicas when not in use
* **Pay for what you use**: Charged only for GPU time when replicas are active
* **Flexible configuration**: Customize autoscaling behavior to match your needs

**Best practices for cost management**:

1. **Leverage default autoscaling**: The system automatically scales down deployments when not in use
2. **Customize carefully**: While you can modify autoscaling behavior using our [configuration options](https://docs.fireworks.ai/guides/ondemand-deployments#customizing-autoscaling-behavior), note that preventing scale-to-zero will result in continuous GPU charges
3. **Consider your use case**: For intermittent or low-frequency usage, serverless deployments might be more cost-effective

For detailed configuration options, see our [deployment guide](https://docs.fireworks.ai/guides/ondemand-deployments#replica-count-horizontal-scaling).


# How does the system scale?
Source: https://docs.fireworks.ai/faq-new/deployment-infrastructure/how-does-the-system-scale



Our system is **horizontally scalable**, meaning it:

* Scales linearly with additional **replicas** of the deployment
* **Automatically allocates resources** based on demand
* Manages **distributed load handling** efficiently


# Are there SLAs for serverless?
Source: https://docs.fireworks.ai/faq-new/deployment-infrastructure/is-latency-guaranteed-for-serverless-models



Our multi-tenant serverless offering does not currently come with Service Level Agreements (SLAs) for latency or availability.

If you have specific performance or availability requirements, we recommend:

* **On-demand deployments**: Provides dedicated resources with predictable performance
* **Contact sales**: [Reach out to discuss](https://fireworks.ai/company/contact-us) custom solutions and enterprise options


# What are the rate limits for on-demand deployments?
Source: https://docs.fireworks.ai/faq-new/deployment-infrastructure/what-are-the-rate-limits-for-on-demand-deployments



On-demand deployments have GPU quotas that determine your maximum allocation.

For detailed information about on-demand deployment quotas and GPU limits, see our [Account quotas guide](/guides/quotas_usage/account-quotas#on-demand-deployment-quotas).

<Tip>
  Need higher GPU allocations? [Contact us](https://fireworks.ai/company/contact-us) to discuss custom solutions for your use case.
</Tip>


# What factors affect the number of simultaneous requests that can be handled?
Source: https://docs.fireworks.ai/faq-new/deployment-infrastructure/what-factors-affect-the-number-of-simultaneous-requests-that-can-be-handled



The request handling capacity is influenced by multiple factors:

* **Model size and type**
* **Number of GPUs** allocated to the deployment
* **GPU type** (e.g., A100 vs. H100)
* **Prompt size** and **generation token length**
* **Deployment type** (serverless vs. on-demand)


# What’s the supported throughput?
Source: https://docs.fireworks.ai/faq-new/deployment-infrastructure/whats-the-supported-throughput



Throughput capacity typically depends on several factors:

* **Deployment type** (serverless or on-demand)
* **Traffic patterns** and **request patterns**
* **Hardware configuration**
* **Model size and complexity**


# Why am I experiencing request timeout errors and slow response times with serverless LLM models?
Source: https://docs.fireworks.ai/faq-new/deployment-infrastructure/why-am-i-experiencing-request-timeout-errors-and-slow-response-times-with-server



Timeout errors and increased response times can occur due to **server load during high-traffic periods**.

With serverless, users are essentially **sharing a pool of GPUs** with models pre-provisioned.
The goal of serverless is to allow users and teams to **seamlessly power their generative applications** with the **latest generative models** in **less than 5 lines of code**.
Deployment barriers should be **minimal** and **pricing is based on usage**.

However there are trade-offs with this approach, namely that in order to ensure users have **consistent access** to the most in-demand models, users are also subject to **minor latency and performance variability** during **high-volume periods**.
With **on-demand deployments**, users are reserving GPUs (which are **billed by rented time** instead of usage volume) and don't have to worry about traffic spikes.

Which is why our two recommended ways to address timeout and response time issues is:

### Current solution (recommended for production)

* **Use on-demand deployments** for more stable performance
* **Guaranteed response times**
* **Dedicated resources** to ensure availability

We are always investing in ways to improve speed and performance.

### Upcoming improvements

* Enhanced SLAs for uptime
* More consistent generation speeds during peak load times

If you experience persistent issues, please include the following details in your support request:

1. Exact **model name**
2. **Timestamp** of errors (in UTC)
3. **Frequency** of timeouts
4. **Average wait times**

### Performance optimization tips

* Consider **batch processing** for handling bulk requests
* Implement **retry logic with exponential backoff**
* Monitor **usage patterns** to identify peak traffic times
* Set **appropriate timeout settings** based on model complexity


# Does Fireworks support custom base models?
Source: https://docs.fireworks.ai/faq-new/models-inference/does-fireworks-support-custom-base-models



Yes, custom base models can be deployed via **firectl**. You can learn more about custom model deployment in our [guide on uploading custom models](https://docs.fireworks.ai/models/uploading-custom-models).


# Does the API support batching and load balancing?
Source: https://docs.fireworks.ai/faq-new/models-inference/does-the-api-support-batching-and-load-balancing



Current capabilities include:

* **Load balancing**: Yes, supported out of the box
* **Continuous batching**: Yes, supported
* **Batch inference**: Yes, supported via the [Batch API](/guides/batch-inference)
* **Streaming**: Yes, supported

For asynchronous batch processing of large volumes of requests, see our [Batch API documentation](/guides/batch-inference).


# How do I control output image sizes when using SDXL ControlNet?
Source: https://docs.fireworks.ai/faq-new/models-inference/how-do-i-control-output-image-sizes-when-using-sdxl-controlnet



When using **SDXL ControlNet** (e.g., canny control), the output image size is determined by the explicit **width** and **height** parameters in your API request:

The input control signal image will be automatically:

* **Resized** to fit your specified dimensions
* **Cropped** to preserve aspect ratio

**Example**: To generate a 768x1344 image, explicitly include these parameters in your request:

```json theme={null}
{
    "width": 768,
    "height": 1344
}
```

*Note*: While these parameters may not appear in the web interface examples, they are supported API parameters that can be included in your requests.


# How to check if a model is available on serverless?
Source: https://docs.fireworks.ai/faq-new/models-inference/how-to-check-if-a-model-is-available-on-serverless



## Web UI

Go to [https://app.fireworks.ai/models?filter=LLM\&serverless=true](https://app.fireworks.ai/models?filter=LLM\&serverless=true)

## API

You can programmatically retrieve all serverless models using the [List Models API](/api-reference/list-models) with the `supports_serverless=true` filter.

<Tabs>
  <Tab title="Python (Fireworks SDK)">
    ```python theme={null}
    from fireworks import Fireworks

    client = Fireworks()

    # List all serverless models
    models = client.models.list(filter="supports_serverless=true")

    for model in models:
        print(model.name)
    ```

    You can also combine filters and customize the response:

    ```python theme={null}
    # List serverless models with pagination
    models = client.models.list(
        filter="supports_serverless=true",
        page_size=50,
    )

    for model in models:
        print(f"{model.name}: {model.display_name}")
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    curl "https://api.fireworks.ai/v1/accounts/fireworks/models?filter=supports_serverless%3Dtrue" \
      -H "Authorization: Bearer $FIREWORKS_API_KEY"
    ```

    With pagination:

    ```bash theme={null}
    curl "https://api.fireworks.ai/v1/accounts/fireworks/models?filter=supports_serverless%3Dtrue&pageSize=50" \
      -H "Authorization: Bearer $FIREWORKS_API_KEY"
    ```
  </Tab>
</Tabs>

The filter parameter uses the [AIP-160 filter syntax](https://google.aip.dev/160). The `supports_serverless` field indicates whether a model is available on serverless infrastructure.

<Tip>
  See the [List Models API reference](/api-reference/list-models) for all available parameters including `order_by`, `page_size`, and `read_mask`.
</Tip>


# There’s a model I would like to use that isn’t available on Fireworks. Can I request it?
Source: https://docs.fireworks.ai/faq-new/models-inference/theres-a-model-i-would-like-to-use-that-isnt-available-on-fireworks-can-i-reques



Fireworks supports a wide array of custom models and actively takes feature requests for new, popular models to add to the platform.

**To request new models**:

1. **Join our [Discord server](https://discord.gg/fireworks-ai)**
2. Let us know which models you’d like to see
3. Provide **use case details**, if possible, to help us prioritize

We regularly evaluate and add new models based on:

* **Community requests**
* **Popular demand**
* **Technical feasibility**
* **Licensing requirements**


# What factors affect the number of simultaneous requests that can be handled?
Source: https://docs.fireworks.ai/faq-new/models-inference/what-factors-affect-the-number-of-simultaneous-requests-that-can-be-handled



Request handling capacity depends on several factors:

* **Model size and type**
* **Number of GPUs allocated** to the deployment
* **GPU type** (e.g., A100, H100)
* **Prompt size**
* **Generation token length**
* **Deployment type** (serverless vs. on-demand)


# Agent Skills
Source: https://docs.fireworks.ai/fine-tuning/agent/use-with-coding-agents

Install the Fireworks training skill for your coding agent.

The `fireworks-training` skill helps coding agents configure, run, and troubleshoot training jobs using current Fireworks best practices.

## Install

### Claude Code

Install the auto-updating cookbook plugin:

```bash theme={null}
claude plugin marketplace add fw-ai/cookbook
claude plugin install fireworks-training@fw-ai-cookbook
```

### Cursor

```bash theme={null}
npx --yes skills add fw-ai/cookbook -g \
  -s fireworks-training -a cursor -y
```

### Codex

```bash theme={null}
npx --yes skills add fw-ai/cookbook -g \
  -s fireworks-training -a codex -y
```

### Other compatible agents

Install to every detected Agent Skills-compatible harness:

```bash theme={null}
npx --yes skills add fw-ai/cookbook -g \
  -s fireworks-training -a '*' -y
```

<Note>
  Skills installed with `npx skills` do not update automatically. Refresh them with `npx --yes skills update -g -y`.
</Note>

## Prerequisites

* [Fireworks CLI (`firectl`)](/tools-sdks/firectl/firectl) installed. Authenticate with either `firectl signin` or `FIREWORKS_API_KEY`.
* Export `FIREWORKS_API_KEY` for Training API Python workflows.

## What it does

Ask in plain language, for example *"Fine-tune qwen3-4b on my `train.jsonl` and deploy it."* The skill configures the job, validates inputs, runs it, and helps troubleshoot failures.

Before creating cost or resources, it shows the resolved parameters and estimated cost for confirmation.

## Usage data and privacy

To improve the skill, authenticated API calls include the skill version and a random session ID. Fireworks uses this metadata with existing account and training job records for internal product analytics. Prompts and datasets are not collected, and usage data is not shared outside Fireworks.

<Note>
  The Fireworks CLI (`firectl`) may block mutating commands inside an AI-agent environment. When that happens, the skill gives you the exact command to run manually, then resumes monitoring and reporting.
</Note>

Use [managed fine-tuning](/fine-tuning/managed-finetuning-intro) for standard jobs, or the [Training API](/fine-tuning/training-api/introduction) for custom loops on [serverless or dedicated infrastructure](/fine-tuning/training-api/introduction#infrastructure).

## See also

<CardGroup>
  <Card title="Training Overview" icon="compass" href="/fine-tuning/finetuning-intro#choose-a-surface">
    Pick managed fine-tuning vs the Training API before you install.
  </Card>

  <Card title="Managed Fine-Tuning" icon="sliders" href="/fine-tuning/managed-finetuning-intro">
    Drive the same training infra directly when you know your config.
  </Card>

  <Card title="Training API" icon="code" href="/fine-tuning/training-api/introduction">
    Write your own Python training loop on Fireworks GPUs.
  </Card>

  <Card title="Choose infrastructure" icon="server" href="/fine-tuning/training-api/introduction#infrastructure">
    Compare serverless and dedicated training.
  </Card>

  <Card title="CLI reference" icon="terminal" href="/tools-sdks/firectl/firectl">
    Automate managed training with `firectl`.
  </Card>

  <Card title="API reference" icon="brackets-curly" href="/api-reference/introduction">
    Automate managed training through REST APIs.
  </Card>

  <Card title="Cookbook" icon="book" href="/fine-tuning/training-api/cookbook/overview">
    Ready-to-run recipes, including the inline-reward RL loop.
  </Card>
</CardGroup>


# Remote Environment Setup
Source: https://docs.fireworks.ai/fine-tuning/connect-environments

Implement the /init endpoint to run evaluations in your infrastructure

If you already have an agent running in your product, or need to run rollouts on your own infrastructure, you can integrate it with RFT using the `RemoteRolloutProcessor`. This delegates rollout execution to an HTTP service you control.

Remote agent are ideal for:

* Multi-turn agentic workflows with tool use
* Access to private databases, APIs, or internal services
* Integration with existing agent codebases
* Complex simulations that require your infrastructure

<Tip>
  New to RFT? Start with [local agent](/fine-tuning/quickstart-math) instead. They're simpler and cover most use cases. Only use remote agent environments when you need access to private infrastructure or have an existing agent to integrate.
</Tip>

## How remote rollouts work

<Frame>
  <img alt="Remote rollout processor flow diagram showing the interaction between Eval Protocol, your remote server, and Fireworks Tracing" />
</Frame>

<Steps>
  <Step title="Fireworks triggers rollout">
    During training, Fireworks calls your service's `POST /init` endpoint with the dataset row and correlation metadata.
  </Step>

  <Step title="Your service processes the rollout">
    Your agent executes the task (e.g., multi-turn conversation, tool calls, simulation steps), logging progress via Fireworks tracing.
  </Step>

  <Step title="Status reporting">
    Your service sends structured logs tagged with rollout metadata to Fireworks so the system can track completion.
  </Step>

  <Step title="Evaluation">
    Once Fireworks detects completion, it pulls the full trace and evaluates it using your scoring logic.
  </Step>
</Steps>

<Note>
  Everything except implementing your remote server is handled automatically by Eval Protocol. You only need to implement the `/init` endpoint and add Fireworks tracing.
</Note>

## Implementing the /init endpoint

Your remote service must implement a single `/init` endpoint that accepts rollout requests.

### Request schema

<ParamField type="object">
  Model configuration including model name and inference parameters like temperature, max\_tokens, etc.
</ParamField>

<ParamField type="array">
  Array of conversation messages to send to the model
</ParamField>

<ParamField type="array">
  Array of available tools for the model (for function calling)
</ParamField>

<ParamField type="string">
  Base URL for making LLM calls through Fireworks tracing (includes correlation metadata)
</ParamField>

<ParamField type="object">
  Rollout execution metadata for correlation (rollout\_id, run\_id, row\_id, etc.)
</ParamField>

<ParamField type="string">
  Fireworks API key to use for model calls
</ParamField>

### Example request

```json theme={null}
{
  "completion_params": {
    "model": "accounts/fireworks/models/qwen3-4b",
    "temperature": 0.7,
    "max_tokens": 2048
  },
  "messages": [
    { "role": "user", "content": "What is the weather in San Francisco?" }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the weather for a city",
        "parameters": {
          "type": "object",
          "properties": {
            "city": { "type": "string" }
          }
        }
      }
    }
  ],
  "model_base_url": "https://tracing.fireworks.ai/rollout_id/brave-night-42/invocation_id/wise-ocean-15/experiment_id/calm-forest-28/run_id/quick-river-07/row_id/bright-star-91",
  "metadata": {
    "invocation_id": "wise-ocean-15",
    "experiment_id": "calm-forest-28",
    "rollout_id": "brave-night-42",
    "run_id": "quick-river-07",
    "row_id": "bright-star-91"
  },
  "api_key": "fw_your_api_key"
}
```

## Metadata correlation

The `metadata` object contains correlation IDs that you must include when logging to Fireworks tracing. This allows Eval Protocol to match logs and traces back to specific evaluation rows.

Required metadata fields:

* `invocation_id` - Identifies the evaluation invocation
* `experiment_id` - Groups related experiments
* `rollout_id` - Unique ID for this specific rollout (most important)
* `run_id` - Identifies the evaluation run
* `row_id` - Links to the dataset row

<Note>
  `RemoteRolloutProcessor` automatically generates these IDs and sends them to your server. You don't need to create them yourself—just pass them through to your logging.
</Note>

## Fireworks tracing integration

Your remote server must use Fireworks tracing to report rollout status. Eval Protocol polls these logs to detect when rollouts complete.

### Basic setup

```python theme={null}
import logging
from eval_protocol import Status, InitRequest, FireworksTracingHttpHandler, RolloutIdFilter

# Configure Fireworks tracing handler globally
fireworks_handler = FireworksTracingHttpHandler()
logging.getLogger().addHandler(fireworks_handler)

@app.post("/init")
def init(request: InitRequest):
    # Create rollout-specific logger with filter
    rollout_logger = logging.getLogger(f"eval_server.{request.metadata.rollout_id}")
    rollout_logger.addFilter(RolloutIdFilter(request.metadata.rollout_id))

    try:
        # Execute your agent logic here
        result = execute_agent(request)

        # Log successful completion with structured status
        rollout_logger.info(
            f"Rollout {request.metadata.rollout_id} completed",
            extra={"status": Status.rollout_finished()}
        )

        return {"status": "success"}

    except Exception as e:
        # Log errors with structured status
        rollout_logger.error(
            f"Rollout {request.metadata.rollout_id} failed: {e}",
            extra={"status": Status.rollout_error(str(e))}
        )
        raise
```

### Key components

1. **FireworksTracingHttpHandler**: Sends logs to Fireworks tracing service
2. **RolloutIdFilter**: Tags logs with the rollout ID for correlation
3. **Status objects**: Structured status reporting that Eval Protocol can parse
   * `Status.rollout_finished()` - Signals successful completion
   * `Status.rollout_error(message)` - Signals failure with error details

### Alternative: Environment variable approach

For simpler setups, you can use the `EP_ROLLOUT_ID` environment variable instead of manual filters.

<Tabs>
  <Tab title="Single rollout per instance">
    If your server processes one rollout at a time (e.g., serverless functions, container per request):

    ```python theme={null}
    import os
    import logging
    from eval_protocol import Status, InitRequest, FireworksTracingHttpHandler

    # Set rollout ID in environment
    os.environ["EP_ROLLOUT_ID"] = request.metadata.rollout_id

    # Configure handler (automatically picks up EP_ROLLOUT_ID)
    fireworks_handler = FireworksTracingHttpHandler()
    logging.getLogger().addHandler(fireworks_handler)

    logger = logging.getLogger(__name__)

    @app.post("/init")
    def init(request: InitRequest):
        # Logs are automatically tagged with rollout_id
        logger.info("Processing rollout...")
        # ... execute agent logic ...
    ```
  </Tab>

  <Tab title="Separate processes">
    If your `/init` handler spawns separate Python processes for each rollout:

    ```python theme={null}
    import os
    import logging
    import multiprocessing
    from eval_protocol import FireworksTracingHttpHandler, InitRequest

    def execute_rollout_step_sync(request):
        # Set EP_ROLLOUT_ID in the child process
        os.environ["EP_ROLLOUT_ID"] = request.metadata.rollout_id
        logging.getLogger().addHandler(FireworksTracingHttpHandler())

        # Execute your rollout logic here
        # Logs are automatically tagged

    @app.post("/init")
    async def init(request: InitRequest):
        # Do NOT set EP_ROLLOUT_ID in parent process
        p = multiprocessing.Process(
            target=execute_rollout_step_sync,
            args=(request,)
        )
        p.start()
        return {"status": "started"}
    ```
  </Tab>
</Tabs>

### How Eval Protocol uses tracing

1. **Your server logs completion**: Uses `Status.rollout_finished()` or `Status.rollout_error()`
2. **Eval Protocol polls**: Searches Fireworks logs by `rollout_id` tag until completion signal found
3. **Status extraction**: Reads structured status fields (`code`, `message`, `details`) to determine outcome
4. **Trace retrieval**: Fetches full trace of model calls and tool use for evaluation

## Complete example

Here's a minimal but complete remote server implementation:

```python theme={null}
from fastapi import FastAPI
from fastapi.responses import JSONResponse
from eval_protocol import InitRequest, FireworksTracingHttpHandler, RolloutIdFilter, Status
import logging

app = FastAPI()

# Setup Fireworks tracing
fireworks_handler = FireworksTracingHttpHandler()
logging.getLogger().addHandler(fireworks_handler)

@app.post("/init")
async def init(request: InitRequest):
    # Create rollout-specific logger
    rollout_logger = logging.getLogger(f"eval_server.{request.metadata.rollout_id}")
    rollout_logger.addFilter(RolloutIdFilter(request.metadata.rollout_id))

    rollout_logger.info(f"Starting rollout {request.metadata.rollout_id}")

    try:
        # Your agent logic here
        # 1. Make model calls using request.model_base_url
        # 2. Call tools, interact with environment
        # 3. Collect results

        result = run_your_agent(
            messages=request.messages,
            tools=request.tools,
            model_config=request.completion_params,
            api_key=request.api_key
        )

        # Signal completion
        rollout_logger.info(
            f"Rollout {request.metadata.rollout_id} completed successfully",
            extra={"status": Status.rollout_finished()}
        )

        return {"status": "success", "result": result}

    except Exception as e:
        # Signal error
        rollout_logger.error(
            f"Rollout {request.metadata.rollout_id} failed: {str(e)}",
            extra={"status": Status.rollout_error(str(e))}
        )
        return JSONResponse(
            status_code=500,
            content={"status": "error", "message": str(e)}
        )

def run_your_agent(messages, tools, model_config, api_key):
    # Implement your agent logic here
    # Make model calls, use tools, etc.
    pass
```

## Testing locally

Before deploying, test your remote server locally:

<Steps>
  <Step title="Start your server">
    ```bash theme={null}
    uvicorn main:app --reload --port 8080
    ```
  </Step>

  <Step title="Configure RemoteRolloutProcessor">
    In your evaluator test, point to your local server:

    ```python theme={null}
    from eval_protocol.pytest import RemoteRolloutProcessor

    rollout_processor = RemoteRolloutProcessor(
        remote_base_url="http://localhost:8080"
    )
    ```
  </Step>

  <Step title="Run test evaluation">
    ```bash theme={null}
    pytest my-evaluator-name.py -vs
    ```

    This sends test rollouts to your local server and verifies the integration works.
  </Step>
</Steps>

## Deploying your service

Once tested locally, deploy to production:

<AccordionGroup>
  <Accordion title="Deployment checklist">
    * ✅ Service is publicly accessible (or accessible via VPN/private network)
    * ✅ HTTPS endpoint with valid SSL certificate (recommended)
    * ✅ Authentication/authorization configured
    * ✅ Monitoring and logging set up
    * ✅ Auto-scaling configured for concurrent rollouts
    * ✅ Error handling and retry logic implemented
    * ✅ Service availability SLA meets training requirements
  </Accordion>

  <Accordion title="Platform-specific guides">
    **Vercel/Serverless**:

    * One rollout per function invocation
    * Use environment variable approach
    * Configure timeout for long-running evaluations

    **AWS ECS/Kubernetes**:

    * Handle concurrent requests with proper worker configuration
    * Use RolloutIdFilter approach
    * Set up load balancing

    **On-premise**:

    * Ensure network connectivity from Fireworks
    * Configure firewall rules
    * Set up VPN if needed for security
  </Accordion>
</AccordionGroup>

## Connecting to RFT

Once your remote server is deployed, create an RFT job that uses it:

```bash theme={null}
eval-protocol create rft \
  --base-model accounts/fireworks/models/qwen3-4b \
  --remote-server-url https://your-evaluator.example.com \
  --dataset my-dataset
```

The RFT job will send all rollouts to your remote server for evaluation during training.

## Troubleshooting

<AccordionGroup>
  <Accordion title="Rollouts timing out">
    **Symptoms**: Rollouts show as timed out or never complete

    **Solutions**:

    * Check that your service is logging `Status.rollout_finished()` correctly
    * Verify Fireworks tracing handler is configured
    * Ensure rollout\_id is included in log tags
    * Check for exceptions being swallowed without logging
  </Accordion>

  <Accordion title="Correlation errors">
    **Symptoms**: Eval Protocol can't match logs to rollouts

    **Solutions**:

    * Verify you're using the exact `rollout_id` from request metadata
    * Check that RolloutIdFilter or EP\_ROLLOUT\_ID is set correctly
    * Ensure logs are being sent to Fireworks (check tracing dashboard)
  </Accordion>

  <Accordion title="Performance issues">
    **Symptoms**: Training is slow, high rollout latency

    **Solutions**:

    * Scale your service to handle concurrent requests
    * Optimize your agent logic (caching, async operations)
    * Add more workers or instances
    * Profile your code to find bottlenecks
  </Accordion>

  <Accordion title="Authentication failures">
    **Symptoms**: Model calls fail, API errors

    **Solutions**:

    * Verify API key is passed correctly from request
    * Check that your service has network access to Fireworks
    * Ensure model\_base\_url is used for traced calls
  </Accordion>
</AccordionGroup>

## Example implementations

Learn by example:

<CardGroup>
  <Card title="SVG Agent (TypeScript)" icon="image" href="/fine-tuning/quickstart-svg-agent">
    Complete walkthrough using a Vercel TypeScript server for SVG generation
  </Card>

  <Card title="Hello World Example" icon="code" href="https://github.com/eval-protocol/remote-rollout-processor-hello-world">
    Minimal Python implementation showing the basics
  </Card>
</CardGroup>

## Next steps

<CardGroup>
  <Card title="Launch training" icon="rocket" href="/fine-tuning/reinforcement-fine-tuning-models#cli">
    Launch your RFT job using the CLI
  </Card>

  <Card title="Monitor training" icon="chart-line" href="/fine-tuning/reinforcement-fine-tuning-models#monitor-training">
    Track rollout progress and debug issues
  </Card>

  <Card title="Eval Protocol docs" icon="book" href="https://evalprotocol.io/tutorial/remote-rollout-processor">
    Full Remote Rollout Processor tutorial
  </Card>

  <Card title="Evaluator best practices" icon="lightbulb" href="/fine-tuning/reinforcement-fine-tuning-models#evaluators">
    Design effective reward functions
  </Card>
</CardGroup>

<h2>
  Agent tracing
</h2>

Remote RFT needs **correlated traces** so the trainer can join rollouts, tool calls, and rewards. Your environment receives `model_base_url` (OpenAI-compatible, on `tracing.fireworks.ai`) and correlation IDs in `/init`.

**Minimum wiring:**

1. Use `model_base_url` from `/init` for all model calls (do not override).
2. Add `FireworksTracingHttpHandler` and log `Status.rollout_finished()` or `Status.rollout_error()` when the rollout ends.
3. Tag logs with `rollout_id` via `RolloutIdFilter` or `EP_ROLLOUT_ID`.

<Tip>
  **Coding agents:** correlation fields, minimal server example, capture checklist, and RemoteRolloutProcessor polling → [RFT agent tracing](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rft-agent-tracing.md) in the Fireworks training skill.
</Tip>

### Next steps

<CardGroup>
  <Card title="Connect remote environment" icon="server" href="/fine-tuning/connect-environments">
    Implement `/init`, tracing, and structured status for remote agents
  </Card>

  <Card title="Small model quickstart" icon="bolt" href="/fine-tuning/quickstart-math">
    Build and deploy a local evaluator in under 10 minutes
  </Card>

  <Card title="Launch training" icon="rocket" href="/fine-tuning/reinforcement-fine-tuning-models#cli">
    Launch your RFT job
  </Card>

  <Card title="Evaluator best practices" icon="code" href="/fine-tuning/reinforcement-fine-tuning-models#evaluators">
    Design effective reward functions for your task
  </Card>
</CardGroup>


# Deploying Fine Tuned Models
Source: https://docs.fireworks.ai/fine-tuning/deploying-loras

Deploy one or multiple LoRA models fine tuned on Fireworks using live merge or multi-LoRA

After fine-tuning your model on Fireworks, deploy it to make it available for inference. Fireworks supports two deployment methods for LoRA fine-tuned models: **live merge** and **multi-LoRA**. Each method has different tradeoffs around performance, cost, and flexibility.

To run training evals before production serving, see [Evaluating Fine Tuned Models](/fine-tuning/evaluating-fine-tuned-models). This page covers production serving.

<Warning>
  Fine-tuned LoRA models, whether created on the Fireworks platform or imported, can **only** be deployed to **on-demand (dedicated) deployments**. Serverless deployment is not supported for LoRA models.
</Warning>

<Note>
  You can also upload and deploy LoRA models fine-tuned outside of Fireworks. See [importing fine-tuned models](/models/uploading-custom-models#importing-fine-tuned-models) for details.
</Note>

## Choosing a deployment method

Fireworks offers two ways to deploy LoRA fine-tuned models. The right choice depends on how many fine-tuned variants you need to serve and your performance requirements.

|                           | **Live merge**                                                                                 | **Multi-LoRA**                                                                                  |
| ------------------------- | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| **How it works**          | LoRA weights are merged into the base model at deployment time, creating a single merged model | Base model is deployed with addon support; LoRA adapters are loaded dynamically at request time |
| **Number of LoRAs**       | One per deployment                                                                             | Multiple per deployment                                                                         |
| **Inference performance** | Matches the base model (no overhead)                                                           | Some overhead per request due to dynamic adapter application                                    |
| **Throughput**            | Same as base model                                                                             | Lower maximum throughput under high concurrency                                                 |
| **Cost efficiency**       | One deployment per fine-tune                                                                   | Share a single deployment across many fine-tunes                                                |
| **Best for**              | Production workloads requiring maximum performance                                             | Experimentation, A/B testing, or serving many variants of the same base model                   |

<Tip>
  If you only need to serve a single fine-tuned model, **live merge is the recommended approach**. It delivers the best performance with the simplest setup.
</Tip>

## Live merge deployment

Live merge is the simplest way to deploy a fine-tuned model. Fireworks automatically merges the LoRA weights into the base model at deployment time, producing a model that performs identically to a natively fine-tuned model with no inference overhead.

### How it works

When you deploy a LoRA model directly, Fireworks:

1. Takes your LoRA adapter weights and the base model
2. Merges them into a single set of weights at deployment time
3. Serves the merged model as a standalone deployment

The result is a deployment that is indistinguishable from a fully fine-tuned model in terms of latency, throughput, and memory usage.

### Deploy with live merge

Deploy your LoRA fine-tuned model with a single command:

```bash theme={null}
firectl deployment create "accounts/<ACCOUNT_ID>/models/<FINE_TUNED_MODEL_ID>"
```

<Check>
  Your deployment will be ready to use once it completes, with performance that matches the base model.
</Check>

### Sending requests

Send inference requests to your live-merge deployment by referencing the deployment directly:

<Tabs>
  <Tab title="Python (Fireworks SDK)">
    ```python theme={null}
    from fireworks import Fireworks

    client = Fireworks()

    response = client.chat.completions.create(
      model="accounts/<ACCOUNT_ID>/models/<FINE_TUNED_MODEL_ID>",
      messages=[{"role": "user", "content": "Hello!"}]
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    curl https://api.fireworks.ai/inference/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $FIREWORKS_API_KEY" \
      -d '{
        "model": "accounts/<ACCOUNT_ID>/models/<FINE_TUNED_MODEL_ID>",
        "messages": [
          {
            "role": "user",
            "content": "Hello!"
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

### When to use live merge

* You need maximum inference performance (latency and throughput matching the base model)
* You are serving a single fine-tuned model in production
* You want the simplest possible deployment workflow

## Multi-LoRA deployment

Multi-LoRA lets you load multiple LoRA adapters onto a single base model deployment. This is useful when you have several fine-tuned variants of the same base model and want to share GPU resources across them rather than creating a separate deployment for each.

### How it works

With multi-LoRA:

1. You deploy the base model with addon support enabled
2. You load one or more LoRA adapters onto the running deployment
3. At inference time, the correct adapter is selected and applied dynamically based on the model specified in the request

Because adapters are applied dynamically rather than merged, there is some performance overhead compared to live merge. This overhead increases with higher request concurrency.

### LoRA addon shape compatibility

Not all deployment shapes support LoRA addons. **FP8 and FP4 quantized shapes do not support `--enable-addons`.**

| Precision | `--enable-addons` supported? |
| --------- | ---------------------------- |
| BF16      | ✅ Yes                        |
| FP8       | ❌ No                         |
| FP4       | ❌ No                         |

Many base models default to FP8 or FP4 shapes. If you need LoRA addon inference on one of these models, you have two options:

**Option 1 — Use a BF16 deployment shape**

```bash theme={null}
# List available shapes for your model
firectl deployment-shape-version list --base-model accounts/fireworks/models/<MODEL_ID>

# Create deployment with a BF16 shape and addons enabled
firectl deployment create "accounts/fireworks/models/<BASE_MODEL_ID>" \
  --deployment-shape <bf16-shape-name> \
  --enable-addons
```

**Option 2 — Merge the adapter into a standalone model**

If no BF16 addon-compatible shape is available, use [live merge](#live-merge-deployment) (recommended for a single adapter) or merge the LoRA into a standalone Fireworks model, then deploy that merged model without `--enable-addons`. See [Uploading custom models](/models/uploading-custom-models#importing-fine-tuned-models) and [`firectl model create`](/tools-sdks/firectl/commands/model-create).

<Note>
  `"addons cannot be enabled with quantized precisions (FP8/FP4)"` — your model's default shape is quantized; use Option 1 or 2 above.

  `"the deployment shape version does not exist or you do not have access to it"` — the shape you requested is not available on your account; contact support.
</Note>

### Deploy with multi-LoRA

<Steps>
  <Step title="Create base model deployment with addon support">
    Deploy the base model with addons enabled:

    ```bash theme={null}
    firectl deployment create "accounts/fireworks/models/<BASE_MODEL_ID>" --enable-addons
    ```
  </Step>

  <Step title="Load LoRA adapters">
    Once the deployment is ready, load your LoRA models onto the deployment:

    ```bash theme={null}
    firectl load-lora <FINE_TUNED_MODEL_ID> --deployment <DEPLOYMENT_ID>
    ```

    Repeat this command for each LoRA adapter you want to load.
  </Step>
</Steps>

### Sending requests

To route inference requests to a specific LoRA adapter on a multi-LoRA deployment, set the `model` field to `<model_name>#<deployment_name>`. The `#` separator tells Fireworks to route the request to the specified adapter on the given deployment.

<Warning>
  **Deprecation notice:** The `deployedModel` request key for routing to LoRA addons is deprecated and will not be supported for any new deployments. Use the `model` field with the `<model_name>#<deployment_name>` format shown below.
</Warning>

<Tabs>
  <Tab title="Python (Fireworks SDK)">
    ```python theme={null}
    from fireworks import Fireworks

    client = Fireworks()

    response = client.chat.completions.create(
      model="accounts/<ACCOUNT_ID>/models/<FINE_TUNED_MODEL_ID>#accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>",
      messages=[{"role": "user", "content": "Hello!"}]
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="Python (OpenAI SDK)">
    ```python theme={null}
    import os
    from openai import OpenAI

    client = OpenAI(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference/v1"
    )

    response = client.chat.completions.create(
        model="accounts/<ACCOUNT_ID>/models/<FINE_TUNED_MODEL_ID>#accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>",
        messages=[{"role": "user", "content": "Hello!"}]
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    import OpenAI from "openai";

    const client = new OpenAI({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference/v1",
    });

    const response = await client.chat.completions.create({
      model: "accounts/<ACCOUNT_ID>/models/<FINE_TUNED_MODEL_ID>#accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>",
      messages: [
        {
          role: "user",
          content: "Hello!",
        },
      ],
    });

    console.log(response.choices[0].message.content);
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    curl https://api.fireworks.ai/inference/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $FIREWORKS_API_KEY" \
      -d '{
        "model": "accounts/<ACCOUNT_ID>/models/<FINE_TUNED_MODEL_ID>#accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>",
        "messages": [
          {
            "role": "user",
            "content": "Hello!"
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

### When to use multi-LoRA

* You need to serve multiple fine-tuned models based on the same base model
* You want to maximize GPU utilization by sharing a single deployment
* You are running experiments or A/B tests across multiple fine-tuned variants
* You can accept some performance overhead compared to live merge

## Downloading model weights

You can download your fine-tuned weights from Fireworks to inspect them, extend the context locally, or serve them outside the platform. There are two things you might want: the **LoRA adapter** on its own, or the **merged (base + adapter) model**.

### Download the LoRA adapter

LoRA adapters are listed alongside models in `firectl model list` (denoted with the type `HF_PEFT_ADDON`). Download one with the same command used for any model:

```bash theme={null}
firectl model download <FINE_TUNED_MODEL_ID> /path/to/checkpoint/
```

See [`firectl model download`](/tools-sdks/firectl/commands/model-download) for flags.

<Note>
  The adapter alone is not enough to run inference. You also need the matching base model. The adapter was trained against a specific base (for example, a vendor checkpoint that may differ from the public Hugging Face weights), so pair the adapter with the exact base it was trained on. If you are unsure which base was used, ask your Fireworks contact before assuming the public Hugging Face weights are identical.
</Note>

### Download the merged (base + adapter) model

On the platform, **the merge happens on the fly at deployment time** (live merge), so serving a fine-tuned model does not require a standalone merged file. To produce a merged copy you can run off-platform, download the base and the adapter, then merge them locally in BF16 with PEFT:

1. Download the base model with `firectl model download`.
2. Download the LoRA adapter with `firectl model download`.
3. Load the base model, wrap it with `PeftModel` to load the adapter, call `merge_and_unload()`, and save the merged model.

```python theme={null}
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("/path/to/base", torch_dtype="bfloat16")
merged = PeftModel.from_pretrained(base, "/path/to/adapter").merge_and_unload()

merged.save_pretrained("/path/to/merged")
AutoTokenizer.from_pretrained("/path/to/base").save_pretrained("/path/to/merged")
```

Merge in BF16; if you need quantized (FP8) weights, quantize the merged result afterward (see FP8 below).

### FP8 (and other quantized) merged weights

If you want an FP8 merged model to run off-platform, merge in BF16 first, then quantize the merged result yourself. For reference, the on-platform serving path is:

1. Keep the BF16 base + BF16 LoRA adapter.
2. At deploy, merge in BF16: `W' = W_bf16 + (B·A)_bf16`.
3. Quantize the merged BF16 weights to FP8 on the fly at serving time.

To reproduce this locally, merge in BF16 first, then quantize the merged weights to FP8.

<Warning>
  **Match the original quantization scheme when serving locally.** Use the same quantization the base model ships with on Hugging Face rather than a generic FP8 cast. For example, a GLM-family MoE base uses **blockwise FP8** for its MoE weights, and casting with a different scheme can silently degrade quality. When in doubt, keep the merged model in BF16 and let your serving stack quantize.
</Warning>

<h2>
  LoRA performance
</h2>

### Why multi-LoRA can feel slower than the base model

Three factors explain most latency gaps:

1. **Unmerged adapters** — dynamic LoRA application adds compute on each request. [Live merge](#live-merge-deployment) removes this entirely.
2. **Speculative decoding** — base deployments often use SD; fine-tuned LoRA deployments may not unless you configure a custom draft model.
3. **Concurrency** — multi-LoRA overhead shows up mainly under sustained load (see below).

### Live merge vs multi-LoRA (performance)

|                         | **Live merge**                  | **Multi-LoRA**                                      |
| ----------------------- | ------------------------------- | --------------------------------------------------- |
| Latency / throughput    | Matches base model              | TTFT often +10–30%; lower max throughput under load |
| Adapter count at deploy | One fine-tune per deployment    | Many adapters on one base deployment                |
| Best for                | Production single-model serving | A/B tests, many variants, shared GPU                |

The number of adapters registered on a deployment has **little effect** on per-request performance; concurrency and merge mode matter more.

### Speculative decoding

Training and deploying a **custom draft model** for your fine-tuned setup can match or beat base-model latency. Speculative decoding for fine-tuned models is an enterprise feature — contact Fireworks for access.

## Troubleshooting

### Silent deployment-shape drop (multi-LoRA lands on the default serving image)

This is a subtle failure mode specific to multi-LoRA deployments. If the deployment shape you request is not **validated for the exact base model version** you are deploying, deployment create does **not** return an error. The unvalidated shape is **silently dropped**, and the deployment quietly falls back to the **default serving image**. That default image's addon loader then rejects addon (multi-LoRA) checkpoints, so you end up seeing base-model behavior or an addon-load failure with no obvious cause.

<Warning>
  This differs from training, where an unvalidated shape returns a **400**. At deployment create time there is no such error. The `skip_shape_validation` override is superuser-only, so you cannot force an unvalidated shape through yourself. The shape must be validated for your exact model version.
</Warning>

**Why it happens.** A deployment shape is validated against a specific base model version, not just a model family. A shape such as `deploymentShapes/<model>-h200-multilora` may have validated versions that bind one model version but **not** another version of the same family. Deploying a model version that no validated shape version binds triggers the silent drop.

**How to detect it.** Before (or after) creating the deployment, confirm a validated shape version exists for the **exact** model version you are deploying, not just the family. List the validated shape versions for your model:

```bash theme={null}
firectl deployment-shape-version list --base-model accounts/<your-account>/models/<your-model-version>
```

Or query the API directly with the `latest_validated=true` filter (see [List Deployment Shape Versions](/api-reference/list-deployment-shape-versions)):

```bash theme={null}
curl -s "https://api.fireworks.ai/v1/accounts/-/deploymentShapes/-/versions?filter=snapshot.base_model%3D%22accounts%2F<your-account>%2Fmodels%2F<your-model-version>%22%20AND%20latest_validated%3Dtrue&order_by=create_time%20desc" \
  -H "Authorization: Bearer $FIREWORKS_API_KEY" | jq .
```

Signs you have hit this failure mode:

* No validated shape version lists your exact model version under `snapshot.base_model` (every validated version binds a **different** version of the same model family).
* The deployment comes up serving base-model behavior instead of your fine-tune.
* Loading an addon (a Tinker or other LoRA checkpoint) is rejected even though the shape you requested supports addons.

**How to avoid landing on the default serving image.**

* Deploy only against a shape version that is validated for your **exact** model version, confirmed with the check above.
* If no validated shape version binds your model version, do **not** rely on the shape argument being honored. Ask your Fireworks account team to **validate a deployment shape version for that model version** first. A shape validated only for a sibling version will be dropped.
* As an alternative that avoids multi-LoRA and the addon loader entirely, [live merge](#live-merge-deployment) the single adapter, which does not go through the addon path.

## Next steps

<CardGroup>
  <Card title="On-Demand Deployments" href="/guides/ondemand-deployments" icon="rocket">
    Deployment configuration, scaling, and hardware
  </Card>

  <Card title="Import Fine-Tuned Models" href="/models/uploading-custom-models#importing-fine-tuned-models" icon="upload">
    Upload LoRA models fine-tuned outside Fireworks
  </Card>
</CardGroup>


# Preference Optimization with DPO or ORPO
Source: https://docs.fireworks.ai/fine-tuning/dpo-fine-tuning

Train on preferred and non-preferred response pairs using managed DPO or ORPO.

Preference optimization fine-tunes models on pairs of preferred and non-preferred responses to the same prompt. Managed jobs support two objectives:

* **DPO** compares the policy against a reference model.
* **ORPO** combines supervised and preference objectives without a separate reference model.

Use either method for:

* Aligning model outputs with brand voice, tone, or style guidelines
* Reducing hallucinations or incorrect reasoning patterns
* Improving response quality where there's no single "correct" answer
* Teaching models to follow specific formatting or structural preferences

## Fine-tuning with DPO or ORPO

<Steps>
  <Step title="Prepare dataset">
    Datasets must adhere strictly to the JSONL format, where each line represents a complete JSON-formatted training example.

    **Minimum Requirements:**

    * **Minimum examples needed:** 3
    * **Maximum examples:** Up to 3 million examples per dataset
    * **File format:** JSONL (each line is a valid JSON object)
    * **Dataset Schema:** Each training sample must include the following fields:
      * An `input` field containing a `messages` array, where each message is an object with two fields:
        * `role`: one of `system`, `user`, or `assistant`
        * `content`: a string representing the message content
      * A `preferred_output` field containing an assistant message with an ideal response
      * A `non_preferred_output` field containing an assistant message with a suboptimal response

    Here’s an example conversation dataset (one training example):

    ```json einstein_dpo.jsonl theme={null}
    {
      "input": {
        "messages": [
          {
            "role": "user",
            "content": "What is Einstein famous for?"
          }
        ],
        "tools": []
      },
      "preferred_output": [
        {
          "role": "assistant",
          "content": "Einstein is renowned for his theory of relativity, especially the equation E=mc²."
        }
      ],
      "non_preferred_output": [
        {
          "role": "assistant",
          "content": "He was a famous scientist."
        }
      ]
    }
    ```

    <Warning>
      We currently only support one-turn conversations for each example, where the preferred and non-preferred messages need to be the last assistant message.
    </Warning>

    Save this dataset as jsonl file locally, for example `einstein_dpo.jsonl`.
  </Step>

  <Step title="Create and upload the dataset">
    There are a couple ways to upload the dataset to Fireworks platform for fine tuning: `firectl`, `Restful API` , `builder SDK` or `UI`.

    <Tabs>
      <Tab title="UI">
        * You can simply navigate to the dataset tab, click `Create Dataset` and follow the wizard.

          <img alt="Dataset Pn" />
      </Tab>

      <Tab title="firectl">
        * Upload dataset using `firectl`

        ```bash theme={null}
        firectl dataset create <dataset-id> /path/to/file.jsonl
        ```
      </Tab>

      <Tab title="Restful API">
        You need to make two separate HTTP requests. One for creating the dataset entry and one for uploading the dataset. Full reference here: [Create dataset](/api-reference/create-dataset). Note that the `exampleCount` parameter needs to be provided by the client.

        ```jsx theme={null}
        // Create Dataset Entry
        const createDatasetPayload = {
          datasetId: "trader-poe-sample-data",
          dataset: { userUploaded: {} }
          // Additional params such as exampleCount
        };
        const urlCreateDataset = `${BASE_URL}/datasets`;
        const response = await fetch(urlCreateDataset, {
          method: "POST",
          headers: HEADERS_WITH_CONTENT_TYPE,
          body: JSON.stringify(createDatasetPayload)
        });
        ```

        ```jsx theme={null}
        // Upload JSONL file
        const urlUpload = `${BASE_URL}/datasets/${DATASET_ID}:upload`;
        const files = new FormData();
        files.append("file", localFileInput.files[0]);

        const uploadResponse = await fetch(urlUpload, {
          method: "POST",
          headers: HEADERS,
          body: files
        });
        ```
      </Tab>
    </Tabs>

    While all of the above approaches should work, `UI` is more suitable for smaller datasets `< 500MB` while `firectl` might work better for bigger datasets.

    Ensure the dataset ID conforms to the [resource id restrictions](/getting-started/concepts#resource-names-and-ids).
  </Step>

  <Step title="Create a DPO or ORPO Job">
    <Tabs>
      <Tab title="firectl">
        ```bash theme={null}
        firectl dpo-job create \
          --loss-method DPO \
          --base-model accounts/account-id/models/base-model-id \
          --dataset accounts/my-account-id/datasets/my-dataset-id \
          --output-model new-model-id
        ```

        For reservation capacity: add `--use-reservation` with firectl (default off). REST and the Python SDK (`>=1.2.8`) default to reservation-first placement; set `useReservation: false` or `use_reservation=False` to opt out. For full-parameter DPO, policy and dedicated reference trainers try independently.

        For ORPO, use the same preference dataset and select the ORPO objective:

        ```bash theme={null}
        firectl dpo-job create \
          --loss-method ORPO \
          --orpo-lambda <approved-value> \
          --base-model accounts/account-id/models/base-model-id \
          --dataset accounts/my-account-id/datasets/my-dataset-id \
          --output-model new-model-id
        ```

        Choose a base model that [Models](/fine-tuning/models) marks as DPO-enabled, and a shape published for it.
      </Tab>
    </Tabs>
  </Step>

  <Step title="Monitor the preference job">
    <Tabs>
      <Tab title="firectl">
        ```bash theme={null}
        firectl dpo-job get dpo-job-id
        ```
      </Tab>
    </Tabs>

    Once the job is complete, the `STATE` will be set to `JOB_STATE_COMPLETED`, and the fine-tuned model can be deployed.
  </Step>

  <Step title="Deploy the DPO fine-tuned model">
    Once training completes, you can create a deployment to interact with the fine-tuned model. Refer to [deploying a fine-tuned model](/fine-tuning/fine-tuning-models#deploying-a-fine-tuned-model) for more details.
  </Step>
</Steps>

## Next Steps

Explore other fine-tuning methods to improve model output for different use cases.

<CardGroup>
  <Card title="Supervised Fine Tuning - Text" icon="message" href="/fine-tuning/fine-tuning-models">
    Train models on input-output examples to improve task-specific performance.
  </Card>

  <Card title="Reinforcement Fine Tuning" icon="brain" href="/fine-tuning/reinforcement-fine-tuning-models">
    Optimize models using AI feedback for complex reasoning and decision-making.
  </Card>

  <Card title="Supervised Fine Tuning - Vision" icon="eye" href="/fine-tuning/fine-tuning-models#vision-fine-tuning">
    Fine-tune vision-language models to understand both images and text.
  </Card>
</CardGroup>


# Evaluating Fine Tuned Models
Source: https://docs.fireworks.ai/fine-tuning/evaluating-fine-tuned-models

Run training evals on a fine-tuned adapter before you create a production deployment.

After training, run evals on the adapter before you create a production deployment. The preemptible deployment on this page is for **training evals only**. It is not a production serving setup, and Fireworks does not offer serverless LoRA inference for a customer fine-tune.

<Warning>
  Do not send production traffic through a preemptible deployment. For production serving after eval, create an on-demand deployment. See [Deploying Fine Tuned Models](/fine-tuning/deploying-loras).
</Warning>

## Preemptible deployment

If you want to eval a fine-tuned model without holding dedicated on-demand capacity, create a **preemptible deployment**. It borrows idle reserved GPU capacity instead of reserving GPUs exclusively for you.

It can be reclaimed (preempted) at any time. Fireworks does not guarantee how many GPUs are available or how long the deployment stays up, but in practice it typically lasts long enough to finish a training eval.

### How it works

Passing `--preemptible` to `firectl deployment create` opts the deployment into capacity borrowing:

* The deployment runs on reserved nodes that are currently idle.
* When the capacity owner needs those GPUs back, your deployment can be preempted.

Because it borrows idle capacity, you do not need to hold dedicated on-demand capacity for the duration of the eval.

### Requirements

`--preemptible` takes effect only on **firectl 1.7.26 or newer**. Check with `firectl version`. The flag is not present in older builds. Upgrade if you are below 1.7.26.

### Behavior

* **Training evals only.** A preemptible deployment can be preempted mid-request. Treat disappearance as a normal outcome, not an error.
* **`--preemptible` is immutable.** It is set at create time and cannot be toggled on or off afterward. To change it, delete the deployment and create a new one.
* **Clean up when done.** Delete the deployment after your eval so you stop holding the borrowed capacity.

### Create the deployment

```bash theme={null}
firectl deployment create accounts/<ACCOUNT_ID>/models/<FINE_TUNED_MODEL_ID> \
  --deployment-id <YOUR_DEPLOYMENT_ID> \
  --display-name <YOUR_DISPLAY_NAME> \
  --deployment-shape <DEPLOYMENT_SHAPE> \
  --min-replica-count 1 \
  --max-replica-count 1 \
  --preemptible
```

Replace the placeholders:

* `<FINE_TUNED_MODEL_ID>`: the fine-tuned model to eval, not a base model.
* `<YOUR_DEPLOYMENT_ID>` / `<YOUR_DISPLAY_NAME>`: a name of your choice for the eval deployment.
* `<DEPLOYMENT_SHAPE>`: the deployment shape to use for that model.

### Worked example

```bash theme={null}
firectl deployment create accounts/<ACCOUNT_ID>/models/<FINE_TUNED_MODEL_ID> \
  -a <ACCOUNT_ID> \
  --deployment-id <FINE_TUNED_MODEL_ID>-eval \
  --display-name <FINE_TUNED_MODEL_ID>-eval \
  --deployment-shape <DEPLOYMENT_SHAPE> \
  --min-replica-count 1 \
  --max-replica-count 1 \
  --preemptible
```

### Run the eval and tear down

Check that the deployment is ready, then send eval requests to the fine-tuned model:

```bash theme={null}
firectl deployment get <YOUR_DEPLOYMENT_ID> -a <ACCOUNT_ID>
```

```bash theme={null}
curl https://api.fireworks.ai/inference/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FIREWORKS_API_KEY" \
  -d '{
    "model": "accounts/<ACCOUNT_ID>/models/<FINE_TUNED_MODEL_ID>",
    "messages": [
      {
        "role": "user",
        "content": "Hello!"
      }
    ]
  }'
```

When the evaluation is finished, delete the deployment to release the borrowed capacity:

```bash theme={null}
firectl deployment delete <YOUR_DEPLOYMENT_ID> -a <ACCOUNT_ID>
```

## Next steps

<CardGroup>
  <Card title="Deploying Fine Tuned Models" href="/fine-tuning/deploying-loras" icon="rocket">
    Live merge or multi-LoRA for production serving
  </Card>
</CardGroup>


# Supervised Fine Tuning - Text
Source: https://docs.fireworks.ai/fine-tuning/fine-tuning-models



This guide will focus on using supervised fine-tuning to fine-tune a model and deploy it to an on-demand (dedicated) deployment, which is the only supported method for serving fine-tuned models.

For the full list of base models supported by managed fine-tuning (SFT, DPO, and RFT) and their max context lengths, see [Models](/fine-tuning/models).

## Fine-tuning a model using SFT

<Steps>
  <Step title="Confirm model support for fine-tuning">
    You can confirm that a base model is available to fine-tune by looking for the `Tunable` tag in the model library or by using:

    ```bash theme={null}
    firectl model get -a fireworks <MODEL-ID>
    ```

    And looking for `Tunable: true`.

    <Note>
      Custom uploaded base models must include a corresponding Hugging Face URL before Fireworks can determine whether they are tunable. Fireworks uses the URL to infer the training renderer and find compatible training shapes. The tunability refresh runs asynchronously about every 30 minutes, so a newly uploaded or updated custom model may take up to 30 minutes to show `Tunable: true`.
    </Note>

    <Note>
      Some base models cannot be tuned on Fireworks (`Tunable: false`) but still list support for LoRA (`Supports Lora: true`). This means that users can tune a LoRA for this base model on a separate platform and upload it to Fireworks for inference. Consult [importing fine-tuned models](/models/uploading-custom-models#importing-fine-tuned-models) for more information.
    </Note>
  </Step>

  <Step title="Prepare a dataset">
    Fireworks uses the **OpenAI-compatible chat completion format** for SFT training data. If you already have datasets formatted for OpenAI fine-tuning, they work on Fireworks with no changes needed.

    Datasets must be in JSONL format, where each line represents a complete JSON-formatted training example. Make sure your data conforms to the following restrictions:

    * **Minimum examples:** 3
    * **Maximum examples:** 3 million per dataset
    * **File format:** `.jsonl`
    * **Message schema:** Each training sample must include a messages array, where each message is an object with two fields:
      * `role`: one of `system`, `user`, or `assistant`. A message with the `system` role is optional, but if specified, it must be the first message of the conversation
      * `content`: the message content. This can be either a plain string **or** a list of content parts in the OpenAI chat completions style, e.g. `[{"type": "text", "text": "..."}]`. Both forms are accepted, and you can mix them freely across messages and even within the same dataset
      * `weight`: optional key with value to be configured in either 0 or 1. message will be skipped if value is set to 0
    * **Sample weight:** Optional key `weight` at the root of the JSON object. It can be any floating point number (positive, negative, or 0) and is used as a loss multiplier for tokens in that sample. If used, this field must be present in all samples in the dataset.

    Here is an example conversation dataset:

    ```json theme={null}
    {
      "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the capital of France?"},
        {"role": "assistant", "content": "Paris."}
      ]
    }
    {
      "messages": [
        {"role": "user", "content": "What is 1+1?"},
        {"role": "assistant", "content": "2", "weight": 0},
        {"role": "user", "content": "Now what is 2+2?"},
        {"role": "assistant", "content": "4"}
      ]
    }
    ```

    #### OpenAI-style structured content

    In addition to plain strings, `content` may also be a list of content parts following the OpenAI chat completions format. For text fine-tuning, use `{"type": "text", "text": "..."}` parts. This is convenient if you already produce data in the OpenAI chat completions shape, or if you generate datasets with the OpenAI SDK. The string form and the list form are equivalent for text models, and you can mix them within the same file (and even within the same conversation):

    ```json theme={null}
    {"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": [{"type": "text", "text": "What is the capital of France?"}]}, {"role": "assistant", "content": [{"type": "text", "text": "Paris."}]}]}
    {"messages": [{"role": "user", "content": [{"type": "text", "text": "What is 1+1?"}]}, {"role": "assistant", "content": [{"type": "text", "text": "2"}], "weight": 0}, {"role": "user", "content": "Now what is 2+2?"}, {"role": "assistant", "content": "4"}]}
    {"messages": [{"role": "user", "content": [{"type": "text", "text": "Say hello "}, {"type": "text", "text": "in French."}]}, {"role": "assistant", "content": "Bonjour."}]}
    ```

    <Note>
      All keys you can use with the string form — including the per-message `weight` and `reasoning_content` — work the same way with the list form. When a single message contains multiple text parts (as in the third example above), the parts are concatenated when the chat template is applied. For text-only fine-tuning, only `{"type": "text", ...}` parts are used; image parts are reserved for [vision fine-tuning](/fine-tuning/fine-tuning-models#vision-fine-tuning).
    </Note>

    Here is an example conversation dataset with sample weights:

    ```json theme={null}
    {
      "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the capital of France?"},
        {"role": "assistant", "content": "Paris."}
      ],
      "weight": 0.5
    }
    {
      "messages": [
        {"role": "user", "content": "What is 1+1?"},
        {"role": "assistant", "content": "2", "weight": 0},
        {"role": "user", "content": "Now what is 2+2?"},
        {"role": "assistant", "content": "4"}
      ],
      "weight": 1.0
    }
    ```

    We also support function calling dataset with a list of tools. An example would look like:

    ```json theme={null}
    {
      "tools": [
        {
          "type": "function",
          "function": {
            "name": "get_car_specs",
            "description": "Fetches detailed specifications for a car based on the given trim ID.",
            "parameters": {
              "trimid": {
                "description": "The trim ID of the car for which to retrieve specifications.",
                "type": "int",
                "default": ""
              }
            }
          }
        },
    ],
      "messages": [
        {
          "role": "user",
          "content": "What is the specs of the car with trim 121?"
        },
        {
          "role": "assistant",
          "tool_calls": [
            {
              "type": "function",
              "function": {
                "name": "get_car_specs",
                "arguments": "{\"trimid\": 121}"
              }
            }
          ]
        }
      ]
    }
    ```

    #### Thinking traces

    For managed supervised fine-tuning (SFT), you can include thinking traces for assistant turns in `reasoning_content`. Thinking traces are optional, but ideally each assistant turn includes one. For example:

    ```json theme={null}
    {
      "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the capital of France?"},
        {"role": "assistant", "content": "Paris.", "reasoning_content": "The user is asking about the capital city of France, it should be Paris."}
      ]
    }
    {
      "messages": [
        {"role": "user", "content": "What is 1+1?"},
        {"role": "assistant", "content": "2", "weight": 0, "reasoning_content": "The user is asking about the result of 1+1, the answer is 2."},
        {"role": "user", "content": "Now what is 2+2?"},
        {"role": "assistant", "content": "4", "reasoning_content": "The user is asking about the result of 2+2, the answer should be 4."}
      ]
    }
    ```

    How earlier thinking appears in later training contexts depends on the base model and its thinking-history mode. This is separate from enabling or disabling thinking generation. Some models offer both Interleaved and Preserved history, some have one fixed mode, and DeepSeek V4 derives its behavior from whether each dataset row declares tools.

    <Card title="Thinking history in fine-tuning" icon="brain" href="/fine-tuning/thinking-history">
      Compare every supported model, understand per-user-turn unrolling, configure the job field, and preview what the trainer will see.
    </Card>
  </Step>

  <Step title="Create and upload a dataset">
    There are a couple ways to upload the dataset to Fireworks platform for fine tuning: `firectl`, `Restful API` , `builder SDK` or `UI`.

    <Tabs>
      <Tab title="UI">
        * You can simply navigate to the dataset tab, click `Create Dataset` and follow the wizard.

          <img alt="Dataset Pn" />
      </Tab>

      <Tab title="firectl">
        ```bash theme={null}
        firectl dataset create <DATASET_ID> /path/to/jsonl/file
        ```
      </Tab>

      <Tab title="Restful API">
        You need to make two separate HTTP requests. One for creating the dataset entry and one for uploading the dataset. Full reference here: [Create dataset](/api-reference/create-dataset). Note that the `exampleCount` parameter needs to be provided by the client.

        ```jsx theme={null}
        // Create Dataset Entry
        const createDatasetPayload = {
          datasetId: "trader-poe-sample-data",
          dataset: { userUploaded: {} }
          // Additional params such as exampleCount
        };
        const urlCreateDataset = `${BASE_URL}/datasets`;
        const response = await fetch(urlCreateDataset, {
          method: "POST",
          headers: HEADERS_WITH_CONTENT_TYPE,
          body: JSON.stringify(createDatasetPayload)
        });
        ```

        ```jsx theme={null}
        // Upload JSONL file
        const urlUpload = `${BASE_URL}/datasets/${DATASET_ID}:upload`;
        const files = new FormData();
        files.append("file", localFileInput.files[0]);

        const uploadResponse = await fetch(urlUpload, {
          method: "POST",
          headers: HEADERS,
          body: files
        });
        ```
      </Tab>
    </Tabs>

    While all of the above approaches should work, `UI` is more suitable for smaller datasets `< 500MB` while `firectl` might work better for bigger datasets.

    Ensure the dataset ID conforms to the [resource id restrictions](/getting-started/concepts#resource-names-and-ids).
  </Step>

  <Step title="Launch a fine-tuning job">
    There are also a couple ways to launch the fine-tuning jobs. We highly recommend creating supervised fine tuning jobs via `UI` .

    <Tabs>
      <Tab title="UI">
        Simply navigate to the `Fine-Tuning` tab, click `Fine-Tune a Model` and follow the wizard from there. You can even pick a LoRA model to start the fine-tuning for continued training.

        <img alt="Fine Tuning Pn" />

        <img alt="Create Sftj Pn" />
      </Tab>

      <Tab title="firectl">
        Ensure the fine tuned model ID conforms to the [resource id restrictions](/getting-started/concepts#resource-names-and-ids). This will return a fine-tuning job ID. For a full explanation of the settings available to control the fine-tuning process, including learning rate and epochs, consult [additional managed fine-tuning job settings](#additional-managed-fine-tuning-job-settings).

        ```bash theme={null}
        firectl sftj create --base-model <MODEL_ID> --dataset <DATASET_ID> --output-model <FINE_TUNED_MODEL_ID>
        ```

        <Tip>
          Similar to UI, instead of tuning a base model, you can also start tuning from a previous LoRA model using

          ```bash theme={null}
          firectl sftj create --warm-start-from <FINE_TUNED_MODEL_ID> --dataset <DATASET_ID> --output-model <FINE_TUNED_MODEL_ID>
          ```

          Notice that we use `--warm-start-from` instead of `--base-model` when creating this job.
        </Tip>
      </Tab>
    </Tabs>

    With `UI`, once the job is created, it will show in the list of jobs. Clicking to view the job details to monitor the job progress.

    <img alt="Sftj Details Pn" />

    <Tip>
      If the fine-tuned model appears to learn the wrong text or ignore the expected assistant response, use **Render Samples** on the job details page to inspect the rendered token IDs and loss masks. See [Debug SFT tokenization](/fine-tuning/fine-tuning-models#debug-sft-tokenization).
    </Tip>

    With `firectl`, you can monitor the progress of the tuning job by running

    ```bash theme={null}
    firectl sftj get <JOB_ID>
    ```

    Once the job successfully completes, you will see the new LoRA model in your model list

    ```bash theme={null}
    firectl model list
    ```
  </Step>
</Steps>

<Tip>
  For a complete Python SDK example that demonstrates the full workflow
  (creating datasets, uploading files, and launching a supervised fine-tuning
  job), see the [Python SDK workflow
  example](https://github.com/fw-ai-external/python-sdk/blob/main/examples/sftj_workflow.py).
</Tip>

## Deploying a fine-tuned model

After fine-tuning completes, [evaluate the model](/fine-tuning/evaluating-fine-tuned-models) before you hold dedicated capacity for production serving.

To deploy for inference:

```bash theme={null}
firectl deployment create <FINE_TUNED_MODEL_ID>
```

This creates a dedicated deployment with performance matching the base model.

<Tip>
  For more details on deploying fine-tuned models, including multi-LoRA
  deployments, see the [Deploying Fine Tuned Models
  guide](/fine-tuning/deploying-loras).
</Tip>

<a />

## Additional managed fine-tuning job settings

Additional tuning settings are available when starting an SFT or preference (DPO/ORPO) job. All of the settings below are optional and have reasonable defaults. For settings that affect tuning quality, such as `epochs` and `learning_rate`, use the defaults first and change them only when the results indicate a clear need. Examples use SFT unless otherwise noted.

<AccordionGroup>
  <Accordion title="Evaluation">
    By default, the fine-tuning job will run evaluation by running the fine-tuned model against an evaluation set that's created by automatically carving out a portion of your training set. You have the option to explicitly specify a separate evaluation dataset to use instead of carving out training data.

    `evaluation_dataset`: The ID of a separate dataset to use for evaluation. Must be pre-uploaded via firectl

    ```shell theme={null}
    firectl sftj create \
      --evaluation-dataset my-eval-set \
      --base-model MY_BASE_MODEL \
      --dataset cancerset \
      --output-model my-tuned-model
    ```
  </Accordion>

  <Accordion title="Max Context Length">
    Depending on the size of the model, the default context size will be different. For most models, the default context size is >= 32768. Training examples will be cut-off at 32768 tokens. Usually you do not need to set the max context length unless out of memory error is encountered with higher lora rank and large max context length.

    ```shell theme={null}
    firectl sftj create \
      --max-context-length 65536 \
      --base-model MY_BASE_MODEL \
      --dataset cancerset \
      --output-model my-tuned-model
    ```
  </Accordion>

  <Accordion title="Batch Size (Samples)">
    Managed SFT and preference tuning use sample-count batching. `batch_size_samples` is the number of SFT samples or preference pairs included in each optimizer step. It is independent of `max_context_length`, which limits the token length of each sample. The UI defaults are 32 samples for SFT and 4 preference pairs for DPO/ORPO.

    ```shell theme={null}
    firectl sftj create \
      --batch-size-samples 32 \
      --base-model MY_BASE_MODEL \
      --dataset cancerset \
      --output-model my-tuned-model
    ```
  </Accordion>

  <Accordion title="Epochs">
    Epochs are the number of passes over the training data. Our default value is 1. If the model does not follow the training data as much as expected, increase the number of epochs by 1 or 2. Non-integer values are supported.

    **Note: we set a max value of 3 million dataset examples × epochs**

    ```shell theme={null}
    firectl sftj create \
      --epochs 2.0 \
      --base-model MY_BASE_MODEL \
      --dataset cancerset \
      --output-model my-tuned-model
    ```
  </Accordion>

  <Accordion title="Learning rate">
    Learning rate controls how fast the model updates from data. We generally do not recommend changing learning rate. The default value is automatically based on your selected model.

    ```shell theme={null}
    firectl sftj create \
      --learning-rate 0.0001 \
      --base-model MY_BASE_MODEL \
      --dataset cancerset \
      --output-model my-tuned-model
    ```
  </Accordion>

  <Accordion title="Learning rate warmup steps">
    Learning rate warmup steps controls the number of training steps during which the learning rate will be linearly ramped up to the set learning rate.

    ```shell theme={null}
    firectl sftj create \
      --learning-rate 0.0001 \
      --learning-rate-warmup-steps 200 \
      --base-model MY_BASE_MODEL \
      --dataset cancerset \
      --output-model my-tuned-model
    ```
  </Accordion>

  <Accordion title="Learning rate scheduler">
    Configure how the learning rate changes over training. Supported schedulers are `constant`, `linear`, and `cosine`. When unset, the trainer uses its legacy constant schedule. The same flags apply to SFT and preference tuning jobs (DPO/ORPO).

    For `linear` and `cosine`, you can optionally set:

    * `--learning-rate-min-lr-ratio`: minimum learning rate as a fraction of `--learning-rate` (0 to 1)
    * `--learning-rate-decay-ratio`: fraction of total training steps over which to decay; `0` decays over the full run

    ```shell theme={null}
    firectl sftj create \
      --base-model MY_BASE_MODEL \
      --dataset cancerset \
      --output-model my-tuned-model \
      --learning-rate 0.0001 \
      --learning-rate-warmup-steps 10 \
      --learning-rate-scheduler cosine \
      --learning-rate-min-lr-ratio 0.1 \
      --learning-rate-decay-ratio 0.8
    ```

    Via the REST API, pass an `lrScheduler` object with one of `constant`, `linear`, or `cosine`:

    ```javascript theme={null}
    const payload = {
      supervisedFineTuningJob: {
        baseModel: "accounts/my-account/models/MY_BASE_MODEL",
        dataset: "accounts/my-account/datasets/cancerset",
        outputModel: "accounts/my-account/models/my-tuned-model",
        learningRate: 0.0001,
        learningRateWarmupSteps: 10,
        lrScheduler: {
          cosine: {
            minLrRatio: 0.1,
            decayRatio: 0.8
          }
        }
      }
    };
    ```
  </Accordion>

  <Accordion title="LoRA Rank">
    LoRA rank refers to the number of parameters that will be tuned in your LoRA add-on. Higher LoRA rank increases the amount of information that can be captured while tuning. LoRA rank must be a power of 2 up to 32. Our default value is 8.

    ```shell theme={null}
    firectl sftj create \
      --lora-rank 16 \
      --base-model MY_BASE_MODEL \
      --dataset cancerset \
      --output-model my-tuned-model
    ```
  </Accordion>

  <Accordion title="Training progress and monitoring">
    The fine-tuning service integrates with Weights & Biases to provide observability into the tuning process. To use this feature, you must have a Weights & Biases account and have provisioned an API key.

    ```shell theme={null}
    firectl sftj create \
      --wandb-entity my-org \
      --wandb-api-key xxx \
      --wandb-project "My Project" \
      --base-model MY_BASE_MODEL \
      --dataset cancerset \
      --output-model my-tuned-model
    ```
  </Accordion>

  <Accordion title="Model ID">
    By default, the fine-tuning job will generate a random unique ID for the model. This ID is used to refer to the model at inference time. You can optionally specify a custom ID, within [ID constraints](/getting-started/concepts#resource-names-and-ids).

    ```shell theme={null}
    firectl sftj create \
      --output-model my-model \
      --base-model MY_BASE_MODEL \
      --dataset cancerset
    ```
  </Accordion>

  <Accordion title="Job ID">
    By default, the fine-tuning job will generate a random unique ID for the fine-tuning job. You can optionally choose a custom ID.

    ```shell theme={null}
    firectl sftj create \
      --job-id my-fine-tuning-job \
      --base-model MY_BASE_MODEL \
      --dataset cancerset \
      --output-model my-tuned-model
    ```
  </Accordion>

  <Accordion title="Reservation placement">
    Try your account's reservation capacity before falling back to shared trainer capacity.

    * **firectl**: add `--use-reservation` (default off).
    * **REST / Python SDK** (`>=1.2.8`): defaults to `useReservation: true` / `use_reservation=True`. Set to `false` to opt out.

    ```shell theme={null}
    firectl sftj create \
      --use-reservation \
      --base-model MY_BASE_MODEL \
      --dataset cancerset \
      --output-model my-tuned-model
    ```
  </Accordion>
</AccordionGroup>

### Deprecated parameters

<Warning>
  These parameters are deprecated. Do not include them in new managed
  fine-tuning requests. The wire fields remain present so existing resources can
  still be read, but Training V2 rejects or ignores non-default values as
  described below.
</Warning>

| Proto / Python field          | Former or legacy `firectl` flag           | Affected jobs                    | Migration behavior                                                                                                                                                                                                                         |
| ----------------------------- | ----------------------------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `batch_size`                  | `--batch-size`                            | SFT and DPO/ORPO Training V2     | Training V2 rejects nonzero values; use `batch_size_samples` / `--batch-size-samples`. RFT and RLOR V1 paths are not affected: they still use `batch_size` as the packed-token budget alongside the optional `batch_size_samples` control. |
| `gradient_accumulation_steps` | `--gradient-accumulation-steps`           | SFT, DPO/ORPO, RFT, RLOR trainer | Legacy V1 accumulation control. SFT and DPO/ORPO V2 reject nonzero values. Use `batch_size_samples` to control samples or preference pairs per optimizer step.                                                                             |
| `jinja_template`              | —                                         | SFT and shared training config   | Training V2 rejects non-empty values. Conversation rendering comes from the base model's registered renderer configuration.                                                                                                                |
| `early_stop`                  | `--early-stop`                            | Managed SFT                      | Early stopping is not supported by managed training. The CLI flag is no longer exposed; omit the field or leave it `false`.                                                                                                                |
| `mtp_enabled`                 | `--mtp-enable`                            | Managed SFT                      | MTP training is no longer supported. The CLI flag was removed, and managed training rejects `true`.                                                                                                                                        |
| `mtp_num_draft_tokens`        | `--mtp-num-draft-tokens`                  | Managed SFT                      | Deprecated with MTP support. The CLI flag was removed; leave the field unset (`0`).                                                                                                                                                        |
| `mtp_freeze_base_model`       | `--mtp-freeze-base-model`                 | Managed SFT                      | Deprecated with MTP support. The CLI flag was removed; leave the field unset (`false`).                                                                                                                                                    |
| `extra_values`                | `--extra-values` (admin-only legacy flag) | Managed SFT                      | Legacy V1 Helm overrides. Training V2 rejects a non-empty map.                                                                                                                                                                             |

## Appendix

* `Python SDK` [references](/tools-sdks/python-sdk)
* `Restful API` [references](/api-reference/introduction)
* `firectl` [references](/tools-sdks/firectl/firectl)
* [Complete Python SDK workflow example](https://github.com/fw-ai-external/python-sdk/blob/main/examples/sftj_workflow.py) for a code-only implementation

<h2>
  Vision fine-tuning
</h2>

Vision-language SFT uses the same managed job flow as text SFT with multimodal content in `messages`. Confirm VLM support and training shapes in the live [Models](/fine-tuning/models) matrix because modality, method, and shape eligibility are model-specific.

Each message `content` is an array of text and `image_url` objects. Images must be base64 data URIs with a MIME type; raw HTTP image URLs are not supported in training datasets.

```json theme={null}
{
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "What is shown?"},
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/jpeg;base64,/9j/4AAQSkZJRg..."
          }
        }
      ]
    },
    {
      "role": "assistant",
      "content": [{"type": "text", "text": "A red bicycle."}]
    }
  ]
}
```

Multiple images and multi-turn conversations use the same content-array shape. Keep the assistant response you want to train as the final message, upload the JSONL dataset, then create the managed SFT job with a VLM-capable base model.

<Warning>
  Download remote images and encode them before upload. Keep each MIME prefix accurate (`image/jpeg`, `image/png`, and so on), and validate the rendered sample before starting a paid job.
</Warning>

For Training API VLM loops, use a VLM-compatible training shape and the model's processor rather than a text-only tokenizer. The same Training API primitives support multimodal SFT, DPO, and RL datums. Start from the relevant cookbook recipe and verify processor output and loss masks before launch. Shape details: [Training Shapes](/fine-tuning/models#vision-and-multimodal-support).

<h2>
  Debug SFT tokenization
</h2>

If the model learns the wrong text or ignores assistant turns, the training renderer may not match inference tokenization.

1. In the dashboard, open your SFT job → **Render Samples** to inspect token boundaries and loss masks.
2. Compare rendered tokens against inference for the same prompt.
3. For custom renderers or agent-driven debugging, use the [training skill — renderer verification](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/renderer-verification.md).

<Note>
  In the REST response, the render preview identifies each datum produced from a source row with `examples[].renderings[].renderedDatums[].datumIndex`. Downloaded **Render Samples** use the legacy `split_index` field. The fields play corresponding roles in different schemas; `split_index` has not been renamed in the downloadable artifact.
</Note>

Common fix: ensure assistant messages you intend to train have non-zero loss weight; system/user turns should be masked out.

### Common findings

| What you see                                        | Likely cause                                                                                                                                                                                                                         | What to do                                                                                                                                                   |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Assistant answer tokens have `token_weights` of `0` | The assistant message has `weight: 0`, the sample has zero weight, or the job is configured to train on different content.                                                                                                           | Check the original JSONL row and remove unintended weights.                                                                                                  |
| User or system tokens have positive `token_weights` | The row schema or training configuration is not representing roles as intended.                                                                                                                                                      | Verify every message has the correct `role`, and avoid putting assistant text in a `user` message.                                                           |
| Expected text is missing from `decoded_tokens`      | The source row may have been split, truncated, or rendered differently by the model chat template.                                                                                                                                   | Check `split_index`, source line number, and the job's max context length.                                                                                   |
| Extra special tokens appear around messages         | The selected model renderer is adding chat template markers.                                                                                                                                                                         | This is often expected. If the markers are wrong for your use case, check that the base model and dataset format match.                                      |
| Thinking traces missing from conversation history   | The job's thinking-history mode and model renderer decide whether earlier turns' `reasoning_content` is retained. Interleaved removes thinking across user-turn boundaries; Preserved retains it. Datum unrolling is model-specific. | Compare the available modes in the render preview, then verify the created job's mode. See [Thinking history in fine-tuning](/fine-tuning/thinking-history). |
| Token boundaries look surprising                    | Many tokenizers encode whitespace, Unicode, and byte fallback pieces in non-obvious ways.                                                                                                                                            | Compare with the same Hugging Face tokenizer using `skip_special_tokens=False`.                                                                              |
| The Render Samples row is missing                   | The job may predate this feature, may have failed before rendering, or may not have captured samples.                                                                                                                                | Create a new supervised fine-tuning job, or contact support with the job ID if the job should have rendered samples.                                         |


# Training Overview
Source: https://docs.fireworks.ai/fine-tuning/finetuning-intro



Fine-tuning adapts a base model to your task by training it on your own data, so it learns your formats, tone, tools, and edge cases instead of relying on prompt instructions alone. Fireworks runs the training for you, without the burden of building and maintaining your own GPU or training infrastructure.

Fine-tuning is worth it when you want:

* **Higher task quality** - beat a general-purpose model on your specific workload, and often match or exceed a larger closed model.
* **Lower latency and cost** - a smaller specialized model can replace a bigger one at a fraction of the per-token cost.
* **Consistent behavior** - bake in formats, style, and tool-use so you stop paying for long prompts and few-shot examples on every request.
* **Ownership and no infra** - you keep the resulting weights, and Fireworks handles the GPUs, scheduling, and checkpointing.

<Info>
  **Coming from OpenAI?** Fireworks uses the same **OpenAI-compatible chat completion format** for training data — the same `messages` array with `role`, `content`, `tool_calls`, and `weight` fields. You can use your existing SFT datasets with no conversion required. See the [SFT dataset format](/fine-tuning/fine-tuning-models#fine-tuning-a-model-using-sft) for the full schema and examples.
</Info>

<h2>
  Choose a method
</h2>

Pick a method based on the data or signal you have. All three run as standard jobs on [Managed Fine-Tuning](/fine-tuning/managed-finetuning-intro), or as custom loops you write yourself on the [Training API](/fine-tuning/training-api/introduction).

|                              | SFT                                                                                                                                                         | DPO                                                                                                         | RL                                                                                                                                                     |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Good for**                 | Classification, extraction, format and tone adherence, distillation                                                                                         | Steering the model toward a goal you cannot measure objectively, such as style, helpfulness, or safety      | Tasks where you have no verified outputs to learn from, but you can tell whether an outcome was good or bad. Pushing the model beyond state-of-the-art |
| **Data you supply**          | Verified input/output pairs, or successful trajectories                                                                                                     | Preference pairs, single-turn only: one prompt, a chosen and a rejected response                            | Prompts, plus an evaluator that can tell a good outcome from a bad one                                                                                 |
| **Dataset size**             | Hundreds of examples, or roughly 10M+ tokens                                                                                                                | Hundreds to thousands of pairs                                                                              | Dozens to thousands of prompts, sometimes more. Often fewer than 100 is enough                                                                         |
| **Consider alternatives if** | You have very few examples, or no high-quality verified outputs to learn from                                                                               | Outputs can be judged objectively, or you already have high-quality verified pairs. Both point to SFT or RL | You have no way at all to judge an outcome, including an LLM judge. Simpler methods are untried, or you want a quick fine-tuning experiment            |
| **Guides**                   | [Text](/fine-tuning/fine-tuning-models) · [Vision](/fine-tuning/fine-tuning-models#vision-fine-tuning) · [Cookbook](/fine-tuning/training-api/cookbook/sft) | [Managed DPO / ORPO](/fine-tuning/dpo-fine-tuning) · [Cookbook](/fine-tuning/training-api/cookbook/dpo)     | [Managed RFT](/fine-tuning/reinforcement-fine-tuning-models) · [Cookbook](/fine-tuning/training-api/cookbook/rl)                                       |

<Tip>
  **Verifiable** means you can reliably judge whether a model output is good (rules, unit tests, programmatic checks). RL fits reasoning and agentic tasks where full ground-truth labels are hard to write.
</Tip>

The Training API also supports custom methods (GRPO, distillation, and others) via the Python SDK. See [Cookbook recipes](/fine-tuning/training-api/cookbook/overview) and [Managed Fine-Tuning](/fine-tuning/managed-finetuning-intro) for model support and pricing.

<h2>
  Choose a surface
</h2>

Pick a **surface** (managed or Training API, serverless or dedicated). The surface decides how much of the model you update and which **interfaces** are available to you.

Answer the question below and the flow takes you to your surface, which links to its guide. Click any answered question to change it, or show every path at once.

<TrainingDecisionFlow />

Compare the last branch in detail on [serverless versus dedicated](/fine-tuning/training-api/introduction#infrastructure), and check per-model support on [Models](/fine-tuning/models).

### Managed Fine-Tuning vs Training API

Fireworks offers two ways to train: **Managed Fine-Tuning** (Fireworks runs the loop) and the **Training API** (you write the loop in Python).

| Choose Managed Fine-Tuning when                                   | Choose the Training API when                                                               |
| ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| You need a standard SFT, DPO, ORPO, or RFT job                    | You need a custom loss, reward, rollout, trajectory, or optimizer-step loop                |
| You want Fireworks to own scheduling, training, and checkpointing | You want to fork or write Python training logic                                            |
| A supported model and managed configuration cover the task        | You need inference in the loop, distillation, per-step diagnostics, or research algorithms |

<CardGroup>
  <Card title="Managed Fine-Tuning" icon="wand-magic-sparkles" href="/fine-tuning/managed-finetuning-intro">
    Standard jobs with a platform-managed loop.
  </Card>

  <Card title="Training API" icon="code" href="/fine-tuning/training-api/introduction">
    Programmable loops built from cookbook recipes or the SDK.
  </Card>
</CardGroup>

### Serverless vs Dedicated infrastructure

Infrastructure applies to the **Training API** only. Managed Fine-Tuning uses platform-managed compute.

| Choose Serverless Training when                                                   | Choose Dedicated Training when                                                     |
| --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| Supported LoRA SFT or RL covers the workload                                      | You need full-parameter, DPO, ORPO, distillation, or broader model support         |
| You want shared pooled compute with no trainer or sampler deployment provisioning | You need explicit trainer, deployment, checkpoint, reconnect, or promotion control |
| Per-token billing fits a small or bursty experiment                               | A sustained, highly utilized time-based run fits the workload                      |
| In-session sampling is sufficient                                                 | You need provisioned rollout or evaluation deployments                             |

<CardGroup>
  <Card title="Serverless Training" icon="bolt" href="/fine-tuning/training-api/serverless">
    Shared pooled trainer, no provisioning, per-token billing.
  </Card>

  <Card title="Dedicated Training" icon="server" href="/fine-tuning/training-api/dedicated">
    Provisioned trainer and deployment resources with explicit lifecycle control.
  </Card>
</CardGroup>

See the detailed [serverless versus dedicated comparison](/fine-tuning/training-api/introduction#infrastructure).

## Choose how to interact

* **Skill** — the only interface that drives both surfaces. Your coding agent configures, runs, and troubleshoots training through the [Fireworks training skill](/fine-tuning/agent/use-with-coding-agents).
* **Fireworks UI, `firectl`, or the REST API** — managed jobs only. Guided creation and monitoring in the UI, reproducible job and resource automation from the CLI or API.
* **Python SDK** — Training API loops only, on serverless or dedicated. Start from a [cookbook recipe](/fine-tuning/training-api/cookbook/overview).

<Note>
  **CLI or API vs Python SDK:** `firectl` and the REST API manage **managed** jobs and platform resources. The **Python SDK** runs **Training API** loops you author yourself (loss, rollouts, optimizer steps).
</Note>

### GPU quota prerequisite

Managed jobs and dedicated Training API runs need training GPU quota, granted automatically by [spending tier](/guides/quotas_usage/account-quotas#training-gpu-quota). [Serverless Training](/fine-tuning/training-api/serverless) uses a shared pool with its own model, concurrency, and rate limits instead of dedicated training GPU quota.

<Accordion title="Training GPU quota by spending tier">
  | Tier              | How to reach it                          | B200 / B300 (Blackwell) | H200 | H100 / A100 |
  | ----------------- | ---------------------------------------- | :---------------------: | :--: | :---------: |
  | No payment method | —                                        |            0            |   0  |      0      |
  | Tier 1            | Valid payment method and billing profile |            0            |  16  |      8      |
  | Tier 2            | Spend or add \$50 in credits             |            16           |  16  |      16     |
  | Tier 3            | Spend or add \$500 in credits            |            24           |  24  |      24     |
  | Tier 4            | Spend or add \$5,000 in credits          |            32           |  32  |      32     |

  Check your quota with the Fireworks CLI (`firectl quota list`). A job rejected with HTTP 429 `quota_exceeded` (sometimes a `403` on the job poll) is a tier issue, not a dataset/config problem.
</Accordion>

<Note>
  Need more training quota than your tier allows? [Reach out for enterprise support](https://fireworks.ai/contact-training) and we'll help size the right allocation for your workload.
</Note>

## Models

Model availability is decided per model and per surface — managed jobs by method (SFT, DPO, RFT), Training API jobs by parameter mode (LoRA or full-parameter). Check the live catalog before you launch.

<Card title="Models" icon="microchip" href="/fine-tuning/models">
  Browse the base model catalog with per-model surface, method, and training-shape support.
</Card>

<h2>
  Training security
</h2>

Across every training surface, one principle holds: **your training data is never used to train Fireworks-owned or shared models**. Inference follows [Zero Data Retention](/guides/security_compliance/data_handling) by default. Step-by-step BYOB IAM, CMEK KMS setup, and secure RFT automation live in the [Fireworks training skill — secure training operations](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/secure-training-operations.md).

### Choose a surface by data-privacy needs

| Surface              | Where your training data lives                                  | What Fireworks retains       | Your deletion controls                                            |
| -------------------- | --------------------------------------------------------------- | ---------------------------- | ----------------------------------------------------------------- |
| **Managed Training** | Fireworks-managed storage (GCS); reference link in our database | Dataset, checkpoints, traces | Delete dataset after job; request checkpoint/trace deletion       |
| **Managed + BYOB**   | Your cloud bucket; read in-place during training only           | Checkpoints and traces only  | Revoke bucket access after job; request checkpoint/trace deletion |
| **Training API**     | No dataset file on Fireworks — transient tokenized batches only | Checkpoints and traces only  | Request checkpoint/trace deletion                                 |

Checkpoints and traces are retained \~30 days by default (deletable on request). Strictest governance: [BYOB](#dataset-storage-byob) (dataset never copied to Fireworks) or the [Training API](/fine-tuning/training-api/introduction) (no stored dataset file).

<h3>
  Bring your own bucket (BYOB)
</h3>

Register an external URL so Fireworks reads your dataset during the job without persisting a copy, then revoke access after the job:

```bash theme={null}
firectl dataset create my-dataset --external-url gs://your-bucket/path/train.jsonl
```

Supported: GCS, AWS S3, and Azure Blob, with least-privilege IAM to Fireworks service accounts provided at onboarding. For AWS S3, lock the IAM trust policy with both `accounts.google.com:sub` and `accounts.google.com:oaud` (your Fireworks account ID) so tokens for other accounts are rejected. Full IAM trust policies, OIDC audience, and rotation are in the [secure training operations skill reference](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/secure-training-operations.md).

<h3>
  Customer-managed encryption keys (CMEK)
</h3>

CMEK encrypts datasets and checkpoints on Fireworks-managed storage with **your** cloud KMS key — revoke the key and Fireworks cannot decrypt. **Supported today:** AWS KMS + Managed SFT (DPO, RFT, Azure Key Vault, and GCP KMS coming soon). It does not cover in-memory training compute or inference request/response. Setup, IAM, and rotation detail: [secure training operations skill reference](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/secure-training-operations.md) · [Data Security Overview](/guides/security_compliance/data_security).

### Secure RFT and customer controls

For RFT under strict governance, combine a [BYOB](#dataset-storage-byob) dataset with evaluators and rollout servers kept in your own environment (see [Remote Environment Setup](/fine-tuning/connect-environments)). To delete checkpoints, traces, or rollout data, contact your Fireworks account team; datasets are deletable from the console or API after a job completes.

## Before launch

Verify current model support, shapes, access status, pricing, limits, and quota in the linked live pages. A coding agent asks for confirmation before upload, registration, paid inference, job creation, promotion, deployment, or another mutation. Material changes require approval again; promotion and deployment are confirmed separately.


# Managed Fine-Tuning Overview
Source: https://docs.fireworks.ai/fine-tuning/managed-finetuning-intro

Fine-tune models with Fireworks-managed infrastructure — no custom code required.

Give Fireworks your data and configuration. The platform handles scheduling, training, checkpointing, and model output. Training data uses the **OpenAI-compatible chat completion format**, so existing OpenAI SFT datasets work with no conversion required.

## How to launch managed training

These interfaces create the same underlying managed jobs:

| Interface        | Use when                                                         |
| ---------------- | ---------------------------------------------------------------- |
| **Fireworks UI** | You want guided configuration and visual monitoring              |
| **CLI or API**   | You want scripted and reproducible job operations                |
| **Your agent**   | You want help configuring, running, and troubleshooting training |

The Fireworks CLI is called `firectl`. [Install the training skill](/fine-tuning/agent/use-with-coding-agents) to use your agent, or continue with the method-specific managed guides below. For custom Python training loops, start with the [Training API overview](/fine-tuning/training-api/introduction).

## Methods

<CardGroup>
  <Card title="Supervised Fine-Tuning" href="/fine-tuning/fine-tuning-models" icon="message">
    Train text and vision models with labeled examples of desired outputs
  </Card>

  <Card title="Preference Optimization (DPO / ORPO)" href="/fine-tuning/dpo-fine-tuning" icon="arrows-left-right">
    Train on preferred and non-preferred response pairs using DPO or ORPO
  </Card>

  <Card title="Reinforcement Fine-Tuning" href="/fine-tuning/reinforcement-fine-tuning-models" icon="brain">
    Train models using custom reward functions for complex reasoning tasks
  </Card>
</CardGroup>

## Supported base models

Fireworks supports fine-tuning for major open source model families, including DeepSeek, Qwen, Kimi, Gemma, GLM, and Llama. Eligibility is decided per model and per method: a model can support SFT without supporting DPO or RFT.

[**Models**](/fine-tuning/models) is the live per-model matrix: the surfaces and methods each base model is enabled for, the training shapes behind it, and each shape's maximum context length. Check it before creating a job, and set the job context from a shape that supports the method you picked, using `firectl sftj create`, `firectl dpo-job create`, or the corresponding RFT command.

Custom models uploaded by users are not automatically tunable. To use managed fine-tuning with an uploaded custom base model, the model must have a corresponding Hugging Face URL. Fireworks uses that URL to infer the training renderer and locate compatible training shapes. A custom model is supported only when Fireworks can resolve both a supported renderer and at least one compatible training shape. After the Hugging Face URL is set, tunability is refreshed by a background operation that runs about every 30 minutes, so the model may take up to 30 minutes to show as `Tunable: true`. We are working to make this refresh faster.

To browse the broader catalog (including non-tunable inference models), visit the [Model Library for text models](https://app.fireworks.ai/models?filter=LLM\&tunable=true) or [vision models](https://app.fireworks.ai/models?filter=vision\&tunable=true).

## Tuning modes and context length

Managed fine-tuning runs **[Low-Rank Adaptation (LoRA)](https://arxiv.org/abs/2106.09685)** only. It supports the full context lengths exposed by the available training shapes, matching the same long-context capabilities used by cookbook recipes.

LoRA gives you efficient adapter training and flexible deployment, including [multiple LoRAs](/fine-tuning/deploying-loras#multi-lora-deployment) on a single base model deployment. For full-parameter tuning, use the [Training API](/fine-tuning/training-api/introduction).

<Warning>
  **Deprecation notice:** The `deployedModel` request key for routing to LoRA addons is deprecated and will not be supported for any new deployments. Please migrate to the `model` field with the `<model_name>#<deployment_name>` format described in [Routing requests to LoRA addons](/fine-tuning/deploying-loras#multi-lora-deployment).
</Warning>


# Models
Source: https://docs.fireworks.ai/fine-tuning/models

Which base models you can train on Fireworks and the surfaces each one is available on.

Managed fine-tuning, the Training API, and serverless training all draw from the same base model catalog, but availability is decided per model: managed jobs by method (SFT, DPO, RFT), Training API jobs by parameter mode (LoRA or full-parameter).

## Model availability

Pick a model to see the surfaces and methods it is enabled for, plus any training shapes that back it. Switch to **All models** for the full matrix.

<ModelsCatalog />

## Vision and multimodal support

Vision support is model- and surface-specific. Use the catalog above to confirm that the selected VLM has a compatible managed method or Training API shape before preparing data.

* Managed VLM SFT dataset schema and launch flow: [Supervised Fine-Tuning: Vision](/fine-tuning/fine-tuning-models#vision-fine-tuning)
* Training API VLM loops: start from a VLM-compatible shape and the same cookbook SFT, DPO, or RL recipe used for text, replacing the text tokenizer with the model processor
* Inference request formats after deployment: [Vision-language models](/guides/querying-vision-language-models)

## Next steps

<CardGroup>
  <Card title="Managed Fine-Tuning" href="/fine-tuning/managed-finetuning-intro" icon="wand-magic-sparkles">
    Hand Fireworks your data and let the platform run the job
  </Card>

  <Card title="Training API" href="/fine-tuning/training-api/introduction" icon="code">
    Write your own training loop against a Tinker-compatible API
  </Card>

  <Card title="Serverless Models" href="/fine-tuning/training-api/serverless#models" icon="bolt">
    Serverless Training API model catalog with per-token pricing
  </Card>

  <Card title="Training Shapes" href="/fine-tuning/training-api/training-shapes" icon="microchip">
    What a shape pins and how to reference one
  </Card>

  <Card title="Dedicated Training" href="/fine-tuning/training-api/dedicated" icon="server">
    Provision a trainer and sampler on reserved GPU capacity
  </Card>

  <Card title="Pricing" href="https://fireworks.ai/pricing" icon="tag">
    Current rates across training and inference
  </Card>
</CardGroup>


# Price comparison vs Tinker
Source: https://docs.fireworks.ai/fine-tuning/multi-turn-cost-comparison

Estimate the cost of multi-turn agentic RL rollouts on Fireworks compared to Tinker's per-token pricing

If you're running RL or agentic post-training on a long-context model and your
provider bills you per token with **no cross-turn prefix cache**, the prefill
cost grows quadratically with the number of turns — every turn re-prefills the
full conversation history. On Fireworks Dedicated, session-affinity routing
keeps an episode pinned to one replica so the KV cache is reused across turns,
and cached prompt tokens contribute essentially zero extra compute.

The calculator below makes that difference concrete. Set your episode shape
(turns, context growth, generation length) and compare:

* **Tinker** — flat per-token billing, no cross-turn cache (re-prefill every turn)
* **Fireworks Dedicated** — on-demand GPU-hour billing; the cache savings show up as more work per hour, not as a discounted token rate

<MultiTurnCostCalculator />

## Performance and benchmarking notes

### Dedicated trainer vs pooled/serverless resourcing

Tinker runs training jobs on a **pooled/serverless** GPU fleet, which lets a
single job burst onto many more GPUs than you would dedicate to a replica on
Fireworks. That burst is what makes individual Tinker steps feel fast — but it
also **caps the maximum training speed you can buy**: you cannot pay to scale
beyond the pool's per-job allocation, and you cannot reserve isolated capacity.

Fireworks dedicated trainers take the opposite trade-off: predictable,
isolated execution with no shared-pool queueing or noisy-neighbor variance,
and the ability to scale **wall-clock time and cost independently** by
adjusting replica count. If you want faster steps on dedicated, increase
replica count and parallelize work.

For **large model training or longer rollouts**, we have consistently found
the dedicated setup like ours is **cheaper overall and can also be faster**
depending on the customer's resourcing needs.

### Context-length benchmarking caveat

Benchmark comparisons are only apples-to-apples when truncation policy and
effective context length are matched. If one system truncates `>32k` samples
and another does not, the non-truncating run is doing more work and will
appear slower.

### Replica count is a speed/cost knob

Users can trade cost and wall-clock time by scaling replicas. A quick
back-of-envelope estimate:

$$
\text{\$ / 1M tokens} \approx \frac{\text{GPU count} \cdot \text{\$ / GPU-hour}}{\text{tokens/sec(cluster)} \cdot 3600} \cdot 10^6
$$

### Check utilization before scaling

Fireworks Dedicated is billed by GPU-hour, so low rollout traffic can make a
job look slow or expensive even when the deployment has spare capacity. Before
adding replicas, first confirm whether the inference deployment is saturated or
waiting for more work from your rollout client.

Useful signals:

* **Per-request performance metrics:** log Fireworks response metrics such as
  prompt tokens, cached prompt tokens, time to first token, and total server
  processing time from your rollout client. Non-streaming requests include
  these in response headers; for streaming requests, set
  [`perf_metrics_in_response`](/guides/querying-text-models#usage--performance-tracking)
  to include them in the final response chunk.
* **Deployment-level metrics:** export
  [Prometheus-style metrics](/deployments/exporting-metrics) for request rate,
  prompt and cached-token rates, queue latency, KV-cache usage, and concurrent
  request count. Low request/concurrency metrics with low queueing usually mean
  the deployment can accept more traffic.
* **Training API efficiency hints:** when available, monitor
  `trainer/training_efficiency/.../effective_batch_fill_ratio:last` and
  `trainer/training_efficiency/.../trainer_waiting_for_work:last`. These are
  returned in the `metrics` dict on your `forward` / `forward_backward`
  responses, not on the deployment dashboard. Low batch fill or a
  trainer-waiting-for-work signal usually points to the rollout side not
  feeding the trainer fast enough. See
  [Reading Training API efficiency metrics](#reading-training-api-efficiency-metrics)
  below for how to access and interpret them.

If the deployment is not saturated, increase rollout traffic first. For managed
RFT and Training API jobs, the main throughput knob is concurrent rollouts; see
[`max_concurrent_rollouts`](/fine-tuning/rft-parameters-reference)
and the Training API [deployment replica guidance](/fine-tuning/training-api/reference/deployment-manager#deployment-shape-and-training-shapes).

#### Reading Training API efficiency metrics

The two `trainer/training_efficiency/...` metrics are returned in the `metrics`
dict on your `forward` / `forward_backward` responses. They do **not** appear on
inference deployment dashboards, the per-request and deployment-level signals
above are separate.

```python theme={null}
# forward / forward_backward return a future, call .result()
result = training_client.forward_backward(datums, "cross_entropy").result()

# result.metrics is a dict; it includes:
#   trainer/training_efficiency/.../effective_batch_fill_ratio:last
#   trainer/training_efficiency/.../trainer_waiting_for_work:last
print(result.metrics)
```

* **`effective_batch_fill_ratio:last`**: the number of tokens in a batch
  divided by the maximum possible. **1.0 means fully saturated**; consistently
  low values across steps indicate under-filling.
* **`trainer_waiting_for_work:last`**: how much time the trainer (GPU) sat idle
  since the last op, i.e. the gap between `forward` calls. More waiting means the
  trainer is starved for work.

Low fill or significant waiting-for-work means the rollout side isn't feeding the
trainer fast enough: raise rollout concurrency
(`max_concurrent_rollouts`) before adding deployment replicas.

## How the numbers come together

### Tinker (the cost customers describe)

Each turn re-prefills the full accumulated context:

$$
\text{Prefill tokens (Tinker)} = \sum_{t=1}^{T} P_t = T \cdot P_1 + \Delta \cdot \frac{T(T-1)}{2}
$$

…where $P_1$ is the initial prompt (system + tools + task), $\Delta$ is the
context added per turn (model response + tool result), and $T$ is the turn
count. This is **quadratic in $T$**.

$$
\text{Cost (Tinker)} = \frac{\text{Prefill tokens}}{10^6} \cdot r_{\text{prefill}} + \frac{\text{Decode tokens}}{10^6} \cdot r_{\text{sample}}
$$

### Fireworks Dedicated — GPU-hour billing

Dedicated deployments are billed per GPU-second, so the prefix cache shows up
as **higher effective throughput** rather than a discount on per-token rates.
Across one episode, each unique token is prefilled at most once — the rest of
the prompt is served from the prefix cache and contributes essentially no GPU
work. The uncached portion that actually hits prefill is:

$$
\text{Uncached prompt} = P_T = P_1 + (T - 1) \Delta
$$

On a saturated cluster:

$$
\text{Cluster-hours} = \frac{\text{Uncached prompt} / \text{prefill TPS}}{3600}
$$

$$
\text{Cost} = \text{Cluster-hours} \cdot N_{\text{GPU}} \cdot r_{\text{GPU/hr}}
$$

Because cached tokens contribute essentially nothing to wall-clock work, the
cluster's effective \$/M token rate falls as utilization rises. For continuous
RL training, where rollouts run at sustained pace, dedicated is typically the
cheapest path at scale.

<Note>
  The calculator's dedicated path uses *saturated* throughput estimates as
  defaults. A small, lightly-loaded test deployment will look more expensive
  per token than these numbers because the cluster is paid for whether it's
  busy or idle. Tune the throughput inputs in the **Advanced** panel to match
  your actual rollout pace.
</Note>

## What's covered

The calculator currently includes the four models for which Tinker publishes
per-token rates:

| Model                    | Tinker prefill / sample (per 1M) |
| ------------------------ | -------------------------------- |
| Kimi K2.6 (128K)         | $5.15 / $12.81                   |
| Kimi K2.5 (128K)         | $5.15 / $12.81                   |
| Qwen3.5-397B-A17B (256K) | $4.00 / $10.00                   |
| GPT-OSS-120B (128K)      | $0.63 / $1.54                    |

All Fireworks-side rates are taken from the public pages linked below and the
constants live in `snippets/multi-turn-cost-calculator.jsx` — update there if
either side's pricing changes.

## FAQ

### What is the fastest way to reduce wall-clock time?

Increase replicas and overlap sampling/training where your workflow allows it.
Those are usually the most direct levers for shortening end-to-end cycle time.

### How should I compare costs between providers?

Use matched assumptions for context length, truncation policy, and effective
resource allocation. The calculator at the top of this page handles the math
once you plug in your episode shape — be sure to also align truncation policy
and effective context window between providers before drawing conclusions.

## Sources

* Tinker pricing: [thinkingmachines.ai/tinker](https://thinkingmachines.ai/tinker)
* Fireworks GPU-hour pricing: [fireworks.ai/pricing](https://fireworks.ai/pricing)
* Related: [RFT Cost Estimator](/fine-tuning/reinforcement-fine-tuning-models#rft-cost-planning) — same idea, but
  for the training-side bill (Fireworks GPU-hour, no comparison column).

<Warning>
  This is an estimator, not a quote (updated). Real costs depend on your exact workload,
  cache hit rate, hardware utilization, and rate-card terms at run time.
</Warning>


# Single-Turn Training Quickstart
Source: https://docs.fireworks.ai/fine-tuning/quickstart-math

Train a model to be an expert at answering GSM8K math questions

<Note>
  **Following the [RFT Overview](/fine-tuning/reinforcement-fine-tuning-models)?** This is the **Single-Turn Training** path—the fastest way to get started with RFT.
</Note>

In this quickstart, you'll train `Qwen3 4B` to solve mathematical reasoning problems from the GSM8K dataset.

## What you'll learn

* How to set up and test an evaluator locally, using the Eval Protocol SDK
* How to take that evaluator and use it in an RFT job, from the command line
* How to monitor training progress and evaluate accuracy improvements

<Tip>
  Prefer a notebook experience? You can also [run this tutorial in Google Colab](https://colab.research.google.com/drive/16xrb9rx6AoAEOtrDXumzo71HjhunaoPi#scrollTo=CP18QX4tgi-0). Note that Colab requires billing enabled on your Google account.
</Tip>

## Prerequisites

* Python 3.10+
* A Fireworks API key (stored in your shell or .env)
* Command-line access (terminal or shell)

## 1. Install dependencies and set up files

<Tabs>
  <Tab title="Clone repository (recommended)">
    Clone the quickstart-gsm8k repository and install dependencies:

    ```bash theme={null}
    git clone https://github.com/eval-protocol/quickstart-gsm8k.git
    cd quickstart-gsm8k
    pip install -r requirements.txt
    ```

    Create the `gsm8k_artifacts/` folder structure and copy files:

    ```bash theme={null}
    mkdir -p gsm8k_artifacts/{tests/pytest/gsm8k,development}
    cp evaluation.py gsm8k_artifacts/tests/pytest/gsm8k/test_pytest_math_example.py
    cp gsm8k_sample.jsonl gsm8k_artifacts/development/gsm8k_sample.jsonl
    ```

    The repository includes:

    * **Evaluator** (`evaluation.py`): Defines how to evaluate math answers
    * **Dataset** (`gsm8k_sample.jsonl`): Contains example math problems to test on
  </Tab>

  <Tab title="Download files manually">
    Install the latest `eval-protocol` SDK, `pytest`, and `requests`:

    ```bash theme={null}
    python -m pip install --upgrade pip
    python -m pip install pytest requests git+https://github.com/eval-protocol/python-sdk.git
    ```

    Download the evaluator and dataset files:

    Run this Python script to download two files from the Eval Protocol repository into a folder on your machine called `gsm8k_artifacts/`.

    * **Test script** (`test_pytest_math_example.py`): Defines how to evaluate math answers
    * **Sample dataset** (`gsm8k_sample.jsonl`): Contains example math problems to test on

    ```python tutorial/download_gsm8k_assets.py theme={null}
    from pathlib import Path
    import requests

    ARTIFACT_ROOT = Path("gsm8k_artifacts")
    TEST_PATH = ARTIFACT_ROOT / "tests" / "pytest" / "gsm8k" / "test_pytest_math_example.py"
    DATASET_PATH = ARTIFACT_ROOT / "development" / "gsm8k_sample.jsonl"

    files_to_download = {
        TEST_PATH: "https://raw.githubusercontent.com/eval-protocol/python-sdk/main/tests/pytest/gsm8k/test_pytest_math_example.py",
        DATASET_PATH: "https://raw.githubusercontent.com/eval-protocol/python-sdk/main/development/gsm8k_sample.jsonl",
    }

    for local_path, url in files_to_download.items():
        local_path.parent.mkdir(parents=True, exist_ok=True)
        response = requests.get(url, timeout=30)
        response.raise_for_status()
        local_path.write_bytes(response.content)
        print(f"Saved {url} -> {local_path}")
    ```

    Expected output:

    ```
    Saved https://raw.githubusercontent.com/.../test_pytest_math_example.py -> gsm8k_artifacts/tests/pytest/gsm8k/test_pytest_math_example.py
    Saved https://raw.githubusercontent.com/.../gsm8k_sample.jsonl -> gsm8k_artifacts/development/gsm8k_sample.jsonl
    ```
  </Tab>
</Tabs>

## 2. Test your evaluator locally

In this step, we will test your evaluator by examining the output locally. Feel free to iterate on the evaluator you downloaded in the last step until it gives the output you want.

<Steps>
  <Step title="Start the local UI server">
    Open a terminal and run:

    ```bash theme={null}
    ep logs
    ```

    This will start a local server, navigate to `http://localhost:8000`. Keep this terminal running.
  </Step>

  <Step title="Run the test script">
    In a **new terminal**, call the test script to run the evaluator on your dataset of sample math problems.

    ```bash theme={null}
    cd gsm8k_artifacts
    ep local-test
    ```

    This command discovers and runs your `@evaluation_test` with pytest.

    As the test runs, you'll see evaluation scores appear in the browser, with detailed logs for each problem the model attempts. `pytest` will also register your evaluator and dataset with Fireworks automatically, so you can use them in the next step for RFT.
  </Step>
</Steps>

<Frame>
  <img alt="GSM8K evaluation UI showing model scores and trajectories" />
</Frame>

## 3. Start training

First, set your Fireworks API key so the Fireworks CLI can authenticate you:

```bash theme={null}
export FIREWORKS_API_KEY="<your-fireworks-key>"
```

Next, launch the RFT job using the evaluator and dataset you registered. This example uses `qwen3-4b`; confirm current RFT eligibility and pricing before launch.

```bash theme={null}
cd ..
eval-protocol create rft \
  --base-model accounts/fireworks/models/qwen3-4b
```

The CLI will output dashboard links where you can monitor your training job in real-time.

<Frame>
  <img alt="GSM8K evaluation score showing upward trajectory" />
</Frame>

<Tip>
  You can also store your API key in a `.env` file instead of exporting it each session.
</Tip>

## Monitor your training progress

Your RFT job is now running. You can monitor progress in the dashboard links provided by the CLI output.

<AccordionGroup>
  <Accordion title="Evaluate accuracy regularly">
    Re-run the pytest evaluation command to measure your model's performance on new checkpoints:

    ```bash theme={null}
    cd gsm8k_artifacts
    pytest -q tests/pytest/gsm8k/test_pytest_math_example.py::test_math_dataset -s
    ```

    This helps you see how your model's accuracy improves over time and decide when to stop training.
  </Accordion>

  <Accordion title="Customize your evaluation">
    You can adjust the evaluation logic to better fit your needs:

    * **Modify reward shaping**: Edit the scoring logic in `test_pytest_math_example.py` to match your answer format expectations
    * **Use your own data**: Replace the sample dataset by either editing the JSONL file locally or passing `--dataset-jsonl` when creating the RFT job
  </Accordion>
</AccordionGroup>

### What's happening behind the scenes

Understanding the training workflow:

1. **Evaluation registration**: The pytest script evaluates a small GSM8K subset using numeric answer checking, then automatically registers both your evaluator and dataset with Fireworks
2. **RFT job creation**: The `create rft` command connects your registered evaluator and dataset to a Reinforcement Fine-Tuning job for your chosen base model
3. **Continuous improvement**: As training progresses, evaluation scores on the held-out set reflect improved accuracy, allowing you to iterate quickly before scaling to larger experiments

## Next steps

<CardGroup>
  <Card title="Customize training" icon="terminal" href="/fine-tuning/reinforcement-fine-tuning-models#cli">
    Learn all CLI options to customize your training parameters
  </Card>

  <Card title="Try remote agents" icon="server" href="/fine-tuning/quickstart-svg-agent">
    Train agents that run in your production infrastructure
  </Card>

  <Card title="Learn RFT concepts" icon="brain" href="/fine-tuning/reinforcement-fine-tuning-models">
    Understand how reinforcement fine-tuning works
  </Card>
</CardGroup>


# Remote Agent Quickstart
Source: https://docs.fireworks.ai/fine-tuning/quickstart-svg-agent

Train an SVG drawing agent running in a remote environment

<Note>
  **Following the [RFT Overview](/fine-tuning/reinforcement-fine-tuning-models)?** This is the **Remote Agent Training** path—for training agents that run in your production infrastructure.
</Note>

In this quickstart, you'll train an agent to generate SVG drawings. Your agent runs in a remote server (Vercel), which means rollouts happen remotely while Fireworks handles the training. This approach lets you train agents that already live in your production environment.

Here's a quick walkthrough:

<Frame>
  <iframe />
</Frame>

## What You'll Learn

* **Apply RFT to production agents** — Train models that work with remote servers and existing infrastructure
* **Remote rollout processing** — Connect your production environment to Fireworks RFT using Eval Protocol
* **Monitor and debug training** — Track progress, inspect rollouts, and debug issues with live logs

## 1. Installation

1. **Clone the quickstart repo**: [https://github.com/eval-protocol/quickstart](https://github.com/eval-protocol/quickstart)

```bash theme={null}
git clone git@github.com:eval-protocol/quickstart.git
cd quickstart
```

2. **Install Eval Protocol**:

```bash theme={null}
pip install "eval-protocol[svgbench]"
```

3. **Environment Setup**:

The `env.example` file is located in the `evaluator/` directory. Make a copy of it in the same directory, name it `.env`, and fill in your API keys:

```bash theme={null}
cp evaluator/env.example evaluator/.env
```

Then edit `evaluator/.env` with your API keys:

```
FIREWORKS_API_KEY=your-fireworks-key-here
OPENAI_API_KEY=your-openai-key-here
```

The create process below automatically reads and uploads these secrets to Fireworks.

For more details on Fireworks Secret Management usage, please refer to [using secret in evaluator](/fine-tuning/reinforcement-fine-tuning-models#using-secrets).

## 2. Test your evaluator locally

Test your evaluator locally before launching training, to verify everything works with your rollout processor.

**Terminal 1** - Start the local UI server to view results:

```bash theme={null}
ep logs
```

**Terminal 2** - Kick off the test:

```bash theme={null}
cd evaluator
ep local-test
```

This command discovers and runs your `@evaluation_test` with pytest. In this case, it builds an image and runs the test in Docker, because a `Dockerfile` is present.

The test automatically uses our Vercel remote server:

```
rollout_processor=RemoteRolloutProcessor(
    remote_base_url="https://vercel-svg-server-ts.vercel.app",
)
```

If you want to use a local development Vercel server instead, see [Local Development Server](#local-development-server).

**Note:**

* If your evaluation setup has custom system dependencies (e.g., Chromium), add a `Dockerfile`. When you run `ep local-test`, it will build an image and run `pytest` inside Docker.
* If you don't need Docker, `ep local-test` will run `pytest` on your host machine by default.
* You can ignore the `Dockerfile` and force host execution with: `ep local-test --ignore-docker`.

<Accordion title="Dockerfile constraints for RFT evaluators">
  RFT evaluators run in sandboxed environments. Your Dockerfile must follow these constraints:

  **Base image:**

  * Only Debian-based images are supported (e.g., Debian, Ubuntu, or `python:3.x-slim`)
  * Alpine, CentOS, and other non-Debian distros are not supported
  * If no Dockerfile is provided, the system uses a default Python environment with common packages pre-installed

  **Supported instructions:**

  * `FROM`: Base image (required, only one allowed)
  * `RUN`: Execute commands
  * `COPY` / `ADD`: Copy files into the image
  * `WORKDIR`: Set working directory
  * `USER`: Set the user
  * `ENV`: Set environment variables
  * `CMD` / `ENTRYPOINT`: Set the start command
  * `ARG`: Build-time variables

  **Unsupported features:**

  | Feature                | Status                                    |
  | ---------------------- | ----------------------------------------- |
  | Non-Debian base images | ❌ Not supported (no Alpine, CentOS, etc.) |
  | Multi-stage builds     | ❌ Not supported (only one `FROM` allowed) |
  | `EXPOSE`               | ⚠️ Ignored                                |
  | `VOLUME`               | ⚠️ Ignored                                |

  **Example Dockerfile:**

  ```dockerfile theme={null}
  FROM python:3.11-slim

  WORKDIR /app

  # Install system dependencies
  RUN apt-get update && apt-get install -y \
      chromium \
      && rm -rf /var/lib/apt/lists/*

  # Install Python dependencies
  COPY requirements.txt .
  RUN pip install --no-cache-dir -r requirements.txt

  # Copy evaluator code
  COPY . .

  CMD ["pytest", "-vs"]
  ```

  <Warning>
    Multi-stage Dockerfiles will fail during the evaluator build. Use a single `FROM` instruction and install all dependencies in one stage.
  </Warning>
</Accordion>

### Expected Test Output

Navigate to [http://localhost:8000](http://localhost:8000) to see the Eval Protocol UI.

```
INFO:eval_protocol.pytest.remote_rollout_processor:Found status log for rollout democratic-way-12: Rollout democratic-way-12 completed
INFO:eval_protocol.pytest.remote_rollout_processor:Found Fireworks log for rollout democratic-way-12 with status code 100.0
INFO:eval_protocol.adapters.fireworks_tracing:Successfully converted 1 traces to evaluation rows | 3/8 [00:19<00:22, 4.52s/rollout]
...
Runs (Parallel): 100%|████████████████████████████████████████████| 1/1 [00:31<00:00, 31.07s/run]
PASSED
```

<img alt="Eval Protocol Logs Interface" />

If you're interested in understanding how Remote Rollout Processing works and how it communicates with the remote server, see [How Remote Rollout Processing Works](#how-remote-rollout-processing-works).

## 3. Start training with a single command

To kickoff training, simply do:

```bash theme={null}
eval-protocol create rft \
  --base-model accounts/fireworks/models/qwen3-4b \
  --chunk-size 10
```

This command:

1. Uploads secrets — reads your `.env` and uploads API keys as Fireworks secrets
2. Uploads evaluator — packages and uploads your evaluation code
3. Waits for build — polls evaluator status until ACTIVE (timeout: 10 minutes)
4. Creates dataset — uploads your `svgbench_dataset.jsonl`
5. Launches RFT job — starts reinforcement fine-tuning with your evaluator

### Configuration & Troubleshooting

**Training Parameters**: We use Eval Protocol's default values for training parameters (batch size, epochs, learning rate, LoRA rank, accelerator count, etc.). For a complete list of available RFT flags you can customize, see [Fireworks RFT Command Documentation](/tools-sdks/firectl/commands/reinforcement-fine-tuning-job-create).

**Changing Evaluators**: If you've made changes to your evaluator code and want to upload a new version:

```bash theme={null}
eval-protocol create rft \
  --base-model accounts/fireworks/models/qwen3-4b \
  --chunk-size 10 \
  --force
```

**Evaluator Upload Timing Out**: If your evaluator takes longer than 10 minutes to build, you'll see:

```
⏰ Timeout after 10.0m - evaluator is not yet ACTIVE

❌ Evaluator is not ready within the timeout period.
📊 Please check the evaluator status at: https://app.fireworks.ai/dashboard/evaluators/test-svgagent-test-svg-generation-evaluation
   Wait for it to become ACTIVE, then run 'eval-protocol create rft' again.
```

In this case, monitor the evaluator upload at the link, and run the command again when ACTIVE.

## 4. Monitor Training Progress

After successful job creation, you'll see:

```
✅ Created Reinforcement Fine-tuning Job
   name: accounts/pyroworks/reinforcementFineTuningJobs/sdnld4yn

📊 Dashboard Links:
   Evaluator: https://app.fireworks.ai/dashboard/evaluators/test-svgagent-test-svg-generation-evaluation
   Dataset:   https://app.fireworks.ai/dashboard/datasets/svgbench-dataset
   RFT Job:   https://app.fireworks.ai/dashboard/fine-tuning/reinforcement/sdnld4yn
```

Click on the **RFT Job** link to view real-time training progress, epoch counts, and rollout data.

### Training Results

After successful training, you should see performance improvements reflected in the training metrics:

<img alt="SVG Agent Training Progress" />

### SVG Quality Improvement

You can inspect individual rollouts to see the dramatic improvement in SVG generation quality. Below is a comparison between the first epoch and the final 8th epoch:

**Before (1st Epoch):**

<img alt="SVG Generation - Before Training" />

**After (8th Epoch):**

<img alt="SVG Generation - After Training" />

The reinforcement fine tuning process significantly improves the model's ability to generate accurate, detailed SVG graphics that better match the input descriptions.

## Debugging Tips

When your training is running, you have several powerful tools to debug and monitor your rollouts:

### Rollout Overview

Clicking on any **Epoch** or **Step** in the training dashboard, then clicking the **table icon** to the right, will show you a comprehensive table of all rollouts. It's a good high-level overview to see if any rollouts failed and for what reason.

<img alt="Rollout Overview Table" />

### Individual Rollout Details

If you click on a specific row in the rollout table, you can see exactly what the prompt was and how the model responded. You can even copy and paste out the SVG code generated and render it yourself to see what the model did. This is how we got the results above in the before and after comparison.

<img alt="Individual Rollout Details" />

### Live Log Streaming

Clicking on **View Logs** takes you to a page of logs being streamed in. Here, you can see precisely what errors are happening to the rollouts. This is useful to debug and fix any issues with your rollouts.

<img alt="Live Log Streaming" />

## Next steps

<CardGroup>
  <Card title="Customize training" icon="terminal" href="/fine-tuning/reinforcement-fine-tuning-models#cli">
    Learn all CLI options to customize your training parameters
  </Card>

  <Card title="Try a single-turn example" icon="laptop-code" href="/fine-tuning/quickstart-math">
    Train models with  Python evaluators for simpler tasks
  </Card>

  <Card title="Learn RFT concepts" icon="brain" href="/fine-tuning/reinforcement-fine-tuning-models">
    Understand how reinforcement fine-tuning works
  </Card>
</CardGroup>

## Additional resources

* [Discord Server](https://discord.gg/mMqQxvFD9A) - Come talk to us in the #eval-protocol channel!
* [Eval Protocol Documentation](https://evalprotocol.io/introduction)
* [Remote Rollout Processor Tutorial](https://evalprotocol.io/tutorial/remote-rollout-processor)
* [SVGBench Dataset](https://github.com/johnbean393/SVGBench) - The original benchmark this project is based on

## Appendix

### How Remote Rollout Processing Works

Eval Protocol enables **reinforcement learning that meets you where you are**. Instead of forcing you to rewrite your agent in a specific framework, you can implement a lightweight remote server wherever your codebase and infrastructure already live.

Your remote server is only responsible for:

* **Executing rollouts** - Run your agent logic (in this case, SVG generation from text prompts)
* **Logging to tracing** - Send structured logs to `tracing.fireworks.ai` for evaluation (see the below linked docs for more information)

In this example, we showcase a **Vercel TypeScript server** that executes single-turn SVG code generation.

> **📖 Learn More**: For a complete deep-dive into Remote Rollout Processing, see the [Remote Rollout Processor Tutorial](https://evalprotocol.io/tutorial/remote-rollout-processor).

### Local Development Server

```bash theme={null}
cd vercel_svg_server_ts
vercel dev
```

Then swap out the `remote_base_url` to point to the local server you just started:

```
rollout_processor=RemoteRolloutProcessor(
    remote_base_url="http://localhost:3000",
)
```

And in a third terminal, run the evaluation:

```bash theme={null}
ep local-test
```

> See [Vercel CLI documentation](https://vercel.com/docs/cli/dev) for more information on local development.


# Reinforcement Fine-Tuning
Source: https://docs.fireworks.ai/fine-tuning/reinforcement-fine-tuning-models

Train models using reinforcement learning in minutes

Fireworks RFT helps you train frontier models like DeepSeek V3 and Kimi K2 to **outperform closed models for your product use case, using reinforcement learning.** Fireworks RFT is powerful and easy to use for developers and enterprises:

* **No infrastructure:** Train frontier models without managing GPUs or RL infra
* **Production-ready:** Built-in tracing, monitoring, security & one-click deploy
* **Fast iteration:** From evaluator setup to deployed model in hours, not weeks

<Tip>
  See how [Genspark](https://fireworks.ai/blog/genspark) and [Vercel](https://fireworks.ai/blog/vercel) used Fireworks RFT to train open models for agentic use cases, outperforming leading closed models.
</Tip>

## Quickstart: pick your training approach

<CardGroup>
  <Card title="Single-Turn Training" icon="laptop-code" href="/fine-tuning/quickstart-math">
    **15 minutes** — test locally with a simple evaluator and small model.
  </Card>

  <Card title="Remote Agents" icon="server" href="/fine-tuning/quickstart-svg-agent">
    **1–2 hours** — multi-turn rollouts in your environment with HTTP tracing.
  </Card>

  <Card title="Training security (BYOB)" icon="shield-check" href="/fine-tuning/finetuning-intro#training-security">
    **2–4 hours** — datasets stay in your GCS/S3 bucket.
  </Card>
</CardGroup>

## How RFT works

In supervised fine-tuning you provide labeled examples of good outputs. In RFT you provide **prompts** and an **evaluator** (reward function) that scores model outputs from 0 to 1. Training iteratively improves outputs that maximize that score.

<Steps>
  <Step title="Design an evaluator">
    Define how to score outputs (rules, tests, LLM-as-judge, or hybrid).
  </Step>

  <Step title="Prepare a dataset">
    JSONL with prompts (`messages` array). Ground-truth completions are optional.
  </Step>

  <Step title="Connect your agent (optional)">
    Local evaluators for simple tasks, or [remote environments](/fine-tuning/connect-environments) for agents.
  </Step>

  <Step title="Launch training">
    Fireworks UI, `firectl rftj`, or [eval-protocol CLI](https://evalprotocol.io/introduction).
  </Step>

  <Step title="Deploy">
    Promote the checkpoint and deploy LoRA on [on-demand hardware](/fine-tuning/deploying-loras).
  </Step>
</Steps>

RFT fits when the task is **verifiable** (you can judge output quality), you lack golden completions, or the task needs multi-step reasoning.

## Launch and operate

Use the docs for path choice and quickstarts. Use the **[Fireworks training skill](https://github.com/fw-ai/cookbook/tree/main/skills/fireworks-training)** for launch flags, validation, monitoring, and troubleshooting.

| Topic                                                | Skill reference                                                                                                                                      |
| ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| Prerequisites, validation, job states, CLI/UI launch | [managed-rft-operations](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/managed-rft-operations.md)                 |
| Evaluator design and preference data                 | [preference-data-and-evaluators](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/preference-data-and-evaluators.md) |
| Remote agent tracing                                 | [rft-agent-tracing](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rft-agent-tracing.md)                           |
| Parameter tuning and method choice                   | [choose-method](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/choose-method.md)                                   |
| Failures and stuck jobs                              | [error-reference](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/error-reference.md)                               |
| Deploy and prove serving                             | [deploy-and-troubleshoot](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/deploy-and-troubleshoot.md)               |

<h3>
  Prerequisites
</h3>

Confirm account access and quota, an RFT-compatible base model, valid prompt JSONL, and a reviewed evaluator that produces non-flat scores on a small probe.

<h3>
  Evaluators
</h3>

Evaluators turn a rollout into a reward. Test full-credit, partial-credit, zero, malformed-output, and edge cases locally before registration. Keep evaluator and preference-data guidance in the [evaluator skill reference](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/preference-data-and-evaluators.md).

<h3>
  Parameter tuning
</h3>

Resolve the current defaults from `firectl rftj create --help` and the live RFT parameter reference. Start with a small run, change one optimization or rollout variable at a time, and compare reward and held-out quality.

<h4>
  Weighted training
</h4>

Use per-example or per-token weighting only when the selected managed contract supports it. Inspect the rendered samples and effective loss mask before launch.

<h3>
  Launch from the CLI
</h3>

Use `firectl rftj create` for a reproducible managed job. Prepare the full base-model, dataset, evaluator, and output-model resource names, run the documented preflight, and confirm the resolved plan before creation.

<h3>
  Launch from the dashboard
</h3>

Use the dashboard for a guided first run: choose the model, dataset, evaluator, rollout settings, and optional W\&B project, then review the summary before launch.

<h3>
  Warm start
</h3>

Warm-start from a compatible promoted LoRA when continuing SFT into RFT. Do not also supply a conflicting base-model field; verify the current CLI contract before creation.

<h3>
  Evaluator secrets
</h3>

Store credentials in the Fireworks secret-management surface and reference them by environment-variable name. Never embed secret values in evaluator code, datasets, or run reports.

<h3>
  Cost planning
</h3>

Estimate rollout inference, training runtime or tokens, candidate count, response length, and any deployment uptime. Use the live pricing page and [Price vs Tinker](/fine-tuning/multi-turn-cost-comparison); do not assume a free tier.

<h3>
  Monitor training
</h3>

Track state plus actual progress: optimizer steps, rollout throughput, reward distribution, validation results, logs, and W\&B metrics when enabled. Inspect representative rollouts for reward hacking and stop when quality regresses or the evaluator saturates.

<Note>
  Using Cursor, Claude Code, or Codex? [Install Agent Skills](/fine-tuning/agent/use-with-coding-agents) to drive managed RFT with the `fireworks-training` skill.
</Note>

Parameter field reference: [RFT parameters](/fine-tuning/rft-parameters-reference) (API tab).

## Related

<CardGroup>
  <Card title="Remote Environment Setup" icon="server" href="/fine-tuning/connect-environments">
    HTTP `/init` contract for remote rollouts
  </Card>

  <Card title="Price vs Tinker" icon="scale-balanced" href="/fine-tuning/multi-turn-cost-comparison">
    Multi-turn RL cost comparison
  </Card>

  <Card title="Training API RL" icon="code" href="/fine-tuning/training-api/cookbook/rl">
    Custom RL loops on dedicated or serverless infrastructure
  </Card>

  <Card title="Models" icon="table" href="/fine-tuning/models">
    Which models support managed RFT
  </Card>
</CardGroup>


# RFT parameters reference
Source: https://docs.fireworks.ai/fine-tuning/rft-parameters-reference

Checkpoint, resume, and GRPO metrics fields for reinforcement fine-tuning recipes.

Use this page for training **checkpoint and resume** knobs and **GRPO metric interpretation** that are easy to miss when running reinforcement fine-tuning (RFT) and cookbook-driven training. For sampling and optimization hyperparameters (learning rate, epochs, temperature, KL targets, etc.), see [Parameter tuning](/fine-tuning/reinforcement-fine-tuning-models#parameter-tuning).

The canonical cookbook reference for save, resume, and promote is [Checkpoints and Resume](/fine-tuning/training-api/cookbook/reference#checkpoints). Low-level SDK APIs are documented in [Saving and loading](/fine-tuning/training-api/dedicated#saving-and-loading).

## `dcp_save_interval`

Controls how often full training state (weights **and** optimizer) is checkpointed using DCP (Distributed Checkpoint) format.

| Property                                      | Value                                                                                                                                                                                                  |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Type**                                      | `integer`                                                                                                                                                                                              |
| **Default**                                   | `0` (disabled)                                                                                                                                                                                         |
| **Typical config — SFT / RL / DPO cookbooks** | `WeightSyncConfig(dcp_save_interval=N)` on the recipe `Config` (see [Cookbook: RL](/fine-tuning/training-api/cookbook/rl) and [Checkpoints](/fine-tuning/training-api/cookbook/reference#checkpoints)) |

When set to `0` (the default), no periodic DCP checkpoints are written for resume. Only sampler and HuggingFace-format weight snapshots may be produced — these preserve model weights but **not optimizer state**.

When set to a positive integer `N`, a full DCP checkpoint is written every `N` steps.

**Why this matters:** If a training job is interrupted, optimizer state is lost unless `dcp_save_interval` is set. The model resumes from the last checkpoint, but the optimizer re-initializes from scratch — which can affect training stability and effective learning rate.

### Example (cookbook `Config`)

```python theme={null}
from training.recipes.rl_loop import Config, main
from training.utils import WeightSyncConfig

cfg = Config(
    log_path="./grpo_logs",
    base_model="accounts/fireworks/models/qwen3-8b",
    # ... other fields ...
    weight_sync=WeightSyncConfig(dcp_save_interval=50),  # full checkpoint every 50 steps
    # ...
)
main(cfg)
```

Some internal or forked recipes may expose the same interval on a nested config type (for example a weight sync block). The field name is always `dcp_save_interval`; see your recipe’s `Config` dataclass for the exact attribute path.

### Job recovery and preemption

For transient control-plane or worker interruptions, the trainer job manager exposes [`reconnect_and_wait`](/fine-tuning/training-api/reference/trainer-job-manager) so your driver can wait for a resumable state and resume cleanly.

<Note>
  `load_state_with_optimizer()` only restores optimizer state from DCP-format checkpoints. If you point it at an HF or sampler snapshot, optimizer state silently won't be restored. Always load from the path returned by `save_state()` when you need full optimizer restore. See [Saving and loading](/fine-tuning/training-api/dedicated#saving-and-loading).
</Note>

***

## Metrics reference

### `ppo_kl` vs `ref_kld`

GRPO training logs two KL divergence metrics that measure different things:

| Metric    | What it measures                                                                                                        | Expected behavior                                                                     |
| --------- | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `ppo_kl`  | KL between the **current policy** and the **previous policy** (importance-sampling ratio inside the PPO clip objective) | Stays near `0` with one minibatch per rollout — this is correct, not a bug            |
| `ref_kld` | KL between the **current policy** and the **reference (base) model**                                                    | Starts near `0`, increases gradually as the policy diverges from base during training |

**Which one to monitor:** `ref_kld` is the metric to watch for policy drift. A sudden large jump in `ref_kld` may indicate reward hacking or that the KL penalty coefficient needs tuning.

The cookbook does not always surface `ref_kld` by default. To add it, you can use the `k3` unbiased estimator:

```python theme={null}
ref_kld = (ref_logp - policy_logp).exp() - (ref_logp - policy_logp) - 1
```


# RL Rollouts with Your Own Trainer
Source: https://docs.fireworks.ai/fine-tuning/rl-rollout-integration

Integrate an external RL trainer with Fireworks inference: hot-load new checkpoints from your bucket and run rollouts via the OpenAI-compatible API.

<Warning>
  **Early Access Feature.** External-bucket hot-load for RL rollouts is a
  private preview. [Contact Fireworks](https://fireworks.ai/contact) to enable
  this path on your account before you use `S3`, `MINIO`, `NEBIUS`, or similar
  non-`FW_HOSTED` storage.
</Warning>

<Tip>
  **Using a code agent?** Follow sections in order: [Prerequisites](#prerequisites)
  → [Quickstart checklist](#quickstart-checklist) → [Hot-load API](#hot-load-api).
  Required env: `FIREWORKS_API_KEY`. After your first full snapshot is serving,
  read [Incremental snapshots (skill)](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-hotload.md#incremental-snapshots-arc2) before
  production training loops. For active stream, session ID, and `reset_prompt_cache`
  semantics, see [KV cache behavior for RL rollouts](/guides/rollout-inference#kv-cache-behavior-for-rl-rollouts).
  For ledger and hot-load status debugging, see [Ledger & debugging (skill)](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-hotload.md).
</Tip>

This guide is for teams already running their own RL trainer (Megatron, TorchTitan, Slime, Verl, etc) who want Fireworks purely for inference during rollouts. It also covers the raw hot-load API, which Training API users can call directly when they need rollout behaviors beyond what `WeightSyncer` and `DeploymentManager` expose.

Fireworks offers three paths for reinforcement learning, along a spectrum that trades convenience for control:

* **BYOT rollouts** — the most hands-on, for teams already running their own trainer who want Fireworks purely for high-scale rollout inference.
* **Training API** — the middle ground, for teams who want to own their training logic without managing infrastructure.
* **Managed RFT** — the turnkey path, for teams who just want a tuned model without running the loop themselves.

## Where this guide fits

| Path                                                         | You own                                                  | Fireworks owns                                                                    | Use this guide?                                               |
| ------------------------------------------------------------ | -------------------------------------------------------- | --------------------------------------------------------------------------------- | ------------------------------------------------------------- |
| **This guide (BYOT rollouts)**                               | Trainer, rewards, environment, checkpoint upload cadence | Hot-load deployment, distributed weight swap, inference, KV cache across rollouts | Yes                                                           |
| [Training API](/fine-tuning/training-api/introduction)       | Training logic (recipes or SDK)                          | GPUs, trainer lifecycle, often `FW_HOSTED` bucket                                 | Only if you need rollout behavior beyond what the SDK exposes |
| [Managed RFT](/fine-tuning/reinforcement-fine-tuning-models) | Dataset and evaluator                                    | End-to-end hosted RL                                                              | No                                                            |

**Why BYOT rollout inference?**

* **Disaggregated:** Your trainer and rollout cluster can run in different regions or clouds; deployments can span multiple regions to pool capacity.
* **Full-parameter or LoRA:** Hot-load full-parameter checkpoints for large models, or hot-load a LoRA adapter on top of a frozen base model—both run on Fireworks inference shapes. See [LoRA rollouts](#lora-rollouts).
* **Fast checkpoint transfer:** Lossless compressed incremental snapshots (`arc_v2`, typically 20×+ compression) over standard object storage—no special RDMA networking between trainer and inference.
* **Async / off-policy friendly:** Background download during rollouts; configurable swap semantics similar in spirit to [PipelineRL](https://arxiv.org/pdf/2509.19128)—see [checkpoint-swap behavior](/fine-tuning/rl-rollout-integration#checkpoint-swap-behavior).

For **Online RL** (live user traffic as rollouts with rolling per-replica updates), the same hot-load infrastructure applies; contact Fireworks for production Online RL setup.

## Supported models

BYOT rollout hot-load is enabled for a curated set of base models from our full [model library](https://fireworks.ai/models). The following are supported today:

| Model         | Base model ID                 |
| ------------- | ----------------------------- |
| Kimi K2.5     | `kimi-k2p5`                   |
| Kimi K2.6     | `kimi-k2p6`                   |
| Kimi K2.7     | `kimi-k2p7-code`              |
| GLM 5.2       | `glm-5p2`                     |
| Qwen3 30B-A3B | `qwen3-30b-a3b-instruct-2507` |

Both full-parameter checkpoints and [LoRA adapters](#lora-rollouts) can be hot-loaded for these models.

<Note>
  **Currently, reach out to the [Fireworks RL team](https://fireworks.ai/contact) to get set up.** We will help provision the deployment shape (GPU, precision, quantization) for your base model and confirm the snapshot format it expects.
</Note>

## Placeholders

Reuse these values in every command below:

| Placeholder                            | Example                                                           |
| -------------------------------------- | ----------------------------------------------------------------- |
| `<account_id>`                         | `my-team`                                                         |
| `<model_id>`                           | `qwen3-30b-a3b`                                                   |
| `<deployment_id>`                      | `rl-rollout-prod`                                                 |
| `<fireworks_api_key>`                  | From [API keys](https://app.fireworks.ai/settings/users/api-keys) |
| `<your_bucket>` / `<your_upload_path>` | Parent prefix configured on the deployment (no trailing slash)    |
| `<checkpoint_id>`                      | Snapshot directory name, e.g. `version_001` (no slashes)          |

## Prerequisites

Complete this checklist before creating a deployment:

1. **Fireworks account** and **API key** — [create a key](https://app.fireworks.ai/settings/users/api-keys) and set `export FIREWORKS_API_KEY="<key>"`.
2. **Account ID** — In the [dashboard](https://app.fireworks.ai/), open your account settings or any resource URL; the account slug is the segment after `/accounts/` (for example `accounts/<account_id>/...`).
3. **Feature enablement** — Request **external-bucket hot-load for RL rollouts** on account `<account_id>`, including your bucket provider (`S3`, `GCS`/`gs://`, or `NEBIUS`).
4. **Object storage read access for Fireworks** — Fireworks needs read-only access to the bucket prefix you will pass as `--hot-load-bucket-url`. At enablement, Fireworks shares the IAM principal to grant access. Typical setup:
   * **Amazon S3:** Grant the Fireworks principal `s3:GetObject` (and `s3:ListBucket` on the prefix) on `s3://<your_bucket>/<your_upload_path>/*`.
   * **Google Cloud Storage:** Grant `roles/storage.objectViewer` on the bucket or prefix to the Fireworks service account provided at onboarding.
   * **Nebius / MinIO:** Equivalent read-only credentials or access key scoped to the upload prefix.
5. **`firectl` installed** — See [firectl](/tools-sdks/firectl/firectl).
6. **Base model and deployment shape** — An RL-capable shape for your model (GPU count, precision). If you omit `--deployment-shape`, `firectl` prompts you to pick one interactively.

## Architecture

```mermaid theme={null}
flowchart LR
  trainer["Your RL Trainer"] -->|"1. Upload checkpoint"| bucket[("External bucket")]
  trainer -->|"2. Signal snapshot ready"| api["Fireworks Hot-Load API"]
  api -->|"3. Load weights"| deployment["Inference Deployment"]
  trainer -->|"4. Rollout via /v1/completions"| deployment
  deployment -->|"Tokens + optional routing_matrix"| trainer
```

**You own:** trainer, reward shaping, checkpoint cadence, rollout orchestration.

**Fireworks owns:** hot-load logistics, distributed weight swap, inference serving, KV cache across rollouts.

## End-to-end loop

1. Create a hot-load deployment.
2. Upload and hot-load an initial **full** snapshot.
3. Run rollouts against that snapshot.
4. For each training step: upload and hot-load the next **incremental** snapshot (see [Incremental snapshots](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-hotload.md)).
5. Run rollouts again.
6. Every 20th or 30th step, publish a **full** snapshot instead of an incremental one. If the incremental chain fails, fall back to a full snapshot.

## Quickstart checklist

Use this table for your **first** rollout end-to-end:

| Step | Action                                                                     | Done when                                                                                   |
| ---- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| 1    | [Create hot-load deployment](#1-create-a-hot-load-deployment)              | `firectl deployment get <deployment_id>` shows a healthy deployment                         |
| 2    | [Upload full HF snapshot](#2-upload-and-hot-load-an-initial-full-snapshot) | All files exist under `.../<checkpoint_id>/` in object storage                              |
| 3    | `POST` [signal snapshot](#hot-load-api)                                    | HTTP 200                                                                                    |
| 4    | `GET` [poll status](#hot-load-api)                                         | Every replica has `readiness: true` and `current_snapshot_identity` matches your `identity` |
| 5    | [Run rollouts](#3-run-rollouts)                                            | Chat/completions returns tokens                                                             |

## 1. Create a hot-load deployment

Create the deployment that will serve rollouts. During preview, `--enable-hot-load` flags may be hidden from CLI help but can still be passed explicitly.

```bash theme={null}
firectl create deployment accounts/<account_id>/models/<model_id> \
  --deployment-shape <shape_name> \
  --deployment-id <deployment_id> \
  --enable-hot-load \
  --hot-load-bucket-type S3 \
  --hot-load-bucket-url s3://<your_bucket>/<your_upload_path> \
  --hot-load-transition-type ASYNC \
  --region US_OHIO_1
```

<Note>
  **Flags**

  * `--deployment-shape` — Optional. If omitted, `firectl` prompts you to pick one.
  * `--hot-load-bucket-type` — `MINIO`, `S3`, `NEBIUS`, or `FW_HOSTED`. This guide focuses on external buckets (`S3`, `gs://`, etc.). `FW_HOSTED` is for Fireworks-managed trainers.
  * `--hot-load-bucket-url` — Required when `--enable-hot-load` is set. Examples: `s3://mybucket/path`, `gs://mybucket/path`. **No trailing slash.** This is the **parent prefix**; each snapshot is a subdirectory named by `identity` (see [snapshot layout](#snapshot-layout)).
  * `--hot-load-transition-type` — `ASYNC` (recommended for RL) or `SYNC`. Defaults to `ASYNC` when hot load is enabled. See [checkpoint-swap behavior](/fine-tuning/rl-rollout-integration#checkpoint-swap-behavior).
  * `--region` — Where the deployment runs (for example `US_OHIO_1`, `US_VIRGINIA_1`). Keep the trainer upload path geographically close to the bucket and deployment.
</Note>

Save the **account ID**, **deployment ID**, and **model ID** from the output for hot-load and rollout calls.

If you do not set a shape, the CLI shows a shape picker:

<Frame>
  <img alt="firectl deployment shape picker" />
</Frame>

## 2. Upload and hot-load an initial full snapshot

Upload a full HuggingFace-format checkpoint, then signal Fireworks to load it.

### Snapshot layout

Place each snapshot under its own subdirectory. The `identity` you signal in the API must match the directory name (a single path segment—no slashes):

```
s3://<your_bucket>/<your_upload_path>/<checkpoint_id>/
├── config.json                   # HF model config (must match the base model)
├── tokenizer.json                # tokenizer files (same as the base model)
├── tokenizer_config.json
├── model.safetensors.index.json  # weight name → shard file map
├── model.weight.spec.json        # weight name → {shape, dtype} for every tensor
├── model-00000.safetensors       # weights, one layer per file
├── model-00001.safetensors
└── ...
```

Example with the recommended path pattern:

```
s3://<your_bucket>/<account_id>/<account_id>-<deployment_id>/version_001/
```

* **`identity` / `<checkpoint_id>`** — Any opaque string (for example `version_001` or `step_00100`).
* **Format** — Same layout as the base model on HuggingFace, plus the two manifest files described below. **No tensor-parallel sharding** in uploaded files.
* **File size** — Split weights into multiple `.safetensors` files, each under about 5 GB. **One layer per file** is required (a single shard must not mix weights from more than one layer) and also minimizes load time.

#### Required files (full snapshot)

A full (non-LoRA) snapshot is validated at POST time; it must contain all of:

| File                           | Purpose                                                                | Validation                                                                                                                                                                                                                               |
| ------------------------------ | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `config.json`                  | HuggingFace model config.                                              | Must be loadable as an `AutoConfig` and **equivalent to the base model's** config (`hidden_size`, `num_hidden_layers`, `rope_parameters`, etc.). A `quantization_config` key is allowed for [quantized snapshots](#quantized-snapshots). |
| `model.safetensors.index.json` | Maps each tensor name to the shard file that stores it (`weight_map`). | Must be a JSON object with a `weight_map`; every shard file may contain weights from only **one** layer.                                                                                                                                 |
| `model.weight.spec.json`       | Describes each tensor's `shape` and `dtype` (`tensor_map`).            | Must be a JSON object with a `tensor_map` that covers **every** weight named in `weight_map`.                                                                                                                                            |
| `model-*.safetensors`          | The weights themselves.                                                | Tensor coverage must match the index/spec; tensors must cover the base model.                                                                                                                                                            |
| tokenizer files                | `tokenizer.json`, `tokenizer_config.json`, etc.                        | Carried over from the base model.                                                                                                                                                                                                        |

`model.safetensors.index.json` (HuggingFace-standard) maps tensors to shards:

```json theme={null}
{
  "weight_map": {
    "model.layers.0.self_attn.q_proj.weight": "model-00001.safetensors",
    "model.layers.0.self_attn.k_proj.weight": "model-00001.safetensors"
  }
}
```

`model.weight.spec.json` gives each tensor's metadata (used to validate transitions and dequantization):

```json theme={null}
{
  "tensor_map": {
    "model.layers.0.self_attn.q_proj.weight": {
      "shape": [2048, 2048],
      "dtype": "float16"
    }
  }
}
```

<h4>
  Incremental snapshots and ARC2
</h4>

An [incremental snapshot](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-hotload.md) is an ARC2-compressed **delta** of the safetensors against a `previous_snapshot_identity` already on the deployment. It keeps the **same `model.safetensors.index.json`** as its parent — the `weight_map`, the file count, and the per-file weight set must be **identical** (only the tensor contents change). Tensor `dtype`s must also match across the transition. Upload only the diff `.safetensors` (plus the unchanged manifests/config) under the new `identity`; signal it with `incremental_snapshot_metadata`. See [Incremental snapshots](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-hotload.md) for the full body and the delta-build utilities.

Optional: call the [per-file hint API](/fine-tuning/rl-rollout-integration#per-file-hints-optional) as each file lands to speed up loading on large models.

### Quantized snapshots

Snapshot **precision must be in a format the shape's loader accepts**, so the exact target format is **shape-dependent** — confirm it with the Fireworks RL team during shape provisioning (see [Supported models](#supported-models)). You can upload weights in the base precision (BF16) and let the shape convert them at load time, or **pre-quantize** in the snapshot to cut upload size and weight-swap time.

For large MoE models such as **GLM** and **Kimi**, the routed MoE expert weights can be **pre-quantized to FP8** in the uploaded snapshot. When you pre-quantize:

* Add the matching `quantization_config` block to `config.json`.
* Make sure `model.weight.spec.json` (`tensor_map`) describes the quantized tensors (`shape` + `dtype`); the snapshot is rejected if a `quantization_config` is present but the spec has no dequantizable tensors or is missing the metadata needed to dequantize them.
* Keep the quantization recipe consistent across every snapshot in a run so the [incremental chain](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-hotload.md) stays valid (dtypes must not change between a snapshot and its parent).

### Signal and poll

Use the [Hot-load API](#hot-load-api) below with `{ "identity": "<checkpoint_id>" }` and poll until all replicas are ready.

## Hot-load API

All hot-load requests use these headers:

| Header                 | Value                                               |
| ---------------------- | --------------------------------------------------- |
| `Authorization`        | `Bearer <fireworks_api_key>`                        |
| `fireworks-model`      | `accounts/<account_id>/models/<model_id>`           |
| `fireworks-deployment` | `accounts/<account_id>/deployments/<deployment_id>` |
| `Content-Type`         | `application/json`                                  |

| Operation                | Method | URL                                                         |
| ------------------------ | ------ | ----------------------------------------------------------- |
| Signal snapshot ready    | `POST` | `https://api.fireworks.ai/hot_load/v1/models/hot_load`      |
| Poll load status         | `GET`  | `https://api.fireworks.ai/hot_load/v1/models/hot_load`      |
| Per-file hint (optional) | `POST` | `https://api.fireworks.ai/hot_load/v1/models/hot_load/hint` |

<h3>
  Per-file hints (optional)
</h3>

For large full-parameter snapshots, notify the hint endpoint as each shard finishes uploading. Hints let replicas begin validating and staging available files before the final snapshot-ready signal. They do not replace the final signal or readiness poll.

### Signal snapshot ready

**Full snapshot** body:

```json theme={null}
{ "identity": "version_001" }
```

**Incremental snapshot** bodies, compression, hints, and `checksum_format` are documented in [Incremental snapshots](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-hotload.md).

<ParamField type="string">
  Snapshot directory name under the configured bucket prefix. Must not contain `/`.
</ParamField>

<ParamField type="object">
  Required for incremental snapshots. Includes `previous_snapshot_identity`, `compression_format` (`arc_v2`), and `checksum_format` (`alder32`). See the incremental snapshots guide.
</ParamField>

<ParamField type="string">
  Prompt-cache policy after the swap: `all` (default), `none`, or `new_session`. See [KV cache behavior for RL rollouts](/guides/rollout-inference#kv-cache-behavior-for-rl-rollouts) for active stream, session ID, and reset-option semantics.
</ParamField>

<ParamField type="string[]">
  Top-level `config.json` fields to ignore during snapshot validation. Only use for known-safe metadata fields.
</ParamField>

```bash theme={null}
curl -X POST https://api.fireworks.ai/hot_load/v1/models/hot_load \
  -H "Authorization: Bearer <fireworks_api_key>" \
  -H "fireworks-model: accounts/<account_id>/models/<model_id>" \
  -H "fireworks-deployment: accounts/<account_id>/deployments/<deployment_id>" \
  -H "Content-Type: application/json" \
  -d '{ "identity": "version_001" }'
```

```python theme={null}
import os
import requests

API_KEY = os.environ["FIREWORKS_API_KEY"]
ACCOUNT = "<account_id>"
MODEL = f"accounts/{ACCOUNT}/models/<model_id>"
DEPLOYMENT = f"accounts/{ACCOUNT}/deployments/<deployment_id>"
HEADERS = {
    "Authorization": f"Bearer {API_KEY}",
    "fireworks-model": MODEL,
    "fireworks-deployment": DEPLOYMENT,
    "Content-Type": "application/json",
}

resp = requests.post(
    "https://api.fireworks.ai/hot_load/v1/models/hot_load",
    headers=HEADERS,
    json={"identity": "version_001"},
    timeout=60,
)
resp.raise_for_status()
```

### Poll load status

Poll until **every** replica has `readiness: true` and `current_snapshot_identity` equals the `identity` you signaled.

```bash theme={null}
curl https://api.fireworks.ai/hot_load/v1/models/hot_load \
  -H "Authorization: Bearer <fireworks_api_key>" \
  -H "fireworks-model: accounts/<account_id>/models/<model_id>" \
  -H "fireworks-deployment: accounts/<account_id>/deployments/<deployment_id>"
```

```python theme={null}
status = requests.get(
    "https://api.fireworks.ai/hot_load/v1/models/hot_load",
    headers=HEADERS,
    timeout=30,
).json()

replicas = status.get("replicas", [])
ready = (
    replicas
    and all(r.get("readiness") for r in replicas)
    and all(r.get("current_snapshot_identity") == "version_001" for r in replicas)
)
```

<h3>
  Checkpoint swap behavior
</h3>

A swap changes which policy serves new rollout work. Wait for every replica to report the requested identity for strict on-policy collection. For bounded off-policy collection, record the policy version returned by inference and apply the reviewed admission rule before training on that rollout.

<h4>
  Async transition (recommended default for RL)
</h4>

Async transition keeps rollout capacity available while replicas adopt the new snapshot. Existing generations may finish across the transition, so preserve policy-version metadata and validate trainer/inference numerics. Use a synchronized transition only when the workflow requires every in-flight generation to finish on one policy version.

### When to start rollouts

* **Default (on-policy):** Wait until all replicas report readiness on the new `identity`.
* **Off-policy / higher utilization:** You may start sending rollouts when a **subset** of replicas is ready—inspect each entry in `replicas` in the `GET` response. Stale-policy rollouts are expected; use async transition mode and monitor policy version in streaming responses (see [Policy version in responses](/guides/rollout-inference#policy-version-in-responses)).

Per-file hints are optional but recommended for large checkpoints—see [Incremental snapshots](/fine-tuning/rl-rollout-integration#per-file-hints-optional).

## 3. Run rollouts

Call the OpenAI-compatible inference API. For multi-turn RL, set session headers so KV cache stays on one replica:

```bash theme={null}
curl https://api.fireworks.ai/inference/v1/chat/completions \
  -H "Authorization: Bearer <fireworks_api_key>" \
  -H "fireworks-model: accounts/<account_id>/models/<model_id>" \
  -H "fireworks-deployment: accounts/<account_id>/deployments/<deployment_id>" \
  -H "x-multi-turn-session-id: <trajectory_id>" \
  -H "x-session-affinity: <trajectory_id>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "accounts/<account_id>/models/<model_id>",
    "messages": [{"role": "user", "content": "..."}]
  }'
```

See [Inference for RL rollouts](/guides/rollout-inference) for session affinity, weight-swap behavior, MoE Router Replay (R3), and policy-version fields.

## Steady-state training loop

After the first full snapshot:

1. **Intermediate steps** — Build and upload an [incremental snapshot](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-hotload.md) (`arc_v2`), signal with `incremental_snapshot_metadata`, poll until ready, then run rollouts.
2. **Every 20th or 30th step** — Publish a new **full** snapshot for faster recovery and chain reset.
3. **On failure** — Fall back to a full snapshot; see [Ledger and debugging](#ledger-debugging).

Brief incremental signal example (full details on the incremental page):

```bash theme={null}
curl -X POST https://api.fireworks.ai/hot_load/v1/models/hot_load \
  -H "Authorization: Bearer <fireworks_api_key>" \
  -H "fireworks-model: accounts/<account_id>/models/<model_id>" \
  -H "fireworks-deployment: accounts/<account_id>/deployments/<deployment_id>" \
  -H "Content-Type: application/json" \
  -d '{
    "identity": "version_002",
    "incremental_snapshot_metadata": {
      "previous_snapshot_identity": "version_001",
      "compression_format": "arc_v2",
      "checksum_format": "alder32"
    }
  }'
```

## LoRA rollouts

Rollouts work with both **full-parameter** and **LoRA** checkpoints. With LoRA you hot-load only the adapter on top of a frozen base model: snapshots are tiny (tens of MB instead of tens of GB), weight swaps are near-instant, and the deployment applies the adapter at request time. This is a good fit for rapid RL iteration and for serving several adapter variants from one base deployment.

The flow is the same as the [end-to-end loop](#end-to-end-loop) above—create a hot-load deployment, upload a snapshot, signal it, poll, and run rollouts—with the differences below.

<Note>
  LoRA rollouts run on a LoRA-capable RL/hot-load shape (adapter serving enabled on the base-model deployment). Confirm the shape for your base model with Fireworks during [feature enablement](#prerequisites).
</Note>

### Auto-detection

You do **not** set a flag to choose LoRA vs full-parameter. Fireworks classifies each snapshot from its contents: a directory containing `adapter_config.json` is loaded as a LoRA adapter; anything else is treated as a full-parameter snapshot. The [Hot-load API](#hot-load-api) is identical for both.

### LoRA snapshot layout

Upload a HuggingFace / PEFT-format adapter under the snapshot `identity` directory (same bucket parent as the [full snapshot layout](#snapshot-layout)):

```
s3://<your_bucket>/<your_upload_path>/<checkpoint_id>/
├── adapter_config.json
└── adapter_model.safetensors
```

* **`adapter_config.json`** — PEFT adapter config. Its presence is what marks the snapshot as LoRA; it must reference the same base model the deployment serves.
* **`adapter_model.safetensors`** — adapter weights. Sharded `adapter_model-*.safetensors` and the legacy `adapter_model.bin` are also accepted.

### Signal and poll

Signal exactly like a full snapshot—just the `identity`:

```bash theme={null}
curl -X POST https://api.fireworks.ai/hot_load/v1/models/hot_load \
  -H "Authorization: Bearer <fireworks_api_key>" \
  -H "fireworks-model: accounts/<account_id>/models/<model_id>" \
  -H "fireworks-deployment: accounts/<account_id>/deployments/<deployment_id>" \
  -H "Content-Type: application/json" \
  -d '{ "identity": "adapter_step_001" }'
```

When [polling load status](#poll-load-status), LoRA deployments report progress per adapter in a `loaded_adapters` array (each entry has an `identity` and a `status`) in addition to the single `current_snapshot_identity` used for full-parameter snapshots. Treat the snapshot as ready when your `identity` appears with `status: "loaded"` on **every** replica.

### No incremental chain

LoRA adapters are small, so there is **no ARC2 incremental/delta chain** for LoRA. Upload the full adapter every step—each LoRA snapshot is complete and self-contained. The [incremental snapshot](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-hotload.md) workflow (and the "every 20th–30th step, publish a full snapshot" cadence) applies only to full-parameter checkpoints.

## Numerics alignment

For best training–inference alignment:

* Match **quantization / precision** between trainer checkpoints and the deployment shape (work with Fireworks if you need a custom shape).
* Measure **logprob divergence** between trainer forward passes and rollout inference on the same tokens.
* For MoE models, use **Router Replay (R3)** during rollouts—see [MoE Router Replay](/guides/rollout-inference#moe-router-replay).

<h2>
  Ledger and debugging
</h2>

The control-plane checkpoint list is the source of truth for snapshot identity, type, creation time, and promotability. When a hot-load stalls:

1. Confirm the deployment is attached to the trainer or bucket scope that produced the snapshot.
2. Compare the requested identity with every replica's reported current identity.
3. For full-parameter chains, retry with a full base snapshot if a delta parent is missing.
4. For LoRA, fix a stale deployment attachment; LoRA snapshots are complete adapters and do not use a delta chain.

Detailed scope mismatch, checkpoint retention, promotion, and recovery procedures live in the [hot-load skill reference](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-hotload.md).

## Next steps

<CardGroup>
  <Card title="Incremental snapshots & ARC2" icon="layer-group" href="https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-hotload.md">
    Delta builds, per-file hints, and incremental signal bodies
  </Card>

  <Card title="Ledger & debugging" icon="bug" href="https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-hotload.md">
    Snapshot history, stuck swaps, and recovery playbooks
  </Card>

  <Card title="Inference for RL rollouts" icon="bolt" href="/guides/rollout-inference">
    Session affinity, policy version, MoE Router Replay (R3)
  </Card>

  <Card title="Training API" icon="flask" href="/fine-tuning/training-api/introduction">
    Fireworks-hosted trainer alternative
  </Card>
</CardGroup>

<Note>
  **Operational depth** (ledger, ARC2 chains, hot-load errors) lives in the [Fireworks training skill — rl-hotload](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-hotload.md). This page is the BYOT happy path.
</Note>


# Thinking history in fine-tuning
Source: https://docs.fireworks.ai/fine-tuning/thinking-history

Choose how historical reasoning is rendered for managed fine-tuning and match training behavior to inference.

Thinking models can store an assistant turn's reasoning in `reasoning_content`. A thinking-history mode controls whether that existing reasoning remains visible in the context of later turns.

This setting does **not** enable or disable thinking generation. It only controls how reasoning already present in the dataset is rendered. Thinking enablement, context-length splitting, and truncation are separate concerns.

## Interleaved and Preserved

* **Interleaved** removes thinking across user-turn boundaries while retaining the complete assistant → tool → assistant trajectory for the current user turn.
* **Preserved** retains thinking across user turns.

These names align with the serving API's `reasoning_history="interleaved"` and `reasoning_history="preserved"` behaviors. Choose the mode that matches how you will replay reasoning history when serving the fine-tuned model.

<Warning>
  Do not interpret Interleaved as "keep only the final assistant's thinking." The boundary is the last real user query. If one user turn contains an assistant tool call, a tool result, and another assistant response, thinking from both assistant messages stays in that turn's trajectory.
</Warning>

## Model capabilities

Models expose one of four capability shapes:

* **Selectable:** the job can explicitly select Interleaved or Preserved.
* **Fixed Interleaved:** only Interleaved is supported.
* **Fixed Preserved:** only Preserved is supported.
* **Automatic:** neither explicit mode describes the model-native behavior; the renderer selects behavior from each dataset row's input shape.

| Base model                | Available modes                | Default                   | Interleaved boundary                                                       | Managed SFT datums                                                    |
| ------------------------- | ------------------------------ | ------------------------- | -------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| GLM 5.1                   | Interleaved, Preserved         | Interleaved               | Last real user query; keeps the complete current assistant/tool trajectory | Both modes unroll per user turn                                       |
| GLM 5.2                   | Interleaved, Preserved         | Interleaved               | Same as GLM 5.1                                                            | Interleaved unrolls; Preserved keeps one complete datum               |
| Qwen 3.5                  | Interleaved only               | Interleaved               | Last real user query; tool results do not reset the boundary               | Unrolls per user turn                                                 |
| Qwen 3.6                  | Interleaved, Preserved         | Interleaved               | Same as Qwen 3.5                                                           | Interleaved unrolls; Preserved keeps one complete datum               |
| Kimi K2.5                 | Interleaved only               | Interleaved               | Keeps thinking in the trailing/current assistant-tool trajectory           | Unrolls per user turn                                                 |
| Kimi K2.6                 | Interleaved, Preserved         | Interleaved               | Same as Kimi K2.5                                                          | Interleaved unrolls; Preserved keeps one complete datum               |
| Kimi K2.7 Code            | Preserved only                 | Preserved                 | Not applicable                                                             | Keeps one complete datum                                              |
| DeepSeek V4 Flash and Pro | Automatic only; no mode toggle | Automatic (`UNSPECIFIED`) | Depends on whether the row declares tools                                  | Without tools, unrolls; with declared tools, keeps one complete datum |

Base models not listed above keep their existing model-default renderer behavior and do not expose an explicit thinking-history choice.

### GLM 5.1 compared with GLM 5.2

Both versions expose the upstream `clear_thinking` template control. Fireworks maps Interleaved to `clear_thinking=true` and Preserved to `clear_thinking=false`. Both default to Interleaved, and both use the last real user query as the clearing boundary.

The difference is in Preserved mode:

* **GLM 5.1 still unrolls.** Its official template can change an earlier assistant turn's rendered think wrapper after a later assistant message is appended. The previously rendered sequence is therefore not always a prefix of the longer conversation.
* **GLM 5.2 keeps one complete datum.** Its Preserved rendering is prefix-stable: appending a later turn does not rewrite tokens already rendered for earlier turns.

GLM 5.1 unrolling does not remove preserved reasoning. It creates one training datum per user turn so that every target is rendered with the correct template context.

### Qwen 3.5 compared with Qwen 3.6

Both versions use the same Interleaved boundary: the last real user query, not the last assistant response. In a user → assistant tool call → tool result → assistant response trajectory, thinking from both assistant messages is retained.

* **Qwen 3.5** has no supported history toggle and is fixed to Interleaved.
* **Qwen 3.6** adds the upstream `preserve_thinking` control. Fireworks maps Interleaved to `preserve_thinking=false` and Preserved to `preserve_thinking=true`. Preserved is prefix-stable and stays one complete datum.

For byte-level parity with the official Qwen 3.6 template, Preserved mode can emit an empty think wrapper for a historical assistant message that has no `reasoning_content`. This does not synthesize reasoning; it preserves the template's exact structure.

### Kimi K2.5 compared with Kimi K2.6 and K2.7 Code

* **Kimi K2.5** is fixed to Interleaved and unrolls per user turn.
* **Kimi K2.6** adds the upstream `preserve_thinking` control. Fireworks maps Interleaved to `preserve_thinking=false` and Preserved to `preserve_thinking=true`; Preserved stays one complete datum.
* **Kimi K2.7 Code** is fixed to Preserved and stays one complete datum.

### DeepSeek V4 automatic behavior

DeepSeek V4 Flash and Pro do not expose an Interleaved/Preserved toggle. Leave the job field omitted or `UNSPECIFIED`.

The model-native renderer makes a deterministic choice for each row:

* A row without declared tools removes thinking across user turns and unrolls per user turn.
* A row with declared tools preserves thinking and remains one complete datum.

This is determined by the row's input shape, not chosen autonomously or stochastically by the model. Render-preview clients should display one read-only automatic option.

## Why some modes unroll

Managed SFT trains all assistant messages. When a renderer removes historical thinking, Fireworks unrolls the conversation at **user boundaries** so every assistant response gets one opportunity to be the current target.

Consider this conversation:

```text theme={null}
U1
A1 (thinking 1, final answer)

U2
A2 (thinking 2, tool call)
Tool result
A3 (thinking 3, final answer)
```

For an Interleaved renderer that unrolls, Fireworks produces:

* Datum 0 trains A1.
* Datum 1 removes thinking 1 from the historical context and trains A2 and A3 together. Thinking 2 and thinking 3 both remain because they belong to the same user turn.

For a prefix-stable Preserved renderer, the row remains one datum and thinking 1, 2, and 3 all remain. GLM 5.1 is the exception: its Preserved renderer keeps the reasoning but still unrolls because of its template's prefix behavior.

Because unrolled datums duplicate conversation context, the tuned token count can exceed the raw dataset token count. Unrolling is independent of context-length truncation or splitting.

## Configure a training job

The optional, immutable REST field is `thinkingTraceHistoryMode`:

* `THINKING_TRACE_HISTORY_MODE_UNSPECIFIED`: use the registered model's default behavior. For an unregistered model, keep its existing renderer fallback.
* `THINKING_TRACE_HISTORY_MODE_INTERLEAVED`: explicitly request Interleaved when supported.
* `THINKING_TRACE_HISTORY_MODE_PRESERVED`: explicitly request Preserved when supported.

`UNSPECIFIED` is the default for clients that omit the field; it is not a third user-selectable history mode.

```json theme={null}
{
  "thinkingTraceHistoryMode": "THINKING_TRACE_HISTORY_MODE_PRESERVED"
}
```

An explicitly requested unsupported mode fails configuration validation before GPU training starts. Managed SFT, DPO, and ORPO use the same field and enum. The unrolling behavior described above applies to managed SFT; RFT does not expose this setting.

## Preview before training

A render-preview request does not select a thinking-history mode. It returns every available option for the base model in `thinkingTraceHistoryModeOptions` and one rendering per option for each example.

* Selectable models return two options for a toggle.
* Fixed and automatic models return one read-only option.
* A rendering can contain multiple `renderedDatums` when that dataset row unrolls.
* Within each datum, concatenate `segments[].text` in order. A `lossWeight` of `0` is masked context; a value greater than `0` is trained.

Select one of the returned modes when creating the training job. After the job starts, use [Render Samples](/fine-tuning/fine-tuning-models#debug-sft-tokenization) to inspect the exact token IDs and loss mask seen by the trainer.

## How Fireworks verifies model behavior

Fireworks treats the official model chat template as the behavior baseline and manually onboards supported model families. Capability is not inferred dynamically from arbitrary template text.

Renderer tests pin upstream Hugging Face template revisions and verify token-for-token parity for single-turn, multi-turn, and tool-trajectory inputs. A shared regression fixture verifies all of the following:

* Interleaved clears thinking before the last real user query.
* Every assistant message after that query remains, including multiple assistants around a tool result.
* Preserved retains every thinking trace.
* Each renderer's prefix behavior produces the documented datum count.

The relevant upstream templates are [GLM 5.1](https://huggingface.co/zai-org/GLM-5.1/blob/main/chat_template.jinja), [GLM 5.2](https://huggingface.co/zai-org/GLM-5.2/blob/main/chat_template.jinja), [Qwen 3.5](https://huggingface.co/Qwen/Qwen3.5-35B-A3B/blob/main/tokenizer_config.json), [Qwen 3.6](https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/tokenizer_config.json), [Kimi K2.5](https://huggingface.co/moonshotai/Kimi-K2.5/blob/main/chat_template.jinja), [Kimi K2.6](https://huggingface.co/moonshotai/Kimi-K2.6/blob/main/chat_template.jinja), and [Kimi K2.7 Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code/blob/main/chat_template.jinja).

<Card title="Prepare an SFT dataset" icon="database" href="/fine-tuning/fine-tuning-models">
  Add `reasoning_content` to assistant messages in your JSONL dataset.
</Card>


# Cookbook: Agentic Reinforcement Learning
Source: https://docs.fireworks.ai/fine-tuning/training-api/cookbook/agentic-rl

Turn multi-turn agent and environment interactions into exact, trainer-ready trajectories without prescribing one agent architecture.

Agentic RL adds a trajectory adapter around an RL loop. The adapter runs the
agent and environment, records each policy generation, assigns the environment
reward, and returns trainer-ready data. The RL loop still owns scheduling,
grouping, advantages, optimization, and weight publication.

The cookbook's
[`async_rl_loop`](/fine-tuning/training-api/cookbook/rl) works with agentic
rollouts, but it does not require a particular agent framework, environment,
session service, or history-reconciliation policy.

The rollout contract is infrastructure-independent. On the shared serverless
pool, use the experimental
[`async_rl_loop_serverless` recipe](https://github.com/fw-ai/cookbook/blob/main/training/recipes/experiment/async_rl_loop_serverless.py);
on dedicated infrastructure, use the primary `async_rl_loop` recipe.

## Correctness boundary

For each logical trajectory, the adapter should:

* preserve the exact generated token IDs and their aligned log probabilities;
* mask prompts, tool results, and environment observations out of the loss;
* keep every segment under one reward, GRPO group member, and advantage;
* define what happens when a later prompt is not an exact token append;
* distinguish an environment outcome from a broken or incomplete trace.

Never silently truncate or positionally zip mismatched token, log-probability,
loss-mask, or routing data. Either repair the trace using an explicit policy or
drop it before training.

## Choose a trajectory architecture

There is no required architecture. Common choices include:

| Approach                    | Behavior                                                                                                              | Tradeoff                                                                                                                                                                                                                                                          |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Split on token divergence   | Start another physical training segment when exact token ancestry breaks. The segments remain one logical trajectory. | Simple and lossless, but creates more trainer sequences. This is the policy used by the cookbook's Harbor/OpenCode example.                                                                                                                                       |
| Realign or fork             | Repair a small, well-defined drift and mask the replaced span; fork for larger or earlier divergence.                 | Preserves longer contiguous sequences, but the repair heuristic must be model- and renderer-safe. [Slime's coding-agent trajectory manager](https://github.com/THUDM/slime/tree/main/examples/coding_agent_rl) is one reference.                                  |
| Session service             | Put message validation, token accumulation, retry rollback, and sample assembly beside the inference router.          | Can preserve prefix caching and reduce client-side trace movement at scale, but adds a stateful service and stricter chat-template integration. [Miles' TITO session server](https://github.com/radixark/miles/tree/main/miles/rollout/session) is one reference. |
| Reject non-append histories | Require strict append-only behavior and retry or drop any violation.                                                  | Strongest invariant and simplest trainer input, but may discard valid agent work.                                                                                                                                                                                 |

Choose and test the policy for your harness. A history rewrite caused by a
subagent, retry, dynamic system field, or context management is not inherently
an error, and should not be labeled as compaction without evidence.

## Logical trajectories and physical segments

One `rollout_fn` call is one logical trajectory and one completion in its prompt
group. It may return multiple physical `RolloutSample` segments when histories
branch or token ancestry diverges. Those segments retain the same reward and
advantage; they do not become extra GRPO completions. If branches share a
generated prefix, train that prefix once and keep it as masked context on later
branches.

Treat retry and failure behavior as an algorithm decision. A malformed trace
should normally be retried within a bounded budget and then dropped. Assigning
reward zero is appropriate only when the environment defines the failure as a
real task outcome, not as a substitute for missing or misaligned training data.

## Start from an example

The cookbook provides a shared
[Harbor + OpenCode adapter](https://github.com/fw-ai/cookbook/tree/main/training/examples/rl/harbor)
and two task-specific entrypoints:

* [Terminal-Bench 2.0](https://github.com/fw-ai/cookbook/tree/main/training/examples/rl/harbor_rl_terminal_bench)
  prepares Harbor tasks with a pinned OpenCode CLI, runs the task environments
  and verifier in local Docker containers, and trains through the serverless
  async RL recipe. Its walkthrough starts with sampling-only calibration so you
  can inspect the exact trajectories before training.
* [DABStep](https://github.com/fw-ai/cookbook/tree/main/training/examples/rl/harbor_rl_opencode)
  keeps its dataset selection and experiment configuration separate while
  reusing the same adapter.

Both entrypoints record OpenCode model calls, build a per-attempt token tree,
and split non-append token histories into segments within one logical rollout.
They demonstrate one integration pattern, not a required Harbor, OpenCode, or
trajectory-reconciliation architecture.

For the loop itself, read
[Cookbook: Reinforcement Learning](/fine-tuning/training-api/cookbook/rl). For
detailed implementation choices, calibration checks, failure policy, and
session/cache guidance, read the [agentic RL skill reference](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-agentic.md).


# Cookbook: Distillation
Source: https://docs.fireworks.ai/fine-tuning/training-api/cookbook/distillation

On-policy distillation recipes with one or more frozen teachers.

## What this is

The cookbook's `training.recipes.distillation_loop` trains one student from its own rollouts while frozen teacher deployments score those same responses.

Use it when you want recipe-managed trainer provisioning, student sampling, teacher scoring, checkpointing, and cleanup for distillation experiments.

## Modes

| Mode                 | Use when                                                   | Teacher signal                                 | Training loss                         |
| -------------------- | ---------------------------------------------------------- | ---------------------------------------------- | ------------------------------------- |
| `sampled_reverse_kl` | You want OPD-style sampled-token distillation              | Teacher logprob on each sampled response token | `importance_sampling`                 |
| `topk_forward_kl`    | You want sparse SDFT soft labels from teacher top-K tokens | Teacher `top_logprobs=K` per response position | `cross_entropy` with `[N, K]` targets |

`sampled_reverse_kl` is the default. The student samples on policy, the teacher scores the sampled tokens, and the recipe trains on the dense per-token gap:

```text theme={null}
teacher_logprob - sampling_logprob
```

For `topk_forward_kl`, set `distill_mode=DistillMode.TOPK_FORWARD_KL` and `sdft_top_k`.

## Current limits and logprobs

The distillation recipe depends on the public inference `logprobs` response:

| Field or request option | Meaning                                                                                                                                                                            |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `top_k`                 | Request-side sampling filter. It limits which next-token logits remain eligible for sampling and redistributes probability mass over that set.                                     |
| `sampling_mask`         | Optional request flag for generated tokens. It can return the count or token IDs still eligible after sampling filters such as `top_p` and `top_k`.                                |
| `logprob`               | Model logprob for the returned token before sampling-temperature and sampling-filter renormalization. In the legacy response, this is `token_logprobs`.                            |
| `sampling_logprob`      | Generation-only logprob of the sampled token after temperature and sampling filters are applied. Use this when comparing against the distribution that actually sampled the token. |
| `top_logprobs`          | Response option for returning likely alternatives at each position. The public inference API currently caps this at `5`, so `sdft_top_k` must be at most `5`.                      |

`top_k` and `top_logprobs` are different knobs: `top_k` changes sampling; `top_logprobs` only controls how many alternatives are returned in the response.

## Minimal example

```python theme={null}
from training.recipes.distillation_loop import Config, main
from training.utils import DeployConfig, TrainerConfig

cfg = Config(
    log_path="./distillation_logs",
    base_model="accounts/fireworks/models/qwen3-8b",
    teacher_model="accounts/fireworks/models/qwen3-32b",
    dataset="/path/to/prompts.jsonl",
    trainer=TrainerConfig(
        training_shape_id="accounts/fireworks/trainingShapes/qwen3-8b-128k",
    ),
    deployment=DeployConfig(tokenizer_model="Qwen/Qwen3-8B"),
    max_rows=100,
    epochs=1,
)

main(cfg)
```

If `teacher_model` is a base model resource, the recipe creates a frozen teacher deployment for scoring. If it is already an inference model or deployment resource, the recipe uses it directly.

## Multi-teacher runs

Set `multi_teacher=MultiTeacherConfig(...)` when you have more than one teacher.

With `sampled_reverse_kl`, multi-teacher OPD is routed: each dataset row is scored by exactly one teacher selected by the configured route key, defaulting to `teacher`. With `topk_forward_kl`, every configured teacher can score the sampled response and the recipe blends sparse top-K probability mass using `TeacherConfig.blend_weight`.

## Dataset contract

Rows are JSONL objects. The only required field is `messages`, the student-visible OpenAI-style chat prompt.

Optional fields:

| Field              | Use                                                                                                                                                                      |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `teacher`          | Default route key for routed sampled reverse-KL MOPD. The value must match a configured `TeacherConfig.route_value`, or the teacher `model` when `route_value` is unset. |
| `teacher_messages` | Teacher-side prompt for privileged-context scoring. If omitted, the teacher scores under `messages`.                                                                     |
| `expected_answer`  | Optional metadata for eval callbacks and smoke checks.                                                                                                                   |

Student and teacher token IDs must use a compatible tokenizer and vocabulary. Prefer teachers from the same model family, and set `TeacherConfig.tokenizer_model` when you want the recipe to validate teacher tokenizers against `DeployConfig.tokenizer_model`.

## Examples

The cookbook includes distillation examples under `training/examples/distillation`:

| Example                     | Path                                    | Use                                                                    |
| --------------------------- | --------------------------------------- | ---------------------------------------------------------------------- |
| Privileged-context OPD/SDFT | `gsm8k_privileged`                      | Student sees the problem; teacher can see privileged solution context. |
| Routed MOPD smoke           | `routed_mopd/train_two_teacher_lora.py` | Tiny generated dataset with two route labels and a LoRA student.       |

Run from the cookbook repository:

```bash theme={null}
cd training
FIREWORKS_API_KEY=... \
python examples/distillation/routed_mopd/train_two_teacher_lora.py
```

## Next steps

* [Cookbook Reference](/fine-tuning/training-api/cookbook/reference) - config classes and common recipe fields
* [Loss Functions](/fine-tuning/training-api/dedicated#loss-functions) - built-in and custom Training API losses
* [Weight sync](/fine-tuning/training-api/cookbook/reference#weight-sync) - how updated weights reach serving deployments


# Cookbook: Preference Optimization
Source: https://docs.fireworks.ai/fine-tuning/training-api/cookbook/dpo

Direct Preference Optimization with pairwise data using the cookbook recipe.

## What this is

This guide walks through DPO (Direct Preference Optimization) training using the cookbook. DPO learns from preference pairs (chosen vs. rejected responses) without a separate reward model.

## How DPO differs from GRPO

|                        | DPO                                                          | GRPO                                                                    |
| ---------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------- |
| **Trainer jobs**       | 1 for LoRA, 2 for full-parameter (policy + frozen reference) | 1-2 trainers plus an inference deployment, depending on reference needs |
| **Data**               | Preference pairs (chosen/rejected)                           | Prompts + reward function                                               |
| **Reference logprobs** | Cached once at initialization                                | Computed every step                                                     |
| **Loss**               | `-log(sigmoid(beta * margin))`                               | Advantage-weighted policy gradient + KL                                 |

## Architecture

```mermaid theme={null}
flowchart LR
  loop[Your Python Loop] -->|forward chosen+rejected| reference[Reference source frozen]
  reference -->|ref logprobs cached at init| loop
  loop -->|forward_backward_custom + optim_step| policyTrainer[Policy Trainer]
```

## Using the recipe

```python theme={null}
from training.recipes.dpo_loop import Config, main
from training.utils import TrainerConfig, WandBConfig

cfg = Config(
    log_path="./dpo_logs",
    base_model="accounts/fireworks/models/qwen3-8b",
    dataset="/path/to/preference_data.jsonl",
    tokenizer_model="Qwen/Qwen3-8B",
    beta=0.1,
    epochs=1,
    batch_size=4,
    max_seq_len=4096,
    trainer=TrainerConfig(
        training_shape_id="accounts/fireworks/trainingShapes/qwen3-8b-128k",
        reference_training_shape_id="accounts/fireworks/trainingShapes/qwen3-8b-128k-h200-forward",
    ),
    wandb=WandBConfig(entity="my-team", project="dpo-experiment"),
)

main(cfg)
```

## Dataset format

DPO expects preference pairs. Supported formats:

**Format 1 — chosen/rejected messages:**

```json theme={null}
{
  "chosen": {"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "good response"}]},
  "rejected": {"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "bad response"}]}
}
```

**Format 2 — input/output split:**

```json theme={null}
{
  "input": {"messages": [{"role": "user", "content": "..."}]},
  "preferred_output": [{"role": "assistant", "content": "good"}],
  "non_preferred_output": [{"role": "assistant", "content": "bad"}]
}
```

## Step-by-step (API-level)

### Provision trainers with `build_service_client`

DPO always needs reference logprobs. Full-parameter DPO uses a policy trainer plus a separate reference trainer backed by the model's LoRA shape with the adapter disabled; LoRA DPO uses one policy trainer and the policy session's shared base reference. Provisioning is owned by the SDK-managed service client — `build_service_client` resolves shapes, attaches or creates the trainer(s), and decides the reference strategy for you:

* **LoRA** (`lora_rank > 0`) with no `reference_training_shape_id` → `create_reference_client` reuses the policy session (no second trainer).
* **Full-parameter**, or an explicit `reference_training_shape_id` → a separate reference trainer is provisioned from the model's LoRA shape (adapter disabled) and its lifecycle is owned by the service client.

```python theme={null}
import os
from training.utils import TrainerConfig, build_service_client

api_key = os.environ["FIREWORKS_API_KEY"]
base_url = os.environ.get("FIREWORKS_BASE_URL", "https://api.fireworks.ai")
base_model = "accounts/fireworks/models/qwen3-8b"

service = build_service_client(
    api_key=api_key,
    base_url=base_url,
    additional_headers=None,
    base_model=base_model,
    tokenizer_model="Qwen/Qwen3-8B",
    lora_rank=0,
    max_context_length=None,
    learning_rate=1e-5,
    trainer=TrainerConfig(
        training_shape_id="accounts/fireworks/trainingShapes/qwen3-8b-128k",
        reference_training_shape_id="accounts/fireworks/trainingShapes/qwen3-8b-128k-h200-forward",
    ),
    # deployment=None  → trainer-only provisioning (DPO has no rollouts)
    cleanup_trainer_on_close=True,  # delete SDK-managed trainers on service.close()
)

policy_client = service.create_training_client(base_model, lora_rank=0)
reference_client = service.create_reference_client(base_model, lora_rank=0)

# ... training loop ...
# service.close()  # tears down the trainers it created
```

<Note>
  The cookbook recipes wrap these clients in `ReconnectableClient.from_training_client(...)` for blocking semantics; for a raw API-level loop you can call `policy_client` / `reference_client` directly.
</Note>

### Cache reference logprobs

Reference logprobs are computed once at initialization and reused throughout training:

```python theme={null}
ref_cache = {}
for i, (chosen_tokens, rejected_tokens, prompt_len) in enumerate(dataset):
    chosen_datum, rejected_datum = build_dpo_datums(
        chosen_tokens, rejected_tokens, prompt_len, max_seq_len=4096,
    )
    fwd = reference_client.forward([chosen_datum, rejected_datum], "cross_entropy")
    ref_cache[i] = {
        "ref_chosen": fwd.loss_fn_outputs[0]["logprobs"].data,
        "ref_rejected": fwd.loss_fn_outputs[1]["logprobs"].data,
        "chosen_tokens": chosen_tokens,
        "rejected_tokens": rejected_tokens,
        "prompt_len": prompt_len,
    }
```

### DPO loss function

```python theme={null}
import torch
import torch.nn.functional as F

def make_dpo_loss_fn(ref_chosen_logprobs, ref_rejected_logprobs, beta=0.1):
    ref_chosen_t = torch.tensor(ref_chosen_logprobs, dtype=torch.float32)
    ref_rejected_t = torch.tensor(ref_rejected_logprobs, dtype=torch.float32)

    def loss_fn(data, logprobs_list):
        pi_chosen, pi_rejected = logprobs_list[0], logprobs_list[1]
        chosen_weights = torch.tensor(data[0].loss_fn_inputs["weights"].data, dtype=torch.float32)
        rejected_weights = torch.tensor(data[1].loss_fn_inputs["weights"].data, dtype=torch.float32)

        pi_chosen_sum = torch.dot(pi_chosen.float(), chosen_weights)
        pi_rejected_sum = torch.dot(pi_rejected.float(), rejected_weights)
        ref_chosen_sum = torch.dot(ref_chosen_t.float(), chosen_weights)
        ref_rejected_sum = torch.dot(ref_rejected_t.float(), rejected_weights)

        margin = (pi_chosen_sum - ref_chosen_sum) - (pi_rejected_sum - ref_rejected_sum)
        dpo_loss = -F.logsigmoid(beta * margin)

        with torch.no_grad():
            accuracy = 1.0 if margin.item() > 0 else 0.0

        return dpo_loss, {"dpo_loss": dpo_loss.item(), "margin": margin.item(), "accuracy": accuracy}

    return loss_fn
```

### Training loop

```python theme={null}
step = 0
accum_count = 0
grad_accum = 4

for idx in ref_cache:
    cached = ref_cache[idx]
    chosen_datum, rejected_datum = build_dpo_datums(
        cached["chosen_tokens"], cached["rejected_tokens"],
        cached["prompt_len"], max_seq_len=4096,
    )
    loss_fn = make_dpo_loss_fn(
        ref_chosen_logprobs=cached["ref_chosen"],
        ref_rejected_logprobs=cached["ref_rejected"],
        beta=0.1,
    )
    result = policy_client.forward_backward_custom([chosen_datum, rejected_datum], loss_fn)
    accum_count += 1

    if accum_count >= grad_accum:
        policy_client.optim_step(
            tinker.AdamParams(learning_rate=1e-5, beta1=0.9, beta2=0.999, eps=1e-8, weight_decay=0.01)
        )
        step += 1
        accum_count = 0
        print(f"Step {step}: {result.metrics}")
```

## Operational guidance

* **Set `trainer.training_shape_id` when you need an explicit policy shape** — otherwise supported recipes auto-select a validated policy shape.
* **Leave `trainer.reference_training_shape_id` unset unless you need a specific reference shape** — full-parameter DPO auto-selects the model's validated LoRA shape as the reference; LoRA DPO uses a shared-session reference by default.
* **DPO does not provision a deployment** — there are no rollout samples or deployment weight syncs in the recipe.
* **Keep a versioned reference cache** tied to tokenizer + base model revision. If the base model changes, recompute reference logprobs.
* **Monitor margin statistics**: increasing margins indicate the policy is learning preferences.
* **DCP checkpoints are disabled by default** (`dcp_save_interval=0`). If you need to resume training from a checkpoint, set `dcp_save_interval` directly on `dpo_loop.Config`.

## Common pitfalls

* **Mismatched formatting** between chosen/rejected sequences corrupts preference signals — ensure identical prompt prefixes.
* **Stale reference cache**: If you warm-start from a different model, cached reference logprobs are invalid.

## Related preference methods

* **ORPO** (`training.recipes.orpo_loop`) — Odds Ratio Preference Optimization. Combines an SFT-style negative-log-likelihood term on the chosen response with a margin term on the odds ratio between chosen and rejected. Unlike DPO, ORPO does **not** require a reference trainer (no cached reference logprobs), so the recipe runs with a single trainer + dataset of preference pairs. See `training.recipes.orpo_loop` in the public [cookbook repo](https://github.com/fw-ai/cookbook/tree/main/training/recipes/orpo_loop.py) for the full configuration.

## Related guides

* [Cookbook RL (GRPO)](/fine-tuning/training-api/cookbook/rl) — reinforcement learning recipes
* [Cookbook Reference](/fine-tuning/training-api/cookbook/reference) — all config classes
* [Loss Functions](/fine-tuning/training-api/dedicated#loss-functions) — API-level DPO loss details


# The Cookbook
Source: https://docs.fireworks.ai/fine-tuning/training-api/cookbook/overview

Ready-to-run training recipes for GRPO, DPO, SFT, and distillation built on top of the Training API.

## What is the Cookbook?

The [Fireworks Cookbook](https://github.com/fw-ai/cookbook/tree/main/training) is a collection of training recipes and utilities built on top of the [Training API](/fine-tuning/training-api/introduction). It provides config-driven training loops that handle trainer provisioning, data loading, tokenization, gradient accumulation, checkpointing, and cleanup automatically.

The cookbook is **optional** — everything it does can be done with the API directly. Use the cookbook when you want a working training loop quickly; use the API when you need full control.

## Installation

```bash theme={null}
git clone https://github.com/fw-ai/cookbook.git
cd cookbook/training && pip install -e .
```

Set your credentials:

```bash theme={null}
export FIREWORKS_API_KEY="your-api-key"
```

## Available recipes

| Recipe                                   | Module                                                 | Use case                                                                                                                                                                                                                                                                                            |
| ---------------------------------------- | ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **RL** *(primary, experimental)*         | `training.recipes.async_rl_loop`                       | Reinforcement learning — you write a rollout function, and the recipe owns completion-driven rollout production, bounded off-policy admission, serialized GRPO training, and weight publication. See [Cookbook RL](/fine-tuning/training-api/cookbook/rl). **No backward-compatibility guarantee.** |
| **Serverless async RL** *(experimental)* | `training.recipes.experiment.async_rl_loop_serverless` | The same rollout and scheduling contract on the shared serverless training and sampling pool. See [Serverless Training](/fine-tuning/training-api/serverless). **No backward-compatibility guarantee.**                                                                                             |
| **Agentic RL integration**               | `async_rl_loop` plus your rollout adapter              | Multi-turn agents, tools, environments, token ancestry, and session architecture. See [Cookbook Agentic RL](/fine-tuning/training-api/cookbook/agentic-rl).                                                                                                                                         |
| **RL** *(simpler, synchronous)*          | `training.recipes.rl_loop`                             | Synchronous on-policy GRPO scaffold — reach for it when you don't need rollout/train overlap                                                                                                                                                                                                        |
| **IGPO**                                 | `training.recipes.igpo_loop`                           | Information Gain-based Policy Optimization — turn-level IG rewards for multi-turn agent trajectories (extends GRPO)                                                                                                                                                                                 |
| **DPO**                                  | `training.recipes.dpo_loop`                            | Direct preference optimization from chosen/rejected pairs                                                                                                                                                                                                                                           |
| **SFT**                                  | `training.recipes.sft_loop`                            | Supervised fine-tuning with cross-entropy loss                                                                                                                                                                                                                                                      |
| **Distillation**                         | `training.recipes.distillation_loop`                   | On-policy distillation with sampled-token OPD, routed multi-teacher MOPD, and top-K SDFT                                                                                                                                                                                                            |
| **ORPO**                                 | `training.recipes.orpo_loop`                           | Odds ratio preference optimization                                                                                                                                                                                                                                                                  |

Each recipe follows the same pattern: import `Config` and `main`, set your config, and call `main(cfg)`. Dedicated recipes attach or create trainer and deployment resources from `TrainerConfig` / `DeployConfig`. Serverless recipes attach to the shared pool and publish session-scoped sampler snapshots instead of provisioning a deployment.

All launch examples below use `trainer=TrainerConfig(training_shape_id=...)` for explicit shape selection. Cookbook recipes can also auto-select validated shapes when `training_shape_id` is unset. The main run-level trainer knob you may set alongside a shape is `replica_count` for replicated HSDP launches; reference shapes can usually be left unset because the cookbook auto-selects or uses a shared-session reference when appropriate.

If you want field-level details about what a training shape controls and what stays configurable, see [Training shapes](/fine-tuning/training-api/training-shapes) and the [Cookbook Reference](/fine-tuning/training-api/cookbook/reference).

<Note>
  `InfraConfig` and the standalone `setup_infra` / `ResourceCleanup` helpers are **deprecated and removed from the recipe surface**. Recipes now take `trainer=TrainerConfig(...)` (and `deployment=DeployConfig(...)` for RL). See [Migrating from the deprecated managed infra](/fine-tuning/training-api/cookbook/reference#deprecated-managed-infra-infraconfig).
</Note>

## Quick example: SFT

```python theme={null}
from training.recipes.sft_loop import Config, main
from training.utils import TrainerConfig

cfg = Config(
    log_path="./sft_quickstart",
    base_model="accounts/fireworks/models/qwen3-8b",
    dataset="/path/to/training_data.jsonl",
    tokenizer_model="Qwen/Qwen3-8B",
    max_seq_len=4096,
    epochs=1,
    batch_size=4,
    trainer=TrainerConfig(
        training_shape_id="accounts/fireworks/trainingShapes/qwen3-8b-128k",
    ),
)

main(cfg)
```

## Quick example: GRPO

```python theme={null}
from training.recipes.rl_loop import Config, main
from training.utils import DeployConfig, TrainerConfig

cfg = Config(
    log_path="./grpo_quickstart",
    base_model="accounts/fireworks/models/qwen3-8b",
    dataset="/path/to/prompts.jsonl",
    max_rows=100,
    trainer=TrainerConfig(
        training_shape_id="accounts/fireworks/trainingShapes/qwen3-8b-128k",
    ),
    deployment=DeployConfig(
        deployment_id="grpo-serving",
        tokenizer_model="Qwen/Qwen3-8B",
    ),
    weight_sync_interval=1,
)

main(cfg)
```

## W\&B logging

All cookbook recipes accept a `WandBConfig` to stream metrics to [Weights & Biases](https://wandb.ai):

```python theme={null}
from training.utils import WandBConfig

cfg = Config(
    # ... same config as above ...
    wandb=WandBConfig(
        entity="my-team",
        project="grpo-experiment",
        run_name="qwen3-8b-sft-v1",  # optional, auto-generated if omitted
    ),
)

main(cfg)
```

## Vision-language model support

All cookbook recipes support VLM fine-tuning. Use a VLM training shape and tokenizer, and provide multimodal datasets with `image_url` content. See [Vision Inputs](/fine-tuning/models) for dataset format and examples.

## Embedding loop

The [`embedding_loop`](https://github.com/fw-ai/cookbook/blob/main/training/recipes/embedding_loop.py) recipe fine-tunes embedding models with contrastive or supervised embedding objectives. Use it when your base model is an embedding endpoint rather than a chat completion model.

## Next steps

* [Cookbook SFT](/fine-tuning/training-api/cookbook/sft) — supervised fine-tuning
* [Cookbook DPO](/fine-tuning/training-api/cookbook/dpo) — preference optimization with pairwise data
* [Cookbook RL (GRPO)](/fine-tuning/training-api/cookbook/rl) — full GRPO walkthrough with reward functions
* [Cookbook Agentic RL](/fine-tuning/training-api/cookbook/agentic-rl) — multi-turn trajectory, token ancestry, and session design
* [Cookbook Distillation](/fine-tuning/training-api/cookbook/distillation) — OPD, routed MOPD, and top-K SDFT
* [Vision Inputs](/fine-tuning/models) — fine-tune VLMs with image and text data
* [Cookbook Reference](/fine-tuning/training-api/cookbook/reference) — all config classes and parameters


# Cookbook: Reference
Source: https://docs.fireworks.ai/fine-tuning/training-api/cookbook/reference

Configuration classes, checkpoint utilities, weight sync, and advanced recipe knobs.

## TrainerConfig

Training-client launch settings: which training shape to use, the optional reference trainer, region, and run-level knobs. Recipes take it as `Config.trainer`:

```python theme={null}
from training.utils import TrainerConfig

trainer = TrainerConfig(
    training_shape_id="accounts/fireworks/trainingShapes/qwen3-8b-128k",
    reference_training_shape_id="accounts/fireworks/trainingShapes/qwen3-8b-128k-h200-forward",
)
```

Use `training_shape_id` for explicit shape selection — this is the primary shape-specific value you set. Pass the full shared path `accounts/fireworks/trainingShapes/<shape>` (the `fireworks` account is the public shared shape catalog). If you leave it unset, supported recipes auto-select a validated shape from the control plane based on `base_model`, `lora_rank`, and `max_seq_len`.

| Field                         | Type                | Default | Description                                                                                                                                                                                                                                                         |
| ----------------------------- | ------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `training_shape_id`           | `str \| None`       | `None`  | Optional full training-shape ID for the policy trainer, typically `accounts/fireworks/trainingShapes/<shape>`. When unset, supported recipes auto-select a validated shape.                                                                                         |
| `reference_training_shape_id` | `str \| None`       | `None`  | Optional full training-shape ID for a separate reference trainer. For full-parameter runs that need a reference, leave unset to auto-select the model's validated LoRA shape; for LoRA runs, leave unset to use the shared-session reference on the policy trainer. |
| `job_id`                      | `str \| None`       | `None`  | Attach to an existing trainer job (resume / reattach) instead of creating a new one.                                                                                                                                                                                |
| `reference_job_id`            | `str \| None`       | `None`  | Attach to an existing reference trainer job.                                                                                                                                                                                                                        |
| `cleanup_reference_on_close`  | `bool`              | `True`  | Delete the SDK-managed reference trainer when the service closes.                                                                                                                                                                                                   |
| `region`                      | `str \| None`       | `None`  | Region override (drives trainer + deployment colocation).                                                                                                                                                                                                           |
| `timeout_s`                   | `float`             | `3600`  | Timeout for trainer provisioning / readiness waits.                                                                                                                                                                                                                 |
| `extra_args`                  | `list[str] \| None` | `None`  | Extra trainer arguments.                                                                                                                                                                                                                                            |
| `replica_count`               | `int \| None`       | `None`  | Data-parallel HSDP replica count for policy trainer launches. This is a run-level knob, not part of the validated training shape; reference trainers are launched without it.                                                                                       |
| `skip_validations`            | `bool`              | `False` | Skip server-side shape validation. Requires elevated permissions.                                                                                                                                                                                                   |
| `use_reservation`             | `bool`              | `True`  | Try reservation capacity first. Set to `False` for shared capacity only.                                                                                                                                                                                            |
| `purpose`                     | `str \| None`       | `None`  | Optional platform purpose enum name for internal scheduling metadata. Most users leave this unset.                                                                                                                                                                  |

To request replicated HSDP for a run:

```python theme={null}
trainer = TrainerConfig(
    training_shape_id="accounts/fireworks/trainingShapes/qwen3-8b-128k",
    replica_count=2,
)
```

<Note>
  On the shape path (`training_shape_id` set or auto-selected), `accelerator_type`, `accelerator_count`, `node_count`, and `custom_image_tag` are derived from the training shape. `TrainerConfig` still exposes those fields for the advanced manual path (`training_shape_id=None`), where they are sent directly and shape validation is skipped.
</Note>

<Tip>
  Migrating from `InfraConfig`? See [Deprecated managed infra (InfraConfig)](#deprecated-managed-infra-infraconfig) for the field-rename table.
</Tip>

## DeployConfig

Deployment settings for sampling and weight sync. Wraps `DeploymentConfig` fields:

```python theme={null}
from training.utils import DeployConfig

deploy_cfg = DeployConfig(
    deployment_id="grpo-serving",
    tokenizer_model="Qwen/Qwen3-8B",
)
```

When `deployment_shape` is set (the recommended path), the shape owns deployment hardware and serving configuration.

| Field                          | Type                     | Default                       | Description                                                                                                                                                   |
| ------------------------------ | ------------------------ | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `weight_sync_scope`            | `WeightSyncScope`        | `WeightSyncScope.PER_TRAINER` | Controls whether the trainer bucket or deployment bucket owns weight sync state. See [Weight sync](/fine-tuning/training-api/cookbook/reference#weight-sync). |
| `deployment_id`                | `str \| None`            | `None`                        | Deployment identifier. If unset, the cookbook auto-derives one from the base model name.                                                                      |
| `tokenizer_model`              | `str \| None`            | `None`                        | HuggingFace model name for client-side tokenization. Required for RL sampling.                                                                                |
| `tokenizer_revision`           | `str \| None`            | `None`                        | Optional HuggingFace tokenizer revision.                                                                                                                      |
| `deployment_shape`             | `str \| None`            | `None`                        | Deployment shape resource name. When set, the shape owns GPU type and serving config.                                                                         |
| `deployment_region`            | `str \| None`            | `None`                        | Region override for the deployment                                                                                                                            |
| `hot_load_bucket_type`         | `str`                    | `"FW_HOSTED"`                 | Weight-sync storage backend                                                                                                                                   |
| `hot_load_trainer_job`         | `str \| None`            | `None`                        | Trainer job name whose weight-sync bucket this deployment should use. Format: `accounts/{account}/rlorTrainerJobs/{job_id}`.                                  |
| `deployment_timeout_s`         | `float`                  | `5400`                        | Timeout for deployment provisioning / readiness waits                                                                                                         |
| `reattach_settle_timeout_s`    | `int`                    | `600`                         | Timeout for the serving pod to settle after re-attaching a deployment to a new trainer bucket.                                                                |
| `deployment_extra_args`        | `list[str] \| None`      | `None`                        | Extra serving arguments                                                                                                                                       |
| `sample_timeout`               | `int`                    | `600`                         | HTTP read timeout for sampling completions                                                                                                                    |
| `disable_speculative_decoding` | `bool`                   | `True`                        | Disable speculative decoding for weight-sync compatibility                                                                                                    |
| `extra_values`                 | `dict[str, str] \| None` | `None`                        | Extra deployment Helm values                                                                                                                                  |
| `replica_count`                | `int \| None`            | `None`                        | If set, pin the deployment to a fixed replica count (sets both min and max).                                                                                  |
| `deployment_accelerator_type`  | `str \| None`            | `None`                        | Manual-path deployment GPU type used only when no `deployment_shape` is set.                                                                                  |

<Note>
  When `deployment_shape` is set, the deployment shape owns GPU type and serving configuration. Use `deployment_accelerator_type` only for advanced manual deployments without a deployment shape.
</Note>

## ConcurrencyConfig

Rollout sampling concurrency settings used by RL-family recipes:

| Field                         | Type          | Default      | Description                                                                                                                                                   |
| ----------------------------- | ------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `mode`                        | `str \| None` | `"adaptive"` | Concurrency mode. RL recipes currently use adaptive concurrency.                                                                                              |
| `initial_window`              | `int \| None` | `None`       | Starting adaptive concurrency window. When unset, recipes derive it from deployment capacity.                                                                 |
| `min_window`                  | `int`         | `1`          | Minimum adaptive concurrency window.                                                                                                                          |
| `max_window`                  | `int`         | `256`        | Maximum adaptive concurrency window.                                                                                                                          |
| `prefill_queue_target`        | `float`       | `0.5`        | Target prefill queue duration in seconds for AIMD adjustment.                                                                                                 |
| `rollout_adjustment_interval` | `int`         | `32`         | Adjust adaptive concurrency every N completed rollout requests. Remaining requests adjust at the step boundary; set to `0` for step-boundary-only adjustment. |
| `max_concurrency`             | `int \| None` | `None`       | Deprecated fixed-concurrency compatibility field.                                                                                                             |

## Checkpoint & weight-sync fields

Weight-sync and checkpoint cadence are **top-level fields on the recipe `Config`** (no nested config object). `rl_loop` and `igpo_loop` expose weight-sync cadence knobs. `async_rl_loop` always syncs once before training and once after every optimizer batch; only its timeout is configurable. Every recipe exposes `dcp_save_interval`:

```python theme={null}
cfg = Config(
    # ... base_model, dataset, trainer, deployment ...
    weight_sync_interval=1,               # rl_loop/igpo_loop: sync weights every N steps
    weight_sync_before_training=False,    # rl_loop/igpo_loop: optional initial sync
    weight_sync_timeout=600,              # RL: per weight-sync timeout (seconds)
    dcp_save_interval=10,                  # all recipes: save resumable DCP checkpoints every N steps
)
```

<Warning>
  `dcp_save_interval` defaults to `0` (off). Without setting it to a positive value, **no DCP checkpoints are saved and training cannot be resumed**. If you need checkpoint-based resume, explicitly set `dcp_save_interval` (e.g. `dcp_save_interval=50`).
</Warning>

| Field                         | Recipes                | Type   | Default | Description                                                                                                                              |
| ----------------------------- | ---------------------- | ------ | ------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `dcp_save_interval`           | All                    | `int`  | `0`     | Save resumable DCP checkpoints every N steps. `0` disables DCP saves. **Set to a positive value to enable resume.**                      |
| `weight_sync_interval`        | `rl_loop`, `igpo_loop` | `int`  | `1`     | Save + sync weights to the deployment every N optimizer steps. `0` disables weight sync. This field is not part of `async_rl_loop`.      |
| `weight_sync_before_training` | `rl_loop`, `igpo_loop` | `bool` | `False` | Save a base checkpoint and sync it to the deployment before the first training step. `async_rl_loop` performs this sync unconditionally. |
| `weight_sync_timeout`         | RL family              | `int`  | `600`   | Timeout for each weight sync (seconds).                                                                                                  |

<Note>
  The old nested `WeightSyncConfig` recipe field is gone. Recipe `Config` objects set the fields above directly, and the SDK-managed service owns the underlying save and weight-sync state.
</Note>

## WandBConfig

Weights & Biases logging settings:

```python theme={null}
from training.utils import WandBConfig

wandb = WandBConfig(
    entity="my-team",
    project="grpo-experiment",
    run_name="qwen3-8b-v1",
)
```

| Field      | Type          | Default | Description                          |
| ---------- | ------------- | ------- | ------------------------------------ |
| `entity`   | `str \| None` | `None`  | W\&B team or user name               |
| `project`  | `str \| None` | `None`  | W\&B project name                    |
| `run_name` | `str \| None` | `None`  | Run name (auto-generated if omitted) |

## ReconnectableClient

Blocking convenience wrapper around `FiretitanTrainingClient`. All cookbook recipes use this as their training client — it dispatches each call and blocks until the result is ready or the timeout expires. Failures propagate to the caller so the training loop can crash cleanly and resume from the last DCP checkpoint.

<Note>
  This is a recipe-internal wrapper. User code should not construct it with trainer managers. Recipes build it from the `FiretitanTrainingClient` returned by the SDK-managed service client.
</Note>

```python theme={null}
from training.utils import ReconnectableClient

client = ReconnectableClient.from_training_client(
    training_client,
    base_model="accounts/fireworks/models/qwen3-8b",
    lora_rank=0,
    job_id=service.trainer_job_id,
    service=service,
)

result = client.forward_backward_custom(datums, loss_fn)
client.optim_step(tinker.AdamParams(...))
```

| Parameter         | Type                             | Default | Description                                                       |
| ----------------- | -------------------------------- | ------- | ----------------------------------------------------------------- |
| `client`          | `FiretitanTrainingClient`        | —       | Training client returned by `service.create_training_client(...)` |
| `job_id`          | `str`                            | —       | RLOR trainer job ID                                               |
| `base_model`      | `str`                            | —       | Base model name                                                   |
| `lora_rank`       | `int`                            | `0`     | LoRA rank (`0` for full-parameter)                                |
| `service`         | `FiretitanServiceClient \| None` | `None`  | Managed service that owns the trainer lifecycle                   |
| `default_timeout` | `int`                            | `3600`  | Timeout in seconds for forward/backward/optim calls               |

**Properties:**

| Property | Type  | Description        |
| -------- | ----- | ------------------ |
| `job_id` | `str` | The trainer job ID |

**Methods:**

| Method                                                         | Description                                  |
| -------------------------------------------------------------- | -------------------------------------------- |
| `forward(data, loss_fn)`                                       | Forward pass, blocks until complete          |
| `forward_backward(data, loss_fn, loss_fn_config)`              | Forward + backward pass                      |
| `forward_backward_custom(data, loss_fn)`                       | Forward + backward with custom loss function |
| `optim_step(params, grad_accumulation_normalization=None)`     | Optimizer step                               |
| `save_state(name, timeout)`                                    | Save DCP checkpoint (default timeout: 2700s) |
| `load_state_with_optimizer(path, timeout)`                     | Load DCP checkpoint (default timeout: 2700s) |
| `save_weights_for_sampler_ext(name, checkpoint_type, timeout)` | Save sampler checkpoint for promotion        |
| `resolve_checkpoint_path(name, source_job_id)`                 | Resolve cross-job checkpoint path            |
| `list_checkpoints()`                                           | List available DCP checkpoints               |

## Checkpoint utilities

For checkpointing, resume, and promote — see the dedicated [Checkpoints and Resume](/fine-tuning/training-api/cookbook/reference#checkpoints) page.

## Skills reference

Agent-facing operational guidance for gradient accumulation normalization lives in the [cookbook skill reference](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-gradient-accumulation.md).

## Deprecated managed infra (InfraConfig)

Earlier cookbook releases provisioned trainers and deployments from the recipe layer using `InfraConfig`, `WeightSyncConfig`, and the standalone helpers `setup_infra` / `ResourceCleanup` / `make_reference_client` / `create_base_reference`. Provisioning now lives entirely behind the **SDK-managed service client** (`build_service_client(...)` → `service.create_*`), and recipes take `trainer=TrainerConfig(...)` plus `deployment=DeployConfig(...)`.

<Warning>
  This is a **breaking change to the recipe-facing interface**. The recipe `Config` no longer accepts `infra=` or `weight_sync=`, and `setup_infra` / `ResourceCleanup` have been removed. If you are **not ready to migrate, simply do not upgrade the SDK + cookbook** — pin your current versions and existing code keeps working. **Upgrading is recommended** (cleaner config, one provisioning path, SDK-owned lifecycle), but it is opt-in: the old and new surfaces do not coexist in one install.
</Warning>

### What to change

| Before (deprecated)                                            | After (current)                                                                                                                                                |
| -------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Config(infra=InfraConfig(...))`                               | `Config(trainer=TrainerConfig(...))`                                                                                                                           |
| `InfraConfig.ref_training_shape_id`                            | `TrainerConfig.reference_training_shape_id`                                                                                                                    |
| `InfraConfig.trainer_timeout_s`                                | `TrainerConfig.timeout_s`                                                                                                                                      |
| `InfraConfig.trainer_replica_count`                            | `TrainerConfig.replica_count`                                                                                                                                  |
| `Config(weight_sync=WeightSyncConfig(weight_sync_interval=N))` | `Config(weight_sync_interval=N)` (top-level, `rl_loop` / `igpo_loop`; `async_rl_loop` always syncs after each optimizer batch)                                 |
| `weight_sync.dcp_save_interval=N`                              | `Config(dcp_save_interval=N)` (top-level, all recipes)                                                                                                         |
| top-level `policy_job_id=...`                                  | `TrainerConfig(job_id=...)`                                                                                                                                    |
| `setup_infra(rlor_mgr, deploy_mgr, ...)`                       | `build_service_client(...)` (see the [DPO API-level example](/fine-tuning/training-api/cookbook/dpo#step-by-step-api-level))                                   |
| `create_base_reference()` / `make_reference_client()`          | `service.create_reference_client(...)`                                                                                                                         |
| `with ResourceCleanup(...)`                                    | `cleanup_trainer_on_close=True` + `service.close()` (see [Cleanup](/fine-tuning/training-api/reference/cleanup#automatic-cleanup-via-the-sdk-managed-service)) |

The `InfraConfig` dataclass is still importable for backward compatibility and now emits a `DeprecationWarning` when constructed; it is no longer accepted by recipe `Config` objects.

### Get help migrating

The cookbook ships one [Fireworks training skill](https://github.com/fw-ai/cookbook/tree/main/skills/fireworks-training) that routes managed and Training API work. Its progressive SDK migration reference walks an agent through porting old `InfraConfig` / `setup_infra` scripts to the new `TrainerConfig` + `build_service_client` surface, alongside weight-sync and checkpoint guidance.

<h2>
  Weight sync
</h2>

During RL training the inference deployment needs updated weights after each optimizer step. Cookbook recipes wire this through a shared bucket (trainer writes, deployment hot-loads).

For `WeightSyncScope`, error strings, and recovery: [rl-hotload skill reference](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-hotload.md).

See also: [RL cookbook](/fine-tuning/training-api/cookbook/rl).

<h2>
  Checkpoints and resume
</h2>

Recipe users: set `dcp_save_interval` and `output_model_id` on the recipe `Config`. Rerun with the same `log_path` to resume.

```python theme={null}
cfg = Config(
    log_path="./my_training",
    dcp_save_interval=10,
    output_model_id="my-model",
    ...
)
```

For DCP vs sampler checkpoints, promote/resume edge cases, and cross-job warm start: [sdk-checkpoints](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/sdk-checkpoints.md) and [training-api-losses](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/training-api-losses.md).


# Cookbook: Reinforcement Learning
Source: https://docs.fireworks.ai/fine-tuning/training-api/cookbook/rl

Run experimental async GRPO with a custom rollout function while the recipe owns scheduling, training, and weight publication.

The cookbook's primary RL recipe is [`async_rl_loop`](https://github.com/fw-ai/cookbook/blob/main/training/recipes/async_rl_loop.py). You provide dataset rows and a rollout function; the recipe runs rollout production independently from serialized training. When a rollout finishes, the producer immediately tries to refill available capacity—even while forward/backward or optimizer work is running.

<Warning>
  `async_rl_loop` is experimental. Its configuration and rollout protocol may
  change without backward-compatibility shims. Pin the cookbook version for
  production workloads.
</Warning>

For the shared serverless pool, use the experimental
[`async_rl_loop_serverless` sibling](https://github.com/fw-ai/cookbook/blob/main/training/recipes/experiment/async_rl_loop_serverless.py).
It keeps the rollout and scheduling contract described here but publishes
session-scoped snapshots instead of hotloading an inference deployment. See
[Serverless Training](/fine-tuning/training-api/serverless) for lifecycle and
limits.

## Responsibilities

| You provide                                                | The recipe owns                                                                    |
| ---------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| Dataset rows through `Config.dataset` or `rows=`           | Trainer and deployment setup, cleanup, and initial weight sync                     |
| `rollout_fn_factory(setup) -> rollout_fn`                  | Rollout fan-out, admission, grouping, and advantages                               |
| Environment interaction, scoring, and aligned rollout data | Reference and old-policy forwards, GRPO/TIS/KL, and optimizer steps                |
| Scheduling and algorithm configuration                     | Training chunks, sampler hotload and version publication, metrics, and checkpoints |
| Optional `dynamic_filter_fn`                               | Bounded handling of transient rollout failures                                     |

## Minimal setup

```python theme={null}
from training.examples.rl.single_turn_token_in.rollout import make_rollout_fn
from training.recipes.async_rl_loop import Config, main
from training.utils import DeployConfig, TrainerConfig

cfg = Config(
    log_path="./async-rl-logs",
    base_model="accounts/<account>/models/<model>",
    completions_per_prompt=8,
    prompt_groups_per_step=8,
    pipeline_chunks_per_step=4,
    max_head_offpolicy_versions=2,
    max_concurrency_rollout_sample=128,
    trainer=TrainerConfig(
        training_shape_id="accounts/<account>/trainingShapes/<shape>",
    ),
    deployment=DeployConfig(tokenizer_model="<tokenizer>"),
)

rows = [...]  # Each row is passed to rollout_fn as sample_prompt.
main(cfg, rollout_fn_factory=make_rollout_fn, rows=rows)
```

The example rollout above expects rows with `prompt_token_ids`. Fork the [single-turn example](https://github.com/fw-ai/cookbook/tree/main/training/examples/rl/single_turn_token_in) or [multi-turn example](https://github.com/fw-ai/cookbook/tree/main/training/examples/rl/multi_turn_message_in) for your environment.

For multi-turn agents, tools, sandboxes, token ancestry, and session design, read
[Cookbook: Agentic Reinforcement Learning](/fine-tuning/training-api/cookbook/agentic-rl).
Agentic RL is a rollout integration concern; it does not change the async
loop's scheduling contract. The
[Terminal-Bench 2.0 example](https://github.com/fw-ai/cookbook/tree/main/training/examples/rl/harbor_rl_terminal_bench)
shows a complete serverless integration with Harbor-managed local containers,
OpenCode tool use, verifier rewards, and multi-segment logical rollouts.

## Rollout contract

The factory receives `RolloutSetup` once and returns an async function:

```python theme={null}
from training.recipes.async_rl_loop import RolloutFn, RolloutSetup
from training.utils.rl.rollout import RolloutRun


def make_rollout_fn(setup: RolloutSetup) -> RolloutFn:
    async def rollout_fn(sample_prompt: dict) -> RolloutRun | None:
        ...

    return rollout_fn
```

The recipe calls `rollout_fn` `completions_per_prompt` times for each dataset row. One call represents one trajectory and returns:

* `RolloutRun(segments=[...])` on success. A run contains one or more `RolloutSample` segments from the same trajectory.
* `None` to drop that trajectory draw.

Each segment carries aligned `tokens`, `logprobs`, and `loss_mask` lists plus a scalar `reward`. Set the mask to `1` only for tokens that should contribute to training. All segments in one run must have the same reward.

## Scheduling controls

These five fields define the rollout/training pipeline:

| Field                            | Default | Meaning                                                                                           |
| -------------------------------- | ------: | ------------------------------------------------------------------------------------------------- |
| `completions_per_prompt`         |     `4` | Trajectories per dataset row. Must be at least `2`.                                               |
| `prompt_groups_per_step`         |     `1` | Dataset rows grouped into one optimizer batch.                                                    |
| `pipeline_chunks_per_step`       |     `1` | Balanced forward/backward chunks prepared for each optimizer batch.                               |
| `max_head_offpolicy_versions`    |     `0` | Number of published policy versions that rollout admission may run ahead. `0` is fully on-policy. |
| `max_concurrency_rollout_sample` |  `None` | Optional cap on in-flight rollout calls. It must fit at least one complete row.                   |

Admission is row-atomic: the scheduler submits a row only when both the staleness budget and concurrency budget can fit all of its completions. `max_head_offpolicy_versions=0` is fully on-policy: every optimizer batch trains groups from its current published policy version. Chunk training can still overlap remaining rollouts from the same optimizer batch.

## Runtime behavior

1. The recipe syncs initial policy weights to the sampler.
2. The producer submits complete rows while both admission budgets allow it.
3. Every completed rollout retries refill. As soon as the first training chunk is ready, serialized trainer work can begin while rollout production continues.
4. Later chunks queue and run in order. One optimizer step follows the final chunk.
5. The recipe hotloads the updated weights and publishes the next policy version. Publication reopens staleness capacity.

There is one sampler hotload per optimizer batch; `async_rl_loop` does not expose a weight-sync interval. Known transient rollout failures are dropped behind a bounded circuit breaker. Invalid rollout data, unexpected cancellation, and unknown errors remain fatal.

## Loss behavior

The stock recipe has one direct client-side GRPO path and no `policy_loss` or `loss_path` selector. `anchor_logp="old_policy"` (the default) snapshots trainer logprobs and applies TIS against rollout behavior logprobs; `anchor_logp="rollout"` reuses aligned rollout logprobs and makes the TIS ratio identity. Set `kl_beta=0` to disable reference-policy KL and reference provisioning.

## Detailed reference

Keep implementation and tuning detail out of the recipe page:

* [Async RL skill reference](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-async.md) — admission math, metrics, tuning, failure policy, and resume semantics
* [Agentic RL skill reference](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-agentic.md) — multi-turn token ancestry, session/cache architectures, mismatch policies, and trace failures
* [Custom RL loss reference](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-custom-loss.md) — fork the recipe deliberately when you need a trainer built-in or another research objective
* [Checkpointing](/fine-tuning/training-api/cookbook/reference#checkpoints) — resumable checkpoints and final model promotion
* [Weight sync](/fine-tuning/training-api/cookbook/reference#weight-sync) — how updated policy weights reach the sampler
* [`rl_loop`](https://github.com/fw-ai/cookbook/blob/main/training/recipes/rl_loop.py) — simpler synchronous GRPO when rollout/training overlap is unnecessary

## IGPO (Information Gain Policy Optimization)

The [`igpo_loop`](https://github.com/fw-ai/cookbook/blob/main/training/recipes/igpo_loop.py) recipe extends the async RL pipeline with turn-level information-gain rewards for multi-turn agent trajectories. Start from [Cookbook RL](/fine-tuning/training-api/cookbook/rl) for the core rollout and weight-sync lifecycle, then switch to `igpo_loop` when your reward depends on per-turn information gain rather than a single scalar at episode end.


# Cookbook: Supervised Fine-Tuning
Source: https://docs.fireworks.ai/fine-tuning/training-api/cookbook/sft

Supervised fine-tuning via the cookbook's sft_loop recipe.

## What this is

Supervised Fine-Tuning (SFT) trains the model to produce desired outputs by minimizing cross-entropy loss on (prompt, response) pairs. The cookbook's `sft_loop` recipe handles data loading, tokenization, batching, gradient accumulation, and checkpointing automatically.

## Using the recipe

```python theme={null}
from training.recipes.sft_loop import Config, main
from training.utils import TrainerConfig

cfg = Config(
    log_path="./sft_logs",
    base_model="accounts/fireworks/models/qwen3-8b",
    dataset="/path/to/training_data.jsonl",
    tokenizer_model="Qwen/Qwen3-8B",
    max_seq_len=4096,
    epochs=1,
    batch_size=4,
    learning_rate=1e-5,
    trainer=TrainerConfig(
        training_shape_id="accounts/fireworks/trainingShapes/qwen3-8b-128k",
    ),
)

main(cfg)
```

<Warning>
  **`batch_size_samples` is not supported in the V2 SFT `CookbookTrainingConfig`.**

  Passing `batch_size_samples` to the V2 config has no effect — the parameter is accepted without error but silently ignored, which can lead to unexpected step counts.

  **How batching works in V2:** Steps are calculated as:

  ```
  steps = (num_samples × num_epochs) / batch_size
  ```

  where `batch_size` is set by the training shape and the recipe’s `batch_size` field — not by `batch_size_samples`.

  **Example:** 10 samples × 5 epochs ÷ batch size of 10 = **5 steps**, not 50.

  To control training length, adjust `epochs` (and related recipe fields). Contact support for custom batch size configurations.
</Warning>

## Dataset format

SFT datasets use the standard messages format (JSONL with one example per line):

```json theme={null}
{"messages": [
  {"role": "user", "content": "What is the capital of France?"},
  {"role": "assistant", "content": "The capital of France is Paris."}
]}
```

Multi-turn conversations are supported:

```json theme={null}
{"messages": [
  {"role": "system", "content": "You are a helpful assistant."},
  {"role": "user", "content": "Hello"},
  {"role": "assistant", "content": "Hi! How can I help?"},
  {"role": "user", "content": "What is 2+2?"},
  {"role": "assistant", "content": "2+2 = 4"}
]}
```

The recipe automatically tokenizes conversations using the chat template, setting token weights to `0.0` for prompt tokens and `1.0` for response tokens.

### Vision datasets

The SFT recipe also supports vision-language model fine-tuning. Use multimodal `content` arrays with `image_url` objects in your JSONL, and specify a VLM training shape and tokenizer. See [Vision Inputs](/fine-tuning/models) for dataset format details and a full walkthrough.

## Checkpointing and resume

The current `sft_loop` recipe manages the trainer-side loop only. It does **not** create a deployment or run weight sync during training, but it does expose DCP checkpointing and resume controls:

```python theme={null}
from training.utils import TrainerConfig, WandBConfig

cfg = Config(
    log_path="./sft_logs",
    base_model="accounts/fireworks/models/qwen3-8b",
    dataset="/path/to/training_data.jsonl",
    tokenizer_model="Qwen/Qwen3-8B",
    max_seq_len=4096,
    epochs=1,
    batch_size=4,
    trainer=TrainerConfig(
        training_shape_id="accounts/fireworks/trainingShapes/qwen3-8b-128k",
    ),
    dcp_save_interval=50,
    init_from_checkpoint="previous-job-id:step-100",  # optional
    wandb=WandBConfig(entity="my-team", project="sft-experiment"),
)

main(cfg)
```

## Operational guidance

* **Set `trainer.training_shape_id`** — cookbook trainer launches use training shapes.
* **Only one trainer job needed** — SFT does not require a reference trainer.
* **The current recipe does not provision a deployment** — use the API directly if you want deployment-side evaluation or weight sync during SFT.
* **Use `batch_size`** to control the number of examples per optimizer step.
* **Resume**: The recipe uses `checkpoint_utils.resolve_resume()` to automatically restore from the last saved state on restart.
* **DCP checkpoints are disabled by default** (`dcp_save_interval=0`). If you need to resume training from a checkpoint, you must explicitly set `dcp_save_interval` to a positive value (e.g., `dcp_save_interval=50`).

## Related guides

* [Vision Inputs](/fine-tuning/models) — VLM fine-tuning with image and text data
* [Cookbook DPO](/fine-tuning/training-api/cookbook/dpo) — preference optimization
* [Cookbook RL (GRPO)](/fine-tuning/training-api/cookbook/rl) — reinforcement learning recipes
* [Cookbook Reference](/fine-tuning/training-api/cookbook/reference) — all config classes and parameters
* [Loss Functions](/fine-tuning/training-api/dedicated#loss-functions) — API-level SFT loss details


# Dedicated Training
Source: https://docs.fireworks.ai/fine-tuning/training-api/dedicated

Run Training API workloads with provisioned trainer and deployment resources, explicit checkpoints, and lifecycle control.

<Info>
  The Training API is currently in **private preview**. [Request access](https://fireworks.ai/contact-training) before running this guide.
</Info>

Dedicated training provisions trainer and deployment resources for your run. Use it when you need broader model or method support, full-parameter training, ORPO or distillation, sustained RL, explicit checkpoint resume, or control over rollout and evaluation deployments.

If LoRA SFT, DPO, or RL on the shared pool is sufficient, compare this path with [Serverless Training](/fine-tuning/training-api/serverless) before provisioning resources.

<Tip>
  **Start from the cookbook.** Clone [`fw-ai/cookbook`](https://github.com/fw-ai/cookbook), fork the closest recipe under [`training/recipes/`](https://github.com/fw-ai/cookbook/tree/main/training/recipes), and use the [Fireworks training skill](https://github.com/fw-ai/cookbook/tree/main/skills/fireworks-training) for losses, checkpoints, and debugging depth.
</Tip>

## How dedicated training runs

<div aria-label="Dedicated Training lifecycle using provisioned trainer and inference resources">
  <div>
    <div>1 · Local</div>
    <strong>Your Python loop</strong>
    <div>Controls data, losses, rewards, and experiment logic.</div>
  </div>

  <div>
    <div>2 · Provisioned</div>
    <strong>Dedicated trainer</strong>
    <div>Runs remote model and optimizer operations on the selected shape.</div>
  </div>

  <div>
    <div>3 · Persistent</div>
    <strong>Training artifacts</strong>
    <div>Stores sampler snapshots or resumable training state.</div>
  </div>

  <div>
    <div>4 · Optional</div>
    <strong>Inference deployment</strong>
    <div>Loads sampler weights for rollouts and evaluation.</div>
  </div>

  <div>
    <div>5 · Explicit</div>
    <strong>Promote and clean up</strong>
    <div>Register the selected model, then release billable compute.</div>
  </div>
</div>

The trainer and inference deployment are separate resources with separate billing and cleanup. Promotion registers a model but does not deploy it.

## What you can run

<CardGroup>
  <Card title="SFT, DPO, and ORPO" icon="messages">
    Use cookbook recipes with LoRA or full-parameter configurations supported by the selected training shape.
  </Card>

  <Card title="RL and RFT" icon="brain">
    Run synchronous or asynchronous rollouts with custom rewards, losses, environments, and deployment sampling.
  </Card>

  <Card title="Distillation" icon="arrows-left-right">
    Provision student and teacher resources for sampled reverse KL, top-k forward KL, and related workflows.
  </Card>

  <Card title="Custom loops" icon="code">
    Use the Tinker-compatible training client directly when a maintained recipe does not express the required behavior.
  </Card>
</CardGroup>

<h2>
  Dedicated training quickstart
</h2>

### Step 1: Install the SDK and cookbook

```bash theme={null}
git clone https://github.com/fw-ai/cookbook
cd cookbook
pip install -e ./training
export FIREWORKS_API_KEY="fw_..."
```

### Step 2: Choose the closest recipe

| Task         | Start here                                                                                                  |
| ------------ | ----------------------------------------------------------------------------------------------------------- |
| SFT          | [`sft_loop.py`](https://github.com/fw-ai/cookbook/blob/main/training/recipes/sft_loop.py)                   |
| DPO          | [`dpo_loop.py`](https://github.com/fw-ai/cookbook/blob/main/training/recipes/dpo_loop.py)                   |
| RL           | [`rl_loop.py`](https://github.com/fw-ai/cookbook/blob/main/training/recipes/rl_loop.py)                     |
| Async RL     | [`async_rl_loop.py`](https://github.com/fw-ai/cookbook/blob/main/training/recipes/async_rl_loop.py)         |
| Distillation | [`distillation_loop.py`](https://github.com/fw-ai/cookbook/blob/main/training/recipes/distillation_loop.py) |

Smoke test (bounded SFT):

```bash theme={null}
python training/examples/sft/train_sft.py \
  --output-model-id dedicated-sft-smoke \
  --max-examples 10 --epochs 1 --lora-rank 8 \
  --training-shape accounts/fireworks/trainingShapes/<shape>
```

### Step 3: Pick model and shape

Choose a shape from [Models](/fine-tuning/models). Pass the full shared shape ID as `training_shape_id`; the SDK resolves the validated version and linked deployment shape.

### Step 4–10: Train, checkpoint, promote, deploy, tear down

Follow the forked recipe. Record trainer and deployment IDs, checkpoint cadence, and cleanup flags before launch.

<h2>
  Training and sampling lifecycle
</h2>

Dedicated recipes use two independently billed resources:

1. Create or reconnect to a trainer from the selected training shape.
2. Create an inference deployment when the loop needs rollouts or evaluation.
3. Run one or more forward/backward calls, then apply one optimizer step.
4. Save sampler weights and refresh the deployment before collecting new rollouts.
5. Save resumable state on the approved cadence, promote the selected checkpoint, then delete or scale down billable resources.

Keep stable trainer and deployment IDs so retries reconnect instead of provisioning duplicates. The trainer owns optimization state; the deployment owns serving and sampling.

Serverless and dedicated sampling use the same `DeploymentSampler` request contract. See [`DeploymentSampler`: RL rollout sampling](/fine-tuning/training-api/reference/deployment-sampler#rl-rollout-sampling) for SDK-specific batching behavior, and use [Inference for RL rollouts](https://docs.fireworks.ai/guides/rollout-inference#inference-for-rl-rollouts) as the canonical session-affinity reference.

<h2>
  Loss functions
</h2>

| Need                   | Start from                                                             |
| ---------------------- | ---------------------------------------------------------------------- |
| SFT                    | `sft_loop.py` and its weighted token objective                         |
| DPO or ORPO            | `dpo_loop.py` or `orpo_loop.py`                                        |
| Standard RL            | `rl_loop.py` or `async_rl_loop.py`                                     |
| New research objective | Fork the closest recipe and use `forward_backward_custom` deliberately |

Call forward/backward multiple times before one `optim_step()` for gradient accumulation. Validate datum fields, token masks, and normalization locally. Detailed loss and datum routing lives in the [Training API losses skill reference](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/training-api-losses.md).

<h2>
  Saving and loading
</h2>

Dedicated training has three distinct checkpoint purposes:

| Purpose               | Contains                    | Use                                               |
| --------------------- | --------------------------- | ------------------------------------------------- |
| Sampler snapshot      | Weights for inference       | Refresh a rollout or evaluation deployment        |
| Resumable state       | Weights and optimizer state | Continue an interrupted training run              |
| Promotable checkpoint | Deployable model artifact   | Register the selected result as a Fireworks model |

Do not pass a sampler snapshot to a resumable-state API. Keep the same trainer ID and log path for exact continuation; use the recipe's explicit initialization option for a new job. See the [checkpoint skill reference](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/sdk-checkpoints.md).

## Deep dives (cookbook skill)

| Topic                                 | Skill reference                                                                                                                |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| Losses, datums, gradient accumulation | [training-api-losses](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/training-api-losses.md) |
| Checkpoints, resume, promote          | [sdk-checkpoints](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/sdk-checkpoints.md)         |
| Custom RL objectives                  | [rl-custom-loss](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-custom-loss.md)           |
| Async RL and concurrency              | [rl-async](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-async.md)                       |
| Recipe catalog                        | [sdk-recipes](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/sdk-recipes.md)                 |

Config class reference (short): [Cookbook Reference](/fine-tuning/training-api/cookbook/reference).

## Compare infrastructure

See [Serverless vs dedicated](/fine-tuning/training-api/introduction#infrastructure).

## Next steps

<CardGroup>
  <Card title="Cookbook overview" icon="book" href="/fine-tuning/training-api/cookbook/overview">
    Recipe entry points
  </Card>

  <Card title="Training Shapes" icon="microchip" href="/fine-tuning/training-api/training-shapes">
    What a shape pins
  </Card>

  <Card title="Models" icon="table" href="/fine-tuning/models">
    Per-model shape catalog
  </Card>

  <Card title="Cleanup" icon="broom" href="/fine-tuning/training-api/reference/cleanup">
    Teardown contract
  </Card>
</CardGroup>


# Overview
Source: https://docs.fireworks.ai/fine-tuning/training-api/introduction

Fireworks Training API — custom training loops with full Python control over objectives, while Fireworks handles distributed GPU infrastructure.

<Info>
  The Training API is currently in **private preview**. [Request early access](https://fireworks.ai/contact-training) to get started.
</Info>

<Tip>
  **Using a coding agent?** Install the [Fireworks training skill](/fine-tuning/agent/use-with-coding-agents) to help configure, run, and troubleshoot training jobs using current Fireworks best practices.
</Tip>

## What is the Training API?

Fireworks Training API lets you write training logic in plain Python on your local machine while model computation runs on remote GPUs managed by Fireworks.

Most users should start from [Cookbook recipes](/fine-tuning/training-api/cookbook/overview), the recommended entry point for standard SFT, DPO, GRPO-style training, and experimental async RL loops for agentic RL. Recipes use the Python SDK and can be run directly or through your agent.

Use the Python SDK directly when you need full control over Training API behavior.

| Starting point      | Best for                                                               | How you use it                                              |
| ------------------- | ---------------------------------------------------------------------- | ----------------------------------------------------------- |
| **Cookbook recipe** | Adapting a working SFT, DPO, GRPO-style, or experimental async RL loop | Run it with the Python SDK directly or through your agent   |
| **Python SDK**      | Full control over training behavior                                    | Write the training flow in Python while Fireworks runs GPUs |

<h2>
  Choose infrastructure
</h2>

After choosing the Training API, decide how compute is provided:

* [**Serverless Training**](/fine-tuning/training-api/serverless): shared pooled trainer, LoRA SFT, DPO, or RL on supported models, no provisioning, per-token billing.
* [**Dedicated Training**](/fine-tuning/training-api/dedicated): provisioned trainer and deployment resources, broader model and method support, explicit checkpoint/resume/deployment control.

<CardGroup>
  <Card title="Serverless Training" icon="bolt" href="/fine-tuning/training-api/serverless">
    Attach to a shared pooled trainer. There is no trainer or rollout deployment to provision.
  </Card>

  <Card title="Dedicated Training" icon="server" href="/fine-tuning/training-api/dedicated">
    Provision trainer and deployment resources for your run, with broader model and method support.
  </Card>
</CardGroup>

### Quick decision

<div aria-label="Decision guide comparing serverless and dedicated Training API infrastructure">
  <div>
    <strong>Start with Serverless Training</strong>

    <div>
      The model is supported, LoRA SFT or RL covers the task, and you want pooled compute with per-token billing.
    </div>
  </div>

  <div>
    <strong>Choose Dedicated Training</strong>

    <div>
      You need full-parameter training, DPO, explicit resume or deployment control, or sustained provisioned compute.
    </div>
  </div>
</div>

### Comparison

| Dimension         | Serverless                                                       | Dedicated                                                                                     |
| ----------------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| Provisioning      | Shared pooled trainer; no trainer or sampler deployment creation | SDK provisions trainer and deployment resources                                               |
| Billing           | Per token; no idle GPU charge                                    | Time-based trainer and deployment billing                                                     |
| Parameter mode    | LoRA only                                                        | LoRA and full-parameter                                                                       |
| Methods           | SFT and RL on the supported serverless surface                   | SFT, DPO, ORPO, RL, distillation, and custom loops supported by the selected shape and recipe |
| Models            | Current serverless model list                                    | Models with an enabled dedicated training shape                                               |
| Capacity          | Shared pool and per-account limits                               | Resources allocated to the run, subject to account quota and platform availability            |
| Checkpoint resume | In-run snapshots; cross-run resume is limited                    | Explicit checkpoint, reconnect, promotion, and deployment lifecycle                           |
| Sampling          | In-session sampler, no deployment to create                      | SDK-managed rollout or evaluation deployment                                                  |
| Teardown          | Session lifecycle is managed by the service                      | You must close trainers and delete or scale down deployments                                  |
| Best fit          | Fast LoRA experiments and first RL iterations                    | Full-parameter work, DPO, sustained RL, larger workloads, explicit lifecycle control          |

Always verify current models, limits, prices, and feature status in the [Serverless Training](/fine-tuning/training-api/serverless) and [Dedicated Training](/fine-tuning/training-api/dedicated) pages before launch.

## Who does what

| Fireworks handles                                                        | Cookbook recipes handle                                                    | Python SDK users implement                                                     |
| ------------------------------------------------------------------------ | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| GPU provisioning and cluster management                                  | Training loop structure for supported recipes                              | Training loop logic (`forward_backward_custom` + `optim_step`)                 |
| Service-mode trainer lifecycle (create, health-check, reconnect, delete) | Resource setup, health checks, reconnect, and cleanup                      | Managed service setup with `FiretitanServiceClient.from_firetitan_config(...)` |
| Distributed forward pass, backward pass, optimizer execution             | Common losses and reward/evaluation plumbing                               | Loss function and batch construction                                           |
| Checkpoint storage and export                                            | Checkpoint save, resume, promotion, and sampler refresh                    | Checkpoint calls (`save_weights_for_sampler`, DCP snapshots)                   |
| Inference deployments and weight sync                                    | Deployment sampling and serving-integrated evaluation for RL recipes       | Custom rollout, sampling, and evaluation logic through the managed service     |
| Preemption recovery and job resume                                       | Resume logic for supported recipe checkpoints                              | Resume policy and state restoration calls                                      |
| Distributed training (multi-node, sharding, FSDP)                        | Config surfaces for learning rate, grad accumulation, context length, W\&B | Hyperparameter schedules, data pipeline, and experiment tracking               |

## System architecture

<div aria-label="Training API lifecycle from a local Python loop through Fireworks training and sampling infrastructure">
  <div>
    <div>1 · Your laptop</div>
    <strong>Python loop</strong>

    <div>
      Loads data, builds batches, computes rewards, and controls the experiment.
    </div>
  </div>

  <div>
    <div>2 · Fireworks API</div>
    <strong>Control plane</strong>

    <div>
      Authenticates requests, routes operations, and manages the selected infrastructure.
    </div>
  </div>

  <div>
    <div>3 · Remote compute</div>
    <strong>GPU trainer</strong>

    <div>
      Runs forward passes, backward passes, and optimizer steps.
    </div>
  </div>

  <div>
    <div>4 · Separate compute</div>
    <strong>Sampling</strong>

    <div>
      Serves saved weights for rollouts and evaluation outside the trainer.
    </div>
  </div>

  <div>
    <div>5 · Persistent output</div>
    <strong>Artifacts</strong>

    <div>
      Stores sampler snapshots and resumable training state for later use.
    </div>
  </div>
</div>

Your Python process stays on your laptop throughout the run. It sends model operations to Fireworks and receives metrics or completions back. Sampling uses separate inference infrastructure rather than the trainer itself.

## How service-mode training works

<Warning>
  **Most common gotchas**

  * Remote operations such as `forward`, `forward_backward`, `optim_step`, sampling, and checkpoint saves return future-like results. Call `.result()` on operations that return one.
  * `token_weights=0` means prompt/no-loss tokens, `token_weights=1` means response/learned tokens.
  * `forward_backward_custom` computes gradients only; you still need `optim_step` to apply updates.
</Warning>

### Minimal training step lifecycle

The shape of the loop is the same on both infrastructures. Toggle between them to see what changes at each stage:

<TrainingLifecycle />

### Datums

A **Datum** is the unit of training data sent to the remote GPU. It wraps tokenized input and per-token weights that your loss function needs.

For SFT, token weight `0.0` marks prompt tokens and `1.0` marks response tokens. Cookbook renderers construct these weights from chat messages.

### Logprobs and forward\_backward\_custom

When you call `forward_backward_custom`, the GPU runs a forward pass and returns **per-token log-probabilities** as PyTorch tensors with `requires_grad=True`. Your loss function computes a scalar loss, the API calls `loss.backward()`, and gradients are sent back to the GPU for the model backward pass.

After accumulating gradients, call `optim_step` to apply the update. See the [Dedicated Training Quickstart](/fine-tuning/training-api/dedicated#quickstart) for one complete runnable Datum, loss, and optimizer loop.

### Futures

Remote training operations such as `forward`, `forward_backward`, `optim_step`, and checkpoint saves return **future-like results**. Call `.result()` on operations that return one so failures surface.

### Checkpointing and weight sync

After training, you export checkpoints for serving:

* **Base snapshot:** a complete chain anchor for the trainable state. For LoRA this is the adapter; for full-parameter training it is model weights.
* **Delta snapshot:** a change relative to a prior full-parameter base snapshot.

The SDK selects base versus delta automatically unless the recipe overrides it.

Checkpoint-to-sampler behavior depends on the infrastructure:

* **Serverless:** save a snapshot and bind an in-session sampling client to that snapshot. There is no deployment weight sync. See [Serverless Training](/fine-tuning/training-api/serverless).
* **Dedicated:** save a snapshot and refresh an SDK-managed deployment sampler, which syncs weights onto the deployment. See [Dedicated Training and Sampling](/fine-tuning/training-api/dedicated#training-and-sampling).

For dedicated RL rollouts that continue across weight sync, see [KV cache behavior for RL rollouts](/guides/rollout-inference#kv-cache-behavior-for-rl-rollouts).

### Dedicated RL rollout transition mode

When a dedicated RL recipe provisions a hot-load rollout deployment, you can set `hot_load_transition_type` to `ASYNC` or `SYNC` in the SDK provisioning config or the cookbook rollout deployment config. Leave it unset to keep the recommended `ASYNC` default; set `SYNC` when a rollout must not span a weight transition. For the tradeoffs and KV-cache behavior, see [Async transition (recommended, default for RL)](/fine-tuning/rl-rollout-integration#async-transition-recommended-default-for-rl).

## Key APIs

| API                                                                             | Purpose                                                                                                                                       |
| ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| [`FiretitanServiceClient`](/fine-tuning/training-api/reference/service-client)  | Recommended direct SDK entry point. Creates or reattaches trainers/deployments and returns training, reference, and sampling clients.         |
| [`FiretitanTrainingClient`](/fine-tuning/training-api/reference/service-client) | Tinker-compatible training client: `forward_backward_custom`, `optim_step`, `save_weights_for_sampler`, `save_state`, and load methods.       |
| [`DeploymentSampler`](/fine-tuning/training-api/reference/deployment-sampler)   | FireTitan-native sampler for tokenized rollout/evaluation from SDK-managed deployments.                                                       |
| [`FireworksClient`](/fine-tuning/training-api/reference/fireworks-client)       | Standalone checkpoint operations such as listing checkpoints or promoting a model without a live training instance.                           |
| [`TrainerJobManager`](/fine-tuning/training-api/reference/trainer-job-manager)  | Legacy/compatibility lifecycle manager. Documented for existing SDK users and advanced debugging; not the recommended user-facing path.       |
| [`DeploymentManager`](/fine-tuning/training-api/reference/deployment-manager)   | Legacy/compatibility deployment manager. Documented for existing SDK users and advanced debugging; normal code uses `FiretitanServiceClient`. |

## Renderers

Chat-template formatting, stop-token handling, and loss-weight masking for SFT/DPO datasets are handled by **renderers** — pluggable per-model classes that turn raw conversations into the trainer's `Datum` shape. Most users never touch a renderer directly; cookbook recipes pick the right one for the `base_model` you set. If you need to author a new one or debug parity against HuggingFace, use the canonical training skill's [renderer implementation reference](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/renderer.md) and [verification reference](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/renderer-verification.md).

## Comparing Training API pricing vs DIY bare metal

When comparing a managed training platform with a self-managed bare-metal stack,
optimize for **cost per successful iteration**, not just headline `$ / GPU-hour`.

### What to compare

* **Time to first deployed model**: include environment setup, training orchestration, checkpoint handoff, and serving integration.
* **Iteration cycle time** (`train -> eval -> deploy -> repeat`): include all retrain/redeploy plumbing, not just GPU runtime.
* **Infra engineering overhead**: include one-time setup and recurring maintenance for containers, runtimes, deployment workflows, and compatibility fixes.
* **Effective `$ / GPU-hour` at real utilization**: include idle capacity, reservation constraints, and burst/overflow behavior.
* **Train/serve parity risk**: account for potential quality drift when training and inference runtimes diverge.
* **Parallel experiment capacity**: compare fixed-reservation throughput against elastic capacity for sweeps and multi-seed runs.

### Useful formulas

```text theme={null}
iterations_per_month = available_working_days / cycle_time_days
effective_cost_per_gpu_hour = total_monthly_spend / gpu_hours_consumed
multi_turn_success ~= (single_turn_success)^turn_count
```

### Keep assumptions explicit

Document assumptions so readers can adjust them for their own workload:

* team size and fully-loaded engineering cost
* average cycle duration in each setup
* expected utilization and burst profile
* average turn count for production agent workflows
* required concurrent experiment count

## FAQ

### Why is my training run "doing nothing" even though code executed?

Usually because `.result()` was not called on futures, so failures were never surfaced.

### What's the difference between base and delta checkpoints, and when should I use each?

Let the SDK select automatically. LoRA snapshots contain the full adapter; full-parameter delta snapshots can accelerate synchronization but are not promotable. See [Saving and Loading](/fine-tuning/training-api/dedicated#saving-and-loading).

### Do I need to manage distributed training infra?

No. You implement training logic while Fireworks manages GPU provisioning and distributed infrastructure.

### Should I start with a Cookbook recipe or the Python SDK?

Start with a Cookbook recipe for most SFT, DPO, or GRPO adaptations. Use the Python SDK directly when you need custom loop semantics and full control.

### Can I evaluate serving behavior during training?

Yes. On serverless, save a snapshot and sample from it in the same session. On dedicated infrastructure, sync a snapshot to the SDK-managed deployment sampler and evaluate there.

### How should I compare Training API pricing vs a DIY bare-metal setup?

Use the framework in [Comparing Training API pricing vs DIY bare metal](#comparing-training-api-pricing-vs-diy-bare-metal). Focus on total iteration economics (cycle time, engineering overhead, utilization-adjusted cost, and quality-parity risk), then plug in your own assumptions.

### How can I compare rollout cost vs other providers?

See the [Price comparison vs Tinker](/fine-tuning/multi-turn-cost-comparison) calculator to estimate scenario-based costs on Fireworks Dedicated against Tinker's per-token pricing.

## Next steps

* [Dedicated quickstart](/fine-tuning/training-api/dedicated#quickstart) — run a minimal dedicated custom loop
* [Choose infrastructure](/fine-tuning/training-api/introduction#infrastructure) — compare serverless and dedicated training
* [Serverless Training](/fine-tuning/training-api/serverless) — shared pooled LoRA training
* [Dedicated Training](/fine-tuning/training-api/dedicated) — provisioned trainer and deployment lifecycle
* [Dedicated Training and Sampling](/fine-tuning/training-api/dedicated#training-and-sampling) — deployment-sampling lifecycle
* [Loss Functions](/fine-tuning/training-api/dedicated#loss-functions) — built-in and custom loss functions
* [Vision Inputs](/fine-tuning/models) — fine-tune vision-language models with image and text data
* [The Cookbook](/fine-tuning/training-api/cookbook/overview) — ready-to-run recipes for SFT, DPO, ORPO, GRPO/IGPO, and async RL (experimental)


# Cleanup and Teardown
Source: https://docs.fireworks.ai/fine-tuning/training-api/reference/cleanup

Delete trainer jobs and deployments after experiments to avoid leaked resources.

## What this is

RLOR trainer jobs and weight-sync-enabled deployments hold GPU resources. Always clean up after experiments — especially if jobs terminate unexpectedly. In new SDK and cookbook code, cleanup is owned by the SDK-managed service client.

## Automatic cleanup via the SDK-managed service

Create the service with cleanup options, then close it in `finally`:

```python theme={null}
from fireworks.training.sdk import FiretitanServiceClient

service = FiretitanServiceClient.from_firetitan_config(
    api_key=api_key,
    base_url=base_url,
    base_model="accounts/fireworks/models/qwen3-8b",
    tokenizer_model="Qwen/Qwen3-8B",
    lora_rank=0,
    training_shape_id="accounts/fireworks/trainingShapes/qwen3-8b-128k",
    deployment_id="research-serving",
    cleanup_trainer_on_close=True,
    cleanup_deployment_on_close="scale_to_zero",
)

try:
    run_training_loop()
finally:
    service.close()
```

`cleanup_trainer_on_close=True` deletes SDK-managed trainers. Separate reference trainers are governed by `cleanup_reference_trainer_on_close` (default `True`). `cleanup_deployment_on_close="scale_to_zero"` releases deployment GPUs while keeping the deployment resource around; use `"delete"` only when you want to remove the deployment entirely.

Cookbook recipes use the same service-client lifecycle internally and close the service through an `ExitStack`.

<Note>
  The standalone `ResourceCleanup` context manager and `setup_infra` helper have been **removed** from the cookbook. Provisioning and teardown now live behind the SDK-managed service client. See [Migrating from the deprecated managed infra](/fine-tuning/training-api/cookbook/reference#deprecated-managed-infra-infraconfig).
</Note>

## Trainer inactivity cleanup

Long-running RLOR trainer jobs are automatically stopped after 10 minutes with no tracked activity. The trainer reports this activity to the control plane, and tracked activity includes trainer API operations and active-session heartbeats.

When creating a trainer through the REST API (`POST /v1/accounts/{account_id}/rlorTrainerJobs`), set `inactivityTimeout` to a positive protobuf JSON duration to choose a different timeout:

```json theme={null}
{
  "inactivityTimeout": "1800s"
}
```

When creating a trainer through the legacy manager API, set `TrainerJobConfig.inactivity_timeout` and pass the config to `TrainerJobManager.create(...)` or `TrainerJobManager.create_and_wait(...)`:

```python theme={null}
from datetime import timedelta
from fireworks.training.sdk import TrainerJobConfig

config = TrainerJobConfig(
    base_model="accounts/fireworks/models/qwen3-8b",
    training_shape_ref="accounts/fireworks/trainingShapes/<shape>/versions/<version>",
    inactivity_timeout=timedelta(minutes=30),
)
```

With `firectl`, use `--inactivity-timeout 30m` or `--inactivity-timeout 2h`. When the value is omitted or set to `0`, Fireworks uses the 10-minute default.

To disable automatic inactivity cleanup, set `disableInactivityCleanup` in the REST API, set `TrainerJobConfig.disable_inactivity_cleanup=True` in the Python SDK, or pass `--disable-inactivity-cleanup` in `firectl`. The trainer will not be stopped due to inactivity, and GPU usage continues to accrue while the trainer is running, so delete the trainer when you no longer need it.

## Manual compatibility cleanup

If you provisioned resources yourself with `TrainerJobManager` / `DeploymentManager` instead of the managed service, delete them directly.

### Cleaning up RLOR trainer jobs

```python theme={null}
import os
from fireworks.training.sdk import TrainerJobManager, DeploymentManager

api_key = os.environ["FIREWORKS_API_KEY"]
base_url = os.environ.get("FIREWORKS_BASE_URL", "https://api.fireworks.ai")

rlor_mgr = TrainerJobManager(api_key=api_key, base_url=base_url)
deploy_mgr = DeploymentManager(api_key=api_key, base_url=base_url)

# Delete known trainer jobs from this run
for job_id in ["<policy-job-id>", "<reference-job-id>"]:
    rlor_mgr.delete(job_id=job_id)
```

### Cleaning up deployments

```python theme={null}
deploy_mgr.delete(deployment_id="<deployment-id>")
```

If you want to keep the deployment resource but release GPUs (lighter alternative to delete):

```python theme={null}
deploy_mgr.scale_to_zero(deployment_id="<deployment-id>")
```

This sets both `minReplicaCount` and `maxReplicaCount` to `0`, releasing all accelerators while keeping the deployment available for future scale-up.

### Manual cleanup with try/finally

```python theme={null}
policy_job_id = "<policy-job-id>"
reference_job_id = "<reference-job-id>"
deployment_id = "research-loop-serving"

try:
    run_training_loop()
finally:
    rlor_mgr.delete(policy_job_id)
    rlor_mgr.delete(reference_job_id)
    deploy_mgr.delete(deployment_id)
```

## Checking for leaked resources

Track the IDs you create (trainer job IDs + deployment ID) and clean those explicitly. For broad account-wide discovery, use the Fireworks console or the managed `fw.*.list()` APIs.

## Operational guidance

* **Delete both policy and reference trainers** when running GRPO (which uses 2 RLOR jobs).
* **Close the managed service** in `finally` so trainer/reference/deployment cleanup runs on Ctrl+C or exceptions.
* **Don't delete a trainer** while a `save_weights_for_sampler` operation is in progress — wait for it to complete first.

## Related Guides

* [FiretitanServiceClient](/fine-tuning/training-api/reference/service-client)
* [Training and Sampling](/fine-tuning/training-api/dedicated#training-and-sampling)


# DeploymentManager (Compatibility)
Source: https://docs.fireworks.ai/fine-tuning/training-api/reference/deployment-manager

Legacy SDK reference for direct deployment lifecycle and weight-sync management.

## Overview

<Warning>
  `DeploymentManager` is a low-level compatibility API. New user code should not wire deployments or weight-sync buckets manually; use [`FiretitanServiceClient.from_firetitan_config(...)`](/fine-tuning/training-api/reference/service-client#from_firetitan_config), then `service.create_sampling_client(model_path=...)` or `service.create_deployment_sampler(model_path=...)`. This page remains for existing integrations, migration support, and advanced deployment debugging.
</Warning>

`DeploymentManager` manages the lifecycle of inference deployments that serve as sampling and weight-sync targets during training. For on-policy training (GRPO), the deployment is synced with the latest policy weights.

```python theme={null}
from fireworks.training.sdk import DeploymentManager, DeploymentConfig
```

## Constructor

`DeploymentManager` supports separate URLs for control-plane, inference, and weight-sync traffic:

```python theme={null}
deploy_mgr = DeploymentManager(
    api_key="<FIREWORKS_API_KEY>",
    base_url="https://api.fireworks.ai",      # Control-plane URL (deployment CRUD)
    inference_url="https://api.fireworks.ai",  # Gateway URL for inference (defaults to base_url)
    hotload_api_url="https://api.fireworks.ai",# Gateway URL for weight-sync ops (defaults to base_url)
)
```

| Parameter            | Type           | Default                      | Description                                                     |
| -------------------- | -------------- | ---------------------------- | --------------------------------------------------------------- |
| `api_key`            | `str`          | —                            | Fireworks API key                                               |
| `base_url`           | `str`          | `"https://api.fireworks.ai"` | Control-plane URL for deployment CRUD                           |
| `inference_url`      | `str \| None`  | `None`                       | Gateway URL for inference completions (defaults to `base_url`)  |
| `hotload_api_url`    | `str \| None`  | `None`                       | Gateway URL for weight-sync operations (defaults to `base_url`) |
| `additional_headers` | `dict \| None` | `None`                       | Extra HTTP headers                                              |
| `verify_ssl`         | `bool \| None` | `None`                       | SSL verification override                                       |

For most users, all three URLs default to `base_url`. Separate URLs are useful when the control-plane and gateway have different endpoints (e.g. personal dev gateways).

## Methods

### `create_or_get(config, force_recreate=False)`

Create a new deployment or retrieve an existing one. Set `force_recreate=True` to delete and recreate if it already exists:

```python theme={null}
deploy_info = deploy_mgr.create_or_get(DeploymentConfig(
    deployment_id="research-loop-serving",
    base_model="accounts/fireworks/models/qwen3-8b",
    min_replica_count=0,
    max_replica_count=1,
))
```

Returns a `DeploymentInfo`.

### `wait_for_ready(deployment_id, timeout_s=600, poll_interval_s=15)`

Poll until the deployment is ready to serve:

```python theme={null}
deploy_mgr.wait_for_ready("research-loop-serving")
```

Returns a `DeploymentInfo`.

### `get(deployment_id)`

Inspect deployment status. Returns a `DeploymentInfo` or `None` if not found:

```python theme={null}
current = deploy_mgr.get("research-loop-serving")
print(current.state if current else "MISSING")
```

### `hotload_and_wait(deployment_id, base_model, snapshot_identity, ...)`

Load a checkpoint onto the deployment and wait for completion:

```python theme={null}
deploy_mgr.hotload_and_wait(
    deployment_id="my-deployment",
    base_model="accounts/fireworks/models/qwen3-8b",
    snapshot_identity=result.snapshot_name,
    timeout_seconds=400,
)
```

For delta weight syncs, pass `incremental_snapshot_metadata`:

```python theme={null}
deploy_mgr.hotload_and_wait(
    deployment_id="my-deployment",
    base_model="accounts/fireworks/models/qwen3-8b",
    snapshot_identity=delta_result.snapshot_name,
    incremental_snapshot_metadata={
        "previous_snapshot_identity": base_result.snapshot_name,
        "compression_format": "arc_v2",
        "checksum_format": "alder32",
    },
    timeout_seconds=400,
)
```

### `hotload_check_status(deployment_id, base_model, timeout=30)`

Current weight-sync status per replica — `current_snapshot_identity`, `readiness`, `loading_state.stage`. Use for ad-hoc inspection or to decide whether a weight sync is needed.

### `wait_for_hotload(deployment_id, base_model, expected_identity, timeout_seconds=400, poll_interval=5)`

Poll until every replica reports `readiness=true` and `current_snapshot_identity == expected_identity`. The "wait half" of `hotload_and_wait` — call directly when you started a sync via `hotload()` and want to block separately.

### `update(deployment_id, body, update_mask)`

Partial PATCH. `update_mask` is **required** (snake-case field paths); without it the server replaces all mutable fields, silently zeroing anything not in `body`. Returns `DeploymentInfo`.

```python theme={null}
deploy_mgr.update("my-deployment",
    body={"minReplicaCount": 2, "maxReplicaCount": 8},
    update_mask="min_replica_count,max_replica_count")
```

### `warmup(model, max_retries=30, retry_interval_s=10.0)`

Send a warmup request to the deployment after weight sync. Retries until the deployment responds or the retry limit is reached. Returns `True` on success, `False` if all retries are exhausted.

### `scale_to_zero(deployment_id)`

Release GPU resources without deleting the deployment:

```python theme={null}
deploy_mgr.scale_to_zero("research-loop-serving")
```

Sets both `minReplicaCount` and `maxReplicaCount` to `0`.

### `delete(deployment_id)`

Delete a deployment entirely:

```python theme={null}
deploy_mgr.delete("research-loop-serving")
```

## DeploymentConfig

`DeploymentManager.create_or_get(...)` accepts a `DeploymentConfig` dataclass:

When `deployment_shape` is set (the recommended path), the shape owns the deployment's hardware and serving configuration. The fields below are what you set as a user:

| Field                          | Type                     | Default               | Description                                                                                                                                                                                        |
| ------------------------------ | ------------------------ | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deployment_id`                | `str`                    | —                     | Stable deployment identifier                                                                                                                                                                       |
| `base_model`                   | `str`                    | —                     | Base model name. Must match the trainer's base model for weight sync compatibility.                                                                                                                |
| `deployment_shape`             | `str \| None`            | `None`                | Deployment shape resource name. When set, the shape owns GPU type, node count, and serving engine config.                                                                                          |
| `region`                       | `str \| None`            | `None`                | Region for the deployment                                                                                                                                                                          |
| `min_replica_count`            | `int`                    | `0`                   | Minimum replicas (set `0` to scale to zero when idle)                                                                                                                                              |
| `max_replica_count`            | `int`                    | `1`                   | Maximum replicas for autoscaling                                                                                                                                                                   |
| `accelerator_type`             | `str`                    | `"NVIDIA_H200_141GB"` | Manual-path deployment GPU type. Do not set when `deployment_shape` is set.                                                                                                                        |
| `hot_load_bucket_type`         | `str \| None`            | `"FW_HOSTED"`         | Weight sync storage backend                                                                                                                                                                        |
| `hot_load_trainer_job`         | `str \| None`            | `None`                | Trainer job name whose weight-sync bucket this deployment should use. Format: `accounts/{account}/rlorTrainerJobs/{job_id}`. When set, the deployment shares the trainer's bucket for weight sync. |
| `disable_speculative_decoding` | `bool`                   | `False`               | Disable speculative decoding                                                                                                                                                                       |
| `extra_args`                   | `list[str] \| None`      | `None`                | Extra serving arguments                                                                                                                                                                            |
| `extra_values`                 | `dict[str, str] \| None` | `None`                | Extra deployment Helm values                                                                                                                                                                       |
| `annotations`                  | `dict[str, str] \| None` | `None`                | Deployment annotations                                                                                                                                                                             |

<Note>
  On the recommended shape path, `deployment_shape` owns the deployment hardware and serving configuration, so do not override `accelerator_type`. Advanced manual deployments can omit `deployment_shape` and set `accelerator_type` directly. `skip_shape_validation` is for internal development and requires elevated permissions.
</Note>

## DeploymentInfo

Returned by `create_or_get`, `wait_for_ready`, and `get`:

| Field                 | Type          | Description                                                              |
| --------------------- | ------------- | ------------------------------------------------------------------------ |
| `deployment_id`       | `str`         | Deployment identifier                                                    |
| `name`                | `str`         | Full resource name                                                       |
| `state`               | `str`         | Deployment state (e.g. `"READY"`, `"CREATING"`)                          |
| `hot_load_bucket_url` | `str \| None` | URL for weight sync storage                                              |
| `inference_model`     | `str \| None` | Model string for completions API (`accounts/{account}/deployments/{id}`) |

## Deployment shape and training shapes

When using a training shape, the linked **deployment shape is determined by the training shape and cannot be changed**. The training shape's `deploymentShapeVersion` locks the GPU type, node count, and serving engine configuration for the inference deployment.

The one thing you **can** adjust is the **replica count**. Use `min_replica_count` and `max_replica_count` to scale up throughput for sampling during RL loops:

```python theme={null}
deploy_mgr.create_or_get(DeploymentConfig(
    deployment_id="rl-serving",
    base_model="accounts/fireworks/models/qwen3-8b",
    deployment_shape="accounts/fireworks/deploymentShapes/<deployment-shape>",
    min_replica_count=1,
    max_replica_count=4,
))
```

You normally do not name a deployment shape at all: on the shape path the SDK reads it from the training shape's `deploymentShapeVersion`. Set it explicitly only when you are managing deployments outside that flow, and take the value from the training shape rather than hand-writing one.

## Operational guidance

* **Prefer `FiretitanServiceClient`** for normal trainer/deployment provisioning and sampler refresh.
* **Keep deployment IDs stable** per experiment family for easier rollbacks.
* **Use `min_replica_count=0`** for development to avoid idle GPU costs.
* **Create the trainer before the deployment** and link the deployment to the trainer's weight-sync bucket via `hot_load_trainer_job`.
* **Use `deployment_shape`** when the control plane has a pre-validated shape for your model.
* **Do not treat shape-owned hardware as a user-facing override surface** — in normal flows, leave `accelerator_type` and placement decisions to the deployment shape and only tune replica counts.
* **Use `scale_to_zero`** after training as a lighter alternative to `delete`.

## Related guides

* [DeploymentSampler](/fine-tuning/training-api/reference/deployment-sampler) — sample from the deployment
* [FiretitanServiceClient](/fine-tuning/training-api/reference/service-client) — recommended managed service path
* [Cleanup](/fine-tuning/training-api/reference/cleanup) — resource cleanup


# DeploymentSampler
Source: https://docs.fireworks.ai/fine-tuning/training-api/reference/deployment-sampler

Client-side tokenized sampling for Training API rollouts and evaluation.

## Overview

`DeploymentSampler` handles client-side tokenization via a HuggingFace tokenizer and returns structured `SampledCompletion` objects with token IDs, logprobs, and completion metadata. Serverless and dedicated Training API sampling clients both use this implementation after their infrastructure-specific setup. Use it in training scripts that need token-level outputs (e.g. GRPO, DPO).

```python theme={null}
from fireworks.training.sdk import DeploymentSampler
```

## Constructor

```python theme={null}
from transformers import AutoTokenizer
from fireworks.training.sdk import DeploymentSampler, AdaptiveConcurrencyController

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B", trust_remote_code=True)

# Adaptive concurrency (recommended) — auto-tunes based on server load
sampler = DeploymentSampler(
    inference_url="https://api.fireworks.ai",
    model="accounts/<account-id>/deployments/<deployment-id>",
    api_key="<FIREWORKS_API_KEY>",
    tokenizer=tokenizer,
    concurrency_controller=AdaptiveConcurrencyController(initial_window=16),
)
```

| Parameter                | Type                                                                  | Description                                                                                                                                       |
| ------------------------ | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `inference_url`          | `str`                                                                 | Gateway URL for inference completions                                                                                                             |
| `model`                  | `str`                                                                 | Deployment model path (`accounts/<id>/deployments/<id>`)                                                                                          |
| `api_key`                | `str`                                                                 | Fireworks API key                                                                                                                                 |
| `tokenizer`              | `PreTrainedTokenizerBase`                                             | HuggingFace tokenizer matching the base model                                                                                                     |
| `concurrency_controller` | `AdaptiveConcurrencyController \| FixedConcurrencyController \| None` | Controls how many concurrent HTTP requests are in-flight. `None` (default) means no limit. See [Concurrency Control](#concurrency-control) below. |

## Concurrency Control

`sample_with_tokens(n=K)` fans out into K individual streaming requests. Without concurrency control, all requests fire simultaneously, which can overload the server. Two controllers are available:

### AdaptiveConcurrencyController (recommended)

Auto-tunes the concurrency window using AIMD (Additive Increase / Multiplicative Decrease) based on the server's `prefill_queue_duration`:

```python theme={null}
from fireworks.training.sdk import AdaptiveConcurrencyController

ctrl = AdaptiveConcurrencyController(
    initial_window=16,        # starting concurrency
    min_window=1,             # minimum window
    max_window=256,           # maximum window
    prefill_queue_target=0.5, # target prefill queue latency (seconds)
    adjustment_interval=32,   # adjust every 32 completed requests
)
sampler = DeploymentSampler(..., concurrency_controller=ctrl)

# Between training steps, flush remaining metrics and reset the interval
summary = ctrl.step_completed()
print(summary)  # {"window": 20, "avg_pq": 0.08, "cache_hit_rate": 0.95, ...}
```

The controller reads `prefill_queue_duration` from server response metrics. When the queue is below target, the window grows proportionally. When above, it halves (multiplicative decrease). By default, it adjusts after every 32 completed requests and at step boundaries. Set `adjustment_interval=0` to adjust only at step boundaries.

### FixedConcurrencyController

Static semaphore — use when you know the right concurrency for your deployment:

```python theme={null}
from fireworks.training.sdk import FixedConcurrencyController

sampler = DeploymentSampler(
    ...,
    concurrency_controller=FixedConcurrencyController(32),
)
```

## `sample_with_tokens(...)`

Sample completions and return structured results with token IDs. This method is `async`, so call it with `await` or wrap it with `asyncio.run(...)` from synchronous code:

```python theme={null}
import asyncio

async def main():
    completions = await sampler.sample_with_tokens(
        messages=[{"role": "user", "content": "Solve: 2+2="}],
        n=4,
        max_tokens=1024,
        temperature=0.7,
    )
    for c in completions:
        print(c.full_tokens)       # prompt + completion token IDs
        print(c.prompt_len)        # number of prompt tokens
        print(c.completion_len)    # number of completion tokens
        print(c.text)              # decoded completion text
        print(c.finish_reason)     # "stop", "length", etc.

asyncio.run(main())
```

### Retrieving inference logprobs

For GRPO importance sampling, pass `logprobs=True`:

```python theme={null}
import asyncio

async def main():
    completions = await sampler.sample_with_tokens(
        messages=[{"role": "user", "content": "Solve: 2+2="}],
        n=4,
        logprobs=True,
        top_logprobs=1,
    )
    for c in completions:
        print(c.inference_logprobs)  # List[float] or None

asyncio.run(main())
```

### Sequence length filtering

`sample_with_tokens` supports `max_seq_len` for automatic filtering:

```python theme={null}
import asyncio

completions = asyncio.run(
    sampler.sample_with_tokens(
        messages=input_messages,
        n=4,
        max_tokens=1024,
        max_seq_len=8192,  # filter out sequences exceeding this length
    )
)
```

Two levels of filtering are applied:

1. **Prompt pre-filter**: If the tokenized prompt already meets or exceeds `max_seq_len`, the method returns an empty list immediately — no inference call is made.
2. **Completion post-filter**: After sampling, any completion whose full token sequence (prompt + completion) exceeds `max_seq_len` is silently dropped.

## `sample_with_prompt_tokens(...)`

Use `sample_with_prompt_tokens` when your renderer has already produced prompt token IDs. Both serverless and dedicated services expose the shared `DeploymentSampler` through the sampling client:

```python theme={null}
sampling_client = service.create_sampling_client(
    model_path=snapshot,
    tokenizer=tokenizer,
)
sampler = sampling_client.deployment_sampler
```

Keep `sampling_client` alive until all calls through `sampler` have finished, then close it to release the underlying HTTP clients.

<h3>
  RL rollout sampling
</h3>

Use [Inference for RL rollouts](https://docs.fireworks.ai/guides/rollout-inference#inference-for-rl-rollouts) as the canonical reference for session affinity, session-ID lifecycle, and KV-cache behavior. The example below only shows how to pass a rollout session value through `DeploymentSampler` and fan out independent samples.

The following example launches two independent trajectories for every pre-tokenized prompt:

```python theme={null}
import asyncio
import secrets

sampling_client = service.create_sampling_client(
    model_path=snapshot,
    tokenizer=tokenizer,
)
sampler = sampling_client.deployment_sampler

async def sample_turn(prompt_token_ids, rollout_session_id):
    return await sampler.sample_with_prompt_tokens(
        prompt_token_ids,
        n=1,
        max_tokens=4096,
        temperature=1.0,
        logprobs=True,
        user=rollout_session_id,
    )

# Scope and retain these IDs according to "Inference for RL rollouts."
trajectory_requests = [
    (prompt_token_ids, f"rl-session-{secrets.token_hex(16)}")
    for prompt_token_ids in batch_prompt_token_ids
    for _ in range(2)
]

try:
    rollout_groups = await asyncio.gather(
        *(
            sample_turn(prompt_token_ids, rollout_session_id)
            for prompt_token_ids, rollout_session_id in trajectory_requests
        )
    )
finally:
    sampling_client.close()
```

Here, `n=1` is intentional because `sample_with_prompt_tokens(n=2, user=...)` gives both child requests the same `user` value. With 32 entries in `batch_prompt_token_ids`, the example creates 64 individual sampling requests. Follow [Inference for RL rollouts](https://docs.fireworks.ai/guides/rollout-inference#inference-for-rl-rollouts) to decide when those requests should use distinct or shared session values.

## SampledCompletion

Each completion returned by `sample_with_tokens` or `sample_with_prompt_tokens`:

| Field                | Type                  | Description                                                                      |
| -------------------- | --------------------- | -------------------------------------------------------------------------------- |
| `text`               | `str`                 | Decoded completion text                                                          |
| `full_tokens`        | `List[int]`           | Prompt + completion token IDs                                                    |
| `prompt_len`         | `int`                 | Number of prompt tokens                                                          |
| `finish_reason`      | `str`                 | `"stop"`, `"length"`, etc.                                                       |
| `completion_len`     | `int`                 | Number of completion tokens                                                      |
| `inference_logprobs` | `List[float] \| None` | Per-token logprobs (when `logprobs=True` is passed)                              |
| `logprobs_echoed`    | `bool`                | `True` when `echo=True` was used — logprobs are training-aligned (P+C-1 entries) |
| `routing_matrices`   | `List[str] \| None`   | Base64-encoded per-token routing matrices for MoE Router Replay (R3)             |

## Related guides

* [FiretitanServiceClient](/fine-tuning/training-api/reference/service-client) — create SDK-managed deployment samplers
* [Serverless Training](/fine-tuning/training-api/serverless) — create an in-session sampler from a serverless checkpoint
* [Training and Sampling](/fine-tuning/training-api/dedicated#training-and-sampling) — end-to-end workflow
* [Cookbook RL recipe](/fine-tuning/training-api/cookbook/rl) — GRPO with sampling pipeline
* [Inference for RL rollouts](https://docs.fireworks.ai/guides/rollout-inference#inference-for-rl-rollouts) — session affinity, KV-cache behavior, and rollout request fields


# FireworksClient
Source: https://docs.fireworks.ai/fine-tuning/training-api/reference/fireworks-client

Account-level operations that don't require a running trainer job.

## Overview

`FireworksClient` provides Fireworks platform operations that are independent of any running trainer job: checkpoint promotion, training shape resolution, and model validation. It is also the base class for the legacy [`TrainerJobManager`](/fine-tuning/training-api/reference/trainer-job-manager), which adds direct trainer job lifecycle methods.

Use `FireworksClient` directly when you don't need to create or manage trainer jobs -- for example, promoting a checkpoint after the trainer has already been deleted, or resolving training shape configuration before deciding whether to launch a job.

```python theme={null}
from fireworks.training.sdk import FireworksClient
```

## Constructor

```python theme={null}
client = FireworksClient(
    api_key="<FIREWORKS_API_KEY>",
    base_url="https://api.fireworks.ai",  # optional
)
```

| Parameter            | Type           | Default                      | Description               |
| -------------------- | -------------- | ---------------------------- | ------------------------- |
| `api_key`            | `str`          | --                           | Fireworks API key         |
| `base_url`           | `str`          | `"https://api.fireworks.ai"` | Control-plane URL         |
| `additional_headers` | `dict \| None` | `None`                       | Extra HTTP headers        |
| `verify_ssl`         | `bool \| None` | `None`                       | SSL verification override |

## Methods

### `promote_checkpoint(*, name, output_model_id, base_model)`

Promote a sampler checkpoint to a deployable Fireworks model. The trainer job does not need to be running -- the checkpoint resource name is enough to resolve the GCS bucket where the files reside.

```python theme={null}
from datetime import datetime

def latest_promotable(rows):
    return max(
        (row for row in rows if row.get("promotable")),
        key=lambda row: datetime.fromisoformat(
            row["createTime"].replace("Z", "+00:00")
        ),
    )

entry = latest_promotable(client.list_checkpoints("<job-id>"))
model = client.promote_checkpoint(
    name=entry["name"],          # accounts/<a>/rlorTrainerJobs/<j>/checkpoints/<c>
    output_model_id="my-fine-tuned-model",
    base_model="accounts/fireworks/models/qwen3-8b",
)
print(f"Model state: {model['state']}, kind: {model['kind']}")
```

| Parameter         | Type  | Description                                                                                                                     |
| ----------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------- |
| `name`            | `str` | Full 4-segment checkpoint resource name (`accounts/<a>/rlorTrainerJobs/<j>/checkpoints/<c>`), as returned by `list_checkpoints` |
| `output_model_id` | `str` | Desired model ID (1-63 chars, lowercase a-z, 0-9, hyphen only)                                                                  |
| `base_model`      | `str` | Base model resource name for metadata inheritance (e.g. `accounts/fireworks/models/qwen3-8b`)                                   |

Returns the model dict from the API (includes `state`, `kind`, `peftDetails`). See [Saving and Loading](/fine-tuning/training-api/dedicated#saving-and-loading) for details, and [Checkpoint kinds](/fine-tuning/training-api/cookbook/reference#checkpoints) for which checkpoints are promotable.

<Note>
  The trainer job can be in any state (running, failed, cancelled, or deleted) as long as the checkpoint files still exist in GCS. Promotion is a file copy -- it does not interact with the trainer process.
</Note>

Validate `output_model_id` with [`validate_output_model_id`](#validate-output-model-id-output-model-id) before calling — a rejected ID (>63 chars or bad charset) orphans the staged sampler blob.

### `list_checkpoints(job_id, *, page_size=200)`

Server-side list of a trainer's checkpoints (sampler + DCP, with promotability metadata). Works on any trainer state — including deleted — while the DB record + GCS blobs survive. Auto-paginates. Distinct from [`FiretitanTrainingClient.list_checkpoints()`](/fine-tuning/training-api/reference/service-client) (live-pod, DCP names only).

```python theme={null}
rows = client.list_checkpoints(job_id)
latest = latest_promotable(rows)
```

Each row has `name`, `createTime` / `updateTime` (RFC3339), `checkpointType` (opaque server enum — filter on `promotable` rather than matching values), and `promotable` (bool, authoritative). Server returns rows **oldest-first** — re-sort client-side for newest-first. Requires `fireworks-ai[training] >= 1.0.0a62`.

### `resolve_training_profile(shape_id)`

Resolve a training shape ID into a full configuration profile:

```python theme={null}
shape_id = "accounts/fireworks/trainingShapes/ts-qwen3-8b-policy"
profile = client.resolve_training_profile(shape_id)
print(profile.accelerator_type)      # e.g. "NVIDIA_B200_192GB"
print(profile.trainer_image_tag)      # e.g. "0.0.0-dev-..."
print(profile.node_count)             # e.g. 1
print(profile.pipeline_parallelism)   # e.g. 1
```

See [Training shapes](/fine-tuning/training-api/training-shapes) for the user-facing shape workflow.

### `validate_output_model_id(output_model_id)`

Client-side validation helper for `promote_checkpoint(..., output_model_id=...)`:

```python theme={null}
from fireworks.training.sdk import validate_output_model_id

errors = validate_output_model_id("my-fine-tuned-model")
if errors:
    raise ValueError("\n".join(errors))
```

Returns a list of formatted error strings. An empty list means the model ID is valid.

## Relationship to managed service clients

Normal training code should use [`FiretitanServiceClient.from_firetitan_config(...)`](/fine-tuning/training-api/reference/service-client#from_firetitan_config), which creates the trainer/deployment and delegates checkpoint listing/promotion through its managed control-plane client.

Use `FireworksClient` directly when you only need platform-level operations outside a live training service, such as promoting a checkpoint from a completed experiment. Use `TrainerJobManager` only for legacy integrations or advanced lifecycle debugging.

```python theme={null}
from fireworks.training.sdk import FireworksClient, TrainerJobManager

# Trainer-free: promote a checkpoint from a completed experiment
client = FireworksClient(api_key=api_key)
entry = latest_promotable(client.list_checkpoints(job_id))
client.promote_checkpoint(name=entry["name"], output_model_id="my-model", base_model=base_model)

# Compatibility lifecycle: create trainer manually, train, promote
mgr = TrainerJobManager(api_key=api_key)
endpoint = mgr.create_and_wait(config)
# ... train ...
entry = latest_promotable(mgr.list_checkpoints(endpoint.job_id))
mgr.promote_checkpoint(name=entry["name"], output_model_id="my-model", base_model=base_model)
mgr.delete(endpoint.job_id)
```

## TrainingShapeProfile

Returned by `resolve_training_profile`:

| Field                          | Type          | Description                                                                                                                                  |
| ------------------------------ | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `training_shape_version`       | `str`         | Resolved shape version                                                                                                                       |
| `trainer_image_tag`            | `str`         | Docker image tag for the trainer                                                                                                             |
| `max_supported_context_length` | `int`         | Maximum supported context length                                                                                                             |
| `node_count`                   | `int`         | Number of trainer nodes                                                                                                                      |
| `deployment_shape_version`     | `str`         | Linked deployment shape                                                                                                                      |
| `deployment_image_tag`         | `str`         | Docker image tag for the linked deployment                                                                                                   |
| `accelerator_type`             | `str`         | GPU type                                                                                                                                     |
| `accelerator_count`            | `int`         | Number of GPUs per node                                                                                                                      |
| `base_model_weight_precision`  | `str`         | Model weight precision                                                                                                                       |
| `pipeline_parallelism`         | `int`         | Pipeline parallelism degree                                                                                                                  |
| `trainer_mode`                 | `str`         | Shape mode: `POLICY_TRAINER` or `LORA_TRAINER`. The legacy `FORWARD_ONLY` mode is deprecated and is no longer surfaced in the shape catalog. |
| `training_shape`               | `str`         | Training shape name (without `/versions/...` suffix)                                                                                         |
| `deployment_shape`             | `str \| None` | Full versioned deployment shape resource name; pass as-is to `DeploymentConfig.deployment_shape` for pinning                                 |
| `supports_lora`                | `bool`        | Whether the shape is LoRA-capable (`trainer_mode == "LORA_TRAINER"`)                                                                         |

## Related guides

* [TrainerJobManager](/fine-tuning/training-api/reference/trainer-job-manager) -- trainer job lifecycle (extends FireworksClient)
* [Saving and Loading](/fine-tuning/training-api/dedicated#saving-and-loading) -- checkpoint save, load, and promote workflows
* [Training shapes](/fine-tuning/training-api/training-shapes) -- what a shape pins, and its deployment linkage


# FiretitanServiceClient & TrainingClient
Source: https://docs.fireworks.ai/fine-tuning/training-api/reference/service-client

Connect to a trainer endpoint and use the training client for forward/backward passes, optimizer steps, and checkpointing.

## Overview

`FiretitanServiceClient` is the recommended direct SDK entry point. In the managed path, it creates or reattaches the FireTitan trainer, optional reference trainer, and optional inference deployment, then returns Tinker-compatible training and sampling clients.

For most direct SDK code, create it with `FiretitanServiceClient.from_firetitan_config(...)`. The bare constructor is still useful when you already have a trainer endpoint URL, but that is a lower-level compatibility path.

```python theme={null}
from fireworks.training.sdk import FiretitanServiceClient, GradAccNormalization
```

## FiretitanServiceClient

### `from_firetitan_config(...)`

Create a lazy SDK-managed service. The trainer and deployment are provisioned on the first client call, usually `create_training_client(...)`:

```python theme={null}
service = FiretitanServiceClient.from_firetitan_config(
    api_key="<FIREWORKS_API_KEY>",
    base_url="https://api.fireworks.ai",
    base_model="accounts/fireworks/models/qwen3-8b",
    tokenizer_model="Qwen/Qwen3-8B",
    lora_rank=0,
    training_shape_id="accounts/fireworks/trainingShapes/qwen3-8b-128k",
    deployment_id="research-serving",   # set create_deployment=False for trainer-only flows
    learning_rate=1e-5,
    replica_count=1,                     # deployment replicas
    cleanup_trainer_on_close=True,
    cleanup_deployment_on_close="scale_to_zero",
)

training_client = service.create_training_client(
    base_model="accounts/fireworks/models/qwen3-8b",
    lora_rank=0,
)
```

Core managed config fields:

| Field                                | Type                                  | Default                    | Description                                                                                              |
| ------------------------------------ | ------------------------------------- | -------------------------- | -------------------------------------------------------------------------------------------------------- |
| `api_key`                            | `str \| None`                         | `FIREWORKS_API_KEY`        | Fireworks API key.                                                                                       |
| `base_url`                           | `str \| None`                         | `https://api.fireworks.ai` | Control-plane URL.                                                                                       |
| `inference_url`                      | `str \| None`                         | `None`                     | Optional inference gateway URL.                                                                          |
| `base_model`                         | `str`                                 | —                          | Fireworks base model resource name.                                                                      |
| `tokenizer_model`                    | `str \| None`                         | `None`                     | HuggingFace tokenizer name used by `get_tokenizer()` and sampler setup.                                  |
| `lora_rank`                          | `int`                                 | `0`                        | `0` for full-parameter training; positive value for LoRA.                                                |
| `training_shape_id`                  | `str \| None`                         | `None`                     | User-facing training shape ID. The SDK resolves the pinned version.                                      |
| `reference_training_shape_id`        | `str \| None`                         | `None`                     | Optional separate reference trainer shape. Use the model's LoRA shape.                                   |
| `trainer_job_id`                     | `str \| None`                         | `None`                     | Reattach to an existing trainer instead of creating one.                                                 |
| `reference_trainer_job_id`           | `str \| None`                         | `None`                     | Reattach to an existing reference trainer.                                                               |
| `create_deployment`                  | `bool`                                | `True`                     | Whether to create or reattach an inference deployment. Set `False` for trainer-only SFT/DPO-style loops. |
| `deployment_id`                      | `str \| None`                         | `None`                     | Create or reattach an inference deployment for sampling and weight sync.                                 |
| `deployment_shape`                   | `str \| None`                         | Linked shape               | Optional deployment shape override. Usually inherited from the training shape.                           |
| `trainer_replica_count`              | `int \| None`                         | `None`                     | Data-parallel HSDP replicas for the trainer.                                                             |
| `replica_count`                      | `int`                                 | `1`                        | Inference deployment replicas.                                                                           |
| `cleanup_trainer_on_close`           | `bool`                                | `False`                    | Delete the SDK-managed policy trainer when `service.close()` runs.                                       |
| `cleanup_reference_trainer_on_close` | `bool`                                | `True`                     | Delete SDK-managed separate reference trainers when released/closed.                                     |
| `cleanup_deployment_on_close`        | `"scale_to_zero" \| "delete" \| None` | `None`                     | Optional deployment cleanup action on close.                                                             |

The managed service exposes resolved metadata after provisioning:

```python theme={null}
print(service.trainer_job_id)
print(service.deployment_id)
print(service.max_context_length)
print(service.reference_trainer_job_id)  # None when the reference is shared
```

### Bare constructor

```python theme={null}
service = FiretitanServiceClient(
    base_url=endpoint.base_url,  # From TrainerJobManager.create_and_wait(...)
    api_key="<FIREWORKS_API_KEY>",
)
```

`base_url` is the trainer endpoint URL from `TrainerServiceEndpoint.base_url`. Use this only when you intentionally manage trainer lifecycle yourself. New user code should use `from_firetitan_config(...)`.

### `create_training_client(base_model, lora_rank, user_metadata)`

Creates a `FiretitanTrainingClient` for training operations:

```python theme={null}
training_client = service.create_training_client(
    base_model="accounts/fireworks/models/qwen3-8b",
    lora_rank=0,  # Must match lora_rank from job creation
)
```

| Parameter       | Type                     | Default | Description                                                 |
| --------------- | ------------------------ | ------- | ----------------------------------------------------------- |
| `base_model`    | `str`                    | —       | Must match the trainer job's `base_model`                   |
| `lora_rank`     | `int`                    | `0`     | Must match trainer creation config (`0` for full-parameter) |
| `user_metadata` | `dict[str, str] \| None` | `None`  | Optional run metadata                                       |

<Warning>
  A `ValueError` is raised if you attempt to create a second training client with the same `(base_model, lora_rank)` on the same `FiretitanServiceClient` instance. Create a new `FiretitanServiceClient` for a separate trainer.
</Warning>

### Connecting to an existing trainer

If you already have a running trainer (e.g. from a previous session), connect directly by URL:

```python theme={null}
service = FiretitanServiceClient(
    base_url="https://<existing-trainer-url>",
    api_key="<FIREWORKS_API_KEY>",
)
training_client = service.create_training_client(
    base_model="accounts/fireworks/models/qwen3-8b",
    lora_rank=0,
)
```

### `create_base_training_client(base_model, user_metadata=None)`

Creates a base-only client on the same trainer session. Use this as a frozen reference for LoRA KL/reference logprobs without launching a separate reference trainer:

```python theme={null}
reference_client = service.create_base_training_client(base_model=base_model)
ref = reference_client.forward(datums, "cross_entropy").result()
```

Do not call `forward_backward`, `forward_backward_custom`, or `optim_step` on this client; it is for reference forward passes only.

### `create_reference_client(base_model, lora_rank=0, user_metadata=None)`

Create a frozen reference client for KL/DPO baseline logprobs:

```python theme={null}
reference_client = service.create_reference_client(base_model, lora_rank=0)
ref = reference_client.forward(datums, "cross_entropy").result()
```

The SDK chooses the backing automatically. LoRA policies without an explicit reference shape reuse the policy trainer with the adapter disabled. Full-parameter policies, explicit `reference_training_shape_id`, or explicit `reference_trainer_job_id` use a separate reference trainer owned by the service, backed by the model's LoRA shape with the adapter disabled.

### `create_sampling_client(model_path=None, ...)`

Return a Tinker-shaped sampling client backed by the SDK-managed deployment. When `model_path` is provided, the SDK first syncs that sampler snapshot to the deployment:

```python theme={null}
saved = training_client.save_weights_for_sampler("step-100").result()
sampler = service.create_sampling_client(model_path=saved.path)
```

This is the replacement for calling a standalone weight-sync helper in user code. The SDK tracks the base/delta chain and builds the weight-sync metadata internally.

### `create_deployment_sampler(model_path=None, tokenizer=None, concurrency_controller=None)`

Return the FireTitan-native `DeploymentSampler` directly. Use this when you need tokenized completions, inference logprobs, routing matrices, or adaptive concurrency:

```python theme={null}
sampler = service.create_deployment_sampler(
    model_path=saved.path,
    tokenizer=tokenizer,
    concurrency_controller=controller,
)
```

### `hotload_sampler_snapshot(model_path)`

Low-level method for syncing a previously saved sampler snapshot into the SDK-managed deployment without constructing a sampler:

```python theme={null}
service.hotload_sampler_snapshot(saved.path)
```

## FiretitanTrainingClient

The training client returned by `create_training_client()`. Core training RPCs like `forward(...)`, `forward_backward_custom(...)`, `optim_step(...)`, `save_state(...)`, and `load_state_with_optimizer(...)` return **futures**. Fireworks convenience helpers like `save_weights_for_sampler_ext(...)`, `list_checkpoints()`, and `resolve_checkpoint_path(...)` return concrete values directly.

### Logical batches and parallel submission

One `forward(...)`, `forward_backward(...)`, or `forward_backward_custom(...)` call defines one logical batch. Pass every datum that should run together in that call.

For large logical batches, the SDK may split the request into smaller transport chunks. It submits those chunks in parallel for forward passes, backward passes used by `forward_backward_custom(...)`, and combined `forward_backward(...)` passes. The chunks remain parts of the same logical batch; they do not become independent trainer batches.

To preserve that boundary, the SDK reserves consecutive sequence IDs, submits chunks `2..N` in parallel, and submits chunk `1` last. The trainer holds the later chunks until chunk `1` arrives, then runs all chunks from that call together as one batch.

Separate API calls define separate logical batches. Do not split one intended batch across several calls and rely on the trainer to merge them. The fragments may run as smaller trainer batches, paying fixed batch overhead more often and using the accelerators less efficiently. This reduces trainer throughput.

### Large-result timeouts

<Note>
  If retrieving a large future result times out, manually enable metadata-only future retrieval for the trainer and client. This mode is currently opt-in while backend support is being rolled out.
</Note>

### `forward(datums, loss_type)`

Forward-only pass (no gradient computation). Useful for computing reference logprobs in GRPO/DPO:

```python theme={null}
result = training_client.forward(datums, "cross_entropy").result()
logprobs = result.loss_fn_outputs[0]["logprobs"].data
```

<Note>
  Built-in loss types like `"cross_entropy"` require datums with `target_tokens` in `loss_fn_inputs`. Datums built with `datum_from_model_input_weights` will fail. Use the target-token `tinker.Datum` example in [Loss Functions](/fine-tuning/training-api/dedicated#loss-functions) for built-in losses, or use `forward_backward_custom` with the weight-based format in [Building datums](/fine-tuning/training-api/dedicated#loss-functions) and the custom-loss pattern in [Example: simple cross-entropy](/fine-tuning/training-api/dedicated#loss-functions).
</Note>

### `forward_backward(datums, loss_type, loss_fn_config=None)`

Run a combined forward and backward pass with a built-in trainer-side loss. The complete `datums` argument is one logical batch, even when the SDK splits it into parallel transport chunks:

```python theme={null}
result = training_client.forward_backward(datums, "cross_entropy").result()
print(result.metrics)
```

### `forward_backward_custom(datums, loss_fn)`

Forward + backward with your custom loss function. See [Loss Functions](/fine-tuning/training-api/dedicated#loss-functions) for details:

```python theme={null}
def my_loss(data, logprobs_list):
    loss = compute_loss(data, logprobs_list)
    return loss, {"loss": float(loss.item())}

result = training_client.forward_backward_custom(datums, my_loss).result()
print(result.metrics)  # {"loss": 0.42}
```

This method performs a forward pass, computes your loss and output gradients locally, and then runs the trainer backward pass. Both remote phases use parallel chunk submission while preserving the input as one logical batch.

For embedding-space objectives, pass `output="embedding"` and choose `pooling="mean"` or `"last"`; your loss function then receives pooled embedding tensors instead of logprobs:

```python theme={null}
result = training_client.forward_backward_custom(
    datums,
    embedding_loss,
    output="embedding",
    pooling="mean",
).result()
```

### `optim_step(adam_params, grad_accumulation_normalization=None)`

Apply optimizer update after accumulating gradients:

```python theme={null}
import tinker

training_client.optim_step(
    tinker.AdamParams(
        learning_rate=1e-5,
        beta1=0.9,
        beta2=0.999,
        eps=1e-8,
        weight_decay=0.01,
    )
).result()
```

Advanced callers may pass `grad_accumulation_normalization` to control how accumulated gradients are normalized. The default `None` leaves gradients unchanged. Pass `GradAccNormalization.NUM_LOSS_TOKENS`, `GradAccNormalization.NUM_SEQUENCES`, or `GradAccNormalization.NONE` rather than raw strings. See the [cookbook skill reference](https://github.com/fw-ai/cookbook/blob/main/skills/fireworks-training/references/rl-gradient-accumulation.md) for operational guidance.

### `save_weights_for_sampler(name, checkpoint_type=None)`

Save serving-compatible sampler weights and return a future. This is the normal Tinker-shaped API:

```python theme={null}
saved = training_client.save_weights_for_sampler(
    "step-100",
    checkpoint_type="base",  # optional: "base" or "delta"
).result()
print(saved.path)  # Snapshot identity for create_sampling_client(model_path=...)
```

Full-parameter training saves a base checkpoint first and deltas after that by default. LoRA training always saves base checkpoints. The returned `path` is a public snapshot identity, not a raw storage URI.

### `save_weights_for_sampler_ext(name, checkpoint_type)`

Fireworks-specific extension that returns a concrete `SaveSamplerResult` instead of a future:

```python theme={null}
result = training_client.save_weights_for_sampler_ext(
    "step-100",
    checkpoint_type="base",  # "base" for full weights, "delta" for incremental
)
print(result.snapshot_name)  # Session-qualified name for weight sync
```

| Parameter         | Type          | Default | Description                                          |
| ----------------- | ------------- | ------- | ---------------------------------------------------- |
| `name`            | `str`         | —       | Checkpoint name (auto-suffixed with session ID)      |
| `checkpoint_type` | `str \| None` | `None`  | `"base"` for full weights, `"delta"` for incremental |

<Note>
  On full-parameter training, only `checkpoint_type="base"` produces a promotable blob; `"delta"` cannot be promoted. LoRA is always promotable. See [Checkpoint kinds](/fine-tuning/training-api/cookbook/reference#checkpoints) for the full promotability matrix.
</Note>

`save_weights_for_sampler_ext` saves the snapshot only; it does not mutate a deployment. To serve the snapshot, pass `result.snapshot_name` to the managed service weight-sync path, or use `create_sampling_client(model_path=...)` / `create_deployment_sampler(model_path=...)`, which sync and return a sampler.

### `save_state(name, timeout=None)`

Save full train state (weights + optimizer) for resume:

```python theme={null}
training_client.save_state("train_state_step_100").result()
```

| Parameter | Type            | Default | Description                                                   |
| --------- | --------------- | ------- | ------------------------------------------------------------- |
| `name`    | `str`           | —       | Checkpoint name                                               |
| `timeout` | `float \| None` | `None`  | If set, block until the save completes or the timeout expires |

### `load_state_with_optimizer(name)`

Restore full train state (weights + optimizer) from a checkpoint:

```python theme={null}
training_client.load_state_with_optimizer("train_state_step_100").result()
```

### `load_state(name)`

Load model weights from a checkpoint without restoring optimizer state. The optimizer is reset so the next `optim_step` starts fresh:

```python theme={null}
training_client.load_state("train_state_step_100").result()
```

### `load_adapter(adapter_ref)`

Load a Hugging Face PEFT adapter model into the current LoRA session. Pass a Fireworks model resource name for a promoted adapter, such as `accounts/<ACCOUNT_ID>/models/<ADAPTER_MODEL_ID>`. This is a weights-only warm start; it does not restore optimizer state, scheduler state, or data cursor.

```python theme={null}
training_client.load_adapter("accounts/<ACCOUNT_ID>/models/<ADAPTER_MODEL_ID>").result()
```

### `list_checkpoints()`

List available DCP checkpoints from the trainer. Returns a `list[str]`:

```python theme={null}
checkpoint_names = training_client.list_checkpoints()
print(checkpoint_names)  # e.g. ["step-2", "step-4"]
```

### `resolve_checkpoint_path(checkpoint_name, source_job_id)`

Resolve a checkpoint path for cross-job resume:

```python theme={null}
checkpoint_ref = training_client.resolve_checkpoint_path(
    "step-4",
    source_job_id="previous-job-id",
)
training_client.load_state_with_optimizer(checkpoint_ref).result()
```

## SaveSamplerResult

Returned by `save_weights_for_sampler_ext`:

| Field           | Type  | Description                                       |
| --------------- | ----- | ------------------------------------------------- |
| `path`          | `str` | Snapshot name from trainer                        |
| `snapshot_name` | `str` | Session-qualified name for weight sync operations |

## GradAccNormalization

Enum for the advanced `optim_step` `grad_accumulation_normalization` parameter:

| Enum                                   | Wire value          | Description                                                     |
| -------------------------------------- | ------------------- | --------------------------------------------------------------- |
| `GradAccNormalization.NUM_LOSS_TOKENS` | `"num_loss_tokens"` | Normalize by total loss tokens across accumulated micro-batches |
| `GradAccNormalization.NUM_SEQUENCES`   | `"num_sequences"`   | Normalize by total sequences across accumulated micro-batches   |
| `GradAccNormalization.NONE`            | `"none"`            | Explicit no normalization (raw gradient sum)                    |

## Related guides

* [Training and Sampling](/fine-tuning/training-api/dedicated#training-and-sampling) — managed service training + sampler refresh walkthrough
* [Loss Functions](/fine-tuning/training-api/dedicated#loss-functions) — built-in and custom loss functions
* [Saving and Loading](/fine-tuning/training-api/dedicated#saving-and-loading) — checkpoint details


# TrainerJobManager (Compatibility)
Source: https://docs.fireworks.ai/fine-tuning/training-api/reference/trainer-job-manager

Legacy SDK reference for service-mode trainer job lifecycle management.

## Overview

<Warning>
  `TrainerJobManager` is a low-level compatibility API. New user code should not create trainer managers directly; use [`FiretitanServiceClient.from_firetitan_config(...)`](/fine-tuning/training-api/reference/service-client#from_firetitan_config) or cookbook recipes instead. This page remains for existing integrations, migration support, and advanced lifecycle debugging.
</Warning>

`TrainerJobManager` manages the lifecycle of service-mode trainer jobs — GPU-backed trainer endpoints that your Python loop connects to with a training client.

`TrainerJobManager` extends [`FireworksClient`](/fine-tuning/training-api/reference/fireworks-client), so all trainer-free operations (checkpoint promotion, training shape resolution) are also available here.

```python theme={null}
from fireworks.training.sdk import TrainerJobManager, TrainerJobConfig
```

## Constructor

```python theme={null}
rlor_mgr = TrainerJobManager(
    api_key="<FIREWORKS_API_KEY>",
    base_url="https://api.fireworks.ai",  # optional, defaults to https://api.fireworks.ai
)
```

| Parameter            | Type           | Default                      | Description               |
| -------------------- | -------------- | ---------------------------- | ------------------------- |
| `api_key`            | `str`          | —                            | Fireworks API key         |
| `base_url`           | `str`          | `"https://api.fireworks.ai"` | Control-plane URL         |
| `additional_headers` | `dict \| None` | `None`                       | Extra HTTP headers        |
| `verify_ssl`         | `bool \| None` | `None`                       | SSL verification override |

## Methods

### `create(config)`

Create a service-mode trainer job and return immediately (without waiting). Returns a `CreatedTrainerJob`:

```python theme={null}
created = rlor_mgr.create(TrainerJobConfig(
    base_model="accounts/fireworks/models/qwen3-8b",
    training_shape_ref="accounts/fireworks/trainingShapes/qwen3-8b-128k",
    lora_rank=0,
    learning_rate=1e-5,
))

print(created.job_id)    # <JOB_ID>
print(created.job_name)  # accounts/<ACCOUNT_ID>/rlorTrainerJobs/<JOB_ID>
```

### `wait_for_ready(job_id, job_name=None, poll_interval_s=5.0, timeout_s=900)`

Poll until a trainer job reaches `RUNNING` state and is healthy. Returns a `TrainerServiceEndpoint`:

```python theme={null}
endpoint = rlor_mgr.wait_for_ready(created.job_id)
```

### `create_and_wait(config, poll_interval_s=5.0, timeout_s=900)`

Create a service-mode trainer and poll until the endpoint is healthy. Combines `create()` + `wait_for_ready()`. Returns a `TrainerServiceEndpoint`.

```python theme={null}
endpoint = rlor_mgr.create_and_wait(TrainerJobConfig(
    base_model="accounts/fireworks/models/qwen3-8b",
    training_shape_ref="accounts/fireworks/trainingShapes/qwen3-8b-128k",
    lora_rank=0,
    learning_rate=1e-5,
    display_name="grpo-policy-trainer",
))

print(endpoint.base_url)  # https://<trainer-endpoint>
print(endpoint.job_id)    # <JOB_ID>
print(endpoint.job_name)  # accounts/<ACCOUNT_ID>/rlorTrainerJobs/<JOB_ID>
```

### `wait_for_existing(job_id, poll_interval_s=5.0, timeout_s=900)`

Wait for an already-existing trainer job to reach `RUNNING` state:

```python theme={null}
existing = rlor_mgr.wait_for_existing(job_id="<existing-job-id>")
print(existing.base_url)
```

### `resume_and_wait(job_id, poll_interval_s=5.0, timeout_s=900)`

Resume a failed/cancelled/paused job and wait until healthy:

```python theme={null}
endpoint = rlor_mgr.resume_and_wait(job_id="<job-id>")
```

### `reconnect_and_wait(job_id, ...)`

Handle pod preemption and transient failures. Waits for the job to reach a resumable state, resumes it, then polls until healthy:

```python theme={null}
endpoint = rlor_mgr.reconnect_and_wait(
    job_id="<job-id>",
    timeout_s=600,
    max_wait_for_resumable_s=120,
)
```

More robust than `resume_and_wait()` — retries when the job is in a transitional state (e.g. the control plane is still processing a pod death).

| Parameter                  | Type    | Default | Description                                   |
| -------------------------- | ------- | ------- | --------------------------------------------- |
| `job_id`                   | `str`   | —       | The RLOR job ID to reconnect                  |
| `poll_interval_s`          | `float` | `5.0`   | Seconds between health checks after resume    |
| `timeout_s`                | `float` | `600`   | Overall timeout for the job to become RUNNING |
| `max_wait_for_resumable_s` | `float` | `120`   | Max seconds to wait for a resumable state     |

### `get(job_id)`

Inspect job status:

```python theme={null}
status = rlor_mgr.get(job_id=endpoint.job_id)
print(status["state"])  # JOB_STATE_RUNNING
```

### `delete(job_id)`

Delete a trainer job and release GPU resources:

```python theme={null}
rlor_mgr.delete(job_id="<job-id>")
```

### `promote_checkpoint(*, name, output_model_id, base_model)`

*Inherited from [`FireworksClient`](/fine-tuning/training-api/reference/fireworks-client).* Promote a sampler checkpoint to a deployable Fireworks model. The trainer job does not need to be running -- the checkpoint resource name resolves the storage location.

```python theme={null}
from datetime import datetime

entry = max(
    (
        row
        for row in rlor_mgr.list_checkpoints(endpoint.job_id)
        if row.get("promotable")
    ),
    key=lambda row: datetime.fromisoformat(
        row["createTime"].replace("Z", "+00:00")
    ),
)
model = rlor_mgr.promote_checkpoint(
    name=entry["name"],
    output_model_id="my-fine-tuned-model",
    base_model="accounts/fireworks/models/qwen3-8b",
)
```

See [`FireworksClient.promote_checkpoint`](/fine-tuning/training-api/reference/fireworks-client#promote_checkpoint-name-output_model_id-base_model) for full parameter docs.

### `resolve_training_profile(shape_id)`

*Inherited from [`FireworksClient`](/fine-tuning/training-api/reference/fireworks-client).* Resolve a training shape ID into a full configuration profile.

```python theme={null}
shape_id = "accounts/fireworks/trainingShapes/ts-qwen3-8b-policy"
profile = rlor_mgr.resolve_training_profile(shape_id)
```

See [`FireworksClient.resolve_training_profile`](/fine-tuning/training-api/reference/fireworks-client#resolve_training_profileshape_id) for full parameter docs.

## TrainerJobConfig

`TrainerJobManager.create_and_wait(...)` accepts a `TrainerJobConfig` dataclass:

Launching through a training shape is the recommended path. In normal user code, you should not hand-author `training_shape_ref`; pass a training shape ID to `resolve_training_profile(...)` and use the returned versioned ref. Advanced manual launches can omit `training_shape_ref` and provide infra fields directly.

When `training_shape_ref` is set (the recommended **shape path**), the training shape owns the trainer's hardware and image configuration. The fields below are what you set as a user:

| Field                        | Type                                | Default | Description                                                                                                                                                                                                                                                                                                                                                     |
| ---------------------------- | ----------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `base_model`                 | `str`                               | —       | Base model name (e.g. `"accounts/fireworks/models/qwen3-8b"`)                                                                                                                                                                                                                                                                                                   |
| `training_shape_ref`         | `str \| None`                       | `None`  | Full training-shape resource name (e.g. `accounts/fireworks/trainingShapes/<shape>` or `.../versions/<ver>`). Use `mgr.resolve_training_profile(...)` to get the pinned versioned ref. See [Training shapes](/fine-tuning/training-api/training-shapes).                                                                                                        |
| `lora_rank`                  | `int`                               | `0`     | LoRA rank. `0` for full-parameter tuning, or a positive integer (e.g. `16`, `64`) for LoRA                                                                                                                                                                                                                                                                      |
| `max_context_length`         | `int \| None`                       | `None`  | Maximum sequence length. Usually inherited from the training shape on the shape path.                                                                                                                                                                                                                                                                           |
| `learning_rate`              | `float`                             | `1e-5`  | Learning rate for the optimizer                                                                                                                                                                                                                                                                                                                                 |
| `display_name`               | `str \| None`                       | `None`  | Human-readable trainer name                                                                                                                                                                                                                                                                                                                                     |
| `region`                     | `str \| None`                       | `None`  | Region for the job                                                                                                                                                                                                                                                                                                                                              |
| `extra_args`                 | `list[str] \| None`                 | `None`  | Extra trainer arguments                                                                                                                                                                                                                                                                                                                                         |
| `forward_only`               | `bool`                              | `False` | Deprecated. Created a forward-only reference trainer. Use the model's LoRA shape with the adapter disabled instead.                                                                                                                                                                                                                                             |
| `inactivity_timeout`         | `datetime.timedelta \| str \| None` | `None`  | Trainer inactivity timeout. The trainer reports tracked activity, including trainer API operations and active-session heartbeats. If no tracked activity is observed for this duration, the trainer is automatically stopped. When unset or `0`, Fireworks uses the 10-minute default. String values must use protobuf JSON duration format, such as `"1800s"`. |
| `disable_inactivity_cleanup` | `bool`                              | `False` | Disable trainer inactivity cleanup. GPU usage continues to accrue while the trainer is running.                                                                                                                                                                                                                                                                 |

<Warning>
  `gradient_accumulation_steps` is deprecated in `TrainerJobConfig`. Do not use it to request server-side accumulation. Accumulate gradients in client code by calling `forward_backward...` multiple times before one `optim_step(...)`; see [Loss Functions](/fine-tuning/training-api/dedicated#loss-functions).
</Warning>

<Note>
  On the recommended shape path, `accelerator_type`, `accelerator_count`, `node_count`, and `custom_image_tag` are automatically configured by the training shape and cannot be overridden. Advanced manual launches can omit `training_shape_ref` and set those fields directly.
</Note>

## CreatedTrainerJob

Returned by `create()`:

| Field      | Type  | Description                                               |
| ---------- | ----- | --------------------------------------------------------- |
| `job_name` | `str` | Full resource name (`accounts/<id>/rlorTrainerJobs/<id>`) |
| `job_id`   | `str` | RLOR trainer job ID                                       |

## TrainerServiceEndpoint

Returned by `create_and_wait`, `wait_for_ready`, `wait_for_existing`, `resume_and_wait`, and `reconnect_and_wait`:

| Field      | Type  | Description                                               |
| ---------- | ----- | --------------------------------------------------------- |
| `base_url` | `str` | Trainer endpoint URL for connecting a training client     |
| `job_id`   | `str` | RLOR trainer job ID                                       |
| `job_name` | `str` | Full resource name (`accounts/<id>/rlorTrainerJobs/<id>`) |

## TrainingShapeProfile

See [`FireworksClient` > TrainingShapeProfile](/fine-tuning/training-api/reference/fireworks-client#trainingshapeprofile) for the full field reference.

## Job states

| State                 | Meaning                                              |
| --------------------- | ---------------------------------------------------- |
| `JOB_STATE_CREATING`  | Resources being provisioned                          |
| `JOB_STATE_PENDING`   | Queued, waiting for GPU availability                 |
| `JOB_STATE_RUNNING`   | Trainer is ready — you can connect a training client |
| `JOB_STATE_IDLE`      | Service-mode job is idle                             |
| `JOB_STATE_COMPLETED` | Job finished successfully                            |
| `JOB_STATE_FAILED`    | Job failed                                           |
| `JOB_STATE_CANCELLED` | Job was cancelled                                    |

## Related guides

* [FiretitanServiceClient](/fine-tuning/training-api/reference/service-client) — create a `FiretitanTrainingClient` for a live trainer endpoint
* [Training shapes](/fine-tuning/training-api/training-shapes) — what a shape pins, and its deployment linkage
* [Cleanup](/fine-tuning/training-api/reference/cleanup) — resource cleanup


# WeightSyncer (Legacy)
Source: https://docs.fireworks.ai/fine-tuning/training-api/reference/weight-syncer

Backward-compatibility reference for the old standalone checkpoint-then-sync helper.

## Overview

<Warning>
  `WeightSyncer` is a legacy low-level helper kept only for backward compatibility in SDK API reference. Do not use it in new cookbook recipes or direct user loops. Use the SDK-managed service flow instead: `training_client.save_weights_for_sampler(...).result()` followed by `service.create_sampling_client(model_path=saved.path)` or `service.create_deployment_sampler(model_path=saved.path)`.
</Warning>

`WeightSyncer` coordinates saving sampler checkpoints and syncing them to a deployment, including automatic base/delta chain state tracking, session-scoped snapshot naming, and post-sync warmup. The managed service client now owns this logic internally.

```python theme={null}
from fireworks.training.sdk import WeightSyncer
```

<Note>
  For full-parameter training, only the first checkpoint (saved as `base`) is promotable; subsequent `delta` checkpoints are not. LoRA checkpoints are always promotable (delta chain is disabled via `lora_rank > 0`). See [Checkpoint kinds](/fine-tuning/training-api/cookbook/reference#checkpoints) for the full promotability matrix.
</Note>

## Constructor

```python theme={null}
tracker = WeightSyncer(
    policy_client=training_client,
    deploy_mgr=deploy_mgr,
    deployment_id="my-deployment",
    base_model="accounts/fireworks/models/qwen3-8b",
    hotload_timeout=600,
    first_checkpoint_type="base",
    warmup_after_hotload=True,
    reset_prompt_cache=True,
    lora_rank=0,  # >0 for LoRA adapters (disables delta chain)
)
```

| Field                   | Type                        | Default    | Description                                                                                                                                                                                                                 |
| ----------------------- | --------------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `policy_client`         | `FiretitanTrainingClient`   | —          | Training client for save operations                                                                                                                                                                                         |
| `deploy_mgr`            | `DeploymentManager \| None` | `None`     | Deployment manager for weight sync (`None` = no weight sync)                                                                                                                                                                |
| `deployment_id`         | `str \| None`               | `None`     | Target deployment for weight sync                                                                                                                                                                                           |
| `base_model`            | `str`                       | `""`       | Model name for weight sync API calls                                                                                                                                                                                        |
| `hotload_timeout`       | `int`                       | `600`      | Timeout in seconds for `hotload_and_wait`                                                                                                                                                                                   |
| `first_checkpoint_type` | `str`                       | `"base"`   | Type for the first checkpoint (`"base"` or `"delta"`)                                                                                                                                                                       |
| `compression_format`    | `str`                       | `"arc_v2"` | Delta compression format                                                                                                                                                                                                    |
| `warmup_after_hotload`  | `bool`                      | `True`     | Send a warmup request after each successful weight sync                                                                                                                                                                     |
| `warmup_max_retries`    | `int`                       | `10`       | Max retries for post-weight-sync warmup                                                                                                                                                                                     |
| `reset_prompt_cache`    | `bool`                      | `True`     | Reset the deployment's prompt cache after each weight sync. See [KV cache behavior for RL rollouts](/guides/rollout-inference#kv-cache-behavior-for-rl-rollouts) for active stream, session ID, and reset-option semantics. |
| `lora_rank`             | `int`                       | `0`        | When > 0, forces all checkpoints to `base` type (no delta chain). LoRA adapter exports are standalone PEFT artifacts that cannot use incremental delta compression.                                                         |

## Methods

### `save_and_hotload(name, checkpoint_type=None)`

Save sampler weights and sync to deployment. Automatically handles base (first) vs delta (subsequent) checkpoint types.

Returns the `snapshot_name` (`str | None`) on success or raises on failure:

```python theme={null}
tracker.save_and_hotload(f"step-{step:05d}")
```

### `save_only(name, checkpoint_type=None)`

Save sampler weights without syncing to deployment:

```python theme={null}
snapshot = tracker.save_only("checkpoint-name", checkpoint_type="base")
```

Returns `snapshot_name` or `None`.

### `hotload(snapshot_name, checkpoint_type)`

Sync a previously saved snapshot to the deployment:

```python theme={null}
tracker.hotload(snapshot, checkpoint_type="base")
```

Returns `True` on success, `False` on failure.

### `check_deployment_state()`

Query the deployment's current weight sync state:

```python theme={null}
current = tracker.check_deployment_state()
print(current)  # current_snapshot_identity or None
```

### `wait_for_hotload_ready(timeout_s=300, poll_interval_s=5)`

Block until the deployment's weight sync manager is initialized.

### `reset_delta_chain()`

Force the next save to be treated as `base`. Call when the deployment's bucket or trainer session changes — for example, after attaching an existing deployment to a new trainer job — otherwise the next `delta` could reference a base checkpoint the deployment never loaded.

## Usage patterns

These patterns are for maintaining older integrations. New code should use the service-client sampler refresh pattern documented in [Training and Sampling](/fine-tuning/training-api/dedicated#training-and-sampling).

### Sync weights every step

To minimize sampler staleness in a synchronous loop, sync a new sampler snapshot after every optimizer step before submitting the next rollout batch. This makes new rollout requests target the latest synced checkpoint, but the loop still owns draining or rejecting any stale in-flight requests before training on them:

```python theme={null}
import asyncio

for step in range(total_steps):
    # ... training step ...
    tracker.save_and_hotload(f"step-{step:05d}")
    completions = asyncio.run(
        sampler.sample_with_tokens(messages=input_messages, n=4)
    )
```

### Interval weight sync

For throughput-oriented loops that tolerate stale sampler weights, sync a new sampler snapshot every N steps. This only controls when new sampler snapshots are saved and synced; it does not prove that already-submitted or in-flight requests were generated by the latest policy:

```python theme={null}
for step in range(total_steps):
    # ... training step ...
    if step % weight_sync_interval == 0:
        tracker.save_and_hotload(f"step-{step:05d}")
```

### Split save and sync

Separate save from weight sync when you need intermediate steps (e.g. warmup):

```python theme={null}
snapshot = tracker.save_only("resume-step-0", checkpoint_type="base")
deploy_mgr.warmup(model)
tracker.hotload(snapshot, checkpoint_type="base")
```

### DCP checkpoints for resume

Save DCP checkpoints at intervals using the training client directly:

```python theme={null}
for step in range(total_steps):
    # ... training step ...
    tracker.save_and_hotload(f"step-{step:05d}")
    if step % dcp_interval == 0:
        training_client.save_state(f"step-{step}")
```

## Related guides

* [DeploymentManager](/fine-tuning/training-api/reference/deployment-manager) — deployment lifecycle and weight-sync API
* [Saving and Loading](/fine-tuning/training-api/dedicated#saving-and-loading) — checkpoint concepts
* [Training and Sampling](/fine-tuning/training-api/dedicated#training-and-sampling) — end-to-end workflow


# Serverless Training
Source: https://docs.fireworks.ai/fine-tuning/training-api/serverless

Run LoRA fine-tuning, preference optimization, and RL on a shared pooled trainer, with no provisioning and per-token pricing.

<Info>
  Serverless training is currently in **private preview** and access is gated per account. [Request access](https://fireworks.ai/contact-training) and select "Serverless Training API."
</Info>

Serverless training connects to a shared, already-running pooled trainer through the gateway and returns training and sampling clients. There is no trainer job or inference deployment to provision or delete. Close each sampler and the service client when finished.

Not sure whether the shared pool fits your workload? [Compare serverless and dedicated training](/fine-tuning/training-api/introduction#infrastructure).

<Info>
  **What you need**

  * **Install the SDK** (same one as the dedicated path, no separate console flow): `pip install "fireworks-ai[training]"`
  * **For the runnable example, clone the cookbook:** `git clone https://github.com/fw-ai/cookbook && pip install -e ./cookbook/training`
  * **Point at the serverless endpoint:** `base_url="https://api.fireworks.ai/training/v1/serverless"`
  * **Pick a base model enabled for serverless training on your account.** See [Models](#models) below. Availability changes during private preview; verify it before launch.
  * Set your API key and run the [Quickstart](#quickstart) below.
</Info>

<Tip>
  **Using a code agent?** Start with the self-contained [`serverless_rl` Countdown example](https://github.com/fw-ai/cookbook/tree/main/training/examples/serverless_rl), or fork the experimental [`async_rl_loop_serverless` recipe](https://github.com/fw-ai/cookbook/blob/main/training/recipes/experiment/async_rl_loop_serverless.py) when you need rollout functions, rollout/training overlap, or agentic trajectories. Both run against the serverless gateway.
</Tip>

## What is serverless training?

You write the training loop, for supervised fine-tuning, preference optimization, or reinforcement learning, and Fireworks runs the forward pass, backward pass, and optimizer on remote GPUs, then serves your latest weights for sampling in the same session.

The [quickstart](#quickstart) shows the exact client setup and operation order. The cookbook provides a compact [`serverless_rl` implementation](https://github.com/fw-ai/cookbook/tree/main/training/examples/serverless_rl) and an experimental [`async_rl_loop_serverless` recipe](https://github.com/fw-ai/cookbook/blob/main/training/recipes/experiment/async_rl_loop_serverless.py) with the same rollout contract as the dedicated async RL recipe.

### Serverless lifecycle

<div aria-label="Serverless Training lifecycle using shared pooled training and sampling infrastructure">
  <div>
    <div>1 · Local</div>
    <strong>Your Python loop</strong>
    <div>Builds batches, rewards, and optimizer requests.</div>
  </div>

  <div>
    <div>2 · Shared</div>
    <strong>Pooled trainer</strong>
    <div>Runs remote LoRA forward, backward, and optimizer work.</div>
  </div>

  <div>
    <div>3 · In session</div>
    <strong>Sampler snapshot</strong>
    <div>Captures the current adapter weights for sampling.</div>
  </div>

  <div>
    <div>4 · Shared</div>
    <strong>Pooled sampler</strong>
    <div>Returns rollouts for local scoring and the next step.</div>
  </div>
</div>

You do not create or delete a trainer job or inference deployment. Close each sampler and the service client when the run finishes.

## What you can run

<CardGroup>
  <Card title="Supervised Fine-Tuning (SFT)" icon="message">
    Run the loop with a cross-entropy loss over your labeled data.
  </Card>

  <Card title="Direct Preference Optimization (DPO)" icon="arrows-left-right">
    Train from chosen/rejected preference pairs. LoRA DPO uses the policy session's shared base reference, so there is no separate reference trainer to provision.
  </Card>

  <Card title="Reinforcement Learning (RL)" icon="brain">
    Sample completions from the current adapter, score them with your own reward function, and train with an importance-sampling loss (GRPO-style). This is the primary serverless use case.
  </Card>
</CardGroup>

All three run as LoRA on the shared pool. For full-parameter training, ORPO, distillation, or the broader Training API method set, use [Dedicated Training](/fine-tuning/training-api/dedicated). For standard platform-managed jobs, use [Managed Fine-Tuning](/fine-tuning/managed-finetuning-intro).

## When to use dedicated

Use [Dedicated Training](/fine-tuning/training-api/dedicated) when you need full-parameter training, broader model or method support, explicit resource lifecycle control, or sustained utilization. See the canonical [serverless versus dedicated comparison](/fine-tuning/training-api/introduction#infrastructure).

## Core concepts

**Session.** `create_lora_training_client(base_model, rank)` attaches you to a pooled trainer for that base model. That attachment is your training session (`service.training_session_id`) — your LoRA state lives there.

**Run.** The training client is your run (`training_client.run_id`). One run is one training trajectory: the `forward_backward` and `optim_step` calls plus the checkpoints you save.

**LoRA adapter.** Serverless is LoRA only. Pass a positive `rank` (e.g. `rank=8`); base weights stay frozen and shared across the pool, and you train an adapter on top.

**Checkpoint / snapshot.** `save_weights_for_sampler(name)` writes your current adapter weights and returns a snapshot path. That path is a public sampler identity, not a raw storage URI — hand it to the sampler to serve exactly those weights. See [Saving and loading checkpoints](#saving-and-loading-checkpoints) for the full save / resume / promote surface.

**Sampling.** `create_sampling_client(model_path=snapshot, tokenizer=...)` returns a sampler bound to that snapshot through the completions API (`/inference/v1/completions`). The snapshot selects the weights to serve; it is not a prompt-cache affinity key. The sampler runs in the same session, so there's no deployment to create or hot-load. Serverless and dedicated sampling share the [`DeploymentSampler` request contract](/fine-tuning/training-api/reference/deployment-sampler).

### Sampling with prompt caching

Check [Inference for RL rollouts](https://docs.fireworks.ai/guides/rollout-inference#inference-for-rl-rollouts) for guidance on session affinity and KV-cache behavior. Its **Training SDK** tab shows how to pass a stable trajectory ID through `DeploymentSampler`.

Install SDK version 1.2.9 or later and restart the client process after upgrading:

```bash theme={null}
pip install --upgrade "fireworks-ai[training]>=1.2.9"
```

<Warning>
  SDK versions 1.2.0 through 1.2.8 derived a shared session-affinity value from the sampler checkpoint, overriding per-request session values. Upgrade before using per-request affinity.
</Warning>

For batched rollout generation, see [`DeploymentSampler`: RL rollout sampling](/fine-tuning/training-api/reference/deployment-sampler#rl-rollout-sampling) for the two-samples-per-prompt pattern and `n=1` behavior.

## Quickstart

### Step 1: Create a key, install, and authenticate

Create an API key in the [Fireworks dashboard](https://app.fireworks.ai/settings/users/api-keys) (click **Create API key** and store it somewhere safe), or run `firectl api-key create`. Then install the SDK and export the key:

```bash theme={null}
git clone https://github.com/fw-ai/cookbook
pip install -e ./cookbook/training
export FIREWORKS_API_KEY="fw_..."   # the key you just created
```

### Step 2: Run the complete serverless RL example

The cookbook includes its dataset, reward, loop, metrics, and cleanup behavior:

```bash theme={null}
cd cookbook/training
python -m examples.serverless_rl.countdown_rl
```

The default is a real paid run. For a cheaper smoke test, fork
`examples/serverless_rl/countdown_rl.py` and reduce `steps`, `group_size`,
`prompt_groups_per_step`, and `max_sample_tokens` before execution.

The remaining snippets explain the core calls used by that runnable example.

### Step 3: Connect to the serverless session

```python theme={null}
import os
from fireworks.training.sdk import FiretitanServiceClient

service = FiretitanServiceClient(
    api_key=os.environ["FIREWORKS_API_KEY"],
    base_url="https://api.fireworks.ai/training/v1/serverless",
)

base_model = "accounts/fireworks/models/qwen3p6-27b"
max_seq_len = 131072  # enforce this when rendering prompts and training datums
training_client = service.create_lora_training_client(base_model=base_model, rank=8)

print(f"session={service.training_session_id} run={training_client.run_id}")
```

This doubles as your setup check: if it prints a session id beginning with `ts-`, serverless routing is working. If the base URL does not end in `/training/v1/serverless`, sampling errors out.

### Step 4: Train, checkpoint, and sample

The complete example defines `datums`, `tinker`, `tokenizer`, `prompt`, and
`params`, and rejects prompt plus completion lengths above `max_seq_len`. The
excerpt below shows the operation order:

```python theme={null}
# One optimizer step
training_client.forward_backward(datums, "importance_sampling").result()
training_client.optim_step(
    tinker.AdamParams(learning_rate=2.5e-5, beta1=0.9, beta2=0.95, eps=1e-8, weight_decay=0.0)
).result()

# Save weights and open a sampler bound to that exact snapshot
snapshot = training_client.save_weights_for_sampler("step-0001").result().path
sampler = service.create_sampling_client(model_path=snapshot, tokenizer=tokenizer)
completions = sampler.sample(
    prompt=prompt,
    num_samples=8,
    sampling_params=params,
).result()
sampler.close()
```

<Warning>
  * Remote training and sampling operations return future-like results. Call `.result()` on operations that return one, or failures can be missed.
  * The tokenizer you pass must match `base_model`. It renders prompts and decodes sampled tokens client-side, so a mismatch silently corrupts your rewards.
  * Give each checkpoint a distinct `name`. Saving twice with the same name currently resolves to the same snapshot path.
</Warning>

## Reinforcement learning example

The end-to-end serverless RL pattern is the standard GRPO / importance-sampling loop: each step saves the current adapter, rolls out a batch of prompts through a sampler bound to that snapshot, scores completions with your reward function, turns group-relative advantages into training datums, and takes one optimizer step.

Track reward over time; improvement depends on the task, data, reward function, and configuration. Use the cookbook [`serverless_rl` example](https://github.com/fw-ai/cookbook/tree/main/training/examples/serverless_rl) for a compact synchronous loop, or [`async_rl_loop_serverless`](https://github.com/fw-ai/cookbook/blob/main/training/recipes/experiment/async_rl_loop_serverless.py) for experimental async scheduling and custom rollout functions. For a supervised loop, use cross-entropy loss. For a preference loop, use the DPO loss over chosen/rejected pairs — see [Cookbook: DPO](/fine-tuning/training-api/cookbook/dpo) for the dataset format and loss details. For the broader RL loss menu and dedicated provisioning, see the [cookbook RL recipes](/fine-tuning/training-api/cookbook/rl).

## Evaluating serverless checkpoints

There is no serverless chat endpoint for your adapter. To evaluate it, open a **sampling client** bound to a sampler checkpoint. `sampler.sample()` generates completions from that checkpoint, which you can score with your own metric.

Use the **sampler checkpoint** returned by `save_weights_for_sampler`, not a promoted model resource (`accounts/<ACCOUNT_ID>/models/<FINE_TUNED_MODEL_ID>`). Promoted models are for on-demand deployment and cannot be passed to `create_sampling_client`.

The following save-and-sample sequence comes from the [quickstart](#step-4-train-checkpoint-and-sample) and the cookbook [`serverless_rl` example](https://github.com/fw-ai/cookbook/blob/main/training/examples/serverless_rl/countdown_rl.py). Set up `prompt`, `tokenizer`, and `params` as shown in that example.

```python theme={null}
snapshot = training_client.save_weights_for_sampler("eval").result().path
sampler = service.create_sampling_client(model_path=snapshot, tokenizer=tokenizer)
try:
    result = sampler.sample(
        prompt=prompt,
        num_samples=1,
        sampling_params=params,
    ).result()
    for seq in result.sequences or []:
        tokens = list(seq.tokens or [])
        completion = get_text_content(renderer.parse_response(tokens)[0])
        # Score `completion` against your held-out label or grader.
finally:
    sampler.close()
```

`sampler.sample(...)` is the evaluation call. Repeat it over held-out prompts, then close the sampler. The Countdown example scores with `composite_reward`; replace that with your evaluation metric.

For checkpoint and promotion details, see [Saving and loading checkpoints](#saving-and-loading-checkpoints).

<Note>
  Sampler checkpoints live in the training session. If the session is gone, you cannot open a sampling client from that checkpoint. Promote checkpoints you need to retain, then evaluate the promoted model with a [preemptible deployment](/fine-tuning/evaluating-fine-tuned-models).
</Note>

## Saving and loading checkpoints

Serverless training writes **two different kinds of checkpoint**, and they are not interchangeable:

|                                                            | **Training checkpoint**                 | **Sampler checkpoint**           |
| ---------------------------------------------------------- | --------------------------------------- | -------------------------------- |
| Saved by                                                   | `save_state(name)`                      | `save_weights_for_sampler(name)` |
| Contains                                                   | Adapter weights **and** optimizer state | Current adapter weights only     |
| Resumable — the trainer can load it back and keep training | ✅ Yes                                   | ❌ No                             |
| Promotable to a deployable model                           | ❌ No                                    | ✅ Yes                            |

<Warning>
  Only **sampler checkpoints** can be promoted to a final model, and only **training checkpoints** can be loaded back by the trainer to resume. A training checkpoint cannot be served or promoted; a sampler checkpoint cannot restore training state (weights + optimizer).
</Warning>

Checkpoint storage is included during private preview.

### Save and resume training checkpoints

Save training checkpoints periodically so an interrupted run can continue:

```python theme={null}
# Save a training checkpoint (adapter weights + optimizer) for resume
training_client.save_state("step-0100").result()

# Resume inside the same run: restores weights and optimizer state
training_client.load_state_with_optimizer("step-0100").result()

# Weights-only load (optimizer resets to zero — a warm start)
training_client.load_state("step-0100").result()
```

Save and load calls return futures — call `.result()` to block and surface failures. `save_state` also accepts a `timeout` to bound the wait. Give each checkpoint a distinct name: overwriting an existing name (`overwrite=True`) is not supported.

To see what checkpoints a run has saved, use the session-scoped control-plane list in [Promote a sampler checkpoint to a model](#promote-a-sampler-checkpoint-to-a-model) below — the trainer-local `training_client.list_checkpoints()` is not routed on the serverless surface.

### Resume a new run from a training checkpoint

To fork a new run from a previous run's training checkpoint, use `create_training_client_from_state` on the service client. The SDK reads the base model and LoRA configuration from the checkpoint itself, creates a fresh run, and loads the saved state:

```python theme={null}
# Continue a previous run's training checkpoint in a new run (weights + optimizer)
resumed_client = service.create_training_client_from_state_with_optimizer(
    "<account>/<run-id>/step-0100",
)

# Or warm start from those weights with a fresh optimizer
resumed_client = service.create_training_client_from_state(
    "<account>/<run-id>/step-0100",
)
```

Checkpoint references are fully qualified as `<account>/<run-id>/<checkpoint-name>`, where `run-id` is the previous run's `training_client.run_id` (`run-<hex>`). `weights_access_token` is not supported — load checkpoints accessible to your API key.

### Promote a sampler checkpoint to a model

Promotion turns a **sampler checkpoint** into a deployable Fireworks model (a standard LoRA addon model). Training checkpoints are not promotable — save a sampler checkpoint first.

<Steps>
  <Step title="Save a sampler checkpoint during training">
    ```python theme={null}
    snapshot = training_client.save_weights_for_sampler("final").result().path
    ```

    Save one at the end of training, and at any intermediate step you may want to deploy later.
  </Step>

  <Step title="List the session's checkpoints">
    Listing and promotion are session-scoped control-plane operations on `FireworksClient` (against the regular API gateway, not the serverless base URL). The session resource name is available as `service.training_session_name`:

    ```python theme={null}
    import os
    from datetime import datetime
    from fireworks.training.sdk import FireworksClient

    fw_client = FireworksClient(api_key=os.environ["FIREWORKS_API_KEY"])

    rows = fw_client.list_training_session_checkpoints(service.training_session_name)
    target = max(
        (row for row in rows if row.get("promotable")),
        key=lambda row: datetime.fromisoformat(row["createTime"].replace("Z", "+00:00")),
    )
    ```

    Each row carries `name` (the full 4-segment resource name), `checkpointName`, `checkpointType`, `promotable`, and `createTime`. Two things to know about the values:

    * `checkpointName` is the server-side checkpoint id, **not** the bare name you passed: it is prefixed with the source run id and, for sampler checkpoints, suffixed with an 8-hex-char session id — a save named `final` surfaces as `run-<hex>-final-<8hex>`. Select rows on `promotable` + `createTime` as above, or with a prefix/substring test — never by equality with your logical name.
    * `checkpointType` is a server enum string: `CHECKPOINT_TYPE_TRAINING_LORA` for training checkpoints, `CHECKPOINT_TYPE_INFERENCE_LORA` for sampler checkpoints. Treat it as opaque and filter on `promotable`, which is authoritative.
  </Step>

  <Step title="Promote the checkpoint">
    ```python theme={null}
    model = fw_client.promote_session_checkpoint(
        name=target["name"],  # accounts/<a>/trainingSessions/
/checkpoints/<c>
        output_model_id="my-serverless-lora",
        base_model="accounts/fireworks/models/qwen3p6-27b",
    )
    ```

    `output_model_id` must be 1-63 characters of lowercase a-z, 0-9, and hyphens. The promoted model appears in your account's model list like any other fine-tuned model.
  </Step>
</Steps>

<Note>
  Session-scoped list and promote require the training session and its bound trainer to still exist. Once the session is deleted or its trainer is drained, both calls return `NOT_FOUND` and the checkpoints are no longer reachable through this API — promote any checkpoint you want to keep before the session is torn down. Cross-run training-checkpoint resume (above) is resolved per run and is not subject to this limit.
</Note>

### Deploy the promoted model to production

A promoted model deploys like any LoRA model fine-tuned on Fireworks: with **live merge**, Fireworks merges the adapter into the base weights at deployment time, so the deployment performs identically to the base model.

<Warning>
  Fine-tuned LoRA models can only be deployed to **on-demand (dedicated) deployments**. Serverless per-token serving of your own fine-tuned LoRA is not available.
</Warning>

Deploy the promoted model directly:

```bash theme={null}
firectl deployment create "accounts/<account-id>/models/my-serverless-lora"
```

Then send requests with the model name:

```python theme={null}
from fireworks import Fireworks

client = Fireworks()
response = client.chat.completions.create(
    model="accounts/<account-id>/models/my-serverless-lora",
    messages=[{"role": "user", "content": "Hello!"}],
)
```

For deployment configuration, performance, and troubleshooting, see [Deploying fine-tuned models](/fine-tuning/deploying-loras).

For the full SDK-level checkpoint reference (base/delta sampler types, weight sync, and cross-job resolution), see [Dedicated Training](/fine-tuning/training-api/dedicated#saving-and-loading) and the [Cookbook Reference](/fine-tuning/training-api/cookbook/reference#checkpoints).

## Pricing

Serverless training is billed per token, across three meters: prefill, sample, and train. Current rates per model are in [Models](#models) below. Available models, meter definitions, and rates can change during private preview, so verify current availability and [pricing](https://fireworks.ai/pricing) before launch.

## Supported models and limits

### Models

Serverless models are not selected by shape. You attach to a shared, always-on trainer pool with a base model and your own `max_seq_len`, and pay only for the tokens you prefill, sample, and train. Serverless is LoRA only, for SFT, DPO, and RL.

<ServerlessModelsTable />

* Checkpoint storage for serverless models is included during private preview.
* The serverless model catalog evolves, and other frontier models are coming soon. For full-parameter training, ORPO, distillation, or a model not on that list, use [Dedicated Training](/fine-tuning/training-api/dedicated).
* For per-model availability across all training surfaces, see the [Models](/fine-tuning/models) catalog.

#### What the meters mean

Serverless training bills three separate token meters, so a run's cost depends on the mix of rollout and training tokens your loop generates:

| Meter       | What it counts                                                                                                                          |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| **Prefill** | Prompt tokens processed by the sampler before generation. Repeated prefixes hit the cache and bill at the cached rate, an 80% discount. |
| **Sample**  | Tokens generated by the sampler during rollouts.                                                                                        |
| **Train**   | Tokens passed through `forward_backward`, i.e. the datums you train on.                                                                 |

### Capacity and rate limits (private preview)

* **Concurrent runs:** The default quota is 8, although account overrides may differ. Slots are released when runs become terminal after session expiration.
* **Request and token limits:** Contact Fireworks for the current limits on your account.
* **Shared-pool capacity:** if the pool is full, `create_lora_training_client` returns an out-of-capacity error; retry, or switch to the dedicated path.

### Behavior to know

* **Set `max_seq_len` explicitly.** Serverless has no dedicated instance to infer sequence length from.
* **Cross-run checkpoint resume.** A training checkpoint can be resumed inside the same run (`load_state_with_optimizer`), or forked into a new run with `create_training_client_from_state` / `create_training_client_from_state_with_optimizer` using a fully qualified `<account>/<run-id>/<checkpoint-name>` reference. See [Saving and loading checkpoints](#saving-and-loading-checkpoints).
* **Serving your trained adapter.** Sample in-session during the run. To serve afterward, [promote a sampler checkpoint to a model](#promote-a-sampler-checkpoint-to-a-model) and [deploy it on an on-demand dedicated deployment](#deploy-the-promoted-model-to-production); serverless per-token serving of your own fine-tuned LoRA is not available.

## Video walkthrough: Train a prompt router

This walkthrough fine-tunes Qwen 3.5 9B with LoRA SFT to classify prompts and route them to a small or large model. It covers the local Python loop, pooled serverless trainer, in-session evaluation, and the before-and-after comparison.

<Frame>
  <iframe title="Train a prompt router with Fireworks Serverless Training" />
</Frame>

<Card title="Open the serverless prompt-router notebook" icon="github" href="https://github.com/fw-ai/cookbook/blob/main/training/case-studies/sft_prompt_router/prompt_router_serverless.ipynb">
  Follow the complete Cookbook example for dataset preparation, LoRA SFT, sampling, and evaluation.
</Card>

## Next steps

* [Serverless RL cookbook example](https://github.com/fw-ai/cookbook/tree/main/training/examples/serverless_rl): runnable serverless loop
* [Async serverless RL recipe](https://github.com/fw-ai/cookbook/blob/main/training/recipes/experiment/async_rl_loop_serverless.py): experimental rollout-function loop with rollout/training overlap
* [Choose infrastructure](/fine-tuning/training-api/introduction#infrastructure): compare serverless and dedicated
* [Dedicated Training](/fine-tuning/training-api/dedicated): the provisioned path from setup through teardown
* [Training and Sampling](/fine-tuning/training-api/dedicated#training-and-sampling): dedicated lifecycle internals
* [Loss Functions](/fine-tuning/training-api/dedicated#loss-functions): built-in and custom losses
* [The Cookbook](/fine-tuning/training-api/cookbook/overview): ready-to-run recipes, including [`serverless_rl`](https://github.com/fw-ai/cookbook/tree/main/training/examples/serverless_rl)


# Training Shapes
Source: https://docs.fireworks.ai/fine-tuning/training-api/training-shapes

Pre-configured GPU and model training profiles that simplify distributed training setup.

In practice, a training shape is the user-facing launch input for trainer jobs. Most users only need to choose a training shape ID such as `accounts/fireworks/trainingShapes/qwen3-8b-128k` and pass it to the API as `training_shape_id`.

The `fireworks` account is the shared public shape catalog. Shapes published under `accounts/fireworks/trainingShapes/<shape>` can be referenced by all users. [See the full training shapes catalog here](/fine-tuning/models#model-availability).

You do not need to know the versioned shape reference, image tag, GPU layout, or linked deployment shape ahead of time. The API resolves those details internally.

## Concepts

**Training shape.** A validated trainer configuration for one base model and one training method. It fixes the GPU layout for each trainer replica and the training context limit for that model and method. Shapes exist so you do not have to size distributed training yourself.

**Shape ID vs. shape version.** The shape ID is the stable name you pass, such as `accounts/fireworks/trainingShapes/qwen3-8b-128k`. Behind it sits a series of pinned versions holding the exact GPU counts, context limit, and trainer image. You pass the ID; the API resolves the validated version at launch.

**Training shape vs. deployment shape.** A training shape configures the trainer. Each one links to a deployment shape that configures the inference instance used for RL rollouts / sampling, and on the shape path that linked shape owns the deployment's GPU type, node count, and serving engine configuration. You normally do not name a deployment shape at all, because the API reads it from the training shape's `deploymentShapeVersion`.

**Training method.** Each shape declares one training method, either LoRA or full-parameter, and a model may offer both. Your `lora_rank` must agree with the method the shape declares: `0` for a full-parameter shape, a positive integer for a LoRA shape. These are not two independent settings. Changing `lora_rank` does not convert a shape to the other method, because the method is part of the profile the shape was validated for and it affects the GPU layout and context limit the shape carries.

Read the method off the shape card before you launch. For the same model, the LoRA shape and the full-parameter shape often differ in both GPU count and maximum context, so the two are rarely interchangeable.

## What You Need To Know

For most users, the workflow is:

1. Pick a training shape ID from the per-model shape cards in the [model catalog](/fine-tuning/models#model-availability). In most cases this should be the full shared path `accounts/fireworks/trainingShapes/<shape>`.
2. Pass it as `training_shape_id` to a cookbook recipe's `TrainerConfig`, or to `FiretitanServiceClient.from_firetitan_config(...)`.
3. Let the API resolve the pinned shape version and linked deployment shape.

The shape ID is normally the only shape-specific value you set. Everything else about the shape is resolved for you.

Whether you can skip the shape ID depends on which of the two paths you use:

* **Cookbook recipe.** You can leave `training_shape_id` unset, and the recipe picks a validated shape for you from your `base_model`, `lora_rank`, and `max_seq_len`.
* **`FiretitanServiceClient` directly.** Set the shape ID yourself. The recipe layer is what does the picking, so there is nothing to pick for you here.

On either path, set the shape ID yourself when you need a specific GPU layout or context limit.

## What A Training Shape Controls

A validated shape version can pin every field below. Together these are the launch profile, and none of them are yours to set. The hardware fields size one trainer replica: `acceleratorCount` × `nodeCount` is the GPU count for a single replica.

| Field                       | What it pins                                                                                                                                                             |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `baseModel`                 | The single base model this shape is validated for. A shape is not portable to another model.                                                                             |
| `trainerMode`               | The training method, shown as Full-Param or LoRA in the catalog.                                                                                                         |
| `acceleratorType`           | The GPU type, such as B200 or B300.                                                                                                                                      |
| `acceleratorCount`          | GPUs per node.                                                                                                                                                           |
| `nodeCount`                 | Nodes per trainer replica.                                                                                                                                               |
| `minTotalAcceleratorCount`  | The smallest total world size, across all data-parallel replicas, that the shape has been validated at, when validation recorded one.                                    |
| `trainerShardingScheme`     | The sharding and parallelism profile validated for the trainer launch.                                                                                                   |
| `baseModelWeightPrecision`  | The precision for base weights during training, when the shape sets one. `BFLOAT16` applies no quantization; `FP8`, `FP4_FP8`, `INT8`, and `NF4` are quantized variants. |
| `maxSupportedContextLength` | The validated training context limit for this model and method.                                                                                                          |
| `trainerImageTag`           | The validated trainer runtime image.                                                                                                                                     |
| `deploymentShapeVersion`    | The linked deployment shape version used for rollouts and sampling.                                                                                                      |

The shape resource also carries metadata that does not affect a launch: `name`, `displayName`, `description`, `createTime`, `updateTime`, `modelType`, and `parameterCount`.

Total GPUs for a run are the shape's per-replica GPUs multiplied by your replica count, so a 4-GPU shape launched with `replica_count=2` consumes 8 GPUs.

For example, `qwen3-8b-128k` is the full-parameter shape for Qwen 3 8B. Below is version `vnuk8fdq` in full.

| Field                       | Value in version `vnuk8fdq`                                                 |
| --------------------------- | --------------------------------------------------------------------------- |
| `name`                      | `accounts/fireworks/trainingShapes/qwen3-8b-128k`                           |
| `baseModel`                 | `accounts/fireworks/models/qwen3-8b`                                        |
| `trainerMode`               | `POLICY_TRAINER`, shown as Full-Param in the catalog. Use `lora_rank=0`.    |
| `acceleratorType`           | `NVIDIA_B200_180GB`, shown as B200                                          |
| `acceleratorCount`          | `4`                                                                         |
| `nodeCount`                 | `1`                                                                         |
| `trainerShardingScheme`     | Tensor 1, pipeline 1, context 4, expert 1, sequence parallelism not enabled |
| `maxSupportedContextLength` | `128000`, shown as 128K (128,000 tokens)                                    |
| `baseModelWeightPrecision`  | `WEIGHT_PRECISION_UNSPECIFIED`, so this shape does not pin a precision      |
| `minTotalAcceleratorCount`  | Not set on this shape                                                       |
| `trainerImageTag`           | `0.383.0`                                                                   |
| `deploymentShapeVersion`    | `accounts/fireworks/deploymentShapes/rft-qwen3-8b/versions/gyiqbrd6`        |

To read the same fields for the version you will actually launch on:

```bash theme={null}
firectl training-shape-version get \
  accounts/fireworks/trainingShapes/qwen3-8b-128k/versions/latest
```

## What You Can And Can't Change

You can still configure normal training-loop fields such as:

* `lora_rank`, which must match the shape's training method
* `learning_rate`
* Trainer replica count (`TrainerConfig.replica_count` or `trainer_replica_count`)
* Deployment replica count (`DeployConfig.replica_count` or `replica_count`)

<Note>
  Shape-owned infra is locked on the shape path. Do not set `accelerator_type`, `accelerator_count`, `node_count`, or `custom_image_tag`; the shape supplies all four. The linked deployment shape can be overridden through `deployment_shape`, but only for deployments you manage outside the normal flow.
</Note>

Gradient accumulation is not a trainer-launch setting. To accumulate gradients, call `forward_backward...` multiple times from your client loop before a single `optim_step(...)`; see [Loss Functions](/fine-tuning/training-api/dedicated#loss-functions).

For field-level behavior and dataclass details, see the [`FiretitanServiceClient`](/fine-tuning/training-api/reference/service-client) and [Cookbook Reference](/fine-tuning/training-api/cookbook/reference).

## Using a Training Shape

At launch, the shape ID resolves in three steps:

1. **You provide the shape ID** (e.g. `accounts/fireworks/trainingShapes/qwen3-8b-128k`), with no version needed.
2. **The API resolves the latest validated version** when the service client provisions the trainer.
3. **The API applies the linked deployment shape** when you request a sampler deployment.

Pass the shape ID to the SDK-managed service client (`FiretitanServiceClient`):

```python theme={null}
from fireworks.training.sdk import FiretitanServiceClient

shape_id = "accounts/fireworks/trainingShapes/qwen3-8b-128k"

service = FiretitanServiceClient.from_firetitan_config(
    api_key=api_key,
    base_model="accounts/fireworks/models/qwen3-8b",
    training_shape_id=shape_id,
    lora_rank=0,
    create_deployment=False,
)
training_client = service.create_training_client(
    base_model="accounts/fireworks/models/qwen3-8b",
    lora_rank=0,
)
```

*Note: You do not need to hand-write a versioned `training_shape_ref` yourself. Advanced compatibility launches can still use manager-level shape refs and direct infra fields, but take that path only when you know the exact hardware and image configuration.*

## Model and shape availability

See the [Models](/fine-tuning/models) page for the searchable per-model matrix and training method support.


# Fire Pass Setup
Source: https://docs.fireworks.ai/firepass

Open weight models for personal agentic coding — Fire Pass

Fire Pass is a pass that gives you access to **Open weight models** for use in agentic coding harnesses like [Claude Code](/ecosystem/fireconnect/claude-code), [OpenCode](/ecosystem/fireconnect/opencode), Cline, Kilo Code, OpenClaw, and [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/models) — with no per-token charges for included models. Check the [Fire Pass page](https://app.fireworks.ai/fire-pass) for the latest enabled models.

<Note>
  **Fire Pass** is an experimental product. Features, availability, and
  pricing are subject to change.
</Note>

## What's Included

With an active Fire Pass you get:

* Zero per-token costs on included open weight models (a powerful reasoning model with a 1M context window, optimized for complex coding tasks)
* A NEW dedicated Fire Pass key that only works for open weight models. Use your normal Fireworks key for all other models
* Full compatibility with OpenAI- and Anthropic-compatible agentic coding tools

## Getting Started

<Steps>
  <Step title="Create an account">
    If you haven't already, sign up for a [Fireworks
    account](https://app.fireworks.ai).
  </Step>

  <Step title="Activate your pass">
    Go to the [**Fire Pass** page](https://app.fireworks.ai/fire-pass),
    review the details and enter your promo code. Your pass activates immediately.
  </Step>

  <Step title="Get a Fire Pass API key">
    Create a new dedicated Fire Pass API key and use it with any of the supported agentic harnesses below. Requests to included open weight models will no longer incur per-token charges while your pass is active.
  </Step>
</Steps>

## Using your Fire Pass

Fire Pass is designed for use with personal agentic coding harnesses. For Claude Code, OpenCode, Codex, and Pi, use [FireConnect](/ecosystem/fireconnect/overview) — it detects `fpk_...` keys and applies Fire Pass defaults automatically. Below are setup guides for other supported tools.

### General Configuration

If your tool supports custom API endpoints, you can configure it manually using these details:

* **Model ID**: `accounts/fireworks/routers/kimi-k3-fast`
* **API Key**: Generate a new Fire Pass API key (generated at [app.fireworks.ai/api-keys](https://app.fireworks.ai/api-keys))
* **Base URL (OpenAI-compatible)**: `https://api.fireworks.ai/inference/v1`
* **Base URL (Anthropic-compatible)**: `https://api.fireworks.ai/inference` (some tools may expect `/v1/messages` appended)

<AccordionGroup>
  <Accordion title="OpenClaw">
    1. **Configure the open weight model as the default model** — run the setup script with your Fireworks API key from the [**API Keys** page](https://app.fireworks.ai/api-keys). Replace `YOUR_FIREWORKS_API_KEY` with your key:

    ```bash theme={null}
    curl -fsSL https://storage.googleapis.com/fireworks-public/openclaw/setup-fireworks.sh | bash -s -- YOUR_FIREWORKS_API_KEY
    ```

    This creates `~/.openclaw/openclaw.json` with your Fireworks credentials.

    <Warning>
      Do not share your API key or commit it to version control. Use the
      placeholder above only as a pattern — substitute your real key locally.
    </Warning>

    2. **Install OpenClaw**

    <Tabs>
      <Tab title="macOS / Linux">
        ```bash theme={null}
        curl -fsSL https://openclaw.ai/install.sh | bash
        ```
      </Tab>

      <Tab title="Windows (PowerShell)">
        ```powershell theme={null}
        iwr -useb https://openclaw.ai/install.ps1 | iex
        ```
      </Tab>
    </Tabs>

    3. **Run the onboarding wizard** — in a terminal:

    ```bash theme={null}
    openclaw onboard --install-daemon
    ```

    Choose **Quickstart** mode, skip model and authentication (already configured), and accept the remaining defaults.

    4. **Open the chat UI** — run:

    ```bash theme={null}
    openclaw dashboard
    ```

    The UI opens at [http://127.0.0.1:18789](http://127.0.0.1:18789). Send a message to confirm the **open weight model** is responding.

    For more about OpenClaw, see [openclaw.ai](https://openclaw.ai).
  </Accordion>

  <Accordion title="OpenCode">
    OpenCode setup lives on the [OpenCode integration guide](/ecosystem/fireconnect/opencode):

    * [FireConnect setup](/ecosystem/fireconnect/opencode#enable-fireworks-routing) (recommended)
    * [Using Fire Pass with OpenCode](/ecosystem/fireconnect/opencode#using-fire-pass)
    * [Built-in provider connection](/ecosystem/fireconnect/opencode#built-in-provider-connection) (`/connect` in OpenCode)

    With FireConnect, Fire Pass keys default to the `kimi-fast-latest` router.
  </Accordion>

  <Accordion title="Cline">
    To configure Cline with your Fire Pass:

    1. Install the Cline extension in VS Code from [cline.bot](https://cline.bot) or the VS Code marketplace
    2. Open VS Code settings and search for "Cline" or use the Cline panel
    3. Configure the following settings:

    **API Configuration:**

    * **API Provider**: OpenAI Compatible
    * **Base URL**: `https://api.fireworks.ai/inference/v1`
    * **OpenAI Compatible API Key**: Your Fireworks API key (from [app.fireworks.ai/api-keys](https://app.fireworks.ai/api-keys))
    * **Model ID**: `accounts/fireworks/routers/kimi-k3-fast`

    **Model Configuration:**

    * **Supports Images**: Enabled (checked)
    * **Context Window Size**: 256000
    * **Max Output Tokens**: 256000
    * **Input Price / 1M tokens**: 0 (Fire Pass covers all costs)
    * **Output Price / 1M tokens**: 0 (Fire Pass covers all costs)
    * **Temperature**: 1

    4. Save your settings and start using Cline — your Fire Pass will automatically apply to requests to the turbo router
  </Accordion>

  <Accordion title="Claude Code">
    All Claude Code setup lives on the [Claude Code integration guide](/ecosystem/fireconnect/claude-code):

    * [FireConnect overview](/ecosystem/fireconnect/overview#install) (recommended)
    * [Using Fire Pass with Claude Code](/ecosystem/fireconnect/claude-code#using-fire-pass)

    With FireConnect, Fire Pass keys route all model aliases to `kimi-fast-latest`.
  </Accordion>

  <Accordion title="Kilo Code">
    To configure Kilo Code with your Fire Pass:

    1. Install the Kilo Code extension in VS Code from the [VS Code marketplace](https://marketplace.visualstudio.com/items?itemName=kilocode.Kilo-Code) or [kilocode.ai](https://kilocode.ai)
    2. Open Kilo Code and select **Bring my own Key** on the "How would you like to get started?" screen
    3. Configure the following settings:

    **API Configuration:**

    * **API Provider**: OpenAI Compatible
    * **Base URL**: `https://api.fireworks.ai/inference/v1`
    * **API Key**: Your Fireworks API key (from [app.fireworks.ai/api-keys](https://app.fireworks.ai/api-keys))
    * **Model**: `accounts/fireworks/routers/kimi-k3-fast`
      * Type or paste the router ID and select "Use custom" when it appears

    **Model Configuration:**

    * **Context Window Size**: 256000 (256K tokens)
    * **Supports images**: Enabled (checked)
    * **Input Price**: 0 (Fire Pass covers all costs)
    * **Output Price**: 0 (Fire Pass covers all costs)

    4. Save your settings and start using Kilo Code — your Fire Pass will automatically apply to requests to the turbo router
  </Accordion>

  <Accordion title="LangChain Deep Agent">
    Set `FIREWORKS_API_KEY` to your API key from [app.fireworks.ai/api-keys](https://app.fireworks.ai/api-keys). Install [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/models) with the Fireworks integration, then pass the turbo router as `fireworks:accounts/fireworks/routers/kimi-k3-fast`:

    ```bash theme={null}
    pip install deepagents langchain-fireworks
    ```

    ```python theme={null}
    from deepagents import create_deep_agent

    agent = create_deep_agent(
        model="fireworks:accounts/fireworks/routers/kimi-k3-fast",
        system_prompt="You are a helpful coding assistant.",
    )
    result = agent.invoke(
        {"messages": [{"role": "user", "content": "Hello"}]},
    )
    ```

    See the LangChain docs for [models](https://docs.langchain.com/oss/python/deepagents/models) and [customization](https://docs.langchain.com/oss/python/deepagents/customization).
  </Accordion>
</AccordionGroup>

## Terms of Use

Fire Pass is intended for **non production coding use only**. By activating Fire Pass you agree to the following:

* **Allowed**: Personal development, experimentation, and coding with agentic harnesses (Claude Code, OpenCode, Codex, Pi, Cline, Kilo Code, OpenClaw, LangChain Deep Agents, and similar tools)
* **Prohibited**: Production workloads and any use that violates the [Fireworks Terms of Service](https://fireworks.ai/terms-of-service)

<Warning>
  Violations of these terms may result in pass revocation.
</Warning>

## FAQ

<AccordionGroup>
  <Accordion title="Can I use Fire Pass with any model?">
    No. Fire Pass only covers the models that are enabled on it.
    Check the Fire Pass page for the latest enabled models.
  </Accordion>

  <Accordion title="Do I need an invite code?">
    Fire Pass is currently by invite only.
  </Accordion>

  <Accordion title="Do I need to use a special model ID?">
    Yes. You must use the specific router ID for the turbo model:
    `accounts/fireworks/routers/kimi-k3-fast`. The Fireworks billing system
    will automatically detect your active Fire Pass and zero out the cost
    for requests to this endpoint.
  </Accordion>

  <Accordion title="How will this appear on my billing dashboard?">
    Your usage of included open weight models will still be logged in your dashboard so you can track your request volume,
    but the associated cost will show as \$0.00 while your pass is active.
    You can also view your pass’s active status and expiration date directly on the [**Billing**
    page](https://app.fireworks.ai/account/billing).
  </Accordion>

  <Accordion title="Can I use other models with my agentic harness?">
    Yes. You can use any Fireworks model with your agentic harness.
    However, only included open weight model requests are covered by Fire Pass.
    Usage of other models will be billed to your account at standard per-token rates.
  </Accordion>
</AccordionGroup>


# Per-User Usage Limits
Source: https://docs.fireworks.ai/fireworks-nexus/usage-limits

Set per-user spending limits on serverless inference — account defaults and per-user overrides

Set spending limits for individual users in your account on serverless (per-token) inference. Nexus accounts start with a **\$100 per-user default** limit. You can change the account-wide default cap and override it for specific users. When a user reaches their limit, their further serverless requests are blocked until the billing period resets.

<Note>
  Per-user usage limits are available on request. Reach out to your Fireworks
  contact to enable them for your account.
</Note>

## Concepts

* **Account default cap** — the per-user spending limit applied to every user who doesn't have their own override. Nexus accounts start with a **\$100** default.
* **Per-user override** — a specific user's own cap. Takes precedence over the account default.
* **Effective limit** — the limit actually applied to a user: their override if set, otherwise the account default.
* **Current-period usage** — how much a user (and the account overall) has spent in the current billing period.

All amounts are in USD, and usage resets at the start of each billing period.

## Supported models

Per-user metering applies to **all serverless models** except **[MiniMax M2.7](https://app.fireworks.ai/models/fireworks/minimax-m2p7)**. Usage on MiniMax M2.7 does not count toward a user's limit and won't trigger blocking.

## Who can do what

* **Account admins** — view and manage everything: update the default cap, set/clear per-user overrides, and view every user's usage.
* **Members (non-admin)** — view the account-level limits and **their own** usage and limit. Members cannot view other users' limits, list all users, or change any limits.

Each command below is annotated with who can run it.

## In the web app

Account admins can view and update the default per-user cap and per-user overrides in the [Usage limits](https://app.fireworks.ai/settings/usage-limits) page under **Settings** in the Fireworks web app. Members can view the account default and their own usage and limit there.

## Using `firectl`

### Account-level

```bash theme={null}
# View the account default cap and account-wide usage — any member
firectl usage-limits get

# Update the default per-user cap — admin only
firectl usage-limits update --default-user-limit=200
```

The default cap applies to every user who doesn't have their own override. New Nexus accounts start with a **\$100** default; change it in [Settings → Usage limits](https://app.fireworks.ai/settings/usage-limits) or with `firectl usage-limits update`.

### Per-user

```bash theme={null}
# List every user with their usage, effective limit, and override — admin only
firectl usage-limits user list

# View a single user — admin, or that user viewing their own
firectl usage-limits user get <USER_ID>

# Give a user their own cap, overriding the account default — admin only
firectl usage-limits user set <USER_ID> --limit=500

# Remove a user's override, reverting them to the account default — admin only
firectl usage-limits user unset <USER_ID>
```

A user record shows:

* **used** — current-period spend
* **effective\_limit** — the limit applied to them (override, else account default)
* **override** — their per-user override, if any
* **exceeded\_until** — set only when the user is currently blocked; shows when the block lifts (the end of the current billing period)

### Per-user overrides

Set an override when a user needs a different cap than the account default. `firectl usage-limits user unset` removes the override and reverts that user to the account default. The account-wide default cannot be cleared — every user without an override uses it.

```bash theme={null}
# Give this user a $50 cap — admin only
firectl usage-limits user set <USER_ID> --limit=50
```

## Using the REST API

### Read

```bash theme={null}
# Account-level usage limits — any member
curl -H "Authorization: Bearer $FIREWORKS_API_KEY" \
  https://api.fireworks.ai/v1/accounts/<ACCOUNT_ID>/usageLimits

# A single user's usage limits — admin, or that user
curl -H "Authorization: Bearer $FIREWORKS_API_KEY" \
  https://api.fireworks.ai/v1/accounts/<ACCOUNT_ID>/users/<USER_ID>/usageLimits

# All users in the account — admin only
curl -H "Authorization: Bearer $FIREWORKS_API_KEY" \
  https://api.fireworks.ai/v1/accounts/<ACCOUNT_ID>/userUsageLimits
```

### Update (admin only)

Updates use `PATCH` with a field mask indicating which fields to change:

* `PATCH /v1/accounts/<ACCOUNT_ID>/usageLimits` — update `default_user_limit` (cannot be cleared)
* `PATCH /v1/accounts/<ACCOUNT_ID>/users/<USER_ID>/usageLimits` — set `limit_override` (an unset value removes the override)

`firectl`, the [web app](#in-the-web-app), and the REST API below all support the same changes.

## How enforcement works

* **A user who reaches their effective limit is blocked** from further serverless requests until the billing period resets. Blocked requests receive **HTTP 402**.
* **Limits are per billing period.** Usage and any blocks reset when the period rolls over.
* **Enforcement is near-real-time, not instantaneous.** After a user crosses their limit there is a short delay (typically a few minutes) before requests start being blocked, and a similar delay before a user is unblocked after you raise their limit. Plan around this lag rather than expecting an immediate cutoff.
* **A `$0` cap** means the user is allowed no serverless spend — they are blocked immediately.
* Only **serverless (per-token) inference on [supported models](#supported-models)** counts toward these limits. **[MiniMax M2.7](https://app.fireworks.ai/models/fireworks/minimax-m2p7)** is excluded. Dedicated deployment (GPU-hour) usage is not metered per user here.

## FAQ

<AccordionGroup>
  <Accordion title="What counts toward a user's limit?">
    Serverless per-token inference spend in the current billing period.
  </Accordion>

  <Accordion title="What does a blocked user see?">
    Their serverless requests return HTTP `402` until the billing period resets or their limit is raised.
  </Accordion>

  <Accordion title="Can a user check their own usage?">
    Yes — a user can read their own usage and limit; they just can't see other users' limits or change any limits.
  </Accordion>

  <Accordion title="Do limits carry over between periods?">
    No. Usage and blocks reset each billing period.
  </Accordion>
</AccordionGroup>


# Concepts
Source: https://docs.fireworks.ai/getting-started/concepts

This document outlines basic Fireworks AI concepts.

This page outlines core Fireworks resources and ideas. For definitions of technical terms (inference, LoRA, token, and many others), see the [**Glossary**](/getting-started/glossary).

## Resources

### Account

Your account is the top-level resource under which other resources are located. Quotas and billing are enforced at the account level, so usage for all users in an account contribute to the same quotas and bill.

* For developer accounts, the account ID is auto-generated from the email address used to sign up.
* Enterprise accounts can optionally choose a custom, unique account ID.

### User

A user is an email address associated with an account. Each user is assigned a role (such as Admin, User, Contributor, or Inference User) that determines their level of access to resources within the account.

### Models and model types

A model is a set of model weights and metadata associated with the model. Each model has a [**globally unique name**](/getting-started/concepts#resource-names-and-ids) of the form `accounts/<ACCOUNT_ID>/models/<MODEL_ID>`. There are two types of models:

**Base models:** A base model consists of the full set of model weights, including models pre-trained from scratch and full fine-tunes.

* Fireworks has a library of common base models that can be used for [**serverless inference**](/models/overview#serverless-inference) as well as [**dedicated deployments**](/models/overview#dedicated-deployments). Model IDs for these models are pre-populated. For example, `llama-v3p1-70b-instruct` is the model ID for the Llama 3.1 70B model that Fireworks provides. The ID for each model can be found on its page ([**example**](https://app.fireworks.ai/models/fireworks/qwen3-coder-480b-a35b-instruct))
* Users can also [upload their own](/models/uploading-custom-models) custom base models and specify model IDs.

**LoRA (low-rank adaptation) addons:** A LoRA addon is a small, fine-tuned model that significantly reduces the amount of memory required to deploy compared to a fully fine-tuned model. Fireworks supports [**training**](/fine-tuning/finetuning-intro), [**uploading**](/models/uploading-custom-models#importing-fine-tuned-models), and [**serving**](/fine-tuning/fine-tuning-models#deploying-a-fine-tuned-model) LoRA addons. LoRA addons must be deployed on a dedicated deployment for its corresponding base model. Model IDs for LoRAs can be either auto-generated or user-specified.

#### A Note on API Model Metadata

When retrieving model details via the API, a model may be listed with both `supportsServerless: true` and `supportsLora: true`. This indicates that the base model is available for serverless inference, AND that the model architecture supports fine-tuning with LoRA.

However, these two features are mutually exclusive in deployment. The `supportsServerless` flag applies **only** to the base model. A LoRA addon fine-tuned from that base model **cannot** be deployed serverlessly and requires a dedicated (on-demand) deployment.

### Deployments and deployment types

A model must be deployed before it can be used for inference. A deployment is a collection (one or more) model servers that host one base model and optionally one or more LoRA addons.

Fireworks supports two types of deployments:

* **Serverless deployments:** Fireworks hosts popular base models on shared "serverless" deployments. Users pay-per-token to query these models and do not need to configure GPUs. See our [Quickstart - Serverless](/getting-started/quickstart) guide to get started.
* **Dedicated deployments:** Dedicated deployments enable users to configure private deployments with a wide array of hardware (see [on-demand deployments guide](/guides/ondemand-deployments)). Dedicated deployments give users performance guarantees and the most flexibility and control over what models can be deployed. Both LoRA addons and base models can be deployed to dedicated deployments. Dedicated deployments are billed by a GPU-second basis (see [**pricing**](https://fireworks.ai/pricing#ondemand) page).

See the [**Querying text models guide**](/guides/querying-text-models) for a comprehensive overview of making LLM inference.

### Deployed model

Users can specify a model to query for inference using the model name and deployment name. Alternatively, users can refer to a "deployed model" name that refers to a unique instance of a base model or LoRA addon that is loaded into a deployment. See [On-demand deployments](/guides/ondemand-deployments) guide for more.

### Dataset

A dataset is an immutable set of training examples that can be used to fine-tune a model.

### Fine-tuning job

A fine-tuning job is an offline training job that uses a dataset to train a LoRA addon model.

## Resource names and IDs

A resource name is a globally unique identifier of a resource. The format of a name also identifies the type and hierarchy of the resource, for example:

Resource IDs must satisfy the following constraints:

* Between 1 and 63 characters (inclusive)
* Consists of a-z, 0-9, and hyphen (-)
* Does not begin or end with a hyphen (-)
* Does not begin with a digit

## Control plane and data plane

The Fireworks API can be split into a control plane and a data plane.

* The **control plane** consists of APIs used for managing the lifecycle of resources. This
  includes your account, models, and deployments.
* The **data plane** consists of the APIs used for inference and the backend services that power
  them.

## Interfaces

Users can interact with Fireworks through one of many interfaces:

* The **web app** at [https://app.fireworks.ai](https://app.fireworks.ai)
* The [`firectl`](/tools-sdks/firectl/firectl) CLI
* [OpenAI compatible API](/tools-sdks/openai-compatibility)
* [Anthropic compatible API](/tools-sdks/anthropic-compatibility)
* [Python SDK](/tools-sdks/python-sdk)


# Glossary
Source: https://docs.fireworks.ai/getting-started/glossary

Definitions for key terms used across Fireworks AI documentation.

This glossary covers terms you'll encounter when working with the Fireworks AI platform — across inference, fine-tuning, deployments, security, and the API.

***

## Account & Billing

**Account**\
Your Fireworks AI organization identity. All deployments, models, fine-tuning jobs, and API keys belong to an account. Referenced as `accounts/<your-account>` in firectl and the API.

**Credit**\
The unit used for prepaid usage on Fireworks. Credits are consumed as you use inference and training resources.

**Rate limiting / 429 errors**\
A `429 Too Many Requests` response means your account or deployment has hit a concurrency or request-per-minute limit. On serverless, limits scale automatically with usage tier. On dedicated deployments, adding replicas or adjusting autoscaling increases capacity.

**Quota**\
A per-account cap on a resource — such as requests per minute (RPM), GPU hours, or concurrent replicas. Quotas can be increased by contacting support.

**ZDR (Zero Data Retention)**\
The default behavior for all open model inference on Fireworks: no prompts, completions, or request logs are stored after the response is returned. ZDR is on by default — it is not an opt-in feature.

**BYOC (Bring Your Own Cloud)**\
An enterprise option to run inference entirely within your own cloud account. Your data never touches Fireworks-managed infrastructure. Contact sales for availability.

***

## Fireworks Platform

**Serverless inference**\
Pay-per-token inference with no deployment management. Requests are routed globally across Fireworks infrastructure for lowest latency and highest availability. Serverless does not support geographic constraints — for region-specific inference, use a dedicated deployment and set placement with the **`--region`** flag on `firectl deployment create` (for example `GLOBAL`, `US`, `EUROPE`, `APAC`).

**Dedicated deployment**\
A deployment you provision and manage, with reserved GPU capacity. Gives you control over model, hardware, placement, autoscaling, and addon support. Created with `firectl deployment create`.

**Multi-region deployment**\
A dedicated deployment configured to run replicas across multiple datacenters. Enabled with `--region GLOBAL` (or a specific mega-region: `US`, `EUROPE`, `APAC`) on `firectl deployment create`. Increases availability and throughput.

**Placement**\
Controls which regions a dedicated deployment is allowed to schedule replicas in. On the CLI, set at creation time with **`--region`** (`GLOBAL`, `US`, `EUROPE`, `APAC`, or a specific region id). Cannot be changed after deployment creation — recreate the deployment to change placement. If not specified, the deployment pins to a single datacenter at creation time.

**Deployment shape**\
The hardware and precision configuration used when creating a dedicated deployment. Shapes encode GPU type, count, precision (BF16, FP8, FP4), and other settings. Specified with `--deployment-shape`. Some shapes do not support LoRA addons.

**Deployment state**\
The lifecycle status of a deployment: `CREATING`, `DEPLOYING`, `DEPLOYED`, `SCALING`, `UPDATING`, `FAILED`, `DELETING`, `DELETED`.

**Replica**\
A single instance of a deployed model. Adding replicas increases concurrency. Replicas can be scaled manually or via autoscaling rules.

**Autoscaling**\
Automatic adjustment of replica count based on traffic. Configured with scale-to-zero, minimum replicas, maximum replicas, and scale-up/down thresholds.

**firectl**\
The Fireworks command-line tool for managing models, deployments, fine-tuning jobs, and account resources.

***

## Models & Inference

**Base model**\
A foundation model available on Fireworks for inference or fine-tuning. Referenced as `accounts/fireworks/models/<model-id>`.

**Addon**\
A LoRA adapter loaded on top of a base model deployment at inference time. Enabled on a deployment with `--enable-addons`. The adapter is specified per request by passing the adapter model ID. FP8 and FP4 quantized shapes do not support addons — use a BF16 shape for LoRA addon inference.

**Multi-LoRA**\
A single base model deployment that serves multiple LoRA adapters. The adapter is selected per request. One deployment, multiple fine-tuned behaviors.

**Quantization**\
Reducing the numerical precision of model weights to decrease memory usage and increase throughput, with some quality tradeoff.

**BF16 (BFloat16)**\
A 16-bit floating point format used for model weights. Provides good quality with moderate memory usage. BF16 deployment shapes support LoRA addons.

**FP8**\
An 8-bit floating point format. Faster and more memory-efficient than BF16, with a small quality tradeoff. FP8 deployment shapes do not support LoRA addons.

**FP4**\
A 4-bit floating point format. Faster and cheaper than FP8, with a larger quality tradeoff. FP4 deployment shapes do not support LoRA addons.

**Speculative decoding**\
An inference acceleration technique where a smaller draft model proposes tokens that are verified by the main model in parallel. Reduces latency with no quality loss.

**KV cache**\
Key-value cache that stores intermediate attention computations across tokens. Reduces re-computation cost on repeated or shared prefixes. Cache hit percentage is reflected in billing. Configurable with `--kv-cache-fraction`.

**Context window**\
The maximum number of tokens (input + output) the model can process in a single request.

**`max_tokens`**\
API parameter that caps how many tokens the model will generate in a single response. Always set this explicitly in agentic workflows — without it, reasoning models and large models may generate very long outputs.

**TTFT (Time to First Token)**\
Latency from when a request is sent to when the first output token is received. Key metric for interactive applications.

**TPS (Tokens Per Second)**\
Throughput metric measuring how many output tokens are generated per second.

**Batch inference**\
Asynchronous large-scale inference via the Fireworks Batch API. Submit a JSONL file of requests; results are returned when the job completes. Lower cost than real-time inference, higher latency, no streaming.

**Streaming**\
Real-time token-by-token output delivery via server-sent events (SSE). Enabled with `stream=True`. Not available in batch inference.

**Tool calling / Function calling**\
The ability for a model to request that the client execute a function and return the result, enabling agentic and multi-step workflows.

**Structured output**\
Model output constrained to a specific JSON schema. Fireworks supports constrained decoding to guarantee valid JSON matching your schema.

**`reasoning_effort`**\
API parameter that controls how much thinking a reasoning model performs before generating its response. Set to `"none"` to disable extended reasoning (useful as a workaround for models prone to reasoning loops).

***

## Fine-tuning

**SFT (Supervised Fine-Tuning)**\
Training a model on labeled input-output pairs to adapt it to a specific task or style. Available via the Fireworks managed fine-tuning pipeline.

**LoRA (Low-Rank Adaptation)**\
A parameter-efficient fine-tuning technique that trains a small set of adapter weights rather than the full model. The adapter can be loaded on top of the base model at inference time.

**LoRA rank**\
A hyperparameter that controls the size of the LoRA adapter. Higher rank = more capacity but larger adapter and more training compute.

**GRPO (Group Relative Policy Optimization)**\
A reinforcement learning variant used in RFT. Optimizes model outputs using group-relative reward scoring rather than a separate value model.

**RFT (Reinforcement Fine-Tuning)**\
Fine-tuning using reinforcement learning signals rather than labeled examples. Trains the model to maximize a reward function — useful for tasks with verifiable outcomes such as math, code, and reasoning.

**Training shape**\
A validated trainer configuration for one base model and one training method. It fixes the GPU type and count per trainer replica, plus the maximum training context length. On the [Training API](/fine-tuning/training-api/training-shapes) you pass a shape ID such as `accounts/fireworks/trainingShapes/qwen3-8b-128k`; managed fine-tuning jobs select a compatible shape for you.

**`dcp_save_interval`**\
RFT training parameter that controls how often full training state (weights + optimizer) is checkpointed. Default is `0` (disabled). Set to a positive integer to enable full checkpoint-and-resume including optimizer state.

**Epoch**\
One complete pass through the training dataset.

**Step**\
A single gradient update during training.

**Checkpoint**\
A saved snapshot of model state (and optionally optimizer state) at a point during training.

**`ppo_kl` vs `ref_kld`**\
Two KL divergence metrics logged during GRPO/RFT training. `ppo_kl` measures divergence between current and previous policy — stays near zero with one minibatch per rollout, which is expected. `ref_kld` measures divergence from the reference/base model — this is the metric to monitor for policy drift.

***

## Deployment

**Cold start**\
The latency incurred when a deployment scales up from zero replicas or provisions a new replica.

**Scale-to-zero**\
A deployment configuration where replica count drops to zero when there is no traffic. Eliminates idle costs but introduces cold-start latency.

**GPU hours**\
The billing unit for dedicated deployment capacity. Charged per GPU per hour of replica uptime.

**Prometheus metrics**\
Per-deployment performance and utilization metrics exposed via a Prometheus-compatible endpoint. Includes TTFT, TPS, GPU utilization, queue depth, and error rates.

***

## API & SDK

**API key**\
An authentication token used to make requests to the Fireworks API. Generated in the Fireworks console under Account > API Keys.

**OpenAI-compatible API**\
The Fireworks inference API is compatible with the OpenAI Chat Completions API format. Use the OpenAI Python SDK with Fireworks by changing `base_url` and `api_key`.

**`reconnect_and_wait()`**\
SDK method for recovering a training job that has been interrupted by pod preemption or a network error. Use in your training loop to make jobs resilient to transient interruptions.

**JSONL**\
JSON Lines format — one JSON object per line. Used for batch inference input files and fine-tuning dataset uploads.

***

## Security & Compliance

**ISO 27001**\
International standard for information security management systems (ISMS). Fireworks has achieved ISO 27001 certification. Certificate available at [trust.fireworks.ai](https://trust.fireworks.ai).

**ISO 27701**\
Extension to ISO 27001 covering privacy information management. Fireworks has achieved ISO 27701 certification. Certificate available at [trust.fireworks.ai](https://trust.fireworks.ai).

**ISO 42001**\
International standard for AI management systems. Fireworks has achieved ISO 42001 certification. Certificate available at [trust.fireworks.ai](https://trust.fireworks.ai).

**SOC 2 Type II**\
An auditing standard that verifies a company's security, availability, and confidentiality controls over time. Fireworks maintains SOC 2 Type II compliance.

**HIPAA**\
U.S. healthcare data regulation. Fireworks supports HIPAA-compliant deployments for covered entities. Contact sales for Business Associate Agreement (BAA) details.

**Audit logs**\
A record of API activity on your account. Useful for security review and compliance reporting. Available via the Fireworks console and API.

**Trust Center**\
Fireworks' central repository for security documentation, compliance certificates, and data handling policies. Available at [trust.fireworks.ai](https://trust.fireworks.ai).

***

## Inference Parameters

**Temperature**\
Controls randomness in token sampling. `0.0` = deterministic. Higher values increase diversity.

**Top-p (nucleus sampling)**\
Limits token sampling to the smallest set of tokens whose cumulative probability exceeds `p`.

**Top-k**\
Limits token sampling to the top `k` most probable tokens at each step.

**Frequency penalty**\
Reduces the likelihood of tokens that have already appeared in the output. Discourages repetition.

**Presence penalty**\
Reduces the likelihood of tokens that have appeared at all in the output. Encourages the model to introduce new topics.

**Stop sequences**\
One or more strings that cause the model to stop generating when produced. Useful for controlling output format.

**System prompt**\
An instruction prepended to the conversation that sets the model's persona, task, or constraints.


# Build with Fireworks AI
Source: https://docs.fireworks.ai/getting-started/introduction

Fast inference and fine-tuning for open source models

Fireworks AI is the fastest platform for building with open source AI models. Get production-ready inference and fine-tuning with best-in-class speed, cost and quality.

## Get started in minutes

<CardGroup>
  <Card title="Start fast with Serverless" href="/getting-started/quickstart" icon="bolt">
    Use popular models instantly with pay-per-token pricing. Perfect for quality vibe testing and prototyping.
  </Card>

  <Card title="Deploy models & autoscale on dedicated GPUs" href="/getting-started/ondemand-quickstart" icon="server">
    Deploy with high performance on dedicated GPUs with fast autoscaling and minimal cold starts. Optimize deployments for speed and throughput.
  </Card>

  <Card title="Fine-tune models for best quality" href="/fine-tuning/finetuning-intro" icon="sliders">
    Boost model quality with supervised and reinforcement fine-tuning of models up to 1T+ parameters. Start training in minutes, deploy immediately.
  </Card>
</CardGroup>

<Tip>
  Not sure where to start? First, pick the right model for your use case with our [**model selection guide**](/guides/recommended-models). Then choose [**Serverless**](/getting-started/quickstart) to prototype quickly, move to [**Deployments**](/getting-started/ondemand-quickstart) to optimize and run production workloads, or use [**Fine-tuning**](/fine-tuning/finetuning-intro) to improve quality.

  New to AI or Fireworks? Look up any term in the [**Glossary**](/getting-started/glossary).

  Need help optimizing deployments, fine-tuning models, or setting up production infrastructure? [Talk to our team](https://fireworks.ai/company/contact-us) - we'll help you get the best performance and reliability.
</Tip>

## What you can build

<CardGroup>
  <Card title="100+ Supported Models" href="https://fireworks.ai/models" icon="books">
    Text, vision, audio, image, and embeddings
  </Card>

  <Card title="Migrate from OpenAI" href="/tools-sdks/openai-compatibility" icon="arrow-right-arrow-left">
    Drop-in replacement for inference and fine-tuning — same API, same SFT data format
  </Card>

  <Card title="Function Calling" href="/guides/function-calling" icon="function">
    Connect models to tools and APIs
  </Card>

  <Card title="Structured Outputs" href="/structured-responses/structured-response-formatting" icon="brackets-curly">
    Reliable JSON responses for agentic workflows
  </Card>

  <Card title="Vision Models" href="/guides/querying-vision-language-models" icon="eye">
    Analyze images and documents
  </Card>

  <Card title="Embeddings & Reranking" href="/guides/querying-embeddings-models" icon="vector">
    Use embeddings & reranking in search & context retrieval
  </Card>

  <Card title="Batch Inference" href="/guides/batch-inference" icon="vector">
    Run async inference jobs at scale, faster and cheaper
  </Card>
</CardGroup>

## Resources & help

<CardGroup>
  <Card title="Which model should I use?" href="/guides/recommended-models" icon="compass">
    Find the best model for your use case
  </Card>

  <Card title="Cookbook" href="https://github.com/fw-ai/cookbook" icon="book-open">
    Code examples and tutorials
  </Card>

  <Card title="API Reference" href="/api-reference/introduction" icon="code">
    Complete API documentation
  </Card>

  <Card title="Discord Community" href="https://discord.gg/fireworks-ai" icon="discord">
    Ask questions and get help from developers
  </Card>

  <Card title="Security & Compliance" href="https://trust.fireworks.ai/" icon="shield-check">
    SOC 2, HIPAA, and audit reports
  </Card>

  <Card title="System Status" href="https://status.fireworks.ai/" icon="signal">
    Check service uptime
  </Card>

  <Card title="Talk to Sales" href="https://fireworks.ai/company/contact-us" icon="building">
    Talk to our team
  </Card>
</CardGroup>


# Deployments Quickstart
Source: https://docs.fireworks.ai/getting-started/ondemand-quickstart

Deploy models on dedicated GPUs in minutes

On-demand deployments are dedicated GPUs that give you better performance, no rate limits, fast autoscaling, and a wider selection of models than serverless. This quickstart will help you spin up your first on-demand deployment in minutes.

## Step 1: Create and export an API key

Before you begin, create an API key in the [Fireworks dashboard](https://app.fireworks.ai/settings/users/api-keys). Click **Create API key** and store it in a safe location.

Once you have your API key, export it as an environment variable in your terminal:

<Tabs>
  <Tab title="macOS / Linux">
    ```bash theme={null}
    export FIREWORKS_API_KEY="your_api_key_here"
    ```
  </Tab>

  <Tab title="Windows">
    ```powershell theme={null}
    setx FIREWORKS_API_KEY "your_api_key_here"
    ```
  </Tab>
</Tabs>

## Step 2: Install the CLI

To create and manage on-demand deployments, you'll need the `firectl` CLI tool. Install it using one of the following methods, based on your platform:

<CodeGroup>
  ```bash homebrew theme={null}
  brew tap fw-ai/firectl
  brew install firectl

  # If you encounter a failed SHA256 check, try first running
  brew update
  ```

  ```bash macOS (Apple Silicon) theme={null}
  curl https://storage.googleapis.com/fireworks-public/firectl/stable/darwin-arm64.gz -o firectl.gz
  gzip -d firectl.gz && chmod a+x firectl
  sudo mv firectl /usr/local/bin/firectl
  sudo chown root: /usr/local/bin/firectl
  ```

  ```bash macOS (x86_64) theme={null}
  curl https://storage.googleapis.com/fireworks-public/firectl/stable/darwin-amd64.gz -o firectl.gz
  gzip -d firectl.gz && chmod a+x firectl
  sudo mv firectl /usr/local/bin/firectl
  sudo chown root: /usr/local/bin/firectl
  ```

  ```bash Linux  (x86_64) theme={null}
  wget -O firectl.gz https://storage.googleapis.com/fireworks-public/firectl/stable/linux-amd64.gz
  gunzip firectl.gz
  sudo install -o root -g root -m 0755 firectl /usr/local/bin/firectl
  ```

  ```Text Windows (64 bit) theme={null}
  wget -L https://storage.googleapis.com/fireworks-public/firectl/stable/firectl.exe
  ```
</CodeGroup>

Then, sign in:

```bash theme={null}
firectl signin
```

## Step 3: Create a deployment

This command will create a deployment of GPT OSS 120B optimized for speed. It will take a few minutes to complete. The resulting deployment will scale up to 1 replica.

```bash theme={null}
firectl deployment create accounts/fireworks/models/gpt-oss-120b \
        --deployment-shape fast \
        --scale-down-window 5m \
        --scale-up-window 30s \
        --min-replica-count 0 \
        --max-replica-count 1 \
        --scale-to-zero-window 5m \
        --wait
```

<Tip>
  `fast` is called a [deployment shape](/guides/ondemand-deployments#deployment-shapes), which is a pre-configured deployment template created by the Fireworks team that sets sensible defaults for most deployment options (such as hardware type).

  You can also pass `throughput` or `cost` to `--deployment-shape`:

  * `throughput` creates a deployment that trades off latency for lower cost-per-token at scale
  * `cost` creates a deployment that trades off latency and throughput for lowest cost-per-token at small scale, usually for early experimentation and prototyping

  While we recommend using a deployment shape, you are also free to pass your own configuration to the deployment via our [deployment guide](/guides/ondemand-deployments).
</Tip>

The response will look like this:

```bash theme={null}
Name: accounts/<YOUR ACCOUNT ID>/deployments/<DEPLOYMENT ID>
Create Time: <CREATION_TIME>
Expire Time: <EXPIRATION_TIME>
Created By: <YOUR EMAIL>
State: CREATING
Status: OK
Min Replica Count: 0
Max Replica Count: 1
Desired Replica Count: 0
Replica Count: 0
Autoscaling Policy:
  Scale Up Window: 30s
  Scale Down Window: 5m0s
  Scale To Zero Window: 5m0s
Base Model: accounts/fireworks/models/gpt-oss-120b
...other fields...
```

Take note of the `Name:` field in the response, as it will be used in the next step to query your deployment.

[Learn more about deployment options→](/guides/ondemand-deployments)

[Learn more about autoscaling options→](/guides/ondemand-deployments#autoscaling)

## Step 4: Query your deployment

Now you can query your on-demand deployment using the same API as serverless models, but using your dedicated deployment. Replace `<DEPLOYMENT_NAME>` in the below snippets with the value from the `Name:` field in the previous step:

<Tabs>
  <Tab title="Python (Fireworks SDK)">
    Install the [Fireworks Python SDK](/tools-sdks/python-sdk):

    <Note>
      The SDK is currently in alpha. Use the `--pre` flag when installing to get the latest version.
    </Note>

    <CodeGroup>
      ```bash pip theme={null}
      pip install --pre fireworks-ai
      ```

      ```bash poetry theme={null}
      poetry add --pre fireworks-ai
      ```

      ```bash uv theme={null}
      uv add --pre fireworks-ai
      ```
    </CodeGroup>

    Then make your first on-demand API call:

    ```python theme={null}
    from fireworks import Fireworks

    client = Fireworks()

    response = client.chat.completions.create(
        model="accounts/fireworks/models/gpt-oss-120b#<DEPLOYMENT_NAME>",
        messages=[{
            "role": "user",
            "content": "Explain quantum computing in simple terms",
        }],
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="Python (OpenAI SDK)">
    ```python theme={null}
    import os
    from openai import OpenAI

    client = OpenAI(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference/v1"
    )

    response = client.chat.completions.create(
        model="<DEPLOYMENT_NAME>",
        messages=[{
            "role": "user",
            "content": "Explain quantum computing in simple terms",
        }],
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    import OpenAI from "openai";

    const client = new OpenAI({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference/v1",
    });

    const response = await client.chat.completions.create({
      model: "<DEPLOYMENT_NAME>",
      messages: [
        {
          role: "user",
          content: "Explain quantum computing in simple terms",
        },
      ],
    });

    console.log(response.choices[0].message.content);
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    curl https://api.fireworks.ai/inference/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $FIREWORKS_API_KEY" \
      -d '{
        "model": "<DEPLOYMENT_NAME>",
        "messages": [
          {
            "role": "user",
            "content": "Explain quantum computing in simple terms"
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

The examples from the Serverless quickstart will work with this deployment as well, just replace the model string with the deployment-specific model string from above.

[Serverless quickstart→](/getting-started/quickstart)

## Common use cases

### Autoscale based on requests per second

```bash theme={null}
firectl deployment create accounts/fireworks/models/gpt-oss-120b \
        --deployment-shape fast \
        --scale-down-window 5m \
        --scale-up-window 30s \
        --scale-to-zero-window 5m \
        --min-replica-count 0 \
        --max-replica-count 4 \
        --load-targets requests_per_second=5 \
        --wait
```

### Autoscale based on concurrent requests

```bash theme={null}
firectl deployment create accounts/fireworks/models/gpt-oss-120b \
        --deployment-shape fast \
        --scale-down-window 5m \
        --scale-up-window 30s \
        --scale-to-zero-window 5m \
        --min-replica-count 0 \
        --max-replica-count 4 \
        --load-targets concurrent_requests=5 \
        --wait
```

## Next steps

Ready to scale to production, explore other modalities, or customize your models?

<CardGroup>
  <Card title="Upload a custom model" href="/models/uploading-custom-models" icon="server">
    Bring your own model and deploy it on Fireworks
  </Card>

  <Card title="Fine-tune Models" href="/fine-tuning/finetuning-intro" icon="sliders">
    Improve model quality with supervised and reinforcement learning
  </Card>

  <Card title="Embeddings & Reranking" href="/guides/querying-embeddings-models" icon="brackets-curly">
    Use embeddings & reranking in search & context retrieval
  </Card>

  <Card title="Batch Inference" href="/guides/batch-inference" icon="list-check">
    Run async inference jobs at scale, faster and cheaper
  </Card>

  <Card title="Browse 100+ Models" href="https://fireworks.ai/models" icon="books">
    Explore all available models across modalities
  </Card>

  <Card title="API Reference" href="/api-reference/introduction" icon="code">
    Complete API documentation
  </Card>
</CardGroup>


# Serverless Quickstart
Source: https://docs.fireworks.ai/getting-started/quickstart

Make your first Serverless API call in minutes

Serverless is the fastest way to get started with using open models. This quickstart will help you make your first API call in minutes.

## Step 1: Create and export an API key

Before you begin, create an API key in the [Fireworks dashboard](https://app.fireworks.ai/settings/users/api-keys). Click **Create API key** and store it in a safe location.

Once you have your API key, export it as an environment variable in your terminal:

<Tabs>
  <Tab title="macOS / Linux">
    ```bash theme={null}
    export FIREWORKS_API_KEY="your_api_key_here"
    ```
  </Tab>

  <Tab title="Windows">
    ```powershell theme={null}
    setx FIREWORKS_API_KEY "your_api_key_here"
    ```
  </Tab>
</Tabs>

## Step 2: Make your first Serverless API call

<Tabs>
  <Tab title="Python (Fireworks SDK)">
    Install the [Fireworks Python SDK](/tools-sdks/python-sdk):

    <Note>
      The SDK is currently in alpha. Use the `--pre` flag when installing to get the latest version.
    </Note>

    <CodeGroup>
      ```bash pip theme={null}
      pip install --pre fireworks-ai
      ```

      ```bash poetry theme={null}
      poetry add --pre fireworks-ai
      ```

      ```bash uv theme={null}
      uv add --pre fireworks-ai
      ```
    </CodeGroup>

    Then make your first Serverless API call:

    ```python theme={null}
    from fireworks import Fireworks

    client = Fireworks()

    response = client.chat.completions.create(
      model="accounts/fireworks/models/deepseek-v3p1",
      messages=[{
        "role": "user",
        "content": "Say hello in Spanish",
      }],
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="Python (OpenAI SDK)">
    Fireworks provides an OpenAI compatible endpoint. Install the [OpenAI Python SDK](https://github.com/openai/openai-python):

    ```bash theme={null}
    pip install openai
    ```

    Then make your first Serverless API call:

    ```python theme={null}
    import os
    from openai import OpenAI

    client = OpenAI(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference/v1"
    )

    response = client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[{
            "role": "user",
            "content": "Say hello in Spanish",
        }],
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="Python (Anthropic SDK)">
    Fireworks provides an Anthropic compatible endpoint. Install the [Anthropic Python SDK](https://github.com/anthropics/anthropic-sdk-python):

    ```bash theme={null}
    pip install anthropic
    ```

    Then make your first Serverless API call:

    ```python theme={null}
    import os
    import anthropic

    client = anthropic.Anthropic(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference"
    )

    response = client.messages.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": "Say hello in Spanish",
        }],
    )

    print(response.content[0].text)
    ```
  </Tab>

  <Tab title="JavaScript (OpenAI SDK)">
    Fireworks provides an OpenAI compatible endpoint. Install the [OpenAI JavaScript / TypeScript SDK](https://github.com/openai/openai-node):

    ```bash theme={null}
    npm install openai
    ```

    Then make your first Serverless API call:

    ```javascript theme={null}
    import OpenAI from "openai";

    const client = new OpenAI({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference/v1",
    });

    const response = await client.chat.completions.create({
      model: "accounts/fireworks/models/deepseek-v3p1",
      messages: [
        {
          role: "user",
          content: "Say hello in Spanish",
        },
      ],
    });

    console.log(response.choices[0].message.content);
    ```
  </Tab>

  <Tab title="JavaScript (Anthropic SDK)">
    Fireworks provides an Anthropic compatible endpoint. Install the [Anthropic JavaScript / TypeScript SDK](https://github.com/anthropics/anthropic-sdk-typescript):

    ```bash theme={null}
    npm install @anthropic-ai/sdk
    ```

    Then make your first Serverless API call:

    ```javascript theme={null}
    import Anthropic from "@anthropic-ai/sdk";

    const client = new Anthropic({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference",
    });

    const response = await client.messages.create({
      model: "accounts/fireworks/models/deepseek-v3p1",
      max_tokens: 1024,
      messages: [
        {
          role: "user",
          content: "Say hello in Spanish",
        },
      ],
    });

    console.log(response.content[0].text);
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    curl https://api.fireworks.ai/inference/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $FIREWORKS_API_KEY" \
      -d '{
        "model": "accounts/fireworks/models/deepseek-v3p1",
        "messages": [
          {
            "role": "user",
            "content": "Say hello in Spanish"
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

You should see a response like: `"¡Hola!"`

<Tip>
  For **Priority tier** (`service_tier: "priority"`) and **Fast**, see [Serverless Serving Paths](/serverless/serving-paths).
</Tip>

## Common use cases

### Streaming responses

Stream responses token-by-token for a better user experience:

<Tabs>
  <Tab title="Python (Fireworks SDK)">
    ```python theme={null}
    from fireworks import Fireworks

    client = Fireworks()

    stream = client.chat.completions.create(
      model="accounts/fireworks/models/deepseek-v3p1",
      messages=[{"role": "user", "content": "Tell me a short story"}],
      stream=True
    )

    for chunk in stream:
      if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
    ```
  </Tab>

  <Tab title="Python (OpenAI SDK)">
    ```python theme={null}
    import os
    from openai import OpenAI

    client = OpenAI(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference/v1"
    )

    stream = client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[{"role": "user", "content": "Tell me a short story"}],
        stream=True
    )

    for chunk in stream:
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="")
    ```
  </Tab>

  <Tab title="Python (Anthropic SDK)">
    ```python theme={null}
    import os
    import anthropic

    client = anthropic.Anthropic(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference"
    )

    with client.messages.stream(
        model="accounts/fireworks/models/deepseek-v3p1",
        max_tokens=1024,
        messages=[{"role": "user", "content": "Tell me a short story"}],
    ) as stream:
        for text in stream.text_stream:
            print(text, end="", flush=True)
    ```
  </Tab>

  <Tab title="JavaScript (OpenAI SDK)">
    ```javascript theme={null}
    import OpenAI from "openai";

    const client = new OpenAI({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference/v1",
    });

    const stream = await client.chat.completions.create({
      model: "accounts/fireworks/models/deepseek-v3p1",
      messages: [{ role: "user", content: "Tell me a short story" }],
      stream: true,
    });

    for await (const chunk of stream) {
      process.stdout.write(chunk.choices[0]?.delta?.content || "");
    }
    ```
  </Tab>

  <Tab title="JavaScript (Anthropic SDK)">
    ```javascript theme={null}
    import Anthropic from "@anthropic-ai/sdk";

    const client = new Anthropic({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference",
    });

    const stream = client.messages.stream({
      model: "accounts/fireworks/models/deepseek-v3p1",
      max_tokens: 1024,
      messages: [{ role: "user", content: "Tell me a short story" }],
    });

    stream.on("text", (text) => {
      process.stdout.write(text);
    });

    await stream.finalMessage();
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    curl https://api.fireworks.ai/inference/v1/chat/completions \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $FIREWORKS_API_KEY" \
        -d '{
        "model": "accounts/fireworks/models/deepseek-v3p1",
        "messages": [
            {
            "role": "user",
            "content": "Tell me a short story"
            }
        ],
        "stream": true
        }'
    ```
  </Tab>
</Tabs>

### Function calling

Connect your models to external tools and APIs:

<Tabs>
  <Tab title="Python (Fireworks SDK)">
    ```python theme={null}
    from fireworks import Fireworks

    client = Fireworks()

    response = client.chat.completions.create(
        model="accounts/fireworks/models/kimi-k2-instruct-0905",
        messages=[
            {"role": "user", "content": "What's the weather in Paris?"}
        ],
        tools=[
            {
                "type": "function",
                "function": {
                    "name": "get_weather",
                    "description": "Get the current weather for a location",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "location": {
                                "type": "string",
                                "description": "City name, e.g. San Francisco",
                            }
                        },
                        "required": ["location"],
                    },
                },
            },
        ],
    )

    print(response.choices[0].message.tool_calls)
    ```
  </Tab>

  <Tab title="Python (OpenAI SDK)">
    ```python theme={null}
    import os
    from openai import OpenAI

    client = OpenAI(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference/v1",
    )

    response = client.chat.completions.create(
        model="accounts/fireworks/models/kimi-k2-instruct-0905",
        messages=[
            {"role": "user", "content": "What's the weather in Paris?"}
        ],
        tools=[
            {
                "type": "function",
                "function": {
                    "name": "get_weather",
                    "description": "Get the current weather for a location",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "location": {
                                "type": "string",
                                "description": "City name, e.g. San Francisco",
                            }
                        },
                        "required": ["location"],
                    },
                },
            },
        ],
    )

    print(response.choices[0].message.tool_calls)
    ```
  </Tab>

  <Tab title="Python (Anthropic SDK)">
    ```python theme={null}
    import os
    import anthropic

    client = anthropic.Anthropic(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference"
    )

    response = client.messages.create(
        model="accounts/fireworks/models/kimi-k2-instruct-0905",
        max_tokens=1024,
        messages=[
            {"role": "user", "content": "What's the weather in Paris?"}
        ],
        tools=[
            {
                "name": "get_weather",
                "description": "Get the current weather for a location",
                "input_schema": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "City name, e.g. San Francisco",
                        }
                    },
                    "required": ["location"],
                },
            },
        ],
    )

    for block in response.content:
        if block.type == "tool_use":
            print(f"Tool: {block.name}, Input: {block.input}")
    ```
  </Tab>

  <Tab title="JavaScript (OpenAI SDK)">
    ```javascript theme={null}
    import OpenAI from "openai";

    const client = new OpenAI({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference/v1",
    });

    const tools = [
      {
        type: "function",
        function: {
          name: "get_weather",
          description: "Get the current weather for a location",
          parameters: {
            type: "object",
            properties: {
              location: {
                type: "string",
                description: "City name, e.g. San Francisco",
              },
            },
            required: ["location"],
          },
        },
      },
    ];

    const response = await client.chat.completions.create({
      model: "accounts/fireworks/models/kimi-k2-instruct-0905",
      messages: [{ role: "user", content: "What's the weather in Paris?" }],
      tools: tools,
    });

    console.log(response.choices[0].message.tool_calls);
    ```
  </Tab>

  <Tab title="JavaScript (Anthropic SDK)">
    ```javascript theme={null}
    import Anthropic from "@anthropic-ai/sdk";

    const client = new Anthropic({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference",
    });

    const response = await client.messages.create({
      model: "accounts/fireworks/models/kimi-k2-instruct-0905",
      max_tokens: 1024,
      messages: [{ role: "user", content: "What's the weather in Paris?" }],
      tools: [
        {
          name: "get_weather",
          description: "Get the current weather for a location",
          input_schema: {
            type: "object",
            properties: {
              location: {
                type: "string",
                description: "City name, e.g. San Francisco",
              },
            },
            required: ["location"],
          },
        },
      ],
    });

    for (const block of response.content) {
      if (block.type === "tool_use") {
        console.log(`Tool: ${block.name}, Input:`, block.input);
      }
    }
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    curl https://api.fireworks.ai/inference/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $FIREWORKS_API_KEY" \
      -d '{
        "model": "accounts/fireworks/models/kimi-k2-instruct-0905",
        "messages": [
          {
            "role": "user",
            "content": "What'\''s the weather in Paris?"
          }
        ],
        "tools": [
          {
            "type": "function",
            "function": {
              "name": "get_weather",
              "description": "Get the current weather for a location",
              "parameters": {
                "type": "object",
                "properties": {
                  "location": {
                    "type": "string",
                    "description": "City name, e.g. San Francisco"
                  }
                },
                "required": ["location"]
              }
            }
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

[Learn more about function calling →](/guides/function-calling)

### Structured outputs (JSON mode)

Get reliable JSON responses that match your schema:

<Tabs>
  <Tab title="Python (Fireworks SDK)">
    ```python theme={null}
    from fireworks import Fireworks

    client = Fireworks()

    response = client.chat.completions.create(
      model="accounts/fireworks/models/deepseek-v3p1",
      messages=[
        {
          "role": "user",
          "content": "Extract the name and age from: John is 30 years old",
        }
      ],
      response_format={
        "type": "json_schema",
        "json_schema": {
          "name": "person",
          "schema": {
            "type": "object",
            "properties": {
              "name": { "type": "string" },
              "age": { "type": "number" }
            },
            "required": ["name", "age"],
          },
        },
      },
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="Python (OpenAI SDK)">
    ```python theme={null}
    import os
    from openai import OpenAI

    client = OpenAI(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference/v1",
    )

    response = client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[
            {
                "role": "user",
                "content": "Extract the name and age from: John is 30 years old",
            }
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "person",
                "schema": {
                    "type": "object",
                    "properties": {"name": {"type": "string"}, "age": {"type": "number"}},
                    "required": ["name", "age"],
                },
            },
        },
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="Python (Anthropic SDK)">
    ```python theme={null}
    import os
    import anthropic

    client = anthropic.Anthropic(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference"
    )

    response = client.messages.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        max_tokens=1024,
        output_config={
            "format": {
                "type": "json_schema",
                "schema": {
                    "type": "object",
                    "properties": {
                        "name": { "type": "string" },
                        "age": { "type": "number" }
                    },
                    "required": ["name", "age"],
                },
            }
        },
        messages=[
            {
                "role": "user",
                "content": "Extract the name and age from: John is 30 years old",
            }
        ],
    )

    print(response.content[0].text)
    ```
  </Tab>

  <Tab title="JavaScript (OpenAI SDK)">
    ```javascript theme={null}
    import OpenAI from "openai";

    const client = new OpenAI({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference/v1",
    });

    const response = await client.chat.completions.create({
      model: "accounts/fireworks/models/deepseek-v3p1",
      messages: [
        {
          role: "user",
          content: "Extract the name and age from: John is 30 years old",
        },
      ],
      response_format: {
        type: "json_schema",
        json_schema: {
          name: "person",
          schema: {
            type: "object",
            properties: {
              name: { type: "string" },
              age: { type: "number" },
            },
            required: ["name", "age"],
          },
        },
      },
    });

    console.log(response.choices[0].message.content);
    ```
  </Tab>

  <Tab title="JavaScript (Anthropic SDK)">
    ```javascript theme={null}
    import Anthropic from "@anthropic-ai/sdk";

    const client = new Anthropic({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference",
    });

    const response = await client.messages.create({
      model: "accounts/fireworks/models/deepseek-v3p1",
      max_tokens: 1024,
      output_config: {
        format: {
          type: "json_schema",
          schema: {
            type: "object",
            properties: {
              name: { type: "string" },
              age: { type: "number" },
            },
            required: ["name", "age"],
          },
        },
      },
      messages: [
        {
          role: "user",
          content: "Extract the name and age from: John is 30 years old",
        },
      ],
    });

    console.log(response.content[0].text);
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    curl https://api.fireworks.ai/inference/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $FIREWORKS_API_KEY" \
      -d '{
        "model": "accounts/fireworks/models/deepseek-v3p1",
        "messages": [
          {
            "role": "user",
            "content": "Extract the name and age from: John is 30 years old"
          }
        ],
        "response_format": {
          "type": "json_schema",
          "json_schema": {
            "name": "person",
            "schema": {
              "type": "object",
              "properties": {
                "name": {
                  "type": "string"
                },
                "age": {
                  "type": "number"
                }
              },
              "required": ["name", "age"]
            }
          }
        }
      }'
    ```
  </Tab>
</Tabs>

[Learn more about structured outputs →](/structured-responses/structured-response-formatting)

### Reasoning

Some models support reasoning, where the model shows its thought process before giving the final answer:

<Tabs>
  <Tab title="Python (Fireworks SDK)">
    ```python theme={null}
    from fireworks import Fireworks

    client = Fireworks()

    response = client.chat.completions.create(
        model="accounts/fireworks/models/glm-5p2",
        messages=[
            {"role": "user", "content": "What is 25 * 37? Show your work."}
        ],
        reasoning_effort="medium",
    )

    msg = response.choices[0].message
    if msg.reasoning_content:
        print("Reasoning:", msg.reasoning_content)
    print("Answer:", msg.content)
    ```
  </Tab>

  <Tab title="Python (OpenAI SDK)">
    ```python theme={null}
    import os
    from openai import OpenAI

    client = OpenAI(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference/v1",
    )

    response = client.chat.completions.create(
        model="accounts/fireworks/models/glm-5p2",
        messages=[
            {"role": "user", "content": "What is 25 * 37? Show your work."}
        ],
        extra_body={"reasoning_effort": "medium"},
    )

    msg = response.choices[0].message
    # Reasoning content is returned in a separate field
    reasoning = getattr(msg, "reasoning_content", None)
    if reasoning is None and hasattr(msg, "model_extra"):
        reasoning = msg.model_extra.get("reasoning_content")

    if reasoning:
        print("Reasoning:", reasoning)
    print("Answer:", msg.content)
    ```
  </Tab>

  <Tab title="Python (Anthropic SDK)">
    The Anthropic SDK uses the `thinking` parameter to enable reasoning:

    ```python theme={null}
    import os
    import anthropic

    client = anthropic.Anthropic(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference"
    )

    response = client.messages.create(
        model="accounts/fireworks/models/glm-5p2",
        max_tokens=16000,
        thinking={"type": "enabled", "budget_tokens": 4096},
        messages=[
            {"role": "user", "content": "What is 25 * 37? Show your work."}
        ],
    )

    for block in response.content:
        if block.type == "thinking":
            print("Thinking:", block.thinking)
        elif block.type == "text":
            print("Answer:", block.text)
    ```
  </Tab>

  <Tab title="JavaScript (OpenAI SDK)">
    ```javascript theme={null}
    import OpenAI from "openai";

    const client = new OpenAI({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference/v1",
    });

    const response = await client.chat.completions.create({
      model: "accounts/fireworks/models/glm-5p2",
      messages: [
        { role: "user", content: "What is 25 * 37? Show your work." },
      ],
      reasoning_effort: "medium",
    });

    const msg = response.choices[0].message;
    if (msg.reasoning_content) {
      console.log("Reasoning:", msg.reasoning_content);
    }
    console.log("Answer:", msg.content);
    ```
  </Tab>

  <Tab title="JavaScript (Anthropic SDK)">
    The Anthropic SDK uses the `thinking` parameter to enable reasoning:

    ```javascript theme={null}
    import Anthropic from "@anthropic-ai/sdk";

    const client = new Anthropic({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference",
    });

    const response = await client.messages.create({
      model: "accounts/fireworks/models/glm-5p2",
      max_tokens: 16000,
      thinking: { type: "enabled", budget_tokens: 4096 },
      messages: [
        { role: "user", content: "What is 25 * 37? Show your work." },
      ],
    });

    for (const block of response.content) {
      if (block.type === "thinking") {
        console.log("Thinking:", block.thinking);
      } else if (block.type === "text") {
        console.log("Answer:", block.text);
      }
    }
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    curl https://api.fireworks.ai/inference/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $FIREWORKS_API_KEY" \
      -d '{
        "model": "accounts/fireworks/models/glm-5p2",
        "messages": [
          {
            "role": "user",
            "content": "What is 25 * 37? Show your work."
          }
        ],
        "reasoning_effort": "medium"
      }'
    ```
  </Tab>
</Tabs>

[Learn more about reasoning →](/guides/reasoning)

### Vision models

Analyze images with vision-language models:

<Tabs>
  <Tab title="Python (Fireworks SDK)">
    ```python theme={null}
    from fireworks import Fireworks

    client = Fireworks()

    response = client.chat.completions.create(
      model="accounts/fireworks/models/kimi-k2p6",
      messages=[
        {
          "role": "user",
          "content": [
            {"type": "text", "text": "What's in this image?"},
            {
              "type": "image_url",
              "image_url": {
                "url": "https://storage.googleapis.com/fireworks-public/image_assets/fireworks-ai-wordmark-color-dark.png"
              },
            },
          ],
        }
      ],
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="Python (OpenAI SDK)">
    ```python theme={null}
    import os
    from openai import OpenAI

    client = OpenAI(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference/v1"
    )

    response = client.chat.completions.create(
        model="accounts/fireworks/models/kimi-k2p6",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "What's in this image?"},
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": "https://storage.googleapis.com/fireworks-public/image_assets/fireworks-ai-wordmark-color-dark.png"
                        },
                    },
                ],
            }
        ],
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="Python (Anthropic SDK)">
    The Anthropic SDK uses its native image format with `type: "image"` and a `source` object:

    ```python theme={null}
    import os
    import anthropic

    client = anthropic.Anthropic(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference"
    )

    response = client.messages.create(
        model="accounts/fireworks/models/kimi-k2p6",
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "What's in this image?"},
                    {
                        "type": "image",
                        "source": {
                            "type": "url",
                            "url": "https://storage.googleapis.com/fireworks-public/image_assets/fireworks-ai-wordmark-color-dark.png",
                        },
                    },
                ],
            }
        ],
    )

    for block in response.content:
        if block.type == "text":
            print(block.text)
    ```
  </Tab>

  <Tab title="JavaScript (OpenAI SDK)">
    ```javascript theme={null}
    import OpenAI from "openai";

    const client = new OpenAI({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference/v1",
    });

    const response = await client.chat.completions.create({
      model: "accounts/fireworks/models/kimi-k2p6",
      messages: [
        {
          role: "user",
          content: [
            { type: "text", text: "What's in this image?" },
            {
              type: "image_url",
              image_url: {
                url: "https://storage.googleapis.com/fireworks-public/image_assets/fireworks-ai-wordmark-color-dark.png",
              },
            },
          ],
        },
      ],
    });

    console.log(response.choices[0].message.content);
    ```
  </Tab>

  <Tab title="JavaScript (Anthropic SDK)">
    ```javascript theme={null}
    import Anthropic from "@anthropic-ai/sdk";

    const client = new Anthropic({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference",
    });

    const response = await client.messages.create({
      model: "accounts/fireworks/models/kimi-k2p6",
      max_tokens: 1024,
      messages: [
        {
          role: "user",
          content: [
            { type: "text", text: "What's in this image?" },
            {
              type: "image",
              source: {
                type: "url",
                url: "https://storage.googleapis.com/fireworks-public/image_assets/fireworks-ai-wordmark-color-dark.png",
              },
            },
          ],
        },
      ],
    });

    for (const block of response.content) {
      if (block.type === "text") {
        console.log(block.text);
      }
    }
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    curl https://api.fireworks.ai/inference/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $FIREWORKS_API_KEY" \
      -d '{
        "model": "accounts/fireworks/models/kimi-k2p6",
        "messages": [
          {
            "role": "user",
            "content": [
              {
                "type": "text",
                "text": "What'\''s in this image?"
              },
              {
                "type": "image_url",
                "image_url": {
                  "url": "https://storage.googleapis.com/fireworks-public/image_assets/fireworks-ai-wordmark-color-dark.png"
                }
              }
            ]
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

[Learn more about vision models →](/guides/querying-vision-language-models)

## Learn more about Serverless

For the model lifecycle policy, billing details, and serverless-specific request/response behavior, see the [Serverless overview](/serverless/overview).

## Next steps

Ready to scale to production, explore other modalities, or customize your models?

<CardGroup>
  <Card title="Deploy and autoscale on Dedicated GPUs" href="/guides/ondemand-deployments" icon="server">
    Deploy with high performance on dedicated GPUs with fast autoscaling and minimal cold starts
  </Card>

  <Card title="Fine-tune Models" href="/fine-tuning/finetuning-intro" icon="sliders">
    Improve model quality with supervised and reinforcement learning
  </Card>

  <Card title="Embeddings & Reranking" href="/guides/querying-embeddings-models" icon="brackets-curly">
    Use embeddings & reranking in search & context retrieval
  </Card>

  <Card title="Batch Inference" href="/guides/batch-inference" icon="list-check">
    Run async inference jobs at scale, faster and cheaper
  </Card>

  <Card title="Browse 100+ Models" href="https://fireworks.ai/models" icon="books">
    Explore all available models across modalities
  </Card>

  <Card title="API Reference" href="/api-reference/introduction" icon="code">
    Complete API documentation
  </Card>
</CardGroup>


# Batch API
Source: https://docs.fireworks.ai/guides/batch-inference

Process large-scale async workloads at a discount

Process large volumes of requests asynchronously at **50% off** [Serverless per-token prices](https://docs.fireworks.ai/serverless/pricing). Batch API is ideal for:

* Data labeling and synthetic data generation
* Training smaller models with larger ones ([distillation guide](https://fireworks.ai/blog/deepseek-r1-distillation-reasoning))
* Large-scale evaluations and benchmarking
* Document processing and more

<Tip>
  Batch jobs automatically use [prompt caching](/guides/prompt-caching) for additional 50% cost savings on cached tokens. Maximize cache hits by placing static content first in your prompts.
</Tip>

## Model compatibility

Not all models support the Batch API. Before submitting a batch job, verify your target model is batch-compatible.

* **Base Models** – Any model that supports [On-Demand Deployments](https://docs.fireworks.ai/guides/ondemand-deployments) in the [Model Library](https://fireworks.ai/models)
* **Custom Models** – Your uploaded or fine-tuned models built on a batch-compatible base model

*Note: Newly added models may have a delay before being supported. See [Quantization](/models/quantization) for precision info.*

<Note>
  If a model does not support batch inference, submitting a job may not produce an immediate error — the job can remain in a pending state and never schedule. Always verify compatibility before submitting.
</Note>

**If your batch job is not running:**

1. If validation failed, check your JSONL input — each line must be a complete, valid JSON object matching the request schema.
2. Batch jobs wait to be scheduled in a "pending" state during the selected time window, so it may not run immediately.
3. If the job has been "creating" a deployment for more than 30 minutes, contact support with your job ID.
   1. Confirm the model supports batch inference (see above).
   2. Check that your account has sufficient quota for batch jobs.
4. Progress may pause while waiting on capacity. The job will resume automatically.

## Getting Started

<AccordionGroup>
  <Accordion title="1. Prepare Your Dataset">
    Datasets must be in JSONL format (one JSON object per line):

    **Requirements:**

    * **File format:** JSONL (each line is a valid JSON object)
    * **Size limit:** Up to 80 GiB
    * **Required fields:** `custom_id` (unique) and `body` (request parameters)
    * **Input mode:** `body.messages` (standard) or `body.prompt_token_ids` (pre-tokenized) — mutually exclusive

    **Example dataset (messages):**

    ```json theme={null}
    {"custom_id": "request-1", "body": {"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"}], "max_tokens": 100}}
    {"custom_id": "request-2", "body": {"messages": [{"role": "user", "content": "Explain quantum computing"}], "temperature": 0.7}}
    {"custom_id": "request-3", "body": {"messages": [{"role": "user", "content": "Tell me a joke"}]}}
    ```

    <Tip>
      If every row shares the same `system` message, you can omit it from the dataset and set it once at the job level with `--system-prompt` instead — see [Job-level system prompt](#job-level-system-prompt). This shrinks your upload and keeps prompt-caching intact.
    </Tip>

    **Example dataset (pre-tokenized):**

    When you need byte-identical inputs (e.g. a custom tokenizer or renderer), use `prompt_token_ids` instead of `messages`. The server skips chat-template rendering and feeds your token IDs directly to generation. Mutually exclusive with `messages` — do not include both.

    ```json theme={null}
    {"custom_id": "request-1", "body": {"prompt_token_ids": [163587, 2482, 163601, 71079, 17522, 13], "max_tokens": 100}}
    ```

    <Warning>
      Token IDs must be valid for the vocabulary of the **specific model** the job targets. An out-of-vocabulary ID fails that row with a terminal 400 (no retry), so re-tokenize when you change models.
    </Warning>

    <Warning>
      A job-level `system_prompt` cannot be combined with `prompt_token_ids` — the prompt is already tokenized, so there is nowhere to prepend a system message. Set `system_prompt` only for `messages`-based jobs.
    </Warning>

    <Warning>
      Set `user` or `prompt_cache_key` in the body. Without message text, the gateway has nothing to hash for content-based cache affinity, so each row routes independently and prompt-cache hit rates drop.
    </Warning>

    Save as `batch_input_data.jsonl` locally.
  </Accordion>

  <Accordion title="2. Upload Your Dataset">
    <Tabs>
      <Tab title="UI">
        You can simply navigate to the dataset tab, click `Create Dataset` and follow the wizard.

        <img alt="Dataset Upload" />
      </Tab>

      <Tab title="firectl">
        ```bash theme={null}
        firectl dataset create batch-input-dataset ./batch_input_data.jsonl
        ```
      </Tab>

      <Tab title="HTTP API">
        You need to make two separate HTTP requests. One for creating the dataset entry and one for uploading the dataset. Full reference here: [Create dataset](/api-reference/create-dataset).

        ```bash theme={null}
        # Create Dataset Entry
        curl -X POST "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/datasets" \
          -H "Authorization: Bearer ${API_KEY}" \
          -H "Content-Type: application/json" \
          -d '{
            "datasetId": "batch-input-dataset",
            "dataset": { "userUploaded": {} }
          }'

        # Upload JSONL file
        curl -X POST "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/datasets/batch-input-dataset:upload" \
          -H "Authorization: Bearer ${API_KEY}" \
          -F "file=@./batch_input_data.jsonl"
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="3. Create a Batch Job">
    <Tabs>
      <Tab title="UI">
        Navigate to the Batch Inference tab and click "Create Batch Inference Job". Choose your batch-eligible model from the dropdown selector:

        <img alt="BIJ Model Selector" title="BIJ Model Selector" />

        Select your dataset:

        <img alt="BIJ Dataset Selector" title="BIJ Dataset Selector" />

        Configure optional settings:

        <img alt="BIJ Optional Settings Selector" title="BIJ Optional Settings Selector" />
      </Tab>

      <Tab title="firectl">
        ```bash theme={null}
        firectl batch-inference-job create \
          --model accounts/fireworks/models/llama-v3p1-8b-instruct \
          --input-dataset-id batch-input-dataset
        ```

        With additional parameters:

        ```bash theme={null}
        firectl batch-inference-job create \
          --job-id my-batch-job \
          --model accounts/fireworks/models/llama-v3p1-8b-instruct \
          --input-dataset-id batch-input-dataset \
          --output-dataset-id batch-output-dataset \
          --system-prompt "You are a helpful assistant." \
          --max-tokens 1024 \
          --temperature 0.7 \
          --top-p 0.9
        ```
      </Tab>

      <Tab title="HTTP API">
        ```bash theme={null}
        curl -X POST "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/batchInferenceJobs?batchInferenceJobId=my-batch-job" \
          -H "Authorization: Bearer ${API_KEY}" \
          -H "Content-Type: application/json" \
          -d '{
            "model": "accounts/fireworks/models/llama-v3p1-8b-instruct",
            "inputDatasetId": "accounts/'${ACCOUNT_ID}'/datasets/batch-input-dataset",
            "outputDatasetId": "accounts/'${ACCOUNT_ID}'/datasets/batch-output-dataset",
            "systemPrompt": "You are a helpful assistant.",
            "inferenceParameters": {
              "maxTokens": 1024,
              "temperature": 0.7,
              "topP": 0.9
            }
          }'
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="4. Monitor Your Job">
    <Tabs>
      <Tab title="UI">
        View all your batch inference jobs in the dashboard:

        <img alt="BIJ List" />
      </Tab>

      <Tab title="firectl">
        ```bash theme={null}
        # Get job status
        firectl batch-inference-job get my-batch-job

        # List all batch jobs
        firectl batch-inference-job list
        ```
      </Tab>

      <Tab title="HTTP API">
        ```bash theme={null}
        # Get specific job
        curl -X GET "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/batchInferenceJobs/my-batch-job" \
          -H "Authorization: Bearer ${API_KEY}"

        # List all jobs
        curl -X GET "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/batchInferenceJobs" \
          -H "Authorization: Bearer ${API_KEY}"
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="5. Download Results">
    <Tabs>
      <Tab title="UI">
        Navigate to the output dataset and download the results:

        <img alt="BIJ Download Output" title="BIJ Download Output" />
      </Tab>

      <Tab title="firectl">
        ```bash theme={null}
        firectl dataset download batch-output-dataset
        ```
      </Tab>

      <Tab title="HTTP API">
        ```bash theme={null}
        # Get download endpoint and save response
        curl -s -X GET "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/datasets/batch-output-dataset:getDownloadEndpoint" \
          -H "Authorization: Bearer ${API_KEY}" \
          -d '{}' > download.json

        # Extract and download all files
        jq -r '.filenameToSignedUrls | to_entries[] | "\(.key) \(.value)"' download.json | \
        while read -r object_path signed_url; do
            fname=$(basename "$object_path")
            echo "Downloading → $fname"
            curl -L -o "$fname" "$signed_url"
        done
        ```
      </Tab>
    </Tabs>

    <Tip>
      The output dataset contains two files: a **results file** (successful responses in JSONL format) and an **error file** (failed requests with debugging info).
    </Tip>
  </Accordion>
</AccordionGroup>

## Job-level system prompt

Set one static `system` message for the entire job instead of repeating it on every dataset row. When provided, the batch runner injects it as a leading `system` message into **every input row that does not already begin with a `system` message** — a row's own leading `system` message always takes precedence, so you can override the job-level prompt per-row when needed.

This is useful when all rows share a large, fixed system prompt (e.g. a rubric or output-schema spec). Removing it from every row shrinks the input dataset (less upload, stays under the 80 GiB limit), and because the injected prefix is byte-identical across rows, [prompt caching](/guides/prompt-caching) still applies.

```bash theme={null}
firectl batch-inference-job create \
  --model accounts/fireworks/models/llama-v3p1-8b-instruct \
  --input-dataset-id batch-input-dataset \
  --system-prompt "You are a helpful assistant that responds in JSON."
```

For a long or multi-line prompt, pass it from a file to preserve newlines and avoid shell-escaping:

```bash theme={null}
firectl batch-inference-job create \
  --model accounts/fireworks/models/llama-v3p1-8b-instruct \
  --input-dataset-id batch-input-dataset \
  --system-prompt "$(cat ./system_prompt.txt)"
```

The equivalent HTTP field is `systemPrompt`:

```json theme={null}
{
  "model": "accounts/fireworks/models/llama-v3p1-8b-instruct",
  "inputDatasetId": "accounts/${ACCOUNT_ID}/datasets/batch-input-dataset",
  "systemPrompt": "You are a helpful assistant that responds in JSON."
}
```

<Note>
  The injected system message counts toward the model's context window. If your system prompt is very large and your rows also include long content (e.g. media), the combined prompt may exceed the model's context length — the job will run but produce no completions. Keep the system prompt + per-row content within the model's context limit, or trim the prompt.
</Note>

## Reference

<AccordionGroup>
  <Accordion title="Job states">
    Batch jobs progress through several states:

    | State          | Description                                                                    |
    | -------------- | ------------------------------------------------------------------------------ |
    | **VALIDATING** | Dataset is being validated for format requirements                             |
    | **PENDING**    | Job is queued and waiting for resources                                        |
    | **RUNNING**    | Actively processing requests                                                   |
    | **COMPLETED**  | All requests successfully processed                                            |
    | **FAILED**     | Unrecoverable error occurred (check status message)                            |
    | **EXPIRED**    | Exceeded chosen time limit (12, 24, 48, 72 hrs). Completed requests are saved. |
  </Accordion>

  <Accordion title="Supported models">
    * **Base Models** – Any model that supports [On-Demand Deployments](https://docs.fireworks.ai/guides/ondemand-deployments) in the [Model Library](https://fireworks.ai/models)
    * **Custom Models** – Your uploaded or fine-tuned models built on a batch-compatible base model

    *Note: Newly added models may have a delay before being supported. See [Quantization](/models/quantization) for precision info.*
  </Accordion>

  <Accordion title="Limits and constraints">
    * **Per-request limits:** Same as [Chat Completion API limits](/api-reference/post-chatcompletions)
    * **Input dataset:** Up to 80 GiB
    * **Output dataset:** Max 8GB (job may expire early if limit is reached)
    * **Job expiration:** Select from 12, 24, 48, 72 hours maximum in Optional Settings
  </Accordion>

  <Accordion title="Handling expired jobs">
    Jobs expire after 24 hours. Completed rows are billed and saved to the output dataset.

    **Resume processing:**

    ```bash theme={null}
    firectl batch-inference-job create \
      --continue-from original-job-id \
      --model accounts/fireworks/models/llama-v3p1-8b-instruct \
      --output-dataset-id new-output-dataset
    ```

    This processes only unfinished/failed requests from the original job.

    **Download complete lineage:**

    ```bash theme={null}
    firectl dataset download output-dataset-id --download-lineage
    ```

    Downloads all datasets in the continuation chain.
  </Accordion>

  <Accordion title="Best practices">
    * **Validate thoroughly:** Check dataset format before uploading
    * **Descriptive IDs:** Use meaningful `custom_id` values for tracking
    * **Optimize tokens:** Set reasonable `max_tokens` limits
    * **Monitor progress:** Track long-running jobs regularly
    * **Cache optimization:** Place static content first in prompts
    * **Shared system prompt:** If every row uses the same system message, set it once with `--system-prompt` instead of repeating it per row — smaller upload and cache-friendly (see [Job-level system prompt](#job-level-system-prompt))
  </Accordion>
</AccordionGroup>

## Next Steps

<CardGroup>
  <Card title="Prompt Caching" icon="bolt" href="/guides/prompt-caching">
    Maximize cost savings with automatic prompt caching
  </Card>

  <Card title="Fine-Tuning" icon="sparkles" href="/fine-tuning/finetuning-intro">
    Create custom models for your batch workloads
  </Card>

  <Card title="API Reference" icon="code" href="/api-reference/create-batch-inference-job">
    Full API documentation for Batch API
  </Card>
</CardGroup>


# Completions API
Source: https://docs.fireworks.ai/guides/completions-api

Use the completions API for raw text generation with custom prompt templates

The completions API provides raw text generation without automatic message formatting. Use this when you need full control over prompt formatting or when working with base models.

## When to use completions

**Use the completions API for:**

* Custom prompt templates with specific formatting requirements
* Base models (non-instruct/non-chat variants)
* Fine-grained control over token-level formatting
* Legacy applications that depend on raw completion format

**For most use cases, use [chat completions](/guides/querying-text-models) instead.** Chat completions handles message formatting automatically and works better with instruct-tuned models.

## Basic usage

<Tabs>
  <Tab title="Python (Fireworks SDK)">
    ```python theme={null}
    from fireworks import Fireworks

    client = Fireworks()

    response = client.completions.create(
      model="accounts/fireworks/models/deepseek-v3p1",
      prompt="Once upon a time"
    )

    print(response.choices[0].text)
    ```
  </Tab>

  <Tab title="Python (OpenAI SDK)">
    ```python theme={null}
    import os
    from openai import OpenAI

    client = OpenAI(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference/v1"
    )

    response = client.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        prompt="Once upon a time"
    )

    print(response.choices[0].text)
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    import OpenAI from "openai";

    const client = new OpenAI({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference/v1",
    });

    const response = await client.completions.create({
      model: "accounts/fireworks/models/deepseek-v3p1",
      prompt: "Once upon a time",
    });

    console.log(response.choices[0].text);
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    curl https://api.fireworks.ai/inference/v1/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $FIREWORKS_API_KEY" \
      -d '{
        "model": "accounts/fireworks/models/deepseek-v3p1",
        "prompt": "Once upon a time"
      }'
    ```
  </Tab>
</Tabs>

<Note>
  Most models automatically prepend the beginning-of-sequence (BOS) token (e.g., `
`) to your prompt. Verify this with the `raw_output` parameter if needed.
</Note>

## Custom prompt templates

The completions API is useful when you need to implement custom prompt formats:

```python theme={null}
# Custom few-shot prompt template
prompt = """Task: Classify the sentiment of the following text.

Text: I love this product!
Sentiment: Positive

Text: This is terrible.
Sentiment: Negative

Text: The weather is nice today.
Sentiment:"""

response = client.completions.create(
    model="accounts/fireworks/models/deepseek-v3p1",
    prompt=prompt,
    max_tokens=10,
    temperature=0
)

print(response.choices[0].text)  # Output: " Positive"
```

## Common parameters

All [chat completions parameters](/guides/querying-text-models) work with completions:

* `temperature` - Control randomness (0-2)
* `max_tokens` - Limit output length
* `top_p`, `top_k`, `min_p` - Sampling parameters
* `stream` - Stream responses token-by-token
* `frequency_penalty`, `presence_penalty` - Reduce repetition

See the [API reference](/api-reference/post-completions) for complete parameter documentation.

## Querying deployments

Use completions with [on-demand deployments](/guides/ondemand-deployments) by specifying the deployment identifier:

```python theme={null}
response = client.completions.create(
    model="accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>",
    prompt="Your prompt here"
)
```

## Next steps

<CardGroup>
  <Card title="Chat Completions" href="/guides/querying-text-models" icon="messages">
    Use chat completions for most use cases
  </Card>

  <Card title="Streaming" href="/guides/querying-text-models#streaming-responses" icon="bolt">
    Stream responses for real-time UX
  </Card>

  <Card title="API Reference" href="/api-reference/post-completions" icon="code">
    Complete API documentation
  </Card>
</CardGroup>


# Tool Calling
Source: https://docs.fireworks.ai/guides/function-calling

Connect models to external tools and APIs

Tool calling (also known as function calling) enables models to intelligently select and use external tools based on user input. You can build agents that access APIs, retrieve real-time data, or perform actions—all through [OpenAI-compatible](https://platform.openai.com/docs/guides/function-calling) tool specifications.

**How it works:**

1. Define tools using [JSON Schema](https://json-schema.org/learn/getting-started-step-by-step) (name, description, parameters)
2. Model analyzes the query and decides whether to call a tool
3. If needed, model returns structured tool calls with parameters
4. You execute the tool and send results back for the final response

## Quick example

Define tools and send a request - the model will return structured tool calls when needed:

Initialize the client:

<CodeGroup>
  ```python Python (Fireworks SDK) theme={null}
  from fireworks import Fireworks

  client = Fireworks()
  ```

  ```python Python (OpenAI SDK) theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
      api_key=os.environ.get("FIREWORKS_API_KEY"),
      base_url="https://api.fireworks.ai/inference/v1"
  )
  ```
</CodeGroup>

Define the tools and make the request:

```python theme={null}
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string", "description": "City name"}
            },
            "required": ["location"]
        }
    }
}]

response = client.chat.completions.create(
    model="accounts/fireworks/models/kimi-k2-instruct-0905",
    messages=[{"role": "user", "content": "What's the weather in San Francisco?"}],
    tools=tools,
    temperature=0.1
)

print(response.choices[0].message.tool_calls)
# Output: [ChatCompletionMessageToolCall(id='call_abc123', function=Function(arguments='{"location":"San Francisco"}', name='get_weather'), type='function')]
```

<Tip>
  For best results with tool calling, use a low temperature (0.0-0.3) to reduce hallucinated parameter values and ensure more deterministic tool selection.
</Tip>

<AccordionGroup>
  <Accordion title="Complete workflow: Execute tools and get final response">
    ```python theme={null}
    import json

    # Step 1: Define your tools
    tools = [{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City name"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
                },
                "required": ["location"]
            }
        }
    }]

    # Step 2: Send initial request
    messages = [{"role": "user", "content": "What's the weather in San Francisco?"}]
    response = client.chat.completions.create(
        model="accounts/fireworks/models/kimi-k2-instruct-0905",
        messages=messages,
        tools=tools,
        temperature=0.1
    )

    # Step 3: Check if model wants to call a tool
    if response.choices[0].message.tool_calls:
        # Step 4: Execute the tool
        tool_call = response.choices[0].message.tool_calls[0]

        # Your actual tool implementation
        def get_weather(location, unit="celsius"):
            # In production, call your weather API here
            return {"temperature": 72, "condition": "sunny", "unit": unit}

        # Parse arguments and call your function
        function_args = json.loads(tool_call.function.arguments)
        function_response = get_weather(**function_args)

        # Step 5: Send tool response back to model
        messages.append(response.choices[0].message)  # Add assistant's tool call
        messages.append({
            "role": "tool",
            "tool_call_id": tool_call.id,
            "content": json.dumps(function_response)
        })

        # Step 6: Get final response
        final_response = client.chat.completions.create(
            model="accounts/fireworks/models/kimi-k2-instruct-0905",
            messages=messages,
            tools=tools,
            temperature=0.1
        )

        print(final_response.choices[0].message.content)
        # Output: "It's currently 72°F and sunny in San Francisco."
    ```
  </Accordion>
</AccordionGroup>

## Defining tools

Tools are defined using [JSON Schema](https://json-schema.org/understanding-json-schema/reference) format. Each tool requires:

* **name**: Function identifier (a-z, A-Z, 0-9, underscores, dashes; max 64 characters)
* **description**: Clear explanation of what the function does (used by the model to decide when to call it)
* **parameters**: JSON Schema object describing the function's parameters

<Tip>
  Write detailed descriptions and parameter definitions. The model relies on these to select the correct tool and provide appropriate arguments.
</Tip>

### Parameter types

JSON Schema supports: `string`, `number`, `integer`, `object`, `array`, `boolean`, and `null`. You can also:

* Use `enum` to restrict values to specific options
* Mark parameters as `required` or optional
* Provide descriptions for each parameter
* Reuse subschemas via `$defs` / `definitions` and `$ref`, including recursive references (e.g. linked lists, trees, mutually recursive types)
* Carry `$id`, `$schema`, and other annotation keywords (no external fetches are performed)

<Tip>
  The same JSON Schema feature set is supported in tool `parameters` and in `response_format={"type": "json_schema", ...}`. See [JSON Schema Support](/structured-responses/structured-response-formatting#json-schema-support) for the full list, including the rules for resolving `$ref` and how to reach definitions placed inside nested subschemas.
</Tip>

<Accordion title="Example: parameters that reuse a subschema via `$defs`">
  A common shape produced by pydantic v2, Instructor, and LangChain `TypeAdapter`. The `$defs` block holds the reusable subschemas and properties reference them by `$ref`:

  ```python theme={null}
  tools = [{
      "type": "function",
      "function": {
          "name": "submit_order",
          "description": "Submit an order with line items and a customer",
          "parameters": {
              "type": "object",
              "$defs": {
                  "Product":  {"type": "object", "properties": {"name": {"type": "string"}, "price": {"type": "number"}}, "required": ["name", "price"]},
                  "Customer": {"type": "object", "properties": {"name": {"type": "string"}, "email": {"type": "string"}}, "required": ["name", "email"]}
              },
              "properties": {
                  "items":    {"type": "array", "items": {"$ref": "#/$defs/Product"}},
                  "customer": {"$ref": "#/$defs/Customer"}
              },
              "required": ["items", "customer"]
          }
      }
  }]
  ```
</Accordion>

<Accordion title="Example: Defining multiple tools">
  ```python theme={null}
  tools = [
      {
          "type": "function",
          "function": {
              "name": "get_weather",
              "description": "Get current weather for a location",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "location": {
                          "type": "string",
                          "description": "City name, e.g. San Francisco"
                      },
                      "unit": {
                          "type": "string",
                          "enum": ["celsius", "fahrenheit"],
                          "description": "Temperature unit"
                      }
                  },
                  "required": ["location"]
              }
          }
      },
      {
          "type": "function",
          "function": {
              "name": "search_restaurants",
              "description": "Search for restaurants by cuisine type",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "cuisine": {
                          "type": "string",
                          "description": "Type of cuisine (e.g., Italian, Mexican)"
                      },
                      "location": {
                          "type": "string",
                          "description": "City or neighborhood"
                      },
                      "price_range": {
                          "type": "string",
                          "enum": ["$", "$$", "$$$", "$$$$"]
                      }
                  },
                  "required": ["cuisine", "location"]
              }
          }
      }
  ]
  ```
</Accordion>

## Additional configurations

### tool\_choice

The [`tool_choice`](/api-reference/post-chatcompletions) parameter controls how the model uses tools:

* **`auto`** (default): Model decides whether to call a tool or respond directly
* **`none`**: Model will not call any tools
* **`required`**: Model must call at least one tool
* **Specific function**: Force the model to call a particular function

```python theme={null}
# Force a specific tool
response = client.chat.completions.create(
    model="accounts/fireworks/models/kimi-k2-instruct-0905",
    messages=[{"role": "user", "content": "What's the weather?"}],
    tools=tools,
    tool_choice={"type": "function", "function": {"name": "get_weather"}},
    temperature=0.1
)
```

<Note>
  Some models support parallel tool calling, where multiple tools can be called in a single response. Check the model's capabilities before relying on this feature.
</Note>

## Streaming

<Accordion title="Using tool calls with streaming responses">
  Tool calls work with streaming responses. Arguments are sent incrementally as the model generates them:

  ```python theme={null}
  import json

  tools = [{
      "type": "function",
      "function": {
          "name": "get_weather",
          "description": "Get the current weather for a city",
          "parameters": {
              "type": "object",
              "properties": {
                  "city": {"type": "string", "description": "City name"},
                  "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
              },
              "required": ["city"]
          }
      }
  }]

  stream = client.chat.completions.create(
      model="accounts/fireworks/models/kimi-k2-instruct-0905",
      messages=[{"role": "user", "content": "What's the weather in San Francisco?"}],
      tools=tools,
      stream=True,
      temperature=0.1
  )

  # Accumulate tool call data
  tool_calls = {}

  for chunk in stream:
      if chunk.choices[0].delta.tool_calls:
          for tool_call in chunk.choices[0].delta.tool_calls:
              index = tool_call.index
              
              if index not in tool_calls:
                  tool_calls[index] = {"id": "", "name": "", "arguments": ""}
              
              if tool_call.id:
                  tool_calls[index]["id"] = tool_call.id
              if tool_call.function and tool_call.function.name:
                  tool_calls[index]["name"] = tool_call.function.name
              if tool_call.function and tool_call.function.arguments:
                  tool_calls[index]["arguments"] += tool_call.function.arguments
      
      if chunk.choices[0].finish_reason == "tool_calls":
          for tool_call in tool_calls.values():
              args = json.loads(tool_call["arguments"])
              print(f"Calling {tool_call['name']} with {args}")
          break
  ```
</Accordion>

## Troubleshooting

<AccordionGroup>
  <Accordion title="Model isn't calling tools when expected">
    * Check that your tool descriptions are clear and detailed
    * Ensure the user query clearly indicates a need for the tool
    * Try using `tool_choice="required"` to force tool usage
    * Verify your model supports tool calling (check `supportsTools` field)
  </Accordion>

  <Accordion title="Tool arguments are incorrect or malformed">
    * Add more detailed parameter descriptions
    * Use lower temperature (0.0-0.3) for more deterministic outputs
    * Provide examples in parameter descriptions
    * Use `enum` to constrain values to specific options
  </Accordion>

  <Accordion title="Getting JSON parsing errors">
    * Always validate tool call arguments before parsing
    * Handle partial or malformed JSON gracefully in production
    * Use try-catch blocks when parsing `tool_call.function.arguments`
  </Accordion>

  <Accordion title="400 &#x22;Error resolving schema reference&#x22; or &#x22;Error validating JSON Schema&#x22;">
    The schema in `parameters` (or `response_format`) is reaching a `$ref` Fireworks cannot resolve. Common causes:

    * **External `$ref` URI** (e.g. `https://example.com/schema.json`). Only in-document JSON Pointer fragments (`#/...`) are supported. Inline the referenced subschema or hoist it into `$defs`.
    * **Wrong fragment path**, e.g. `#/components/schemas/Foo` when the document has no `components` key. Match the pointer to the actual document layout.
    * **Older deployment image.** Recursive `$ref`, root `$id`, and nested `$defs` placed under a property all became supported in mid-2026 and earlier images returned 400 on those shapes. If you control a self-hosted/dedicated deployment, redeploy on a current image.

    For the full set of supported reference forms, see [JSON Schema Support](/structured-responses/structured-response-formatting#json-schema-support).
  </Accordion>
</AccordionGroup>

## Next steps

<CardGroup>
  <Card title="Structured Outputs" icon="brackets-curly" href="/structured-responses/structured-response-formatting">
    Enforce JSON schemas for consistent responses
  </Card>

  <Card title="Text Models" icon="message" href="/guides/querying-text-models">
    Learn about chat completions and other APIs
  </Card>

  <Card title="Deployments" icon="server" href="/guides/ondemand-deployments">
    Deploy models on dedicated GPUs
  </Card>

  <Card title="API Reference" icon="code" href="/api-reference/post-chatcompletions">
    Full chat completions API documentation
  </Card>
</CardGroup>


# Inference Error Codes
Source: https://docs.fireworks.ai/guides/inference-error-codes

Common error codes, their meanings, and resolutions for inference requests

Understanding error codes helps you quickly diagnose and resolve issues when making inference requests to the Fireworks API.

## Common error codes

| **Code** | **Error Name**        | **Possible Issue(s)**                                                                                                                            | **How to Resolve**                                                                                                                                                            |
| -------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `400`    | Bad Request           | Invalid input or malformed request.                                                                                                              | Review the request parameters and ensure they match the expected format.                                                                                                      |
| `401`    | Unauthorized          | Invalid API key or insufficient permissions.                                                                                                     | Verify your API key and ensure it has the correct permissions.                                                                                                                |
| `402`    | Payment Required      | Account is not on a paid plan or has exceeded usage limits.                                                                                      | Check your billing status and ensure your payment method is up to date. Upgrade your plan if necessary.                                                                       |
| `403`    | Forbidden             | Authentication issues.                                                                                                                           | Verify you have the correct API key.                                                                                                                                          |
| `404`    | Not Found             | The API endpoint path doesn't exist, the model doesn't exist, the model is not deployed, or you don't have permission to access it.              | Verify the URL path in your request and ensure you are using the correct API endpoint. Check if the model exists and is available. Ensure you have the necessary permissions. |
| `405`    | Method Not Allowed    | Using an unsupported HTTP method (e.g., using GET instead of POST).                                                                              | Check the API documentation for the correct HTTP method.                                                                                                                      |
| `408`    | Request Timeout       | The request took too long to complete, possibly due to server overload or network issues.                                                        | Retry the request after a brief wait. Consider increasing the timeout value if applicable.                                                                                    |
| `412`    | Precondition Failed   | Account is suspended or there's an issue with account status. This error also occurs when attempting to invoke a LoRA model that failed to load. | Check your account status and billing information. For LoRA models, ensure the model was uploaded correctly and is compatible. Contact support if the issue persists.         |
| `413`    | Payload Too Large     | Input data exceeds the allowed size limit.                                                                                                       | Reduce the size of the input payload (e.g., by trimming large text or image data).                                                                                            |
| `429`    | Too Many Requests     | Rate limited (serverless) or deployment capacity exceeded (dedicated/on-demand).                                                                 | See [understanding 429 errors](#understanding-429-errors) below.                                                                                                              |
| `500`    | Internal Server Error | Server-side code bug that is unlikely to resolve on its own.                                                                                     | Contact Fireworks support immediately, as this error typically requires intervention from the engineering team.                                                               |
| `502`    | Bad Gateway           | The server received an invalid response from an upstream server.                                                                                 | Wait and retry the request. If the error persists, it may indicate a server outage.                                                                                           |
| `503`    | Service Unavailable   | The service is down for maintenance or experiencing issues.                                                                                      | Retry the request after some time. Check the [status page](https://status.fireworks.ai) for maintenance announcements.                                                        |
| `504`    | Gateway Timeout       | The server did not receive a response in time from an upstream server.                                                                           | Wait briefly and retry the request. Consider using a shorter input prompt if applicable.                                                                                      |
| `520`    | Unknown Error         | An unexpected error occurred with no clear explanation.                                                                                          | Retry the request. If the issue persists, contact support for further assistance.                                                                                             |

## Understanding 429 errors

HTTP 429 (`Too Many Requests`) can be returned on both serverless and dedicated/on-demand deployments, but the cause and recommended action differ.

### Serverless deployments

On serverless, a 429 means your account has exceeded its current serverless request or TPM limit. Standard serverless, Priority tier, and Fast all use the same public rate-limit policy, which combines request-rate limits with adaptive TPM limits and is designed to prevent very spiky traffic. To resolve:

* Wait briefly and retry with exponential backoff
* Smooth sudden bursts or spread traffic more evenly over time
* Check the rate-limit response headers returned with your requests
* Review [Serverless rate limits](/serverless/rate-limits) if you need more headroom, or use an [on-demand deployment](/guides/ondemand-deployments) for dedicated capacity

### Dedicated and on-demand deployments

On dedicated and on-demand deployments, **there are no account-level rate limits**. A 429 instead indicates that your deployment's processing capacity is saturated. The inference server returns 429 when the number of queued and active requests exceeds what the deployment's GPUs can handle at that moment.

This is a capacity signal, not quota enforcement. To resolve:

* **Reduce burst concurrency** — lower the number of parallel requests or add client-side rate limiting with backoff
* **Scale up the deployment** — add more replicas or GPUs to increase throughput
* **Optimize request patterns** — use shorter prompts, reduce max output tokens, or batch requests to lower per-request resource consumption

<Tip>
  If you consistently see 429 errors on a dedicated or on-demand deployment, it's an indicator that your current GPU allocation is undersized for your traffic. [Contact us](https://fireworks.ai/company/contact-us) to discuss increasing your deployment capacity.
</Tip>

## Troubleshooting tips

If you encounter an error not listed here:

* Review the API documentation for the correct usage of endpoints and parameters
* Check the [Fireworks status page](https://status.fireworks.ai) for any ongoing service disruptions
* Contact support at [support@fireworks.ai](mailto:support@fireworks.ai) or join our [Discord](https://discord.gg/fireworks-ai)

<Tip>
  Enable detailed error logging in your application to capture the full error response, including error messages and request IDs, which helps with debugging.
</Tip>


# Deployments
Source: https://docs.fireworks.ai/guides/ondemand-deployments

Configure and manage on-demand deployments on dedicated GPUs

<Info>
  **New to deployments?** Start with our [Deployments Quickstart](/getting-started/ondemand-quickstart) to deploy and query your first model in minutes, then return here to learn about configuration options.
</Info>

On-demand deployments give you dedicated GPUs for your models, providing several advantages over serverless:

* **Better performance** – Lower latency, higher throughput, and predictable performance unaffected by other users
* **No hard rate limits** – Only limited by your deployment's capacity
* **Cost-effective at scale** – Cheaper under high utilization. Unlike serverless models (billed per token), on-demand deployments are [billed by GPU-second](https://fireworks.ai/pricing).
* **Broader model selection** – Access models not available on serverless
* **Custom models** – Upload your own models (for supported architectures) from Hugging Face or elsewhere

Need higher GPU quotas or want to reserve capacity? [Contact us](https://fireworks.ai/contact).

## Creating & querying deployments

**Create a deployment:**

```bash theme={null}
# This command returns your accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID> - save it for querying
firectl deployment create accounts/fireworks/models/<MODEL_NAME> --wait
```

<Warning>
  **Deployment placement (`--region`) must be set at creation time and cannot be changed in place.**

  If you do not specify `--region`, the deployment is pinned to a single datacenter at creation time and will not be automatically migrated later.

  For production workloads that need geographic availability or capacity failover, always set `--region` explicitly:

  ```bash theme={null}
  firectl deployment create accounts/fireworks/models/<MODEL_NAME> --region GLOBAL   # recommended default
  firectl deployment create accounts/fireworks/models/<MODEL_NAME> --region US
  firectl deployment create accounts/fireworks/models/<MODEL_NAME> --region EUROPE
  firectl deployment create accounts/fireworks/models/<MODEL_NAME> --region APAC
  ```
</Warning>

### Check current placement

```bash theme={null}
firectl deployment get <DEPLOYMENT_ID>
```

The deployment metadata shows where the deployment is currently allowed to schedule replicas (placement / region configuration).

### Change placement

There is no supported command to change region placement on an existing deployment. To change placement, recreate the deployment:

```bash theme={null}
# 1. Create replacement with correct region
firectl deployment create accounts/fireworks/models/<MODEL_NAME> \
  --deployment-shape <shape> \
  --region GLOBAL \
  --min-replica-count 1

# 2. Verify it's healthy, then point your app at the new endpoint

# 3. Delete old deployment
firectl deployment delete <OLD_DEPLOYMENT_ID>
```

See [Regions](/deployments/regions) for mega-regions and hardware availability.

See [Deployment shapes](#deployment-shapes) below to optimize for speed, throughput, or cost.

**Query your deployment:**

After creating a deployment, query it using this format:

```
accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>
```

<Tip>
  You can find your deployment name anytime with `firectl deployment list` and `firectl deployment get <DEPLOYMENT_ID>`.
</Tip>

**Example:**

```
accounts/alice/deployments/12345678
```

### Code examples

<Tabs>
  <Tab title="Python (Fireworks SDK)">
    ```python theme={null}
    from fireworks import Fireworks

    client = Fireworks()

    response = client.chat.completions.create(
      model="accounts/fireworks/models/gpt-oss-120b#<DEPLOYMENT_NAME>",
      messages=[{"role": "user", "content": "Explain quantum computing in simple terms"}]
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="Python (OpenAI SDK)">
    ```python theme={null}
    import os
    from openai import OpenAI

    client = OpenAI(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference/v1"
    )

    response = client.chat.completions.create(
        model="accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>",
        messages=[{"role": "user", "content": "Explain quantum computing in simple terms"}]
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    import OpenAI from "openai";

    const client = new OpenAI({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference/v1",
    });

    const response = await client.chat.completions.create({
      model: "accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>",
      messages: [
        {
          role: "user",
          content: "Explain quantum computing in simple terms",
        },
      ],
    });

    console.log(response.choices[0].message.content);
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    curl https://api.fireworks.ai/inference/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $FIREWORKS_API_KEY" \
      -d '{
        "model": "accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>",
        "messages": [
          {
            "role": "user",
            "content": "Explain quantum computing in simple terms"
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

### Deployment status states

Deployment states from the Gateway API spec:

* `CREATING` - still being created
* `READY` - ready to be used
* `UPDATING` - in-progress updates happening
* `DELETING` - being deleted
* `DELETED` - soft-deleted
* `FAILED` - creation failed (see status for details)

UI-only states are display labels derived from deployment fields:

* `Inactive`: `state == READY && max_replica_count == 0 && ready_replica_count == 0`
* `Scaled to 0`: `state == READY && min_replica_count == 0 && max_replica_count > 0 && desired_replica_count == 0 && ready_replica_count == 0`

These are display labels computed from deployment fields; they are not new backend `Deployment.State` enum values.

## Deployment shapes

Deployment shapes are the primary way to configure deployments. They're pre-configured templates optimized for speed, cost, or efficiency, including hardware, quantization, and other [performance factors](/faq/deployment/performance/optimization#performance-factors).

* **Fast** – Low latency for interactive workloads
* **Throughput** – Cost-per-token at scale for high-volume workloads
* **Minimal** – Lowest cost for testing or light workloads

**Usage:**

```bash theme={null}
# List available shapes
firectl deployment-shape-version list --base-model <model-id>

# Create with a shape (shorthand)
firectl deployment create accounts/fireworks/models/deepseek-v3 --deployment-shape throughput

# Create with full shape ID
firectl deployment create accounts/fireworks/models/llama-v3p3-70b-instruct \
  --deployment-shape accounts/fireworks/deploymentShapes/llama-v3p3-70b-instruct-fast

# View shape details
firectl deployment-shape-version get <full-deployment-shape-version-id>
```

<Tip>
  Need even better performance with tailored optimizations? [Contact our team](https://fireworks.ai/contact).
</Tip>

## Managing & configuring deployments

### Basic management

```bash theme={null}
# List all deployments
firectl deployment list

# Check deployment status
firectl deployment get <DEPLOYMENT_ID>

# Delete a deployment
firectl deployment delete <DEPLOYMENT_ID>
```

<Note>
  By default, deployments scale to zero if unused for 1 hour. Deployments with min replicas set to 0 are automatically deleted after 7 days of no traffic.
</Note>

<Warning>
  When a deployment is scaled to zero, requests return a `503` error immediately while the deployment scales up. Your application should implement retry logic to handle this. See [Scaling from zero behavior](/deployments/autoscaling#scaling-from-zero-behavior) for implementation details.
</Warning>

### GPU hardware

Choose GPU type with `--accelerator-type`:

* `NVIDIA_A100_80GB`
* `NVIDIA_H100_80GB`
* `NVIDIA_H200_141GB`

GPU availability varies by [region](/deployments/regions). See [Hardware selection guide→](https://docs.fireworks.ai/faq/deployment/ondemand/hardware-options#hardware-selection)

### Autoscaling

Control replica counts, scale timing, and load targets for your deployment.

See the [Autoscaling guide](/deployments/autoscaling) for configuration options.

### Multiple GPUs per replica

Use multiple GPUs to improve latency and throughput:

```bash theme={null}
firectl deployment create <MODEL_NAME> --accelerator-count 2
```

More GPUs = faster generation. Note that scaling is sub-linear (2x GPUs ≠ 2x performance).

## Preemptible deployments (eval / batch)

For the fine-tuning eval workflow, see [Evaluating Fine Tuned Models](/fine-tuning/evaluating-fine-tuned-models).

Preemptible deployments let you run **evaluation and batch workloads without holding dedicated on-demand capacity**. Passing `--preemptible` at creation opts the deployment into *borrowing idle reserved GPUs* instead of reserving GPUs exclusively for you, so you do not pay to hold dedicated capacity for an eval that only runs occasionally.

Because the capacity is borrowed, it can be **reclaimed (preempted) at any time** and is not guaranteed. In practice a preemptible deployment usually stays up long enough to run an eval end to end, but you must treat disappearance as a normal outcome, not an error.

<Warning>
  **Eval and batch workloads only.** A preemptible deployment can be preempted mid-request and disappear with no warning. **Never** point production or latency-sensitive traffic at a preemptible deployment.
</Warning>

### Guarantees vs. risks

|                                                      | Preemptible (`--preemptible`)        | Standard on-demand                             |
| ---------------------------------------------------- | ------------------------------------ | ---------------------------------------------- |
| **Capacity model**                                   | Borrows idle reserved GPUs           | Reserves dedicated GPUs for you                |
| **Cost**                                             | No charge to hold dedicated capacity | Billed by GPU-second while replicas are active |
| **Can be reclaimed mid-request?**                    | Yes, at any time, without warning    | No                                             |
| **Availability guarantee**                           | None (treat as best-effort)          | Held for you once scheduled                    |
| **Safe for production / latency-sensitive traffic?** | No                                   | Yes                                            |
| **Intended for**                                     | Evaluation, batch, one-off runs      | Production and interactive serving             |

<Note>
  **No unique availability risk.** Preemptible does not make capacity *harder* to get. If there is no capacity for a preemptible deployment, there is no capacity for an on-demand one either. The only difference is that a preemptible deployment can also be taken back after it starts.
</Note>

### Requirements

* **`firectl` >= 1.7.26.** `--preemptible` is a newer flag and is **silently ignored** on older builds (no error, and the deployment is created as standard on-demand). Check with `firectl version` and upgrade if you are below this version.

### Create a preemptible deployment

```bash theme={null}
firectl deployment create accounts/fireworks/models/<MODEL> \
  -a <ACCOUNT_ID> \
  --deployment-id <NAME> --display-name <NAME> \
  --deployment-shape accounts/fireworks/deploymentShapes/<SHAPE> \
  --min-replica-count 1 --max-replica-count 1 \
  --preemptible --wait
```

`--wait` blocks until the deployment is ready (1 hour default; tune with `--wait-timeout`).

<Warning>
  **`--preemptible` is immutable.** It is set at creation time and cannot be toggled on or off later. To change it, delete the deployment and recreate it.
</Warning>

Check status and then send eval requests as you would to any deployment:

```bash theme={null}
firectl deployment get <NAME> -a <ACCOUNT_ID>
```

When the eval is done, delete the deployment to release the borrowed capacity:

```bash theme={null}
firectl deployment delete <NAME> -a <ACCOUNT_ID>
```

## Advanced

* **[Speculative decoding](/deployments/speculative-decoding)** - Speed up text generation using draft models or n-gram speculation
* **[Quantization](/models/quantization)** - Reduce model precision (e.g., FP16 to FP8) to improve speeds and reduce costs by 30-50%
* **[Performance benchmarking](/deployments/benchmarking)** - Measure and optimize your deployment's performance with load testing
* **[Managing default deployments](/deployments/managing-default-deployments)** - Control which deployment handles queries when using just the model name
* **[Publishing deployments](/deployments/publishing-deployments)** - Make your deployment accessible to other Fireworks users

## Next steps

<CardGroup>
  <Card title="Autoscaling" href="/deployments/autoscaling" icon="arrows-up-down">
    Configure autoscaling for optimal cost and performance
  </Card>

  <Card title="Upload custom models" href="/models/uploading-custom-models" icon="cloud-arrow-up">
    Deploy your own models from Hugging Face
  </Card>

  <Card title="Quantization" href="/models/quantization" icon="compress">
    Reduce costs with model quantization
  </Card>

  <Card title="Regions" href="/deployments/regions" icon="earth-americas">
    Choose deployment regions for optimal latency
  </Card>

  <Card title="Reserved capacity" href="/deployments/reservations" icon="calendar-check">
    Purchase reserved GPUs for guaranteed capacity
  </Card>

  <Card title="Fine-tuning" href="/fine-tuning/finetuning-intro" icon="wand-magic-sparkles">
    Fine-tune models for your specific use case
  </Card>
</CardGroup>


# Using predicted outputs
Source: https://docs.fireworks.ai/guides/predicted-outputs

Use Predicted Outputs to boost output generation speeds for editing / rewriting use cases

<Tip>
  This feature is in beta and we are working on improvements. We welcome your feedback on [Discord](https://discord.gg/fireworks-ai)
</Tip>

In cases where large parts of the LLM output are known in advance, e.g. editing or rewriting a document or code snippet, you can improve output generation speeds with predicted outputs. Predicted outputs allows you to provide strong "guesses" of what output may look like.

To use Predicted Outputs, set the `prediction` field in the Fireworks API with the predicted output. For example, you may want to edit a survey and add an option to contact users by text message:

```
{
  "questions": [
    {
      "question": "Name",
      "type": "text"
    },
    {
      "question": "Age",
      "type": "number"
    },
    {
      "question": "Feedback",
      "type": "text_area"
    },
    {
      "question": "How to Contact",
      "type": "multiple_choice",
      "options": ["Email", "Phone"],
      "optional": true
    }
  ]
}
```

In this case, we expect most of the code will remain the same. We set the ‘prediction’ field to be the original survey code. The output generation speed increases using predicted outputs.

```python Python (Fireworks) theme={null}
from fireworks.client import Fireworks
 
code = """{
"questions": [
    {
      "question": "Name",
      "type": "text"
    },
    {
      "question": "Age",
      "type": "number"
    },
    {
      "question": "Feedback",
      "type": "text_area"
    },
    {
      "question": "How to Contact",
      "type": "multiple_choice",
      "options": ["Email", "Phone"],
      "optional": true
    }
  ]
}
"""

client = Fireworks(api_key="<FIREWORKS_API_KEY>")

response = client.chat.completions.create(
  model="accounts/fireworks/models/llama-v3p1-70b-instruct",
  messages=[{
      "role": "user",
      "content": "Edit the How to Contact question to add an option called Text Message. Output the full edited code, with no markdown or explanations.",
    },
    {
      "role": "user",
      "content": code
    }
  ],
  temperature=0,
  prediction={"type": "content", "content": code}
)

print(response.choices[0].message.content)
```

### Additional information on Predicted Outputs:

* Using Predicted Outputs is free at this time
* We recommend setting `temperature=0` for best results for most intended use cases of Predicted Outputs. In these cases, using Predicted Outputs does not impact the quality of outputs generated
* If the prediction is substantially different from the generated output, output generation speed may decrease
* The max length of the `prediction` field is set by `max_tokens` and is 2048 by default, and needs to be updated if you have a longer input and prediction.
* If you are using an on-demand deployment, you can set `rewrite_speculation=True` and potentially get even faster output generation. We are working on rolling this out to Serverless soon.


# Prompt caching
Source: https://docs.fireworks.ai/guides/prompt-caching



Prompt caching is a performance optimization feature that allows Fireworks to
respond faster to requests with prompts that share common prefixes. In many
situations, it can reduce time to first token (TTFT) by as much as 80%.

Prompt caching is enabled by default for all Fireworks models and deployments.

For serverless models, cached prompt tokens are discounted compared to regular prompt tokens. The default discount is 50%, but the exact discount varies by model. Check the [Model Library](https://fireworks.ai/models) for model-specific cached token pricing.

For dedicated deployments, prompt caching frees up resources, leading to higher
throughput on the same hardware. Cached tokens on dedicated deployments are close to
free for you, because they affect context length but do not need extra processing.

## Using prompt caching

### Common use cases

Requests to LLMs often share a large portion of their prompt. For example:

* Long system prompts with detailed instructions
* Descriptions of available tools for function calling
* Growing previous conversation history for chat use cases
* Shared per-user context, like a current file for a coding assistant

Prompt caching avoids re-processing the cached prefix of the prompt and starts output generation much sooner.

### Structuring prompts for caching

Prompt caching works only for exact prefix matches within a prompt. To
realize caching benefits, place static content like instructions and examples at
the beginning of your prompt, and put variable content, such as user-specific
information, at the end.

For function calling models, tools are considered part of the prompt.

### Optimizing inference request for caching

Prompt caching only works within 1 replica. If you are using serverless or a
deployment with multiple replicas, you need to give us hints for where to send
the traffic to maximize prompt cache hit rates. To do this, you can send us a
unique identifier for each user or session, when you expect the prompts to share
the same prefix.

You may place this identifier either in the `user` field of the request body or
in the `x-session-affinity` header.

<Tip>
  If you're collecting RL rollouts and want per-trajectory metrics plus
  stickiness across multi-turn agent interactions, also set
  `x-multi-turn-session-id`. See [Inference for RL
  rollouts](/guides/rollout-inference#session-affinity).
</Tip>

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    import os
    from openai import OpenAI

    client = OpenAI(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference/v1",
    )

    response = client.chat.completions.create(
        model="accounts/fireworks/models/<MODEL_ID>",
        messages=[{
            "role": "user",
            "content": "Explain quantum computing in simple terms"
        }],
        # A single client can make requests with different sessions.
        extra_headers={
            "x-session-affinity": "session-id-123"
        }
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    import OpenAI from "openai";

    const client = new OpenAI({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference/v1",
    });

    const response = await client.chat.completions.create({
      model: "accounts/fireworks/models/<MODEL_ID>",
      messages: [
        {
          role: "user",
          content: "Explain quantum computing in simple terms",
        },
      ],
      // A single client can make requests with different sessions.
      extra_headers: {
        x-session-affinity: "session-id-123"
      }
    });

    console.log(response.choices[0].message.content);
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    curl https://api.fireworks.ai/inference/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $FIREWORKS_API_KEY" \
      -H "x-session-affinity: session-id-123" \
      -d '{
        "model": "accounts/fireworks/models/",
        "messages": [
          {
            "role": "user",
            "content": "Explain quantum computing in simple terms"
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

### Resetting or isolating cache state

When you want new traffic to stop reusing an older cached prefix, Fireworks supports a few options:

* Change the session key in `user` or `x-session-affinity`. This moves traffic onto a different sticky routing key.
* Set a new `x-prompt-cache-isolation-key` header or `prompt_cache_isolation_key` request field. This forces cache separation even when the prompt text is otherwise identical.
* For hot-load or weight-sync workflows, set `reset_prompt_cache` to `all`, `none`, or `new_session` when you publish a new snapshot.

## Prompt optimization techniques for maximum cache hits

Due to the autoregressive nature of LLMs, even a single-token difference can invalidate the cache from that token onward. Here are key strategies to maximize your cache hit rates:

### Keep your prompt prefix stable

The most critical rule for effective prompt caching is maintaining a stable prefix. Any change to the beginning of your prompt will invalidate the entire cache chain that follows.

<Warning>
  **Common mistake:** Including timestamps or other dynamic content at the beginning of your system prompt.

  ```python theme={null}
  # ❌ DON'T: This kills cache hit rates
  system_prompt = f"""
  Current time: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}
  You are a helpful AI assistant...
  """
  ```

  Even a one-second difference in the timestamp will invalidate the entire cache, making it completely ineffective.
</Warning>

### Structure prompts for caching success

**✅ DO:** Place static content first, dynamic content last

```python theme={null}
# ✅ Good: Static content first
system_prompt = """
You are a helpful AI assistant with expertise in software development.

Your guidelines:
- Provide clear, concise explanations
- Include practical examples when helpful
- Ask clarifying questions when requirements are unclear

Available tools:
- web_search: Search the internet for current information
- code_executor: Run code snippets safely
- file_manager: Read and write files
"""

# Build the complete prompt
user_message = ""

# Add dynamic content at the end
if user_context:
    user_message += f"User context: {user_context}\n\n"

if current_time_needed:
    user_message += f"Current time: {datetime.now().isoformat()}\n\n"

# User query goes last
user_message += user_query

response = client.chat.completions.create(
    model="accounts/fireworks/models/<MODEL_ID>",
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_message}
    ]
)
```

### Smart timestamp handling

When you need to provide current time information, consider these strategies:

**Option 1: Rounded timestamps**

```python theme={null}
# ✅ Round to larger intervals to increase cache hits
current_hour = datetime.now().replace(minute=0, second=0, microsecond=0)
system_prompt = f"""
You are a helpful assistant.
Current hour: {current_hour.strftime('%Y-%m-%d %H:00')}
...
"""
```

**Option 2: Conditional time injection**

```python theme={null}
# ✅ Only add time when the query actually needs it
def build_prompt(user_query, system_base):
    prompt = system_base

    # Only add timestamp for time-sensitive queries
    time_keywords = ['today', 'now', 'current', 'latest', 'recent']
    if any(keyword in user_query.lower() for keyword in time_keywords):
        prompt += f"\nCurrent time: {datetime.now().isoformat()}"

    prompt += f"\nUser: {user_query}"
    return prompt
```

**Option 3: Move time to user message**

```python theme={null}
# ✅ Keep system prompt static, add time context to user message
system_prompt = """
You are a helpful AI assistant...
""" # This stays the same

user_message = f"""
Current time: {datetime.now().isoformat()}

User query: {user_query}
"""

response = client.chat.completions.create(
    model="accounts/fireworks/models/<MODEL_ID>",
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_message}
    ]
)
```

By following these optimization techniques, you can significantly improve your application's performance through effective prompt caching while maintaining the quality and functionality of your AI system.

### How it works

Fireworks will automatically find the longest prefix of the request that is
present in the cache and reuse it. The remaining portion of the prompt will be
processed as usual.

The entire prompt is stored in the cache for future reuse. Cached prompts
usually stay in the cache for at least several minutes. Depending on the model,
load level, and deployment configuration, it can be up to several hours. The
oldest prompts are evicted from the cache first.

Prompt caching doesn't alter the result generated by the model. The response you
receive will be identical to what you would get if prompt caching was not used.
Each generation is sampled from the model independently on each request and is not
cached for future usage.

## Monitoring

For dedicated deployments, information about prompt caching is returned in the
response headers. The header `fireworks-prompt-tokens` contains the number of tokens
in the prompt, out of which `fireworks-cached-prompt-tokens` are cached.

Aggregated metrics are also available in the [usage dashboard](https://fireworks.ai/account/usage?type=deployments).

## Data privacy

Serverless deployments maintain separate caches for each Fireworks account to prevent data leakage and timing attacks.

Dedicated deployments by default share a single cache across all requests.
Because prompt caching doesn't change the outputs, privacy is preserved even
if the deployment powers a multi-tenant application. It does open a minor risk
of a timing attack: potentially, an adversary can learn that a particular prompt
is cached by observing the response time. To ensure full isolation, you can pass
the `x-prompt-cache-isolation-key` header or the [`prompt_cache_isolation_key`](/api-reference/post-chatcompletions)
field in the body of the request. It can contain an arbitrary string that acts
as an additional cache key, i.e., no sharing will occur between requests with
different IDs.

## Advanced: Migration and traffic management

When migrating between deployments that use prompt caching, it's crucial to implement proper traffic routing to maintain optimal cache hit rates. When gradually routing traffic to a new deployment, use consistent user/session-based sampling rather than random sampling.

Here's the recommended implementation for traffic routing:

```python theme={null}
import hashlib

# Configure traffic fraction (e.g., 20% to new deployment)
fireworks_traffic_fraction = 0.2
user_id = "session-id-123"

# Generate deterministic hash from user_id
hashed_user_id = int(hashlib.md5(user_id.encode()).hexdigest(), 16) # MD5 hash on user-id and convert to integer
MAX_HASH = 2**128 - 1  # MD5 hash maximum value

# Compute ratio for consistent routing
ratio = hashed_user_id / MAX_HASH # Returns 0.0 to 1.0

if (ratio < fireworks_traffic_fraction):
    send_to_new_deployment(user=hashed_user_id)  # Pass user ID for caching
else:
    send_elsewhere()  # Route to old deployment or serverless
```

<Warning>
  Avoid random sampling for traffic routing as it can negatively impact cache hit rates:

  ```python theme={null}
  # Don't do this:
  if random() < fireworks_traffic_fraction:  # ❌ Reduces cache effectiveness
    send_to_new_deployment(user=hashed_user_id)
  ```

  Using consistent user-based routing ensures complete user sessions are maintained on the same deployment, optimizing prompt cache performance regardless of the traffic fraction.
</Warning>


# Embeddings & Reranking
Source: https://docs.fireworks.ai/guides/querying-embeddings-models

Generate embeddings and rerank results for semantic search

Fireworks hosts embedding and reranking models, which are useful for tasks like RAG and semantic search.

## Generating embeddings

Embeddings models take text as input and output a vector of floating point numbers to use for tasks like similarity comparisons and search. Our embedding service is OpenAI compatible. Refer to OpenAI's embeddings [guide](https://platform.openai.com/docs/guides/embeddings)  and OpenAI's [embeddings documentation](https://platform.openai.com/docs/api-reference/embeddings) for more information on using these models.

### Choosing a model

Every option below uses the same endpoint, `POST https://api.fireworks.ai/inference/v1/embeddings`. What changes is the value you pass as `model`.

| Path                                    | `model` value                                     | In the Model Library? | Best for                                                          |
| --------------------------------------- | ------------------------------------------------- | --------------------- | ----------------------------------------------------------------- |
| **Qwen3 Embedding**                     | `fireworks/qwen3-embedding-8b` and similar        | Yes                   | Default choice for serverless semantic search                     |
| **Voyage AI**                           | Your deployment path                              | Yes                   | Retrieval quality with asymmetric query and document embeddings   |
| **Legacy BERT / sentence-transformers** | Hugging Face id, such as `BAAI/bge-small-en-v1.5` | **No**                | Small, cheap vectors and existing sentence-transformers pipelines |
| **Generative LLM**                      | `fireworks/gpt-oss-20b` and similar               | Yes, as chat models   | Hidden-state vectors when you are already using the model         |
| **Your own upload**                     | `accounts/<your-account>/models/<model-id>`       | In your account       | Custom or fine-tuned embedders                                    |

<Tabs>
  <Tab title="Qwen3">
    The Qwen3 Embedding family is hosted by Fireworks, with the 8B model available on serverless.

    | Model                                                                                  | Model ID                                                                                           | Context | Resizable | Availability             |
    | -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ------- | --------- | ------------------------ |
    | [Qwen3 Embedding 8B](https://app.fireworks.ai/models/fireworks/qwen3-embedding-8b)     | [`fireworks/qwen3-embedding-8b`](https://app.fireworks.ai/models/fireworks/qwen3-embedding-8b)     | 40k     | Yes       | Serverless and dedicated |
    | [Qwen3 Embedding 4B](https://app.fireworks.ai/models/fireworks/qwen3-embedding-4b)     | [`fireworks/qwen3-embedding-4b`](https://app.fireworks.ai/models/fireworks/qwen3-embedding-4b)     | 40k     | Yes       | Dedicated                |
    | [Qwen3 Embedding 0.6B](https://app.fireworks.ai/models/fireworks/qwen3-embedding-0p6b) | [`fireworks/qwen3-embedding-0p6b`](https://app.fireworks.ai/models/fireworks/qwen3-embedding-0p6b) | 32k     | Yes       | Dedicated                |

    Resizable models accept the `dimensions` parameter to return shorter vectors.

    Serverless usage of `qwen3-embedding-8b` is billed at \$0.10 per million tokens. The 4B and 0.6B models require a [dedicated deployment](/guides/ondemand-deployments), and you can also deploy the 8B model that way for dedicated capacity.
  </Tab>

  <Tab title="Voyage AI">
    Fireworks also hosts the Voyage AI by MongoDB family of embedding models and rerankers.

    | Model                                                                                    | Model ID                                                                                             | Context | Type       |
    | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | ------- | ---------- |
    | [Voyage 4 Large](https://app.fireworks.ai/models/fireworks/voyage-4-large)               | [`fireworks/voyage-4-large`](https://app.fireworks.ai/models/fireworks/voyage-4-large)               | 40k     | Embeddings |
    | [Voyage 4](https://app.fireworks.ai/models/fireworks/voyage-4)                           | [`fireworks/voyage-4`](https://app.fireworks.ai/models/fireworks/voyage-4)                           | 40k     | Embeddings |
    | [Voyage 4 Lite](https://app.fireworks.ai/models/fireworks/voyage-4-lite)                 | [`fireworks/voyage-4-lite`](https://app.fireworks.ai/models/fireworks/voyage-4-lite)                 | 40k     | Embeddings |
    | [Voyage 4 Nano](https://app.fireworks.ai/models/fireworks/voyage-4-nano)                 | [`fireworks/voyage-4-nano`](https://app.fireworks.ai/models/fireworks/voyage-4-nano)                 | 40k     | Embeddings |
    | [Voyage Multimodal 3.5](https://app.fireworks.ai/models/fireworks/voyage-multimodal-3-5) | [`fireworks/voyage-multimodal-3-5`](https://app.fireworks.ai/models/fireworks/voyage-multimodal-3-5) | 32k     | Embeddings |
    | [Voyage ReRank 2.5](https://app.fireworks.ai/models/fireworks/voyage-rerank-2-5)         | [`fireworks/voyage-rerank-2-5`](https://app.fireworks.ai/models/fireworks/voyage-rerank-2-5)         | 32k     | Reranking  |

    Voyage models are not available on serverless, so they always run on a [dedicated deployment](/guides/ondemand-deployments). They are also trained for asymmetric retrieval, so pass `input_type="document"` when you embed a corpus and `input_type="query"` when you embed a search query. The model prepends task-specific instructions internally, which improves retrieval quality over embedding both sides the same way.
  </Tab>
</Tabs>

### Making a request

How you address the model depends on whether it runs on serverless or on a dedicated deployment.

<Tabs>
  <Tab title="Serverless">
    Pass the model name directly.

    ```python Python theme={null}
    import requests

    url = "https://api.fireworks.ai/inference/v1/embeddings"

    payload = {
        "input": "The quick brown fox jumped over the lazy dog",
        "model": "fireworks/qwen3-embedding-8b",
    }

    headers = {
        "Authorization": "Bearer <FIREWORKS_API_KEY>",
        "Content-Type": "application/json"
    }

    response = requests.post(url, json=payload, headers=headers)

    print(response.json())
    ```
  </Tab>

  <Tab title="Dedicated">
    Pass your deployment path instead of a model name. Voyage models additionally take `input_type`.

    ```python Python theme={null}
    import requests

    url = "https://api.fireworks.ai/inference/v1/embeddings"

    # Copy both IDs from your deployment's page in the Fireworks dashboard.
    ACCOUNT_ID = "<YOUR_ACCOUNT_ID>"
    DEPLOYMENT_ID = "<YOUR_DEPLOYMENT_ID>"

    payload = {
        "input": "The quick brown fox jumped over the lazy dog",
        "model": f"accounts/{ACCOUNT_ID}/deployments/{DEPLOYMENT_ID}",
        "input_type": "document",
    }

    headers = {
        "Authorization": "Bearer <FIREWORKS_API_KEY>",
        "Content-Type": "application/json"
    }

    response = requests.post(url, json=payload, headers=headers)

    print(response.json())
    ```

    <Note>
      Dedicated deployments scale to zero when idle, so the first request after a quiet period can return a `503` while the deployment cold-starts. Retry with backoff.
    </Note>
  </Tab>
</Tabs>

To generate variable-length embeddings, you can add the `dimensions` parameter to the request, for example, `dimensions: 128`. Only resizable models accept it: the Qwen3 embedding models do, and among the legacy models only `nomic-ai/nomic-embed-text-v1.5` does. Any other model rejects the request with `model: <id> is not resizable`.

The API usage for embedding models is identical for BERT-based and LLM-based embeddings. Simply use the `/v1/embeddings` endpoint with your chosen model.

<Note>
  Not every model returns unit-length vectors. Qwen3 embeddings are not normalized, so divide by the L2 norm before using a dot product as a cosine similarity, or use a similarity function that normalizes for you. Most of the legacy BERT models do return normalized vectors.
</Note>

### Other embedding options

<AccordionGroup>
  <Accordion title="Use any LLM as an embeddings model">
    You can retrieve embeddings from many LLMs in our model library. Here are some examples that work with the embeddings API:

    * `fireworks/gpt-oss-20b`
    * `fireworks/gpt-oss-120b`
    * `fireworks/glm-5p2`

    A few caveats before you rely on this:

    * These vectors come from the model's last-token hidden states, not a dedicated embedding head, so they are not trained for retrieval. Prefer a purpose-built embedding model when retrieval quality matters.
    * Raw hidden states are not unit vectors. Pass `"normalize": true` if you plan to compare them with cosine similarity.
    * Not every model in the library supports `/v1/embeddings`. Unsupported models return an `embedding is not supported` error.

    <CodeGroup>
      ```python Python theme={null}
      import os

      from openai import OpenAI

      client = OpenAI(
          api_key=os.environ["FIREWORKS_API_KEY"],
          base_url="https://api.fireworks.ai/inference/v1",
      )

      resp = client.embeddings.create(
          model="fireworks/glm-5p2",
          input=["First chunk to embed", "Second chunk to embed"],
          extra_body={"normalize": True},
      )

      for row in resp.data:
          print(row.index, len(row.embedding))
      ```

      ```bash cURL theme={null}
      curl -X POST https://api.fireworks.ai/inference/v1/embeddings \
        -H "Authorization: Bearer $FIREWORKS_API_KEY" \
        -H "Content-Type: application/json" \
        -d '{
          "model": "fireworks/glm-5p2",
          "input": ["First chunk to embed", "Second chunk to embed"],
          "normalize": true
        }'
      ```
    </CodeGroup>

    <Note>
      `normalize` is not part of the OpenAI embeddings schema, so the Python SDK needs it in `extra_body` rather than as a top-level argument.
    </Note>
  </Accordion>

  <Accordion title="Bring your own model">
    You can also retrieve embeddings from any models you bring yourself through [custom model upload](/models/uploading-custom-models).
  </Accordion>

  <Accordion title="BERT-based models (legacy)">
    <Note>
      **These models are not in the Model Library.** You will not find them at [fireworks.ai/models](https://fireworks.ai/models), but they still work on serverless if you pass the exact Hugging Face style id in the `model` field, for example `nomic-ai/nomic-embed-text-v1.5` rather than `accounts/fireworks/models/...`.
    </Note>

    These models are serverless only; you cannot create a dedicated deployment for them.

    | Model id                             | Dimensions | Parameters | Resizable | Normalized |
    | ------------------------------------ | ---------- | ---------- | --------- | ---------- |
    | `BAAI/bge-small-en-v1.5`             | 384        | 33M        | No        | Yes        |
    | `thenlper/gte-base`                  | 768        | 109M       | No        | Yes        |
    | `BAAI/bge-base-en-v1.5`              | 768        | 109M       | No        | Yes        |
    | `nomic-ai/nomic-embed-text-v1.5`     | 768        | 137M       | Yes       | Yes        |
    | `nomic-ai/nomic-embed-text-v1`       | 768        | 137M       | No        | Yes        |
    | `thenlper/gte-large`                 | 1024       | 335M       | No        | Yes        |
    | `WhereIsAI/UAE-Large-V1`             | 1024       | 335M       | No        | No         |
    | `mixedbread-ai/mxbai-embed-large-v1` | 1024       | 335M       | No        | No         |

    **Resizable** means the model accepts the `dimensions` parameter. Non-resizable models reject it with `model: <id> is not resizable`. **Normalized** means the returned vectors are unit length, so a dot product is already the cosine similarity; for the two that are not, normalize the output yourself before comparing.

    **Retired.** `sentence-transformers/all-MiniLM-L6-v2` and `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` are no longer enabled on the shared serverless pool and return a `not available` error. Use `BAAI/bge-small-en-v1.5` instead for a similarly small model.

    **Quickstart**

    <CodeGroup>
      ```python Python theme={null}
      import os

      from openai import OpenAI

      client = OpenAI(
          api_key=os.environ["FIREWORKS_API_KEY"],
          base_url="https://api.fireworks.ai/inference/v1",
      )

      MODEL = "BAAI/bge-small-en-v1.5"  # 384-dim; not listed in the Model Library

      resp = client.embeddings.create(
          model=MODEL,
          input=["first sentence", "second sentence"],
      )

      for row in resp.data:
          print(row.index, len(row.embedding))
      ```

      ```bash cURL theme={null}
      curl -X POST https://api.fireworks.ai/inference/v1/embeddings \
        -H "Authorization: Bearer $FIREWORKS_API_KEY" \
        -H "Content-Type: application/json" \
        -d '{
          "model": "BAAI/bge-small-en-v1.5",
          "input": ["first sentence", "second sentence"]
        }'
      ```
    </CodeGroup>

    **What to expect**

    * Vector length and normalization vary by model, as shown in the table above.
    * Billing follows [serverless embeddings pricing](/serverless/pricing#embeddings) by parameter bucket. The five models up to 150M parameters bill at $0.008 per 1M input tokens, and the three 335M models bill at $0.016.
    * Serverless only, with no dedicated deployment option.
  </Accordion>
</AccordionGroup>

## Reranking documents

Reranking models are used to rerank a list of documents based on a query. The `/rerank` endpoint provides a simple interface for this.

<Note>
  The `/rerank` endpoint does not yet support all models and parallelism options. For more flexibility, use the `/embeddings` endpoint with `return_logits` as shown in the next section.
</Note>

<Tabs>
  <Tab title="Qwen3">
    | Model                                                                                | Model ID                                                                                         | Context | Availability             |
    | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------ | ------- | ------------------------ |
    | [Qwen3 Reranker 8B](https://app.fireworks.ai/models/fireworks/qwen3-reranker-8b)     | [`fireworks/qwen3-reranker-8b`](https://app.fireworks.ai/models/fireworks/qwen3-reranker-8b)     | 40k     | Serverless and dedicated |
    | [Qwen3 Reranker 4B](https://app.fireworks.ai/models/fireworks/qwen3-reranker-4b)     | [`fireworks/qwen3-reranker-4b`](https://app.fireworks.ai/models/fireworks/qwen3-reranker-4b)     | 40k     | Dedicated                |
    | [Qwen3 Reranker 0.6B](https://app.fireworks.ai/models/fireworks/qwen3-reranker-0p6b) | [`fireworks/qwen3-reranker-0p6b`](https://app.fireworks.ai/models/fireworks/qwen3-reranker-0p6b) | 40k     | Dedicated                |

    Serverless usage of `qwen3-reranker-8b` is billed at \$0.20 per million tokens. The 4B and 0.6B models require a dedicated deployment.
  </Tab>

  <Tab title="Voyage AI">
    | Model                                                                            | Model ID                                                                                     | Context | Availability |
    | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ------- | ------------ |
    | [Voyage ReRank 2.5](https://app.fireworks.ai/models/fireworks/voyage-rerank-2-5) | [`fireworks/voyage-rerank-2-5`](https://app.fireworks.ai/models/fireworks/voyage-rerank-2-5) | 32k     | Dedicated    |

    Voyage ReRank 2.5 goes through the same `/rerank` endpoint once you deploy it.
  </Tab>
</Tabs>

### Making a request

<Tabs>
  <Tab title="Serverless">
    ```python Python theme={null}
    import requests

    url = "https://api.fireworks.ai/inference/v1/rerank"

    payload = {
        "model": "fireworks/qwen3-reranker-8b",
        "query": "What is the capital of France?",
        "documents": [
            "Paris is the capital and largest city of France, home to the Eiffel Tower and the Louvre Museum.",
            "France is a country in Western Europe known for its wine, cuisine, and rich history.",
            "The weather in Europe varies significantly between northern and southern regions.",
            "Python is a popular programming language used for web development and data science."
        ],
        "top_n": 3,
        "return_documents": True
    }

    headers = {
        "Authorization": "Bearer <FIREWORKS_API_KEY>",
        "Content-Type": "application/json"
    }

    response = requests.post(url, json=payload, headers=headers)

    print(response.json())
    ```
  </Tab>

  <Tab title="Dedicated">
    The request body is identical; only the `model` changes to your reranker's deployment path.

    ```python Python theme={null}
    import requests

    url = "https://api.fireworks.ai/inference/v1/rerank"

    # Copy both IDs from your deployment's page in the Fireworks dashboard.
    ACCOUNT_ID = "<YOUR_ACCOUNT_ID>"
    DEPLOYMENT_ID = "<YOUR_RERANK_DEPLOYMENT_ID>"

    payload = {
        "model": f"accounts/{ACCOUNT_ID}/deployments/{DEPLOYMENT_ID}",
        "query": "What is the capital of France?",
        "documents": [
            "Paris is the capital and largest city of France, home to the Eiffel Tower and the Louvre Museum.",
            "France is a country in Western Europe known for its wine, cuisine, and rich history.",
            "The weather in Europe varies significantly between northern and southern regions.",
            "Python is a popular programming language used for web development and data science."
        ],
        "top_n": 3,
        "return_documents": True
    }

    headers = {
        "Authorization": "Bearer <FIREWORKS_API_KEY>",
        "Content-Type": "application/json"
    }

    response = requests.post(url, json=payload, headers=headers)

    print(response.json())
    ```
  </Tab>
</Tabs>

### Using the `/embeddings` endpoint

You can also use the `/embeddings` endpoint with the `return_logits` parameter to rerank documents. This approach supports more models and parallelism options.

The embedding model takes in token IDs for "yes" and "no" and outputs associated logits indicating how likely the document is relevant or not relevant to the query. You can obtain these token IDs using `tokenizer.convert_tokens_to_ids()` with the transformers library and the Qwen3 tokenizer.

<Tabs>
  <Tab title="Simple">
    ```python Python theme={null}
    import requests

    url = "https://api.fireworks.ai/inference/v1/embeddings"

    query = "What is the capital of France?"
    documents = [
        "Paris is the capital and largest city of France, home to the Eiffel Tower and the Louvre Museum.",
        "France is a country in Western Europe known for its wine, cuisine, and rich history.",
        "The weather in Europe varies significantly between northern and southern regions.",
        "Python is a popular programming language used for web development and data science."
    ]

    # Format prompts as query-document pairs using the Qwen3 Reranker format
    instruction = "Given a web search query, retrieve relevant passages that answer the query"
    prompts = [
        f"<Instruct>: {instruction}\n<Query>: {query}\n<Document>: {doc}"
        for doc in documents
    ]

    # Token IDs for "no" and "yes" in Qwen3 reranker models
    token_false_id = 2753   # "no"
    token_true_id = 9454    # "yes"

    payload = {
        "model": "fireworks/qwen3-reranker-8b",
        "input": prompts,
        "return_logits": [token_false_id, token_true_id],
        "normalize": True  # Applies softmax to the selected logits
    }

    headers = {
        "Authorization": "Bearer <FIREWORKS_API_KEY>",
        "Content-Type": "application/json"
    }

    response = requests.post(url, json=payload, headers=headers).json()

    # Extract relevance scores (probability of "yes" token)
    results = []
    for i, item in enumerate(response["data"]):
        probs = item["embedding"]  # [no_prob, yes_prob]
        relevance_score = probs[1]  # "yes" probability is the relevance score
        results.append({
            "index": i,
            "relevance_score": relevance_score,
            "document": documents[i]
        })

    # Sort by relevance score (highest first)
    results.sort(key=lambda x: x["relevance_score"], reverse=True)

    for result in results:
        print(f"Score: {result['relevance_score']:.4f} - {result['document'][:80]}...")

    ```
  </Tab>

  <Tab title="Parallel (asyncio)">
    For large document sets, you can improve throughput by sending multiple requests in parallel using minibatches:

    ```python Python theme={null}
    import asyncio
    import aiohttp

    url = "https://api.fireworks.ai/inference/v1/embeddings"

    query = "What is the capital of France?"
    documents = [...]  # Your list of documents

    # Format prompts as query-document pairs using the Qwen3 Reranker format
    instruction = "Given a web search query, retrieve relevant passages that answer the query"
    prompts = [
        f"<Instruct>: {instruction}\n<Query>: {query}\n<Document>: {doc}"
        for doc in documents
    ]

    # Token IDs for "no" and "yes" in Qwen3 reranker models
    token_false_id = 2753   # "no"
    token_true_id = 9454    # "yes"

    headers = {
        "Authorization": "Bearer <FIREWORKS_API_KEY>",
        "Content-Type": "application/json"
    }

    async def rerank_batch(session, batch_prompts):
        payload = {
            "model": "fireworks/qwen3-reranker-8b",
            "input": batch_prompts,
            "return_logits": [token_false_id, token_true_id],
            "normalize": True
        }
        async with session.post(url, json=payload, headers=headers) as response:
            return await response.json()

    async def rerank_parallel(prompts, batch_size=100):
        batches = [prompts[i:i+batch_size] for i in range(0, len(prompts), batch_size)]

        async with aiohttp.ClientSession() as session:
            tasks = [rerank_batch(session, batch) for batch in batches]
            results = await asyncio.gather(*tasks)

        # Combine results from all batches
        all_scores = []
        for result in results:
            for item in result["data"]:
                all_scores.append(item["embedding"][1])  # "yes" probability

        return all_scores

    scores = asyncio.run(rerank_parallel(prompts))
    ```
  </Tab>
</Tabs>

With `normalize=True`, the endpoint applies softmax to the selected logits, returning probabilities that sum to 1. The "yes" probability directly represents the relevance score.

## Building a two-stage retrieval pipeline

Embeddings and reranking are usually combined: vector search recalls a broad candidate set quickly, then a reranker reorders those candidates for precision. This cookbook builds that pipeline end to end with Voyage models on Fireworks and MongoDB vector search.

<Card title="Two-stage retrieval with Voyage embeddings and reranking" icon="magnifying-glass" href="https://colab.research.google.com/github/fw-ai/cookbook/blob/main/partners/voyage-ai/voyage_two_stage_retrieval.ipynb">
  Embed a corpus with a Voyage embedding model, retrieve with MongoDB `$vectorSearch`, and rerank the candidates with Voyage ReRank 2.5.
</Card>

## Deploying embeddings and reranking models

While Qwen3 Embedding 8b and Qwen3 Reranker 8b are available on serverless, you also have the option to deploy them via [on-demand deployments](/guides/ondemand-deployments). Voyage models require a dedicated deployment.

## Troubleshooting

<AccordionGroup>
  <Accordion title="I can't find my embedding model in the Model Library">
    Legacy BERT and sentence-transformers embedders are not listed in the Model Library, but they still serve on serverless. Pass the Hugging Face style id directly, for example `BAAI/bge-small-en-v1.5`. See [BERT-based models (legacy)](#other-embedding-options) for the current list.
  </Accordion>

  <Accordion title="What is the difference between `model not found` and `not available`?">
    `not found` means the id does not resolve, which usually points to a typo or a deployment path for a deployment that does not exist in your account. `not available` means the id is real but is not enabled on the shared serverless pool, which is what retired legacy embedders return. In that case, switch to another model from the legacy list or to `fireworks/qwen3-embedding-8b`.
  </Accordion>

  <Accordion title="My dedicated deployment returns a 503">
    Dedicated deployments scale to zero when idle, so the first request after a quiet period can fail while the deployment cold-starts. Retry with backoff rather than treating it as a hard failure.
  </Accordion>
</AccordionGroup>


# Text Models
Source: https://docs.fireworks.ai/guides/querying-text-models

Query, track and manage inference for text models

<Info>
  New to Fireworks? Start with the [Serverless Quickstart](/getting-started/quickstart) for a step-by-step guide to making your first API call.
</Info>

Fireworks provides fast, cost-effective access to leading open-source text models through OpenAI-compatible APIs. Query models via serverless inference or dedicated deployments using the chat completions API (recommended), completions API, or responses API.

[Browse 100+ available models →](https://fireworks.ai/models)

## Chat Completions API

<Tabs>
  <Tab title="Python (Fireworks SDK)">
    ```python theme={null}
    from fireworks import Fireworks

    client = Fireworks()

    response = client.chat.completions.create(
      model="accounts/fireworks/models/deepseek-v3p1",
      messages=[{"role": "user", "content": "Explain quantum computing in simple terms"}]
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="Python (OpenAI SDK)">
    ```python theme={null}
    import os
    from openai import OpenAI

    client = OpenAI(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference/v1"
    )

    response = client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[{
            "role": "user",
            "content": "Explain quantum computing in simple terms"
        }]
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    import OpenAI from "openai";

    const client = new OpenAI({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference/v1",
    });

    const response = await client.chat.completions.create({
      model: "accounts/fireworks/models/deepseek-v3p1",
      messages: [
        {
          role: "user",
          content: "Explain quantum computing in simple terms",
        },
      ],
    });

    console.log(response.choices[0].message.content);
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    curl https://api.fireworks.ai/inference/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $FIREWORKS_API_KEY" \
      -d '{
        "model": "accounts/fireworks/models/deepseek-v3p1",
        "messages": [
          {
            "role": "user",
            "content": "Explain quantum computing in simple terms"
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

<Tip>
  Most models automatically format your messages with the correct template. To verify the exact prompt used, enable the [`echo`](/guides/querying-text-models) parameter.
</Tip>

For **Priority tier** (`service_tier: "priority"`) and **Fast**, see [Serverless Serving Paths](/serverless/serving-paths).

## Alternative query methods

Fireworks also supports [Completions API](/guides/completions-api) and [Responses API](/guides/response-api).

## Querying dedicated deployments

For consistent performance, guaranteed capacity, or higher throughput, you can query [on-demand deployments](/guides/ondemand-deployments) instead of serverless models. Deployments use the same APIs with a deployment-specific identifier:

```
accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>
```

For example:

```python theme={null}
response = client.chat.completions.create(
    model="accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>",
    messages=[{"role": "user", "content": "Hello"}]
)
```

## Common patterns

### Multi-turn conversations

Maintain conversation history by including all previous messages:

```python theme={null}
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What's the capital of France?"},
    {"role": "assistant", "content": "The capital of France is Paris."},
    {"role": "user", "content": "What's its population?"}
]

response = client.chat.completions.create(
    model="accounts/fireworks/models/deepseek-v3p1",
    messages=messages
)

print(response.choices[0].message.content)
```

The model uses the full conversation history to provide contextually relevant responses.

### System prompts

Override the default system prompt by setting the first message with `role: "system"`:

```python theme={null}
messages = [
    {"role": "system", "content": "You are a helpful Python expert who provides concise code examples."},
    {"role": "user", "content": "How do I read a CSV file?"}
]

response = client.chat.completions.create(
    model="accounts/fireworks/models/deepseek-v3p1",
    messages=messages
)
```

To completely omit the system prompt, set the first message's `content` to an empty string.

### Streaming responses

Stream tokens as they're generated for real time, interactive UX. Covered in detail in the [Serverless Quickstart](/getting-started/quickstart#streaming-responses).

```python theme={null}
stream = client.chat.completions.create(
    model="accounts/fireworks/models/deepseek-v3p1",
    messages=[{"role": "user", "content": "Tell me a story"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

**Aborting streams:** Close the connection to stop generation and avoid billing for ungenerated tokens:

```python theme={null}
for chunk in stream:
    print(chunk.choices[0].delta.content, end="")
    if some_condition:
        stream.close()
        break
```

### Async requests

Use async clients to make multiple concurrent requests for better throughput:

<Tabs>
  <Tab title="Python (Fireworks SDK)">
    ```python theme={null}
    from fireworks import AsyncFireworks

    client = AsyncFireworks()

    async def main():
      response = await client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[{"role": "user", "content": "Hello"}]
      )
      print(response.choices[0].message.content)

    asyncio.run(main())
    ```
  </Tab>

  <Tab title="Python (OpenAI SDK)">
    ```python theme={null}
    import asyncio
    from openai import AsyncOpenAI

    client = AsyncOpenAI(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference/v1"
    )

    async def main():
        response = await client.chat.completions.create(
            model="accounts/fireworks/models/deepseek-v3p1",
            messages=[{"role": "user", "content": "Hello"}]
        )
        print(response.choices[0].message.content)

    asyncio.run(main())
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    import OpenAI from "openai";

    const client = new OpenAI({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference/v1",
    });

    async function main() {
      const response = await client.chat.completions.create({
        model: "accounts/fireworks/models/deepseek-v3p1",
        messages: [{ role: "user", content: "Hello" }],
      });
      console.log(response.choices[0].message.content);
    }

    main();
    ```
  </Tab>
</Tabs>

### Usage & performance tracking

Every response includes token usage information and performance metrics for debugging and observability. For aggregate metrics over time, see the [usage dashboard](https://app.fireworks.ai/account/usage).

<Note>
  **The analytics and usage dashboard measures server-acknowledged requests, not every client-observed outcome.**

  The dashboard counts requests that successfully reached the Fireworks API. It does not capture connection timeouts before the request lands on the server, client-side retries before a successful attempt, or failures on the network path between your application and the API.

  If your application reports failures but the dashboard looks healthy, check client timeout configuration and network connectivity. For dedicated deployments, [Prometheus-style metrics](/deployments/exporting-metrics) reflect server-side behavior for that deployment.
</Note>

**Token usage** (prompt, completion, total tokens) is included in the response body for all requests.

**Performance metrics** (latency, time-to-first-token, etc.) are included in response headers for non-streaming requests. For streaming requests, use the [`perf_metrics_in_response`](/api-reference/post-chatcompletions) parameter to include all metrics in the response body.

<Tabs>
  <Tab title="Non-streaming">
    ```python theme={null}
    response = client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[{"role": "user", "content": "Hello"}]
    )

    # Token usage (always included)
    print(response.usage.prompt_tokens)      # Tokens in your prompt
    print(response.usage.completion_tokens)  # Tokens generated
    print(response.usage.total_tokens)       # Total tokens billed

    # Performance metrics are in response headers:
    # fireworks-prompt-tokens, fireworks-server-time-to-first-token, etc.
    ```
  </Tab>

  <Tab title="Streaming (usage only)">
    ```python theme={null}
    stream = client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[{"role": "user", "content": "Hello"}],
        stream=True
    )

    for chunk in stream:
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="")
        
        # Usage is included in the final chunk
        if chunk.usage:
            print(f"\n\nTokens used: {chunk.usage.total_tokens}")
            print(f"Prompt: {chunk.usage.prompt_tokens}, Completion: {chunk.usage.completion_tokens}")
    ```
  </Tab>

  <Tab title="Streaming (with performance metrics)">
    ```python theme={null}
    stream = client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[{"role": "user", "content": "Hello, world!"}],
        stream=True,
        extra_body={"perf_metrics_in_response": True}
    )

    for chunk in stream:
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="")
        
        # Both usage and performance metrics are in the final chunk
        if chunk.choices[0].finish_reason:
            if chunk.usage:
                print(f"\n\nTokens: {chunk.usage.total_tokens}")
            if hasattr(chunk, 'perf_metrics'):
                print(f"Performance: {chunk.perf_metrics}")
    ```
  </Tab>
</Tabs>

<Note>
  Usage information is automatically included in the final chunk for streaming responses (the chunk with `finish_reason` set). This is a Fireworks extension - OpenAI SDK doesn't return usage for streaming by default.
</Note>

For all available metrics and details, see the [API reference documentation](/api-reference/post-chatcompletions).

<Tip>
  If you encounter errors during inference, see [Inference Error Codes](/guides/inference-error-codes) for common issues and resolutions.
</Tip>

## Advanced capabilities

Extend text models with additional features for structured outputs, tool integration, and performance optimization:

<CardGroup>
  <Card title="Tool calling" href="/guides/function-calling" icon="function">
    Connect models to external tools and APIs with type-safe parameters
  </Card>

  <Card title="Structured outputs" href="/structured-responses/structured-response-formatting" icon="brackets-curly">
    Enforce JSON schemas for reliable data extraction
  </Card>

  <Card title="Responses API" href="/guides/response-api" icon="brain">
    Multi-step reasoning for complex problem-solving
  </Card>

  <Card title="Predicted outputs" href="/guides/predicted-outputs" icon="bolt">
    Speed up edits by predicting unchanged sections
  </Card>

  <Card title="Prompt caching" href="/guides/prompt-caching" icon="database">
    Cache common prompts to reduce latency and cost
  </Card>

  <Card title="Batch inference" href="/guides/batch-inference" icon="list-check">
    Process large volumes of requests asynchronously
  </Card>
</CardGroup>

## Configuration & debugging

<AccordionGroup>
  <Accordion title="Sampling parameters">
    Control how the model generates text. Fireworks automatically uses recommended sampling parameters from each model's HuggingFace `generation_config.json` when you don't specify them explicitly, ensuring optimal performance out-of-the-box.

    We pull `temperature`, `top_k`, `top_p`, `min_p`, and `typical_p` from the model's configuration when not explicitly provided.

    ### Temperature

    Adjust randomness (0 = deterministic, higher = more creative):

    ```python theme={null}
    response = client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[{"role": "user", "content": "Write a poem"}],
        temperature=0.7  # Override model default
    )
    ```

    ### Max tokens

    Control the maximum number of tokens in the generated completion:

    ```python theme={null}
    max_tokens=100  # Generate at most 100 tokens
    ```

    **Important notes:**

    * Default value is 2048 tokens if not specified
    * Most models support up to their full context window (e.g., 128K for DeepSeek R1)
    * When the limit is reached, you'll see `"finish_reason": "length"` in the response

    <Tip>
      Set `max_tokens` appropriately for your use case to avoid truncated responses. Check the model's context window in the [Model Library](https://fireworks.ai/models).
    </Tip>

    ### Top-p (nucleus sampling)

    Consider only the most probable tokens summing to `top_p` probability mass:

    ```python theme={null}
    top_p=0.9  # Consider top 90% probability mass
    ```

    ### Top-k

    Consider only the k most probable tokens:

    ```python theme={null}
    top_k=50  # Consider top 50 tokens
    ```

    ### Min-p

    Exclude tokens below a probability threshold:

    ```python theme={null}
    min_p=0.05  # Exclude tokens with <5% probability
    ```

    ### Typical-p

    Use typical sampling to select tokens with probability close to the entropy of the distribution:

    ```python theme={null}
    typical_p=0.95  # Consider tokens with typical probability
    ```

    ### Repetition penalties

    Reduce repetitive text with `frequency_penalty`, `presence_penalty`, or `repetition_penalty`:

    ```python theme={null}
    frequency_penalty=0.5,   # Penalize frequent tokens (OpenAI compatible)
    presence_penalty=0.5,    # Penalize any repeated token (OpenAI compatible)
    repetition_penalty=1.1   # Exponential penalty from prompt + output
    ```

    ### Sampling options header

    The `fireworks-sampling-options` header contains the actual default sampling parameters used for the model, including values from the model's HuggingFace `generation_config.json`:

    <Tabs>
      <Tab title="Python">
        ```python theme={null}
        response = client.chat.completions.with_raw_response.create(
          model="accounts/fireworks/models/deepseek-v3p1",
          messages=[{"role": "user", "content": "Hello"}]
        )

        # Access headers from the raw response
        sampling_options = response.headers.get('fireworks-sampling-options')
        print(sampling_options)  # e.g., '{"temperature": 0.7, "top_p": 0.9}'

        completion = response.parse()  # get the parsed response object
        print(completion.choices[0].message.content)
        ```
      </Tab>

      <Tab title="JavaScript">
        ```javascript theme={null}
        import OpenAI from "openai";

        const client = new OpenAI({
          apiKey: process.env.FIREWORKS_API_KEY,
          baseURL: "https://api.fireworks.ai/inference/v1",
        });

        const response = await client.chat.completions.with_raw_response.create({
          model: "accounts/fireworks/models/deepseek-v3p1",
          messages: [{ role: "user", content: "Hello" }],
        });

        // Access headers from the raw response
        const samplingOptions = response.headers.get('fireworks-sampling-options');
        console.log(samplingOptions); // e.g., '{"temperature": 0.7, "top_p": 0.9}'

        const completion = response.parse(); // get the parsed response object
        console.log(completion.choices[0].message.content);
        ```
      </Tab>
    </Tabs>

    See the [API reference](/api-reference/post-chatcompletions) for detailed parameter descriptions.
  </Accordion>

  <Accordion title="Multiple generations">
    Generate multiple completions in one request:

    ```python theme={null}
    response = client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[{"role": "user", "content": "Tell me a joke"}],
        n=3  # Generate 3 different jokes
    )

    for choice in response.choices:
        print(choice.message.content)
    ```
  </Accordion>

  <Accordion title="Token probabilities (logprobs)">
    Inspect token probabilities for debugging or analysis:

    ```python theme={null}
    response = client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[{"role": "user", "content": "Hello"}],
        logprobs=True,
        top_logprobs=5  # Show top 5 alternatives per token
    )

    for content in response.choices[0].logprobs.content:
        print(f"Token: {content.token}, Logprob: {content.logprob}")
    ```
  </Accordion>

  <Accordion title="Prompt inspection (echo & raw_output)">
    Verify how your prompt was formatted:

    **Echo:** Return the prompt along with the generation:

    ```python theme={null}
    response = client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[{"role": "user", "content": "Hello"}],
        echo=True
    )
    ```

    **Token IDs:** Return prompt and completion token IDs:

    ```python theme={null}
    response = client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[{"role": "user", "content": "Hello"}],
        return_token_ids=True
    )

    print(response.prompt_token_ids)      # Prompt token IDs
    print(response.choices[0].token_ids)  # Completion token IDs
    ```

    **Raw output:** See prompt fragments and raw completion:

    <Warning>
      Experimental API - may change without notice.
    </Warning>

    ```python theme={null}
    response = client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[{"role": "user", "content": "Hello"}],
        raw_output=True
    )

    print(response.choices[0].raw_output.completion)  # Raw completion
    ```
  </Accordion>

  <Accordion title="Ignore EOS token">
    Force generation to continue past the end-of-sequence token (useful for benchmarking):

    ```python theme={null}
    response = client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[{"role": "user", "content": "Hello"}],
        ignore_eos=True,
        max_tokens=100  # Will always generate exactly 100 tokens
    )
    ```

    <Note>
      Output quality may degrade when ignoring EOS. This API is experimental and should not be relied upon for production use cases.
    </Note>
  </Accordion>

  <Accordion title="Logit bias">
    Modify token probabilities to encourage or discourage specific tokens:

    ```python theme={null}
    response = client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[{"role": "user", "content": "Hello"}],
        logit_bias={
            123: 10.0,   # Strongly encourage token ID 123
            456: -50.0   # Strongly discourage token ID 456
        }
    )
    ```
  </Accordion>

  <Accordion title="Mirostat sampling">
    Control perplexity dynamically using the [Mirostat algorithm](https://arxiv.org/abs/2007.14966):

    ```python theme={null}
    response = client.chat.completions.create(
        model="accounts/fireworks/models/deepseek-v3p1",
        messages=[{"role": "user", "content": "Hello"}],
        mirostat_target=5.0,  # Target perplexity
        mirostat_lr=0.1       # Learning rate for adjustments
    )
    ```
  </Accordion>
</AccordionGroup>

## Understanding tokens

Language models process text in chunks called **tokens**. In English, a token can be as short as one character or as long as one word. Different model families use different **tokenizers**, so the same text may translate to different token counts depending on the model.

**Why tokens matter:**

* Models have maximum context lengths measured in tokens
* Pricing is based on token usage (prompt + completion)
* Token count affects response time

For Llama models, use [this tokenizer tool](https://belladoreai.github.io/llama-tokenizer-js/example-demo/build/) to estimate token counts. Actual usage is returned in the `usage` field of every API response.

## OpenAI SDK migration

Fireworks provides an OpenAI-compatible API, making migration from OpenAI straightforward. For detailed information on setup, usage examples, and API compatibility notes, see the [OpenAI compatibility guide](/tools-sdks/openai-compatibility).

## Next steps

<CardGroup>
  <Card title="Vision models" href="/guides/querying-vision-language-models" icon="image">
    Process images alongside text
  </Card>

  <Card title="Embeddings" href="/guides/querying-embeddings-models" icon="database">
    Generate vector representations for search
  </Card>

  <Card title="On-demand deployments" href="/guides/ondemand-deployments" icon="server">
    Deploy models on dedicated GPUs
  </Card>

  <Card title="Fine-tuning" href="/fine-tuning/finetuning-intro" icon="sliders">
    Customize models for your use case
  </Card>

  <Card title="Error codes" href="/guides/inference-error-codes" icon="circle-exclamation">
    Troubleshoot common inference errors
  </Card>

  <Card title="API Reference" href="/api-reference/post-chatcompletions" icon="code">
    Complete API documentation
  </Card>
</CardGroup>


# Vision Models
Source: https://docs.fireworks.ai/guides/querying-vision-language-models

Query vision-language models to analyze images and visual content

Vision-language models (VLMs) process both text and images in a single request, enabling image captioning, visual question answering, document analysis, chart interpretation, OCR, and content moderation. Use VLMs via serverless inference or [dedicated deployments](/getting-started/ondemand-quickstart).

[Browse available vision models →](https://app.fireworks.ai/models?filter=Vision)

## Chat Completions API

Provide images via URL or base64 encoding. The request structure is identical to OpenAI's vision API.

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    from fireworks import Fireworks

    client = Fireworks()

    response = client.chat.completions.create(
        model="accounts/fireworks/models/kimi-k2p5",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Can you describe this image?"},
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": "https://images.unsplash.com/photo-1582538885592-e70a5d7ab3d3?w=800"
                        }
                    }
                ]
            }
        ]
    )

    print(response.choices[0].message.content)
    ```

    <Tip>
      You can also use the [OpenAI SDK](/tools-sdks/openai-compatibility) with Fireworks by changing the base URL and API key.
    </Tip>
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    import OpenAI from "openai";

    const client = new OpenAI({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference/v1",
    });

    const response = await client.chat.completions.create({
      model: "accounts/fireworks/models/kimi-k2p5",
      messages: [
        {
          role: "user",
          content: [
            { type: "text", text: "Can you describe this image?" },
            {
              type: "image_url",
              image_url: {
                url: "https://images.unsplash.com/photo-1582538885592-e70a5d7ab3d3?w=800"
              }
            }
          ]
        }
      ]
    });

    console.log(response.choices[0].message.content);
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    curl https://api.fireworks.ai/inference/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $FIREWORKS_API_KEY" \
      -d '{
        "model": "accounts/fireworks/models/kimi-k2p5",
        "messages": [
          {
            "role": "user",
            "content": [
              {"type": "text", "text": "Can you describe this image?"},
              {
                "type": "image_url",
                "image_url": {
                  "url": "https://images.unsplash.com/photo-1582538885592-e70a5d7ab3d3?w=800"
                }
              }
            ]
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

### Using base64-encoded images

For local files, encode them as base64 with the appropriate MIME type prefix:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    import base64
    from fireworks import Fireworks

    def encode_image(image_path):
        with open(image_path, "rb") as image_file:
            return base64.b64encode(image_file.read()).decode("utf-8")

    image_base64 = encode_image("your_image.jpg")

    client = Fireworks()

    response = client.chat.completions.create(
        model="accounts/fireworks/models/kimi-k2p5",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Can you describe this image?"},
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/jpeg;base64,{image_base64}"
                        }
                    }
                ]
            }
        ]
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={null}
    import OpenAI from "openai";
    import fs from "fs";

    const client = new OpenAI({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference/v1",
    });

    const imageBase64 = fs.readFileSync("your_image.jpg").toString("base64");

    const response = await client.chat.completions.create({
      model: "accounts/fireworks/models/kimi-k2p5",
      messages: [
        {
          role: "user",
          content: [
            { type: "text", text: "Can you describe this image?" },
            {
              type: "image_url",
              image_url: {
                url: `data:image/jpeg;base64,${imageBase64}`
              }
            }
          ]
        }
      ]
    });

    console.log(response.choices[0].message.content);
    ```
  </Tab>
</Tabs>

## Working with images

Vision-language models support [prompt caching](/guides/prompt-caching) to improve performance for requests with repeated content. Both text and image portions can benefit from caching to reduce time to first token by up to 80%.

**Tips for optimal performance:**

* **Use URLs for long conversations** – Reduces latency compared to base64 encoding
* **Downsize images** – Smaller images use fewer tokens and process faster
* **Structure prompts for caching** – Place static instructions at the beginning, variable content at the end
* **Include metadata in prompts** – Add context about the image directly in your text prompt

## Working with PDFs

VLMs do not natively accept PDF files as input. To analyze PDF documents, convert each page to an image and pass the images to the model using base64 encoding.

<Note>
  Remember the [30-image limit per request](#known-limitations). For long documents, process pages in batches or select only the relevant pages.
</Note>

<Tabs>
  <Tab title="Python">
    Install [PyMuPDF](https://pymupdf.readthedocs.io/):

    ```bash theme={null}
    pip install pymupdf fireworks-ai
    ```

    ```python theme={null}
    import base64
    import fitz
    from fireworks.client import Fireworks


    def pdf_pages_to_base64(pdf_path, dpi=200):
        doc = fitz.open(pdf_path)
        images = []
        for page in doc:
            pix = page.get_pixmap(dpi=dpi)
            images.append(base64.b64encode(pix.tobytes("png")).decode("utf-8"))
        doc.close()
        return images


    page_images = pdf_pages_to_base64("document.pdf")

    client = Fireworks()

    content = [{"type": "text", "text": "Summarize this document."}]
    for img in page_images:
        content.append({
            "type": "image_url",
            "image_url": {"url": f"data:image/png;base64,{img}"}
        })

    response = client.chat.completions.create(
        model="accounts/fireworks/models/kimi-k2p5",
        messages=[{"role": "user", "content": content}]
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="JavaScript">
    Install [pdf-to-img](https://www.npmjs.com/package/pdf-to-img) and [openai](https://www.npmjs.com/package/openai):

    ```bash theme={null}
    npm install pdf-to-img openai
    ```

    ```javascript theme={null}
    import { pdf } from "pdf-to-img";
    import OpenAI from "openai";

    const client = new OpenAI({
      apiKey: process.env.FIREWORKS_API_KEY,
      baseURL: "https://api.fireworks.ai/inference/v1",
    });

    const pages = [];
    for await (const page of await pdf("document.pdf", { scale: 2.0 })) {
      pages.push(Buffer.from(page).toString("base64"));
    }

    const content = [
      { type: "text", text: "Summarize this document." },
      ...pages.map((base64) => ({
        type: "image_url",
        image_url: { url: `data:image/png;base64,${base64}` },
      })),
    ];

    const response = await client.chat.completions.create({
      model: "accounts/fireworks/models/kimi-k2p5",
      messages: [{ role: "user", content }],
    });

    console.log(response.choices[0].message.content);
    ```
  </Tab>
</Tabs>

## Advanced capabilities

<CardGroup>
  <Card title="Vision fine-tuning" href="/fine-tuning/fine-tuning-models#vision-fine-tuning" icon="sliders">
    Fine-tune VLMs for specialized visual tasks
  </Card>

  <Card title="LoRA adapters" href="/models/uploading-custom-models" icon="layer-group">
    Deploy custom LoRA adapters for vision models
  </Card>

  <Card title="Dedicated deployments" href="/getting-started/ondemand-quickstart" icon="server">
    Deploy VLMs on dedicated GPUs for better performance
  </Card>

  <Card title="Video & audio inputs" href="/guides/video-audio-inputs" icon="video">
    Process video and audio content with multimodal models
  </Card>
</CardGroup>

## Alternative query methods

For the Completions API, manually insert the image token `<image>` in your prompt and supply images as an ordered list:

```python theme={null}
response = client.completions.create(
    model="accounts/fireworks/models/kimi-k2p5",
    prompt="SYSTEM: Hello\n\nUSER:<image>\ntell me about the image\n\nASSISTANT:",
    extra_body={
        "images": ["https://images.unsplash.com/photo-1582538885592-e70a5d7ab3d3?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=1770&q=80"]
    }
)

print(response.choices[0].text)
```

## Known limitations

1. **Maximum images per request**: 30 images maximum, regardless of format (base64 or URL)
2. **Base64 size limit**: Total base64-encoded images must be less than 10MB
3. **URL size and timeout**: Each image URL must be smaller than 5MB and download within 1.5 seconds
4. **Supported formats**: `.png`, `.jpg`, `.jpeg`, `.gif`, `.bmp`, `.tiff`, `.ppm`
5. **Llama 3.2 Vision models**: Pass images before text in the content field to avoid refusals (temporary limitation)


# Account quotas
Source: https://docs.fireworks.ai/guides/quotas_usage/account-quotas

Account-wide request limits, spending tiers, spend limits, and on-demand GPU quotas

<a />

Fireworks uses different controls for serverless and on-demand deployments. This page is the canonical reference for spending tiers, monthly spend limits, on-demand GPU quotas, and account-wide request limits.

For serverless TPM and adaptive limits, see [Serverless rate limits](/serverless/rate-limits).

## Check your current limits

View your account's current quotas and limits:

```bash theme={null}
firectl quota list
```

This shows your rate limits, GPU quotas, spend limits, and usage across serverless and on-demand deployments.

## Spending tiers

Your spend tier controls available quotas and capacity. For legacy self-serve postpaid accounts, it also determines the maximum monthly spend limit. Prepaid accounts can set a monthly spend limit independently of tier.

| Tier      | Criteria                                                                 | Legacy Postpaid Max Monthly Spend Limit |
| --------- | ------------------------------------------------------------------------ | --------------------------------------- |
| Tier 1    | [Valid payment method and billing profile](https://fireworks.ai/billing) | \$50                                    |
| Tier 2    | Spend or add \$50 in credits                                             | \$500                                   |
| Tier 3    | Spend or add \$500 in credits                                            | \$5,000                                 |
| Tier 4    | Spend or add \$5,000 in credits                                          | \$50,000                                |
| Unlimited | [Contact us](https://fireworks.ai/company/contact-us)                    | Unlimited                               |

<Tip>
  Add prepaid credits to unlock a higher tier. For example, adding \$100 moves you from Tier 1 to Tier 2. Your new tier activates within minutes.
</Tip>

<Note>
  These spending tiers control the maximum [serverless TPM upper bounds](/serverless/rate-limits) your account can reach. The spend-limit maximum shown above applies only to legacy self-serve postpaid accounts.

  Fireworks operates on a pre-paid credits billing system. Contracted customers may have the option to move to post-paid billing — [contact our sales team](https://fireworks.ai/company/contact-us) to discuss your options.
</Note>

### Training GPU quota

Fine-tuning jobs use training GPU quota (separate from on-demand deployment quota), granted automatically by spending tier:

| Tier              | How to reach it                                       | B200 / B300 (Blackwell) |  H200  | H100 / A100 |
| ----------------- | ----------------------------------------------------- | :---------------------: | :----: | :---------: |
| No payment method | —                                                     |            0            |    0   |      0      |
| Tier 1            | Valid payment method and billing profile              |            0            |   16   |      8      |
| Tier 2            | Spend or add \$50 in credits                          |            16           |   16   |      16     |
| Tier 3            | Spend or add \$500 in credits                         |            24           |   24   |      24     |
| Tier 4            | Spend or add \$5,000 in credits                       |            32           |   32   |      32     |
| Enterprise        | [Contact us](https://fireworks.ai/company/contact-us) |          Custom         | Custom |    Custom   |

Counts are GPUs of that type available to training jobs. Blackwell (B200/B300) is `0` until Tier 2, and current managed fine-tuning shapes run on Blackwell, so most fine-tuning needs Tier 2. If a job is rejected with HTTP 429 `quota_exceeded`, raise your tier and resubmit.

<Note>
  Need more training quota than your tier allows? [Reach out for enterprise support](https://fireworks.ai/contact-training) and we'll help size the right allocation for your workload.
</Note>

### Enterprise accounts

Enterprise accounts can configure monthly spend alerts, but those alerts do not pause service. They track the cost of Fireworks usage, including usage paid for with credits. Adding credits itself does not count as spend.

For Enterprise quota details, see [Enterprise quotas](/faq/enterprise/service/quotas) or contact your account representative.

## Manage your quotas

### Account-wide request limits

All API usage on your account shares a single request-throughput envelope:

| Account state                     |  Request-rate limit |
| --------------------------------- | ------------------: |
| No payment method or no credits   |              10 RPM |
| Payment method and active credits | 6,000 RPM (maximum) |

The **6,000 RPM** cap applies account-wide—it is **not** a separate serverless-only limit—and it is a **fixed** ceiling, not adaptive. Per-minute request volume above this cap is rejected (for example HTTP 429), regardless of your spending tier.

### Monthly spend limit

Set a monthly spend limit that fits your needs and adjust it anytime.

For prepaid accounts, the limit tracks the dollar value of your usage regardless of whether credits pay for that usage. Adding credits increases your available balance, but does not raise or reset your monthly spend limit.

Auto Reload is separate from the spend limit. Auto Reload purchases credits when your balance is low; it does not change how much usage your account can accrue during the month.

### View and adjust your spend limit

Check your current spend limit:

```bash theme={null}
firectl quota list
```

Set a custom monthly spend limit:

```bash theme={null}
firectl quota update monthly-spend-usd --value <AMOUNT>
```

For example, to set a \$200 monthly spend limit:

```bash theme={null}
firectl quota update monthly-spend-usd --value 200
```

### Warnings and suspension

By default, Fireworks sends a warning email when usage reaches 80% of your monthly spend limit. You can add other notification amounts from the Billing page.

When usage reaches 100% of the limit, all API requests pause automatically across serverless inference, deployments, and fine-tuning. Raise the monthly spend limit to resume usage. If your credit balance is also depleted, you must add credits as well.

<Note>
  The suspension behavior described here does not apply to Enterprise accounts. Their monthly spend alerts are informational.
</Note>

### On-demand deployment quotas

On-demand deployments have GPU quotas instead of rate limits:

| GPU Type          | Default Quota |
| ----------------- | ------------- |
| Nvidia A100       | 8 GPUs        |
| Nvidia H100       | 8 GPUs        |
| Nvidia H200       | 8 GPUs        |
| Nvidia B200       | 8 GPUs        |
| LoRAs (on-demand) | 100           |

<Tip>
  Need more GPUs? [Contact us](https://fireworks.ai/company/contact-us) to request a quota increase.
</Tip>

<Callout type="info">
  On-demand and dedicated deployments are **not limited by adaptive serverless TPM upper bounds**. If you receive HTTP **429** on those endpoints, it typically means **deployment saturation** (GPUs busy) rather than hitting a TPM tier cap. Requests still count toward [account-wide request limits](#account-wide-request-limits). See [understanding 429 errors](/guides/inference-error-codes#understanding-429-errors) for details and resolution steps.
</Callout>

### Account recovery

If your account is suspended due to payment issues:

1. Go to [Billing](https://fireworks.ai/billing)
2. Resolve failed payment methods and add credits (or pay outstanding invoices for postpaid accounts)
3. Your account reactivates automatically within an hour

<Tip>
  Still suspended after resolving payment issues? Contact support via [Discord](https://discord.gg/fireworks-ai) or email [inquiries@fireworks.ai](mailto:inquiries@fireworks.ai).
</Tip>


# Reasoning
Source: https://docs.fireworks.ai/guides/reasoning

How to use reasoning with Fireworks models

For thinking/reasoning models, Fireworks provides access to the model's
reasoning process through the `reasoning_content` field. This field contains the
model's internal reasoning, which would otherwise appear in `<think></think>`
tags within the content field. For some models, the reasoning content may
instead be included directly in the content field itself.

### Prerequisites

We recommend using the [Fireworks Python SDK](/tools-sdks/python-sdk) to work with
reasoning, as it supports Fireworks-specific parameters and response fields.

<Note>
  The SDK is currently in alpha. Use the `--pre` flag when installing to get the latest version.
</Note>

<CodeGroup>
  ```bash pip theme={null}
  pip install --pre fireworks-ai
  ```

  ```bash poetry theme={null}
  poetry add --pre fireworks-ai
  ```

  ```bash uv theme={null}
  uv add --pre fireworks-ai
  ```
</CodeGroup>

### Basic usage

Select a reasoning model from our [serverless model library](https://app.fireworks.ai/models/?filter=Serverless).

```python lines highlight={16-18} theme={null}
from fireworks import Fireworks

client = Fireworks()

completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What is 25 * 37?",
        }
    ],
    model="accounts/fireworks/models/<reasoning-model>",
)

for choice in completion.choices:
    # Access the reasoning content (thinking process)
    if choice.message.reasoning_content:
        print("Reasoning:", choice.message.reasoning_content)
    print("Answer:", choice.message.content)
```

### Controlling reasoning effort

You can control the reasoning token length using either the `reasoning_effort` parameter or the Anthropic-compatible `thinking` parameter.

Coming from NVIDIA NIM or vLLM? Fields like `chat_template_kwargs.enable_thinking` are mapped onto `reasoning_effort` automatically. See [NIM compatibility](/tools-sdks/nim-compatibility).

#### Using `reasoning_effort`

The [`reasoning_effort`](/api-reference/post-chatcompletions) parameter accepts string values like `"low"`, `"medium"`, or `"high"`:

```python lines highlight={9} theme={null}
completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "Solve this step by step: If a train travels at 60 mph for 2.5 hours, how far does it go?",
        }
    ],
    model="accounts/fireworks/models/<reasoning-model>",
    reasoning_effort="medium",
)
```

See the [reasoning\_effort](/api-reference/post-chatcompletions) parameter for more details.

#### Using `thinking` (Anthropic-compatible)

Alternatively, you can use the [`thinking`](/api-reference/post-chatcompletions) parameter, which provides an Anthropic-compatible format for controlling reasoning behavior:

```python lines highlight={9-12} theme={null}
completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "Solve this step by step: If a train travels at 60 mph for 2.5 hours, how far does it go?",
        }
    ],
    model="accounts/fireworks/models/<reasoning-model>",
    thinking={
        "type": "enabled",
        "budget_tokens": 4096,  # Must be >= 1024
    },
)
```

See the [thinking](/api-reference/post-chatcompletions) parameter for more details.

<Warning>
  You cannot specify both `thinking` and `reasoning_effort` in the same request. If both are provided, a validation error will be raised.
</Warning>

### Streaming with reasoning content

When streaming, the reasoning content is available in each chunk's delta:

```python lines highlight={21-23} theme={null}
from fireworks import Fireworks

client = Fireworks()

stream = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What is the square root of 144?",
        }
    ],
    model="accounts/fireworks/models/<reasoning-model>",
    reasoning_effort="medium",
    stream=True,
)

reasoning_parts = []
content_parts = []

for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.reasoning_content:
        reasoning_parts.append(delta.reasoning_content)
    if delta.content:
        content_parts.append(delta.content)

print("Reasoning:", "".join(reasoning_parts))
print("Answer:", "".join(content_parts))
```

### Interleaved thinking

When building multi-turn tool-calling agents with models that support
interleaved thinking, you **MUST** include the `reasoning_content` from previous
assistant turns in subsequent requests. This enables the model to think between tool calls and after receiving tool results, allowing for more complex, step-by-step reasoning.

<Frame>
  <img alt="Diagram showing interleaved thinking where reasoning is preserved within Turn 1 across multiple steps but starts fresh in Turn 2" />
</Frame>

You can preserve reasoning context in two ways:

1. **Pass the `Message` object directly** (recommended) - The SDK message object
   already contains the `reasoning_content` field alongside `content` and `tool_calls`
2. **Manually include `reasoning_content`** - When constructing messages as
   dictionaries, explicitly add the `reasoning_content` field

<Note>
  Interleaved thinking is triggered when the last message in your API request
  has `"role": "tool"`, enabling the model to use its previous reasoning process
  when responding to the tool result. If a model does not support interleaved
  thinking, it simply ignores the extra reasoning context so this pattern is
  safe to use broadly.
</Note>

<Note>
  **Fine-tuned models:** whether historical `reasoning_content` was visible during
  training depends on the base model and the job's thinking-history mode.
  Interleaved removes thinking across user-turn boundaries while retaining the
  assistant/tool trajectory within the current turn; Preserved retains thinking
  across user turns. Datum unrolling is model-specific. For reliable results,
  serve a fine-tuned model consistently with how it was trained; see
  [thinking history in fine-tuning](/fine-tuning/thinking-history)
  for per-model behavior.
</Note>

Here's how to preserve reasoning context using both approaches:

<Tabs>
  <Tab title="Pass Message object">
    ```python theme={null}
    # First turn: Get a response with reasoning_content
    first_response = client.chat.completions.create(
        messages=[{"role": "user", "content": "What is 15 + 27?"}],
        model="accounts/fireworks/models/<reasoning-model>",
        tools=tools,
    )

    # The assistant message contains reasoning_content, content, and tool_calls
    assistant_message = first_response.choices[0].message
    # assistant_message.reasoning_content -> "The user is asking for addition..."
    # assistant_message.tool_calls -> [ToolCall(id="...", function=...)]

    # Second turn: Pass the Message object directly
    # This automatically includes reasoning_content alongside the message
    second_response = client.chat.completions.create(
        messages=[
            {"role": "user", "content": "What is 15 + 27?"},
            assistant_message,  # Pass the complete Message object
            {"role": "tool", "content": "42", "tool_call_id": assistant_message.tool_calls[0].id},
        ],
        model="accounts/fireworks/models/<reasoning-model>",
        tools=tools,
    )
    ```
  </Tab>

  <Tab title="Manual dictionary">
    ```python theme={null}
    # First turn: Get a response with reasoning_content
    first_response = client.chat.completions.create(
        messages=[{"role": "user", "content": "What is 15 + 27?"}],
        model="accounts/fireworks/models/<reasoning-model>",
        tools=tools,
    )

    assistant_message = first_response.choices[0].message

    # Second turn: Manually construct the assistant message dict
    # Include reasoning_content explicitly alongside role, content, and tool_calls
    second_response = client.chat.completions.create(
        messages=[
            {"role": "user", "content": "What is 15 + 27?"},
            {
                "role": "assistant",
                "content": assistant_message.content,
                "reasoning_content": assistant_message.reasoning_content,  # Include reasoning
                "tool_calls": assistant_message.tool_calls,
            },
            {"role": "tool", "content": "42", "tool_call_id": assistant_message.tool_calls[0].id},
        ],
        model="accounts/fireworks/models/<reasoning-model>",
        tools=tools,
    )
    ```
  </Tab>

  <Tab title="Anthropic SDK">
    The Anthropic SDK uses `thinking` content blocks instead of `reasoning_content`. Pass the full `content` array (including thinking blocks) back as the assistant message, and send tool results as `tool_result` content blocks:

    ```python theme={null}
    import anthropic

    client = anthropic.Anthropic(
        api_key=os.environ.get("FIREWORKS_API_KEY"),
        base_url="https://api.fireworks.ai/inference",
    )

    tools = [
        {
            "name": "calculator",
            "description": "Perform basic arithmetic operations",
            "input_schema": {
                "type": "object",
                "properties": {
                    "operation": {"type": "string", "enum": ["add", "subtract", "multiply", "divide"]},
                    "a": {"type": "number"},
                    "b": {"type": "number"},
                },
                "required": ["operation", "a", "b"],
            },
        }
    ]

    # First turn: thinking + tool_use
    first_response = client.messages.create(
        model="accounts/fireworks/models/<reasoning-model>",
        max_tokens=16000,
        thinking={"type": "enabled", "budget_tokens": 4096},
        messages=[{"role": "user", "content": "What is 15 + 27? Use the calculator."}],
        tools=tools,
    )

    # Response content includes [thinking_block, tool_use_block]
    for block in first_response.content:
        if block.type == "thinking":
            print(f"Thinking: {block.thinking[:100]}...")
        elif block.type == "tool_use":
            print(f"Tool: {block.name}({block.input})")

    tool_use = next(b for b in first_response.content if b.type == "tool_use")

    # Second turn: pass back the full content array (with thinking blocks) + tool result
    second_response = client.messages.create(
        model="accounts/fireworks/models/<reasoning-model>",
        max_tokens=16000,
        thinking={"type": "enabled", "budget_tokens": 4096},
        messages=[
            {"role": "user", "content": "What is 15 + 27? Use the calculator."},
            {"role": "assistant", "content": first_response.content},
            {
                "role": "user",
                "content": [
                    {"type": "tool_result", "tool_use_id": tool_use.id, "content": "42"}
                ],
            },
        ],
        tools=tools,
    )

    # The model thinks again (interleaved) and produces a text answer
    for block in second_response.content:
        if block.type == "thinking":
            print(f"Thinking: {block.thinking[:100]}...")
        elif block.type == "text":
            print(f"Answer: {block.text}")
    ```
  </Tab>
</Tabs>

<Warning>
  If you construct the assistant message manually as a dictionary but omit the
  `reasoning_content` field, the model will not have access to its previous
  reasoning process.
</Warning>

The following script demonstrates this behavior and validates that the
`reasoning_content` from the first turn is included in subsequent requests:

```python main.py expandable theme={null}
"""Test that reasoning_content is passed in multi-turn conversations.

This test proves that reasoning_content from previous turns is included
in subsequent requests by examining the raw prompt sent to the model.
"""

from fireworks import Fireworks
from dotenv import load_dotenv

load_dotenv()

client = Fireworks()

MODEL = "accounts/fireworks/models/kimi-k2-thinking"
# MODEL = "accounts/fireworks/models/minimax-m2"

# Define tools to enable interleaved thinking
tools = [
    {
        "type": "function",
        "function": {
            "name": "calculator",
            "description": "Perform basic arithmetic operations",
            "parameters": {
                "type": "object",
                "properties": {
                    "operation": {
                        "type": "string",
                        "enum": ["add", "subtract", "multiply", "divide"],
                    },
                    "a": {"type": "number"},
                    "b": {"type": "number"},
                },
                "required": ["operation", "a", "b"],
            },
        },
    }
]


def print_header(title: str, char: str = "═", width: int = 60):
    """Print a formatted section header."""
    print(f"\n{char * width}")
    print(f"  {title}")
    print(f"{char * width}")


def print_field(label: str, value: str, indent: int = 2):
    """Print a labeled field with optional indentation."""
    prefix = " " * indent
    print(f"{prefix}{label}: {value}")


def print_multiline(label: str, content: str, max_preview: int = 200, indent: int = 2):
    """Print multiline content with a label and optional truncation."""
    prefix = " " * indent
    print(f"{prefix}{label}:")
    preview = content[:max_preview] + "..." if len(content) > max_preview else content
    for line in preview.split("\n"):
        print(f"{prefix}  │ {line}")


# First turn - get a response with reasoning_content
print_header("FIRST TURN", "═")
first_response = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What is 15 + 27?",
        }
    ],
    model=MODEL,
    tools=tools,
)

print_field("📝 Content", first_response.choices[0].message.content or "(none)")

reasoning = first_response.choices[0].message.reasoning_content
print_multiline("💭 Reasoning", reasoning)

# Print tool call (verified) from the first response
tool_calls = first_response.choices[0].message.tool_calls
assert tool_calls, "No tool calls in first response!"
print(f"\n  🔧 Tool Calls ({len(tool_calls)}):")
for i, tc in enumerate(tool_calls, 1):
    print(f"    [{i}] id={tc.id}")
    print(f"        function={tc.function.name}")
    print(f"        arguments={tc.function.arguments}")
tool_call_id = first_response.choices[0].message.tool_calls[0].id

# Verify we got reasoning_content
assert reasoning and len(reasoning) > 0, "No reasoning_content in first response!"
print("\n  ✓ First response has reasoning_content")

# Second turn - include the first assistant message
print_header("SECOND TURN", "═")
second_response = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What is 15 + 27?",
        },
        first_response.choices[0].message,  # Includes reasoning_content
        {"role": "tool", "content": "42", "tool_call_id": tool_call_id},
    ],
    model=MODEL,
    tools=tools,
    raw_output=True,
)

print_field("📝 Answer", second_response.choices[0].message.content or "(none)")

# Extract and display the raw prompt that was sent to the model
raw_prompt = second_response.choices[0].raw_output.prompt_fragments[0]
print_header("RAW PROMPT SENT TO MODEL", "─")
print(raw_prompt)

# Check if reasoning_content from first turn is in the raw prompt
has_reasoning_content = reasoning[:50] in raw_prompt

print_header("RESULT", "═")
if has_reasoning_content:
    print("  ✅ SUCCESS: reasoning_content IS included in subsequent requests!")
else:
    print("  ❌ FAILED: reasoning_content not found in raw prompt")
print()
```

Below is the expected output:

<CodeGroup>
  ```txt Kimi-K2-Thinking (output) expandable theme={null}
  ════════════════════════════════════════════════════════════
    FIRST TURN
  ════════════════════════════════════════════════════════════
    📝 Content: (none)
    💭 Reasoning:
      │ The user is asking for a simple addition calculation: 15 + 27.
      │ 
      │ I should use the calculator function with:
      │ - operation: "add"
      │ - a: 15
      │ - b: 27

    🔧 Tool Calls (1):
      [1] id=functions.calculator:0
          function=calculator
          arguments={"operation": "add", "a": 15, "b": 27}

    ✓ First response has reasoning_content

  ════════════════════════════════════════════════════════════
    SECOND TURN
  ════════════════════════════════════════════════════════════
    📝 Answer: 15 + 27 = 42

  ────────────────────────────────────────────────────────────
    RAW PROMPT SENT TO MODEL
  ────────────────────────────────────────────────────────────
  
tool_declare
[{"function":{"description":"Perform basic arithmetic operations","name":"calculator","parameters":{"properties":{"a":{"type":"number"},"b":{"type":"number"},"operation":{"enum":["add","subtract","multiply","divide"],"type":"string"}},"required":["operation","a","b"],"type":"object"}},"type":"function"}]

user
What is 15 + 27?

assistant
<think>The user is asking for a simple addition calculation: 15 + 27.

  I should use the calculator function with:
  - operation: "add"
  - a: 15
  - b: 27</think>

functions.calculator:0
{"operation": "add", "a": 15, "b": 27}



tool
## Return of None
  42

assistant


  ════════════════════════════════════════════════════════════
    RESULT
  ════════════════════════════════════════════════════════════
    ✅ SUCCESS: reasoning_content IS included in subsequent requests!
  ```

  ```txt Minimax-M2 (output) expandable theme={null}
  ════════════════════════════════════════════════════════════
    FIRST TURN
  ════════════════════════════════════════════════════════════
    📝 Content: (none)
    💭 Reasoning:
      │ 
      │ Okay, the user is asking a simple arithmetic question: "What is 15 + 27?". This is a basic addition operation. I have a calculator tool available that can perform arithmetic operations. This is the p...

    🔧 Tool Calls (1):
      [1] id=chatcmpl-tool-249757d9ac8f4ca9afbdb580ced40ae6
          function=calculator
          arguments={"operation": "add", "a": 15, "b": 27}

    ✓ First response has reasoning_content

  ════════════════════════════════════════════════════════════
    SECOND TURN
  ════════════════════════════════════════════════════════════
    📝 Answer: 42

  ────────────────────────────────────────────────────────────
    RAW PROMPT SENT TO MODEL
  ────────────────────────────────────────────────────────────
  ]~b]system
  You are a helpful assistant.

  # Tools
  You may call one or more tools to assist with the user query.
  Here are the tools available in JSONSchema format:

  <tools>
  <tool>{"name": "calculator", "description": "Perform basic arithmetic operations", "parameters": {"type": "object", "properties": {"operation": {"type": "string", "enum": ["add", "subtract", "multiply", "divide"]}, "a": {"type": "number"}, "b": {"type": "number"}}, "required": ["operation", "a", "b"]}}</tool>
  </tools>

  When making tool calls, use XML format to invoke tools and pass parameters:

  <minimax:tool_call>
  <invoke name="tool-name-1">
  <parameter name="param-key-1">param-value-1</parameter>
  <parameter name="param-key-2">param-value-2</parameter>
  ...
  </invoke>
  </minimax:tool_call>[e~[
  ]~b]user
  What is 15 + 27?[e~[
  ]~b]ai
  <think>

  Okay, the user is asking a simple arithmetic question: "What is 15 + 27?". This is a basic addition operation. I have a calculator tool available that can perform arithmetic operations. This is the perfect tool for this task.

  Looking at the calculator tool parameters, I need to provide:
  1. The operation: In this case, it would be "add" since we're adding two numbers
  2. The first number: 15
  3. The second number: 27

  The calculator tool will handle the actual computation. Addition is one of the most basic arithmetic operations, so I'm confident the calculator tool can handle this correctly. I don't need to do the calculation manually since we have a dedicated tool for this.

  The user might be expecting me to simply state the answer, but using the calculator tool will ensure accuracy and follows the guidelines of using available tools when appropriate. The tool will take the two numbers (15 and 27) and add them together.

  So I'll make a tool call to the calculator with the operation "add", a=15, and b=27. The calculator will perform the addition and return the result, which I can then provide to the user as the final answer.

  This is a straightforward request that aligns perfectly with the calculator tool's functionality. No additional clarification is needed from the user, and the operation doesn't require any special handling.

  </think>


  <minimax:tool_call>
  <invoke name="calculator">
  <parameter name="operation">add</parameter>
  <parameter name="a">15</parameter>
  <parameter name="b">27</parameter>
  </invoke>
  </minimax:tool_call>[e~[
  ]~b]tool
  <response>42</response>[e~[
  ]~b]ai


  ════════════════════════════════════════════════════════════
    RESULT
  ════════════════════════════════════════════════════════════
    ✅ SUCCESS: reasoning_content IS included in subsequent requests!
  ```
</CodeGroup>

### Preserved thinking

While interleaved thinking preserves reasoning within a single turn (across tool calls), **preserved thinking** extends this concept across multiple user turns. This allows the model to retain reasoning content from previous assistant turns in the conversation context, enabling more coherent multi-turn reasoning.

<Frame>
  <img alt="Diagram showing preserved thinking where reasoning from Turn 1 is included in the input context for Turn 2" />
</Frame>

#### Controlling reasoning history

You can control how historical reasoning content is included in subsequent requests using the [`reasoning_history`](/api-reference/post-chatcompletions) parameter:

```python lines highlight={8} theme={null}
completion = client.chat.completions.create(
    messages=[
        {"role": "user", "content": "What is 15 + 27?"},
        assistant_message,  # Contains reasoning_content from previous turn
        {"role": "user", "content": "Now multiply that by 2"},
    ],
    model="accounts/fireworks/models/<reasoning-model>",
    reasoning_history="preserved",  # Retain all previous reasoning content
)
```

See the [`reasoning_history`](/api-reference/post-chatcompletions) parameter for all accepted values.

<Note>
  When `reasoning_history` is set to `"preserved"`, the model receives the full reasoning context from all previous turns. This is particularly useful for complex multi-turn conversations where maintaining reasoning continuity is important.
</Note>

The following script demonstrates preserved thinking across multiple turns and validates that
`reasoning_content` from all previous turns is included in subsequent requests:

```python main.py expandable theme={null}
"""Test that reasoning_content is passed in multi-turn conversations.

This test proves that reasoning_content from previous turns is included
in subsequent requests by examining the raw prompt sent to the model.
"""

from fireworks import Fireworks
from dotenv import load_dotenv

load_dotenv()

client = Fireworks()

MODEL = "accounts/fireworks/models/glm-4p7"


# Define tools to enable interleaved thinking
tools = [
    {
        "type": "function",
        "function": {
            "name": "calculator",
            "description": "Perform basic arithmetic operations",
            "parameters": {
                "type": "object",
                "properties": {
                    "operation": {
                        "type": "string",
                        "enum": ["add", "subtract", "multiply", "divide"],
                    },
                    "a": {"type": "number"},
                    "b": {"type": "number"},
                },
                "required": ["operation", "a", "b"],
            },
        },
    }
]


def print_header(title: str, char: str = "═", width: int = 60):
    """Print a formatted section header."""
    print(f"\n{char * width}")
    print(f"  {title}")
    print(f"{char * width}")


def print_field(label: str, value: str, indent: int = 2):
    """Print a labeled field with optional indentation."""
    prefix = " " * indent
    print(f"{prefix}{label}: {value}")


def print_multiline(label: str, content: str, max_preview: int = 200, indent: int = 2):
    """Print multiline content with a label and optional truncation."""
    prefix = " " * indent
    print(f"{prefix}{label}:")
    preview = content[:max_preview] + "..." if len(content) > max_preview else content
    for line in preview.split("\n"):
        print(f"{prefix}  │ {line}")


# First turn - get a response with reasoning_content
print_header("FIRST TURN", "═")
first_response = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What is 15 + 27?",
        }
    ],
    model=MODEL,
    tools=tools,
)

print_field("📝 Content", first_response.choices[0].message.content or "(none)")

reasoning = first_response.choices[0].message.reasoning_content
print_multiline("💭 Reasoning", reasoning)

# Print tool call (verified) from the first response
tool_calls = first_response.choices[0].message.tool_calls
assert tool_calls, "No tool calls in first response!"
print(f"\n  🔧 Tool Calls ({len(tool_calls)}):")
for i, tc in enumerate(tool_calls, 1):
    print(f"    [{i}] id={tc.id}")
    print(f"        function={tc.function.name}")
    print(f"        arguments={tc.function.arguments}")
tool_call_id = first_response.choices[0].message.tool_calls[0].id

# Verify we got reasoning_content
assert reasoning and len(reasoning) > 0, "No reasoning_content in first response!"
print("\n  ✓ First response has reasoning_content")

# Second turn - include the first assistant message
print_header("SECOND TURN", "═")
second_response = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What is 15 + 27?",
        },
        first_response.choices[0].message,  # Includes reasoning_content
        {"role": "tool", "content": "42", "tool_call_id": tool_call_id},
    ],
    model=MODEL,
    tools=tools,
    raw_output=True,
)

print_field("📝 Answer", second_response.choices[0].message.content or "(none)")

# Extract and display the raw prompt that was sent to the model
raw_prompt = second_response.choices[0].raw_output.prompt_fragments[0]
print_header("RAW PROMPT SENT TO MODEL", "─")
print(raw_prompt)

# Check if reasoning_content from first turn is in the raw prompt
has_reasoning_content = reasoning[:50] in raw_prompt

print_header("RESULT", "═")
if has_reasoning_content:
    print("  ✅ SUCCESS: reasoning_content IS included in subsequent requests!")
else:
    print("  ❌ FAILED: reasoning_content not found in raw prompt")
print()

# Third turn - ask for another calculation
print_header("THIRD TURN", "═")
third_response = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What is 15 + 27?",
        },
        first_response.choices[0].message,  # Includes reasoning_content
        {"role": "tool", "content": "42", "tool_call_id": tool_call_id},
        {
            "role": "user",
            "content": "What is 20 * 5?",
        },
    ],
    model=MODEL,
    tools=tools,
)

print_field("📝 Answer", third_response.choices[0].message.content or "(none)")

reasoning_third = third_response.choices[0].message.reasoning_content
print_multiline("💭 Reasoning", reasoning_third)

# Print tool call from the third response
tool_calls_third = third_response.choices[0].message.tool_calls
assert tool_calls_third, "No tool calls in third response!"
print(f"\n  🔧 Tool Calls ({len(tool_calls_third)}):")
for i, tc in enumerate(tool_calls_third, 1):
    print(f"    [{i}] id={tc.id}")
    print(f"        function={tc.function.name}")
    print(f"        arguments={tc.function.arguments}")
tool_call_id_third = third_response.choices[0].message.tool_calls[0].id
print()

# Fourth turn - include the third assistant message and tool response
print_header("FOURTH TURN", "═")
fourth_response = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What is 15 + 27?",
        },
        first_response.choices[0].message,  # Includes reasoning_content
        {"role": "tool", "content": "42", "tool_call_id": tool_call_id},
        {
            "role": "user",
            "content": "What is 20 * 5?",
        },
        third_response.choices[0].message,  # Includes reasoning_content from third turn
        {"role": "tool", "content": "100", "tool_call_id": tool_call_id_third},
    ],
    model=MODEL,
    tools=tools,
    raw_output=True,
    reasoning_history="preserved",
)

print_field("📝 Answer", fourth_response.choices[0].message.content or "(none)")

# Extract and display the raw prompt that was sent to the model
raw_prompt_fourth = fourth_response.choices[0].raw_output.prompt_fragments[0]
print_header("RAW PROMPT SENT TO MODEL (FOURTH TURN)", "─")
print(raw_prompt_fourth)

# Check if reasoning_content from both first and third turns are in the raw prompt
has_reasoning_content_first = reasoning[:50] in raw_prompt_fourth
has_reasoning_content_third = reasoning_third[:50] in raw_prompt_fourth

print_header("RESULT (FOURTH TURN)", "═")
if has_reasoning_content_first and has_reasoning_content_third:
    print(
        "  ✅ SUCCESS: reasoning_content from both first and third turns IS included in fourth turn requests!"
    )
elif has_reasoning_content_first:
    print("  ⚠️  PARTIAL: Only first turn reasoning_content found in raw prompt")
elif has_reasoning_content_third:
    print("  ⚠️  PARTIAL: Only third turn reasoning_content found in raw prompt")
else:
    print("  ❌ FAILED: reasoning_content not found in raw prompt")
print()
```

Below is the expected output:

```txt GLM-4.7 (output) expandable theme={null}
════════════════════════════════════════════════════════════
  FIRST TURN
════════════════════════════════════════════════════════════
  📝 Content: (none)
  💭 Reasoning:
    │ The user is asking for the sum of 15 and 27. This is a simple arithmetic calculation that I can perform using the calculator function. I need to:
    │ 
    │ - Call the calculator function
    │ - Set operation to "ad...

  🔧 Tool Calls (1):
    [1] id=call_6iGeMOel4nRUNw9B1UZlphUp
        function=calculator
        arguments={"operation": "add", "a": 15, "b": 27}

  ✓ First response has reasoning_content

════════════════════════════════════════════════════════════
  SECOND TURN
════════════════════════════════════════════════════════════
  📝 Answer: 15 + 27 = 42

────────────────────────────────────────────────────────────
  RAW PROMPT SENT TO MODEL
────────────────────────────────────────────────────────────
[gMASK]<sop>

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "calculator", "description": "Perform basic arithmetic operations", "parameters": {"type": "object", "properties": {"operation": {"type": "string", "enum": ["add", "subtract", "multiply", "divide"]}, "a": {"type": "number"}, "b": {"type": "number"}}, "required": ["operation", "a", "b"]}}}
</tools>

For each function call, output the function name and arguments within the following XML format:
<tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>
What is 15 + 27?
<think>The user is asking for the sum of 15 and 27. This is a simple arithmetic calculation that I can perform using the calculator function. I need to:

- Call the calculator function
- Set operation to "add" 
- Set a to 15
- Set b to 27

Let me make the function call.</think><tool_call>calculator<arg_key>operation</arg_key><arg_value>add</arg_value><arg_key>a</arg_key><arg_value>15</arg_value><arg_key>b</arg_key><arg_value>27</arg_value></tool_call>
<tool_response>42</tool_response>
<think>

════════════════════════════════════════════════════════════
  RESULT
════════════════════════════════════════════════════════════
  ✅ SUCCESS: reasoning_content IS included in subsequent requests!


════════════════════════════════════════════════════════════
  THIRD TURN
════════════════════════════════════════════════════════════
  📝 Answer: (none)
  💭 Reasoning:
    │ The user is asking for 20 * 5. I need to use the calculator function with:
    │ - operation: "multiply"
    │ - a: 20
    │ - b: 5

  🔧 Tool Calls (1):
    [1] id=call_s3VSYmOARo0gZUl0S4pai8jk
        function=calculator
        arguments={"operation": "multiply", "a": 20, "b": 5}


════════════════════════════════════════════════════════════
  FOURTH TURN
════════════════════════════════════════════════════════════
  📝 Answer: 100

────────────────────────────────────────────────────────────
  RAW PROMPT SENT TO MODEL (FOURTH TURN)
────────────────────────────────────────────────────────────
[gMASK]<sop>

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "calculator", "description": "Perform basic arithmetic operations", "parameters": {"type": "object", "properties": {"operation": {"type": "string", "enum": ["add", "subtract", "multiply", "divide"]}, "a": {"type": "number"}, "b": {"type": "number"}}, "required": ["operation", "a", "b"]}}}
</tools>

For each function call, output the function name and arguments within the following XML format:
<tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>
What is 15 + 27?
<think>The user is asking for the sum of 15 and 27. This is a simple arithmetic calculation that I can perform using the calculator function. I need to:

- Call the calculator function
- Set operation to "add" 
- Set a to 15
- Set b to 27

Let me make the function call.</think><tool_call>calculator<arg_key>operation</arg_key><arg_value>add</arg_value><arg_key>a</arg_key><arg_value>15</arg_value><arg_key>b</arg_key><arg_value>27</arg_value></tool_call>
<tool_response>42</tool_response>
What is 20 * 5?
<think>The user is asking for 20 * 5. I need to use the calculator function with:
- operation: "multiply"
- a: 20
- b: 5</think><tool_call>calculator<arg_key>operation</arg_key><arg_value>multiply</arg_value><arg_key>a</arg_key><arg_value>20</arg_value><arg_key>b</arg_key><arg_value>5</arg_value></tool_call>
<tool_response>100</tool_response>
<think>

════════════════════════════════════════════════════════════
  RESULT (FOURTH TURN)
════════════════════════════════════════════════════════════
  ✅ SUCCESS: reasoning_content from both first and third turns IS included in fourth turn requests!
```


# Which model should I use?
Source: https://docs.fireworks.ai/guides/recommended-models

Find the best open models for your use case or migrate from closed source models like Claude, GPT, and Gemini

Looking for the right open source model? Whether you're exploring by use case or migrating from closed source models like Claude, GPT, or Gemini, this guide provides recommendations based on **Fireworks internal testing**, **customer deployments**, and **external benchmarks**. We update it regularly as new models emerge.

<Tip>
  Medium and small models typically offer faster responses and lower cost, with some tradeoff in capability for more complex tasks.
</Tip>

## Choose by Use Case

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Category</th>
      <th>Use Case</th>
      <th>Recommended Models</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td><strong>Code & Development</strong></td>
      <td><strong>Code generation, reasoning & agentic tasks</strong></td>
      <td><a href="https://app.fireworks.ai/models/fireworks/deepseek-v4-pro">DeepSeek V4 Pro</a>, <a href="https://app.fireworks.ai/models/fireworks/kimi-k3">Kimi K3</a>, <a href="https://app.fireworks.ai/models/fireworks/glm-5p2">GLM 5.2</a>, <a href="https://app.fireworks.ai/models/fireworks/minimax-m2p7">MiniMax M2.7</a></td>
    </tr>

    <tr>
      <td><strong>AI Applications</strong></td>
      <td><strong>AI agents with tool use</strong></td>
      <td><a href="https://app.fireworks.ai/models/fireworks/kimi-k2p6">Kimi K2.6</a>, <a href="https://app.fireworks.ai/models/fireworks/deepseek-v4-pro">DeepSeek V4 Pro</a>, <a href="https://app.fireworks.ai/models/fireworks/glm-5p2">GLM 5.2</a>, <a href="https://app.fireworks.ai/models/fireworks/minimax-m2p7">MiniMax M2.7</a></td>
    </tr>

    <tr>
      <td />

      <td><strong>General reasoning & planning</strong></td>
      <td><a href="https://app.fireworks.ai/models/fireworks/deepseek-v4-pro">DeepSeek V4 Pro</a>, <a href="https://app.fireworks.ai/models/fireworks/kimi-k2p6">Kimi K2.6</a>, <a href="https://app.fireworks.ai/models/fireworks/glm-5p2">GLM 5.2</a>, <a href="https://app.fireworks.ai/models/fireworks/gpt-oss-120b">GPT-OSS 120B</a> <em>(medium)</em></td>
    </tr>

    <tr>
      <td />

      <td><strong>Long context & summarization</strong></td>
      <td><a href="https://app.fireworks.ai/models/fireworks/deepseek-v4-pro">DeepSeek V4 Pro</a>, <a href="https://app.fireworks.ai/models/fireworks/kimi-k2p6">Kimi K2.6</a>, <a href="https://app.fireworks.ai/models/fireworks/qwen3p6-plus">Qwen3.6 Plus</a>, <a href="https://app.fireworks.ai/models/fireworks/glm-5p2">GLM 5.2</a>, <a href="https://app.fireworks.ai/models/fireworks/deepseek-v4-flash">DeepSeek V4 Flash</a></td>
    </tr>

    <tr>
      <td />

      <td><strong>Fast extraction, classification & search</strong></td>
      <td><a href="https://app.fireworks.ai/models/fireworks/deepseek-v4-flash">DeepSeek V4 Flash</a>, <a href="https://app.fireworks.ai/models/fireworks/minimax-m2p5">MiniMax M2.5</a>, <a href="https://app.fireworks.ai/models/fireworks/kimi-k2p5">Kimi K2.5</a>, <a href="https://app.fireworks.ai/models/fireworks/step-3p7-flash-nvfp4">Step 3.7 Flash</a>, <a href="https://app.fireworks.ai/models/fireworks/gpt-oss-20b">GPT-OSS 20B</a> <em>(small)</em></td>
    </tr>

    <tr>
      <td><strong>Vision & Multimodal</strong></td>
      <td><strong>Vision & document understanding</strong></td>
      <td><a href="https://app.fireworks.ai/models/fireworks/kimi-k2p6">Kimi K2.6</a>, <a href="https://app.fireworks.ai/models/fireworks/qwen3p6-plus">Qwen3.6 Plus</a>, <a href="https://app.fireworks.ai/models/fireworks/step-3p7-flash-nvfp4">Step 3.7 Flash</a>, <a href="https://app.fireworks.ai/models/fireworks/gemma-4-31b-it">Gemma 4 31B</a> <em>(small)</em></td>
    </tr>

    <tr>
      <td />

      <td><strong>Audio & video understanding</strong></td>
      <td><a href="https://fireworks.ai/models/fireworks/qwen3-omni-30b-a3b-instruct">Qwen3 Omni 30B A3B Instruct</a>, <a href="https://fireworks.ai/models/fireworks/nvidia-nemotron-3-nano-omni-30b-a3b">NVIDIA Nemotron 3 Nano Omni 30B A3B</a></td>
    </tr>

    <tr>
      <td><strong>Search & Retrieval</strong></td>
      <td><strong>Embeddings & reranking</strong></td>
      <td><a href="https://app.fireworks.ai/models/fireworks/qwen3-embedding-8b">Qwen3 Embedding 8B</a>, <a href="https://app.fireworks.ai/models/fireworks/qwen3-reranker-8b">Qwen3 Reranker 8B</a></td>
    </tr>
  </tbody>
</table>

<Tip>
  For audio/video workloads, start with `Qwen3 Omni 30B A3B Instruct`. `NVIDIA Nemotron 3 Nano Omni 30B A3B` is a newer omni option, but its reasoning mode is currently limited to text+image inputs; use `enable_thinking: false` for video/audio requests.
</Tip>

<Tip>
  Looking for a smaller MiniLM, BGE, or nomic style embedder? Those legacy models are not shown in the Model Library but still serve on serverless. See [embeddings and reranking](/guides/querying-embeddings-models) for the supported ids.
</Tip>

***

## Migrating from Closed Models?

If you're currently using Claude, OpenAI / GPT, or Gemini models, here's a guide to the best open source alternatives on Fireworks by use case and latency requirements.

### Claude Alternatives

| **Closed Source**                | **Use Case**                                             | **Latency Budget** | **Open Source Alternative**                                                                                                                                                                                                                                                                                                                                                        |
| -------------------------------- | -------------------------------------------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Claude Opus 4.8 / Sonnet 4.6** | • Agentic use cases<br />• Coding<br />• Research agents | High               | • [DeepSeek V4 Pro](https://app.fireworks.ai/models/fireworks/deepseek-v4-pro)<br />• [Kimi K3](https://app.fireworks.ai/models/fireworks/kimi-k3)<br />• [GLM 5.2](https://app.fireworks.ai/models/fireworks/glm-5p2)<br />• [MiniMax M2.7](https://app.fireworks.ai/models/fireworks/minimax-m2p7)<br />• [Qwen3.6 Plus](https://app.fireworks.ai/models/fireworks/qwen3p6-plus) |
| **Claude Haiku 4.5**             | • Agentic use cases<br />• Coding<br />• Research agents | Low                | • [Step 3.7 Flash](https://app.fireworks.ai/models/fireworks/step-3p7-flash-nvfp4)<br />• [DeepSeek V4 Flash](https://app.fireworks.ai/models/fireworks/deepseek-v4-flash)<br />• [MiniMax M2.5](https://app.fireworks.ai/models/fireworks/minimax-m2p5)<br />• [GPT-OSS 20B](https://app.fireworks.ai/models/fireworks/gpt-oss-20b) <em>(small)</em>                              |

### OpenAI GPT Alternatives

| **Closed Source**         | **Use Case**                                          | **Latency Budget** | **Open Source Alternative**                                                                                                                                                                                                                                                                                                                           |
| ------------------------- | ----------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **GPT-5.5 / GPT-5.5 Pro** | • Agentic use cases<br />• Research agents            | High               | • [DeepSeek V4 Pro](https://app.fireworks.ai/models/fireworks/deepseek-v4-pro)<br />• [Kimi K2.6](https://app.fireworks.ai/models/fireworks/kimi-k2p6)<br />• [GLM 5.2](https://app.fireworks.ai/models/fireworks/glm-5p2)<br />• [MiniMax M2.7](https://app.fireworks.ai/models/fireworks/minimax-m2p7)                                              |
| **GPT-5.4 mini & nano**   | • Chatbots<br />• Intent classification<br />• Search | Low                | • [Step 3.7 Flash](https://app.fireworks.ai/models/fireworks/step-3p7-flash-nvfp4)<br />• [DeepSeek V4 Flash](https://app.fireworks.ai/models/fireworks/deepseek-v4-flash)<br />• [MiniMax M2.5](https://app.fireworks.ai/models/fireworks/minimax-m2p5)<br />• [GPT-OSS 20B](https://app.fireworks.ai/models/fireworks/gpt-oss-20b) <em>(small)</em> |

### Google Gemini Alternatives

| **Closed Source**                     | **Use Case**                                                 | **Latency Budget** | **Open Source Alternative**                                                                                                                                                                                                                                                                                                                                                            |
| ------------------------------------- | ------------------------------------------------------------ | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Gemini 3.1 Pro**                    | • Agentic use cases<br />• Research agents<br />• Multimodal | High               | • [DeepSeek V4 Pro](https://app.fireworks.ai/models/fireworks/deepseek-v4-pro)<br />• [Kimi K2.6](https://app.fireworks.ai/models/fireworks/kimi-k2p6)<br />• [GLM 5.2](https://app.fireworks.ai/models/fireworks/glm-5p2)<br />• [Qwen3.6 Plus](https://app.fireworks.ai/models/fireworks/qwen3p6-plus)<br />• [MiniMax M2.7](https://app.fireworks.ai/models/fireworks/minimax-m2p7) |
| **Gemini 3.5 Flash & 3.1 Flash-Lite** | • Chatbots<br />• Intent classification<br />• Search        | Low                | • [Step 3.7 Flash](https://app.fireworks.ai/models/fireworks/step-3p7-flash-nvfp4)<br />• [DeepSeek V4 Flash](https://app.fireworks.ai/models/fireworks/deepseek-v4-flash)<br />• [MiniMax M2.5](https://app.fireworks.ai/models/fireworks/minimax-m2p5)<br />• [GPT-OSS 20B](https://app.fireworks.ai/models/fireworks/gpt-oss-20b) <em>(small)</em>                                  |

**Understanding Latency Budget:**

* **High latency budget**: Quality is priority. Best for complex reasoning, multi-step workflows, and research tasks where accuracy matters more than speed.
* **Low latency budget**: Speed is priority. Best for user-facing applications like chatbots, real-time search, and high-throughput classification.

***

<Tip>
  You can explore and filter every available model in the [Fireworks Model Library](https://app.fireworks.ai/models).
</Tip>

*Last updated: July 2026*


# Reliability and Error Handling
Source: https://docs.fireworks.ai/guides/reliability

Recommended patterns for timeouts, retries, and error handling when building production applications on the Fireworks API.

Building reliable applications requires handling network conditions, transient errors, and long-running requests. This guide covers recommended patterns for production use.

## Timeout configuration

Set timeouts based on your workload type:

| Workload                             | Recommended client timeout     |
| ------------------------------------ | ------------------------------ |
| Interactive / chat                   | 30–60 seconds                  |
| Agentic (tool calls, multi-step)     | 5–30 minutes                   |
| Large model inference (long context) | 10–30 minutes                  |
| Batch job submission                 | 60 seconds (results are async) |

### Python SDK

```python theme={null}
from openai import OpenAI
import httpx

client = OpenAI(
    base_url="https://api.fireworks.ai/inference/v1",
    api_key="<your-api-key>",
    timeout=httpx.Timeout(
        connect=10.0,
        read=1800.0,   # 30 min for long generations
        write=30.0,
        pool=10.0,
    ),
)
```

### Raw HTTP

```python theme={null}
import requests

response = requests.post(
    "https://api.fireworks.ai/inference/v1/chat/completions",
    headers={"Authorization": "Bearer <your-api-key>"},
    json={"model": "...", "messages": [...]},
    timeout=(10, 1800),  # (connect, read) in seconds
)
```

## Retry logic

### Which errors are retryable

| Status | Meaning               | Retry?                           |
| ------ | --------------------- | -------------------------------- |
| `429`  | Rate limit            | ✅ Yes — with backoff             |
| `500`  | Internal server error | ✅ Yes — transient                |
| `502`  | Bad gateway           | ✅ Yes — transient                |
| `503`  | Service unavailable   | ✅ Yes — with backoff             |
| `504`  | Gateway timeout       | ✅ Yes — transient                |
| `400`  | Bad request           | ❌ No — fix the request           |
| `401`  | Unauthorized          | ❌ No — check API key             |
| `404`  | Not found             | ❌ No — check model/deployment ID |
| `422`  | Unprocessable entity  | ❌ No — fix the request body      |

### Exponential backoff with jitter

```python theme={null}
import time, random
from openai import OpenAI, RateLimitError, APIStatusError

def call_with_retry(client, max_retries=5, base_delay=1.0, **kwargs):
    for attempt in range(max_retries):
        try:
            return client.chat.completions.create(**kwargs)
        except RateLimitError:
            if attempt == max_retries - 1:
                raise
            delay = base_delay * (2 ** attempt) + random.uniform(0, 1)
            time.sleep(delay)
        except APIStatusError as e:
            if e.status_code in (500, 502, 503, 504):
                if attempt == max_retries - 1:
                    raise
                delay = base_delay * (2 ** attempt) + random.uniform(0, 1)
                time.sleep(delay)
            else:
                raise
```

### OpenAI SDK built-in retry

```python theme={null}
client = OpenAI(
    base_url="https://api.fireworks.ai/inference/v1",
    api_key="<your-api-key>",
    max_retries=3,
)
```

## Handling 429 rate limits

**On serverless:** Limits scale automatically with sustained usage. For immediate capacity, contact support or switch to a dedicated deployment.

**On dedicated deployments:** Increase concurrency by raising replica counts (for example with `firectl deployment update` and autoscaling settings). See [Autoscaling](/deployments/autoscaling).

## Long-running training jobs

For RL / RFT trainer jobs, use [`reconnect_and_wait`](/fine-tuning/training-api/reference/trainer-job-manager) on the job manager to recover from preemption or transient failures. See [Trainer job manager](/fine-tuning/training-api/reference/trainer-job-manager) for parameters and examples.

To preserve optimizer state across interruptions, set `dcp_save_interval` in your training config. See [RFT parameters reference](/fine-tuning/rft-parameters-reference).

## The analytics dashboard vs. client-side failures

The Fireworks analytics and usage views count **server-acknowledged requests**. They do not capture connection errors that occur before a request reaches the server — those appear as failures on the client but may show as zero or reduced traffic in the console.

If your client shows failures but the dashboard looks clean, the issue is likely client-side: timeout before connection, DNS resolution failure, or network path problems.

Use [Exporting metrics](/deployments/exporting-metrics) for per-deployment Prometheus metrics that reflect what Fireworks infrastructure observed for dedicated deployments.


# Responses API
Source: https://docs.fireworks.ai/guides/response-api



Fireworks.ai offers a powerful Responses API that allows for more complex and stateful interactions with models. This guide will walk you through the key features and how to use them.

<Warning title="Data Retention Policy">
  The Responses API has a different data retention policy than the chat completions endpoint. See [Data Privacy & security](/guides/security_compliance/data_handling#response-api-data-retention).
</Warning>

## Overview

The Responses API is designed for building conversational applications and complex workflows. It allows you to:

* **Continue conversations**: Maintain context across multiple turns without resending the entire history.
* **Use external tools**: Integrate with external services and data sources through MCP/SSE tools (server-executed) or function tools (client-executed).
* **Stream responses**: Receive results as they are generated, enabling real-time applications.
* **Control tool usage**: Set limits on tool calls with `max_tool_calls` parameter.
* **Manage data retention**: Choose whether to store conversations (default) or opt-out with `store=false`.

## Basic Usage

You can interact with the Response API using the Fireworks Python SDK or by making direct HTTP requests.

### Creating a Response

To start a new conversation, you use the `client.responses.create` method. For a complete example, see the [getting started notebook](https://github.com/fw-ai/cookbook/blob/main/archived/learn/response-api/fireworks_mcp_examples.ipynb).

```python OpenAI SDK theme={null}
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.fireworks.ai/inference/v1",
    api_key=os.getenv("FIREWORKS_API_KEY", "YOUR_FIREWORKS_API_KEY_HERE")
)

response = client.responses.create(
    model="accounts/fireworks/models/qwen3-235b-a22b",
    input="What is reward-kit and what are its 2 main features? Keep it short Please analyze the fw-ai-external/reward-kit repository.",
    tools=[{"type": "sse", "server_url": "https://gitmcp.io/docs"}]
)

print(response.output[-1].content[0].text.split("</think>")[-1])
```

### Using Function Tools

Function tools follow the OpenAI-compatible format and are returned to the client for execution.

```python OpenAI SDK theme={null}
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.fireworks.ai/inference/v1",
    api_key=os.getenv("FIREWORKS_API_KEY", "YOUR_FIREWORKS_API_KEY_HERE")
)

response = client.responses.create(
    model="accounts/fireworks/models/qwen3-235b-a22b",
    input="What is the weather like in San Francisco?",
    tools=[{
        "type": "function",
        "name": "get_weather",
        "description": "Get the current weather for a location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA"
                }
            },
            "required": ["location"]
        }
    }],
    tool_choice="auto"
)

# Check if the model wants to call a function
for item in response.output:
    if hasattr(item, 'type') and item.type == "tool_call":
        print(f"Function: {item.function.name}")
        print(f"Arguments: {item.function.arguments}")
```

### Continuing a Conversation with `previous_response_id`

To continue a conversation, you can use the `previous_response_id` parameter. This tells the API to use the context from a previous response, so you don't have to send the entire conversation history again. For a complete example, see the [previous response ID notebook](https://github.com/fw-ai/cookbook/blob/main/archived/learn/response-api/fireworks_previous_response_cookbook.ipynb).

```python OpenAI SDK theme={null}
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.fireworks.ai/inference/v1",
    api_key=os.getenv("FIREWORKS_API_KEY", "YOUR_FIREWORKS_API_KEY_HERE")
)

# First, create an initial response
initial_response = client.responses.create(
    model="accounts/fireworks/models/qwen3-235b-a22b",
    input="What are the key features of reward-kit?",
    tools=[{"type": "sse", "server_url": "https://gitmcp.io/docs"}]
)
initial_response_id = initial_response.id

# Now, continue the conversation
continuation_response = client.responses.create(
    model="accounts/fireworks/models/qwen3-235b-a22b",
    input="How do I install it?",
    previous_response_id=initial_response_id,
    tools=[{"type": "sse", "server_url": "https://gitmcp.io/docs"}]
)

print(continuation_response.output[-1].content[0].text.split("</think>")[-1])
```

## Streaming Responses

For real-time applications, you can stream the response as it's being generated. For a complete example, see the [streaming example notebook](https://github.com/fw-ai/cookbook/blob/main/archived/learn/response-api/fireworks_streaming_example.ipynb).

```python OpenAI SDK theme={null}
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.fireworks.ai/inference/v1",
    api_key=os.getenv("FIREWORKS_API_KEY", "YOUR_FIREWORKS_API_KEY_HERE")
)

stream = client.responses.create(
    model="accounts/fireworks/models/qwen3-235b-a22b",
    input="give me 5 interesting facts on modelcontextprotocol/python-sdk -- keep it short!",
    stream=True,
    tools=[{"type": "mcp", "server_url": "https://mcp.deepwiki.com/mcp"}]
)

for chunk in stream:
    print(chunk)
```

## Cookbook Examples

For more in-depth examples, check out the following notebooks:

* [General MCP Examples](https://github.com/fw-ai/cookbook/blob/main/archived/learn/response-api/fireworks_mcp_examples.ipynb)
* [Using `previous_response_id`](https://github.com/fw-ai/cookbook/blob/main/archived/learn/response-api/fireworks_previous_response_cookbook.ipynb)
* [Streaming Responses](https://github.com/fw-ai/cookbook/blob/main/archived/learn/response-api/fireworks_streaming_example.ipynb)
* [Using `store=False`](https://github.com/fw-ai/cookbook/blob/main/archived/learn/response-api/mcp_server_with_store_false_argument.ipynb)
* [MCP with Streaming](https://github.com/fw-ai/cookbook/blob/main/archived/learn/response-api/fireworks_mcp_with_streaming.ipynb)

## Storing Responses

By default, responses are stored and can be referenced by their ID. You can disable this by setting `store=False`. If you do this, you will not be able to use the `previous_response_id` to continue the conversation. For a complete example, see the [store=False notebook](https://github.com/fw-ai/cookbook/blob/main/archived/learn/response-api/mcp_server_with_store_false_argument.ipynb).

```python OpenAI SDK theme={null}
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.fireworks.ai/inference/v1",
    api_key=os.getenv("FIREWORKS_API_KEY", "YOUR_FIREWORKS_API_KEY_HERE")
)

response = client.responses.create(
    model="accounts/fireworks/models/qwen3-235b-a22b",
    input="give me 5 interesting facts on modelcontextprotocol/python-sdk -- keep it short!",
    store=False,
    tools=[{"type": "mcp", "server_url": "https://mcp.deepwiki.com/mcp"}]
)

# This will fail because the previous response was not stored
try:
    continuation_response = client.responses.create(
        model="accounts/fireworks/models/qwen3-235b-a22b",
        input="Explain the second fact in more detail.",
        previous_response_id=response.id
    )
except Exception as e:
    print(e)
```

## Deleting Stored Responses

When responses are stored (the default behavior with `store=True`), you can immediately delete them from storage using the DELETE endpoint. This permanently removes the conversation data.

<CodeGroup>
  ```python Python (OpenAI SDK) theme={null}
  import os
  from openai import OpenAI
  import requests

  client = OpenAI(
      base_url="https://api.fireworks.ai/inference/v1",
      api_key=os.getenv("FIREWORKS_API_KEY", "YOUR_FIREWORKS_API_KEY_HERE")
  )

  # Create a response
  response = client.responses.create(
      model="accounts/fireworks/models/qwen3-235b-a22b",
      input="What is the capital of France?",
      store=True  # This is the default
  )

  response_id = response.id
  print(f"Created response with ID: {response_id}")

  # Delete the response immediately
  headers = {
      "Authorization": f"Bearer {os.getenv('FIREWORKS_API_KEY')}",
      "x-fireworks-account-id": "your-account-id"
  }
  delete_response = requests.delete(
      f"https://api.fireworks.ai/inference/v1/responses/{response_id}",
      headers=headers
  )

  if delete_response.status_code == 200:
      print("Response deleted successfully")
  else:
      print(f"Failed to delete response: {delete_response.status_code}")
  ```

  ```bash cURL theme={null}
  # First create a response and capture the ID
  RESPONSE=$(curl -X POST https://api.fireworks.ai/inference/v1/responses \
    -H "Authorization: Bearer $FIREWORKS_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "accounts/fireworks/models/qwen3-235b-a22b",
      "input": "What is the capital of France?",
      "store": true
    }')

  # Extract the response ID (using jq or similar JSON parser)
  RESPONSE_ID=$(echo $RESPONSE | jq -r '.id')

  # Delete the response
  curl -X DELETE "https://api.fireworks.ai/inference/v1/responses/$RESPONSE_ID" \
    -H "Authorization: Bearer $FIREWORKS_API_KEY" \
    -H "x-fireworks-account-id: your-account-id"
  ```
</CodeGroup>

<Warning>
  Once a response is deleted, it cannot be recovered.
</Warning>

## Response Structure

All response objects include the following fields:

* `id`: Unique identifier for the response (e.g., `resp_abc123...`)
* `created_at`: Unix timestamp when the response was created
* `status`: Status of the response (typically `"completed"`)
* `model`: The model used to generate the response
* `output`: Array of message objects, tool calls, and tool outputs
* `usage`: Token usage information:
  * `prompt_tokens`: Number of tokens in the prompt
  * `completion_tokens`: Number of tokens in the completion
  * `total_tokens`: Total tokens used
  * `prompt_tokens_details`: Details about prompt tokens:
    * `cached_tokens`: Number of prompt tokens served from cache
* `previous_response_id`: ID of the previous response in the conversation (if any)
* `store`: Whether the response was stored (boolean)
* `max_tool_calls`: Maximum number of tool calls allowed (if set)

### Example Response

```json theme={null}
{
  "id": "resp_abc123...",
  "created_at": 1735000000,
  "status": "completed",
  "model": "accounts/fireworks/models/qwen3-235b-a22b",
  "output": [
    {
      "id": "msg_xyz789...",
      "role": "user",
      "content": [{"type": "input_text", "text": "What is 2+2?"}],
      "status": "completed"
    },
    {
      "id": "msg_def456...",
      "role": "assistant",
      "content": [{"type": "output_text", "text": "2 + 2 equals 4."}],
      "status": "completed"
    }
  ],
  "usage": {
    "prompt_tokens": 15,
    "completion_tokens": 8,
    "total_tokens": 23,
    "prompt_tokens_details": {
      "cached_tokens": 0
    }
  },
  "previous_response_id": null,
  "store": true,
  "max_tool_calls": null
}
```


# Inference for RL Rollouts
Source: https://docs.fireworks.ai/guides/rollout-inference

Session affinity, KV-cache behavior, weight-swap behavior, and MoE Router Replay for rollout traffic on Fireworks inference deployments.

When you use Fireworks inference to collect RL rollouts, the regular [`/v1/completions`](/api-reference/post-completions) and [`/v1/chat/completions`](/api-reference/post-chatcompletions) endpoints expose a few extra features tailored to multi-turn, stateful rollout traffic. You can use these whether or not the underlying deployment is a hot-load deployment.

<Note>
  These features are fully compatible with the OpenAI SDKs — they're all
  attached as either request headers or optional body fields, so no OpenAI SDK
  upgrade is required.
</Note>

## Session affinity

Multi-turn rollouts typically reuse a long prefix between turns (same system prompt, same trajectory so far). To get the KV cache to hit, all turns of a trajectory should land on the same inference replica. Two headers are relevant here:

* `x-multi-turn-session-id` — identifies the agent trajectory. Set this once per trajectory and keep it constant across turns. If both headers are present, Fireworks currently prefers this value when deriving the request's session-affinity key.
* `x-session-affinity` — fallback sticky routing key when `x-multi-turn-session-id` is absent. In most RL rollout setups, set it to the same trajectory ID.

<Tabs>
  <Tab title="OpenAI SDK">
    ```python theme={null}
    from openai import OpenAI

    client = OpenAI(
        api_key="<FIREWORKS_API_KEY>",
        base_url="https://api.fireworks.ai/inference/v1",
    )

    trajectory_id = "traj-42f1"

    for turn in trajectory:
        response = client.chat.completions.create(
            model="accounts/<account_id>/models/<model_id>",
            messages=turn.messages,
            extra_headers={
                "x-multi-turn-session-id": trajectory_id,
                "x-session-affinity": trajectory_id,
                "fireworks-deployment": "accounts/<account_id>/deployments/<deployment_id>",
            },
        )
    ```
  </Tab>

  <Tab title="Training SDK">
    ```python theme={null}
    import secrets

    sampling_client = service.create_sampling_client(
        model_path=snapshot,
        tokenizer=tokenizer,
    )
    sampler = sampling_client.deployment_sampler

    trajectory_id = f"rl-session-{secrets.token_hex(16)}"

    try:
        for prompt_token_ids in trajectory_prompt_token_ids:
            completions = await sampler.sample_with_prompt_tokens(
                prompt_token_ids,
                n=1,
                max_tokens=4096,
                temperature=1.0,
                logprobs=True,
                user=trajectory_id,
            )
    finally:
        sampling_client.close()
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    curl https://api.fireworks.ai/inference/v1/chat/completions \
      -H "Authorization: Bearer <fireworks_api_key>" \
      -H "fireworks-model: accounts/<account_id>/models/<model_id>" \
      -H "fireworks-deployment: accounts/<account_id>/deployments/<deployment_id>" \
      -H "x-multi-turn-session-id: traj-42f1" \
      -H "x-session-affinity: traj-42f1" \
      -H "Content-Type: application/json" \
      -d '{
        "model": "accounts/<account_id>/models/<model_id>",
        "messages": [{"role": "user", "content": "..."}]
      }'
    ```
  </Tab>
</Tabs>

The Training SDK's Tinker-compatible `sampler.sample(...)` method does not expose arbitrary completions request fields. Use the underlying `DeploymentSampler`, as shown above, to pass the trajectory ID in `user`. For independent trajectories, use distinct IDs and separate `n=1` calls; see [`DeploymentSampler`: RL rollout sampling](/fine-tuning/training-api/reference/deployment-sampler#rl-rollout-sampling) for a batched example. Serverless sampling requires `fireworks-ai[training]` 1.2.9 or later for per-request affinity.

<Tip>
  `x-session-affinity` on its own is already documented for general [prompt
  caching](/guides/prompt-caching#optimizing-inference-request-for-caching). In
  RL rollouts you typically also want `x-multi-turn-session-id` so that per-turn
  metrics (TTFT, generation latency) are aggregated by trajectory, while
  preserving the current serving preference when both headers are supplied.
</Tip>

## KV cache behavior for RL rollouts

The active request stream, the session ID, and `reset_prompt_cache` are tightly coupled in rollout workflows, but they are not the same mechanism. Keep the three layers separate:

| Layer                 | Scope                                                  | What it controls                                                        | What it does not control                               |
| --------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------- | ------------------------------------------------------ |
| Single request stream | One HTTP request that has already started decoding     | Active in-flight KV/state for that stream                               | Future prompt-prefix reuse after the stream ends       |
| Session ID            | Later requests that use the same stable trajectory key | Sticky routing to the same replica and `new_session` namespace behavior | A cache hit by itself, or active-stream recompute      |
| `reset_prompt_cache`  | Requests admitted after a checkpoint swap              | Which reusable prompt-prefix KV namespace later requests can use        | The active in-flight KV for a request already decoding |

### Active request stream

An active request stream is one in-flight HTTP request. Its active KV/state is private to that running decode.

When a checkpoint swap happens under **async transition**, the stream pauses, weights swap, and the same HTTP stream resumes with its existing active KV/state. `reset_prompt_cache` does not flush, invalidate, or recompute that active KV. Passing the same `x-multi-turn-session-id` also does not change this active-stream behavior.

When a checkpoint swap happens under **sync transition**, the server waits for in-flight requests to finish on the old weights before swapping. New requests that arrive during the swap can receive HTTP `425 Too Early` and should retry.

If every token in a turn must come from exactly one policy version, do not let a long request cross an async hot-load boundary. Use sync transition, end the turn before signaling the next snapshot, or track the streamed snapshot identity and filter rollouts accordingly.

### Session ID

For rollout traffic, use one stable session ID per trajectory:

* `x-multi-turn-session-id`: identifies the trajectory and is preferred when Fireworks derives the session-affinity key.
* `x-session-affinity`: fallback sticky routing key when `x-multi-turn-session-id` is absent. In RL rollouts, set it to the same trajectory ID.
* `user`: request-body affinity key used by the Training SDK's `DeploymentSampler`. Keep it stable per trajectory when the client does not expose the headers above.

The session ID is coupled to prompt-cache sharing in two ways:

1. **Sticky routing:** later turns route back to the same serving replica, so they can see that replica's local prompt-prefix KV.
2. **Namespace behavior:** with `reset_prompt_cache=new_session`, later requests with an existing session ID can stay pinned to the previous prompt-cache namespace after a checkpoint swap.

A stable session ID is required for reliable reuse, but it is not sufficient by itself. A reusable prompt-prefix KV hit still requires the request to reach a replica that owns the cached prefix, the prompt tokens to match, and the prompt-cache isolation key to match.

Within one already-running HTTP stream, the session ID is not what preserves active KV. The active stream preserves its own KV/state because it is the same in-flight request.

### Behavior during weight swap

If your rollout traffic hits a hot-load deployment, a new checkpoint can arrive mid-rollout. What happens to your requests depends on the deployment's configured transition mode:

* **Async transition (recommended for RL):** in-flight requests pause then resume on the same HTTP connection using the new weights. The active turn keeps its current KV state, so it continues rather than restarting. New requests queue up. You see elevated TTFT but no errors.
* **Synchronous transition:** in-flight requests finish on the old weights; new requests get HTTP `425 Too Early` until the swap is done. Your client should retry with back-off, ideally keeping the same session-affinity key so it lands on a replica that has already finished the swap.

See [Checkpoint-swap behavior](/fine-tuning/rl-rollout-integration#checkpoint-swap-behavior) for the full hot-load reference.

### `reset_prompt_cache`

Configure `reset_prompt_cache` per snapshot in `POST /hot_load/v1/models/hot_load`, for example:

```json theme={null}
{ "identity": "version_002", "reset_prompt_cache": "new_session" }
```

This setting applies after the checkpoint swap and controls reusable prompt-prefix KV for later requests.

| `reset_prompt_cache` | Active in-flight request crossing the swap                                                                                      | Later request with the same `x-multi-turn-session-id`          | Later request with a new session ID                           |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------- |
| `all` (default)      | Not recomputed by this setting. Async pauses and resumes with existing active KV; sync lets the request finish before the swap. | Recomputes prompt-prefix KV under the new snapshot namespace.  | Recomputes prompt-prefix KV under the new snapshot namespace. |
| `new_session`        | Not recomputed by this setting. Same active-stream behavior as `all`.                                                           | Can reuse eligible prompt-prefix KV for that existing session. | Recomputes prompt-prefix KV under the new snapshot namespace. |
| `none`               | Not recomputed by this setting. Same active-stream behavior as `all`.                                                           | Can reuse eligible prompt-prefix KV.                           | Can reuse eligible prompt-prefix KV.                          |

For RL rollouts, the usual policy is:

* Use `new_session` when an episode may continue across a weight sync and later turns in that same episode should keep eligible prompt-prefix reuse, while newly started episodes use the latest snapshot namespace.
* Use `all` when the next request should recompute prompt-prefix KV even if it uses the same `x-multi-turn-session-id`.
* Use `none` only when both existing and new sessions should keep using the previous prompt-cache namespace after the swap.

If you do not send a stable `x-multi-turn-session-id` for the trajectory, later requests are treated like new sessions for `new_session` semantics.

## MoE Router Replay

For Mixture-of-Experts models, training-inference divergence often comes from the router picking different top-K experts at the same token position between trainer and inference. Aligning those choices across rollouts and training is known as [Rollout Router Replay (R3)](https://arxiv.org/abs/2510.11370).

Fireworks inference supports returning the selected MoE experts for every token and every MoE layer. Pass `include_routing_matrix: true` together with `logprobs: true` on your request:

```bash theme={null}
curl https://api.fireworks.ai/inference/v1/chat/completions \
  -H "Authorization: Bearer <fireworks_api_key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "accounts/<account_id>/models/<model_id>",
    "messages": [{"role": "user", "content": "..."}],
    "include_routing_matrix": true,
    "logprobs": true
  }'
```

The selected expert indices for each token are returned alongside logprobs. For `/v1/chat/completions` you find them at `choices[i].logprobs.content[j].routing_matrix`; for `/v1/completions` the structure is analogous. Each value is a flattened, base64-encoded `uint8` array of shape `[num_layers_with_moe, num_active_experts]`.

### Example response (DeepSeek V3)

```json theme={null}
{
  "object": "text_completion",
  "model": "...my-deepseek-v3-model...",
  "choices": [
    {
      "index": 0,
      "logprobs": {
        "content": [
          {
            "token": " ",
            "logprob": -0.00014507,
            "sampling_logprob": -0.0001450882,
            "token_id": 223,
            "routing_matrix": "CYvWPzaOl8g/o7q2XPVTMJ7w/Y8G..."
          }
        ]
      }
    }
  ]
}
```

### Decoding the routing matrix

DeepSeek V3 has 58 MoE layers (the first 3 of 61 total are dense) and selects 8 active experts per token, so each decoded buffer is `58 * 8 = 464` bytes.

```python theme={null}
import base64
import numpy as np

num_layers_with_moe = 58
num_active_experts = 8

encoded = choice["logprobs"]["content"][0]["routing_matrix"]
raw_bytes = base64.b64decode(encoded)
routing_matrix = np.frombuffer(raw_bytes, dtype=np.uint8).reshape(
    num_layers_with_moe, num_active_experts
)
# routing_matrix[layer_idx] -> array of 8 expert indices for that token
```

### Other API modes

* **Completions API (`/v1/completions`)**: same mechanism — `include_routing_matrix` and `logprobs` are top-level body fields.
* **Streaming (`stream: true`)**: `routing_matrix` is included on each streamed token chunk's `logprobs.content` entry.
* **Prompt tokens (`echo: true`)**: returns expert selection for the prompt tokens too. Combine with `echo_last: N` to only include expert selection for the last N prompt tokens.

## Policy version in responses

On **hot-load** deployments, track which snapshot served each token—useful for off-policy RL and debugging stale rollouts.

### Streaming

Each streamed chunk includes the loaded snapshot in the `model` field as `accounts/<account_id>/models/<model_id>@<snapshot_identity>`:

```text theme={null}
data: {"object":"text_completion","model":"accounts/<account_id>/models/<model_id>@version_002","choices":[{"index":0,"text":"...","finish_reason":null}],...}
```

Parse the suffix after `@` as the policy version for that token. If a weight swap happens mid-stream under async transition, later chunks may reflect the new snapshot.

### Non-streaming

Non-streaming responses are adding the same `model@snapshot_identity` convention; until your deployment shape exposes it, rely on streaming or correlate rollout timing with your hot-load poll timestamps.

## Related pages

<CardGroup>
  <Card title="Quickstart (BYOT)" icon="rotate" href="/fine-tuning/rl-rollout-integration">
    Prerequisites, hot-load deployment, and rollout loop.
  </Card>

  <Card title="Incremental snapshots" icon="layer-group" href="/fine-tuning/rl-rollout-integration#arc2">
    ARC2 compression and incremental hot-load signals.
  </Card>

  <Card title="Ledger & checkpoint swap" icon="bug" href="/fine-tuning/rl-rollout-integration#ledger-debugging">
    Detailed semantics of request behavior across weight swaps.
  </Card>

  <Card title="Prompt caching" icon="memory" href="/guides/prompt-caching">
    Session-affinity patterns for general cache hit optimization.
  </Card>
</CardGroup>


# Audit & Access Logs
Source: https://docs.fireworks.ai/guides/security_compliance/audit_logs

Monitor and track account activities with audit logging for Enterprise accounts

Audit logs are available for Enterprise accounts. This feature enhances security visibility, incident investigation, and compliance reporting.

Audit logs include data access logs. All read, write, and delete operations on storage are logged, normalized, and enriched with account context for complete visibility.

## View audit logs

You can view audit logs, including data access logs, using the Fireworks CLI:

```bash theme={null}
firectl ls audit-logs
```

<Frame>
  <img alt="Audit logs table showing data access activities with columns for timestamp, principal, response code, resource path, and message" />
</Frame>


# Zero Data Retention
Source: https://docs.fireworks.ai/guides/security_compliance/data_handling

Data retention policies at Fireworks

## Zero data retention

Fireworks has Zero Data Retention by default. Specifically, this means

* Fireworks does not log or store prompt or generation data for any open models, without explicit user opt-in.
  * More technically: prompt and generation data exist only in volatile memory for the duration of the request. If [prompt caching](https://docs.fireworks.ai/guides/prompt-caching#data-privacy) is active, some prompt data (and associated KV caches) can be stored in volatile memory for several minutes. In either case, prompt and generation data are not logged into any persistent storage.
* Fireworks logs metadata (e.g. number of tokens in a request) as required to deliver the service.
* Users can explicitly opt-in to log prompt and generation data for certain advanced features (e.g. FireOptimizer).

## Response API data retention

For the Response API specifically, Fireworks retains conversation data with the following policy when the API request has `store=True` (the default):

* **What is stored**: Conversation messages include the complete conversation data:
  * User prompts
  * Model responses
  * Tools called by the model
* **Opt-out option**: You can disable data storage by setting `store=False` in your API requests to prevent any conversation data from being retained.
* **Retention period**: All stored conversation data is automatically deleted after 30 days.
* **Immediate deletion**: You can immediately delete stored conversation data using the DELETE API endpoint by providing the `response_id`. This will permanently remove the record.

This retention policy is designed to be consistent with the OpenAI API while providing users control over their data storage preferences.

<Note>
  The Response API retention policy only applies to conversation data when using the Response API endpoints. All other Fireworks services follow the zero data retention policy described above.
</Note>


# Data Security
Source: https://docs.fireworks.ai/guides/security_compliance/data_security

How we secure and handle your data for inference and training

At Fireworks, protecting customer data is at the core of our platform. We design all of our systems, infrastructure, and business processes to ensure customer trust through verifiable security & compliance.

This page provides an overview of our key security measures. For documentation and audit reports, see our [Trust Center](https://trust.fireworks.ai/).

## Zero Data Retention

Fireworks does not log or store prompt or generation data for open models, without explicit user opt-in. See our [Zero Data Retention Policy](https://docs.fireworks.ai/guides/security_compliance/data_handling).

For US-only inference residency, see [US-only Serverless](/serverless/us-only-serverless).

## Secure Data Handling

**Data Ownership & Control:** Customers maintain ownership of their data. Customer data stored as part of an active workflow can be permanently deleted with auditable confirmation, and secure wipe processes ensure deleted assets cannot be reconstructed.

**Encryption**: Data is encrypted in transit (TLS 1.2+) and at rest (AES-256).

**Bring Your Own Bucket:** Customers may integrate their own cloud storage to retain governance and apply their own compliance frameworks.

* Datasets: [Bring your own bucket (BYOB)](/fine-tuning/finetuning-intro#dataset-storage-byob) (GCS, AWS S3, Azure)
* Models: [External AWS S3 Bucket Integration](/models/uploading-custom-models#uploading-your-model)

**Bring Your Own Key:** Customers may integrate their own AWS KMS key to retain governance and apply their own encryption policies.

* Datasets and checkpoints: [Customer-Managed Encryption Keys for Fine-Tuning](/fine-tuning/finetuning-intro#cmek) (SFT supported; DPO and RFT coming soon)
* Final model weights: Coming soon

**Access Logging:** All customer data access is logged, monitored, and protected against tampering. See [Audit & Access Logs](https://docs.fireworks.ai/guides/security_compliance/audit_logs).

## Workload Isolation

Dedicated workloads run in logically isolated environments, preventing cross-customer access or data leakage.

## Secure Training

Fireworks enables secure model training, including fine-tuning and reinforcement learning, while maintaining customer control over sensitive components and data. This approach builds on our [Zero Data Retention](#zero-data-retention) policy to ensure sensitive training data never persists on our platform.

**Customer-Controlled Architecture:** For advanced training workflows like reinforcement learning, critical components remain under customer control:

* Reward models and reward functions are kept proprietary and not shared
* Rollout servers and training metrics are built and managed by customers
* Model checkpoints are managed through secure cloud storage registries

**Minimal Data Sharing:** Training data is shared via controlled bucket access with minimal sharing and step-wise retention, limiting data exposure while enabling effective training workflows.

**API-Based Integration:** Customers leverage Fireworks' training APIs while maintaining full control over sensitive components, ensuring no cross-component data leakage.

<Tip>
  For detailed guidance on secure reinforcement fine-tuning and using your own cloud storage, see [Training security](/fine-tuning/finetuning-intro#training-security).
</Tip>

## Technical Safeguards

* **Device Trust**: Only approved, secured devices with strong authentication can access sensitive Fireworks systems.
* **Identity & Access Management**: Fine-grained access controls are enforced across all Fireworks environments, following the principle of least privilege.
* **Network Security**
  * Private network isolation for customer workloads.
  * Firewalls and security groups prevent unauthorized inbound/outbound traffic.
  * DDoS protection is in place across core services.
* **Monitoring & Detection**: Real-time monitoring and anomaly detection systems alert on suspicious activity
* **Vulnerability Management**: Continuous scanning and patching processes keep infrastructure up to date against known threats.

## Operational Security

* **Security Reviews & Testing**: Regular penetration testing validates controls.
* **Incident Response**: A formal incident response plan ensures swift containment, customer notification, and remediation if an issue arises.
* **Employee Access**: Only a minimal subset of Fireworks personnel have access to production systems, and all access is logged and periodically reviewed.
* **Third-Party Risk Management**: Vendors and subprocessors undergo rigorous due diligence and contractual security obligations.

## Compliance & Certifications

Fireworks aligns with leading industry standards to support customer compliance obligations.

## ISO Certifications

Fireworks has achieved all three ISO certifications covering security, privacy, and AI management:

| Certification | Scope                                                             | Status     |
| ------------- | ----------------------------------------------------------------- | ---------- |
| **ISO 27001** | Information Security Management Systems (ISMS)                    | ✅ Achieved |
| **ISO 27701** | Privacy Information Management (extension to ISO 27001)           | ✅ Achieved |
| **ISO 42001** | AI Management Systems — responsible AI development and deployment | ✅ Achieved |

Certificate PDFs for all three certifications are available for download from the [Fireworks Trust Center](https://trust.fireworks.ai).

These certifications are maintained through continuous monitoring and annual audits — not one-time assessments.

### What these certifications cover

**ISO 27001** verifies that Fireworks has implemented a documented, risk-managed information security program with controls covering access, encryption, incident response, and vendor management.

**ISO 27701** extends ISO 27001 to cover privacy — including how Fireworks collects, processes, and protects personal data from customers and end users.

**ISO 42001** covers the responsible development and deployment of AI systems, including risk assessment, transparency, and bias mitigation practices.

### Other programs

* **SOC 2 Type II** (certified)
* **HIPAA Support**: Fireworks is HIPAA compliant and supports healthcare and life sciences organizations in leveraging our rapid inference capabilities with confidence.
* **Regulatory Alignment**: Controls are mapped to GDPR, CCPA, and other international data protection frameworks

<Note>
  Documentation and audit reports are available in our [Trust Center](https://trust.fireworks.ai/).
</Note>


# Video & Audio Inputs
Source: https://docs.fireworks.ai/guides/video-audio-inputs

Query multimodal models to process video and audio content directly

Some Omni/multimodal models can process audio and/or video inputs directly, enabling video captioning, scene analysis, content understanding, and multimodal question answering. A good example is Qwen3 Omni (`qwen3-omni-30b-a3b-instruct`), which supports video, audio, and text inputs in a single request. Deploy these models using [dedicated deployments](/getting-started/ondemand-quickstart) for production workloads.

## Available models

| Model                                                                                            | Input support      | Notes                         |
| ------------------------------------------------------------------------------------------------ | ------------------ | ----------------------------- |
| [Qwen3 Omni 30B A3B Instruct](https://fireworks.ai/models/fireworks/qwen3-omni-30b-a3b-instruct) | Video, audio, text | Dedicated deployment required |
| [Molmo2-4B](https://fireworks.ai/models/fireworks/molmo2-4b)                                     | Video, text        | Dedicated deployment required |
| [Molmo2-8B](https://fireworks.ai/models/fireworks/molmo2-8b)                                     | Video, text        | Dedicated deployment required |

<Note>
  Qwen3 Omni supports native video and audio inputs. Molmo2 models are video-only, so use the same request structure as below, but omit `audio_url`. Molmo2 models cannot understand audio from videos.
</Note>

## Create a deployment

Video and audio models require dedicated deployments. Create one using firectl:

```bash theme={null}
firectl deployment create qwen3-omni-30b-a3b-instruct \
  --account-id <YOUR_ACCOUNT_ID> \
  --min-replica-count 1 \
  --max-replica-count 1 \
  --deployment-shape qwen3-omni-30b-a3b-instruct-minimal
```

<Note>
  Make sure to use the predefined `qwen3-omni-30b-a3b-instruct-minimal` deployment shape for your deployment to work correctly.
</Note>

## Chat Completions API

Provide video and audio as base64-encoded data URLs. The model accepts `video_url`, `audio_url`, and `text` content types.

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    import os
    import base64
    import requests

    # Load and encode your preprocessed video and audio
    with open("processed_video.mp4", "rb") as f:
        video_b64 = base64.b64encode(f.read()).decode("utf-8")

    with open("audio.ogg", "rb") as f:
        audio_b64 = base64.b64encode(f.read()).decode("utf-8")

    # API configuration
    url = "https://api.fireworks.ai/inference/v1/chat/completions"
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {os.environ['FIREWORKS_API_KEY']}",
    }

    # Request payload
    payload = {
        "model": "accounts/<YOUR_ACCOUNT_ID>/models/qwen3-omni-30b-a3b-instruct#accounts/<YOUR_ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>",
        "max_tokens": 1000,
        "temperature": 0.3,
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "video_url", "video_url": {"url": f"data:video/mp4;base64,{video_b64}"}},
                    {"type": "audio_url", "audio_url": {"url": f"data:audio/ogg;base64,{audio_b64}"}},
                    {"type": "text", "text": "Describe what happens in this video."},
                ],
            },
        ],
    }

    # Send request
    response = requests.post(url, headers=headers, json=payload)
    print(response.json()["choices"][0]["message"]["content"])
    ```
  </Tab>

  <Tab title="curl">
    ```bash theme={null}
    # Encode your files (run these separately)
    VIDEO_B64=$(base64 -i processed_video.mp4)
    AUDIO_B64=$(base64 -i audio.ogg)

    curl https://api.fireworks.ai/inference/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $FIREWORKS_API_KEY" \
      -d '{
        "model": "accounts/<YOUR_ACCOUNT_ID>/models/qwen3-omni-30b-a3b-instruct#accounts/<YOUR_ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>",
        "max_tokens": 1000,
        "temperature": 0.3,
        "messages": [
          {
            "role": "user",
            "content": [
              {"type": "video_url", "video_url": {"url": "data:video/mp4;base64,'$VIDEO_B64'"}},
              {"type": "audio_url", "audio_url": {"url": "data:audio/ogg;base64,'$AUDIO_B64'"}},
              {"type": "text", "text": "Describe what happens in this video."}
            ]
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

## Working with videos

Video models perform best with preprocessed inputs that balance quality and token efficiency. Use ffmpeg to optimize your video and audio before sending requests.

### Preprocessing video

Extract frames at 1 FPS and downscale to 360p for efficient processing:

```bash theme={null}
ffmpeg -y -i input_video.mp4 \
  -t 60 \
  -vf "fps=1,scale=-1:360" \
  -c:v libx264 -preset fast \
  -an \
  processed_video.mp4
```

| Parameter      | Description                                     |
| -------------- | ----------------------------------------------- |
| `-t 60`        | Limit to first 60 seconds                       |
| `fps=1`        | Extract 1 frame per second                      |
| `scale=-1:360` | Downscale to 360p height, maintain aspect ratio |
| `-an`          | Remove audio track (extracted separately)       |

### Preprocessing audio

Extract audio as Opus in an Ogg container for optimal compression:

```bash theme={null}
ffmpeg -y -i input_video.mp4 \
  -t 60 \
  -vn \
  -c:a libopus \
  -b:a 24k \
  -ar 16000 \
  -ac 1 \
  audio.ogg
```

| Parameter      | Description               |
| -------------- | ------------------------- |
| `-t 60`        | Limit to first 60 seconds |
| `-vn`          | Remove video track        |
| `-c:a libopus` | Use Opus codec            |
| `-b:a 24k`     | 24 kbps bitrate           |
| `-ar 16000`    | 16 kHz sample rate        |
| `-ac 1`        | Mono audio                |

### Complete preprocessing example

```python theme={null}
import subprocess
import tempfile
import base64
import os

def preprocess_video(video_path: str) -> tuple[str, str]:
    """
    Preprocess video for optimal model input.
    
    Returns:
        Tuple of (video_base64, audio_base64)
    """
    with tempfile.NamedTemporaryFile(suffix=".mp4", delete=False) as tmp_video:
        processed_video_path = tmp_video.name
    with tempfile.NamedTemporaryFile(suffix=".ogg", delete=False) as tmp_audio:
        audio_path = tmp_audio.name
    
    try:
        # Process video: 1 FPS, 360p, max 60 seconds
        subprocess.run([
            "ffmpeg", "-y", "-i", video_path,
            "-t", "60",
            "-vf", "fps=1,scale=-1:360",
            "-c:v", "libx264", "-preset", "fast",
            "-an",
            processed_video_path
        ], check=True, capture_output=True)
        
        # Extract audio: Opus/Ogg, mono, 16kHz, 24kbps
        subprocess.run([
            "ffmpeg", "-y", "-i", video_path,
            "-t", "60",
            "-vn",
            "-c:a", "libopus",
            "-b:a", "24k",
            "-ar", "16000",
            "-ac", "1",
            audio_path
        ], check=True, capture_output=True)
        
        with open(processed_video_path, "rb") as f:
            video_b64 = base64.b64encode(f.read()).decode("utf-8")
        
        with open(audio_path, "rb") as f:
            audio_b64 = base64.b64encode(f.read()).decode("utf-8")
        
        return video_b64, audio_b64
    
    finally:
        os.unlink(processed_video_path)
        os.unlink(audio_path)
```

<Tip>
  Preprocessing is highly recommended to reduce latency and ensure consistent performance.
</Tip>

## Performance considerations

**Tips for optimal throughput:**

* **Preprocess all videos** – 1 FPS at 360p provides good quality with minimal tokens
* **Extract audio separately** – Opus/Ogg at 24kbps offers excellent compression
* **Limit video duration** – Cap at 60 seconds for consistent performance
* **Use dedicated deployments** – Scale replicas based on your throughput needs

## Known limitations

1. **Video duration**: Maximum 60 seconds recommended for optimal performance
2. **Supported formats**: `.mp4` for video, `.ogg` (Opus) for audio
3. **Base64 size**: Total encoded payload should be under 10MB
4. **Deployment required**: Video models are not available on serverless; dedicated deployment required

## Related resources

<CardGroup>
  <Card title="Chat with Video using Qwen3 Omni" href="https://colab.research.google.com/github/fw-ai/cookbook/blob/main/archived/multimedia/video/Qwen3-Omni-Chat-With-Video-Cookbook.ipynb" icon="video">
    Interactive notebook for video and audio analysis
  </Card>

  <Card title="Vision models" href="/guides/querying-vision-language-models" icon="image">
    Query models with image inputs
  </Card>

  <Card title="Dedicated deployments" href="/getting-started/ondemand-quickstart" icon="server">
    Deploy models on dedicated GPUs
  </Card>
</CardGroup>


# Kimi K2 family
Source: https://docs.fireworks.ai/models/kimi-k2

Using Kimi K2 family models in agentic and tool-calling workflows on Fireworks.

## Using the Kimi K2 family in agentic workflows

### Always set `max_tokens`

Models in the **Kimi K2 family** can produce very long reasoning traces before arriving at a final answer. In agentic workflows where output is parsed and passed to downstream steps, always set `max_tokens` explicitly:

```python theme={null}
response = client.chat.completions.create(
    model="accounts/fireworks/models/kimi-k2-instruct",
    messages=messages,
    max_tokens=512,
    tools=tools,
)
```

Starting points by output type:

| Output type         | Suggested `max_tokens` |
| ------------------- | ---------------------- |
| Tool call responses | 256–512                |
| Short text          | 512–1024               |
| Structured JSON     | 1024–2048              |
| Long-form reasoning | 4096+                  |

### Tool schema design

Kimi K2 family models perform best when tools have clearly distinct names, descriptions, and parameter schemas. When tools have overlapping surface areas the model may select the wrong one.

```python theme={null}
# Less clear — overlapping descriptions
tools = [
    {"type": "function", "function": {"name": "read", "description": "Read data from a source", "parameters": {"type": "object", "properties": {}}}},
    {"type": "function", "function": {"name": "exec", "description": "Execute an operation on a source", "parameters": {"type": "object", "properties": {}}}},
]

# More clear — distinct names and explicit scope
tools = [
    {
        "type": "function",
        "function": {
            "name": "read_file",
            "description": "Read the contents of a file at a given path. Use this to inspect existing content before making changes. Do not use this to run code.",
            "parameters": {"type": "object", "properties": {}},
        },
    },
    {
        "type": "function",
        "function": {
            "name": "execute_command",
            "description": "Run a shell command and return its output. Use this to run scripts, tests, or system operations. Do not use this to read file contents.",
            "parameters": {"type": "object", "properties": {}},
        },
    },
]
```

Best practices:

* Name tools by their primary action, not their domain (`read_file` not `file_tool`).
* Write descriptions that distinguish tools from each other, including what each tool is *not* for.
* Avoid optional parameters that make two tools look identical with only a flag difference.

### Timeouts for agentic loops

Inference for Kimi K2 family models can be slow on large inputs. For multi-step agents, set your client read timeout to at least 10–30 minutes per call. See [Reliability and error handling](/guides/reliability).


# Quantization
Source: https://docs.fireworks.ai/models/quantization

Reduce model precision to improve performance and lower costs

Quantization reduces the number of bits used to serve a model, improving performance and reducing cost by 30-50%. However, this can change model numerics which may introduce small changes to the output.

<Tip>
  Read our [blog post](https://fireworks.ai/blog/fireworks-quantization) for a detailed treatment of how quantization affects model quality.
</Tip>

## Checking available precisions

Models may support different numerical precisions like FP16, FP8, BF16, or INT8, which affect memory usage and inference speed.

**Check default precision:**

```bash theme={null}
firectl model get accounts/fireworks/models/llama-v3p1-8b-instruct | grep "Default Precision"
```

**Check supported precisions:**

```bash theme={null}
firectl model get accounts/fireworks/models/llama-v3p1-8b-instruct | grep -E "(Supported Precisions|Supported Precisions With Calibration)"
```

The `Precisions` field indicates what precisions the model has been prepared for.

## Quantizing a model

A model can be quantized to 8-bit floating-point (FP8) precision.

<Tabs>
  <Tab title="firectl">
    ```bash theme={null}
    firectl prepare-model <MODEL_ID>
    ```
  </Tab>

  <Tab title="Python (REST API)">
    ```python theme={null}
    import os
    import requests

    ACCOUNT_ID = os.environ.get("FIREWORKS_ACCOUNT_ID")
    API_KEY = os.environ.get("FIREWORKS_API_KEY")
    MODEL_ID = "<YOUR_MODEL_ID>" # The ID of the model you want to prepare

    response = requests.post(
      f"https://api.fireworks.ai/v1/accounts/{ACCOUNT_ID}/models/{MODEL_ID}:prepare",
      headers={
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json"
      },
      json={
        "precision": "FP8"
      }
    )

    print(response.json())
    ```
  </Tab>
</Tabs>

<Note>This is an additive process that enables creating deployments with additional precisions. The original FP16 checkpoint is still available for use.</Note>

You can check on the status of preparation by running:

<Tabs>
  <Tab title="firectl">
    ```bash theme={null}
    firectl model get <MODEL_ID>
    ```
  </Tab>

  <Tab title="Python (REST API)">
    ```python theme={null}
    import os
    import requests

    ACCOUNT_ID = os.environ.get("FIREWORKS_ACCOUNT_ID")
    API_KEY = os.environ.get("FIREWORKS_API_KEY")
    MODEL_ID = "<YOUR_MODEL_ID>" # The ID of the model you want to get

    response = requests.get(
      f"https://api.fireworks.ai/v1/accounts/{ACCOUNT_ID}/models/{MODEL_ID}",
      headers={
        "Authorization": f"Bearer {API_KEY}"
      }
    )

    print(response.json())
    ```
  </Tab>
</Tabs>

and checking if the state is still in `PREPARING`. A successfully prepared model will have the desired precision added
to the `Precisions` list.

## Creating an FP8 deployment

By default, creating a deployment uses the FP16 checkpoint. To use a quantized FP8 checkpoint, first ensure the model has been prepared for FP8 (see [Checking available precisions](#checking-available-precisions) above), then pass the `--precision` flag when creating your deployment:

<Tabs>
  <Tab title="firectl">
    ```bash theme={null}
    firectl deployment create <MODEL> --accelerator-type NVIDIA_H100_80GB --precision FP8
    ```
  </Tab>

  <Tab title="Python (REST API)">
    ```python theme={null}
    import os
    import requests

    ACCOUNT_ID = os.environ.get("FIREWORKS_ACCOUNT_ID")
    API_KEY = os.environ.get("FIREWORKS_API_KEY")
    # The ID of the model you want to deploy.
    # The model must be prepared for FP8 precision.
    MODEL_ID = "<YOUR_MODEL_ID>"
    DEPLOYMENT_NAME = "My FP8 Deployment"

    response = requests.post(
      f"https://api.fireworks.ai/v1/accounts/{ACCOUNT_ID}/deployments",
      headers={
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json"
      },
      json={
        "displayName": DEPLOYMENT_NAME,
        "baseModel": MODEL_ID,
        "acceleratorType": "NVIDIA_H100_80GB",
        "precision": "FP8",
      }
    )

    print(response.json())
    ```
  </Tab>
</Tabs>

<Note>Quantized deployments can only be served using H100 GPUs.</Note>


# Custom Models
Source: https://docs.fireworks.ai/models/uploading-custom-models

Upload, verify, and deploy your own models from Hugging Face or elsewhere

Upload your own models from Hugging Face or elsewhere to deploy fine-tuned or custom-trained models optimized for your use case.

* **Multiple upload options** – Upload from local files or directly from S3 buckets or Azure Blob Storage
* **Secure uploads** – All uploads are encrypted and models remain private to your account by default

## Requirements

### Supported architectures

Fireworks supports most popular model architectures, including:

* [DeepSeek V1, V2 & V3](https://huggingface.co/deepseek-ai)
* [Qwen, Qwen2, Qwen2.5, Qwen2.5-VL, Qwen3](https://huggingface.co/Qwen)
* [Kimi K2 family](https://huggingface.co/moonshotai)
* [GLM 4.X family](https://huggingface.co/zai-org)
* [Llama 1, 2, 3, 3.1, 4](https://huggingface.co/docs/transformers/en/model_doc/llama2)
* [Mistral & Mixtral](https://huggingface.co/docs/transformers/en/model_doc/mistral)
* [Gemma](https://huggingface.co/docs/transformers/en/model_doc/gemma)
* [GPT-OSS 120B and 20B](https://huggingface.co/openai/gpt-oss-120b)

<Accordion title="View all supported architectures">
  - [DBRX](https://huggingface.co/docs/transformers/en/model_doc/dbrx)
  - [DeepSeek V1, V2 & V3](https://huggingface.co/deepseek-ai)
  - [Falcon](https://huggingface.co/docs/transformers/en/model_doc/falcon)
  - [Gemma](https://huggingface.co/docs/transformers/en/model_doc/gemma)
  - [GPT NeoX](https://huggingface.co/docs/transformers/en/model_doc/gpt_neox)
  - [Idefics3](https://huggingface.co/docs/transformers/en/model_doc/idefics3)
  - [Llama 1, 2, 3, 3.1, 4](https://huggingface.co/docs/transformers/en/model_doc/llama2)
  - [LLaVA](https://huggingface.co/docs/transformers/main/en/model_doc/llava)
  - [Mistral](https://huggingface.co/docs/transformers/en/model_doc/mistral) & [Mixtral](https://huggingface.co/docs/transformers/en/model_doc/mixtral)
  - [Phi, Phi-3, Phi-3V, Phi-4](https://huggingface.co/docs/transformers/en/model_doc/phi)
  - [Pythia](https://huggingface.co/docs/transformers/en/model_doc/gpt_neox)
  - [Qwen](https://huggingface.co/docs/transformers/en/model_doc/qwen), [Qwen2](https://huggingface.co/docs/transformers/en/model_doc/qwen2), [Qwen2.5](https://huggingface.co/collections/Qwen/qwen25-66e81a666513e518adb90d9e), [Qwen2.5-VL](https://huggingface.co/collections/Qwen/qwen25-vl-6795ffac22b334a837c0f9a5), [Qwen3](https://huggingface.co/Qwen)
  - [Solar](https://huggingface.co/upstage/SOLAR-10.7B-v1.0)
  - [StableLM](https://huggingface.co/docs/transformers/main/en/model_doc/stablelm)
  - [Starcoder (GPTBigCode)](https://huggingface.co/docs/transformers/en/model_doc/gpt_bigcode) & [Starcoder2](https://huggingface.co/docs/transformers/main/en/model_doc/starcoder2)
  - [Vision Llama](https://huggingface.co/docs/transformers/en/model_doc/llama2)
</Accordion>

### Required files

You'll need standard Hugging Face model files: `config.json`, model weights (`.safetensors` or `.bin`), and tokenizer files.

<Accordion title="View detailed file requirements">
  The model files you will need to provide depend on the model architecture. In general, you will need:

  * **Model configuration**: `config.json`

    <Note>
      Fireworks does not support the `quantization_config` option in `config.json`.
    </Note>
  * **Model weights** in one of the following formats:
    * `*.safetensors`
    * `*.bin`
  * **Weights index**: `*.index.json`
  * **Tokenizer file(s)**, e.g.:
    * `tokenizer.model`
    * `tokenizer.json`
    * `tokenizer_config.json`
  * **Vision / multimodal models** additionally require the image (and, where applicable, video) preprocessor configuration files that their architecture's processor loads, for example:

    * `preprocessor_config.json` (image preprocessor)
    * `video_preprocessor_config.json` (video preprocessor, for architectures that support video)

    The exact set is architecture-specific. Provide the **complete and consistent** set that ships with a known-working checkpoint of the **same architecture**, and don't add extra processor config files the architecture doesn't use. For example, Qwen3-VL–family models load an image processor and a video processor and expect both `preprocessor_config.json` **and** `video_preprocessor_config.json`; a missing or mismatched set can cause deployment to fail during startup validation.

  If the requisite files are not present, model deployment may fail.
</Accordion>

<Note>
  **Vision (multimodal) models run an image warmup at startup.** When you deploy a model that Fireworks detects as multimodal (from `config.json` — e.g. it has a `vision_config` and image token ids), startup validation automatically runs a sample image through the model. If the model's image/video preprocessor configuration files are missing or inconsistent with each other, this warmup fails and the deployment enters a `FAILED` state **even though the weights loaded and text generation works**.

  If a multimodal upload fails to become `READY` (or serves text but fails on image inputs), verify that its preprocessor config files match those of a known-working checkpoint of the **same architecture** — the simplest fix is to re-export your checkpoint with the same non-weight files as a reference checkpoint and swap in only your weights.
</Note>

### Customizing base model configuration

For base models (not LoRA adapters), you can customize the chat template and generation defaults by modifying the standard Hugging Face configuration files:

* **Chat template**: Add or modify the `chat_template` field in `tokenizer_config.json`. See the Hugging Face guide on [Templates for Chat Models](https://huggingface.co/docs/transformers/main/en/chat_templating) for details.
* **Generation defaults**: Modify `generation_config.json` to set default generation parameters like `max_new_tokens`, `temperature`, `top_p`, etc.

You can also use a `fireworks.json` file with base models. If present, `fireworks.json` takes priority over `generation_config.json`. See [Customizing generation defaults with fireworks.json](#customizing-generation-defaults-with-fireworksjson) for the full `fireworks.json` schema.

<Warning>
  For LoRA adapters, you must use `fireworks.json` to customize generation defaults. Modifying `generation_config.json` in the adapter folder won't work because adapters inherit these settings from their base model.
</Warning>

## Uploading your model

For larger models, you can upload directly from cloud storage (S3 or Azure Blob Storage) for faster transfer instead of uploading from your local machine.

<Tabs>
  <Tab title="Local files (CLI)">
    Upload from your local machine:

    ```bash theme={null}
    firectl model create <MODEL_ID> /path/to/files/
    ```
  </Tab>

  <Tab title="S3 bucket (CLI)">
    Upload directly from an Amazon S3 bucket:

    ```bash theme={null}
    firectl model create <MODEL_ID> s3://<BUCKET_NAME>/<PATH_TO_MODEL>/ \
      --aws-access-key-id <ACCESS_KEY_ID> \
      --aws-secret-access-key <SECRET_ACCESS_KEY>
    ```

    See the [AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id-credentials-access-keys-update.html) for how to generate an access key ID and secret access key pair.

    <Note>
      Ensure the IAM user has read access to the S3 bucket containing the model.
    </Note>
  </Tab>

  <Tab title="Azure Blob Storage (CLI)">
    Upload directly from Azure Blob Storage using either SAS token or federated identity authentication.

    <Tabs>
      <Tab title="SAS token">
        First, create a Fireworks secret containing your Azure SAS token:

        ```bash theme={null}
        firectl secret create --name <SECRET_NAME> --value <SAS_TOKEN>
        ```

        Then, upload the model using the secret:

        ```bash theme={null}
        firectl model create <MODEL_ID> https://<STORAGE_ACCOUNT>.blob.core.windows.net/<CONTAINER>/<PATH> \
          --azure-sas-token-secret accounts/<ACCOUNT_ID>/secrets/<SECRET_NAME>
        ```

        See the [Azure documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview) for how to generate a SAS token.

        <Note>
          Ensure the SAS token has read access to the Azure Blob Storage container containing the model.
        </Note>
      </Tab>

      <Tab title="Federated identity">
        Use Azure AD federated identity for passwordless authentication.

        **Setup:** Register an Azure AD application and add a federated credential with values provided by Fireworks during onboarding:

        * **Issuer:** `https://accounts.google.com`
        * **Subject:** `<FIREWORKS_GCP_SERVICE_ACCOUNT_UNIQUE_ID>`
        * **Audience:** `api://AzureADTokenExchange`

        Contact [inquiries@fireworks.ai](mailto:inquiries@fireworks.ai) if you need the exact subject identifier.

        Then grant **Storage Blob Data Reader** role to the application on your storage account.

        ```bash theme={null}
        firectl model create <MODEL_ID> https://<STORAGE_ACCOUNT>.blob.core.windows.net/<CONTAINER>/<PATH> \
          --azure-client-id <CLIENT_ID> \
          --azure-tenant-id <TENANT_ID>
        ```
      </Tab>
    </Tabs>
  </Tab>

  <Tab title="REST API">
    For programmatic uploads (automation, CI/CD pipelines, notebooks), use the Fireworks REST API.

    See [Upload via REST API](/models/uploading-custom-models-api) for the full pipeline and a ready-to-use Python example.
  </Tab>
</Tabs>

<Note>
  If you're uploading an embedding model, add the `--embedding` flag.
</Note>

<Note>
  To use Fireworks managed fine-tuning with an uploaded custom base model, provide the corresponding Hugging Face model URL when you create the model:

  ```bash theme={null}
  firectl model create <MODEL_ID> /path/to/files/ \
    --hugging-face-url https://huggingface.co/<ORG>/<MODEL>
  ```

  Fireworks uses this URL to infer the training renderer and determine whether a compatible training shape exists. Custom uploads without a Hugging Face URL can be deployed for inference, but they are not eligible for managed fine-tuning until the URL is set and tunability is refreshed. This refresh runs asynchronously about every 30 minutes, so `Tunable: true` may take up to 30 minutes to appear after you provide the URL. We are working to make this refresh faster.
</Note>

## Verifying your upload

After uploading, verify your model is ready to deploy:

```bash theme={null}
firectl model get accounts/<ACCOUNT_ID>/models/<MODEL_NAME>
```

Look for `State: READY` in the output. Once ready, you can create a deployment.

<Note>
  **Model states:**

  * `UPLOADING` – Files are being uploaded or awaiting validation (for REST API uploads)
  * `READY` – Model is validated and ready to deploy
  * `FAILED` – Upload or validation failed (check error details)

  If using the REST API and your model remains in `UPLOADING` state, you most likely haven't called the validation endpoint yet — REST uploads don't become deployable until they're validated. See [Why validation is a separate step](/models/uploading-custom-models-api#why-validation-is-a-separate-step) for details.
</Note>

## Deploying your model

Once your model shows `State: READY`, create a deployment:

```bash theme={null}
firectl deployment create accounts/<ACCOUNT_ID>/models/<MODEL_NAME> --wait
```

See the [On-demand deployments guide](/guides/ondemand-deployments) for configuration options like GPU types, autoscaling, and quantization.

## Publishing your model

By default, models are private to your account. Publish a model to make it available to other Fireworks users.

**When published:**

* Listed in the public model catalog
* Deployable by anyone with a Fireworks account
* Still hosted and controlled by your account

**Publish a model:**

```bash theme={null}
firectl model update <MODEL_ID> --public
```

**Unpublish a model:**

```bash theme={null}
firectl model update <MODEL_ID> --public=false
```

## Importing fine-tuned models

In addition to models you fine-tune on the Fireworks platform, you can also upload your own custom fine-tuned models as LoRA adapters.

<Note>
  Uploaded LoRA adapters can only be deployed to on-demand (dedicated) deployments. Serverless deployment is not supported.
</Note>

### Requirements

Your custom LoRA addon must contain the following files:

* `adapter_config.json` - The Hugging Face adapter configuration file
* `adapter_model.bin` or `adapter_model.safetensors` - The saved addon file

The `adapter_config.json` must contain the following fields:

* `r` - The number of LoRA ranks. Must be an integer between 4 and 64, inclusive
* `target_modules` - A list of target modules. Currently the following target modules are supported:
  * `q_proj`
  * `k_proj`
  * `v_proj`
  * `o_proj`
  * `up_proj` or `w1`
  * `down_proj` or `w2`
  * `gate_proj` or `w3`
  * `block_sparse_moe.gate`
  * `lm_head` only for base model families where FireAttention supports `lm_head` LoRA live-merge, including Llama, Mistral, Qwen2, Qwen2-VL, and Kimi K2.5.

Embedding target modules, such as `embed_tokens`, are not supported for LoRA addons.

Additional fields may be specified but are ignored.

### Customizing generation defaults with fireworks.json

For LoRA adapters, use a `fireworks.json` file to customize generation defaults. This is the recommended approach because adapters inherit configuration from their base model—modifying `generation_config.json` in the adapter folder won't work.

Add a `fireworks.json` file to the directory containing your adapter files:

```json fireworks.json theme={null}
{
  "defaults": {
    "stop": ["
", "
"],
    "max_tokens": 1024,
    "temperature": 0.7,
    "top_k": 50,
    "top_p": 0.9,
    "min_p": 0.0,
    "typical_p": 1.0,
    "frequency_penalty": 0.0,
    "presence_penalty": 0.0,
    "repetition_penalty": 1.0
  },
  "model_arch": null,
  "model_config_name": null,
  "has_lora": true,
  "has_teft": false
}
```

<AccordionGroup>
  <Accordion title="defaults options">
    These defaults are applied when the user doesn't specify values in their API request:

    | Field                | Type    | Example                    | Description                           |
    | -------------------- | ------- | -------------------------- | ------------------------------------- |
    | `stop`               | array   | `["<\|im_end\|>", "
"]` | Default stop sequences                |
    | `max_tokens`         | integer | `1024`                     | Default maximum tokens to generate    |
    | `temperature`        | float   | `0.7`                      | Default sampling temperature          |
    | `top_k`              | integer | `50`                       | Default top-k sampling                |
    | `top_p`              | float   | `0.9`                      | Default nucleus sampling probability  |
    | `min_p`              | float   | `0.0`                      | Default minimum probability threshold |
    | `typical_p`          | float   | `1.0`                      | Default typical sampling probability  |
    | `frequency_penalty`  | float   | `0.0`                      | Default frequency penalty             |
    | `presence_penalty`   | float   | `0.0`                      | Default presence penalty              |
    | `repetition_penalty` | float   | `1.0`                      | Default repetition penalty            |
  </Accordion>

  <Accordion title="Additional options">
    | Field               | Default | Description                                                                            |
    | ------------------- | ------- | -------------------------------------------------------------------------------------- |
    | `model_arch`        | null    | Model architecture (e.g., `"qwen2"`, `"llama"`). Usually auto-detected from base model |
    | `model_config_name` | null    | Model configuration name (e.g., `"4B"`). Usually auto-detected from base model         |
    | `has_lora`          | true    | Set to `true` for LoRA adapters                                                        |
    | `has_teft`          | false   | Set to `true` if using TEFT (Token-Efficient Fine-Tuning)                              |
  </Accordion>
</AccordionGroup>

<Note>
  All fields in `fireworks.json` are optional. Include only the fields you need to override.
</Note>

### Uploading the LoRA adapter

To upload a LoRA addon, run the following command. The MODEL\_ID is an arbitrary [resource ID](/getting-started/concepts#resource-names-and-ids) to refer to the model within Fireworks.

<Note>
  Only some base models support LoRA addons.
</Note>

```bash theme={null}
firectl model create <MODEL_ID> /path/to/files/ --base-model "accounts/fireworks/models/<BASE_MODEL_ID>"
```

## Next steps

<CardGroup>
  <Card title="Deploy your model" icon="rocket" href="/guides/ondemand-deployments">
    Configure GPU types, autoscaling, and optimization
  </Card>

  <Card title="Quantization" icon="compress" href="/models/quantization">
    Reduce serving costs with model quantization
  </Card>

  <Card title="Fine-tuning" icon="wand-magic-sparkles" href="/fine-tuning/finetuning-intro">
    Fine-tune models before deploying them
  </Card>
</CardGroup>


# Upload via REST API
Source: https://docs.fireworks.ai/models/uploading-custom-models-api

Programmatically upload custom models using the Fireworks REST API

Use the Fireworks REST API to upload custom models programmatically — ideal for automation, CI/CD pipelines, and notebooks. If you just want the quickest path from a terminal, use [`firectl`](/models/uploading-custom-models#uploading-your-model) instead.

## The upload pipeline

Uploading a model via the REST API is a four-step pipeline:

1. **Create the model** – Register a model object that references the files you're about to upload.
2. **Get upload URLs** – Request a signed URL for each file.
3. **Upload the files** – Upload each file directly to its signed URL.
4. **Validate the upload** – Finalize the upload and move the model to `READY`.

<Warning>
  Step 4 is required. Until you call `validateUpload`, your model stays in the `UPLOADING` state and **cannot be deployed** — even after every file has finished uploading.
</Warning>

## Why validation is a separate step

Transfer and validation are decoupled because uploads can arrive several ways (signed-URL upload, S3 import, Azure import) and the transfer runs asynchronously on Fireworks infrastructure. The `validateUpload` call is what:

* Confirms every expected file has landed and is intact.
* Parses your model config to detect architecture, precision, and context length.
* Transitions the model from `UPLOADING` to `READY` so it can be deployed.

Because the transfer is asynchronous, `validateUpload` returns `FAILED_PRECONDITION` while files are still landing. Poll it until it returns `200` — at that point the model is `READY`.

## Full example

This script runs the entire pipeline end to end. Set your API key, account ID, model ID, and the local folder containing your Hugging Face model files.

```python theme={null}
import os
import time
import requests

# --- Configuration ---
API_KEY = os.environ["FIREWORKS_API_KEY"]
ACCOUNT_ID = "your-account-id"
MODEL_ID = "my-custom-model"
MODEL_PATH = "/path/to/model/files/"  # folder of Hugging Face files: config.json, *.safetensors, tokenizer, ...

BASE_URL = "https://api.fireworks.ai/v1"
HEADERS = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}

files = [f for f in os.listdir(MODEL_PATH) if os.path.isfile(os.path.join(MODEL_PATH, f))]
file_sizes = {f: os.path.getsize(os.path.join(MODEL_PATH, f)) for f in files}

# --- Step 1: Create the model ---
requests.post(
    f"{BASE_URL}/accounts/{ACCOUNT_ID}/models",
    headers=HEADERS,
    json={
        "modelId": MODEL_ID,
        "model": {
            "kind": "HF_BASE_MODEL",
            "huggingFaceUrl": "https://huggingface.co/<ORG>/<MODEL>",
            "baseModelDetails": {
                "checkpointFormat": "HUGGINGFACE",
                "worldSize": 1,
                "huggingfaceFiles": files,
            },
        },
    },
).raise_for_status()
print(f"Created model {MODEL_ID}")

# --- Step 2: Get a signed upload URL for each file ---
resp = requests.post(
    f"{BASE_URL}/accounts/{ACCOUNT_ID}/models/{MODEL_ID}:getUploadEndpoint",
    headers=HEADERS,
    json={"filenameToSize": file_sizes, "enableResumableUpload": False},
)
resp.raise_for_status()
upload_urls = resp.json()["filenameToSignedUrls"]

# --- Step 3: Upload each file to its signed URL ---
for filename, url in upload_urls.items():
    size = file_sizes[filename]
    with open(os.path.join(MODEL_PATH, filename), "rb") as f:
        requests.put(
            url,
            data=f,
            headers={
                "Content-Type": "application/octet-stream",
                "x-goog-content-length-range": f"{size},{size}",
            },
        ).raise_for_status()
    print(f"Uploaded {filename}")

# --- Step 4: Validate — poll until the model is READY ---
validate_url = f"{BASE_URL}/accounts/{ACCOUNT_ID}/models/{MODEL_ID}:validateUpload"
while True:
    resp = requests.get(validate_url, headers=HEADERS)
    if resp.status_code == 200:
        print("Model validated and ready to deploy!")
        break
    if "FAILED_PRECONDITION" in resp.text:  # files still landing
        print("Files still finalizing, retrying in 30s...")
        time.sleep(30)
        continue
    resp.raise_for_status()  # any other error is fatal
```

<Note>
  Set `huggingFaceUrl` if this uploaded custom base model should be considered for Fireworks managed fine-tuning. Fireworks uses the Hugging Face URL to infer the training renderer and locate compatible training shapes. The tunability check runs asynchronously about every 30 minutes, so `Tunable: true` may take up to 30 minutes to appear after the URL is set.
</Note>

<Tip>
  Every step is safe to retry. `create` and `getUploadEndpoint` no-op if the model already exists (returning `ALREADY_EXISTS`), so an automated uploader that gets interrupted can simply re-run from the top and keep polling `validateUpload` until it succeeds.
</Tip>

## Troubleshooting: model stuck in `UPLOADING`

The most common cause is that `validateUpload` was never called — files transferring successfully does not move the model to `READY` on its own. Call the endpoint (and poll it):

```bash theme={null}
curl -X GET "https://api.fireworks.ai/v1/accounts/<ACCOUNT_ID>/models/<MODEL_ID>:validateUpload" \
  -H "Authorization: Bearer $FIREWORKS_API_KEY"
```

If it returns `FAILED_PRECONDITION`, files are still landing — wait and retry. If it keeps failing after the transfer should be complete, confirm that every [required file](/models/uploading-custom-models#required-files) was included.

## Next steps

1. **Verify**: `firectl model get accounts/<ACCOUNT_ID>/models/<MODEL_ID>` — look for `State: READY`.
2. **Deploy**: see the [Custom Models guide](/models/uploading-custom-models#deploying-your-model) for deployment instructions.


# Serverless Inference Overview
Source: https://docs.fireworks.ai/serverless/overview

How Serverless Inference works on Fireworks: serving paths, billing, request/response headers, prompt caching, model lifecycle, and when to choose Serverless Inference over On-demand

## What is Serverless Inference

Serverless Inference is multi-tenant inference for popular open models running on Fireworks-managed infrastructure. You point your client at `api.fireworks.ai`, send tokens, and pay only for what you use — no GPUs to size, no autoscaler to tune, no cold starts to wait through. Models eligible for Serverless carry the **Serverless** tag in the [model library](https://fireworks.ai/models). To make your first call, see the [Serverless quickstart](/getting-started/quickstart).

## Serverless products at a glance

Three serving paths run on the same Serverless framework. They share the same rate-limit policy, but route and price differently:

* **Standard** — the default serving path. No `service_tier` parameter needed.
* **Priority** — higher reliability during peak periods. Opt in by setting `service_tier: "priority"` on chat completions. Priced at a premium.
* **Fast** — high-speed deployments for latency-sensitive workloads. Selected by switching the `model` ID to a Fast variant (for example, `accounts/fireworks/routers/kimi-k2p6-fast`).

For usage examples and the full list of supported models, see [Serverless Serving Paths](/serverless/serving-paths). For pricing by serving path, see [Serverless pricing](/serverless/pricing).

## Billing

Serverless is priced per token. Three dimensions are billed:

* **Input tokens** — what you send to the model.
* **Cached input tokens** — input tokens served from prompt cache, discounted (default 50% of input on text and vision models, unless a model lists a different cached rate).
* **Generated tokens** — what the model produces.

Other things to know:

* The `usage` object in each response is the source of truth for what was billed (`prompt_tokens`, `completion_tokens`, `total_tokens`).
* **Batch inference** is billed at 50% of standard Serverless rates on both input and output. See [Batch inference](/guides/batch-inference).
* Your **spend tier** influences Serverless capacity caps. Your monthly spend limit is a separate account control. Higher spend tiers unlock higher TPM upper bounds. See [Account quotas](/guides/quotas_usage/account-quotas) and [Serverless rate limits](/serverless/rate-limits).

## Request and response headers

Headers a Serverless caller will set or read.

### Request headers

| Header                                     | Notes                                                                                                                                                   |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Authorization: Bearer $FIREWORKS_API_KEY` | Required for all requests.                                                                                                                              |
| `x-session-affinity`                       | Optional sticky-routing key. Pin repeated requests to the same replica to maximize prompt-cache hit rate. See [Prompt caching](/guides/prompt-caching). |

### Response headers

Fireworks sets the following on Serverless inference responses:

| Header                                           | What it tells you                                                                     |
| ------------------------------------------------ | ------------------------------------------------------------------------------------- |
| `fireworks-prompt-tokens`                        | Input tokens for the request.                                                         |
| `fireworks-cached-prompt-tokens`                 | Cached portion of the input. See [Prompt caching](/guides/prompt-caching#monitoring). |
| `X-Ratelimit-Limit-Tokens-Prompt`                | Your current Total Prompt Tokens per Second Limit.                                    |
| `X-Ratelimit-Limit-Tokens-Cache-Adjusted-Prompt` | Your current Total Uncached Prompt Tokens per Second Limit.                           |
| `X-Ratelimit-Limit-Tokens-Generated`             | Your current Total Generated Tokens per Second Limit.                                 |

<Tip>
  Streaming responses don't carry per-request perf headers. To get the same metrics in the streaming response body, set the `perf_metrics_in_response` parameter on the request. See [Querying text models](/guides/querying-text-models#usage--performance-tracking).
</Tip>

## Prompt caching

Prompt caching is on by default for every Serverless model. Cached input tokens are billed at the discounted rate (default 50% of input). Caching is replica-local, so to maximize hit rate you should route repeated prompts to the same replica — pass a stable identifier in `x-session-affinity` (or in the OpenAI `user` field) for each user or session whose prompts share a prefix.

For the full guide, including how to structure prompts for cache hits and how to read cache metrics, see [Prompt caching](/guides/prompt-caching).

## Serverless model lifecycle

Serverless models are managed by the Fireworks team and may be updated or deprecated as new models are released. We provide **at least 2 weeks advance notice** before removing any model, with longer notice periods for popular models based on usage.

For production workloads requiring long-term model stability, we recommend [on-demand deployments](/guides/ondemand-deployments), which give you full control over model versions and updates.

## Serverless vs On-demand

| When Serverless fits                                           | When On-demand fits                                                                   |
| -------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| Pay per token, only for what you use                           | Pay per GPU-hour for dedicated capacity                                               |
| You're using popular base models that Fireworks already hosts  | You're running custom base models or fine-tuned LoRA models (LoRA requires On-demand) |
| You don't want to manage scaling, replicas, or hardware sizing | You have custom latency requirements and want control over hardware and replicas      |

For dedicated infrastructure, see [On-demand deployments](/guides/ondemand-deployments).

## Next steps

<CardGroup>
  <Card title="Serverless quickstart" icon="rocket" href="/getting-started/quickstart">
    Make your first Serverless API call.
  </Card>

  <Card title="Serverless Serving Paths" icon="bolt" href="/serverless/serving-paths">
    Higher-reliability and higher-speed serving paths.
  </Card>

  <Card title="US-only Serverless" icon="flag-usa" href="/serverless/us-only-serverless">
    Dedicated US routers for inference residency.
  </Card>

  <Card title="Pricing" icon="circle-dollar" href="/serverless/pricing">
    Per-token rates for text, vision, embeddings, and Priority.
  </Card>

  <Card title="Rate limits" icon="gauge" href="/serverless/rate-limits">
    Adaptive TPM bounds and how the limit ramps with usage.
  </Card>

  <Card title="Prompt caching" icon="database" href="/guides/prompt-caching">
    How caching works and how to maximize hit rate.
  </Card>

  <Card title="On-demand deployments" icon="server" href="/guides/ondemand-deployments">
    Dedicated GPUs for predictable throughput and custom models.
  </Card>
</CardGroup>


# Serverless Pricing
Source: https://docs.fireworks.ai/serverless/pricing

Per-token serverless pricing for text, vision, and embedding models, including Priority and Fast serving paths

## Overview

Serverless inference is priced per token. For how Standard, Priority, and Fast serving paths work and how to select one, see [Serverless Serving Paths](/serverless/serving-paths).

Every text or vision request is billed across three dimensions:

* **Input tokens** — what you send to the model.
* **Cached input tokens** — input tokens served from [prompt cache](/guides/prompt-caching), priced lower.
* **Output tokens** — what the model generates.

Embeddings are billed only on input tokens.

## How pricing works

* Prices below are **per 1 million tokens** in US dollars.
* **Batch inference** is billed at **50% of serverless pricing** on both input and output. See [Batch inference](/guides/batch-inference).

## Text and vision models

Per-model pricing for headline models. Fast variants appear as adjacent rows. In each **Standard** or **Priority** cell, prices are **input / cached input / output** (USD per 1M tokens), in that order.

| Model                                                                                                             | Standard                  | Priority                      |
| ----------------------------------------------------------------------------------------------------------------- | ------------------------- | ----------------------------- |
| [Kimi K3](https://app.fireworks.ai/models/fireworks/kimi-k3)                                                      | \$3.00 / \$0.30 / \$15.00 | \$3.75 / \$0.375 / \$18.75    |
| [Kimi K3 Fast](https://app.fireworks.ai/models/fireworks/kimi-k3)                                                 | \$4.50 / \$0.45 / \$22.50 | —                             |
| [Kimi K3 US](https://app.fireworks.ai/models/fireworks/kimi-k3)                                                   | \$3.30 / \$0.33 / \$16.50 | \$4.125 / \$0.4125 / \$20.625 |
| [Kimi K2.7 Code](https://app.fireworks.ai/models/fireworks/kimi-k2p7-code)                                        | \$0.95 / \$0.19 / \$4.00  | \$1.425 / \$0.285 / \$6.00    |
| [Kimi K2.7 Code Fast](https://app.fireworks.ai/models/fireworks/kimi-k2p7-code)                                   | \$1.90 / \$0.38 / \$8.00  | —                             |
| [Kimi K2.6](https://app.fireworks.ai/models/fireworks/kimi-k2p6)                                                  | \$0.95 / \$0.16 / \$4.00  | \$1.50 / \$0.22 / \$6.00      |
| [Kimi K2.6 Fast](https://app.fireworks.ai/models/fireworks/kimi-k2p6)                                             | \$2.00 / \$0.30 / \$8.00  | —                             |
| [DeepSeek V4 Pro](https://app.fireworks.ai/models/fireworks/deepseek-v4-pro)                                      | \$1.74 / \$0.145 / \$3.48 | \$2.61 / \$0.218 / \$5.22     |
| [DeepSeek V4 Flash (0731)](https://app.fireworks.ai/models/fireworks/deepseek-v4-flash-0731)                      | \$0.14 / \$0.028 / \$0.28 | \$0.21 / \$0.042 / \$0.42     |
| [GLM 5.2](https://app.fireworks.ai/models/fireworks/glm-5p2)                                                      | \$1.40 / \$0.14 / \$4.40  | \$1.75 / \$0.18 / \$5.50      |
| [GLM 5.2 Fast](https://app.fireworks.ai/models/fireworks/glm-5p2)                                                 | \$2.10 / \$0.21 / \$6.60  | —                             |
| [GLM 5.2 Fast US](https://app.fireworks.ai/models/fireworks/glm-5p2)                                              | \$2.10 / \$0.21 / \$6.60  | —                             |
| [GLM 5.1](https://app.fireworks.ai/models/fireworks/glm-5p1)                                                      | \$1.40 / \$0.26 / \$4.40  | \$2.10 / \$0.39 / \$6.60      |
| [GLM 5.1 Fast](https://app.fireworks.ai/models/fireworks/glm-5p1)                                                 | \$2.80 / \$0.52 / \$8.80  | —                             |
| [Qwen 3.7 Plus](https://app.fireworks.ai/models/fireworks/qwen3p7-plus)                                           | \$0.40 / \$0.08 / \$1.60  | —                             |
| [MiniMax M3](https://app.fireworks.ai/models/fireworks/minimax-m3)                                                | \$0.30 / \$0.06 / \$1.20  | \$0.45 / \$0.09 / \$1.80      |
| [MiniMax M2.7](https://app.fireworks.ai/models/fireworks/minimax-m2p7)                                            | \$0.30 / \$0.06 / \$1.20  | \$0.45 / \$0.09 / \$1.80      |
| [OpenAI GPT OSS 120B](https://app.fireworks.ai/models/fireworks/gpt-oss-120b)                                     | \$0.15 / \$0.015 / \$0.60 | \$0.18 / \$0.018 / \$0.72     |
| [OpenAI GPT OSS 20B](https://app.fireworks.ai/models/fireworks/gpt-oss-20b)                                       | \$0.07 / \$0.035 / \$0.30 | —                             |
| [Muse Glimmer 30B](https://app.fireworks.ai/models/fireworks/muse-glimmer-30b)                                    | \$0.35 / \$0.04 / \$1.50  | \$0.525 / \$0.06 / \$2.25     |
| [NVIDIA Nemotron 3.5 Lightning 30B A3B](https://app.fireworks.ai/models/fireworks/nemotron-lightning-3p5-30b-a3b) | \$0.05 / \$0.01 / \$0.20  | —                             |
| [NVIDIA Nemotron 3 Ultra (Preview)](https://app.fireworks.ai/models/fireworks/nemotron-3-ultra-nvfp4)             | \$0.60 / \$0.12 / \$2.40  | —                             |

**—** in the Priority column means Priority is not available for that model. This pricing table is the source of truth for Priority availability.

## Other base models — by size and architecture

For any text or vision model not listed individually, pricing is set by parameter count and architecture. These size-based prices apply uniformly to input and output (no separate cached-input rate):

| Model                                                  | \$ / 1M tokens |
| ------------------------------------------------------ | -------------- |
| Less than 4B parameters                                | \$0.10         |
| 4B – 16B parameters                                    | \$0.20         |
| More than 16B parameters                               | \$0.90         |
| MoE up to 56B parameters (e.g. Mixtral 8x7B)           | \$0.50         |
| MoE 56.1B – 176B parameters (e.g. DBRX, Mixtral 8x22B) | \$1.20         |

## Embeddings

Embeddings are billed per 1M input tokens.

| Base model parameter count | \$ / 1M input tokens |
| -------------------------- | -------------------- |
| up to 150M                 | \$0.008              |
| 150M – 350M                | \$0.016              |
| Qwen3 8B                   | \$0.10               |

## Notes

* [US-only Serverless](/serverless/us-only-serverless) endpoints are priced at a 10% premium to the base model serverless prices, except GLM 5.2 Fast US, which matches global GLM 5.2 Fast pricing.
* For account-level controls (spend tiers, monthly spend limits, on-demand GPU quotas), see [Account quotas](/guides/quotas_usage/account-quotas).


# Serverless Rate Limits
Source: https://docs.fireworks.ai/serverless/rate-limits

Adaptive rate limits grow and shrink with your usage

<a />

When using Serverless, you may experience `429 Too Many Requests` or `503 Service Overloaded`. To avoid 429s, you need to stay below our adaptive rate limits. To reduce the likelihood of 503s, you can upgrade to [Priority tier](/serverless/serving-paths).

## What are your rate limits?

There are three metrics we use to rate limit accounts:

* **Total Prompt TPM** — input tokens per minute (cached + uncached).
* **Uncached Prompt TPM** — uncached input tokens per minute.
* **Generated TPM** — output tokens per minute.

**Default ceilings:** 21.6M Total Prompt TPM, 5.4M Uncached Prompt TPM, 216k Generated TPM (\~360k / \~90k / \~3.6k TPS). **Enforcement uses TPM**, not TPS.

Based on your usage, your adaptive limits will grow and shrink. If your traffic ramps up too quickly, you will get 429s.

<img alt="kimi-k2p6 usage and rate limits" />

Your current effective rate limits (described in tokens per second) are in the response headers `X-Ratelimit-Limit-Tokens-Prompt`, `X-Ratelimit-Limit-Tokens-Cache-Adjusted-Prompt`, and `X-Ratelimit-Limit-Tokens-Generated`.

Adaptive rate limits have an upper and lower bound. A higher account [Spending Tier](/guides/quotas_usage/account-quotas#spending-tiers) correlates with higher upper bound rate limits; **enterprise accounts** get higher upper bounds automatically.

## FAQ

<AccordionGroup>
  <Accordion title="Am I guaranteed successful responses up to my rate limit?">
    **No.** Staying within your rate limits does not guarantee that every request succeeds. When a deployment is busy, your traffic can still be **load shed**, and those responses are **`503 Service Overloaded`**. To **decrease the chance** of being load shed, you can use [Priority tier](/serverless/serving-paths), which is prioritized during high load.
  </Accordion>

  <Accordion title="How are rate limits scoped?">
    Rate limits are scoped **per account** and **per model**. **Fast** and **regular** model variants have **separate** limits. **Priority tier** and **regular** requests share the **same** rate limits for a given model.
  </Accordion>

  <Accordion title="What should I do first when I see 429s?">
    First, try **exponential backoff** when retrying.
  </Accordion>

  <Accordion title="How do I get higher limits sooner?">
    Reach out to [inquiries@fireworks.ai](mailto:inquiries@fireworks.ai) for a custom solution if either of these applies:

    * **You need higher than the defaults from day one.** Your launch traffic exceeds the starting limit and you can't wait for the adaptive ramp.
    * **You're ramping past the highest upper bound.** You are already at the highest account Spending Tier and the adaptive rate limits are not growing.
  </Accordion>
</AccordionGroup>


# Serverless Serving Paths
Source: https://docs.fireworks.ai/serverless/serving-paths

Standard, Priority, and Fast serving paths on Fireworks Serverless

Fireworks Serverless offers three serving paths:

* **Standard** is the default serving path. No `service_tier` parameter is needed.
* **Priority tier** is for workloads that require higher reliability during peak traffic.
* **Fast** is for workloads that require higher speeds.

## Priority tier

Priority tier is for workloads that require higher reliability during peak traffic periods, at a higher price point. Priority tier is prioritized above Standard traffic and is less likely to be load shed (503 server overloaded).

To use priority tier, set `service_tier` to `"priority"`. Supported on OpenAI-compatible chat completions and on the [Anthropic-compatible](/tools-sdks/anthropic-compatibility) `messages` API:

```bash theme={null}
curl https://api.fireworks.ai/inference/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FIREWORKS_API_KEY" \
  -d '{
    "model": "accounts/fireworks/models/glm-5p2",
    "service_tier": "priority",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
```

Priority tier is available on select models. Models and pricing are listed on the [Serverless pricing](/serverless/pricing) page.

## Fast

Fast is a high-speed serving path, useful for interactive applications that require fast response speeds, at a higher price point. Fast variants aim for **100+ tokens per second** of generated throughput. It is not a different model and the quality of the model remains the same.

Fast is available for select models. To use Fast, change the `model` ID as listed below.

| Model          | `model` ID                                  |
| -------------- | ------------------------------------------- |
| Kimi K2.6 Fast | `accounts/fireworks/routers/kimi-k2p6-fast` |
| GLM 5.2 Fast   | `accounts/fireworks/routers/glm-5p2-fast`   |
| GLM 5.1 Fast   | `accounts/fireworks/routers/glm-5p1-fast`   |

```bash theme={null}
curl https://api.fireworks.ai/inference/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FIREWORKS_API_KEY" \
  -d '{
    "model": "accounts/fireworks/routers/kimi-k2p7-code-fast",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
```

Pricing is listed on the [Serverless pricing](/serverless/pricing) page.

## Related

* [Serverless overview](/serverless/overview)
* [US-only Serverless](/serverless/us-only-serverless)
* [Serverless quickstart](/getting-started/quickstart)
* [Text models](/guides/querying-text-models)
* [Anthropic compatibility](/tools-sdks/anthropic-compatibility) — `service_tier` is supported on both OpenAI-compatible chat completions and the Anthropic `messages` API.


# US-only Serverless
Source: https://docs.fireworks.ai/serverless/us-only-serverless

Run Serverless inference exclusively in the US

US-only Serverless serves inference exclusively from the US, making it a good fit for compliance needs.

## Available models

| Model        | `model` ID                                   |
| ------------ | -------------------------------------------- |
| Kimi K3      | `accounts/fireworks/routers/kimi-k3-us`      |
| GLM 5.2 Fast | `accounts/fireworks/routers/glm-5p2-fast-us` |

## How to use it

Use a US router from the table above as the `model` ID.

```bash theme={null}
curl https://api.fireworks.ai/inference/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FIREWORKS_API_KEY" \
  -d '{
    "model": "accounts/fireworks/routers/kimi-k3-us",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
```

```python theme={null}
from fireworks import Fireworks

client = Fireworks()

response = client.chat.completions.create(
    model="accounts/fireworks/routers/kimi-k3-us",
    messages=[{"role": "user", "content": "Hello"}],
)

print(response.choices[0].message.content)
```

## Pricing

US-only endpoints are priced at a 10% premium to the base model serverless prices, except GLM 5.2 Fast US, which matches global GLM 5.2 Fast pricing. See [Serverless pricing](/serverless/pricing).

## Other regions

For EU-only Serverless, contact [sales](https://fireworks.ai/company/contact-us).

## Related

* [Serverless Serving Paths](/serverless/serving-paths)
* [Serverless pricing](/serverless/pricing)
* [Serverless overview](/serverless/overview)
* [Data Security](/guides/security_compliance/data_security)


# Structured Outputs
Source: https://docs.fireworks.ai/structured-responses/structured-response-formatting

Enforce output formats using JSON schemas or custom grammars

Structured outputs ensure model responses conform to your specified format, making them easy to parse and integrate into your application. Fireworks supports two methods: **JSON mode** (using JSON schemas) and **Grammar mode** (using custom BNF grammars).

<Info>
  New to structured outputs? Check out the [Serverless Quickstart](/getting-started/quickstart#structured-outputs-json-mode) for a quick introduction.
</Info>

## Quick Start

Force model output to conform to a [JSON schema](https://json-schema.org/):

```python theme={null}
import os
from openai import OpenAI
from pydantic import BaseModel

client = OpenAI(
    api_key=os.environ.get("FIREWORKS_API_KEY"),
    base_url="https://api.fireworks.ai/inference/v1"
)

# Define your schema
class Result(BaseModel):
    winner: str

# Make the request
response = client.chat.completions.create(
    model="accounts/fireworks/models/kimi-k2p5",
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "Result",
            "schema": Result.model_json_schema()
        }
    },
    messages=[{
        "role": "user",
        "content": "Who won the US presidential election in 2012? Reply in JSON format."
    }]
)

print(response.choices[0].message.content)
# Output: {"winner": "Barack Obama"}
```

<Tip>
  Include the schema in **both** your prompt and the `response_format` for best results. The model doesn't automatically "see" the schema—it's enforced during generation.
</Tip>

## Response Format Options

Fireworks supports two JSON mode variants:

* **`json_object`** – Force any valid JSON output (no specific schema)
* **`json_schema`** – Enforce a specific JSON schema (recommended)

<Warning>
  Always instruct the model to produce JSON in your prompt. Without this, the model may generate whitespace indefinitely until hitting token limits.
</Warning>

<AccordionGroup>
  <Accordion title="Using arbitrary JSON mode">
    To force JSON output without a specific schema:

    ```python theme={null}
    response = client.chat.completions.create(
        model="accounts/fireworks/models/kimi-k2p5",
        response_format={"type": "json_object"},
        messages=[{
            "role": "user",
            "content": "List the top 3 programming languages in JSON format."
        }]
    )
    ```

    This is similar to [OpenAI's JSON mode](https://platform.openai.com/docs/guides/text-generation/json-mode).
  </Accordion>

  <Accordion title="Important notes and limitations">
    **Token limits:** If `finish_reason="length"`, the response may be truncated and invalid JSON. Increase `max_tokens` if needed.

    **Completions API:** JSON mode works with both Chat Completions and Completions APIs.

    **Function calling:** When using [Tool Calling](/guides/function-calling), JSON mode is enabled automatically—these guidelines don't apply.
  </Accordion>
</AccordionGroup>

## JSON Schema Support

Fireworks supports most [JSON Schema 2020-12](https://json-schema.org/specification) constructs, and also accepts Draft-7 keyword aliases (`definitions`) for backward compatibility:

**Supported:**

* Types: `string`, `number`, `integer`, `boolean`, `object`, `array`, `null`
* Object constraints: `properties`, `required`, `additionalProperties`
* Array constraints: `items`
* Length/size constraints: `minLength`, `maxLength`, `minItems`, `maxItems`
* Regular expressions: `pattern`
* Composition: `anyOf`, `allOf`, `oneOf`
* Reuse: `$defs` (Draft 2020-12) and `definitions` (Draft 7), referenced via `$ref`
* Annotations: `$id`, `$schema`, `description`, `title`, `default`
* Recursive references (self-recursive `$ref`, mutually recursive `$defs`)

**Not yet supported:**

* External `$ref` URIs (HTTP/file). Only in-document JSON Pointer fragments are resolved.

<Tip>
  Fireworks automatically prevents hallucinated fields by treating schemas with `properties` as if `"unevaluatedProperties": false` is set.
</Tip>

<Note>
  `pattern` is enforced on a best-effort basis: regexes that can be compiled into the underlying grammar are enforced, and constructs that can't (e.g. lookahead/lookbehind) are safely ignored — the field falls back to an unconstrained string and the request still succeeds.
</Note>

### JSON Pointer references (`$ref`)

`$ref` follows the [RFC 6901 JSON Pointer](https://datatracker.ietf.org/doc/html/rfc6901) syntax used by JSON Schema 2020-12. Fireworks supports two pointer forms in the same document:

1. **Fully-qualified pointer (strict-spec).** A `$ref` whose fragment is a complete JSON Pointer to the definition's actual location, e.g. `#/properties/A/$defs/Foo`. Always resolves to that exact location, regardless of any other definitions in the document. This is fully portable across any conformant JSON Schema 2020-12 implementation.

2. **Bare shorthand `#/$defs/Foo` (Fireworks extension).** When `Foo` is placed inside a nested subschema (e.g. `properties.A.$defs.Foo`) instead of at the document root, Fireworks lifts every nested `$defs` / `definitions` entry into a root pool so that the bare pointer resolves. Most converters (OpenAPI, Instructor, LangChain `TypeAdapter`, pydantic v2) emit this shape, so the shorthand keeps them working without rewrites. **This shorthand is not portable** — strict validators will return `PointerToNowhere` for the same pointer.

**Name-collision precedence for the bare shorthand:**

* **Root-level `$defs[Foo]` always wins.** If `Foo` exists at both the document root and a nested location, `#/$defs/Foo` resolves to the root entry. The fully-qualified pointer to the nested entry continues to work and addresses the nested definition.
* **Nested vs. nested:** if `Foo` appears at multiple nested locations and there is no root entry, the first occurrence in document order wins (and the server logs a warning). If you need both, give them distinct names or hoist the shared definition to the root.
* **`$defs` and `definitions` are independent pools** — `#/$defs/Foo` and `#/definitions/Foo` address two different schemas, exactly as the spec defines.

<Tip>
  For maximum portability, place all `$defs` at the document root and reference them with `#/$defs/<Name>`. This is the shape OpenAI, pydantic v2, and the JSON Schema spec recommend, and works on every conformant validator.
</Tip>

<Accordion title="Example: recursive schema (linked list)">
  Self-recursive and mutually recursive schemas are supported. The example below is the canonical linked-list shape and works in both `response_format` and tool calling parameters:

  ```json theme={null}
  {
    "$defs": {
      "linked_list_node": {
        "type": "object",
        "properties": {
          "value": {"type": "integer"},
          "next": {
            "anyOf": [
              {"$ref": "#/$defs/linked_list_node"},
              {"type": "null"}
            ]
          }
        },
        "required": ["value"]
      }
    },
    "$ref": "#/$defs/linked_list_node"
  }
  ```
</Accordion>

<Accordion title="Example: pydantic v2 schemas with `$id`">
  Schemas emitted by `pydantic.BaseModel.model_json_schema()` typically carry a root `$id` annotation. This is treated as an identity annotation only — Fireworks never attempts to fetch the URI — and `#/$defs/...` references resolve correctly within the document:

  ```python theme={null}
  from pydantic import BaseModel

  class Person(BaseModel):
      name: str
      age: int

  class Response(BaseModel):
      person: Person

  # Response.model_json_schema() returns a schema with `$id`, `$defs`, and `$ref`.
  # Pass it directly into `response_format={"type": "json_schema", ...}`.
  ```
</Accordion>

<Accordion title="Advanced: Reasoning Model JSON Mode">
  Some models support generating structured JSON outputs alongside their reasoning process. The [Fireworks Python SDK](/tools-sdks/python-sdk) exposes the model's reasoning via the `reasoning_content` field, keeping it separate from the JSON output in the `content` field.

  <Warning>
    Using `response_format` with `json_schema` disables reasoning output. To get **both** reasoning and structured JSON, include the schema in your prompt instead and omit the `response_format` parameter.
  </Warning>

  #### Example Usage

  ````python theme={null}
  import json
  from fireworks import Fireworks
  from pydantic import BaseModel

  client = Fireworks()

  # Define the output schema
  class QAResult(BaseModel):
      question: str
      answer: str

  # Include the schema in the prompt to preserve reasoning
  schema = QAResult.model_json_schema()

  response = client.chat.completions.create(
      model="accounts/fireworks/models/kimi-k2p5",
      messages=[{
          "role": "user",
          "content": (
              "Who wrote 'Pride and Prejudice'?\n\n"
              f"Reply in JSON matching this schema:\n{json.dumps(schema, indent=2)}"
          )
      }],
      max_tokens=1000
  )

  # The Fireworks SDK separates reasoning into its own field
  reasoning = response.choices[0].message.reasoning_content
  content = response.choices[0].message.content

  # Strip markdown code fences if the model wraps the JSON
  json_str = content.strip()
  if json_str.startswith("```"):
      json_str = json_str.split("\n", 1)[1].rsplit("```", 1)[0].strip()

  # Parse into Pydantic model
  qa_result = QAResult.model_validate_json(json_str)

  if reasoning:
      print("Reasoning:", reasoning)
  print("Result:", qa_result.model_dump_json(indent=2))
  ````

  <Tip>
    If you don't need reasoning and just want guaranteed schema-conformant JSON, use the `response_format` parameter as shown in the [Quick Start](#quick-start) above. The `response_format` approach enforces the schema during generation, eliminating the need to parse the JSON yourself.
  </Tip>

  #### Use Cases

  Reasoning mode is useful for:

  * **Debugging**: Understanding why the model generated specific outputs
  * **Auditing**: Documenting the decision-making process
  * **Complex tasks**: Scenarios where the reasoning is as valuable as the final answer

  See the [Reasoning guide](/guides/reasoning) for more on working with reasoning models.
</Accordion>

## Grammar Mode

For advanced use cases where JSON isn't sufficient, use [Grammar mode](/structured-responses/structured-output-grammar-based) to constrain outputs using custom BNF grammars. Grammar mode is ideal for:

* **Classification tasks** – Limit output to a predefined list of options
* **Language-specific output** – Force output in specific languages or character sets
* **Custom formats** – Define arbitrary output structures beyond JSON

[Learn more about Grammar mode →](/structured-responses/structured-output-grammar-based)

## Related features

Check out [Tool Calling](/guides/function-calling) for multi-turn capabilities and routing across multiple schemas.


# Anthropic compatibility
Source: https://docs.fireworks.ai/tools-sdks/anthropic-compatibility

Use Anthropic SDKs with Fireworks, and understand the supported surface for the Anthropic-compatible Messages API.

You can use the [Anthropic Python SDK](https://github.com/anthropics/anthropic-sdk-python) or [Anthropic TypeScript SDK](https://github.com/anthropics/anthropic-sdk-typescript) to interact with Fireworks, making it easy to migrate applications that already use Anthropic's Messages API.

Fireworks exposes an Anthropic-compatible endpoint at `POST /v1/messages`.

## Quickstart

Install the Anthropic SDK for your language:

<Tabs>
  <Tab title="Python">
    ```bash theme={null}
    pip install anthropic
    ```
  </Tab>

  <Tab title="JavaScript / TypeScript">
    ```bash theme={null}
    npm install @anthropic-ai/sdk
    ```
  </Tab>
</Tabs>

Then make your first request:

<CodeGroup>
  ```python Python theme={null}
  import os
  import anthropic

  client = anthropic.Anthropic(
      api_key=os.environ["FIREWORKS_API_KEY"],
      base_url="https://api.fireworks.ai/inference",
  )

  response = client.messages.create(
      model="accounts/fireworks/models/kimi-k2p5",
      max_tokens=256,
      messages=[
          {"role": "user", "content": "Say hello in Spanish. Reply in one word."}
      ],
  )

  print(response.content[0].text)
  ```

  ```javascript JavaScript theme={null}
  import Anthropic from "@anthropic-ai/sdk";

  const client = new Anthropic({
    apiKey: process.env.FIREWORKS_API_KEY,
    baseURL: "https://api.fireworks.ai/inference",
  });

  const response = await client.messages.create({
    model: "accounts/fireworks/models/kimi-k2p5",
    max_tokens: 256,
    messages: [
      { role: "user", content: "Say hello in Spanish. Reply in one word." },
    ],
  });

  console.log(response.content[0].text);
  ```

  ```bash cURL theme={null}
  curl --request POST \
    --url https://api.fireworks.ai/inference/v1/messages \
    --header "Authorization: Bearer $FIREWORKS_API_KEY" \
    --header "Content-Type: application/json" \
    --data '{
      "model": "accounts/fireworks/models/kimi-k2p5",
      "max_tokens": 256,
      "messages": [
        {
          "role": "user",
          "content": "Say hello in Spanish. Reply in one word."
        }
      ]
    }'
  ```
</CodeGroup>

<Note>
  The base URL for the Anthropic SDK is `https://api.fireworks.ai/inference` (without the `/v1` suffix). The SDK appends `/v1/messages` automatically.
</Note>

## Usage

Use the Anthropic SDK as you normally would. Set `model` to a Fireworks model resource name, such as `accounts/fireworks/models/kimi-k2p5`.

The [Serverless Quickstart](/getting-started/quickstart) includes Anthropic SDK examples for common use cases:

* [Messages](/getting-started/quickstart)
* [Streaming](/getting-started/quickstart#streaming-responses)
* [Function calling](/getting-started/quickstart#function-calling)
* [Structured outputs](/getting-started/quickstart#structured-outputs-json-mode)
* [Reasoning](/getting-started/quickstart#reasoning)
* [Vision](/getting-started/quickstart#vision-models)

## API compatibility

### Supported endpoint

Fireworks supports the Anthropic [`/v1/messages`](/api-reference/anthropic-messages) endpoint, including non-streaming and streaming (SSE) responses.

### Deployment support

Anthropic compatibility is supported for serverless and on-demand deployments. Requests must go through `api.fireworks.ai/inference` (direct route endpoints are not supported for this surface).

### Differences from Anthropic

The following parameters and fields are handled differently or are not supported:

* **`model`**: Must be a Fireworks model identifier (for example, `accounts/fireworks/models/deepseek-v3p2`) instead of an Anthropic model name. See the [Fireworks Model Library](https://app.fireworks.ai/models) for available models.
* **`max_tokens`**: Optional on Fireworks (required on Anthropic).
* **`anthropic-version` header**: Not required. Fireworks ignores this header.
* **`usage` field**: Included in both non-streaming and streaming responses. See [Token usage](#token-usage) for details.
* **`service_tier`**: Supported. Set `service_tier: "priority"` to opt into [Priority tier](/serverless/serving-paths).
* **`inference_geo`**: Not supported.

### Reasoning effort mapping

When you use the `thinking` parameter with `output_config.effort`, Anthropic effort values map to Fireworks [`reasoning_effort`](/api-reference/post-chatcompletions#body-reasoning-effort-one-of-0):

| Anthropic effort | Fireworks mapping |
| ---------------- | ----------------- |
| `low`            | `low`             |
| `medium`         | `medium`          |
| `high`           | `high`            |
| `max`            | `high`            |

<Note>
  The `adaptive` thinking type is not supported yet.
</Note>

For more details on reasoning, including interleaved thinking with tool use, see the [Reasoning guide](/guides/reasoning).

### Tool search and deferred tool loading

Tool definition schemas usually live at the top of a model's chat template, ahead of the conversation. Carrying every schema on every turn bloats that prefix and destabilizes the prompt cache for clients with large tool sets. Fireworks supports the [tool search](https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool) pattern for on-demand tool discovery—used by [Claude Code's MCP tool search](https://code.claude.com/docs/en/mcp#scale-with-mcp-tool-search) and the [Agent SDK's tool search](https://code.claude.com/docs/en/agent-sdk/tool-search)—to lazy-load schemas instead.

* **`defer_loading`**: Tools marked `defer_loading: true` are omitted from the request's tool definitions when a tool-search tool is present. Rather than placing every schema at the top of the template up front, the deferred schemas are loaded lazily through tool results once the model identifies which tools it needs.
* **`tool_reference` expansion**: When a tool result returns `tool_reference` blocks (the payload a tool-search call emits), each reference is expanded inline into the referenced tool's schema within the tool-result message. That makes the newly loaded schema visible to the model through the conversation, so it can produce tool calls in line with that schema—without the client re-sending the full `tools` array and shifting the prefix.

This covers both Anthropic-native `tool_search_tool_*` tool names and clients that name their discovery tool `ToolSearch` (for example, Claude Code).

<Note>
  Fireworks translates the **client-side** tool-search discovery and deferred-loading wire format only. Anthropic's **server-side** tool search and server-side tool use—where the provider executes the search and tool calls on its side—are not supported. Server-side execution of the other server tool families (web search, code execution, memory, web fetch) is likewise not supported; see [Unsupported features](#unsupported-features).
</Note>

<Note>
  An explicitly forced `tool_choice` naming a deferred tool overrides the drop: the forced tool stays callable in the request's tool definitions so the forced choice validates.
</Note>

### Unsupported features

The following Anthropic features are not available on Fireworks:

* **Server tools**: Server-side execution of tool families such as code execution, memory, web fetch, and web search is not supported. Tool search discovery and deferred tool loading are supported — see [Tool search and deferred tool loading](#tool-search-and-deferred-tool-loading).
* **Server-tool metadata**: Fields such as `caller` and `container` are not supported.
* **Tool schema fields**: `eager_input_streaming`, `cache_control`, `allowed_callers`, and `input_examples` are not supported.
* **`server_tool_use`**: Not included in usage tracking.
* **`speed`**: The `output_config.speed` option is not supported yet.

### Fireworks extensions

The following Fireworks-specific extension is available on the Anthropic-compatible endpoint:

* **`raw_output`**: A request parameter (boolean) that returns low-level details of what the model sees, including formatted prompts and function call data.

## Token usage

Token usage (`input_tokens` and `output_tokens`) is included in both non-streaming and streaming responses.

### Non-streaming

For non-streaming requests, usage is returned on the response object:

<CodeGroup>
  ```python Python theme={null}
  response = client.messages.create(
      model="accounts/fireworks/models/kimi-k2p5",
      max_tokens=256,
      messages=[{"role": "user", "content": "Say hello"}],
  )

  print(f"Input tokens:  {response.usage.input_tokens}")
  print(f"Output tokens: {response.usage.output_tokens}")
  ```

  ```javascript JavaScript theme={null}
  const response = await client.messages.create({
    model: "accounts/fireworks/models/kimi-k2p5",
    max_tokens: 256,
    messages: [{ role: "user", content: "Say hello" }],
  });

  console.log(`Input tokens:  ${response.usage.input_tokens}`);
  console.log(`Output tokens: ${response.usage.output_tokens}`);
  ```
</CodeGroup>

### Streaming

For streaming requests, token usage is included in the final `message_delta` event:

<CodeGroup>
  ```python Python theme={null}
  stream = client.messages.create(
      model="accounts/fireworks/models/kimi-k2p5",
      max_tokens=256,
      messages=[{"role": "user", "content": "Say hello"}],
      stream=True,
  )

  for event in stream:
      if event.type == "message_delta":
          print(f"Input tokens:  {event.usage.input_tokens}")
          print(f"Output tokens: {event.usage.output_tokens}")
  ```

  ```javascript JavaScript theme={null}
  const stream = client.messages.stream({
    model: "accounts/fireworks/models/kimi-k2p5",
    max_tokens: 256,
    messages: [{ role: "user", content: "Say hello" }],
  });

  for await (const event of stream) {
    if (event.type === "message_delta") {
      console.log(`Input tokens:  ${event.usage.input_tokens}`);
      console.log(`Output tokens: ${event.usage.output_tokens}`);
    }
  }
  ```
</CodeGroup>

<Note>
  There is only one `message_delta` event per stream (the last event before `message_stop`), and it always contains the actual token counts. The `message_start` event also includes a `usage` field, but its values are always `0` and should be ignored for metering purposes.
</Note>

## Next steps

<CardGroup>
  <Card title="Quickstart" href="/getting-started/quickstart" icon="rocket">
    Get started with your first API call
  </Card>

  <Card title="Reasoning" href="/guides/reasoning" icon="brain">
    Use reasoning with thinking models
  </Card>

  <Card title="API reference" href="/api-reference/anthropic-messages" icon="brackets-curly">
    Full Anthropic Messages API reference
  </Card>
</CardGroup>


# firectl account get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/account-get

Prints information about an account.

```
firectl account get [flags]
```

### Examples

```
firectl account get
firectl account get my-account
firectl account get accounts/my-account
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl account list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/account-list

Prints all accounts the current signed-in user has access to.

```
firectl account list [flags]
```

### Examples

```
firectl account list
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl api-key create
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/api-key-create

Creates an API key for the signed in user or a specified service account user.

```
firectl api-key create [flags]
```

### Examples

```
firectl api-key create
firectl api-key create --service-account=my-service-account
firectl api-key create --key-name="Production Key" --service-account=ci-bot
firectl api-key create --key-name="Temporary Key" --expire-time="2025-12-31 23:59:59"
```

### Flags

```
      --dry-run                  Print the request proto without running it.
      --expire-time string       If specified, the time at which the API key will automatically expire. Specified in YYYY-MM-DD[ HH:MM:SS] format.
  -h, --help                     help for create
      --key-name string          The name of the key to be created. Defaults to "default"
  -o, --output Output            Set the output format to "text", "json", or "flag". (default text)
      --service-account string   Admin only: Create API key for the specified service account user
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl api-key delete
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/api-key-delete

Deletes an API key.

```
firectl api-key delete [flags]
```

### Examples

```
firectl api-key delete key-id
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for delete
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl api-key get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/api-key-get

Prints information about an API key.

```
firectl api-key get [flags]
```

### Examples

```
firectl api-key get <key-id>
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl api-key list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/api-key-list

Prints all API keys for the signed in user.

```
firectl api-key list [flags]
```

### Examples

```
firectl api-key list
firectl api-key list --all-users
```

### Flags

```
      --all-users           Admin only: list API keys for all users in the account
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl audit-logs list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/audit-logs-list

Lists audit logs for the signed in user.

### Usage

Lists audit logs for the signed in user with filtering support.

```
firectl audit-logs list [flags]
```

### Examples

```
# List all audit logs from the past 30 days
firectl audit-logs list

# Filter by resource (exact or substring match)
firectl audit-logs list --filter 'resource="accounts/my-account/deployments/abc123"'
firectl audit-logs list --filter 'resource="accounts/my-account/models/model123"'
firectl audit-logs list --filter 'resource:model123"'

# Filter by message (exact or substring match)
firectl audit-logs list --filter 'message="CreateDeployment"'
firectl audit-logs list --filter 'message:"Create"'

# Filter by user email
firectl audit-logs list --filter 'email="user@example.com"'

# List logs from a specific date range with filters
firectl audit-logs list --start 2025-01-01 --end 2025-01-02

# Combine multiple filters
firectl audit-logs list --start 2025-01-01 --filter 'resource:"deployment-id"'
```

### Flags

```
      --end string          The end date for audit logs in YYYY-MM-DD format
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
      --start string        The start date for audit logs in YYYY-MM-DD format. If unspecified, the default is 30 days before now.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# Authentication
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/authentication

Authentication for access to your account

### Signing in

Users using Google SSO can run:

```
firectl signin
```

If you are using [custom SSO](/accounts/sso), also specify the account ID:

```
firectl signin my-enterprise-account
```

### Authenticate with API Key

To authenticate with an API key, append `--api-key` to any firectl command.

```
firectl --api-key API_KEY <command>
```

To persist the API key for all subsequent commands, run:

```
firectl set-api-key API_KEY
```


# firectl batch-inference-job create
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/batch-inference-job-create

Creates a batch inference job.

```
firectl batch-inference-job create [flags]
```

### Examples

```
firectl batch-inference-job create --input-dataset-id my-dataset --output-dataset-id my-output-dataset --model my-model \
			--job-id my-job --max-tokens 1024 --temperature 0.7 --top-p 0.9 --top-k 50 --n 2 --precision FP16 \
			--extra-body '{"stop": ["\n"], "presence_penalty": 0.5}'
```

### Flags

```
      --job-id string              The ID of the batch inference job. If not set, it will be autogenerated.
      --display-name string        The display name of the batch inference job.
  -m, --model string               The model to use for inference.
  -d, --input-dataset-id string    The input dataset ID.
  -x, --output-dataset-id string   The output dataset ID. If not provided, a default one will be generated.
      --continue-from string       Continue from an existing batch inference job (by job ID or resource name).
      --max-tokens int32           Maximum number of tokens to generate per response.
      --temperature float32        Sampling temperature (typically between 0 and 2).
      --top-p float32              Top-p sampling parameter (typically between 0 and 1).
      --top-k int32                Top-k sampling parameter, limits the token selection to the top k tokens.
      --n int32                    Number of response candidates to generate per input.
      --extra-body string          Additional inference parameters as a JSON string (e.g., '{"stop": ["\n"]}').
      --precision string           The precision with which the model should be served. If not specified, a suitable default will be chosen based on the model.
      --quiet                      If set, only errors will be printed.
      --dry-run                    Print the request proto without running it.
  -o, --output Output              Set the output format to "text", "json", or "flag". (default text)
  -h, --help                       help for create
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl batch-inference-job delete
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/batch-inference-job-delete

Deletes a batch inference job.

```
firectl batch-inference-job delete [flags]
```

### Examples

```
firectl batch-inference-job delete my-batch-job
firectl batch-inference-job delete accounts/my-account/batchInferenceJobs/my-batch-job
```

### Flags

```
      --dry-run                 Print the request proto without running it.
  -h, --help                    help for delete
  -o, --output Output           Set the output format to "text", "json", or "flag". (default text)
      --wait                    Wait until the batch inference job is deleted.
      --wait-timeout duration   Maximum time to wait when using --wait flag. (default 30m0s)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl batch-inference-job get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/batch-inference-job-get

Retrieves information about a batch inference job.

```
firectl batch-inference-job get [flags]
```

### Examples

```
firectl batch-inference-job get my-batch-job
firectl batch-inference-job get accounts/my-account/batchInferenceJobs/my-batch-job
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl batch-inference-job list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/batch-inference-job-list

Lists all batch inference jobs in an account.

```
firectl batch-inference-job list [flags]
```

### Examples

```
firectl batch-inference-job list
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl billing export-metrics
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/billing-export-metrics

Exports billing metrics

```
firectl billing export-metrics [flags]
```

### Examples

```
firectl billing export-metrics
```

### Flags

```
      --end-time string     The end time (exclusive).
      --filename string     The file name to export to. (default "billing_metrics.csv")
  -h, --help                help for export-metrics
      --start-time string   The start time (inclusive).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl billing get-usage
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/billing-get-usage

Prints account usage and rated costs for a time range.

```
firectl billing get-usage [flags]
```

### Examples

```
firectl billing get-usage --start-time 2026-05-01 --end-time 2026-06-01
firectl billing get-usage --start-time 2026-05-01 --end-time 2026-06-01 --usage-type serverless --group-by model_name
firectl billing get-usage --start-time 2026-05-01 --end-time 2026-06-01 --usage-type dedicated-deployment --filter deployment_name=accounts/my-account/deployments/my-deployment
```

### Flags

```
      --account-costs-only   Only print account-level cumulative costs for the date range.
      --dry-run              Print the request proto without running it.
      --end-time string      The end time (exclusive), as YYYY-MM-DD or 'YYYY-MM-DD hh:mm:ss'.
      --filter stringArray   Filter in key=value form. Can be repeated; repeated values for the same key are OR'ed.
      --group-by strings     Dimension to group usage by. Can be repeated; valid dimensions are server-validated.
  -h, --help                 help for get-usage
  -o, --output Output        Set the output format to "text", "json", or "flag". (default text)
      --start-time string    The start time (inclusive), as YYYY-MM-DD or 'YYYY-MM-DD hh:mm:ss'.
      --timezone string      IANA timezone identifier for daily aggregation, for example "America/Los_Angeles". Defaults to UTC.
      --usage-type string    Usage type to query: all, serverless, or dedicated-deployment.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl billing list-invoices
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/billing-list-invoices

Prints information about invoices.

```
firectl billing list-invoices [flags]
```

### Examples

```
firectl billing list-invoices
```

### Flags

```
  -h, --help           help for list-invoices
      --show-pending   If true, only pending invoices are shown.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl billing notification-settings
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/billing-notification-settings

Manage notification settings.

### Flags

```
  -h, --help   help for notification-settings
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl billing notification-settings get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/billing-notification-settings-get

Get notification settings for an account.

```
firectl billing notification-settings get [flags]
```

### Examples

```
firectl billing notification-settings get
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl billing notification-settings update
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/billing-notification-settings-update

Update notification settings for an account.

```
firectl billing notification-settings update [flags]
```

### Examples

```
firectl billing notification-settings update --monthly-spend-usd-thresholds=500,800
firectl billing notification-settings update --monthly-spend-usd-thresholds=500 --monthly-spend-usd-thresholds=800
firectl billing notification-settings update --monthly-spend-usd-thresholds=""
```

### Flags

```
      --dry-run                                   Print the request proto without running it.
  -h, --help                                      help for update
      --monthly-spend-usd-thresholds int64Slice   Spend alert thresholds in whole USD (e.g., 500,800). Use "" to clear. (default [])
  -o, --output Output                             Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl credit-redemption list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/credit-redemption-list

Lists credit code redemptions for the current account.

```
firectl credit-redemption list [flags]
```

### Examples

```
firectl credit-redemption list
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl credit-redemption redeem
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/credit-redemption-redeem

Redeems a credit code.

```
firectl credit-redemption redeem [flags]
```

### Examples

```
firectl credit-redemption redeem PROMO2025
```

### Flags

```
  -h, --help   help for redeem
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl dataset create
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/dataset-create

Creates and uploads a dataset.

```
firectl dataset create [flags]
```

### Examples

```
firectl dataset create my-dataset /path/to/dataset.jsonl
firectl dataset create --trace-from-model-id model_abc --format chat --date 2024-01-10 my-dataset
firectl dataset create my-dataset --external-url gs://bucket-name/object-name
```

### Flags

```
      --display-name string    The display name of the dataset.
      --dry-run                Print the request proto without running it.
      --end-time string        The end time for which to trace data (format: YYYY-MM-DD). Only specify for traced dataset.
      --eval-protocol-output   If true, the dataset is in eval protocol output format.
      --external-url string    The GCS URI that points to the dataset file.
      --filter string          Filter condition to apply to the source dataset.
  -h, --help                   help for create
  -o, --output Output          Set the output format to "text", "json", or "flag". (default text)
      --quiet                  If true, does not print the upload progress bar.
      --source string          Source dataset ID to filter from.
      --start-time string      The start time for which to trace data (format: YYYY-MM-DD). Only specify for traced dataset.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl dataset delete
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/dataset-delete

Deletes a dataset.

```
firectl dataset delete [flags]
```

### Examples

```
firectl dataset delete my-dataset
firectl dataset delete accounts/my-account/datasets/my-dataset
```

### Flags

```
      --dry-run                 Print the request proto without running it.
  -h, --help                    help for delete
  -o, --output Output           Set the output format to "text", "json", or "flag". (default text)
      --wait                    Wait until the dataset is deleted.
      --wait-timeout duration   Maximum time to wait when using --wait flag. (default 30m0s)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl dataset download
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/dataset-download

Downloads a dataset to a local directory.

```
firectl dataset download [flags]
```

### Examples

```
# Download a single dataset
firectl dataset download my-dataset --output-dir /path/to/download

# Download entire lineage chain (only for batch inference continuation jobs)
firectl dataset download my-dataset --download-lineage --output-dir /path/to/download
```

### Flags

```
      --download-lineage    If true, downloads entire lineage chain (all related datasets)
  -h, --help                help for download
      --output-dir string   Directory to download dataset files to (default ".")
      --quiet               If true, does not show download progress
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl dataset get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/dataset-get

Prints information about a dataset.

```
firectl dataset get [flags]
```

### Examples

```
firectl dataset get my-dataset
firectl dataset get accounts/my-account/datasets/my-dataset
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl dataset list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/dataset-list

Prints all datasets in an account.

```
firectl dataset list [flags]
```

### Examples

```
firectl dataset list
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl dataset update
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/dataset-update

Updates a dataset.

```
firectl dataset update [flags]
```

### Examples

```
firectl dataset update my-dataset
firectl dataset update accounts/my-account/datasets/my-dataset
```

### Flags

```
      --display-name string   The display name of the dataset.
      --dry-run               Print the request proto without running it.
  -h, --help                  help for update
  -o, --output Output         Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl deployed-model get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/deployed-model-get

Prints information about a deployed model.

```
firectl deployed-model get [flags]
```

### Examples

```
firectl deployed-model get my-deployed-model
firectl deployed-model get accounts/my-account/deployedModels/my-deployed-model
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl deployed-model list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/deployed-model-list

Prints all deployed models in the account.

```
firectl deployed-model list [flags]
```

### Examples

```
firectl deployed-model list
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl deployed-model update
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/deployed-model-update

Update a deployed model.

```
firectl deployed-model update [flags]
```

### Examples

```
firectl deployed-model update my-deployed-model
firectl deployed-model update accounts/my-account/deployedModels/my-deployed-model
```

### Flags

```
      --default #<deployment>   If true, this is the default deployment when querying this model without the #<deployment> suffix.
      --description string      Description of the deployed model. Must be fewer than 1000 characters long.
      --display-name string     Human-readable name of the deployed model. Must be fewer than 64 characters long.
      --dry-run                 Print the request proto without running it.
  -h, --help                    help for update
  -o, --output Output           Set the output format to "text", "json", or "flag". (default text)
      --public                  If true, the deployed model will be publicly reachable.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl deployment create
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/deployment-create

Creates a new deployment.

```
firectl deployment create [flags]
```

### Examples

```
firectl deployment create falcon-7b
firectl deployment create accounts/fireworks/models/falcon-7b
firectl deployment create falcon-7b --file=/path/to/deployment-config.json
firectl deployment create falcon-7b --deployment-shape=falcon-7b-shape
```

### Flags

```
      --accelerator-count int32                  The number of accelerators to use per replica.
      --accelerator-type string                  The type of accelerator to use. Must be one of {NVIDIA_A100_80GB, NVIDIA_H100_80GB, NVIDIA_H200_141GB, AMD_MI300X_192GB}
  -c, --cluster-id string                        The Fireworks cluster ID.
      --deployment-id string                     The ID of the deployment. If not specified, a random ID will be generated.
      --deployment-shape string                  The deployment shape to use for this deployment.
      --deployment-template string               The deployment template to use.
      --description string                       Description of the deployment.
      --direct-route-api-keys stringArray        The API keys for the direct route. Only available to enterprise accounts.
      --direct-route-type string                 If set, this deployment will expose an endpoint that bypasses our API gateway. Must be one of {INTERNET, GCP_PRIVATE_SERVICE_CONNECT, AWS_PRIVATELINK}. Only available to enterprise accounts.
      --disable-speculative-decoding             If true, speculative decoding is disabled.
      --display-name string                      Human-readable name of the deployment. Must be fewer than 64 characters long.
      --draft-model string                       The draft model to use for speculative decoding. If the model is under your account, you can specify the model ID. If the model is under another account, you can specify the full resource name (e.g. accounts/other-account/models/falcon-7b).
      --draft-token-count int32                  The number of tokens to generate per step for speculative decoding.
      --dry-run                                  Print the request proto without running it.
      --enable-addons                            If true, enable addons for this deployment.
      --enable-session-affinity                  If true, does sticky routing based on the 'user' field. Only available to enterprise accounts.
      --file string                              Path to a JSON configuration file containing deployment settings.
  -h, --help                                     help for create
      --load-targets Map                         Map of autoscaling load metric names to their target utilization factors.
      --long-prompt                              Whether this deployment is optimized for long prompts.
      --max-context-length int32                 The maximum context length supported by the model (context window). If set to 0 or not specified, the model's default maximum context length will be used.
      --max-replica-count int32                  Maximum number of replicas for the deployment. If min-replica-count > 0 defaults to 0, otherwise defaults to 1.
      --max-with-revocable-replica-count int32   Maximum number of replicas including revocable replicas. Any replicas in excess of max_replica_count are revocable.
      --min-replica-count int32                  Minimum number of replicas for the deployment. If min-replica-count < max-replica-count the deployment will automatically scale between the two replica counts based on load.
      --ngram-speculation-length int32           The length of previous input sequence to be considered for N-gram speculation.
  -o, --output Output                            Set the output format to "text", "json", or "flag". (default text)
      --precision string                         The precision with which the model is served. If specified, must be one of {FP8, FP16, FP4, BF16}.
      --region string                            Placement: 'global', region group (us), or specific region (us-iowa-1).
      --scale-down-window duration               The duration the autoscaler will wait before scaling down a deployment after observing decreased load. Default is 10m.
      --scale-to-zero-window duration            The duration after which there are no requests that the deployment will be scaled down to zero replicas, if min-replica-count is 0. Default 1h.
      --scale-up-window duration                 The duration the autoscaler will wait before scaling up a deployment after observing increased load. Default is 30s.
      --validate-only                            If true, this will not create the deployment, but will return the deployment that would be created.
      --wait                                     Wait until the deployment is ready.
      --wait-timeout duration                    Maximum time to wait when using --wait flag. (default 1h0m0s)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl deployment delete
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/deployment-delete

Deletes a deployment.

```
firectl deployment delete [flags]
```

### Examples

```
firectl deployment delete my-deployment
firectl deployment delete accounts/my-account/deployments/my-deployment
```

### Flags

```
      --dry-run                 Print the request proto without running it.
      --hard                    Hard delete the deployment
  -h, --help                    help for delete
      --ignore-checks           Skip checking if the deployment is in use before deleting
  -o, --output Output           Set the output format to "text", "json", or "flag". (default text)
      --wait                    Wait until the deployment is deleted.
      --wait-timeout duration   Maximum time to wait when using --wait flag. (default 1h0m0s)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl deployment get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/deployment-get

Prints information about a deployment.

```
firectl deployment get [flags]
```

### Examples

```
firectl deployment get my-deployment
firectl deployment get accounts/my-account/deployments/my-deployment
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl deployment list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/deployment-list

Prints all deployments in the account.

```
firectl deployment list [flags]
```

### Examples

```
firectl deployment list
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
      --show-deleted        If true, DELETED deployments will be included.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl deployment scale
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/deployment-scale

Scales a deployment to a specified number of replicas.

```
firectl deployment scale [flags]
```

### Examples

```
firectl deployment scale my-deployment --replica-count=3
firectl deployment scale accounts/my-account/deployments/my-deployment --replica-count=3
```

### Flags

```
  -h, --help                  help for scale
      --replica-count int32   The desired number of replicas. Must be non-negative.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl deployment-shape-version get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/deployment-shape-version-get

Prints information about a deployment shape version.

```
firectl deployment-shape-version get [flags]
```

### Examples

```
firectl deployment-shape-version get accounts/my-account/deploymentShapes/my-deployment-shape/versions/my-version
firectl deployment-shape-version get accounts/my-account/deploymentShapes/my-deployment-shape/versions/latest
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl deployment-shape-version list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/deployment-shape-version-list

Prints all deployment shape versions of this deployment shape.

```
firectl deployment-shape-version list [flags]
```

### Examples

```
firectl deployment-shape-version list my-deployment-shape
firectl deployment-shape-version list accounts/my-account/deploymentShapes/my-deployment-shape
firectl deployment-shape-version list
```

### Flags

```
      --base-model string   If specified, will filter out versions not matching the given base model.
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl deployment undelete
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/deployment-undelete

Undeletes a deployment.

```
firectl deployment undelete [flags]
```

### Examples

```
firectl deployment undelete my-deployment
```

### Flags

```
  -h, --help                    help for undelete
      --wait                    Wait until the deployment is undeleted.
      --wait-timeout duration   Maximum time to wait when using --wait flag. (default 1h0m0s)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl deployment update
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/deployment-update

Update a deployment.

```
firectl deployment update [flags]
```

### Examples

```
firectl deployment update my-deployment
firectl deployment update accounts/my-account/deployments/my-deployment
firectl deployment update my-deployment --file=/path/to/deployment.json
```

### Flags

```
      --accelerator-count int32                  The number of accelerators to use per replica.
      --accelerator-type string                  The type of accelerator to use. Must be one of {NVIDIA_A100_80GB, NVIDIA_H100_80GB, NVIDIA_H200_141GB, AMD_MI300X_192GB}
      --deployment-shape string                  The deployment shape to use for this deployment.
      --description string                       Description of the deployment.
      --direct-route-api-keys stringArray        The API keys for the direct route. Only available to enterprise accounts.
      --direct-route-type string                 If set, this deployment will expose an endpoint that bypasses our API gateway. Must be one of {INTERNET, GCP_PRIVATE_SERVICE_CONNECT, AWS_PRIVATELINK}. Only available to enterprise accounts.
      --display-name string                      Human-readable name of the deployment. Must be fewer than 64 characters long.
      --draft-model string                       The draft model to use for speculative decoding. If the model is under your account, you can specify the model ID. If the model is under another account, you can specify the full resource name (e.g. accounts/other-account/models/falcon-7b).
      --draft-token-count int32                  The number of tokens to generate per step for speculative decoding.
      --dry-run                                  Print the request proto without running it.
      --enable-addons                            If true, enable addons for this deployment.
      --enable-session-affinity                  If true, does sticky routing based on the 'user' field. Only available to enterprise accounts.
  -h, --help                                     help for update
      --load-targets Map                         Map of autoscaling load metric names to their target utilization factors.
      --long-prompt                              Whether this deployment is optimized for long prompts.
      --max-context-length int32                 The maximum context length supported by the model (context window). If set to 0 or not specified, the model's default maximum context length will be used.
      --max-replica-count int32                  Maximum number of replicas for the deployment. If min-replica-count > 0 defaults to 0, otherwise defaults to 1.
      --max-with-revocable-replica-count int32   Maximum number of replicas including revocable replicas. Any replicas in excess of max_replica_count are revocable.
      --min-replica-count int32                  Minimum number of replicas for the deployment. If min-replica-count < max-replica-count the deployment will automatically scale between the two replica counts based on load.
      --ngram-speculation-length int32           The length of previous input sequence to be considered for N-gram speculation.
  -o, --output Output                            Set the output format to "text", "json", or "flag". (default text)
      --precision string                         The precision with which the model is served. If specified, must be one of {FP8, FP16, FP4, BF16}.
      --region string                            Placement: 'global', region group (us), or specific region (us-iowa-1).
      --scale-down-window duration               The duration the autoscaler will wait before scaling down a deployment after observing decreased load. Default is 10m.
      --scale-to-zero-window duration            The duration after which there are no requests that the deployment will be scaled down to zero replicas, if min-replica-count is 0. Default 1h.
      --scale-up-window duration                 The duration the autoscaler will wait before scaling up a deployment after observing increased load. Default is 30s.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl dpo-job cancel
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/dpo-job-cancel

Cancels a running dpo job.

```
firectl dpo-job cancel [flags]
```

### Examples

```
firectl dpo-job cancel my-dpo-job
firectl dpo-job cancel accounts/my-account/dpoJobs/my-dpo-job
```

### Flags

```
  -h, --help                    help for cancel
      --wait                    Wait until the dpo job is cancelled.
      --wait-timeout duration   Maximum time to wait when using --wait flag. (default 10m0s)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl dpo-job create
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/dpo-job-create

Creates a dpo job.

```
firectl dpo-job create [flags]
```

### Examples

```
firectl dpo-job create \
	--base-model llama-v3-8b-instruct \
	--dataset sample-dataset \
	--output-model name-of-the-trained-model

# Create from source job:
firectl dpo-job create \
	--source-job my-previous-job \
	--output-model new-model

```

### Flags

```
      --base-model string                         The base model for the dpo job. Only one of base-model or warm-start-from should be specified.
      --dataset string                            The dataset for the dpo job. (Required)
      --output-model string                       The output model for the dpo job.
      --job-id string                             The ID of the dpo job. If not set, it will be autogenerated.
      --warm-start-from string                    The model to warm start from. If set, base-model must not be set.
      --source-job string                         The source dpo job to copy configuration from. If other flags are set, they will override the source job's configuration.
      --epochs int32                              The number of epochs for the dpo job.
      --learning-rate float32                     The learning rate for the dpo job.
      --max-context-length int32                  Maximum token length per sequence. When unset, defaults to the validated training shape's max supported context length. Capped (not pinned) by the shape: smaller user values are honored.
      --batch-size-samples int32                  Number of preference pairs per optimizer step. This is the supported Training V2 batch knob.
      --learning-rate-warmup-steps int32          The number of learning rate warmup steps for the dpo job.
      --learning-rate-scheduler string            Learning rate scheduler to use: constant, linear, or cosine. When unset, the trainer uses its legacy constant schedule.
      --learning-rate-min-lr-ratio float32        Minimum learning rate as a ratio of --learning-rate for linear/cosine schedules. Must be between 0 and 1.
      --learning-rate-decay-ratio float32         Fraction of total training steps over which to decay for linear/cosine schedules. 0 uses the full run.
      --lora-rank int32                           The rank of the LoRA layers for the dpo job. Use --full-parameter for full parameter tuning. (default 8)
      --optimizer-weight-decay float32            Weight decay (L2 regularization) for the optimizer. Default in trainer is 0.01.
      --full-parameter                            Enable full parameter fine-tuning instead of LoRA. Equivalent to --lora-rank=0. Requires bf16 precision.

      --wandb-api-key string                      [WANDB_API_KEY] WandB API Key. (Required if any WandB flag is set)
      --wandb-project string                      [WANDB_PROJECT] WandB Project. (Required if any WandB flag is set)
      --wandb-entity string                       [WANDB_ENTITY] WandB Entity. (Required if any WandB flag is set)
      --wandb                                     Enable WandB


      --aws-credentials-secret string             [AWS_CREDENTIALS_SECRET] AWS credentials secret (mutually exclusive with --aws-iam-role)
      --aws-iam-role string                       [AWS_IAM_ROLE_ARN] AWS IAM role ARN (mutually exclusive with --aws-credentials-secret)


      --azure-credentials-secret string           [AZURE_CREDENTIALS_SECRET] Azure credentials secret
      --azure-managed-identity-client-id string   [AZURE_MANAGED_IDENTITY_CLIENT_ID] Azure managed identity client ID for Workload Identity Federation
      --azure-tenant-id string                    [AZURE_TENANT_ID] Azure tenant ID (required with --azure-managed-identity-client-id)

      --display-name string                       The display name of the dpo job.
      --quiet                                     If set, only errors will be printed.
      --loss-method string                        Loss method for the job (DPO, GRPO, or ORPO). Defaults to DPO for new jobs; when using --source-job, inherits the source job's method unless overridden.
      --dpo-beta float32                          Typed DPO beta override. Must be > 0 and < 0.5. Applies to loss_config.dpo.beta.
      --orpo-lambda float32                       Typed ORPO lambda override. Applies to loss_config.orpo.lambda.
      --use-reservation                           Try the account's reservation capacity before falling back to shared trainer capacity.
      --dry-run                                   Print the request proto without running it.
  -o, --output Output                             Set the output format to "text", "json", or "flag". (default text)
  -h, --help                                      help for create
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl dpo-job delete
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/dpo-job-delete

Deletes a dpo job.

```
firectl dpo-job delete [flags]
```

### Examples

```
firectl dpo-job delete my-dpo-job
firectl dpo-job delete accounts/my-account/dpoJobs/my-dpo-job
```

### Flags

```
      --dry-run                 Print the request proto without running it.
  -h, --help                    help for delete
  -o, --output Output           Set the output format to "text", "json", or "flag". (default text)
      --wait                    Wait until the dpo job is deleted.
      --wait-timeout duration   Maximum time to wait when using --wait flag. (default 30m0s)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl dpo-job export-metrics
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/dpo-job-export-metrics

Exports metrics for a dpo job.

```
firectl dpo-job export-metrics [flags]
```

### Examples

```
firectl dpo-job export-metrics my-dpo-job
firectl dpo-job export-metrics accounts/my-account/dpoJobs/my-dpo-job
```

### Flags

```
      --filename string   The file name to export to. (default "metrics.jsonl")
  -h, --help              help for export-metrics
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl dpo-job get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/dpo-job-get

Retrieves information about a dpo job.

```
firectl dpo-job get [flags]
```

### Examples

```
firectl dpo-job get my-dpo-job
firectl dpo-job get accounts/my-account/dpoJobs/my-dpo-job
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl dpo-job list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/dpo-job-list

Lists all dpo jobs in an account.

```
firectl dpo-job list [flags]
```

### Examples

```
firectl dpo-job list
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl dpo-job resume
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/dpo-job-resume

Resumes a dpo job.

```
firectl dpo-job resume [flags]
```

### Examples

```
firectl dpo-job resume my-dpo-job
firectl dpo-job resume accounts/my-account/dpoJobs/my-dpo-job
```

### Flags

```
  -h, --help   help for resume
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl evaluator-revision alias
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/evaluator-revision-alias

Alias an evaluator revision

```
firectl evaluator-revision alias [flags]
```

### Examples

```
firectl evaluator-revision alias accounts/my-account/evaluators/my-evaluator/versions/abc123 --alias-id current
```

### Flags

```
  -h, --help   help for alias
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl evaluator-revision delete
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/evaluator-revision-delete

Delete an evaluator revision

```
firectl evaluator-revision delete [flags]
```

### Examples

```
firectl evaluator-revision delete accounts/my-account/evaluators/my-evaluator/versions/abc123
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for delete
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl evaluator-revision get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/evaluator-revision-get

Get an evaluator revision

```
firectl evaluator-revision get [flags]
```

### Examples

```
firectl evaluator-revision get accounts/my-account/evaluators/my-evaluator/versions/latest
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl evaluator-revision list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/evaluator-revision-list

List evaluator revisions

```
firectl evaluator-revision list [flags]
```

### Examples

```
firectl evaluator-revision list accounts/my-account/evaluators/my-evaluator
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl identity-provider create
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/identity-provider-create

Creates a new identity provider.

```
firectl identity-provider create [flags]
```

### Examples

```
# Create SAML identity provider with metadata URL
firectl identity-provider create --display-name="Company SAML" \
  --saml-metadata-url="https://company.okta.com/app/xyz/sso/saml/metadata"

# Create SAML identity provider with IdP-initiated SSO
firectl identity-provider create --display-name="Company SAML" \
  --saml-metadata-url="https://company.okta.com/app/xyz/sso/saml/metadata" \
  --enable-idp-initiated-sso

# Create SAML identity provider with metadata XML file (when URL is not publicly accessible)
firectl identity-provider create --display-name="Company SAML" \
  --saml-metadata-xml-file="/path/to/metadata.xml"

# Create OIDC identity provider
firectl identity-provider create --display-name="Company OIDC" \
  --oidc-issuer="https://auth.company.com" \
  --oidc-client-id="abc123" \
  --oidc-client-secret="secret456"

# Create OIDC identity provider with multiple domains
firectl identity-provider create --display-name="Example OIDC" \
  --oidc-issuer="https://accounts.google.com" \
  --oidc-client-id="client123" \
  --oidc-client-secret="secret456" \
  --tenant-domains="example.com,example.co.uk"
```

### Flags

```
      --display-name string             The display name of the identity provider (required)
      --dry-run                         Print the request proto without running it.
      --enable-idp-initiated-sso        Enable IdP-initiated SAML SSO. When enabled, users can login from their identity provider's portal. Only supported for SAML providers.
      --enable-jit-user-provisioning    Enable Just-In-Time (JIT) user provisioning. When enabled, users are automatically created on first SSO login if they don't already exist.
      --enforce-sso                     Enforce SSO authentication and restrict account access to approved email domains only
  -h, --help                            help for create
      --jit-default-role string         Default role for JIT-provisioned users (admin, user, contributor, inference-user). Defaults to the least privileged role available for the account.
      --oidc-client-id string           The OIDC client ID for OIDC providers
      --oidc-client-secret string       The OIDC client secret for OIDC providers
      --oidc-issuer string              The OIDC issuer URL for OIDC providers
  -o, --output Output                   Set the output format to "text", "json", or "flag". (default text)
      --saml-metadata-url string        The SAML metadata URL for SAML providers
      --saml-metadata-xml-file string   Path to SAML metadata XML file. Use when your IdP metadata URL is not publicly accessible (e.g., behind a VPN).
      --tenant-domains string           Comma-separated list of allowed domains for the organization (e.g., 'example.com,example.co.uk'). If not provided, domain will be derived from account email.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl identity-provider get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/identity-provider-get

Prints information about an identity provider.

```
firectl identity-provider get [flags]
```

### Examples

```
firectl identity-provider get my-provider
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl identity-provider list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/identity-provider-list

List identity providers for an account

```
firectl identity-provider list [flags]
```

### Examples

```
firectl identity-provider list
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl model create
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/model-create

Creates and uploads a model.

```
firectl model create [flags]
```

### Examples

```
firectl model create my-model /path/to/checkpoint/
firectl model create my-model s3://bucket/path --role-arn arn:aws:iam::123456789012:role/MyRole
firectl model create my-model https://storage-account.blob.core.windows.net/container/path --azure-sas-token-secret accounts/{account}/secrets/{secret}
firectl model create my-model https://storage-account.blob.core.windows.net/container/path --azure-client-id <client-id> --azure-tenant-id <tenant-id>
```

### Flags

```
      --base-model string                 If specified, then the model will be considered a PEFT addon with the given base model.
      --default-draft-model string        The default speculative draft model to use when creating a deployment.
      --default-draft-token-count int32   The default speculative draft token count when creating a deployment.
      --description string                The description of the model.
      --display-name string               The display name of the model.
      --dry-run                           Print the request proto without running it.
      --embedding                         If true, sets the model kind to an embeddings base model.
      --enable-resumable-upload           If true, uses resumable upload for the model.
      --github-url string                 The GitHub URL of the model.
  -h, --help                              help for create
      --hugging-face-url string           The Hugging Face URL of the model.
  -o, --output Output                     Set the output format to "text", "json", or "flag". (default text)
      --poll-duration duration            The duration to poll for model import operation completion. Default is 2 hours. (default 2h0m0s)
      --public                            Whether the model is publicly accessible.
      --quiet                             If true, does not print the upload progress bar.
      --supports-image-input              Whether the model supports image inputs.
      --supports-tools                    Whether the model supports function calling.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl model delete
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/model-delete

Deletes a model.

```
firectl model delete [flags]
```

### Examples

```
firectl model delete my-model
firectl model delete accounts/my-account/models/my-model
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for delete
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl model download
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/model-download

Download a model.

```
firectl model download [flags]
```

### Examples

```
firectl model download my-model /path/to/checkpoint/
```

### Flags

```
  -h, --help    help for download
      --quiet   If true, does not print the upload progress bar.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl model get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/model-get

Prints information about a model.

```
firectl model get [flags]
```

### Examples

```
firectl model get my-model
firectl model get accounts/fireworks/models/my-model
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl model list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/model-list

Prints all models in an account.

```
firectl model list [flags]
```

### Examples

```
# List all models
firectl model list

# Search by name (partial match)
firectl model list --search deepseek

# Filter by kind
firectl model list --kind HF_PEFT_ADDON

# Filter by state
firectl model list --state READY

# Combine filters
firectl model list --search deepseek --kind HF_PEFT_ADDON

# Use raw filter for advanced queries
firectl model list --filter 'create_time > timestamp("2025-01-01T00:00:00Z")'
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --kind string         Filter by model kind (e.g., HF_PEFT_ADDON, HF_BASE_MODEL, DRAFT_ADDON)
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
      --search string       Filter models by name (searches both model_id and display_name)
      --state string        Filter by state (e.g., READY, UPLOADING)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl model load-lora
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/model-load-lora

Loads a LoRA model.

### Usage

Loads a LoRA model to a dedicated deployment. If successful, a DeployedModel resource will be created.

```
firectl model load-lora [flags]
```

### Examples

```
firectl model load-lora my-lora --deployment my-deployment
```

### Flags

```
      --deployment string       The resource ID of the deployment where the LoRA model is to be loaded.
  -h, --help                    help for load-lora
      --public                  If true, the LoRA model will be publicly available for inference.
      --replace-merged-addon    Required when loading an addon to a hot reload deployment. If there is already an existing addon, it will be replaced.
      --wait                    Wait until the model is deployed.
      --wait-timeout duration   Maximum time to wait when using --wait flag. (default 30m0s)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl model prepare
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/model-prepare

Prepare models for different precisions

```
firectl model prepare [flags]
```

### Examples

```
firectl model prepare my-model
firectl model prepare accounts/my-account/models/my-model
```

### Flags

```
  -h, --help                    help for prepare
      --wait                    Wait until the model preparation is complete.
      --wait-timeout duration   Maximum time to wait when using --wait flag. (default 30m0s)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl model unload-lora
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/model-unload-lora

Unloads a LoRA model.

### Usage

Unloads a LoRA model from a dedicated deployment.

```
firectl model unload-lora [flags]
```

### Examples

```
firectl model unload-lora my-lora --deployment abcd1234
```

### Flags

```
      --deployment string       The resource name of the deployment where the model is to be undeployed.
  -h, --help                    help for unload-lora
      --wait                    Wait until the model is deployed.
      --wait-timeout duration   Maximum time to wait when using --wait flag. (default 30m0s)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl model update
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/model-update

Updates a model.

```
firectl model update [flags]
```

### Examples

```
firectl model update my-model --display-name="New Name"
firectl model update accounts/my-account/models/my-model --display-name="New Name"
```

### Flags

```
      --default-draft-model string        The default speculative draft model to use when creating a deployment.
      --default-draft-token-count int32   The default speculative draft token count when creating a deployment.
      --description string                The description of the model.
      --display-name string               The display name of the model.
      --dry-run                           Print the request proto without running it.
      --github-url string                 The GitHub URL of the model.
  -h, --help                              help for update
      --hugging-face-url string           The Hugging Face URL of the model.
  -o, --output Output                     Set the output format to "text", "json", or "flag". (default text)
      --public                            Whether the model is publicly accessible.
      --supports-image-input              Whether the model supports image inputs.
      --supports-tools                    Whether the model supports function calling.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl model upload
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/model-upload

Resumes or completes a model upload.

### Usage

Resumes or completes a model upload for an existing model. This command should be used when a previous upload was interrupted.

```
firectl model upload [flags]
```

### Examples

```
firectl model upload my-model /path/to/checkpoint/
```

### Flags

```
  -h, --help    help for upload
      --quiet   If true, does not print the upload progress bar.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl quota get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/quota-get

Prints information about a quota.

```
firectl quota get [flags]
```

### Examples

```
firectl quota get serverless-inference-rpm
firectl quota get accounts/my-account/quotas/serverless-inference-rpm
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl quota list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/quota-list

Prints all quotas.

```
firectl quota list [flags]
```

### Examples

```
firectl quota list
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl quota update
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/quota-update

Updates a quota.

```
firectl quota update [flags]
```

### Examples

```
firectl quota update serverless-inference-rpm --value 300
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for update
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
      --value int       The quota allowed to be used by this account. Must be less than max_value.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl reinforcement-fine-tuning-job cancel
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/reinforcement-fine-tuning-job-cancel

Cancels a running reinforcement fine-tuning job.

```
firectl reinforcement-fine-tuning-job cancel [flags]
```

### Examples

```
firectl reinforcement-fine-tuning-job cancel my-rftj
firectl reinforcement-fine-tuning-job cancel accounts/my-account/reinforcementFineTuningJobs/my-rftj
```

### Flags

```
  -h, --help   help for cancel
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl reinforcement-fine-tuning-job create
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/reinforcement-fine-tuning-job-create

Creates a reinforcement fine-tuning job.

```
firectl reinforcement-fine-tuning-job create [flags]
```

### Examples

```
firectl reinforcement-fine-tuning-job create \
	--base-model llama-v3-8b-instruct \
	--dataset sample-dataset \
	--epochs 5 \
	--output-model name-of-the-trained-model \
	--evaluator accounts/my-account/evaluators/abc123

# Create from source job:
firectl reinforcement-fine-tuning-job create \
	--source-job my-previous-job \
	--output-model new-model

```

### Flags

```
      --base-model string                         The base model for the reinforcement fine-tuning job. Only one of base-model or warm-start-from should be specified.
      --dataset string                            The dataset for the reinforcement fine-tuning job. (Required)
      --output-model string                       The output model for the reinforcement fine-tuning job.
      --job-id string                             The ID of the reinforcement fine-tuning job. If not set, it will be autogenerated.
      --warm-start-from string                    The model to warm start from. If set, base-model must not be set.
      --source-job string                         The source reinforcement fine-tuning job to copy configuration from. If other flags are set, they will override the source job's configuration.
      --evaluator string                          The evaluator resource name to use for the reinforcement fine-tuning job. (Required)
      --epochs int32                              The number of epochs for the reinforcement fine-tuning job. (default 1)
      --learning-rate float32                     The learning rate for the reinforcement fine-tuning job. (default 0.0001)
      --max-context-length int32                  Maximum token length per sequence. When unset, defaults to the validated training shape's max supported context length. Capped (not pinned) by the shape: smaller user values are honored.
      --batch-size int32                          Legacy V1 token budget used by managed RFT. Limits packed tokens per microbatch and remains active when --batch-size-samples is set.
      --batch-size-samples int32                  Optional number of rollout samples per gradient batch. Works alongside --batch-size; with --gradient-accumulation-steps=N, the target samples per optimizer step is this value multiplied by N.
      --gradient-accumulation-steps int32         The number of batches to accumulate gradients before updating the model parameters. The effective batch size will be batch-size multiplied by this value. (default 1)
      --learning-rate-warmup-steps int32          The number of learning rate warmup steps for the reinforcement fine-tuning job.
      --lora-rank int32                           The rank of the LoRA layers for the reinforcement fine-tuning job. (default 8)
      --optimizer-weight-decay float32            Weight decay (L2 regularization) for the optimizer. Default in trainer is 0.01.
                                                  
      --wandb-api-key string                      [WANDB_API_KEY] WandB API Key. (Required if any WandB flag is set)
      --wandb-project string                      [WANDB_PROJECT] WandB Project. (Required if any WandB flag is set)
      --wandb-entity string                       [WANDB_ENTITY] WandB Entity. (Required if any WandB flag is set)
      --wandb                                     Enable WandB
                                                  
                                                  
      --aws-credentials-secret string             [AWS_CREDENTIALS_SECRET] AWS credentials secret (mutually exclusive with --aws-iam-role)
      --aws-iam-role string                       [AWS_IAM_ROLE_ARN] AWS IAM role ARN (mutually exclusive with --aws-credentials-secret)
                                                  
                                                  
      --azure-credentials-secret string           [AZURE_CREDENTIALS_SECRET] Azure credentials secret
      --azure-managed-identity-client-id string   [AZURE_MANAGED_IDENTITY_CLIENT_ID] Azure managed identity client ID for Workload Identity Federation
      --azure-tenant-id string                    [AZURE_TENANT_ID] Azure tenant ID (required with --azure-managed-identity-client-id)
                                                  
      --temperature float32                       The randomness of the model's word or token selection during text generation.
      --top-p float32                             Top-p sampling, selecting the smallest set of candidate words whose cumulative probability exceeds the top-p.
      --response-candidates-count int32           The number of response candidates to generate per input.
      --max-output-tokens int32                   The maximum number of tokens to generate in the response.
      --top-k int32                               Top-k sampling parameter, limits the token selection to the top k tokens.
      --extra-body string                         Additional parameters for the inference request as a JSON string. For example: '{"stop": ["\n"]}'
      --quiet                                     If set, only errors will be printed.
      --max-concurrent-rollouts int32             Maximum number of concurrent rollouts during the RFT job. If not set, defaults to the value set in @evaluation_test header.
      --max-concurrent-evaluations int32          Maximum number of concurrent evaluations during the RFT job. If not set, defaults to the value set in @evaluation_test header.
      --rl-loss-method string                     RL loss method for underlying trainers. One of {grpo,dapo,gspo-token}.
      --rl-kl-beta float32                        Override KL beta for GRPO-like methods. Must be >= 0.
      --chunk-size int32                          The minimum chunk size to split the dataset before the RL flow. Set to -1 to disable chunking. (default 200)
      --max-inference-replica-count int32         Maximum number of replicas used for batch inference. If not specified, defaults to 1.
      --dry-run                                   Print the request proto without running it.
  -o, --output Output                             Set the output format to "text", "json", or "flag". (default text)
  -h, --help                                      help for create
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl reinforcement-fine-tuning-job delete
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/reinforcement-fine-tuning-job-delete

Deletes a reinforcement fine-tuning job.

```
firectl reinforcement-fine-tuning-job delete [flags]
```

### Examples

```
firectl reinforcement-fine-tuning-job delete my-rftj
firectl reinforcement-fine-tuning-job delete accounts/my-account/reinforcementFineTuningJobs/my-rftj
```

### Flags

```
      --dry-run                 Print the request proto without running it.
  -h, --help                    help for delete
  -o, --output Output           Set the output format to "text", "json", or "flag". (default text)
      --wait                    Wait until the reinforcement fine-tuning job is deleted.
      --wait-timeout duration   Maximum time to wait when using --wait flag. (default 30m0s)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl reinforcement-fine-tuning-job get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/reinforcement-fine-tuning-job-get

Retrieves information about a reinforcement fine-tuning job.

```
firectl reinforcement-fine-tuning-job get [flags]
```

### Examples

```
firectl reinforcement-fine-tuning-job get my-rftj
firectl reinforcement-fine-tuning-job get accounts/my-account/reinforcementFineTuningJobs/my-rftj
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl reinforcement-fine-tuning-job list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/reinforcement-fine-tuning-job-list

Lists all reinforcement fine-tuning jobs in an account.

```
firectl reinforcement-fine-tuning-job list [flags]
```

### Examples

```
firectl reinforcement-fine-tuning-job list
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl reinforcement-fine-tuning-job resume
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/reinforcement-fine-tuning-job-resume

Resumes a failed reinforcement fine-tuning job.

```
firectl reinforcement-fine-tuning-job resume [flags]
```

### Examples

```
firectl reinforcement-fine-tuning-job resume my-rftj
firectl reinforcement-fine-tuning-job resume accounts/my-account/reinforcementFineTuningJobs/my-rftj
```

### Flags

```
  -h, --help   help for resume
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl reinforcement-fine-tuning-job update
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/reinforcement-fine-tuning-job-update

Update fields on a reinforcement fine-tuning job.

```
firectl reinforcement-fine-tuning-job update [flags]
```

### Examples

```
firectl-admin rftj update my-job --training-accelerator-type=NVIDIA_B200_180GB
firectl-admin rftj update accounts/my-account/reinforcementFineTuningJobs/my-job --training-accelerator-type=NVIDIA_B200_180GB --toleration=fireworks.ai/rftj
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for update
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl reservation get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/reservation-get

Prints information about a reservation.

```
firectl reservation get [flags]
```

### Examples

```
firectl reservation get abcdef
firectl reservation get accounts/my-account/reservations/abcdef
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl reservation list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/reservation-list

Prints active reservations.

```
firectl reservation list [flags]
```

### Examples

```
firectl reservation list
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
      --show-inactive       Show all reservations
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl rlor-trainer-job cancel
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/rlor-trainer-job-cancel

Cancels a running rlor trainer job.

```
firectl rlor-trainer-job cancel [flags]
```

### Examples

```
firectl rlor-trainer-job cancel my-rlor-job
firectl rlor-trainer-job cancel accounts/my-account/rlorTrainerJobs/my-rlor-job
```

### Flags

```
  -h, --help                    help for cancel
      --wait                    Wait until the rlor trainer job is cancelled.
      --wait-timeout duration   Maximum time to wait when using --wait flag. (default 10m0s)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl rlor-trainer-job create
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/rlor-trainer-job-create

Creates a rlor trainer job.

```
firectl rlor-trainer-job create [flags]
```

### Examples

```
firectl rlor-trainer-job create \
	--base-model llama-v3-8b-instruct \
	--dataset sample-dataset \
	--output-model name-of-the-trained-model
	--rewards key1=0.5,key2=0.5

```

### Flags

```
      --base-model string                         The base model for the rlor trainer job. Only one of base-model or warm-start-from should be specified.
      --dataset string                            The dataset for the rlor trainer job. (Required)
      --output-model string                       The output model for the rlor trainer job.
      --job-id string                             The ID of the rlor trainer job. If not set, it will be autogenerated.
      --warm-start-from string                    The model to warm start from. If set, base-model must not be set.
      --evaluation-dataset string                 The evaluation dataset for the fine-tuning job.
      --epochs int32                              The number of epochs for the fine-tuning job.
      --learning-rate float32                     The learning rate for the fine-tuning job.
      --max-context-length int32                  Maximum token length per sequence. When unset, defaults to the validated training shape's max supported context length. Capped (not pinned) by the shape: smaller user values are honored.
      --batch-size int32                          Legacy V1 token budget. On V1 RLOR paths, limits packed tokens per microbatch and remains active when --batch-size-samples is set. Training V2 rejects nonzero values.
      --batch-size-samples int32                  Sample-count batching control. On V1 RLOR paths it limits samples per gradient batch and works alongside --batch-size; on Training V2 SFT and DPO paths it maps to examples or preference pairs per optimizer step.
      --gradient-accumulation-steps int32         The number of batches to accumulate gradients before updating the model parameters. The effective batch size will be batch-size multiplied by this value. (default 1)
      --learning-rate-warmup-steps int32          The number of learning rate warmup steps for the rlor trainer job.
      --lora-rank int32                           The rank of the LoRA layers for the fine-tuning job.
                                                   (default 8)
      --wandb-api-key string                      [WANDB_API_KEY] WandB API Key. (Required if any WandB flag is set)
      --wandb-project string                      [WANDB_PROJECT] WandB Project. (Required if any WandB flag is set)
      --wandb-entity string                       [WANDB_ENTITY] WandB Entity. (Required if any WandB flag is set)
      --wandb                                     Enable WandB
                                                  
                                                  
      --aws-credentials-secret string             [AWS_CREDENTIALS_SECRET] AWS credentials secret (mutually exclusive with --aws-iam-role)
      --aws-iam-role string                       [AWS_IAM_ROLE_ARN] AWS IAM role ARN (mutually exclusive with --aws-credentials-secret)
                                                  
                                                  
      --azure-credentials-secret string           [AZURE_CREDENTIALS_SECRET] Azure credentials secret
      --azure-managed-identity-client-id string   [AZURE_MANAGED_IDENTITY_CLIENT_ID] Azure managed identity client ID for Workload Identity Federation
      --azure-tenant-id string                    [AZURE_TENANT_ID] Azure tenant ID (required with --azure-managed-identity-client-id)
                                                  
      --rewards strings                           A list of reward weights as key=value pairs (e.g., --rewards key1=1.0,key2=0.5).
      --display-name string                       The display name of the supervised fine-tuning job.
      --quiet                                     If set, only errors will be printed.
      --eval-auto-carveout                        If set, the evaluation dataset will be auto-carved.
      --training-shape string                     Training shape selector for service-mode launches. Accepts training shape names or training shape version names; a shape name uses its latest validated version.
      --trainer-replicas int32                    Data-parallel replica count for service-mode trainers. If not specified, defaults to 1.
      --inactivity-timeout duration               Trainer inactivity timeout (for example, 30m or 2h). The trainer reports tracked activity, including trainer API operations and active-session heartbeats. If no tracked activity is observed for this duration, the trainer is automatically stopped. If omitted or 0, defaults to 10 minutes. Use --disable-inactivity-cleanup to disable automatic cleanup. GPU usage continues to accrue while the trainer is running.
      --disable-inactivity-cleanup                Disable trainer inactivity cleanup. The trainer is not automatically stopped due to inactivity, and GPU usage continues to accrue while the trainer is running.
      --dry-run                                   Print the request proto without running it.
  -o, --output Output                             Set the output format to "text", "json", or "flag". (default text)
  -h, --help                                      help for create
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl rlor-trainer-job delete
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/rlor-trainer-job-delete

Deletes a rlor trainer job.

```
firectl rlor-trainer-job delete [flags]
```

### Examples

```
firectl rlor-trainer-job delete my-rlor-job
firectl rlor-trainer-job delete accounts/my-account/rlorTrainerJobs/my-rlor-job
```

### Flags

```
      --dry-run                 Print the request proto without running it.
  -h, --help                    help for delete
  -o, --output Output           Set the output format to "text", "json", or "flag". (default text)
      --wait                    Wait until the rlor trainer job is deleted.
      --wait-timeout duration   Maximum time to wait when using --wait flag. (default 30m0s)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl rlor-trainer-job get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/rlor-trainer-job-get

Retrieves information about a rlor trainer job.

```
firectl rlor-trainer-job get [flags]
```

### Examples

```
firectl rlor-trainer-job get my-rlor-job
firectl rlor-trainer-job get accounts/my-account/rlorTrainerJobs/my-rlor-job
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl rlor-trainer-job list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/rlor-trainer-job-list

Lists all rlor trainer jobs in an account.

```
firectl rlor-trainer-job list [flags]
```

### Examples

```
firectl rlor-trainer-job list
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl router create
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/router-create

Creates a router.

```
firectl router create [flags]
```

### Examples

```
firectl router create --deployments=my-deployment1,my-deployment2
firectl router create --strategy=even-load --deployments=my-deployment1,my-deployment2
firectl router create --strategy=weighted-random --deployments=my-deployment1,my-deployment2
```

### Flags

```
      --display-name string   The display name of the router.
      --model string          The model to route traffic to.
      --strategy string       The strategy to use for routing traffic. (default "weighted-random")
      --deployments strings   The deployment names to be covered by the router.
      --router-id string      The ID of the router. If not specified, a random ID will be generated.
      --public                Whether the router is public. If not specified, the router is private.
      --dry-run               Print the request proto without running it.
  -o, --output Output         Set the output format to "text", "json", or "flag". (default text)
  -h, --help                  help for create
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl router delete
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/router-delete

Deletes a router.

```
firectl router delete [flags]
```

### Examples

```
firectl router delete my-router
firectl router delete accounts/my-account/routers/my-router
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for delete
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl router get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/router-get

Prints information about a router.

```
firectl router get [flags]
```

### Examples

```
firectl router get my-router
firectl router get accounts/my-account/routers/my-router
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl router list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/router-list

Prints all routers in the account.

```
firectl router list [flags]
```

### Examples

```
firectl router list
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl router update
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/router-update

Update a router.

```
firectl router update [flags]
```

### Examples

```
firectl router update my-router --deployments=my-deployment1,my-deployment2
firectl router update my-router --strategy=weighted-random
firectl router update my-router --strategy=even-load
```

### Flags

```
      --display-name string   The display name of the router.
      --strategy string       The strategy to use for routing traffic. (default "weighted-random")
      --deployments strings   The deployment names to be covered by the router.
      --public                Whether the router is public. If not specified, the router is private.
      --session-affinity      Enable regional-level session affinity for this router.
      --dry-run               Print the request proto without running it.
  -o, --output Output         Set the output format to "text", "json", or "flag". (default text)
  -h, --help                  help for update
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl secret create
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/secret-create

Creates a secret for the signed in user.

```
firectl secret create [flags]
```

### Examples

```
firectl secret create --name MY_SECRET --value mysecretvalue
firectl secret create --name AWS_CREDS --from-file aws-credentials.json
firectl secret create --name AWS_CREDS --aws-access-key-id AKIA... --aws-secret-access-key ...
```

### Flags

```
      --aws-access-key-id string       AWS access key ID (automatically formats as JSON with --aws-secret-access-key)
      --aws-secret-access-key string   AWS secret access key (automatically formats as JSON with --aws-access-key-id)
      --dry-run                        Print the request proto without running it.
      --from-file string               Path to a file containing the secret value
  -h, --help                           help for create
      --name string                    The name of the secret to be created
  -o, --output Output                  Set the output format to "text", "json", or "flag". (default text)
      --value string                   The value of the secret
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl secret delete
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/secret-delete

Deletes a secret.

```
firectl secret delete [flags]
```

### Examples

```
firectl secret delete MY_SECRET
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for delete
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl secret get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/secret-get

Retrieves a secret by name.

```
firectl secret get [flags]
```

### Examples

```
firectl secret get MY_SECRET
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl secret list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/secret-list

Lists all secrets for the signed in user.

```
firectl secret list [flags]
```

### Examples

```
firectl secret list
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl secret update
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/secret-update

Updates an existing secret.

```
firectl secret update [flags]
```

### Examples

```
firectl secret update --id MY_SECRET --value newvalue
firectl secret update --id AWS_CREDS --from-file aws-credentials.json
firectl secret update --id AWS_CREDS --aws-access-key-id AKIA... --aws-secret-access-key ...
```

### Flags

```
      --aws-access-key-id string       AWS access key ID (automatically formats as JSON with --aws-secret-access-key)
      --aws-secret-access-key string   AWS secret access key (automatically formats as JSON with --aws-access-key-id)
      --dry-run                        Print the request proto without running it.
      --from-file string               Path to a file containing the secret value
  -h, --help                           help for update
      --id string                      The id of the secret to be updated
  -o, --output Output                  Set the output format to "text", "json", or "flag". (default text)
      --value string                   The new value of the secret
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl set-api-key
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/set-api-key

Sets the default API key in ~/.fireworks/auth.ini.

```
firectl set-api-key [flags]
```

### Examples

```
firectl set-api-key API_KEY
```

### Flags

```
  -h, --help   help for set-api-key
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl supervised-fine-tuning-job cancel
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/supervised-fine-tuning-job-cancel

Cancels a running supervised fine-tuning job.

```
firectl supervised-fine-tuning-job cancel [flags]
```

### Examples

```
firectl supervised-fine-tuning-job cancel my-sft-job
firectl supervised-fine-tuning-job cancel accounts/my-account/supervisedFineTuningJobs/my-sft-job
```

### Flags

```
  -h, --help   help for cancel
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl supervised-fine-tuning-job create
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/supervised-fine-tuning-job-create

Creates a supervised fine-tuning job.

```
firectl supervised-fine-tuning-job create [flags]
```

### Examples

```
firectl supervised-fine-tuning-job create \
	--base-model llama-v3-8b-instruct \
	--dataset sample-dataset \
	--output-model name-of-the-trained-model

# Create from source job:
firectl supervised-fine-tuning-job create \
	--source-job my-previous-job \
	--output-model new-model

# Cosine learning rate schedule:
firectl supervised-fine-tuning-job create \
	--base-model llama-v3-8b-instruct \
	--dataset sample-dataset \
	--output-model name-of-the-trained-model \
	--learning-rate 0.0001 \
	--learning-rate-warmup-steps 10 \
	--learning-rate-scheduler cosine \
	--learning-rate-min-lr-ratio 0.1 \
	--learning-rate-decay-ratio 0.8

```

### Flags

```
      --base-model string                         The base model for the supervised fine-tuning job. Only one of base-model or warm-start-from should be specified.
      --dataset string                            The dataset for the supervised fine-tuning job. (Required)
      --output-model string                       The output model for the supervised fine-tuning job.
      --job-id string                             The ID of the supervised fine-tuning job. If not set, it will be autogenerated.
      --warm-start-from string                    The model to warm start from. If set, base-model must not be set.
      --source-job string                         The source supervised fine-tuning job to copy configuration from. If other flags are set, they will override the source job's configuration.
      --evaluation-dataset string                 The evaluation dataset for the supervised fine-tuning job.
      --epochs int32                              The number of epochs for the supervised fine-tuning job.
      --learning-rate float32                     The learning rate for the supervised fine-tuning job.
      --max-context-length int32                  Maximum token length per sequence. When unset, defaults to the validated training shape's max supported context length. Capped (not pinned) by the shape: smaller user values are honored.
      --batch-size-samples int32                  Number of training samples per optimizer step. This is the supported Training V2 batch knob.
      --learning-rate-warmup-steps int32          The number of learning rate warmup steps for the supervised fine-tuning job.
      --learning-rate-scheduler string            Learning rate scheduler to use: constant, linear, or cosine. When unset, the trainer uses its legacy constant schedule.
      --learning-rate-min-lr-ratio float32        Minimum learning rate as a ratio of --learning-rate for linear/cosine schedules. Must be between 0 and 1.
      --learning-rate-decay-ratio float32         Fraction of total training steps over which to decay for linear/cosine schedules. 0 uses the full run.
      --lora-rank int32                           The rank of the LoRA layers for the supervised fine-tuning job. Use --full-parameter for full parameter tuning. (default 8)
      --optimizer-weight-decay float32            Weight decay (L2 regularization) for the optimizer. Default in trainer is 0.01.
      --full-parameter                            Enable full parameter fine-tuning instead of LoRA. Equivalent to --lora-rank=0. Requires bf16 precision.

      --wandb-api-key string                      [WANDB_API_KEY] WandB API Key. (Required if any WandB flag is set)
      --wandb-project string                      [WANDB_PROJECT] WandB Project. (Required if any WandB flag is set)
      --wandb-entity string                       [WANDB_ENTITY] WandB Entity. (Required if any WandB flag is set)
      --wandb                                     Enable WandB


      --aws-credentials-secret string             [AWS_CREDENTIALS_SECRET] AWS credentials secret (mutually exclusive with --aws-iam-role)
      --aws-iam-role string                       [AWS_IAM_ROLE_ARN] AWS IAM role ARN (mutually exclusive with --aws-credentials-secret)


      --azure-credentials-secret string           [AZURE_CREDENTIALS_SECRET] Azure credentials secret
      --azure-managed-identity-client-id string   [AZURE_MANAGED_IDENTITY_CLIENT_ID] Azure managed identity client ID for Workload Identity Federation
      --azure-tenant-id string                    [AZURE_TENANT_ID] Azure tenant ID (required with --azure-managed-identity-client-id)

      --display-name string                       The display name of the supervised fine-tuning job.
      --quiet                                     If set, only errors will be printed.
      --eval-auto-carveout                        If set, the evaluation dataset will be auto-carved.
      --use-reservation                           Try the account's reservation capacity before falling back to shared trainer capacity.
      --dry-run                                   Print the request proto without running it.
  -o, --output Output                             Set the output format to "text", "json", or "flag". (default text)
  -h, --help                                      help for create
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl supervised-fine-tuning-job delete
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/supervised-fine-tuning-job-delete

Deletes a supervised fine-tuning job.

```
firectl supervised-fine-tuning-job delete [flags]
```

### Examples

```
firectl supervised-fine-tuning-job delete my-sft-job
firectl supervised-fine-tuning-job delete accounts/my-account/supervisedFineTuningJobs/my-sft-job
```

### Flags

```
      --dry-run                 Print the request proto without running it.
  -h, --help                    help for delete
  -o, --output Output           Set the output format to "text", "json", or "flag". (default text)
      --wait                    Wait until the supervised fine-tuning job is deleted.
      --wait-timeout duration   Maximum time to wait when using --wait flag. (default 30m0s)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl supervised-fine-tuning-job get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/supervised-fine-tuning-job-get

Retrieves information about a supervised fine-tuning job.

```
firectl supervised-fine-tuning-job get [flags]
```

### Examples

```
firectl supervised-fine-tuning-job get my-sft-job
firectl supervised-fine-tuning-job get accounts/my-account/supervisedFineTuningJobs/my-sft-job
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl supervised-fine-tuning-job list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/supervised-fine-tuning-job-list

Lists all supervised fine-tuning jobs in an account.

```
firectl supervised-fine-tuning-job list [flags]
```

### Examples

```
firectl supervised-fine-tuning-job list
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl training-shape clone
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/training-shape-clone

Clones an existing training shape to a new shape.

### Usage

Fetches the source training shape and creates a new training shape with the same configuration. Override flags can be used to change specific fields.

```
firectl training-shape clone <source-training-shape-id> [flags]
```

### Examples

```
firectl training-shape clone my-source-shape --training-shape-id my-new-shape
firectl training-shape clone my-source-shape --training-shape-id my-new-shape --trainer-image-tag 0.24.11
firectl training-shape clone accounts/my-account/trainingShapes/my-source-shape --training-shape-id my-new-shape --accelerator-count 16
```

### Flags

```
      --accelerator-count int32           Number of accelerators (overrides source)
      --accelerator-type string           Accelerator type (overrides source)
      --cp int32                          Context-parallel degree (overrides source)
      --deployment-shape-version string   Deployment shape version (overrides source)
      --description string                Description (overrides source)
      --display-name string               Human-readable display name (overrides source)
      --ep int32                          Expert-parallel degree (overrides source)
  -h, --help                              help for clone
      --max-context-length int32          Max supported context length (overrides source)
      --node-count int32                  Node count (overrides source)
      --pp int32                          Pipeline-parallel degree (overrides source)
      --sequence-parallel                 Enable sequence parallelism (overrides source)
      --tp int32                          Tensor-parallel degree (overrides source)
      --trainer-image-tag string          Trainer image tag (overrides source)
      --trainer-mode string               Trainer mode (overrides source)
      --training-shape-id string          ID for the new training shape (required)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl training-shape create
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/training-shape-create

Creates a new training shape.

```
firectl training-shape create [flags]
```

### Examples

```
firectl training-shape create --base-model llama-v2-7b --deployment-shape-version accounts/fireworks/deploymentShapes/rft-qwen3-4b/versions/ra6uiv8w --trainer-image-tag 0.24.10
firectl training-shape create --base-model accounts/fireworks/models/llama-v2-7b --deployment-shape-version accounts/fireworks/deploymentShapes/my-shape/versions/v1 --trainer-image-tag 0.24.10 --accelerator-count 8 --max-context-length 4096 --node-count 1
firectl training-shape create --base-model qwen3-30b-a3b-instruct --deployment-shape-version accounts/fireworks/deploymentShapes/rft-qwen3-30b/versions/v1 --trainer-image-tag 0.32.9 --trainer-mode forward_only --tp 2 --pp 1 --accelerator-type NVIDIA_H200_141GB
```

### Flags

```
      --accelerator-count int32           Number of accelerators
      --accelerator-type string           Accelerator type (e.g. NVIDIA_H200_141GB, NVIDIA_H100_80GB, NVIDIA_A100_80GB)
      --base-model string                 Base model name or ID (required)
      --cp int32                          Context-parallel degree (default server-side 1)
      --deployment-shape-version string   Validated deployment shape version resource name (required)
      --description string                Description of the training shape
      --display-name string               Human-readable display name
      --dry-run                           Print the request proto without running it.
      --ep int32                          Expert-parallel degree (default server-side 1)
  -h, --help                              help for create
      --max-context-length int32          Max supported context length
      --node-count int32                  Node count for multi-node training (default 1)
  -o, --output Output                     Set the output format to "text", "json", or "flag". (default text)
      --pp int32                          Pipeline-parallel degree (default server-side 1)
      --sequence-parallel                 Enable sequence parallelism
      --tp int32                          Tensor-parallel degree (default server-side 1)
      --trainer-image-tag string          Validated trainer runtime image tag (required)
      --trainer-mode string               Trainer mode: policy_trainer, forward_only, or lora_trainer
      --training-shape-id string          Training shape ID (optional, auto-generated if not set)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl training-shape delete
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/training-shape-delete

Deletes a training shape and all its versions.

```
firectl training-shape delete <training-shape-id> [flags]
```

### Examples

```
firectl training-shape delete my-shape
firectl training-shape delete accounts/my-account/trainingShapes/my-shape
```

### Flags

```
  -h, --help   help for delete
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl training-shape get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/training-shape-get

Prints information about a training shape.

```
firectl training-shape get <training-shape-id> [flags]
```

### Examples

```
firectl training-shape get my-shape
firectl training-shape get accounts/my-account/trainingShapes/my-shape
```

### Flags

```
  -h, --help   help for get
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl training-shape list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/training-shape-list

Lists training shapes in the account.

```
firectl training-shape list [flags]
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl training-shape update
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/training-shape-update

Updates a training shape (mutable fields only). Creates a new version automatically.

```
firectl training-shape update [flags]
```

### Examples

```
firectl training-shape update my-shape --trainer-image-tag 0.24.11
firectl training-shape update my-shape --trainer-mode forward_only --node-count 2 --tp 2 --pp 4
```

### Flags

```
      --accelerator-count int32              Number of accelerators
      --accelerator-type string              Accelerator type (e.g. NVIDIA_H200_141GB, NVIDIA_H100_80GB)
      --base-model-weight-precision string   Base model weight precision (e.g. BFLOAT16, FP8)
      --cp int32                             Context-parallel degree (default server-side 1)
      --deployment-shape-version string      Validated deployment shape version resource name
      --description string                   Description of the training shape
      --display-name string                  Human-readable display name
      --dry-run                              Print the request proto without running it.
      --ep int32                             Expert-parallel degree (default server-side 1)
  -h, --help                                 help for update
      --max-context-length int32             Max supported context length
      --node-count int32                     Node count for multi-node training
  -o, --output Output                        Set the output format to "text", "json", or "flag". (default text)
      --pp int32                             Pipeline-parallel degree (default server-side 1)
      --sequence-parallel                    Enable sequence parallelism
      --tp int32                             Tensor-parallel degree (default server-side 1)
      --trainer-image-tag string             Validated trainer runtime image tag
      --trainer-mode string                  Trainer mode: policy_trainer or forward_only
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl training-shape-version get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/training-shape-version-get

Prints information about a training shape version.

```
firectl training-shape-version get [flags]
```

### Examples

```
firectl training-shape-version get accounts/my-account/trainingShapes/my-shape/versions/my-version
firectl training-shape-version get accounts/my-account/trainingShapes/my-shape/versions/latest
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl training-shape-version list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/training-shape-version-list

Lists training shape versions.

```
firectl training-shape-version list [flags]
```

### Flags

```
      --base-model string   If specified, filters versions by the given base model or a compatible latest-validated bucket.
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl training-shape-version update
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/training-shape-version-update

Update a training shape version.

```
firectl training-shape-version update [flags]
```

### Examples

```
firectl training-shape-version update accounts/my-account/trainingShapes/my-shape/versions/my-version --public=true
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for update
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
      --public          Will set the version as public.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl upgrade
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/upgrade

Upgrades the firectl binary to the latest version.

```
firectl upgrade [flags]
```

### Examples

```
sudo firectl upgrade
```

### Flags

```
  -h, --help   help for upgrade
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl user create
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/user-create

Creates a new user.

```
firectl user create [flags]
```

### Examples

```
firectl user create --email="alice.cullen@gmail.com"
firectl user create --service-account --user-id="my-bot"
firectl user create --service-account --user-id="my-agent" --permission-preset=agent
```

### Flags

```
      --display-name string        The display name of the user.
      --dry-run                    Print the request proto without running it.
      --email string               The email address of the user (not required for service accounts).
  -h, --help                       help for create
  -o, --output Output              Set the output format to "text", "json", or "flag". (default text)
      --permission-preset string   Permission preset for the service account. Automatically sets role to "custom".
      --role string                The user's role, must be one of "user", "admin", "contributor", "inference-user", or "custom". (default "user")
      --service-account            Admin only: Create as a service account (email will be auto-generated)
      --user-id string             The ID of the user (required for service accounts).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl user delete
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/user-delete

Deletes a user.

```
firectl user delete [flags]
```

### Examples

```
firectl user delete my-user
firectl user delete accounts/my-account/users/my-user
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for delete
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl user get
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/user-get

Prints information about a user.

```
firectl user get [flags]
```

### Examples

```
firectl user get my-user
firectl user get accounts/my-account/users/my-user
```

### Flags

```
      --dry-run         Print the request proto without running it.
  -h, --help            help for get
  -o, --output Output   Set the output format to "text", "json", or "flag". (default text)
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl user list
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/user-list

Prints all users in the account.

```
firectl user list [flags]
```

### Examples

```
firectl user list
```

### Flags

```
      --filter string       Only resources satisfying the provided filter will be listed. See https://google.aip.dev/160 for the filter grammar.
  -h, --help                help for list
      --no-paginate         List all resources without pagination.
      --order-by string     A list of fields to order by. To specify a descending order for a field, append a " desc" suffix
  -o, --output string       Set the output format to "text" or "json". (default "text")
      --page-size int32     The maximum number of resources to list.
      --page-token string   The page to list. A number from 0 to the total number of pages (number of entities / page size).
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl user update
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/user-update

Updates a user.

```
firectl user update [flags]
```

### Examples

```
firectl user update my-user --display-name="Alice Cullen"
firectl user update accounts/my-account/users/my-user --display-name="Alice Cullen"
firectl user update my-agent --permission-preset=agent
```

### Flags

```
      --display-name string        The display name of the user.
      --dry-run                    Print the request proto without running it.
  -h, --help                       help for update
  -o, --output Output              Set the output format to "text", "json", or "flag". (default text)
      --permission-preset string   Permission preset for the service account. Automatically sets role to "custom".
      --role string                The role of the user. Must be one of {user, admin, contributor, inference-user, custom}.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl version
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/version

Prints the version of firectl

```
firectl version [flags]
```

### Examples

```
firectl version
```

### Flags

```
  -h, --help   help for version
  -l, --long   Print build details.
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# firectl whoami
Source: https://docs.fireworks.ai/tools-sdks/firectl/commands/whoami

Shows the currently authenticated user

```
firectl whoami [flags]
```

### Examples

```
firectl whoami
```

### Flags

```
  -h, --help   help for whoami
```

### Global flags

```
  -a, --account-id string   The Fireworks account ID. If not specified, reads account_id from ~/.fireworks/auth.ini.
      --api-key string      An API key used to authenticate with Fireworks.
  -p, --profile string      fireworks auth and settings profile to use.
```


# Getting started
Source: https://docs.fireworks.ai/tools-sdks/firectl/firectl

Learn to create, deploy, and manage resources using Firectl

Firectl can be installed several ways based on your choice and platform.

<CodeGroup>
  ```bash homebrew theme={null}
  brew tap fw-ai/firectl
  brew install firectl

  # If you encounter a failed SHA256 check, try first running
  brew update
  ```

  ```bash macOS (Apple Silicon) theme={null}
  curl https://storage.googleapis.com/fireworks-public/firectl/stable/darwin-arm64.gz -o firectl.gz
  gzip -d firectl.gz && chmod a+x firectl
  sudo mv firectl /usr/local/bin/firectl
  sudo chown root: /usr/local/bin/firectl
  ```

  ```bash macOS (x86_64) theme={null}
  curl https://storage.googleapis.com/fireworks-public/firectl/stable/darwin-amd64.gz -o firectl.gz
  gzip -d firectl.gz && chmod a+x firectl
  sudo mv firectl /usr/local/bin/firectl
  sudo chown root: /usr/local/bin/firectl
  ```

  ```bash Linux  (x86_64) theme={null}
  wget -O firectl.gz https://storage.googleapis.com/fireworks-public/firectl/stable/linux-amd64.gz
  gunzip firectl.gz
  sudo install -o root -g root -m 0755 firectl /usr/local/bin/firectl
  ```

  ```Text Windows (64 bit) theme={null}
  wget -L https://storage.googleapis.com/fireworks-public/firectl/stable/firectl.exe
  ```
</CodeGroup>

### Sign into Fireworks account

To sign into your Fireworks account:

```bash theme={null}
firectl signin
```

If you have set up [Custom SSO](/accounts/sso) then also pass your account ID:

```bash theme={null}
firectl signin <ACCOUNT_ID>
```

### Check you have signed in

To show which account you have signed into:

```bash theme={null}
firectl whoami
```

### Check your installed version

```bash theme={null}
firectl version
```

### Upgrade to the latest version

```bash theme={null}
sudo firectl upgrade
```


# NIM compatibility
Source: https://docs.fireworks.ai/tools-sdks/nim-compatibility

Point NVIDIA NIM and vLLM OpenAI clients at Fireworks. We translate common extras like chat_template_kwargs so requests don't fail with Extra inputs are not permitted.

If you already call models through **NVIDIA NIM**, **vLLM**, or another OpenAI-compatible stack that sends NIM-style extras, you can point that client at Fireworks without rewriting every request.

Fireworks Chat Completions is OpenAI-compatible. NIM and vLLM often add fields that aren't in the OpenAI schema — for example `chat_template_kwargs` to turn thinking off. Without translation, those requests fail with `400 Extra inputs are not permitted`. Fireworks accepts the common ones and maps them onto native parameters.

For first-party Fireworks apps, prefer native controls such as [`reasoning_effort`](/api-reference/post-chatcompletions). See the [Reasoning guide](/guides/reasoning).

Also see: [OpenAI compatibility](/tools-sdks/openai-compatibility) · [Anthropic compatibility](/tools-sdks/anthropic-compatibility).

## Quickstart

Use any OpenAI-compatible client. Set the base URL to Fireworks and use your Fireworks API key:

```python theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="https://api.fireworks.ai/inference/v1",
    api_key="<FIREWORKS_API_KEY>",
)

response = client.chat.completions.create(
    model="accounts/fireworks/models/kimi-k3",
    messages=[{"role": "user", "content": "Say hi in one word."}],
    # NIM-style: disable thinking without rewriting to reasoning_effort
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

print(response.choices[0].message.content)
```

```bash theme={null}
curl https://api.fireworks.ai/inference/v1/chat/completions \
  -H "Authorization: Bearer $FIREWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "accounts/fireworks/models/kimi-k3",
    "messages": [{"role": "user", "content": "Say hi in one word."}],
    "chat_template_kwargs": {"enable_thinking": false}
  }'
```

Set `model` to a Fireworks model resource name (for example `accounts/fireworks/models/kimi-k3`). Streaming works the same way — translation happens before the response starts.

<Tip>
  When you control the request body yourself, prefer native Fireworks fields (`reasoning_effort`, `prompt_truncate_len`, `response_format`). Use NIM field names when you're migrating an existing NIM or vLLM client and want drop-in behavior.
</Tip>

## Thinking and reasoning

NIM clients usually toggle thinking with `chat_template_kwargs`. Fireworks maps those onto [`reasoning_effort`](/api-reference/post-chatcompletions):

| What you send                                                               | What Fireworks does                                                                          |
| --------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| `chat_template_kwargs: {"enable_thinking": false}` (or `"thinking": false`) | Sets `reasoning_effort` to `"none"` and drops the kwargs object                              |
| `{"enable_thinking": true}` alone (or `"thinking": true`)                   | Drops the kwargs object and leaves thinking at the **model default** (does not force a tier) |
| `{"reasoning_budget": 512}`                                                 | Sets `reasoning_effort` to that integer token budget (when thinking isn't off)               |
| `{"low_effort": true}`                                                      | Sets `reasoning_effort` to `"low"` (when no budget is set)                                   |
| `reasoning_effort: "auto"`                                                  | Drops `"auto"` (Fireworks doesn't accept it) so the model default applies                    |

Other keys inside `chat_template_kwargs` (for example `parallel_reasoning_mode`) are dropped with the object — they aren't applied.

When thinking is on, responses may include `message.reasoning_content`. When thinking is off (`reasoning_effort: "none"`), `reasoning_content` is typically null. Details: [Reasoning](/guides/reasoning).

<Note>
  If both `reasoning_effort: "auto"` and a thinking kwargs mapping are present, Fireworks drops `"auto"` first, then applies the kwargs mapping (so `enable_thinking: false` still becomes `"none"`).
</Note>

### Example: tools with thinking off

```bash theme={null}
curl https://api.fireworks.ai/inference/v1/chat/completions \
  -H "Authorization: Bearer $FIREWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "accounts/fireworks/models/kimi-k3",
    "messages": [{"role": "user", "content": "Call the say tool with x=hi."}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "say",
        "description": "Echo text",
        "parameters": {
          "type": "object",
          "properties": {"x": {"type": "string"}},
          "required": ["x"]
        }
      }
    }],
    "tool_choice": "required",
    "chat_template_kwargs": {"enable_thinking": false}
  }'
```

## Prompt truncation

| NIM / vLLM               | Fireworks                        |
| ------------------------ | -------------------------------- |
| `truncate_prompt_tokens` | Renamed to `prompt_truncate_len` |

If you send both, `prompt_truncate_len` wins.

## Structured outputs (`guided_*`)

NIM/vLLM guided decoding fields map onto Fireworks [`response_format`](/structured-responses/structured-response-formatting):

| NIM / vLLM                  | Fireworks `response_format`                                                                                    |
| --------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `guided_json: <schema>`     | `{"type": "json_schema", "json_schema": {"name": "response", "schema": <schema>}}`                             |
| `guided_grammar: "<abnf>"`  | `{"type": "grammar", "grammar": "<abnf>"}`                                                                     |
| `guided_choice: ["a", "b"]` | `{"type": "json_schema", "json_schema": {"name": "choice", "schema": {"type": "string", "enum": ["a", "b"]}}}` |

If more than one `guided_*` field is present, priority is `guided_json` → `guided_grammar` → `guided_choice`. An explicit `response_format` in the request always wins over guided fields.

`guided_regex` is **not** supported (Fireworks grammars are ABNF-style). That field is stripped so the request doesn't 400 — use `guided_grammar` or native `response_format` instead.

## Fields we ignore (stripped)

These NIM/vLLM extras have no Fireworks Chat Completions equivalent. Fireworks removes them so your request isn't rejected. They do **not** change generation:

`guided_regex`, `guided_decoding_backend`, `stop_token_ids`, `include_stop_str_in_output`, `skip_special_tokens`, `spaces_between_special_tokens`, `best_of`, `use_beam_search`, `add_generation_prompt`, `continue_final_message`, `add_special_tokens`, `detokenize`, `allowed_token_ids`, `bad_words`, `include_reasoning`, `nvext`

<Note>
  `min_tokens` is **kept**. Fireworks supports it natively — it is not stripped.
</Note>

Anything else Fireworks already supports (`top_k`, `min_p`, `repetition_penalty`, and other native fields) passes through unchanged.

## Prefer native parameters when you can

```python theme={null}
client.chat.completions.create(
    model="accounts/fireworks/models/kimi-k3",
    messages=[...],
    reasoning_effort="none",   # instead of chat_template_kwargs.enable_thinking=false
    prompt_truncate_len=4096,  # instead of truncate_prompt_tokens
    response_format={          # instead of guided_json / guided_grammar
        "type": "json_schema",
        "json_schema": {"name": "answer", "schema": {...}},
    },
)
```

## Good to know

* **Models that can't disable reasoning.** Mapping `enable_thinking: false` → `reasoning_effort: "none"` returns 400 on models that don't allow `"none"` (same as sending `reasoning_effort: "none"` directly).
* **`enable_thinking: true` vs LiteLLM.** Fireworks leaves model-default thinking (omits `reasoning_effort`). Some LiteLLM Fireworks mappings send `"medium"` instead. Set an explicit `reasoning_effort` when you need a specific tier.
* **Accepted string efforts** include `low`, `medium`, `high`, `xhigh`, `max`, `none`, and (when the model supports it) `adaptive`. `"auto"` is dropped to the model default.
* **OpenAI-standard traffic is unchanged.** Normal `reasoning_effort` values (`"high"`, `"low"`, …) and `tool_choice: "auto"` are not rewritten.

## Next steps

<CardGroup>
  <Card title="Reasoning" href="/guides/reasoning" icon="brain">
    Native reasoning\_effort, streaming, and tool use
  </Card>

  <Card title="Structured responses" href="/structured-responses/structured-response-formatting" icon="brackets-curly">
    json\_schema and grammar response\_format
  </Card>

  <Card title="OpenAI compatibility" href="/tools-sdks/openai-compatibility" icon="code">
    Use the OpenAI SDK with Fireworks
  </Card>
</CardGroup>


# OpenAI compatibility
Source: https://docs.fireworks.ai/tools-sdks/openai-compatibility



You can use the [OpenAI Python client library](https://github.com/openai/openai-python) to interact with Fireworks. This makes migration of existing applications already using OpenAI particularly easy.

For Anthropic SDK support, see [Anthropic compatibility](/tools-sdks/anthropic-compatibility). For NVIDIA NIM / vLLM request fields such as `chat_template_kwargs`, see [NIM compatibility](/tools-sdks/nim-compatibility).

## Specify endpoint and API key

### Using the OpenAI client

You can use the OpenAI client by initializing it with your Fireworks configuration:

```python theme={null}
from openai import OpenAI

# Initialize with Fireworks parameters
client = OpenAI(
    base_url="https://api.fireworks.ai/inference/v1",
    api_key="<YOUR_FIREWORKS_API_KEY>",
)
```

You can also use environment variables with the client:

```python theme={null}
import os
from openai import OpenAI

# Initialize using environment variables
client = OpenAI(
    base_url=os.environ.get("OPENAI_API_BASE", "https://api.fireworks.ai/inference/v1"),
    api_key=os.environ.get("OPENAI_API_KEY"),  # Set to your Fireworks API key
)
```

### Using environment variables

```shell theme={null}
export OPENAI_API_BASE="https://api.fireworks.ai/inference/v1"
export OPENAI_API_KEY="<YOUR_FIREWORKS_API_KEY>"
```

### Alternative approach

```python theme={null}
import openai

# warning: it has a process-wide effect
openai.api_base = "https://api.fireworks.ai/inference/v1"
openai.api_key = "<YOUR_FIREWORKS_API_KEY>"
```

## Usage

Use OpenAI's SDK how you'd normally would. Just ensure that the `model` parameter refers to one of [Fireworks models](https://fireworks.ai/models).

### Completion

Simple completion API that doesn't modify provided prompt in any way:

```python theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="https://api.fireworks.ai/inference/v1",
    api_key="<YOUR_FIREWORKS_API_KEY>",
)

completion = client.completions.create(
    model="accounts/fireworks/models/llama-v3p1-8b-instruct",
    prompt="The quick brown fox",
)
print(completion.choices[0].text)
```

### Chat Completion

Works best for models fine-tuned for conversation (e.g. llama\*-chat variants):

```python theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="https://api.fireworks.ai/inference/v1",
    api_key="<YOUR_FIREWORKS_API_KEY>",
)

chat_completion = client.chat.completions.create(
    model="accounts/fireworks/models/llama-v3p1-8b-instruct",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant.",
        },
        {
            "role": "user",
            "content": "Say this is a test",
        },
    ],
)
print(chat_completion.choices[0].message.content)
```

## Fine-tuning compatibility

Fireworks fine-tuning uses the same **OpenAI-compatible chat completion format** for training data. If you have datasets formatted for OpenAI SFT, they work on Fireworks with no conversion required — the same `messages` array with `role`, `content`, `tool_calls`, and `weight` fields.

Fireworks also supports additional features in the training schema:

* **Thinking traces** via `reasoning_content` on assistant messages (for models like DeepSeek R1 and Qwen3)
* **Per-message weights** to control which turns the model learns from
* **Per-sample weights** for weighted training
* **Vision inputs** using the same OpenAI-compatible multimodal content format

To get started with fine-tuning, see the [Supervised Fine-Tuning guide](/fine-tuning/fine-tuning-models) or the [Fine-Tuning overview](/fine-tuning/finetuning-intro).

## API compatibility

### Differences

The following options have minor differences:

* `max_tokens`: behaves differently if the model context length is exceeded. If the length of `prompt` or `messages` plus `max_tokens` is higher than the model's context window, `max_tokens` will be adjusted lower accordingly. OpenAI returns an invalid request error in this situation. Control this behavior with the `context_length_exceeded_behavior` parameter:
  * `truncate` (default): Automatically adjusts `max_tokens` to fit within the context window
  * `error`: Returns an error like OpenAI does

### Token usage for streaming responses

OpenAI API returns usage stats (number of tokens in prompt and completion) for non-streaming responses but doesn't for the streaming ones (see [forum post](https://community.openai.com/t/chat-completion-stream-api-token-usage/352964)).

Fireworks API returns usage stats in both cases. For streaming responses, the `usage` field is returned in the very last chunk on the response (i.e. the one having `finish_reason` set). For example:

```bash cURL theme={null}
curl --request POST \
     --url https://api.fireworks.ai/inference/v1/completions \
     --header "accept: application/json" \
     --header "authorization: Bearer $API_KEY" \
     --header "content-type: application/json" \
     --data '{"model": "accounts/fireworks/models/starcoder-16b-w8a16", "prompt": "def say_hello_world():", "max_tokens": 100, "stream": true}'
```

```
data: {..., "choices":[{"text":"\n  print('Hello,","index":0,"finish_reason":null,"logprobs":null}],"usage":null}

data: {..., "choices":[{"text":" World!')\n\n\n","index":0,"finish_reason":null,"logprobs":null}],"usage":null}

data: {..., "choices":[{"text":"say_hello_","index":0,"finish_reason":null,"logprobs":null}],"usage":null}

data: {..., "choices":[{"text":"world()\n","index":0,"finish_reason":"stop","logprobs":null}],"usage":{"prompt_tokens":7,"total_tokens":24,"completion_tokens":17}}

data: [DONE]
```

<Note>
  Note, that if you're using OpenAI SDK, they `usage` field won't be listed in the SDK's structure definition. But it can be accessed directly. For example:

  <CodeGroup>
    ```python Python theme={null}
    for chunk in client.chat.completions.create(stream=True, ...):
        if chunk.usage:  # Available in final chunk
            print(f"Tokens: {chunk.usage.total_tokens}")
    ```

    ```typescript TypeScript theme={null}
    for await (const chunk of await openai.chat.completions.create(...)) {
        console.log((chunk as any).usage);
    }
    ```
  </CodeGroup>
</Note>


# Python SDK
Source: https://docs.fireworks.ai/tools-sdks/python-sdk



The official Python SDK for the Fireworks AI API is available on [GitHub](https://github.com/fw-ai-external/python-sdk) and [PyPI](https://pypi.org/project/fireworks-ai/).

## Fireworks vs. OpenAI SDK

Fireworks is [OpenAI-compatible](/tools-sdks/openai-compatibility), so you can use the OpenAI SDK with Fireworks. The Fireworks SDK offers additional benefits:

* **Better concurrency defaults** — Optimized connection pooling for high-throughput workloads
* **Fireworks-exclusive features** — Access parameters and response fields not available in the OpenAI API
* **Platform automation** — Manage datasets, evals, fine-tuning, and deployments programmatically

## Installation

Requires Python 3.9+ and an API key. See [Getting your API key](/api-reference/introduction#getting-your-api-key) for instructions.

<Note>
  The SDK is currently in alpha. Use the `--pre` flag when installing to get the latest version.
</Note>

<CodeGroup>
  ```bash pip theme={null}
  pip install --pre fireworks-ai
  ```

  ```bash poetry theme={null}
  poetry add --pre fireworks-ai
  ```

  ```bash uv theme={null}
  uv add --pre fireworks-ai
  ```
</CodeGroup>

For detailed usage instructions, see the
[README.md](https://github.com/fw-ai-external/python-sdk#readme). To quickly get
started with serverless, see our [Serverless
Quickstart](/getting-started/quickstart).


# Changelog
Source: https://docs.fireworks.ai/updates/changelog



<Update label="2026-08-14">
  # Serverless deprecation: DeepSeek V4 Flash

  **DeepSeek V4 Flash** is deprecated from serverless. Migrate to **[DeepSeek V4 Flash (0731)](https://app.fireworks.ai/models/fireworks/deepseek-v4-flash-0731)**.
</Update>

<Update label="2026-07-16">
  # SFT learning rate scheduler documentation

  Documented learning rate scheduler settings for supervised fine-tuning jobs, including `constant`, `linear`, and `cosine` schedules via `firectl` and the REST API `lrScheduler` object.
</Update>

<Update label="2026-06-26">
  # Serverless deprecation: Kimi K2.5 and Qwen 3.6 Plus

  **Kimi K2.5** and **Qwen 3.6 Plus** are deprecated from serverless.

  ## **Recommended migrations**

  * **[Kimi K2.5](https://app.fireworks.ai/models/fireworks/kimi-k2p5)** — migrate to **[Kimi K2.6](https://app.fireworks.ai/models/fireworks/kimi-k2p6)**
  * **[Qwen 3.6 Plus](https://app.fireworks.ai/models/fireworks/qwen3p6-plus)** — migrate to **[Qwen 3.7 Plus](https://app.fireworks.ai/models/fireworks/qwen3p7-plus)**
</Update>

<Update label="2026-06-17">
  # Serverless deprecation: MiniMax M2.5

  **MiniMax M2.5** is deprecated from serverless. Migrate to **[MiniMax M2.7](https://app.fireworks.ai/models/fireworks/minimax-m2p7)**.
</Update>

<Update label="2026-06-15">
  # New model: GLM 5.2

  **[GLM 5.2](https://app.fireworks.ai/models/fireworks/glm-5p2)** is now available in the Model Library.
</Update>

<Update label="2026-06-12">
  # New models: Kimi K2.7 Code, MiniMax M3, and Qwen 3.7 Plus

  The following models are now available in the Model Library:

  * **[Kimi K2.7 Code](https://app.fireworks.ai/models/fireworks/kimi-k2p7-code)**
  * **[MiniMax M3](https://app.fireworks.ai/models/fireworks/minimax-m3)**
  * **[Qwen 3.7 Plus](https://app.fireworks.ai/models/fireworks/qwen3p7-plus)**
</Update>

<Update label="2026-06-10">
  # Audio inference and image generation deprecation

  Audio inference and image generation are deprecated.
</Update>

<Update label="2026-05-14">
  # Serverless deprecation: legacy models removed May 14, 2026

  Several legacy serverless models will be decommissioned on **May 14, 2026** to make room for newer, higher-performance releases. This applies **only to serverless** usage; **dedicated deployments are unaffected**.

  ## **Action required**

  If you use any of the models below on serverless, migrate to a recommended replacement **before May 14, 2026**. After that date, they will no longer be available via serverless endpoints.

  ## **Serverless migration guide**

  * **DeepSeek V3.1** — migrate to **[Kimi K2.6](https://app.fireworks.ai/models/fireworks/kimi-k2p6)** or **[GLM 5.1](https://app.fireworks.ai/models/fireworks/glm-5p1)**
  * **DeepSeek V3.2** — migrate to **[Kimi K2.6](https://app.fireworks.ai/models/fireworks/kimi-k2p6)** or **[GLM 5.1](https://app.fireworks.ai/models/fireworks/glm-5p1)**
  * **GLM 4.7** — migrate to **[GLM 5.1](https://app.fireworks.ai/models/fireworks/glm-5p1)**
  * **GLM 5** — migrate to **[GLM 5.1](https://app.fireworks.ai/models/fireworks/glm-5p1)**
  * **Qwen3 8B** — migrate to **[GPT-OSS 20B](https://app.fireworks.ai/models/fireworks/gpt-oss-20b)**
  * **Qwen3 VL 30B A3B Thinking** — migrate to **[Kimi K2.6](https://app.fireworks.ai/models/fireworks/kimi-k2p6)**
  * **Qwen3 VL 30B A3B Instruct** — migrate to **[Kimi K2.6](https://app.fireworks.ai/models/fireworks/kimi-k2p6)**
  * **Llama 3.3 70B Instruct** — migrate to **[GPT-OSS 120B](https://app.fireworks.ai/models/fireworks/gpt-oss-120b)**

  We are continuing to invest in faster, more capable models and appreciate your support during this transition.
</Update>

<Update label="2026-02-05">
  # Video & Audio Models, AWS S3 Training Integration, and SSO Improvements

  ## **Video & Audio Input Models**

  You can now query multimodal models with video and audio inputs for video captioning, scene analysis, and multimodal question answering. Deploy models like Qwen3 Omni and Molmo2 to process video and audio content directly using the Chat Completions API.

  See the [Video & Audio Inputs guide](/guides/video-audio-inputs) for deployment instructions and code examples.

  ## **AWS S3 Integration for Training Datasets**

  Training datasets can now be stored in your own AWS S3 buckets using GCP-to-AWS OIDC federation. This Bring Your Own Bucket (BYOB) approach keeps your data private while enabling secure access during Supervised Fine-Tuning and Reinforcement Fine-Tuning jobs—no long-lived credentials required.

  See the [Secure Training (BYOB) documentation](/fine-tuning/finetuning-intro#training-security) for IAM role setup and usage examples.

  ## **Just-In-Time (JIT) User Provisioning for SSO** (Enterprise)

  JIT user provisioning automatically creates user accounts when users sign in through SSO for the first time. Enable this when configuring your identity provider to eliminate manual user creation.

  See the [SSO documentation](/accounts/sso) for setup instructions.

  ## 📚 Documentation Updates

  * **Video & Audio Inputs:** New guide for processing video and audio with Qwen3 Omni and Molmo2 models ([Video & Audio Inputs](/guides/video-audio-inputs))
  * **AWS S3 Bucket Integration:** BYOB dataset storage for training via OIDC federation ([Secure Training](/fine-tuning/finetuning-intro#training-security))
  * **Rate Limits Clarification:** Expanded documentation on adaptive serverless limits, upper bounds, and monitoring ([Serverless rate limits](/serverless/rate-limits))
  * **Anthropic-Compatible Thinking Parameter:** Control reasoning with `thinking` parameter alongside `reasoning_effort` ([Reasoning](/guides/reasoning))
  * **Scaling from Zero Behavior:** Deployments scaled to zero return `503` immediately with retry guidance ([Autoscaling](/deployments/autoscaling))
  * **Custom Model Configuration:** Generation defaults via `fireworks.json` for LoRA adapters ([Uploading Custom Models](/models/uploading-custom-models))
  * **Service Account Roles:** Role assignment and management for service accounts ([Service Accounts](/accounts/service-accounts))
  * **Free RFT:** Documentation highlighting free Reinforcement Fine-Tuning for models under 16B parameters
  * **Serverless Model Retrieval:** Python SDK and curl examples for programmatically listing serverless models ([FAQ](/faq-new/models-inference/how-to-check-if-a-model-is-available-on-serverless))
  * **Function Tools in Responses API:** Client-executed function tools following OpenAI-compatible format ([Responses API](/guides/response-api))
  * **Autoscaling Metric:** New `prompt_tokens_per_second` load target for prefill-heavy workloads ([Autoscaling](/deployments/autoscaling))
  * **firectl CLI Syntax:** All documentation updated to `firectl <noun> <verb>` syntax

  <Accordion title="Bug Fixes & Minor Improvements">
    - **Model Library Enhancements:** Skeleton loader for faster perceived loading, new filter categories for Embeddings, Reranks, and Agents (Web App)
    - **Service Account Roles:** Role column and role assignment in the Users management page (Web App)
    - **JIT User Provisioning Toggle:** Enable JIT provisioning when configuring SSO identity providers (Web App, Enterprise)
    - **Autoscaling Configuration:** Autoscaling metric and threshold settings in deployment creation form (Web App)
    - **RFT Chunk Size:** Configurable chunk size setting with tooltips for Reinforcement Fine-Tuning creation (Web App)
    - **Batch Inference Form:** Redesigned step-based form flow for batch inference job creation (Web App)
    - **Full Precision Shape:** "Full Precision" deployment shape option displayed in deployment configuration (Web App)
    - **Batch Inference Error Banner:** Error banners now display on batch inference job detail pages (Web App)
    - **Audit Log Resource Column:** Improved readability with tooltip for long resource names (Web App)
    - **API Key Error Messages:** Clearer permission error messages when users lack API key management access (Web App)
    - **Fine-Tuning Cost Display:** Fixed estimated cost display for fine-tuning jobs (Web App)
    - **Batch Inference Max Tokens:** Fallback max value increased to 128,000 tokens (Web App)
    - **Playground Model Select:** Improved performance for model selection dropdown (Web App)
    - **Embedding Model Shapes:** Deployment shapes correctly filtered by embedding model type (Web App)
    - **Audit Logs Command:** New `firectl audit-logs list` command for querying audit logs (CLI)
  </Accordion>
</Update>

<Update label="2026-01-20">
  # Warm-Start Training and Azure Model Uploads

  ## **Warm-Start Training for Reinforcement Fine-Tuning**

  You can now warm-start Reinforcement Fine-Tuning jobs from previously supervised fine-tuned checkpoints using the `--warm-start-from` flag. This enables a streamlined SFT-to-RFT workflow where you first train a model with supervised fine-tuning, then continue training with reinforcement learning.

  See the [Warm-Start Training guide](/fine-tuning/reinforcement-fine-tuning-models#warm-start) for details.

  ## **Azure Federated Identity for Model Uploads**

  Model uploads from Azure Blob Storage now support Azure AD federated identity authentication as an alternative to SAS tokens. This eliminates the need for credential rotation and enables secure, credential-less authentication.

  See the [Uploading Custom Models documentation](/models/uploading-custom-models) for setup instructions.

  ## 📚 Documentation Updates

  * **Warm-Start Training:** New guide for SFT-to-RFT workflows ([Warm-Start Training](/fine-tuning/reinforcement-fine-tuning-models#warm-start))
  * **Azure Federated Identity:** Setup instructions for Azure AD authentication ([Uploading Custom Models](/models/uploading-custom-models))
  * **Preserved Thinking:** Multi-turn reasoning with preserved thinking context ([Reasoning](/guides/reasoning))
  * **GLM 4.7:** Added to models supporting `reasoning_effort` parameter

  <Accordion title="Bug Fixes & Minor Improvements">
    - **RFT Cost Display:** Reinforcement Fine-Tuning job pages now show approximate final cost (Web App)
    - **GPU Information:** Deployments table displays GPU type and count (Web App)
    - **DPO Job Resume:** Preference Fine-Tuning jobs can now be resumed after stopping (Web App, API)
    - **Free Tuning Filter:** New filter in RFT model selector for free-to-fine-tune models (Web App)
    - **Playground Inputs:** Editable number inputs for temperature, top\_p, and other parameters (Web App)
    - **Clone Fine-Tuning Jobs:** Fixed field population when cloning jobs (Web App)
    - **Evaluation Job Errors:** Errors now display with alert banners (Web App)
    - **Invoice CSV Export:** Improved download experience (Web App)
    - **Playground Validation:** Prevents querying deployments with 0 replicas (Web App)
    - **List Models Filter:** `firectl model list` supports `--name` and `--public-only` flags (CLI)
    - **RFT Concurrency:** New `--max-concurrent-rollouts` and `--max-concurrent-evaluations` flags (CLI)
  </Accordion>
</Update>

<Update label="2025-12-22">
  # Playground Categories, New User Roles, Fine-Tuning Improvements, and New Models

  ## **Playground Categories**

  The Playground now features category tabs (LLM, Image, TTS, STT) in the header for easier switching between model types. The playground automatically detects the appropriate category based on the selected model and provides smart defaults for each category.

  ## **User Roles: Contributor and Inference**

  New user roles provide more granular access control for team collaboration:

  * **Contributor**: Read and write access to resources without administrative privileges
  * **Inference**: Read-only access with the ability to run inference on deployments

  Assign these roles when inviting team members to provide appropriate access levels.

  ## **Fine-Tuning Improvements**

  Fine-tuning workflows have been enhanced with several new capabilities:

  * **Stop and Resume Jobs**: Stop running fine-tuning jobs and resume them later from where they left off. Available for Supervised Fine-Tuning and Reinforcement Fine-Tuning jobs.
  * **Clone Jobs**: Quickly create new fine-tuning jobs based on existing job configurations using the Clone action.
  * **Download Output Datasets**: Download output datasets from Reinforcement Fine-Tuning jobs, including individual files or bulk download as a ZIP archive.
  * **Download Rollout Logs**: Download rollout logs from Reinforcement Fine-Tuning jobs for offline analysis.

  ## ✨ New Models

  * **[Gemma 3 12B Instruct](https://app.fireworks.ai/models/fireworks/gemma-3-12b-it)** is now available in the Model Library
  * **[Gemma 3 4B Instruct](https://app.fireworks.ai/models/fireworks/gemma-3-4b-it)** is now available in the Model Library
  * **[Qwen3 Omni 30B A3B Instruct](https://app.fireworks.ai/models/fireworks/qwen3-omni-30b-a3b-instruct)** is now available in the Model Library

  ## 📚 Documentation Updates

  * **Deployment Shapes API:** Added [List Deployment Shape Versions](/api-reference/list-deployment-shape-versions) and [Get Deployment Shape](/api-reference/get-deployment-shape) endpoints for querying available deployment shapes
  * **Evaluator APIs:** Added [Create Evaluator](/api-reference/create-evaluator), [Update Evaluator](/api-reference/update-evaluator), and helper endpoints for evaluator source code, build logs, and upload validation
  * **Fine-Tuning APIs:** Added [Resume DPO Job](/api-reference/resume-dpo-job), [Resume Reinforcement Fine-Tuning Step](/api-reference/resume-reinforcement-fine-tuning-step), [Execute Reinforcement Fine-Tuning Step](/api-reference/execute-reinforcement-fine-tuning-step), and [Get Evaluation Job Log Endpoint](/api-reference/get-evaluation-job-log-endpoint)
  * **SDK Examples:** Added Python SDK example links for direct routing and [supervised fine-tuning](/fine-tuning/fine-tuning-models) workflows

  <Accordion title="Bug Fixes & Minor Improvements">
    - **Deployment Progress Display:** Deployment details page now shows live deployment progress with replica status (pending, downloading, initializing, ready) and error banners (Web App)
    - **Multi-LoRA Display:** Deployment details page now shows all deployed models with expandable sections, not just the default model (Web App)
    - **Audio Usage Charts:** Audio usage metrics are now displayed in a dedicated Voice tab on the Usage page with filtering support (Web App)
    - **Deployment Shape Search:** Improved deployment shape discovery for models where exact base model matches aren't found, using parameter count bucketing (Web App)
    - **Vision Model Auto-Detection:** Vision-language models (Qwen VL, LLaVA, Phi-3 Vision, etc.) now automatically have image input support enabled when uploaded (API)
    - **Dataset Loading UX:** Output dataset tables now stream results with a progress indicator for faster perceived loading (Web App)
    - **File Size Limit:** Dataset uploads now enforce a clear file size limit with improved error messaging (Web App)
    - **Number Input Fields:** Improved number input validation across forms (Web App)
    - **Combobox Responsiveness:** Improved combobox dropdown height on smaller screens (Web App)
    - **Evaluator Editor Scroll:** Prevented accidental page navigation when scrolling inside the evaluator code editor (Web App)
    - **Evaluator Save Dialog:** Fixed overflow issues in the save evaluator dialog (Web App)
    - **Evaluator Selector Labels:** Fixed label rendering in async evaluator select components (Web App)
    - **Deploy Button Validation:** Deploy button is now disabled when the model is not ready (Web App)
    - **Model Metadata:** Fixed missing model metadata display on deployment pages (Web App)
    - **Invoice Display:** Invoice list now shows "paid" status for contract payments (Web App)
    - **Color Palette Update:** Updated UI color palette for improved visual consistency (Web App)
  </Accordion>
</Update>

<Update label="2025-12-15">
  # Reasoning Guide, Prompt Caching Updates, New Models and CLI Updates

  ## **Reasoning Guide**

  A new [Reasoning guide](/guides/reasoning) is now available in the documentation. This comprehensive guide covers:

  * Accessing `reasoning_content` from thinking/reasoning models
  * Controlling reasoning effort with the `reasoning_effort` parameter
  * Streaming with reasoning content
  * Interleaved thinking for multi-step tool-calling workflows

  The guide provides code examples using the Fireworks Python SDK and explains how to work with models that support extended reasoning capabilities.

  ## **Prompt Caching Updates**

  Prompt caching documentation has been updated with expanded guidance:

  * Cached prompt tokens on serverless now cost 50% less than uncached tokens
  * Session affinity routing via the `user` field or `x-session-affinity` header for improved cache hit rates
  * Prompt optimization techniques for maximizing cache efficiency

  See the [Prompt Caching guide](/guides/prompt-caching) for details.

  ## ✨ New Models

  * **[Devstral Small 2 24B Instruct 2512](https://app.fireworks.ai/models/fireworks/devstral-small-2-24b-instruct-2512)** is now available in the Model Library
  * **[NVIDIA Nemotron Nano 3 30B A3B](https://app.fireworks.ai/models/fireworks/nemotron-nano-3-30b-a3b)** is now available in the Model Library

  ## 📚 Documentation Updates

  * **Reasoning Guide:** New documentation for working with reasoning models, including `reasoning_content`, `reasoning_effort`, streaming, and interleaved thinking ([Reasoning](/guides/reasoning))
  * **Recommended Models:** Updated recommendations to include DeepSeek V3.2 for code generation and Kimi K2 Thinking as a GPT-5 alternative ([Recommended Models](/guides/recommended-models))
  * **OpenAI Compatibility:** Removed `stop` sequence documentation as Fireworks is now 1:1 compatible with OpenAI's behavior ([OpenAI Compatibility](/tools-sdks/openai-compatibility))
  * **Evaluator APIs:** Added REST API documentation for Evaluator and Evaluation Job CRUD operations ([Evals API Reference](/api-reference/list-evaluators))
  * **firectl CLI Reference:** Updated with new commands including `cancel dpo-job`, `cancel supervised-fine-tuning-job`, `set-api-key`, `redeem-credit-code`, and evaluator revision management

  <Accordion title="Bug Fixes & Minor Improvements">
    - **Audio Usage Charts:** Added a Voice modality tab to the Usage page for viewing audio-specific usage metrics and charts (Web App)
    - **Deployment Page Redesign:** Redesigned deployment details page with inline action buttons (Edit, Delete, Enable/Disable), reordered metadata sections, and collapsible API examples (Web App)
    - **Deployment API Examples:** API code examples now use the deployment route as the canonical model identifier for clearer usage patterns (Web App)
    - **LoRA Addons Tab:** Renamed "Serverless LoRA" tab to "LoRA Addons" on the deployments dashboard for clarity (Web App)
    - **Evaluator Selector:** Improved evaluator selector UI to display both display name and evaluator ID for easier identification (Web App)
    - **Evaluator Delete Confirmation:** Added confirmation modal with success/error feedback when deleting evaluators (Web App)
    - **Evaluator Code Viewer:** Evaluator source files now load asynchronously, preventing browser freezes with large files (Web App)
    - **Dataset Image Preview:** Dataset preview now properly renders image content in message bubbles and comparison views (Web App)
    - **Model Search:** Improved model search accuracy by restricting results to display name and model ID matches only (Web App)
    - **Fine-Tuning Progress Status:** Fine-tuning job detail pages now show the initial job status immediately on page load (Web App)
    - **Evaluator Test Controls:** Dataset selection and pagination are now disabled while an evaluator test is running to prevent conflicts (Web App)
    - **Repository Name Validation:** Evaluator repository names now validate against GitHub naming conventions (Web App)
    - **Billing Contracts Tab:** Added a Contracts tab to the billing page alongside Invoices for viewing contract details (Web App)
    - **Dataset Size Limit:** Enforced 1GB maximum file size for dataset uploads with clear error messaging (Web App)
    - **Evaluator Documentation Link:** Fixed evaluator documentation links to point to the correct location (Web App)
    - **Session Update Fix:** Fixed an issue with session state updates in the web app (Web App)
    - **Popover Fix:** Fixed popover components nested in dialogs not displaying correctly (Web App)
    - **Per-Replica Status:** Deployment status now shows per-replica counts (pending, downloading, initializing, ready) for better visibility into deployment progress (CLI, API)
    - **set-api-key Command:** The `set-api-key` command is now visible in firectl help output (CLI)
    - **DPO Resume:** Added support for resuming cancelled DPO fine-tuning jobs (API)
    - **Reasoning Effort Normalization:** `reasoning_effort` parameter now accepts boolean values in addition to strings and integers (API)
  </Accordion>
</Update>

<Update label="2025-12-08">
  # DeepSeek V3.2 on Serverless, Cached Token Pricing, and New Models

  ## ☁️ Serverless

  * **[DeepSeek V3.2](https://app.fireworks.ai/models/fireworks/deepseek-v3p2)** is now available on serverless

  ## **Cached Token Pricing Display**

  The Model Library and model detail pages now display cached and uncached input
  token pricing for serverless models that support prompt caching. This gives you
  better visibility into potential cost savings when using prompt caching with
  supported models.

  ## **Evaluations Dashboard Improvements**

  The Evaluations dashboard has been enhanced with new filtering and status tracking capabilities:

  * Status column showing evaluator build state (Active, Building, Failed)
  * Quick filters to filter evaluators and evaluation jobs by status
  * Improved table layout with actions integrated into the status column

  ## ✨ New Models

  * **[DeepSeek V3.2](https://app.fireworks.ai/models/fireworks/deepseek-v3p2)** is now available in the Model Library
  * **[Ministral 3 14B Instruct 2512](https://app.fireworks.ai/models/fireworks/ministral-3-14b-instruct-2512)** is now available in the Model Library
  * **[Ministral 3 8B Instruct 2512](https://app.fireworks.ai/models/fireworks/ministral-3-8b-instruct-2512)** is now available in the Model Library
  * **[Ministral 3 3B Instruct 2512](https://app.fireworks.ai/models/fireworks/ministral-3-3b-instruct-2512)** is now available in the Model Library
  * **[Mistral Large 3 675B Instruct](https://app.fireworks.ai/models/fireworks/mistral-large-3-fp8)** is now available in the Model Library
  * **[Qwen3-VL-32B-Instruct](https://app.fireworks.ai/models/fireworks/qwen3-vl-32b-instruct)** is now available in the Model Library
  * **[Qwen3-VL-8B-Instruct](https://app.fireworks.ai/models/fireworks/qwen3-vl-8b-instruct)** is now available in the Model Library

  ## 📚 Documentation Updates

  * **Reranking Guide:** Added documentation for using the `/rerank` endpoint and `/embeddings` endpoint with `return_logits` for reranking, including parallel batching examples ([Querying Embeddings Models](/guides/querying-embeddings-models))

  <Accordion title="Bug Fixes & Minor Improvements">
    - **Deployment Page Enhancements:** Redesigned deployment detail page with new model header, quick actions (Playground, Go to Model), tabbed API/Info interface, collapsible code examples, and improved GPU count display (Web App)
    - **Login Page Redesign:** New marketing panel with customer testimonials carousel, highlighted platform capabilities, and improved visual design (Web App)
    - **Playground CTA:** Added "Deploy on Demand" button next to "Try the API" for eligible models, making it easier to deploy models directly from the playground (Web App)
    - **Console Navigation Icons:** Updated sidebar icons with a refreshed icon set for improved visual consistency (Web App)
    - **Reinforcement Fine-Tuning Defaults:** Changed default epochs to 1 and increased maximum inference N from 8 to 32 for rollout configuration (Web App)
    - **Failed Job Visibility:** Training progress and loss curves now display for failed and cancelled fine-tuning jobs, helping with debugging (Web App)
    - **Large Dataset Uploads:** Improved upload handling for large JSONL files with progress tracking and direct-to-storage uploads (Web App)
    - **Dataset Preview:** Fixed page freeze issue when previewing datasets with very long text content (Web App)
    - **Loss Chart Y-Axis:** Fixed y-axis scaling on loss charts to properly display the full range of values (Web App)
    - **Model Deletion Dialog:** Improved custom model delete confirmation dialog with better validation and feedback (Web App)
    - **Billing Date Range:** Fixed date range calculation errors on the first day of the month in billing usage views (Web App)
    - **Login Session:** Fixed an issue where expired sessions required manual cookie clearing to log in again (Web App)
    - **Rollout Detail Panel:** Improved rollout log viewing with resizable split panels and better log formatting (Web App)
    - **Checkpoint Promotion:** Added validation and error messages when promoting checkpoints with missing target modules or base model (Web App)
    - **Model Validation:** Added validation before deploying fine-tuned models to ensure the model ID is valid (Web App)
    - **Quota Error Message:** Improved error message clarity when request quota is exceeded in the playground (Web App)
    - **Safari Layout:** Fixed extra spacing in login page marketing panel on Safari browsers (Web App)
  </Accordion>
</Update>

<Update label="2025-12-01">
  # Audit Logs, Dataset Download, Weighted Training for Reinforcement Fine-Tuning, and New Model

  ## **Audit Logs in Web App**

  You can now view and search audit logs directly from the Fireworks web app. The new Audit Logs page provides:

  * Search and filter logs by status and timeframe
  * Detailed view panel for individual log entries
  * Easy navigation from the console sidebar under Account settings

  See the [Audit Logs documentation](/guides/security_compliance/audit_logs) for more information.

  ## **Dataset Download**

  You can now download datasets directly from the Fireworks web app. The new download functionality allows you to:

  * Download individual files from a dataset
  * Download all files at once with "Download All"
  * Access downloads from the Datasets table in the dashboard

  ## **Weighted Training for Reinforcement Fine-Tuning**

  Reinforcement Fine-Tuning now supports per-example weighting, giving you more control over which samples have greater influence during training. This feature mirrors the weighted training functionality already available in Supervised Fine-Tuning.

  See the [Weighted Training documentation](/fine-tuning/reinforcement-fine-tuning-models#weighted-training) for details on the weight field format.

  ## ✨ New Models

  * **[KAT Coder](https://app.fireworks.ai/models/fireworks/kat-coder)** is now available in the Model Library

  <Accordion title="Bug Fixes & Minor Improvements">
    - **Console Navigation:** Redesigned sidebar with organized groups (CREATE, EXPLORE, MANAGE) for easier navigation (Web App)
    - **Fine-Tuning Progress Display:** Training progress (progress/epoch) now displays inline in the job title while Supervised Fine-Tuning jobs are running (Web App)
    - **Reinforcement Fine-Tuning Evaluator Selection:** Evaluators that are still building are now disabled in the selector with a tooltip and status badge (Web App)
    - **Loss Chart Smoothing:** Large datasets (>1000 metrics) now show EMA smoothing by default for improved visibility (Web App)
    - **Checkpoint Restore:** Fixed base model resolution when promoting checkpoints from fine-tuning jobs (Web App)
    - **Deployment Usage Charts:** Fixed usage graph display on deployment details page (Web App)
    - **Evaluation Job Share Links:** Fixed incorrect output dataset share links and improved deep-link behavior for evaluation jobs (Web App)
    - **Embedding Model Deployments:** Embedding models can now be deployed directly from the UI (Web App)
    - **Dataset Download State:** Download option is now disabled for datasets that are still uploading (Web App)
    - **GPU Hints:** Removed invalid GPU hints from region selector in deployment form (Web App)
    - **PEFT Model Shapes:** Fixed deployment shape lookup for PEFT Addon and Live Merge models (Web App)
    - **LoRA Validation:** Improved error messages when LoRA checkpoints are missing the required `language_model.` prefix, with actionable conversion instructions (API)
    - **Reinforcement Fine-Tuning Timeout:** Extended maximum job timeout from 4 to 7 days for longer training runs (API)
    - **Training Job Cancellation:** Added ability to cancel Supervised Fine-Tuning, DPO, and Reinforcement Fine-Tuning jobs via API (API)
    - **Resource Errors:** Improved error messages for capacity-related issues during training and deployment (API)
  </Accordion>
</Update>

<Update label="2025-11-24">
  # Evaluator Improvements, Kimi K2 Thinking on Serverless, and New API Endpoints

  ## **Improved Evaluator Creation Experience**

  The evaluator creation workflow has been significantly enhanced with GitHub template integration. You can now:

  * Fork evaluator templates directly from GitHub repositories
  * Browse and preview templates before using them
  * Create evaluators with a streamlined save dialog
  * View evaluators in a new sortable and paginated table

  ## **MLOps & Observability Integrations**

  New documentation for integrating Fireworks with MLOps and observability tools:

  * [Weights & Biases (W\&B)](/ecosystem/integrations/wandb) integration for experiment tracking during fine-tuning
  * MLflow integration for model management and experiment logging

  ## ✨ New Models

  * **[Kimi K2 Thinking](https://app.fireworks.ai/models/fireworks/kimi-k2-thinking)** is now available in the Model Library
  * **[KAT Dev 32B](https://app.fireworks.ai/models/fireworks/kat-dev-32b)** is now available in the Model Library
  * **[KAT Dev 72B Exp](https://app.fireworks.ai/models/fireworks/kat-dev-72b-exp)** is now available in the Model Library

  ## ☁️ Serverless

  * **[Kimi K2 Thinking](https://app.fireworks.ai/models/fireworks/kimi-k2-thinking)** is now available on serverless

  ## 📚 New REST API Endpoints

  New REST API endpoints are now available for managing Reinforcement Fine-Tuning Steps and deployments:

  * [Create Reinforcement Fine-Tuning Step](/api-reference/create-reinforcement-fine-tuning-step)
  * [List Reinforcement Fine-Tuning Steps](/api-reference/list-reinforcement-fine-tuning-steps)
  * [Get Reinforcement Fine-Tuning Step](/api-reference/get-reinforcement-fine-tuning-step)
  * [Delete Reinforcement Fine-Tuning Step](/api-reference/delete-reinforcement-fine-tuning-step)
  * [Scale Deployment](/api-reference/scale-deployment)
  * [List Deployment Shape Versions](/api-reference/list-deployment-shape-versions)
  * [Get Deployment Shape Version](/api-reference/get-deployment-shape-version)
  * [Get Dataset Download Endpoint](/api-reference/get-dataset-download-endpoint)

  <Accordion title="Bug Fixes & Minor Improvements">
    - **Deployment Region Selector:** Added GPU accelerator hints to the region selector, with Global set as default for optimal availability (Web App)
    - **Preference Fine-Tuning (DPO):** Added to the Fine-Tuning page for training models with human preference data (Web App)
    - **Redeem Credits:** Credit code redemption is now available to all users from the Billing page (Web App)
    - **Model Library Search:** Improved fuzzy search with hybrid matching for better model discovery (Web App)
    - **Cogito Models:** Added Cogito namespace to the Model Library for easier discovery (Web App)
    - **Custom Model Editing:** You can now edit display name and description inline on custom model detail pages (Web App)
    - **Loss Curve Charts:** Fixed an issue where loss curves were not updating in real-time during fine-tuning jobs (Web App)
    - **Deployment Shapes:** Fixed deployment shape selection for fine-tuned models (PEFT and live-merge) (Web App)
    - **Usage Charts:** Fixed replica calculation in multi-series usage charts (Web App)
    - **Session Management:** Removed auto-logout on inactivity for improved user experience (Web App)
    - **Onboarding:** Updated onboarding survey with improved profile and questionnaire flow (Web App)
    - **Fine-Tuning Form:** Max context length now defaults to and is capped by the selected base model's context length (Web App)
    - **Secrets for Evaluators:** Added documentation for using secrets in evaluators to securely call external services (Docs)
    - **Region Selection:** Deprecated regions are now filtered from deployment options (Web App)
    - **Playground:** Embedding and reranker models are now filtered from playground model selection (Web App)
    - **LoRA Rank:** Updated valid LoRA rank range to 4-32 in documentation (Docs)
    - **SFT Documentation:** Added documentation for batch size, learning rate warmup, and gradient accumulation settings (Docs)
    - **Direct Routing:** Added OpenAI SDK code examples for direct routing (Docs)
    - **Recommended Models:** Updated model recommendations with migration guidance from Claude, GPT, and Gemini (Docs)
  </Accordion>
</Update>

<Update label="2025-11-12">
  ## ☀️ Sunsetting Build SDK

  The Build SDK is being deprecated in favor of a new Python SDK generated
  directly from our REST API. The new SDK is more up-to-date, flexible, and
  continuously synchronized with our REST API. Please note that the last version
  of the Build SDK will be `0.19.20`, and the new SDK will start at `1.0.0`.
  Python package managers will not automatically update to the new SDK, so you
  will need to manually update your dependencies and refactor your code.

  Existing codebases using the Build SDK will continue to function as before and
  will not be affected unless you choose to upgrade to the new SDK version.

  The new SDK replaces the Build SDK's `LLM` and `Dataset` classes with REST
  API-aligned methods. If you upgrade to version `1.0.0` or later, you will need
  to migrate your code.

  ## 🚀 Improved RFT Experience

  We've drastically improved the RFT experience with better reliability,
  developer-friendly SDK for hooking up your existing agents, support for
  multi-turn training, better observability in our Web App, and better overall
  developer experience.

  See [Reinforcement Fine-Tuning](/fine-tuning/reinforcement-fine-tuning-models) for more details.
</Update>

<Update label="2025-08-22">
  ## Supervised Fine-Tuning

  We now support supervised fine tuning with separate thinking traces for reasoning models (e.g. DeepSeek R1, GPT OSS, Qwen3 Thinking etc) that ensures training-inference consistency. An example including thinking traces would look like:

  ```json theme={null}
    {
      "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the capital of France?"}, 
        {"role": "assistant", "content": "Paris.", "reasoning_content": "The user is asking about the capital city of France, it should be Paris."}
      ]
    }
    {
      "messages": [
        {"role": "user", "content": "What is 1+1?"},
        {"role": "assistant", "content": "2", "weight": 0, "reasoning_content": "The user is asking about the result of 1+1, the answer is 2."},
        {"role": "user", "content": "Now what is 2+2?"},
        {"role": "assistant", "content": "4", "reasoning_content": "The user is asking about the result of 2+2, the answer should be 4."}
      ]
    }
  ```

  We are also properly supporting multi-turn fine tuning (with or without thinking traces) for GPT OSS model family that ensures training-inference consistency.
</Update>

<Update label="2025-08-10">
  ## Supervised Fine-Tuning

  We now support Qwen3 MoE model (Qwen3 dense models are already supported) and GPT OSS models for supervised fine-tuning. GPT OSS model fine tunning support is single-turn without thinking traces at the moment.
</Update>

<Update label="2025-07-29">
  ## 🎨 Vision-Language Model Fine-Tuning

  You can now fine-tune Vision-Language Models (VLMs) on Fireworks AI using the Qwen 2.5 VL model family.
  This extends our Supervised Fine-tuning V2 platform to support multimodal training with both images and text data.

  **Supported models:**

  * Qwen 2.5 VL 3B Instruct
  * Qwen 2.5 VL 7B Instruct
  * Qwen 2.5 VL 32B Instruct
  * Qwen 2.5 VL 72B Instruct

  **Features:**

  * Fine-tune on datasets containing both images and text in JSONL format with base64-encoded images
  * Support for up to 64K context length during training
  * Built on the same Supervised Fine-tuning V2 infrastructure as text models

  See the [VLM fine-tuning documentation](/fine-tuning/fine-tuning-models#vision-fine-tuning) for setup instructions and dataset formatting requirements.

  ## 🔧 Build SDK: Deployment Configuration Application Requirement

  The Build SDK now requires you to call `.apply()` to apply any deployment configurations to Fireworks when using `deployment_type="on-demand"` or `deployment_type="on-demand-lora"`. This change ensures explicit control over when deployments are created and helps prevent accidental deployment creation.

  **Key changes:**

  * `.apply()` is now required for on-demand and on-demand-lora deployments
  * Serverless deployments do not require `.apply()` calls
  * If you do not call `.apply()`, you are expected to set up the deployment through the deployment page at [https://app.fireworks.ai/dashboard/deployments](https://app.fireworks.ai/dashboard/deployments)

  **Migration guide:**

  * Add `llm.apply()` after creating LLM instances with `deployment_type="on-demand"` or `deployment_type="on-demand-lora"`
  * No changes needed for serverless deployments
  * See updated documentation for examples and best practices

  This change improves deployment management and provides better control over resource creation.

  <Note>
    This applies to Python SDK version `>=0.19.14`.
  </Note>
</Update>

<Update label="2025-07-23">
  ## 🚀 Bring Your Own Rollout and Reward Development for Reinforcement Learning

  You can now develop your own custom rollout and reward functionality while using
  Fireworks to manage the training and deployment of your models. This gives you
  full control over your reinforcement learning workflows while leveraging
  Fireworks' infrastructure for model training and deployment.

  See the new [LLM.reinforcement\_step()](/tools-sdks/python-client/sdk-reference#reinforcement_step) method and [ReinforcementStep](/tools-sdks/python-client/sdk-reference#reinforcementstep) class for usage examples and details.
</Update>

<Update label="2025-07-16">
  ## Supervised Fine-Tuning V2

  We now support Llama 4 MoE model supervised fine-tuning (Llama 4 Scout, Llama 4 Maverick, Text only).
</Update>

<Update label="2025-07-10">
  ## 🏗️ Build SDK `LLM` Deployment Logic Refactor

  Based on early feedback from users and internal testing, we've refactored the
  `LLM` class deployment logic in the Build SDK to make it easier to understand.

  **Key changes:**

  * The `id` parameter is now required when `deployment_type` is `"on-demand"`
  * The `base_id` parameter is now required when `deployment_type` is `"on-demand-lora"`
  * The `deployment_display_name` parameter is now optional and defaults to the filename where the LLM was instantiated

  A new deployment will be created if a deployment with the same `id` does not
  exist. Otherwise, the existing deployment will be reused.
</Update>

<Update label="2025-07-02">
  ## 🚀 Support for Responses API in Python SDK

  You can now use the Responses API in the Python SDK. This is useful if you want to use the Responses API in your own applications.

  See the [Responses API guide](/guides/response-api) for usage examples and details.
</Update>

<Update label="2025-07-01">
  ## Support for LinkedIn authentication

  You can now log in to Fireworks using your LinkedIn account. This is useful if
  you already have a LinkedIn account and want to use it to log in to Fireworks.

  To log in with LinkedIn, go to the [Fireworks login
  page](https://fireworks.ai/login) and click the "Continue with LinkedIn"
  button.

  You can also log in with LinkedIn from the CLI using the `firectl login`
  command.

  **How it works:**

  * Fireworks uses your LinkedIn primary email address for account identification
  * You can switch between different Fireworks accounts by changing your LinkedIn primary email
  * See our [LinkedIn authentication FAQ](/faq-new/account-access/what-email-does-linkedin-authentication-use) for detailed instructions on managing email addresses
</Update>

<Update label="2025-06-30">
  ## Support for GitHub authentication

  You can now log in to Fireworks using your GitHub account. This is useful if
  you already have a GitHub account and want to use it to log in to Fireworks.

  To log in with GitHub, go to the [Fireworks login
  page](https://fireworks.ai/login) and click the "Continue with GitHub"
  button.

  You can also log in with GitHub from the CLI using the `firectl login`
  command.

  ## 🚨 Document Inlining Deprecation

  Document Inlining has been deprecated and is no longer available on the Fireworks platform. This feature allowed LLMs to process images and PDFs through the chat completions API by appending `#transform=inline` to document URLs.

  **Migration recommendations:**

  * For image processing: Use Vision Language Models (VLMs) like [Qwen2.5-VL 32B Instruct](https://app.fireworks.ai/models/fireworks/qwen2p5-vl-32b-instruct)
  * For PDF processing: Use dedicated PDF processing libraries combined with text-based LLMs
  * For structured extraction: Leverage our [structured responses](/structured-responses/structured-response-formatting) capabilities

  For assistance with migration, please contact our support team or visit our [Discord community](https://discord.gg/fireworks-ai).
</Update>

<Update label="2025-06-24">
  ## 🎯 Build SDK: Reward-kit integration for evaluator development

  The Build SDK now natively integrates with [reward-kit](https://github.com/fw-ai-external/reward-kit) to simplify evaluator development for [Reinforcement Fine-Tuning (RFT)](/fine-tuning/reinforcement-fine-tuning-models). You can now create custom evaluators in Python with automatic dependency management and seamless deployment to Fireworks infrastructure.

  **Key features:**

  * Native reward-kit integration for evaluator development
  * Automatic packaging of dependencies from `pyproject.toml` or `requirements.txt`
  * Local testing capabilities before deployment
  * Direct integration with Fireworks datasets and evaluation jobs
  * Support for third-party libraries and complex evaluation logic

  See our [Developing Evaluators](/tools-sdks/python-client/developing-evaluators) guide to get started with your first evaluator in minutes.

  ## Added new Responses API for advanced conversational workflows and integrations

  * Continue conversations across multiple turns using the `previous_response_id` parameter to maintain context without resending full history
  * Stream responses in real time as they are generated for responsive applications
  * Control response storage with the `store` parameter—choose whether responses are retrievable by ID or ephemeral

  See the [Response API guide](/guides/response-api) for usage examples and details.
</Update>

<Update label="2025-06-13">
  ## Supervised Fine-Tuning V2

  Supervised Fine-Tuning V2 released.

  **Key features:**

  * Supports Qwen 2/2.5/3 series, Phi 4, Gemma 3, the Llama 3 family, Deepseek V2, V3, R1
  * Longer context window up to full context length of the supported models
  * Multi-turn function calling fine-tuning
  * Quantization aware training

  More details in the [blogpost](https://fireworks.ai/blog/supervised-finetuning-v2).

  ## Reinforcement Fine-Tuning (RFT)

  Reinforcement Fine-Tuning released. Train expert models to surpass closed source frontier models through verifiable reward. More details in [blospost](https://fireworks.ai/blog/reinforcement-fine-tuning-models).
</Update>

<Update label="2025-05-20">
  ## Diarization and batch processing support added to audio inference

  See our [blog post](https://fireworks.ai/blog/audio-summer-updates-and-new-features) for details.
</Update>

<Update label="2025-05-19">
  ## 🚀 Easier & faster LoRA fine-tune deployments on Fireworks

  You can now deploy a LoRA fine-tune with a single command and get speeds that approximately match the base model:

  ```bash theme={null}
  firectl deployment create "accounts/fireworks/models/<MODEL_ID of lora model>"
  ```

  Previously, this involved two distinct steps, and the resulting deployment was slower than the base model:

  1. Create a deployment using `firectl deployment create "accounts/fireworks/models/<MODEL_ID of base model>" --enable-addons`
  2. Then deploy the addon to the deployment: `firectl load-lora <MODEL_ID> --deployment <DEPLOYMENT_ID>`

  For more information, see our [deployment documentation](https://docs.fireworks.ai/models/deploying#deploying-to-on-demand).

  <Note>
    This change is for dedicated deployments with a single LoRA. You can still deploy multiple LoRAs on a deployment as described in the documentation.
  </Note>
</Update>