# Tinybird Documentation (FWD)

Generated on: 2026-08-14T13:58:06.290Z

URL: https://www.tinybird.co/docs/forward/support
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Support \| Tinybird"
theme-color: "#171612"
description: "Tinybird support plans, scope, channels, hours, and how to request help."
inkeep:version: "forward"
---

# Support (https://www.tinybird.co/docs/forward/support#support)

Tinybird provides support through different channels depending on your plan. Free workspaces can request support only through the Community Slack. Developer includes email support. SaaS and Enterprise include Standard or Premier support with priority email, live chat, and a dedicated customer Slack channel.

See [Pricing plans](/docs/forward/pricing) for plan details.

## Support scope (https://www.tinybird.co/docs/forward/support#support-scope)

We divide support into three categories. Each covers different needs and engagement levels.

### Technical support (https://www.tinybird.co/docs/forward/support#technical-support)

Technical support resolves issues you encounter while using Tinybird:

- Production incident investigation
- Troubleshooting errors and blocked workflows
- Potential product bug triage
- Operational and authentication issues
- Questions about existing features
- Supported usage and configuration guidance

### Customer Success (https://www.tinybird.co/docs/forward/support#customer-success)

Customer Success focuses on long-term outcomes, not urgent incidents:

- Adoption guidance
- Success planning
- High-level architecture reviews
- Usage optimization
- Performance recommendations

### Professional Services (https://www.tinybird.co/docs/forward/support#professional-services)

Professional Services is a separate paid engagement, not included with Standard or Premier support:

- Hands-on implementation
- Migrations
- Detailed architecture design
- Deep optimization
- Pipeline building
- Training

## Support plans (https://www.tinybird.co/docs/forward/support#support-plans)

We offer two support plans for SaaS and Enterprise: Standard and Premier.

Standard is included with SaaS and Enterprise. Premier is available for SaaS S-24+ and Enterprise. Your customer agreement defines your support plan, pricing, and terms.

### Standard (https://www.tinybird.co/docs/forward/support#standard)

Standard includes:

- Priority email
- Live chat
- Dedicated customer Slack channel

### Premier (https://www.tinybird.co/docs/forward/support#premier)

Premier includes everything in Standard plus:

- Designated Customer Success engineer
- Prioritized SLA
- On-call requests
- Monthly partnership meeting
- Onboarding support
Premier pricing depends on your plan, deployment, and customer agreement. Support charges don't depend on ticket volume or platform activity. On-call allowances vary by deployment and agreement. For details, use the [pricing calculator](https://www.tinybird.co/pricing/calculator) or [contact sales](https://www.tinybird.co/contact-us).

## Support channels (https://www.tinybird.co/docs/forward/support#support-channels)

| Plan | Channels |
| --- | --- |
| Free | [Community Slack](/docs/community) only |
| Developer | Community Slack and email support at [support@tinybird.co](mailto:support@tinybird.co) |
| SaaS and Enterprise (Standard or Premier) | Priority email, live chat, Community Slack, and a dedicated customer Slack channel |

## Support hours and response commitments (https://www.tinybird.co/docs/forward/support#support-hours-and-response-commitments)

Our email support at [support@tinybird.co](mailto:support@tinybird.co) is available from 9:00 AM to 6:30 PM CET on a Business Day. Other agreed support channels follow those hours unless your customer agreement says otherwise.

Response commitments depend on your plan, incident severity, and customer agreement. We don't provide default 24/7 support. Out-of-hours support is only available when specifically agreed and for Critical or High severity incidents.

Your customer agreement is the source of truth for hours, response commitments, and severity definitions. See the [Terms and Conditions](https://www.tinybird.co/terms-and-conditions).

## How to contact support (https://www.tinybird.co/docs/forward/support#how-to-contact-support)

To help us investigate quickly, include the following when you contact support:

- Organization or Workspace ID. See [Workspace ID](/docs/forward/core-concepts/workspaces#workspace-id).
- Affected resources (Data Sources, Pipes, Endpoints, Connections)
- Timestamps and timezone
- Request or query IDs
- Exact error messages
- Expected and actual behavior
- Recent changes (deploys, schema changes, configuration updates)
- Steps to reproduce the issue
Never send API tokens or other secrets to support. We never ask for them.

## Integrated troubleshooting (https://www.tinybird.co/docs/forward/support#integrated-troubleshooting)

Tinybird tries to give you direct feedback and notifications if it spots anything going wrong. Use Tinybird's [Service Data Sources](/docs/forward/monitoring/service-datasources) to get more details on what's going on in your data and queries.

## Recover deleted items (https://www.tinybird.co/docs/forward/support#recover-deleted-items)

Tinybird creates and backs up daily snapshots and retains them for 7 days.

## Recover data from quarantine (https://www.tinybird.co/docs/forward/support#recover-data-from-quarantine)

The quickest way to recover rows from quarantine is to fix the cause of the errors and then reingest the data. See [Recover data from quarantine](/docs/forward/guides/quarantine#recovering-rows-from-quarantine).

You can also use the [Service Data Sources](/docs/forward/monitoring/service-datasources), like `datasources_ops_log`.

## Useful links (https://www.tinybird.co/docs/forward/support#useful-links)

- [Community Slack](/docs/community)
- email support: [support@tinybird.co](mailto:support@tinybird.co)
- [Service status](https://status.tinybird.co/)
- [Terms and Conditions](https://www.tinybird.co/terms-and-conditions)
- [Pricing calculator](https://www.tinybird.co/pricing/calculator)
- [Contact sales](https://www.tinybird.co/contact-us)
- [Service Data Sources](/docs/forward/monitoring/service-datasources)

---

URL: https://www.tinybird.co/docs/forward/quickstarts
Last update: 2026-07-29T12:18:04.000Z
Content:
---
title: "Quickstarts \| Tinybird"
theme-color: "#171612"
description: "Start building on managed ClickHouse with Tinybird."
inkeep:version: "forward"
---

# Quickstarts (https://www.tinybird.co/docs/forward/quickstarts#quickstarts)

Tinybird combines managed ClickHouse® with a developer experience and tooling fine-tuned for agentic coding and modern development workflows. Build analytics features by ingesting data, transforming it with SQL, and exposing the results to applications, tools, and agents.

## What Tinybird adds to ClickHouse (https://www.tinybird.co/docs/forward/quickstarts#what-tinybird-adds-to-clickhouse)

- Ingest streaming and batch data through the Events API and managed [Connections](/docs/forward/core-concepts/connections) for Kafka, Amazon S3, and Google Cloud Storage.
- Define Data Sources, Connections, Pipes, API Endpoints, and Materialized Views as code with TypeScript, Python, or datafiles.
- Develop with [Tinybird Local](/docs/forward/core-concepts/tinybird-local) and isolated [Cloud Branches](/docs/forward/development-workflow/using-cloud-branches), then deploy through [CI/CD](/docs/forward/development-workflow/cicd).
- Query data through Endpoints, the [Query API](/docs/api-reference/query-api), the [ClickHouse® interface](/docs/forward/query-data/clickhouse-interface), or [MCP](/docs/forward/query-data/mcp).

## Choose how to build (https://www.tinybird.co/docs/forward/quickstarts#choose-how-to-build)

- <a href="/docs/forward/quickstarts/typescript-sdk"></a> TypeScript SDK

Define Data Sources and Endpoints with typed resources.
- <a href="/docs/forward/quickstarts/python-sdk"></a> Python SDK

Define resources in Python and deploy with the Tinybird CLI.
- <a href="/docs/forward/quickstarts/cli"></a> Tinybird CLI

Use datafiles, the Tinybird CLI, and cloud branches.

## Not sure where to start? (https://www.tinybird.co/docs/forward/quickstarts#not-sure-where-to-start)

- Prefer SQL-first workflows and datafiles: start with [Tinybird CLI](/docs/forward/quickstarts/cli).
- Want typed resources and a TypeScript-native workflow: start with [TypeScript SDK](/docs/forward/quickstarts/typescript-sdk).
- Want Python-native resource definitions and runtime usage: start with [Python SDK](/docs/forward/quickstarts/python-sdk).

## Work with Tinybird using AI agents (https://www.tinybird.co/docs/forward/quickstarts#work-with-tinybird-using-ai-agents)

[Tinybird Agent Skills](/docs/forward/development-workflow/agent-skills) give compatible coding agents instructions for creating and reviewing Tinybird project files, SQL, tests, and deployments. Install them in your coding environment:

Install Tinybird Agent Skills
```
npx skills add tinybirdco/tinybird-agent-skills
```
The [Tinybird MCP server](/docs/forward/query-data/mcp) gives an agent Token-scoped access to a live Workspace. The agent can inspect and query data and call published Endpoints according to the Token's permissions. Use Agent Skills and MCP together when an agent needs both project guidance and access to live Workspace data.

## Frequently asked questions about Tinybird (https://www.tinybird.co/docs/forward/quickstarts#frequently-asked-questions-about-tinybird)

### Why use Tinybird instead of managing ClickHouse yourself? (https://www.tinybird.co/docs/forward/quickstarts#why-use-tinybird-instead-of-managing-clickhouse-yourself)

Self-hosting ClickHouse requires you to operate clusters, sharding, replicas, security, monitoring, updates, backups, and deployment pipelines. Tinybird manages the ClickHouse infrastructure and adds ingestion, API publishing, authentication, and development tooling so you can focus on building analytics features. See [Migrate from ClickHouse self-hosted to Tinybird](/docs/forward/guides/migrate-from-clickhouse).

### When should you choose Tinybird instead of ClickHouse Cloud? (https://www.tinybird.co/docs/forward/quickstarts#when-should-you-choose-tinybird-instead-of-clickhouse-cloud)

Choose Tinybird when you need an app development workflow around ClickHouse®: managed ingestion, resources as code, authenticated API Endpoints, Tinybird Local, Cloud Branches, CI/CD, and MCP. You can still connect existing SQL clients and BI tools through the [ClickHouse® interface](/docs/forward/query-data/clickhouse-interface).

### Does Tinybird support data branching and schema iteration? (https://www.tinybird.co/docs/forward/quickstarts#does-tinybird-support-data-branching-and-schema-iteration)

Yes. [Cloud Branches](/docs/forward/core-concepts/branches) create isolated environments where you can change schemas, queries, APIs, and connector ingestion without affecting production. A Branch can attach the latest production partition without copying the underlying data, so you can validate changes before deploying them to your main Workspace.

### How do you define and deploy Tinybird resources as code? (https://www.tinybird.co/docs/forward/quickstarts#how-do-you-define-and-deploy-tinybird-resources-as-code)

Define Tinybird resources with the [TypeScript SDK](/docs/forward/quickstarts/typescript-sdk), [Python SDK](/docs/forward/quickstarts/python-sdk), or [datafiles](/docs/forward/dev-reference/datafiles). Commit them to Git, test with Tinybird Local or Cloud Branches, and deploy with the Tinybird CLI or [CI/CD](/docs/forward/development-workflow/cicd).

### Which ingestion sources and managed Connections does Tinybird support? (https://www.tinybird.co/docs/forward/quickstarts#which-ingestion-sources-and-managed-connections-does-tinybird-support)

Use the [Events API](/docs/forward/ingest-data/events-api) for HTTP streaming, managed Connections for [Kafka](/docs/forward/ingest-data/connectors/kafka), [Amazon S3](/docs/forward/ingest-data/connectors/s3), and [Google Cloud Storage](/docs/forward/ingest-data/connectors/gcs), or upload local files. Connections store credentials and integration settings and deploy with the rest of your Tinybird project.

### How can apps and AI agents query Tinybird? (https://www.tinybird.co/docs/forward/quickstarts#how-can-apps-and-ai-agents-query-tinybird)

Apps can call published [Endpoints](/docs/forward/core-concepts/api-endpoints), run SQL through the [Query API](/docs/api-reference/query-api), or use the ClickHouse® interface. AI agents can use the [Tinybird MCP server](/docs/forward/query-data/mcp) to query permitted data and call published Endpoints. [Tokens](/docs/forward/core-concepts/tokens) control which resources each client or agent can access.

### Which Tinybird pricing plans are available? (https://www.tinybird.co/docs/forward/quickstarts#which-tinybird-pricing-plans-are-available)

Tinybird offers Free, Developer, and SaaS plans on shared infrastructure, plus an Enterprise plan with a dedicated ClickHouse® cluster. See [Pricing plans](/docs/forward/pricing) for plan features and the current [Tinybird pricing](https://www.tinybird.co/pricing) for prices.

---

URL: https://www.tinybird.co/docs/forward/query-data
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Query data \| Tinybird"
theme-color: "#171612"
description: "Learn the ways to query data in Tinybird."
inkeep:version: "forward"
---

# Query data (https://www.tinybird.co/docs/forward/query-data#query-data)

Tinybird gives you several ways to query data, depending on where the query runs and who consumes the result.

Use [API Endpoints](/docs/forward/query-data/api-endpoints) when an app, dashboard, or agent needs a stable HTTP API. Use the Query API or ClickHouse® interface when a tool needs to run ad hoc SQL. Use Playgrounds, Explorations, and Time Series when you are investigating data inside Tinybird Cloud.

## Query from outside Tinybird (https://www.tinybird.co/docs/forward/query-data#query-from-outside-tinybird)

**[API Endpoints](/docs/forward/query-data/api-endpoints)** publish the result of a Pipe as a REST API. They are the production path for serving query results to applications, user interfaces, dashboards, and agents. Use Tinybird's [templating language](/docs/forward/core-concepts/templating-language) to add typed query parameters, filters, sorting, and limits.

**[Query API](/docs/forward/query-data/sql-api)** runs SQL directly against Pipes and Data Sources in your Workspace through the `/v0/sql` endpoint. Use it for internal tools, automation, notebooks, scripts, and ad hoc queries that do not need a published Endpoint contract.

**[ClickHouse® interface](/docs/forward/query-data/clickhouse-interface)** lets ClickHouse-compatible clients, BI tools, and SQL IDEs query Tinybird Data Sources over HTTPS.

**[MCP](/docs/forward/query-data/mcp)** lets AI agents query your Workspace and use published Endpoints as tools through the [Model Context Protocol](https://modelcontextprotocol.io/).

## Query inside Tinybird (https://www.tinybird.co/docs/forward/query-data#query-inside-tinybird)

**[Playgrounds](/docs/forward/query-data/playgrounds)** let you write and test SQL interactively before moving finished logic into Pipes.

**[Explorations](/docs/forward/query-data/explorations)** is a conversational UI in Tinybird Cloud for asking questions about your data in natural language.

**[Time Series](/docs/forward/query-data/time-series)** visualizes time-based metrics from Data Sources, Service Data Sources, and Endpoints.

## Next steps (https://www.tinybird.co/docs/forward/query-data#next-steps)

- Learn how to [query Endpoints](/docs/forward/query-data/api-endpoints).
- Learn about the [Query API](/docs/forward/query-data/sql-api).
- Learn about the [templating language](/docs/forward/core-concepts/templating-language).
- Learn about the [ClickHouse interface](/docs/forward/query-data/clickhouse-interface).
- Learn about [Explorations](/docs/forward/query-data/explorations), [Playgrounds](/docs/forward/query-data/playgrounds), and [Time Series](/docs/forward/query-data/time-series).

---

URL: https://www.tinybird.co/docs/forward/pricing
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Pricing Plans \| Tinybird"
theme-color: "#171612"
description: "Choose the Tinybird plan that fits your needs. Available plans are Free, Developer, SaaS, and Enterprise."
inkeep:version: "forward"
---

# Pricing plans (https://www.tinybird.co/docs/forward/pricing#pricing-plans)

Plans describe what your organization can do in Tinybird, the compute resources at your disposal, and how usage is billed. All plan features, limits, and resources are applied at the organization level and shared across all workspaces within your organization.

To see current prices, see [Pricing](https://www.tinybird.co/pricing).

## Available plans (https://www.tinybird.co/docs/forward/pricing#available-plans)

| Plan | Infrastructure | What you get | When to upgrade |
| --- | --- | --- | --- |
| [Free](/docs/forward/pricing/free) | Shared | - All Tinybird features, with limitations.
  - Use Tinybird without time limits.
  - You can upgrade at any time. | - You need more resources.
  - You're ready for production.
  - You want Tinybird support. |
| [Developer](/docs/forward/pricing/developer) | Shared infrastructure without a commitment | - More resources and fewer limits.
  - Select compute needs of up to 8 vCPUs.
  - vCPU overages for usage exceeding your plan size. | - You need more than 8 vCPUs on shared infrastructure.
  - You need credit-based billing with a commitment.
  - You need a dedicated cluster. |
| [SaaS](/docs/forward/pricing/saas) | Shared infrastructure with a commitment | - Shared infrastructure, 4+ vCPUs.
  - Credit-based billing with a 12-month commitment.
  - QPS and active minutes overages. | - You need a dedicated cluster.
  - You need private networking. |
| [Enterprise](/docs/forward/pricing/enterprise) | Dedicated | - Your own ClickHouse® cluster.
  - No resource sharing.
  - No predefined limits; scale with replica SKUs. | - You need to adjust cluster size or add private networking. |
You can upgrade from Free and Developer plans at any time. SaaS and Enterprise plans require [contacting sales](https://www.tinybird.co/contact-us).

## Features included in all plans (https://www.tinybird.co/docs/forward/pricing#features-included-in-all-plans)

- Unlimited seats
- Unlimited source connections
- Unlimited tables, pipes, and endpoints
- Multi-cloud, multi-region support
- RBAC and Workspace administrator controls. See [Organizations](/docs/forward/core-concepts/organizations).
- SOC2 Type II and HIPAA compliance. See [Compliance](/docs/forward/compliance).

## Deployment options (https://www.tinybird.co/docs/forward/pricing#deployment-options)

Plans run on one of two infrastructure types:

- **Shared infrastructure for Free, Developer, and SaaS plans**: You share resources with other customers. Each plan page documents billing and plan-specific limits.
- **Dedicated infrastructure for Enterprise plans**: Your own isolated ClickHouse® cluster. There are no predefined resource limits. Billing depends on replica SKUs and usage. See [Enterprise](/docs/forward/pricing/enterprise).

## On-demand CPUs (https://www.tinybird.co/docs/forward/pricing#on-demand-cpus)

On-demand CPUs cover temporary compute capacity for jobs that need dedicated compute instances. On-demand compute is available on both shared and dedicated infrastructure.

This currently applies to:

- Deployment populate jobs.
- Copy Pipe jobs run with `--on-demand-compute`.
Deployment populates use on-demand compute automatically for large data migrations when one of these thresholds is met:

- Data size exceeds 50 GB.
- Row count exceeds 100 million rows.
Copy Pipe jobs use on-demand compute only when you explicitly run them with `--on-demand-compute`.

On-demand compute uses 64-core instances by default. You can request smaller instances for less resource-intensive workloads. Contact Tinybird support for configuration options.

Billing is based on the actual compute time used, measured in credits per core per minute. Pricing varies by region:

| Region | Price (Credits per core per minute) |
| --- | --- |
| `aws-us-east-1` | 0.0029 |
| `aws-us-west-2` | 0.0029 |
| `aws-eu-west-1` | 0.0032 |
| `aws-eu-central-1` | 0.0035 |
| `aws-ap-east-1` | 0.004 |
| `aws-ap-southeast-2` | 0.0036 |
| `gcp-us-east4` | 0.0027 |
| `gcp-europe-west3` | 0.0031 |
| `gcp-europe-west2` | 0.0031 |
| `gcp-northamerica-northeast2` | 0.0027 |
Example calculations:

- A 2-hour deployment populate in AWS `us-east-1`: 0.0029 credits × 120 minutes × 64 cores = **22.3 credits**.
- A 30-minute Copy Pipe job in AWS `us-east-1`: 0.0029 credits × 30 minutes × 64 cores = **5.57 credits**.
The credits are automatically deducted from your monthly invoice.

## Next steps (https://www.tinybird.co/docs/forward/pricing#next-steps)

- [Free plan](/docs/forward/pricing/free): What's included and how to upgrade.
- [Developer plan](/docs/forward/pricing/developer): Plan sizes, billing, and resize options.
- [SaaS plan](/docs/forward/pricing/saas): Shared infrastructure with 4+ vCPUs and a commitment.
- [Enterprise plan](/docs/forward/pricing/enterprise): Dedicated cluster, replica SKUs, and Standard or Premier support.
- [Limits](/docs/forward/pricing/limits): Platform and operational limits.

---

URL: https://www.tinybird.co/docs/forward/monitoring
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Monitoring \| Tinybird"
theme-color: "#171612"
description: "Learn how to monitor your Tinybird data platform."
inkeep:version: "forward"
---

# Monitoring (https://www.tinybird.co/docs/forward/monitoring#monitoring)

Tinybird is built around the idea of data that changes or grows continuously. Use the built-in Tinybird tools to monitor your data ingestion and API Endpoint processes.

- [Endpoint latency](/docs/forward/monitoring/latency)
- [Endpoint performance](/docs/forward/monitoring/analyze-endpoints-performance)
- [Health checks](/docs/forward/monitoring/health-checks)
- [Kafka logs](/docs/forward/monitoring/kafka-clickhouse-monitoring)
- [Logs](/docs/forward/monitoring/query-log)
- [Organization consumption](/docs/forward/monitoring/organization-consumption)
- [Service Data Sources](/docs/forward/monitoring/service-datasources)
- [System tables](/docs/forward/monitoring/query-log)
- [Workspace jobs](/docs/forward/monitoring/jobs)

---

URL: https://www.tinybird.co/docs/forward/migrations
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Migrations \| Tinybird"
theme-color: "#171612"
description: "Guides for migrating to or within Tinybird."
inkeep:version: "forward"
---

# Migrations (https://www.tinybird.co/docs/forward/migrations#migrations)

Use these guides to migrate your data and workspaces to Tinybird or between Tinybird infrastructure types.

## Available migration guides (https://www.tinybird.co/docs/forward/migrations#available-migration-guides)

- [Migrate from Tinybird Classic](/docs/forward/guides/migrate-from-classic): Migrate your existing Classic workspace to Tinybird Forward.
- [Migrate from Shared to Dedicated Infrastructure](/docs/forward/guides/migrate-from-shared-to-dedicated): Move your data from shared to dedicated infrastructure without disrupting live services.

---

URL: https://www.tinybird.co/docs/forward/ingest-data
Last update: 2026-06-18T10:43:45.000Z
Content:
---
title: "Get data into Tinybird \| Tinybird"
theme-color: "#171612"
description: "Learn how to get data into Tinybird."
inkeep:version: "forward"
---

# Get data into Tinybird (https://www.tinybird.co/docs/forward/ingest-data#get-data-into-tinybird)

Choose the ingestion method that matches how your data is produced and how often it changes:

- **Events**: Send high-frequency JSON or NDJSON events with the [Events API](/docs/forward/ingest-data/events-api).
- **Files**: Append or replace data from local files, remote URLs, CSV, NDJSON, or Parquet with [file ingestion](/docs/forward/ingest-data/local-file).
- **Kafka**: Stream from Kafka-compatible platforms with the [Kafka connector](/docs/forward/ingest-data/connectors/kafka).
- **S3 and GCS**: Ingest files from object storage with the [S3 connector](/docs/forward/ingest-data/connectors/s3) or [GCS connector](/docs/forward/ingest-data/connectors/gcs).
- **DynamoDB**: Mirror an Amazon DynamoDB table into Tinybird with an initial backfill plus Change Data Capture using the [DynamoDB connector](/docs/forward/ingest-data/connectors/dynamodb).
- **External databases and URLs**: Read from external systems with [Table functions](/docs/forward/ingest-data/table-functions) and schedule the sync with Copy Pipes.
- **Ingestion protection**: Learn how Tinybird handles retries, backpressure, and overload in [Ingestion protection](/docs/forward/ingest-data/ingestion-protection).

---

URL: https://www.tinybird.co/docs/forward/guides
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Guides \| Tinybird"
theme-color: "#171612"
description: "Practical guides for building, migrating, optimizing, and monitoring Tinybird projects."
inkeep:version: "forward"
---

# Guides (https://www.tinybird.co/docs/forward/guides#guides)

Use these guides for practical workflows that span Tinybird concepts, resources, and deployment options.

- [Add Tinybird to your Open Source Project](/docs/forward/guides/integrate-with-open-source)
- [Connect Grafana to Tinybird](/docs/forward/guides/connect-grafana)
- [Connect Hex to Tinybird](/docs/forward/guides/connect-hex)
- [Connect Metabase to Tinybird](/docs/forward/guides/connect-metabase)
- [Connect Superset to Tinybird](/docs/forward/guides/connect-superset)
- [Connect DataGrip to Tinybird](/docs/forward/guides/connect-datagrip)
- [Connect DBeaver to Tinybird](/docs/forward/guides/connect-dbeaver)
- [Connect ClickHouse® JS Client to Tinybird](/docs/forward/guides/connect-clickhouse-js)
- [Connect ClickHouse® Go Client to Tinybird](/docs/forward/guides/connect-clickhouse-go)
- [Connect ClickHouse® Python Client to Tinybird](/docs/forward/guides/connect-clickhouse-python)
- [Connect Microsoft Power BI to Tinybird](/docs/forward/guides/connect-powerbi)
- [Connect Redash to Tinybird](/docs/forward/guides/connect-redash)
- [Connect Chartbrew to Tinybird](/docs/forward/guides/connect-chartbrew)
- [Connect Deepnote to Tinybird](/docs/forward/guides/connect-deepnote)
- [Connect Draxlr to Tinybird](/docs/forward/guides/connect-draxlr)
- [Connect Fabi.ai to Tinybird](/docs/forward/guides/connect-fabi)
- [Connect Holistics to Tinybird](/docs/forward/guides/connect-holistics)
- [Connect Luzmo to Tinybird](/docs/forward/guides/connect-luzmo)
- [Connect Mitzu to Tinybird](/docs/forward/guides/connect-mitzu)
- [Connect Tableau to Tinybird](/docs/forward/guides/connect-tableau)
- [Consume API endpoints in Prometheus format](/docs/forward/guides/consume-api-endpoints-in-prometheus-format)
- [Consume APIs in a Notebook](/docs/forward/guides/consume-apis-in-a-notebook)
- [Consume APIs in a Next.js frontend with JWTs](/docs/forward/guides/consume-apis-nextjs)
- [Multi-tenant real-time APIs with Clerk and Tinybird](/docs/forward/guides/multitenant-real-time-apis-with-clerk-and-tinybird)
- [Reliable scheduling with Trigger.dev](/docs/forward/guides/reliable-scheduling-with-trigger)
- [Share API endpoints documentation](/docs/forward/guides/share-endpoint-documentation)
- [Advanced template functions for dynamic API endpoint](/docs/forward/guides/advanced-dynamic-endpoints-functions)
- [Cluster management](/docs/forward/guides/cluster-management)
- [Add a self-managed region manually](/docs/forward/guides/add-self-managed-region-manually)
- [Use the CLI to add a self-managed region](/docs/forward/guides/add-self-managed-region-cli)
- [Test your project](/docs/forward/guides/test-your-project)
- [Evolve Data Sources](/docs/forward/guides/evolve-data-source)
- [Confluent Cloud setup](/docs/forward/guides/confluent-cloud-setup)
- [AWS MSK setup](/docs/forward/guides/aws-msk-setup)
- [Redpanda setup](/docs/forward/guides/redpanda-setup)
- [Migrate from Classic](/docs/forward/guides/migrate-from-classic)
- [Migrate from PostgreSQL](/docs/forward/guides/migrate-from-postgres)
- [Migrate from ClickHouse® self-hosted](/docs/forward/guides/migrate-from-clickhouse)
- [Migrate from Shared to Dedicated Infrastructure](/docs/forward/guides/migrate-from-shared-to-dedicated)
- [Lambda architecture](/docs/forward/guides/lambda-architecture)
- [Deduplication strategies](/docs/forward/guides/deduplication-strategies)
- [Copy and export data](/docs/forward/guides/copy-and-export-data)
- [Monitoring examples](/docs/forward/guides/monitoring-examples)
- [Quarantine Data Sources](/docs/forward/guides/quarantine)
- [Replace and delete data](/docs/forward/guides/replace-and-delete-data)
- [GDPR-compliant data deletion](/docs/forward/guides/gdpr-compliant-data-deletion)
- [Send Auth0 Log Streams to Tinybird](/docs/forward/guides/ingest-auth0-logs)
- [Ingest AWS ELB logs](/docs/forward/guides/ingest-aws-elb-logs)
- [Stream from AWS Kinesis](/docs/forward/guides/ingest-from-aws-kinesis)
- [Ingest from BigQuery using Google Cloud Storage](/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage)
- [Send Clerk webhooks to Tinybird](/docs/forward/guides/ingest-from-clerk)
- [Ingest CSV files](/docs/forward/guides/ingest-from-csv-files)
- [Send Dub webhooks to Tinybird](/docs/forward/guides/ingest-from-dub)
- [Working with DynamoDB Single-Table Design](/docs/forward/guides/ingest-from-dynamodb-single-table-design)
- [Send GitHub Events to Tinybird](/docs/forward/guides/ingest-from-github)
- [Send GitLab Events to Tinybird](/docs/forward/guides/ingest-from-gitlab)
- [Ingest from Google Pub/Sub](/docs/forward/guides/ingest-from-google-pubsub)
- [Send Knock Events to Tinybird](/docs/forward/guides/ingest-from-knock)
- [Send Mailgun Events to Tinybird](/docs/forward/guides/ingest-from-mailgun)
- [Ingest data from MongoDB](/docs/forward/guides/ingest-from-mongodb)
- [Ingest from OpenTelemetry](/docs/forward/guides/ingest-from-opentelemetry)
- [Send Orb events to Tinybird](/docs/forward/guides/ingest-from-orb)
- [Send PagerDuty events to Tinybird](/docs/forward/guides/ingest-from-pagerduty)
- [Send Resend webhooks to Tinybird](/docs/forward/guides/ingest-from-resend)
- [Stream from RudderStack](/docs/forward/guides/ingest-from-rudderstack)
- [Send Sentry Webhooks to Tinybird](/docs/forward/guides/ingest-from-sentry)
- [Ingest from Snowflake using AWS S3](/docs/forward/guides/ingest-from-snowflake-using-aws-s3)
- [Ingest from Snowflake using Azure Blob Storage](/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage)
- [Ingest from Snowflake using incremental updates](/docs/forward/guides/ingest-from-snowflake-using-incremental-updates)
- [Send Stripe Events to Tinybird](/docs/forward/guides/ingest-from-stripe)
- [Ingest data using Vector.dev](/docs/forward/guides/ingest-from-vector)
- [Send Vercel Webhooks to Tinybird](/docs/forward/guides/ingest-from-vercel)
- [Send LiteLLM Events to Tinybird](/docs/forward/guides/ingest-litellm)
- [Instrument LLM calls with the Vercel AI SDK and Tinybird](/docs/forward/guides/ingest-vercel-ai-sdk)
- [Send Vercel log drains to Tinybird](/docs/forward/guides/ingest-vercel-logdrains)
- [Ingest with Estuary](/docs/forward/guides/ingest-with-estuary)
- [Postgres CDC with Redpanda Connect](/docs/forward/guides/postgres-cdc-with-redpanda-connect)

---

URL: https://www.tinybird.co/docs/forward/development-workflow
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Development workflow \| Tinybird"
theme-color: "#171612"
description: "Develop in a branch, test your changes, and deploy to your Cloud Workspace."
inkeep:version: "forward"
---

# Development workflow (https://www.tinybird.co/docs/forward/development-workflow#development-workflow)

Tinybird follows a branch-based development workflow. You edit project files in your local repo that is connected to Git, test changes in Tinybird Local or a Cloud Branch, and deploy a new version to your Cloud Workspace.

Use this workflow when you change Data Source schemas, iterate on Pipe logic, add Endpoints, or validate connector ingestion before it reaches production.

## Choose an environment (https://www.tinybird.co/docs/forward/development-workflow#choose-an-environment)

| Environment | Use it for | Example |
| --- | --- | --- |
| [Tinybird Local](/docs/forward/development-workflow/local-development) | Fast local iteration and CI checks with fixture data. | Change a Pipe query, run `tb build`, and call the local Endpoint until it returns the expected result. |
| [Cloud Branches](/docs/forward/development-workflow/using-cloud-branches) | Testing with production-like data without changing production. | Check out a Git branch, run `tb build`, change a Data Source schema, and query the branch before deploying. |
| [Staging deployments](/docs/forward/development-workflow/manual-deployment#staging-deployments) | Validating the deployment that can become live. | Run `tb --cloud deployment create --wait`, write compatible test events to staging, then promote. |
Local and branch environments are for development. Deployments are for moving a validated project version toward production.

1
### Develop in a branch (https://www.tinybird.co/docs/forward/development-workflow#develop-in-a-branch)

Choose the development loop that fits the data you need:

- [Local development](/docs/forward/development-workflow/local-development): Run Tinybird Local with fixture data and local APIs.
- [Cloud Branches](/docs/forward/development-workflow/using-cloud-branches): Use a Cloud environment for production-like data, connector validation, preview apps, and pull request review.
2
### Test your changes (https://www.tinybird.co/docs/forward/development-workflow#test-your-changes)

Test Tinybird changes as part of your app workflow, the same way you test the rest of your app. Test locally with fixtures, or test against a Cloud Branch when you need production-like data. You can also validate your data project independently with fixture files, data quality tests, and deployment checks. See [Test your project](/docs/forward/guides/test-your-project).

3
### Deploy to your Cloud Workspace (https://www.tinybird.co/docs/forward/development-workflow#deploy-to-your-cloud-workspace)

Push your changes to Tinybird Cloud with automatic schema migrations. Run a deployment check first when the change affects Data Source schemas, Materialized Views, engine settings, or connector configuration.

- [Examples](/docs/forward/development-workflow/examples): See concrete workflows for Data Source evolution, Pipe iteration, SDK projects, and deployments.
- [Evolve Data Sources](/docs/forward/guides/evolve-data-source): Understand how schema, engine, and Materialized View changes are migrated during deployments.
- [Manual deployment](/docs/forward/development-workflow/manual-deployment): Deploy directly from the CLI.
- [CI/CD](/docs/forward/development-workflow/cicd): Automate testing and deployment with GitHub Actions or GitLab CI.

## Examples (https://www.tinybird.co/docs/forward/development-workflow#examples)

See [Development workflow examples](/docs/forward/development-workflow/examples) for common loops with datafiles, the TypeScript SDK, and the Python SDK.

## Core concepts (https://www.tinybird.co/docs/forward/development-workflow#core-concepts)

- [Tinybird Local](/docs/forward/core-concepts/tinybird-local) explains the local runtime, services, installation, Docker usage, persistence, and local Tokens.
- [Branches](/docs/forward/core-concepts/branches) explains the Branch resource model and data isolation semantics.

---

URL: https://www.tinybird.co/docs/forward/dev-reference
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Tinybird developer reference \| Tinybird"
theme-color: "#171612"
description: "The Tinybird developer reference covers all available commands and tools for developers."
inkeep:version: "forward"
---

# Developer reference (https://www.tinybird.co/docs/forward/dev-reference#developer-reference)

Use this reference to understand the commands and tools available in Tinybird, including their features, options, and arguments.

## Config files (https://www.tinybird.co/docs/forward/dev-reference#config-files)

- Configure Tinybird projects: [Config files](/docs/forward/dev-reference/config-files)

## Tinybird CLI (https://www.tinybird.co/docs/forward/dev-reference#tinybird-cli)

- Learn the datafile formats: [Datafiles](/docs/forward/dev-reference/datafiles)
- Browse CLI commands: [CLI commands](/docs/forward/dev-reference/commands)

## TypeScript SDK (https://www.tinybird.co/docs/forward/dev-reference#typescript-sdk)

- Define resources in TypeScript: [TypeScript SDK resources](/docs/forward/dev-reference/typescript-sdk-resources)
- Browse SDK CLI commands: [TypeScript SDK CLI commands](/docs/forward/dev-reference/commands/typescript-sdk-cli)

## Python SDK (https://www.tinybird.co/docs/forward/dev-reference#python-sdk)

- Define resources in Python: [Python SDK resources](/docs/forward/dev-reference/python-sdk-resources)
- Browse SDK CLI commands: [Python SDK CLI commands](/docs/forward/dev-reference/commands/python-sdk-cli)
- Follow the full setup flow: [Quick start: Python SDK](/docs/forward/quickstarts/python-sdk)

---

URL: https://www.tinybird.co/docs/forward/core-concepts
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Core concepts of Tinybird \| Tinybird"
theme-color: "#171612"
description: "Learn the core concepts of Tinybird, how it works, and how to use it."
inkeep:version: "forward"
---

# Core concepts of Tinybird (https://www.tinybird.co/docs/forward/core-concepts#core-concepts-of-tinybird)

Tinybird gives you the tooling and infrastructure you need to ship analytics features in your app:

- An OLAP database (ClickHouse®)
- An API gateway
- A real-time ingestion system
- An auth system
- Tooling to build, iterate, and deploy data projects
Tinybird is built around several core concepts that abstract the underlying infrastructure and data storage. To make the most of Tinybird, it's important to understand these core concepts.

## Develop anywhere, deploy to Tinybird Cloud (https://www.tinybird.co/docs/forward/core-concepts#develop-anywhere-deploy-to-tinybird-cloud)

Tinybird is a platform for building analytics features in your app. You can build with:

- Tinybird CLI (datafiles)
- TypeScript SDK (typed resources)
- Python SDK (typed resources)
There are three main environments: Tinybird Cloud for production workloads, Tinybird Local for local development, and Tinybird Branches for shared preview environments.

- Tinybird Cloud is the hosted platform (see available [regions](/docs/api-reference#regions-and-endpoints)) where your production workspace lives.
- [Tinybird Local](/docs/forward/core-concepts/tinybird-local) is a local runtime for development and testing.
- Tinybird Branches are cloud environments for development, collaboration, and pre-production validation.
The typical workflow is as follows:

1. Define your resources using [datafiles](/docs/forward/dev-reference/datafiles) or the TypeScript SDK.
2. Iterate and test [locally](/docs/forward/development-workflow/local-development) or in [Cloud Branches](/docs/forward/development-workflow/using-cloud-branches).
3. [Deploy your data project](/docs/forward/core-concepts/deployments) to Tinybird Cloud so your endpoints and ingest pipelines are live. Strongly recommended via [CI/CD](/docs/forward/development-workflow/cicd).
4. Use [Tinybird Cloud](https://cloud.tinybird.co/) to inspect data, manage deployments, and monitor usage.
The following diagram illustrates the development and usage flow, from local development or branches to deployment in Tinybird Cloud.

## Your projects live in workspaces (https://www.tinybird.co/docs/forward/core-concepts#your-projects-live-in-workspaces)

A [workspace](/docs/forward/core-concepts/workspaces) contains the resources, data, and state of your Tinybird project. You can have as many workspaces as you need and invite users to collaborate.

At a minimum, a workspace contains a data source (a database table). You write data to it and read data from it. The following diagram illustrates the relationship between resources in a workspace.

## Data enters through data sources (https://www.tinybird.co/docs/forward/core-concepts#data-enters-through-data-sources)

[Data Sources](/docs/forward/core-concepts/data-sources) are where you ingest and store data in Tinybird. All your data lives inside a data source, and you write SQL queries against data sources. You can bulk upload or stream data into a data source.

You can bring data in from the following sources:

- Events sent to the [Events API](/docs/forward/ingest-data/events-api).
- Events streamed from Kafka. See [Ingest data from Kafka](/docs/forward/ingest-data/connectors/kafka).
- Files in your local file system. See [Ingest data from files](/docs/forward/ingest-data/local-file).
- Files in your cloud storage bucket. See [Connections](/docs/forward/core-concepts/connections).
Data sources can be defined in datafiles or with the SDKs. See [Data Sources](/docs/forward/core-concepts/data-sources) for more information.

## Use pipes to process your data (https://www.tinybird.co/docs/forward/core-concepts#use-pipes-to-process-your-data)

[Pipes](/docs/forward/core-concepts/pipes) are how you write SQL logic in Tinybird. They're a collection of one or more SQL queries chained together, or nodes. Pipes let you break larger queries down into smaller queries that are easier to read.

With pipes you can:

- Publish Endpoints.
- Create optimized Data Sources with Materialized Views or Copy Pipes.
- Send data to Cloud Storage via Sinks.
- Create Views.
[Pipes](/docs/forward/core-concepts/pipes) can be defined in datafiles or with the SDKs. See [Pipes](/docs/forward/core-concepts/pipes) for more information.

## Query your data (https://www.tinybird.co/docs/forward/core-concepts#query-your-data)

When your data is ready to be consumed, expose it through Endpoints or query it directly from Tinybird Cloud, SQL clients, BI tools, and agents.

To query from outside Tinybird:

- [Endpoints](/docs/forward/core-concepts/api-endpoints), which you can call using custom Query Parameters from any app.
- [Query API](/docs/api-reference/query-api) for running SQL directly against Data Sources and Pipes over HTTP.
- [ClickHouse interface](/docs/forward/query-data/clickhouse-interface) for SQL clients and BI tools.
- [MCP server](/docs/forward/query-data/mcp) for agent and tool integrations.
To query inside Tinybird Cloud:

- [Playgrounds](/docs/forward/query-data/playgrounds) for writing and testing SQL interactively.
- [Explorations](/docs/forward/query-data/explorations) for AI-assisted data exploration.
- [Time Series](/docs/forward/query-data/time-series) for visualizing time-based metrics.

## Tokens protect your data (https://www.tinybird.co/docs/forward/core-concepts#tokens-protect-your-data)

[Tokens](/docs/forward/core-concepts/tokens) are how you authenticate and control access to your Tinybird resources. Without tokens, nobody can send data to your data sources or query your endpoints.

There are two types of tokens:

- **[Static tokens](/docs/forward/core-concepts/static-tokens)**: Permanent tokens for backend integrations. Use them to send data, manage resources via CLI, or read data directly from your backend applications.
- **[JSON Web Tokens, JWTs](/docs/forward/core-concepts/jwt)**: Short-lived tokens. Use them for browser access and multi-tenant applications where each user or tenant needs scoped endpoint access, optional filtering, and per-user rate limits.
When you create a workspace, Tinybird automatically creates default tokens for you, including:

- **Workspace admin token**: Full access to everything in your workspace
- **Your personal admin token**: Stored in your `.tinyb` file when you run `tb login`
For development, you can use these admin tokens. For production, create specific tokens with limited permissions (scopes) for security.

### Token workflow (https://www.tinybird.co/docs/forward/core-concepts#token-workflow)

1. **Development**: Use admin tokens to build and test your project locally
2. **Production**: Create resource-specific tokens with minimal required permissions
3. **Frontend access**: Generate JWTs from your backend for browser-based queries
See [Tokens](/docs/forward/core-concepts/tokens) for detailed token management.

## Templates make resources dynamic (https://www.tinybird.co/docs/forward/core-concepts#templates-make-resources-dynamic)

Use the [templating language](/docs/forward/core-concepts/templating-language) to add typed parameters, conditional SQL, settings, and secrets to resources defined as code. Use templates to create parameterized Endpoints, run Copy Pipes with runtime values, and keep connector credentials out of source control.

## To go live, create and promote a deployment (https://www.tinybird.co/docs/forward/core-concepts#to-go-live-create-and-promote-a-deployment)

After you develop your project, you can deploy it to Tinybird Cloud. This creates a deployment, which is a version of your project resources and data running on local or cloud infrastructure.

When you're ready to go live, you can promote your deployment in Tinybird Cloud. This makes your deployment available to your users. In other words, the deployment is live.

See [Deployments](/docs/forward/core-concepts/deployments) for more information.

## Monitor with service data sources (https://www.tinybird.co/docs/forward/core-concepts#monitor-with-service-data-sources)

Tinybird exposes built-in data sources for monitoring usage and performance:

- **`tinybird.*` service data sources** for workspace-level metrics and logs.
- **`system.*` tables** for ClickHouse system-level information.
- **`organization.*` service data sources** for org-wide usage and billing signals.
See [Service data sources](/docs/forward/monitoring/service-datasources) and [Organizations](/docs/forward/core-concepts/organizations) for details.

---

URL: https://www.tinybird.co/docs/forward/copy-export-data
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Copy and export data \| Tinybird"
theme-color: "#171612"
description: "Move data inside Tinybird with Copy Pipes and export query results to Kafka, S3, or GCS with Sinks."
inkeep:version: "forward"
---

# Copy and export data (https://www.tinybird.co/docs/forward/copy-export-data#copy-and-export-data)

After data is in Tinybird, you often need to reshape it, store query results, or send processed data to another system. Use this section when you want to move data after ingestion.

## Move data internally (https://www.tinybird.co/docs/forward/copy-export-data#move-data-internally)

Use [Copy Pipes](/docs/forward/core-concepts/copy-pipes) to capture the result of a Pipe at a specific point in time and write it to a target Data Source. Copy Pipes can run on a schedule or on demand.

Copy Pipes are useful when you need to:

- Build scheduled snapshots of source data.
- Backfill or rebuild a derived Data Source.
- Deduplicate data with a replace strategy.
- Run heavy transformations outside request-time query paths.
- Test a new shape before replacing an existing Data Source.
For continuously updated derived Data Sources, use [Materialized Views](/docs/forward/core-concepts/materialized-views). Materialized Views process new rows as they arrive. Copy Pipes rerun a query and write the result when the copy job executes.

## Sinks (https://www.tinybird.co/docs/forward/copy-export-data#sinks)

Use [Sinks](/docs/forward/core-concepts/sinks) to export query results from Tinybird to external systems. A Sink is a Pipe with `TYPE SINK` and destination-specific export settings.

Tinybird supports these Sink destinations:

- [Kafka Sink](/docs/forward/copy-export-data/kafka-sink): Export events to a Kafka topic.
- [S3 Sink](/docs/forward/copy-export-data/s3-sink): Export files to an S3 bucket.
- [GCS Sink](/docs/forward/copy-export-data/gcs-sink): Export files to a Google Cloud Storage bucket.
Use Sinks when another system needs processed Tinybird data, for example to feed a data lake, publish events to a streaming platform, or deliver scheduled extracts to customers or partners.

## Choose the right option (https://www.tinybird.co/docs/forward/copy-export-data#choose-the-right-option)

| Need | Use |
| --- | --- |
| Store query results in another Tinybird Data Source | [Copy Pipes](/docs/forward/core-concepts/copy-pipes) |
| Export query results to Kafka, S3, or GCS | [Sinks](/docs/forward/core-concepts/sinks) |
| Keep a derived Data Source updated as rows are ingested | [Materialized Views](/docs/forward/core-concepts/materialized-views) |
| Query data without storing the result | [API Endpoints](/docs/forward/core-concepts/api-endpoints) or [Query API](/docs/forward/query-data/sql-api) |

---

URL: https://www.tinybird.co/docs/forward/compliance
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Compliance and certifications \| Tinybird"
theme-color: "#171612"
description: "Tinybird is committed to the highest data security and safety. See what compliance certifications are available."
inkeep:version: "forward"
---

# Compliance and certifications (https://www.tinybird.co/docs/forward/compliance#compliance-and-certifications)

Data security and privacy are paramount in today's digital landscape. Tinybird's commitment to protecting your sensitive information is backed by the following compliance certifications, which demonstrate compliance with rigorous industry standards for data security, privacy, and operational excellence.

## SOC 2 Type II (https://www.tinybird.co/docs/forward/compliance#soc-2-type-ii)

Tinybird has obtained a SOC 2 Type II certification, in accordance with attestation standards established by the American Institute of Certified Public Accountants (AICPA), that are relevant to security, availability, processing integrity, confidentiality, and privacy for Tinybird's real-time platform for user-facing analytics. Compliance is monitored continually—with reports published annually—to confirm the robustness of Tinybird's data security. This independent assessment provides Tinybird users with assurance that their sensitive information is being handled responsibly and securely.

## HIPAA (https://www.tinybird.co/docs/forward/compliance#hipaa)

Tinybird supports its customers' Health Insurance Portability and Accountability Act (HIPAA) compliance efforts by offering Business Associate Agreements (BAAs). Additionally, Tinybird's offering allows customers to process their data constituting personal health information (PHI) in AWS, Azure, or Google Cloud—entities which themselves have entered into BAAs with Tinybird.

## Trust center (https://www.tinybird.co/docs/forward/compliance#trust-center)

To learn more about Tinybird security controls and certifications, visit the [Tinybird Trust Center](https://trust.tinybird.co/).

---

URL: https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Quickstart: TypeScript SDK \| Tinybird"
theme-color: "#171612"
description: "Build Tinybird projects using the TypeScript SDK and typed resources."
inkeep:version: "forward"
---

# Quickstart: TypeScript SDK (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#quickstart-typescript-sdk)

This quickstart walks through creating a Tinybird project using the TypeScript SDK. You define Data Sources and Endpoints in TypeScript, sync them to Tinybird, and use a typed client for ingest and queries.

Looking for other ways to start? See [Quickstarts](/docs/forward/quickstarts).

If you use a coding agent like Cursor, Claude Code, Amp, or Open Code, install Tinybird agent skills so your agent understands Tinybird project structure and workflows:

```
npx skills add tinybirdco/tinybird-agent-skills
```

## Before you begin (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#before-you-begin)

To get started, you need the following:

- Node.js 20 LTS or later
- TypeScript 4.9 or later

## Install the SDK (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#install-the-sdk)

Install the SDK globally so you can run the `tinybird` CLI directly.

- pnpm
- yarn
- npm
- bun

```
pnpm add -g @tinybirdco/sdk
```
Alternatively, add it to your repository:

- pnpm
- yarn
- npm
- bun

```
pnpm add @tinybirdco/sdk
```
To follow the quickstart, install it in the repository so that commands work exactly as they are in the examples.

## Initialize a Tinybird TypeScript project (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#initialize-a-tinybird-typescript-project)

Create a new directory and initialize the project:

```
mkdir tinybird-typescript-quickstart
cd tinybird-typescript-quickstart
git init
pnpm add @tinybirdco/sdk

npx tinybird init
```
During initialization, the CLI authenticates you against Tinybird Cloud and can optionally add CI/CD workflow files.

This creates:

- `tinybird.config.json` configuration file
- `lib/tinybird.ts` for Data Sources, Pipes, Connectors, and Client definitions
- `.env.local` for authentication against your Tinybird Cloud Workspace

## Check your config file (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#check-your-config-file)

In `tinybird.config.json`, the default `"include"` is `lib/tinybird.ts` to generate resources from that file. The default `"devMode"` is `"branch"` to develop in Tinybird Cloud Branches, although you can use `"local"` too with Tinybird Local.

```
{
  "include": [
    "lib/tinybird.ts"
  ],
  "token": "${TINYBIRD_TOKEN}",
  "baseUrl": "${TINYBIRD_URL}",
  "devMode": "branch"
}
```

## Check your Tinybird token (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#check-your-tinybird-token)

Create a `.env.local` file with your Tinybird token if it doesn't exist yet:

```
TINYBIRD_TOKEN=p.your_token_here
TINYBIRD_URL=https://api.tinybird.co
```
The CLI reads `.env.local` automatically.

## Inspect resource definitions (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#inspect-resource-definitions)

### Data sources (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#data-sources)

Check the Data Source definition in `lib/tinybird.ts`. The schema and engine stay the same regardless of how you ingest data. The difference is how data arrives: via the Events API, from a Kafka topic, or from an S3 bucket.

- Events
- Kafka
- S3

```
import { defineDatasource, engine, t, type InferRow } from "@tinybirdco/sdk";

export const pageViews = defineDatasource("page_views", {
  description: "Page view tracking data",
  schema: {
    timestamp: t.dateTime(),
    session_id: t.string(),
    pathname: t.string(),
    referrer: t.string().nullable(),
  },
  engine: engine.mergeTree({
    sortingKey: ["pathname", "timestamp"],
  }),
});

export type PageViewsRow = InferRow<typeof pageViews>;
```

### API Endpoints (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#api-endpoints)

Check the Endpoint definition in `lib/tinybird.ts`:

```
import {
  defineEndpoint,
  node,
  p,
  t,
  type InferParams,
  type InferOutputRow,
} from "@tinybirdco/sdk";

export const topPages = defineEndpoint("top_pages", {
  description: "Get the most visited pages",
  params: {
    start_date: p.dateTime().describe("Start of date range"),
    end_date: p.dateTime().describe("End of date range"),
    limit: p.int32().optional(10).describe("Number of results"),
  },
  nodes: [
    node({
      name: "aggregated",
      sql: `
        SELECT
          pathname,
          count() AS views
        FROM page_views
        WHERE timestamp >= {{DateTime(start_date)}}
          AND timestamp <= {{DateTime(end_date)}}
        GROUP BY pathname
        ORDER BY views DESC
        LIMIT {{Int32(limit, 10)}}
      `,
    }),
  ],
  output: {
    pathname: t.string(),
    views: t.uint64(),
  },
});

export type TopPagesParams = InferParams<typeof topPages>;
export type TopPagesOutput = InferOutputRow<typeof topPages>;
```

### Typed client (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#typed-client)

Check the client in `lib/tinybird.ts`:

```
import { Tinybird } from "@tinybirdco/sdk";

export const tinybird = new Tinybird({
  datasources: { pageViews },
  pipes: { topPages },
});
```

## Build the project (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#build-the-project)

Check out a new git branch:

```
git add tinybird.config.json && git commit -m "Initial commit" && git checkout -b tinybird_intro
```
Run a build to create your resources in your Tinybird development environment:

```
npx tinybird build
```
If you want your changes to sync automatically while you work, run dev mode instead:

```
npx tinybird dev
```

### About the development flow (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#about-the-development-flow)

Your default target (local or branch) is set in `tinybird.config.json`. You can override it with flags on the command line.

`tinybird build` syncs to a Tinybird branch, either in Cloud or in Tinybird Local.

You can also pass a flag to explicitly use the desired development environment.

```
npx tinybird build --local # Tinybird Local Branches
# or
npx tinybird build --branch  # Tinybird Cloud Branches
```
If you haven't installed Tinybird Local yet, see [Tinybird Local](/docs/forward/core-concepts/tinybird-local).

When you're ready to promote changes:

- Use `tinybird build` to push to a branch (good for review or CI).
- Use CI/CD to deploy to your main Tinybird Cloud Workspace. You can also use `tinybird deploy` directly.

#### Git integration (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#git-integration)

You may have noticed that the quickstart example had a `git checkout -b tinybird_intro`. If Tinybird detects you are in a Git repository, it creates a branch with the git branch name in your Tinybird development environment.

## Ingest data and query (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#ingest-data-and-query)

### Ingest (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#ingest)

Use the typed client to send events via the Events API:

```
import { tinybird } from "./lib/tinybird";

await tinybird.pageViews.ingest({
  timestamp: new Date(),
  pathname: "/home",
  session_id: "abc123",
  referrer: "https://google.com",
});
```
If you configured a Kafka or S3 connector, data is ingested automatically after you deploy. You don't need to call `ingest` manually.

### Query (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#query)

Query the Endpoint from your app code:

```
const result = await tinybird.topPages.query({
  start_date: new Date("2026-01-01"),
  end_date: new Date(),
  limit: 5,
});

result.data.forEach((row) => {
  console.log(row.pathname, row.views);
});
```

## Iterate in a branch (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#iterate-in-a-branch)

Add a materialized view in another branch, test the app against that branch, and deploy to Tinybird Cloud when you're finished.

```
git checkout -b materialization
```
Edit `lib/tinybird.ts` to add a new Data Source, a new materialized view, and update the Endpoint to read from the materialized view.

```
import {
  defineDatasource,
  defineEndpoint,
  defineMaterializedView,
  engine,
  node,
  p,
  t,
} from "@tinybirdco/sdk";

export const pageViews = defineDatasource("page_views", {
  description: "Page view tracking data",
  schema: {
    timestamp: t.dateTime(),
    session_id: t.string(),
    pathname: t.string(),
    referrer: t.string().nullable(),
  },
  engine: engine.mergeTree({
    sortingKey: ["pathname", "timestamp"],
  }),
});

export const dailyPageViews = defineDatasource("daily_page_views_ds", {
  description: "Daily page view counts populated by a materialized view",
  schema: {
    date: t.date(),
    pathname: t.string(),
    views: t.simpleAggregateFunction("sum", t.uint64()),
  },
  engine: engine.aggregatingMergeTree({
    sortingKey: ["date", "pathname"],
  }),
  jsonPaths: false,
});

export const dailyPageViewsMv = defineMaterializedView("daily_page_views_mv", {
  datasource: dailyPageViews,
  nodes: [
    node({
      name: "aggregate",
      sql: `
        SELECT
          toDate(timestamp) AS date,
          pathname,
          count() AS views
        FROM page_views
        GROUP BY date, pathname
      `,
    }),
  ],
});

export const topPages = defineEndpoint("top_pages", {
  description: "Get the most visited pages from MV-backed datasource",
  params: {
    start_date: p.dateTime(),
    end_date: p.dateTime(),
    limit: p.int32().optional(10),
  },
  nodes: [
    node({
      name: "aggregated",
      sql: `
        SELECT
          pathname,
          sum(views) AS views
        FROM daily_page_views_ds
        WHERE date >= toDate({{DateTime(start_date)}})
          AND date <= toDate({{DateTime(end_date)}})
        GROUP BY pathname
        ORDER BY views DESC
        LIMIT {{Int32(limit, 10)}}
      `,
    }),
  ],
  output: {
    pathname: t.string(),
    views: t.uint64(),
  },
});
```
Build against the branch:

```
npx tinybird build
```
Once validated, you can deploy to Tinybird Cloud with `npx tinybird deploy` if you haven't set up a CI/CD script.

## Next steps (https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk#next-steps)

- Review the SDK repository for advanced usage. See [tinybird-sdk-typescript](https://github.com/tinybirdco/tinybird-sdk-typescript).
- Review the TypeScript SDK CLI reference. See [TypeScript SDK CLI commands](/docs/forward/dev-reference/commands/typescript-sdk-cli).
- Learn about core Tinybird concepts. See [Core concepts](/docs/forward/core-concepts).
- Configure deployments and branches. See [Development workflow](/docs/forward/development-workflow).

---

URL: https://www.tinybird.co/docs/forward/quickstarts/python-sdk
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Quickstart: Python SDK \| Tinybird"
theme-color: "#171612"
description: "Build Tinybird projects with the Python SDK and use the bundled CLI commands."
inkeep:version: "forward"
---

# Quickstart: Python SDK (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#quickstart-python-sdk)

This quickstart walks through creating a Tinybird project with the Python SDK. You define Data Sources and Endpoints in Python, use the CLI to build and deploy, and call your Endpoint from app code.

Looking for other ways to start? See [Quickstarts](/docs/forward/quickstarts).

If you use a coding agent like Cursor, Claude Code, Amp, or Open Code, install Tinybird agent skills so your agent understands Tinybird project structure and workflows:

```
npx skills add tinybirdco/tinybird-agent-skills
```

## Before you begin (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#before-you-begin)

To get started, you need:

- Python 3.11, 3.12, or 3.13
- A Tinybird Workspace
- git
- `uv` (recommended [Python package manager](https://docs.astral.sh/uv/))
Python 3.14 is not supported yet for this workflow. The Python SDK relies on the Tinybird CLI under the hood, and current CLI support covers Python 3.11 to 3.13.

## Initialize a Python project and add the SDK (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#initialize-a-python-project-and-add-the-sdk)

Create a new directory and initialize a Python SDK project:

```
mkdir tinybird-python-quickstart
cd tinybird-python-quickstart
git init

uv init --name tinybird-sdk-python-sample --python 3.13
uv add tinybird-sdk python-dotenv

uv run tinybird init --folder src/tinybird/
```
This creates:

- `tinybird.config.json` with an `include` path for Python resources
- `.env.local` for your Tinybird credentials

## Check your config file (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#check-your-config-file)

In `tinybird.config.json`, `"include"` points to the folder with your Python resource definitions. The default `"devMode"` is `"branch"` to develop in Tinybird Cloud Branches, although you can use `"local"` too with Tinybird Local.

## Check your Tinybird token (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#check-your-tinybird-token)

Create a `.env.local` file with your Tinybird token if it doesn't exist yet:

```
TINYBIRD_TOKEN=p.your_token_here
TINYBIRD_URL=https://api.tinybird.co
```
The CLI reads `.env.local` automatically.

## Define resources in Python (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#define-resources-in-python)

Create `src/tinybird/tinybird_resources.py`. The schema and engine stay the same regardless of how you ingest data. The difference is how data arrives: via the Events API, from a Kafka topic, or from an S3 bucket.

### Data source (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#data-source)

- Events
- Kafka
- S3

```
from tinybird_sdk import define_datasource, engine, t

page_views = define_datasource(
    "page_views",
    {
        "description": "Page view tracking data",
        "schema": {
            "timestamp": t.date_time(),
            "session_id": t.string(),
            "pathname": t.string(),
            "referrer": t.string().nullable(),
        },
        "engine": engine.merge_tree(
            {
                "sorting_key": ["pathname", "timestamp"],
            }
        ),
    },
)
```

### Endpoint (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#endpoint)

The endpoint reads from the Data Source regardless of how data is ingested:

```
from tinybird_sdk import define_endpoint, node, p, t

top_pages = define_endpoint(
    "top_pages",
    {
        "description": "Get the most visited pages",
        "params": {
            "start_date": p.date_time(),
            "end_date": p.date_time(),
            "limit": p.int32().optional(10),
        },
        "nodes": [
            node(
                {
                    "name": "aggregated",
                    "sql": """
                        SELECT pathname, count() AS views
                        FROM page_views
                        WHERE timestamp >= {{DateTime(start_date)}}
                          AND timestamp <= {{DateTime(end_date)}}
                        GROUP BY pathname
                        ORDER BY views DESC
                        LIMIT {{Int32(limit, 10)}}
                    """,
                }
            )
        ],
        "output": {
            "pathname": t.string(),
            "views": t.uint64(),
        },
    },
)
```
Create `src/tinybird/client.py`:

```
import os

from tinybird_sdk import Tinybird
from src.tinybird.tinybird_resources import page_views, top_pages

tinybird = Tinybird(
    {
        "datasources": {"page_views": page_views},
        "pipes": {"top_pages": top_pages},
        "base_url": os.getenv("TINYBIRD_URL", "https://api.tinybird.co"),
        "token": os.getenv("TINYBIRD_TOKEN"),
    }
)
```

## Build the project (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#build-the-project)

Commit your project files and check out a new git branch:

- `pyproject.toml` — project metadata and dependencies
- `uv.lock` — pinned dependency versions (like `package-lock.json`)
- `.python-version` — ensures consistent Python version
- `.gitignore`
- Your source code and tests

```
git add pyproject.toml uv.lock .python-version .gitignore src/ tinybird.config.json main.py
git commit -m "Initial commit"
git checkout -b tinybird_intro
```
Run a build to create your resources in your Tinybird development environment:

```
uv run tinybird build
```

#### Git integration (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#git-integration)

You may have noticed that the quickstart example had a `git checkout -b tinybird_intro`. If Tinybird detects you are in a Git repository, it creates a branch with the git branch name in your Tinybird development environment.

## Ingest data and query (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#ingest-data-and-query)

### Ingest (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#ingest)

Use the runtime client to send events via the Events API:

```
from datetime import datetime, timezone
from src.tinybird.client import tinybird

now = datetime.now(timezone.utc).isoformat(timespec="milliseconds")

tinybird.page_views.ingest(
    {
        "timestamp": now,
        "session_id": "abc123",
        "pathname": "/home",
        "referrer": "https://google.com",
    }
)
```
If you configured a Kafka or S3 connector, data is ingested automatically after you deploy. You don't need to call `ingest` manually.

### Query (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#query)

Query the Endpoint from your app code:

```
result = tinybird.top_pages.query(
    {
        "start_date": "2026-01-01 00:00:00",
        "end_date": now,
        "limit": 5,
    }
)

for row in result["data"]:
    print(row["pathname"], row["views"])
```

## Iterate in a branch (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#iterate-in-a-branch)

Add a materialized view in another branch, test the app against that branch, and deploy to Tinybird Cloud when you're finished.

```
git checkout -b materialization
```
Edit `tinybird_resources.py` to add a new Data Source, a new materialized view, and update the Endpoint to read from the materialized view.

```
from tinybird_sdk import (
    define_datasource,
    define_endpoint,
    define_materialized_view,
    engine,
    node,
    p,
    t,
)

page_views = define_datasource(
    "page_views",
    {
        "description": "Page view tracking data",
        "schema": {
            "timestamp": t.date_time(),
            "session_id": t.string(),
            "pathname": t.string(),
            "referrer": t.string().nullable(),
        },
        "engine": engine.merge_tree(
            {
                "sorting_key": ["pathname", "timestamp"],
            }
        ),
    },
)

daily_page_views_ds = define_datasource(
    "daily_page_views_ds",
    {
        "description": "Daily page view counts populated by a materialized view",
        "schema": {
            "date": t.date(),
            "pathname": t.string(),
            "views": t.simple_aggregate_function("sum", t.uint64()),
        },
        "engine": engine.aggregating_merge_tree(
            {
                "sorting_key": ["date", "pathname"],
            }
        ),
        "json_paths": False,
    },
)

daily_page_views_mv = define_materialized_view(
    "daily_page_views_mv",
    {
        "description": "Materialized rollup of page views by day and pathname",
        "datasource": daily_page_views_ds,
        "nodes": [
            node(
                {
                    "name": "aggregate",
                    "sql": """
                        SELECT
                            toDate(timestamp) AS date,
                            pathname,
                            count() AS views
                        FROM page_views
                        GROUP BY date, pathname
                    """,
                }
            )
        ],
    },
)

top_pages = define_endpoint(
    "top_pages",
    {
        "description": "Get the most visited pages from MV-backed datasource",
        "params": {
            "start_date": p.date_time(),
            "end_date": p.date_time(),
            "limit": p.int32().optional(10),
        },
        "nodes": [
            node(
                {
                    "name": "aggregated",
                    "sql": """
                        SELECT
                            pathname,
                            sum(views) AS views
                        FROM daily_page_views_ds
                        WHERE date >= toDate({{DateTime(start_date)}})
                          AND date <= toDate({{DateTime(end_date)}})
                        GROUP BY pathname
                        ORDER BY views DESC
                        LIMIT {{Int32(limit, 10)}}
                    """,
                }
            )
        ],
        "output": {
            "pathname": t.string(),
            "views": t.uint64(),
        },
    },
)
```
Build against the branch:

```
uv run tinybird build
```
Once validated, you can deploy to Tinybird Cloud with `uv run tinybird deploy` if you haven't set up a CI/CD script.

## Next steps (https://www.tinybird.co/docs/forward/quickstarts/python-sdk#next-steps)

- Learn Tinybird concepts: [Core concepts](/docs/forward/core-concepts)
- Review Python resource definitions: [Python SDK resources](/docs/forward/dev-reference/python-sdk-resources)
- Review command options: [Python SDK CLI commands](/docs/forward/dev-reference/commands/python-sdk-cli)
- Configure branches and deployments: [Development workflow](/docs/forward/development-workflow)
- Review the SDK repository for advanced usage: [tinybird-sdk-python](https://github.com/tinybirdco/tinybird-sdk-python)

---

URL: https://www.tinybird.co/docs/forward/quickstarts/cli
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Quickstart: Tinybird CLI \| Tinybird"
theme-color: "#171612"
description: "Follow this step-by-step tutorial to get started with Tinybird."
inkeep:version: "forward"
---

# Quickstart: Tinybird CLI (https://www.tinybird.co/docs/forward/quickstarts/cli#quickstart-tinybird-cli)

Follow these steps to install the Tinybird CLI, build your first data project in a Tinybird branch, and deploy it to Tinybird Cloud.

See [Core concepts](/docs/forward/core-concepts) for a complete overview of Tinybird.

This quickstart is based on the Tinybird CLI and datafiles. If you're looking for a different approach with SDKs, see [Quickstarts](/docs/forward/quickstarts).

If you use a coding agent like Cursor, Claude Code, Amp, or Open Code, install Tinybird agent skills so your agent understands Tinybird project structure and workflows:

```
npx skills add tinybirdco/tinybird-agent-skills
```

## Before you begin (https://www.tinybird.co/docs/forward/quickstarts/cli#before-you-begin)

To get started, you need the following:

- A Tinybird account. You can create one at [cloud.tinybird.co](https://cloud.tinybird.co/).
- Git.

## Deploy a new project in five minutes (https://www.tinybird.co/docs/forward/quickstarts/cli#deploy-a-new-project-in-five-minutes)

### Install Tinybird CLI (https://www.tinybird.co/docs/forward/quickstarts/cli#install-tinybird-cli)

Run the following command to install the Tinybird CLI:

- For macOS and Linux
- For Windows

```
curl https://tinybird.co | sh
```
**Already have Tinybird Classic CLI installed?** Both Classic and Forward CLIs use the `tb` command. Run `which tb` to check if you already have a `tb` binary installed. If you do, see [Managing CLI versions](/docs/forward/guides/migrate-from-classic#install-the-tinybird-forward-cli) to avoid conflicts.

### Create a project (https://www.tinybird.co/docs/forward/quickstarts/cli#create-a-project)

Create a new directory and initialize the project:

```
mkdir tinybird-cli-quickstart
cd tinybird-cli-quickstart
git init

tb init
```
This creates the standard project folders (`datasources/`, `pipes/`, `endpoints/`, `fixtures/`, `tests/`, and others), plus other useful resources like CI/CD templates or Agent Skills. It also creates a `tinybird.config.json` containing your project settings, like the folder where the project is located and the default development environment.

```
{
  "dev_mode": "branch",
  "include": [
    "tinybird"
  ]
}
```
If you log in during this step, a file called `.tinyb` is created. This contains your auth details, so never commit it to git.

First things first, you need a data source. The schema and engine stay the same regardless of how you ingest data. The difference is how data arrives: via the Events API, from a Kafka topic, or from an S3 bucket.

Create the `.datasource` file. If you want to ingest from Kafka or S3 instead of the Events API, add a `.connection` file and reference it in the `.datasource`:

- Events
- Kafka
- S3
datasources/page_views.datasource
```
DESCRIPTION >
    Page view tracking data

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `pathname` String `json:$.pathname`,
    `referrer` Nullable(String) `json:$.referrer`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "pathname, timestamp"
```
Data Sources define the schema and properties of the database tables where you store your data. For more information, see [Data Source files](/docs/forward/dev-reference/datafiles/datasource-files).

### Create an endpoint (https://www.tinybird.co/docs/forward/quickstarts/cli#create-an-endpoint)

Create a new file `endpoints/top_pages.pipe`:

endpoints/top_pages.pipe
```
TOKEN top_pages_read READ

DESCRIPTION >
    Get the most visited pages

NODE aggregated
SQL >
    %
    SELECT
        pathname,
        count() AS views
    FROM page_views
    WHERE timestamp >= {{DateTime(start_date, '2026-01-01 00:00:00')}}
      AND timestamp <= {{DateTime(end_date, '2026-12-31 23:59:59')}}
    GROUP BY pathname
    ORDER BY views DESC
    LIMIT {{Int32(limit, 10)}}

TYPE endpoint
```

### Build the project (https://www.tinybird.co/docs/forward/quickstarts/cli#build-the-project)

Checkout a new git branch:

```
git add tinybird.config.json && git commit -m "initial commit" && git checkout -b tinybird_intro
```
Run a build to create your resources in your Tinybird development environment:

```
tb build
```

#### Git integration (https://www.tinybird.co/docs/forward/quickstarts/cli#git-integration)

You may have noticed that the quickstart example had a `git checkout -b tinybird_intro`. Tinybird detects you're on a git project and creates a branch with the git branch name in your Tinybird development environment.

### Ingest data and query (https://www.tinybird.co/docs/forward/quickstarts/cli#ingest-data-and-query)

#### Ingest (https://www.tinybird.co/docs/forward/quickstarts/cli#ingest)

Send events using the Events API with `curl`:

```
TB_HOST=$(cat .tinyb | jq -r ".host")
tb --branch=tinybird_intro token copy "admin"
TB_TOKEN="<paste-your-admin-token-here>"

curl -X POST "$TB_HOST/v0/events?name=page_views&token=$TB_TOKEN" \
  -d '{"timestamp": "2026-01-15 10:00:00", "session_id": "s1", "pathname": "/home", "referrer": "https://google.com"}
{"timestamp": "2026-01-15 10:01:00", "session_id": "s2", "pathname": "/docs", "referrer": null}
{"timestamp": "2026-01-15 10:02:00", "session_id": "s1", "pathname": "/pricing", "referrer": "/home"}'
```
If you configured a Kafka or S3 connector, data is ingested automatically after you deploy. You don't need to call the Events API manually.

#### Query (https://www.tinybird.co/docs/forward/quickstarts/cli#query)

Query the endpoint with `curl`. All Tinybird API calls require a **token** for authentication.

```
curl -X GET "$TB_HOST/v0/pipes/top_pages.json?token=$TB_TOKEN"
```

```
{
  "meta": [
    { "name": "pathname", "type": "String" },
    { "name": "views", "type": "UInt64" }
  ],
  "data": [
    { "pathname": "/home", "views": 1 },
    { "pathname": "/docs", "views": 1 },
    { "pathname": "/pricing", "views": 1 }
  ]
}
```

### Deploy to Tinybird Cloud (https://www.tinybird.co/docs/forward/quickstarts/cli#deploy-to-tinybird-cloud)

```
tb deploy
```
Open the project in Tinybird Cloud:

```
tb --cloud open
```

## Iterate in a branch (https://www.tinybird.co/docs/forward/quickstarts/cli#iterate-in-a-branch)

Add a materialized view in another branch, test the app against that branch, and deploy to Tinybird Cloud when you're finished.

```
git checkout -b materialization
```
Create a new Data Source for the materialized rollup in `datasources/daily_page_views.datasource`:

datasources/daily_page_views.datasource
```
DESCRIPTION >
    Daily page view counts populated by a materialized view

SCHEMA >
    `date` Date,
    `pathname` String,
    `views` AggregateFunction(count)

ENGINE "AggregatingMergeTree"
ENGINE_SORTING_KEY "date, pathname"
```
Create the materialized view in `pipes/daily_page_views_mv.pipe`:

pipes/daily_page_views_mv.pipe
```
DESCRIPTION >
    Materialized rollup of page views by day and pathname

NODE aggregate
SQL >
    SELECT
        toDate(timestamp) AS date,
        pathname,
        countState() AS views
    FROM page_views
    GROUP BY date, pathname

TYPE materialized
DATASOURCE daily_page_views
```
Update the endpoint in `endpoints/top_pages.pipe` to read from the materialized Data Source:

endpoints/top_pages.pipe
```
TOKEN top_pages_read READ

DESCRIPTION >
    Get the most visited pages from MV-backed datasource

NODE aggregated
SQL >
    %
    SELECT
        pathname,
        countMerge(views) AS views
    FROM daily_page_views
    WHERE date >= toDate({{DateTime(start_date, '2026-01-01 00:00:00')}})
      AND date <= toDate({{DateTime(end_date, '2026-12-31 23:59:59')}})
    GROUP BY pathname
    ORDER BY views DESC
    LIMIT {{Int32(limit, 10)}}

TYPE endpoint
```
Build against the branch:

```
tb build
```
Once validated, you can deploy to Tinybird Cloud with `tb deploy` if you haven't set up a CI/CD script.

## Next steps (https://www.tinybird.co/docs/forward/quickstarts/cli#next-steps)

- Familiarize yourself with Tinybird concepts. See [Core concepts](/docs/forward/core-concepts).
- Learn about datafiles, like .datasource and .pipe files. See [Datafiles](/docs/forward/dev-reference/datafiles).
- Get data into Tinybird from a variety of sources. See [Get data in](/docs/forward/ingest-data).
- Learn about authentication and securing your Endpoints. See [Tokens](/docs/forward/core-concepts/tokens).
- Browse the Tinybird CLI commands reference. See [Commands reference](/docs/forward/dev-reference/commands).

---

URL: https://www.tinybird.co/docs/forward/query-data/time-series
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Time Series \| Tinybird"
theme-color: "#171612"
description: "Visualize time-based data with Tinybird Time Series."
inkeep:version: "forward"
---

# Time series (https://www.tinybird.co/docs/forward/query-data/time-series#time-series)

Time Series help analyze a sequence of data points collected over an interval of time. Use them to visualize any time series Data Source in your Workspace, including Service Data Sources and Endpoints.

## Getting started (https://www.tinybird.co/docs/forward/query-data/time-series#getting-started)

1. Go to [Tinybird Cloud](https://cloud.tinybird.co/) and log in to your account.
2. Select **Time Series** in the sidebar.
3. Select **New time series**.
4. Pick a Data Source and the date or datetime column.
5. Choose the metric and any grouping columns.

![Time Series editor with the example Ingestion Metrics query](/docs/_next/image?url=%252Fdocs%252Fimg%252Ftimeseries-fwd.png&w=3840&q=75)

Time Series uses SQL under the hood. For statements, functions, and settings, see the [SQL reference](/docs/sql-reference).

## Time series workflow (https://www.tinybird.co/docs/forward/query-data/time-series#time-series-workflow)

**Shape the chart**

Adjust the date range and granularity, then refine metrics, conditions, and groupings to match the story you need to tell. Select bar or line charts and set the number of results to show.

**Zoom and inspect**

Use smart zoom to drill into spikes or drops and keep the time selector in sync as you explore. Activate it by clicking and dragging the mouse over the chart.

**Cancel long runs**

When a query is running, a **Cancel** button appears. Use it to stop long-running queries mid execution when you need to adjust filters or aggregation settings.

**Open from observability**

Open any observability chart in Time Series to investigate trends with the same interactive controls.

**Share with your team**

Use the sidebar to switch between **Favorites**, **Private**, and **Workspace** Time Series.

- **Private**: Time Series are visible only to you.
- **Workspace**: Time Series are visible and editable by any Workspace member, but only the owner can delete them or stop sharing.
**Work with context**

Select the source to open a side panel, then review in-context metadata like columns, settings, and query plans to validate results.

<!-- <figure> -->

![Video]()
Controls: true

<!-- <figcaption> -->
The side panel shows extra context when you click a Data Source.
<!-- </figcaption> -->

<!-- </figure> -->

## Next steps (https://www.tinybird.co/docs/forward/query-data/time-series#next-steps)

- Learn about [Playgrounds](/docs/forward/query-data/playgrounds) to write your own SQL queries.
- Learn about [Pipes](/docs/forward/core-concepts/pipes).
- Learn about [API Endpoints](/docs/forward/core-concepts/api-endpoints).

---

URL: https://www.tinybird.co/docs/forward/query-data/sql-api
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Query API \| Tinybird"
theme-color: "#171612"
description: "Run SQL queries against Tinybird over HTTP."
inkeep:version: "forward"
---

# Query API (https://www.tinybird.co/docs/forward/query-data/sql-api#query-api)

The [Query API](/docs/api-reference/query-api) runs SQL directly against Pipes and Data Sources in your Workspace through the `/v0/sql` endpoint. Use it for internal tools, automation, and ad hoc queries that do not need a published Endpoint contract.

Use API Endpoints for production query contracts that applications call repeatedly. Use the Query API when the SQL is owned by the caller, such as an internal tool, notebook, script, or debugging workflow.

Query API requests can also use typed [Query parameters](/docs/forward/query-data/query-parameters) when you need validated request-time values.

## Run a SQL query (https://www.tinybird.co/docs/forward/query-data/sql-api#run-a-sql-query)

Send a request to `/v0/sql` with a `q` parameter containing the SQL query. Use a Token with read permissions for the resources you query. Use a [static Token](/docs/forward/core-concepts/static-tokens) for backend services and trusted internal tools. Use a [JWT](/docs/forward/core-concepts/jwt) for browser, user-facing, or multi-tenant access where each user or tenant needs scoped access, filters, expiration, or rate limits.

For short queries, use `GET` and pass the SQL in the `q` query parameter:

```
curl \
  -H "Authorization: Bearer <read token or JWT>" \
  --get \
  --data-urlencode "q=SELECT * FROM <pipe_or_datasource> FORMAT JSON" \
  https://api.tinybird.co/v0/sql
```
Use `POST` when the SQL query or request parameters are large. This avoids URL length limits and keeps long queries out of the URL:

```
curl \
  -X POST \
  -H "Authorization: Bearer <read token or JWT>" \
  --data-urlencode "q=SELECT * FROM <pipe_or_datasource> FORMAT JSON" \
  https://api.tinybird.co/v0/sql
```
You can query Data Sources and Pipes by name. Add a `FORMAT` clause when you need a specific response format.

## Available response formats (https://www.tinybird.co/docs/forward/query-data/sql-api#available-response-formats)

Set the response format with a ClickHouse® `FORMAT` clause at the end of the SQL query:

```
SELECT * FROM events LIMIT 10 FORMAT JSON
```
Common formats include:

| Format | Use when |
| --- | --- |
| `JSON` | You need the default JSON response with metadata. |
| `CSV` | You need comma-separated values without headers. |
| `CSVWithNames` | You need comma-separated values with column headers. |
| `JSONEachRow` | You need newline-delimited JSON. |
| `Parquet` | You need a binary columnar format for larger exports or downstream analytics tools. |
If you don't specify a `FORMAT`, Tinybird returns the default Query API format. For the full endpoint contract, see the [Query API reference](/docs/api-reference/query-api).

## Use query parameters (https://www.tinybird.co/docs/forward/query-data/sql-api#use-query-parameters)

Query API requests can use Tinybird's [templating language](/docs/forward/core-concepts/templating-language) and receive request parameters the same way API Endpoints do. This is useful when an internal tool or script owns the SQL but still needs typed, validated inputs.

```
curl \
  -X POST \
  -H "Authorization: Bearer <read token or JWT>" \
  --data-urlencode "q=% SELECT count() FROM events WHERE timestamp >= {{DateTime(start_date)}} FORMAT JSON" \
  --data-urlencode "start_date=2026-01-01 00:00:00" \
  https://api.tinybird.co/v0/sql
```
For parameter syntax, defaults, and helper functions, see [Templating language](/docs/forward/core-concepts/templating-language) and [Query parameters](/docs/forward/query-data/query-parameters).

## Query a specific Pipe (https://www.tinybird.co/docs/forward/query-data/sql-api#query-a-specific-pipe)

You can also run a SQL query against a specific Pipe. Send the request to `/v0/pipes/<pipe>.json` and use `_` as the Pipe input in the `q` parameter:

```
curl \
  -X POST \
  -H "Authorization: Bearer <read token or JWT>" \
  --data "q=SELECT count() FROM _" \
  https://api.tinybird.co/v0/pipes/<pipe>.json
```
This is useful when you want to reuse a Pipe's logic but apply an extra aggregation, filter, or format at request time.

## Track Query API usage (https://www.tinybird.co/docs/forward/query-data/sql-api#track-query-api-usage)

Query API requests are logged as `query_api` in Service Data Sources. Add metadata as request parameters when you want to group requests by app, customer, or workflow.

```
curl \
  -X POST \
  -H "Authorization: Bearer <read token or JWT>" \
  --data "q=SELECT count() FROM events FORMAT JSON" \
  "https://api.tinybird.co/v0/sql?app_name=internal_tool"
```
See [Endpoint performance](/docs/forward/monitoring/analyze-endpoints-performance#use-the-query-api-with-metadata-parameters) for monitoring examples.

## Next steps (https://www.tinybird.co/docs/forward/query-data/sql-api#next-steps)

- Review the [Query API reference](/docs/api-reference/query-api).
- Add typed parameters with [Query parameters](/docs/forward/query-data/query-parameters).
- Use the [ClickHouse® interface](/docs/forward/query-data/clickhouse-interface) for SQL clients and BI tools.

---

URL: https://www.tinybird.co/docs/forward/query-data/query-parameters
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Query parameters \| Tinybird"
theme-color: "#171612"
description: "Add dynamic query parameters to Tinybird queries."
inkeep:version: "forward"
---

# Query parameters (https://www.tinybird.co/docs/forward/query-data/query-parameters#query-parameters)

Query parameters define values that your app can control when it calls an API Endpoint or the Query API. For example, one Endpoint or Query API request can answer different questions by changing a filter, limit, sort order, or grouping value at request time.

Query parameters are part of Tinybird's [templating language](/docs/forward/core-concepts/templating-language). Use the concept page for the mental model, and this page for common patterns and examples.

Use query parameters to do things like:

- Filtering as part of a `WHERE` clause.
- Changing the number of results as part of a `LIMIT` clause.
- Sorting order as part of an `ORDER BY` clause.
- Selecting specific columns for `ORDER BY` or `GROUP BY` clauses.

## Define query parameters (https://www.tinybird.co/docs/forward/query-data/query-parameters#define-query-parameters)

To make a `.pipe` query dynamic, start the query with a `%` character. That signals the engine that it needs to parse potential parameters. In SDK projects, declare parameters in the `params` object and use the same SQL template syntax inside each node.

After you make a query dynamic, define parameters by using the following pattern `{{<data_type>(<name_of_parameter>[,<default_value>, description=<"This is a description">, required=<True|False>])}}`. For example:

Simple select clause using dynamic parameters
```
%
SELECT * FROM TR LIMIT {{Int32(lim, 10, description="Limit the number of rows in the response", required=False)}}
```
The previous query returns 10 results by default, or however many are specified on the `lim` parameter when requesting data.

Boolean parameters don't support the `description` or `required` arguments.

## Call a query with parameters (https://www.tinybird.co/docs/forward/query-data/query-parameters#call-a-query-with-parameters)

When using an Endpoint that uses parameters, pass in the desired parameters.

Using the previous example where `lim` sets the maximum number of rows, the request looks like this:

- cURL
- TypeScript SDK
- Python SDK

```
curl "/v0/pipes/tr_pipe.json?lim=20&token=<your_token>"
```
You can specify parameters in more than one node in a Pipe. When invoking the Endpoint, the passed parameters are included in the request.

When using the Query API, pass parameters together with the `q` parameter. See [Query API](/docs/forward/query-data/sql-api#use-query-parameters).

You can't use query parameters in Materialized Views.

## Use parameters for observability (https://www.tinybird.co/docs/forward/query-data/query-parameters#use-parameters-for-observability)

In addition to using query parameters in your Endpoints and Query API requests, you can use them downstream for monitoring.

When you pass a parameter to your queries, you can build Pipes to reference the parameters and query the Service Data Sources with them, even if you don't use them in the Endpoints themselves.

Review [Service Data Sources](/docs/forward/monitoring/service-datasources) to use the available options. For example, the `user_agent` column on `pipe_stats_rt` shows which user agent made the request. Pass any additional values you need as parameters to improve visibility into incidents and Workspace performance.

Example query to the pipe_stats_rt Service data source leveraging a passed 'referrer' parameter
```
SELECT
  toStartOfMinute(start_datetime) AS date,
  count() AS requests,
  parameters['referrer'] AS referrer
FROM tinybird.pipe_stats_rt
WHERE
  status_code != 429
  AND start_datetime > now() - INTERVAL 1 HOUR
  AND (
    pipe_id = '<pipe_id_here>'
    OR pipe_name = '<pipe_name_here>'
  )
GROUP BY date, referrer
ORDER BY requests DESC, date DESC
```

## Available parameter types (https://www.tinybird.co/docs/forward/query-data/query-parameters#available-parameter-types)

Tinybird supports the following data types for query parameters:

- `Boolean`: Accepts `True` and `False` as values, as well as strings like `'TRUE'`, `'FALSE'`, `'true'`, `'false'`, `'1'`, or `'0'`, or the integers `1` and `0`.
- `String`: For any string values.
- `DateTime64`, `DateTime` and `Date`: Accepts values like `YYYY-MM-DD HH:MM:SS.MMM`, `YYYY-MM-DD HH:MM:SS` and `YYYYMMDD` respectively.
- `Float32` and `Float64`: Accepts floating point numbers of either 32 or 64 bit precision.
- `Int` or `Integer`: Accepts integer numbers of any precision.
- `Int8`, `Int16`, `Int32`, `Int64`, `Int128`, `Int256` and `UInt8`, `UInt16`, `UInt32`, `UInt64`, `UInt128`, `UInt256`: Accepts signed or unsigned integer numbers of the specified precision.

### Column parameters (https://www.tinybird.co/docs/forward/query-data/query-parameters#column-parameters)

You can use `column` to pass along column names of a defined type as parameters, like:

Using column dynamic parameters
```
%
SELECT * FROM TR
ORDER BY {{column(order_by, 'timestamp')}}
LIMIT {{Int32(lim, 10)}}
```
Always define the `column` function's second argument, the one for the default value. The alternative for not defining the argument is to validate that the first argument is defined, but this only has an effect on query execution. A placeholder is used in the development of the Pipes.

Validate the column parameter when not defining a default value
```
%
SELECT * FROM TR
{% if defined(order_by) %}
ORDER BY {{column(order_by)}}
{% end %}
```

### Array parameters (https://www.tinybird.co/docs/forward/query-data/query-parameters#array-parameters)

You can pass along a list of values with the `Array` function for parameters, like so:

Passing arrays as dynamic parameters
```
%
SELECT * FROM TR WHERE
access_type IN {{Array(access_numbers, 'Int32', default='101,102,110')}}
```

### JSON parameters (https://www.tinybird.co/docs/forward/query-data/query-parameters#json-parameters)

Consider the following stringified JSON:

```
"filters": [
    {
        "operand": "date",
        "operator": "equals",
        "value": "2018-01-02"
    },
    {
        "operand": "high",
        "operator": "greater_than",
        "value": "100"
    },
    {
        "operand": "symbol",
        "operator": "in_list",
        "value": "AAPL,AMZN"
    }
]
```
You can use the `JSON()` function to use `filters` as a query parameter. The following example shows to use the `filters` field from the JSON snippet with the stock_prices_1m sample dataset.

```
%
SELECT symbol, date, high
FROM stock_prices_1m
WHERE
    1
    {% if defined(filters) %}
        {% for item in JSON(filters, '[]') %}
            {% if item.get('operator', '') == 'equals' %}
                AND {{ column(item.get('operand', '')) }} == {{ item.get('value', '') }}
            {% elif item.get('operator') == 'greater_than' %}
                AND {{ column(item.get('operand', '')) }} > {{ item.get('value', '') }}
            {% elif item.get('operator') == 'in_list' %}
                AND {{ column(item.get('operand', '')) }} IN splitByChar(',',{{ item.get('value', '') }})
            {% end %}
        {% end %}
    {% end %}
```
When accessing the fields in a JSON object, use the following syntax:

```
item.get('Field', 'Default value to avoid SQL errors').
```

## Pagination (https://www.tinybird.co/docs/forward/query-data/query-parameters#pagination)

You paginate results by adding `LIMIT` and `OFFSET` clauses to your query. You can parameterize the values of these clauses, allowing you to pass pagination values as query parameters to your Endpoint or Query API request.

Use the `LIMIT` clause to select only the first `n` rows of a query result. Use the `OFFSET` clause to skip `n` rows from the beginning of a query result. Together, you can dynamically chunk the results of a query up into pages.

For example, the following query introduces two dynamic parameters, `page_size` and `page`. Use them to control the pagination of a query result.

Paging results using dynamic parameters
```
%
SELECT * FROM TR
LIMIT {{Int32(page_size, 100)}}
OFFSET {{Int32(page, 0) * Int32(page_size, 100)}}
```
You can also use pages to perform calculations such as `count()`. The following example counts the total number of pages:

Operation on a paginated endpoint
```
%
SELECT count() as total_rows, ceil(total_rows/{{Int32(page_size, 100)}}) pages FROM endpoint_to_paginate
```
The addition of a `LIMIT` clause to a query also adds the `rows_before_limit_at_least` field to the response metadata. `rows_before_limit_at_least` is the lower bound on the number of rows returned by the query after transformations but before the limit was applied, and can be useful for response handling calculations.

**Example:**

```
SELECT * FROM users WHERE active = true LIMIT 10
```
Response might include:

- `rows`: 10 (actual rows returned)
- `rows_before_limit_at_least`: 1,000 (at least this many active users exist)
**Why "at least"?** ClickHouse® can stop counting once it knows the answer exceeds the `LIMIT` value. This makes it a lower bound, not an exact count.

**Useful for:**

- Pagination logic ("showing 10 of at least 1,247 results")
- Determining if more results exist without running a separate `COUNT()` query
This metadata is returned automatically with any `LIMIT` query, saving you from running an additional `COUNT(*)` query to check if more results are available.

To get consistent pagination results, add an `ORDER BY` clause to your paginated queries.

## Advanced templating (https://www.tinybird.co/docs/forward/query-data/query-parameters#advanced-templating)

To build more complex queries, use flow control operators like `if`, `else` and `elif` in combination with the `defined()` function, which helps you check whether a parameter has been received and act accordingly.

Tinybird's templating system is based on the [Tornado Python framework](https://github.com/tornadoweb/tornado), and uses Python syntax. You must enclose control statements in curly brackets with percentages `{%..%}` as in the following example:

Advanced templating using dynamic parameters
```
%
SELECT
  toDate(start_datetime) as day,
  countIf(status_code < 400) requests,
  countIf(status_code >= 400) errors,
  avg(duration) avg_duration
FROM
  log_events
WHERE
  endsWith(user_email, {{String(email, 'gmail.com')}}) AND
  start_datetime >= {{DateTime(start_date, '2019-09-20 00:00:00')}} AND
  start_datetime <= {{DateTime(end_date, '2019-10-10 00:00:00')}}
  {% if method != 'All' %} AND method = {{String(method,'POST')}} {% end %}
GROUP BY
  day
ORDER BY
  day DESC
```

### Validate required parameters (https://www.tinybird.co/docs/forward/query-data/query-parameters#validate-required-parameters)

Validate if a param is in the query
```
%
SELECT * FROM table
{% if defined(my_filter) %}
WHERE attr > {{Int32(my_filter)}}
{% end %}
```
When you call the query with `my_filter=20`, it applies the filter.

### Set default values and understand placeholders (https://www.tinybird.co/docs/forward/query-data/query-parameters#set-default-values-and-understand-placeholders)

Following best practices, you should set default parameter values as follows:

Default parameter values
```
%
SELECT * FROM table
WHERE attr > {{Int32(my_filter, 10)}}
```
When you call the query without setting any value to `my_filter`, it automatically applies the default value of 10.

If you don't set a default value for a parameter, you should validate that the parameter is defined before using it in the query as explained previously.

If you don't validate the parameter and it's not defined, the query might fail. Tinybird populates the parameter with a placeholder value based on the data type. For instance, numerical data types are populated with 0, strings with `__no_value__`, and date and timestamps with `2019-01-01` and `2019-01-01 00:00:00` respectively. You could try yourself with a query like this:

Get placeholder values
```
%
  SELECT
      {{String(param)}} as placeholder_string,
      {{Int32(param)}} as placeholder_num,
      {{Boolean(param)}} as placeholder_bool,
      {{Float32(param)}} as placeholder_float,
      {{Date(param)}} as placeholder_date,
      {{DateTime(param)}} as placeholder_ts,
      {{Array(param)}} as placeholder_array
```
This returns the following values:

```
{
  "placeholder_string": "__no_value__",
  "placeholder_num": 0,
  "placeholder_bool": 0,
  "placeholder_float": 0,
  "placeholder_date": "2019-01-01",
  "placeholder_ts": "2019-01-01 00:00:00",
  "placeholder_array": ["__no_value__0","__no_value__1"]
}
```

### Test parameters in the UI (https://www.tinybird.co/docs/forward/query-data/query-parameters#test-parameters-in-the-ui)

Any dynamic parameters you create appear in the UI. Select **Test new values** to open a test dialog populated with the default value of your parameters. The test dialog helps you test different Pipe values than the default ones without impacting production environments.

Use the View API page to see Endpoint metrics resulting from that specific combination of parameters. Close the dialog to bring the Pipe back to its default production state.

When testing parameters, you can modify both the SQL code and the parameters.

### Encode special characters in test parameters (https://www.tinybird.co/docs/forward/query-data/query-parameters#encode-special-characters-in-test-parameters)

When using the Test UI for parameters, certain special characters need to be encoded:

| Character | Status in Test UI | URI Encoding |
| --- | --- | --- |
| `#` | Truncates the value if not encoded | `%23` |
| `&` | Used as a parameter separator | `%26` |
| `+` | Interpreted as a space character | `%2B` |
| `%` | Reserved for escape sequences | `%25` |
| `"` | Needs encoding | `%22` |
The Forward UI supports special characters without encoding, except `%` and `"`. These character values are not supported.

### Cascade parameters across Pipes (https://www.tinybird.co/docs/forward/query-data/query-parameters#cascade-parameters-across-pipes)

Parameters with the same name in different Pipes are cascaded down the dependency chain.

For example, publish Pipe A with the parameter `foo`, then publish Pipe B with the same parameter and Pipe A as a Data Source. When you call Pipe B's API Endpoint with `foo=bar`, the value of `foo` is `bar` in both Pipes.

### Return custom errors (https://www.tinybird.co/docs/forward/query-data/query-parameters#return-custom-errors)

The following example stops query processing and returns a 400 error:

Validate if a param is defined and throw an error if it's not defined
```
%
{% if not defined(my_filter) %}
{{ error('my_filter (int32) query param is required') }}
{% end %}
SELECT * FROM table
WHERE attr > {{Int32(my_filter)}}
```
The `custom_error` function is an advanced version of `error` where you can customize the response and other aspects. The function gets an object as the first argument, which is sent as JSON, and the status_code as a second argument, which defaults to 400.

Validate if a param is defined and throw an error if it's not defined
```
%
{% if not defined(my_filter) %}
{{ custom_error({'error_id': 10001, 'error': 'my_filter (int32) query param is required'}) }}
{% end %}
SELECT * FROM table
WHERE attr > {{Int32(my_filter)}}
```

## Limits (https://www.tinybird.co/docs/forward/query-data/query-parameters#limits)

You can't use query parameters in nodes that are published as [Materialized Views](/docs/forward/core-concepts/materialized-views), only in Endpoints, Query API requests, or on-demand copies or sinks.

You can use query parameters in scheduled sinks and copies, but must have a default. That default is used in the scheduled execution. The preview step fails if the default doesn't exist.

---

URL: https://www.tinybird.co/docs/forward/query-data/playgrounds
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Playgrounds \| Tinybird"
theme-color: "#171612"
description: "Prototype and test SQL queries in Tinybird Playgrounds."
inkeep:version: "forward"
---

# Playgrounds (https://www.tinybird.co/docs/forward/query-data/playgrounds#playgrounds)

Playgrounds are interactive SQL workspaces in Tinybird Cloud. Use them to explore data, iterate on query logic, and validate results before turning the SQL into a Pipe or Endpoint.

## Getting started (https://www.tinybird.co/docs/forward/query-data/playgrounds#getting-started)

1. Go to [Tinybird Cloud](https://cloud.tinybird.co/) and log in to your account.
2. Select **Playgrounds** in the sidebar.
3. Select **New playground**.
4. Give the playground a title and optional description.
5. Start writing SQL.

![Playgrounds editor with the playground list, description, and SQL editor](/docs/_next/image?url=%252Fdocs%252Fimg%252Fplaygrounds-fwd.png&w=3840&q=75)

For the statements, functions, and settings you can use in queries, see the [SQL reference](/docs/sql-reference).

## Playground workflow (https://www.tinybird.co/docs/forward/query-data/playgrounds#playground-workflow)

**Organize your playground**

Use the list on the left to switch between playgrounds. Mark favorites for quick access and use filters like **Favorites** or **Private** to focus the list.

**Build queries with nodes**

Each playground is made of nodes. Keep nodes focused and readable so you can build multi-step logic and iterate quickly.

**Run and review results**

Use **Run** to execute a node and preview results. Adjust the row limit (for example, **Limited to 100**) or export results when you need to validate outputs outside Tinybird.

**Format and document**

Use **Format** to keep SQL readable. Add node descriptions, or use **Update node description using AI** to keep your work self-documented as it grows.

**Share with your team**

Use **Share** to make a playground available to your workspace when it is ready to review.

**Work with context**

Select the source to open a side panel, then review in-context metadata like columns, settings, and query plans to validate results.

<!-- <figure> -->

![Video]()
Controls: true

<!-- <figcaption> -->
The side panel shows extra context when you click a Data Source.
<!-- </figcaption> -->

<!-- </figure> -->

## From playground to production (https://www.tinybird.co/docs/forward/query-data/playgrounds#from-playground-to-production)

When a query is ready, move the SQL into a Pipe so it can be versioned, deployed, and used by Endpoints or other resources.

## Next steps (https://www.tinybird.co/docs/forward/query-data/playgrounds#next-steps)

- Learn about [Time Series](/docs/forward/query-data/time-series) to quickly visualize data.
- Learn about [Pipes](/docs/forward/core-concepts/pipes).
- Learn about [API Endpoints](/docs/forward/core-concepts/api-endpoints).

---

URL: https://www.tinybird.co/docs/forward/query-data/mcp
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "MCP \| Tinybird"
theme-color: "#171612"
description: "Connect AI agents to Tinybird with the remote MCP server."
inkeep:version: "forward"
---

# MCP (https://www.tinybird.co/docs/forward/query-data/mcp#mcp)

The Tinybird remote MCP server lets AI agents query your Workspace and use published API Endpoints as tools. MCP stands for [Model Context Protocol](https://modelcontextprotocol.io/), a standard interface for exposing tools and context to agents.

Use MCP when you want an agent to explore data, call analytics Endpoints, or answer questions over Tinybird data without writing a custom API integration.

The Tinybird MCP server supports Streamable HTTP. If your MCP client doesn't support Streamable HTTP, use `mcp-remote` as a bridge.

## Before you start (https://www.tinybird.co/docs/forward/query-data/mcp#before-you-start)

You need:

- A Tinybird account and Workspace.
- A Token with the scopes required by the tools you want to expose.
- An MCP client that supports Streamable HTTP, or a client that can run `mcp-remote`.

## Connect to the MCP server (https://www.tinybird.co/docs/forward/query-data/mcp#connect-to-the-mcp-server)

Use this URL in your MCP client:

```
https://mcp.tinybird.co?token=TINYBIRD_TOKEN
```
Replace `TINYBIRD_TOKEN` with a Tinybird Token.

Use a resource-scoped static Token or a JWT when you want to restrict which Endpoints or Data Sources the agent can access. Use an admin Token only for trusted internal tools.

### Clients that need a bridge (https://www.tinybird.co/docs/forward/query-data/mcp#clients-that-need-a-bridge)

For MCP clients that do not support Streamable HTTP directly, configure `mcp-remote`:

```
{
  "mcpServers": {
    "tinybird": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://mcp.tinybird.co?token=TINYBIRD_TOKEN"
      ]
    }
  }
}
```
New Tinybird Tokens include the Tinybird API host. For older Tokens, pass the API host as a query parameter: `https://mcp.tinybird.co?token=TINYBIRD_TOKEN&host=https://api.tinybird.co`. See [Regions and endpoints](/docs/api-reference#regions-and-endpoints).

### JWTs and non-default regions (https://www.tinybird.co/docs/forward/query-data/mcp#jwts-and-non-default-regions)

JWTs do not contain region information. When you use a JWT outside the default region, include the `host` parameter:

```
https://mcp.tinybird.co?token=JWT_TOKEN&host=https://api.REGION.PROVIDER.tinybird.co
```
Examples:

- EU Central 1 AWS: `https://mcp.tinybird.co?token=JWT_TOKEN&host=https://api.eu-central-1.aws.tinybird.co`
- US East 4 GCP: `https://mcp.tinybird.co?token=JWT_TOKEN&host=https://api.us-east.tinybird.co`
See [Regions and endpoints](/docs/api-reference#regions-and-endpoints) for the full list.

## Authentication and Token scopes (https://www.tinybird.co/docs/forward/query-data/mcp#authentication-and-token-scopes)

The tools exposed by the MCP server depend on the Token scopes.

Use static Tokens for backend and internal tools. An admin Token exposes all available tools. Resource-scoped Tokens expose only the resources allowed by the Token.

Use JWTs for user-facing or multi-tenant agents. JWTs can restrict access to specific Endpoints, add fixed parameters, filter Data Sources, and set rate limits. See [Tokens](/docs/forward/core-concepts/tokens) and [JWTs](/docs/forward/core-concepts/jwt).

| Tool | JWT | Admin Token |
| --- | --- | --- |
| Endpoint tools | Specific Endpoints only | ✓ |
| `list_endpoints` | ✓ | ✓ |
| `list_datasources` | ✓ | ✓ |
| `list_service_datasources` | ✓ | ✓ |
| `execute_query` | ✓ | ✓ |
| `explore_data` | ✓ | ✓ |
| `text_to_sql` | ✓ | ✓ |

## Available tools (https://www.tinybird.co/docs/forward/query-data/mcp#available-tools)

The MCP server exposes Endpoint tools and core tools.

### Endpoint tools (https://www.tinybird.co/docs/forward/query-data/mcp#endpoint-tools)

Every published Endpoint in your Workspace can become a tool with the Endpoint name.

Endpoint tools:

- Accept the same parameters as the Endpoint.
- Return the same response as direct API calls.
- Return CSV by default. JSON is also supported.
- Respect Endpoint rate limits and Token scopes.
For example, an Endpoint named `daily_active_users` becomes a tool named `daily_active_users`.

### Core tools (https://www.tinybird.co/docs/forward/query-data/mcp#core-tools)

`explore_data`

: Ask questions about your data and get a natural language answer.

`text_to_sql`

: Convert a natural language question into a SQL query.

`execute_query`

: Run SQL against the Tinybird Query API. The `sql` parameter is required. The optional `format` parameter controls the response format. The default is `CSVWithNames`.

`list_datasources`

: List Data Sources in your Workspace.

`list_service_datasources`

: List Organization and Tinybird service Data Sources available to your Workspace.

`list_endpoints`

: List published Endpoints with names, parameters, and descriptions.

## Use MCP or direct APIs (https://www.tinybird.co/docs/forward/query-data/mcp#use-mcp-or-direct-apis)

Use MCP when:

- You are building an AI agent that needs access to analytics data.
- You want a conversational interface over Data Sources or Endpoints.
- You are prototyping data analysis workflows with an LLM.
- You want Endpoints to become tools without writing tool wrappers.
Use direct APIs when:

- You are building a production app with fixed query patterns.
- You need the lowest possible latency.
- You need custom caching, retries, observability, or request handling.

## Monitor MCP usage (https://www.tinybird.co/docs/forward/query-data/mcp#monitor-mcp-usage)

MCP requests include `from=mcp`. Use Tinybird service Data Sources to monitor queries executed by agents.

```
SELECT *
FROM tinybird.pipe_stats_rt
WHERE url LIKE '%from=mcp%'
AND start_datetime > now() - INTERVAL 1 HOUR
```
See [Service Data Sources](/docs/forward/monitoring/service-datasources).

## Troubleshooting (https://www.tinybird.co/docs/forward/query-data/mcp#troubleshooting)

### Workspace not found with a JWT (https://www.tinybird.co/docs/forward/query-data/mcp#workspace-not-found-with-a-jwt)

If you receive an error like this:

```
Error calling TinybirdAPI: invalid authentication token. Workspace with ID ... not found
```
You might be using a JWT without a `host` parameter. JWTs do not include region information. Add the regional API host to the MCP URL:

```
https://mcp.tinybird.co?token=JWT_TOKEN&host=https://api.REGION.PROVIDER.tinybird.co
```
See [JWTs and non-default regions](https://www.tinybird.co/docs/forward/query-data/mcp#jwts-and-non-default-regions).

## See also (https://www.tinybird.co/docs/forward/query-data/mcp#see-also)

- [Tokens](/docs/forward/core-concepts/tokens)
- [Endpoints](/docs/forward/core-concepts/api-endpoints)
- [Query API](/docs/api-reference/query-api)
- [Model Context Protocol documentation](https://modelcontextprotocol.io/)

---

URL: https://www.tinybird.co/docs/forward/query-data/explorations
Last update: 2026-07-31T11:10:11.000Z
Content:
---
title: "Explorations \| Tinybird"
theme-color: "#171612"
description: "Explore and understand your data with natural language in Tinybird."
inkeep:version: "forward"
---

# Explorations (https://www.tinybird.co/docs/forward/query-data/explorations#explorations)

Explorations is a conversational UI feature in Tinybird. With Explorations, explore and interact with your data using natural language.

## Getting started (https://www.tinybird.co/docs/forward/query-data/explorations#getting-started)

To start using Explorations:

1. Go to [Tinybird Cloud](https://cloud.tinybird.co/) and log in to your account
2. Select **Explorations** in the sidebar
3. Start with a suggested prompt or ask your own question
4. Begin exploring your data

![Explorations initial screen](/docs/_next/image?url=%252Fdocs%252Fimg%252Fexplorations-empty.png&w=3840&q=75)

## How it works (https://www.tinybird.co/docs/forward/query-data/explorations#how-it-works)

Explorations provides conversational analysis through a chat interface. Ask questions in natural language and receive structured, step-by-step responses. The AI breaks down complex analyses into clear, practical steps that build upon each other. The chat also suggests follow-up questions that reference previous results, creating a natural flow where each question deepens your understanding of the data.

![An example of a conversation in Explorations](/docs/_next/image?url=%252Fdocs%252Fimg%252Fexplorations-chat.png&w=3840&q=75)

If you want to write your own SQL queries, use [Playgrounds](/docs/forward/query-data/playgrounds). For time-based data visualization, use Time Series.

## Get the most out of Explorations (https://www.tinybird.co/docs/forward/query-data/explorations#get-the-most-out-of-explorations)

### Custom workspace rules (https://www.tinybird.co/docs/forward/query-data/explorations#custom-workspace-rules)

Fine-tune how the AI assistant responds to get faster and more relevant results. Inside an Exploration, select the **workspace rules** link in the chat prompt input area to add your custom rules. The assistant uses this context in all your prompts.

### Add context (https://www.tinybird.co/docs/forward/query-data/explorations#add-context)

Select the `@` icon near the submit button to add specific resources to your query. The LLM prioritizes them for that specific prompt.

### Getting better results (https://www.tinybird.co/docs/forward/query-data/explorations#getting-better-results)

Start with specific questions rather than broad exploratory queries. Instead of asking "show the data," try "what are the top 5 pages by traffic in the last 7 days?" The AI performs better with clear, focused requests that have measurable outcomes.

## Usage and billing (https://www.tinybird.co/docs/forward/query-data/explorations#usage-and-billing)

At the moment, Explorations usage is free for Developer and Enterprise organizations. Free organizations receive a limited number of credits per month per organization. Tinybird restricts them once they consume all credits. Read more in [Free plan LLM usage](/docs/forward/pricing/free#llm-usage).

Usage can be tracked using the <a href="/docs/forward/monitoring/service-datasources?q=service#tinybird-llm-usage">`llm_usage`</a> service data source.

## Limitations and privacy (https://www.tinybird.co/docs/forward/query-data/explorations#limitations-and-privacy)

Explorations are available to all roles within a Workspace.

The Explorations AI chat is only available in Tinybird Cloud. Tinybird anonymously processes a limited sample of your data to deliver faster, more accurate results.

Currently, Explorations cannot be shared or exported. Each user can only access their own Explorations.

## Next steps (https://www.tinybird.co/docs/forward/query-data/explorations#next-steps)

- Learn about [Playgrounds](/docs/forward/query-data/playgrounds) to write your own SQL queries.
- Learn about [Time Series](/docs/forward/query-data/time-series) to quickly visualize data.
- Learn about [Pipes](/docs/forward/core-concepts/pipes).
- Learn about [Materialized views](/docs/forward/core-concepts/materialized-views).

---

URL: https://www.tinybird.co/docs/forward/query-data/clickhouse-interface
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "ClickHouse® interface \| Tinybird"
theme-color: "#171612"
description: "Learn how to connect BI tools and SQL clients to Tinybird using the ClickHouse HTTP interface for data visualization and analysis."
inkeep:version: "forward"
---

# ClickHouse® interface (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#clickhouse-interface)

Tinybird is compatible with the ClickHouse® HTTP interface, so BI tools, SQL clients, and data visualization platforms can query Tinybird Data Sources directly.

Use the ClickHouse interface when a tool already supports ClickHouse and needs read-only SQL access to Tinybird.

The ClickHouse connection to Tinybird is read-only. You can use it to query, visualize, and analyze data from your Tinybird Data Sources, but you cannot modify data through this connection.

## Connection parameters (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#connection-parameters)

To connect any ClickHouse®-compatible tool to Tinybird, use these standard connection parameters:

### Basic Connection settings (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#basic-connection-settings)

```
Protocol: HTTPS
Host: clickhouse.<REGION>.tinybird.co
Port: 443 (HTTPS)
SSL/TLS: Required (enabled)
```
Replace `<REGION>` with your workspace region (e.g., `eu-west-1.aws`, `us-east-1`).

### ClickHouse interface hosts (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts)

Here are the ClickHouse HTTP hosts for each Tinybird region:

| Region | Provider | Provider region | ClickHouse interface URL |
| --- | --- | --- | --- |
| Europe | GCP | europe-west2 | clickhouse.europe-west2.gcp.tinybird.co |
| Europe | GCP | europe-west3 | clickhouse.tinybird.co |
| US East | GCP | us-east4 | clickhouse.us-east.tinybird.co |
| North America | GCP | northamerica-northeast2 | clickhouse.northamerica-northeast2.gcp.tinybird.co |
| Europe | AWS | eu-central-1 | clickhouse.eu-central-1.aws.tinybird.co |
| Europe | AWS | eu-west-1 | clickhouse.eu-west-1.aws.tinybird.co |
| US East | AWS | us-east-1 | clickhouse.us-east.aws.tinybird.co |
| US West | AWS | us-west-2 | clickhouse.us-west-2.aws.tinybird.co |

### Authentication (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#authentication)

```
Username: <WORKSPACE_NAME>  # Optional, for identification purposes
Password: <TOKEN>           # Your Tinybird Auth Token
```
The username field is not used for authentication but helps identify the connection. The actual authentication is done via the Auth Token in the password field.

### Auth Token requirements (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#auth-token-requirements)

Your Auth Token must have the following permissions:

- **Read access** to Workspace Data Sources
- **Read access** to Service Data Sources (if needed)
- **Read access** to Organization Data Sources (if needed)
Create a token with these permissions:

```
tb --cloud token create static <token_name> --scope "WORKSPACE:READ_ALL" --scope "ORG_DATASOURCES:READ"
```

### Tinybird Local (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#tinybird-local)

To connect to your local development environment using Tinybird Local, use these connection parameters:

```
Protocol: HTTP
Host: localhost
Port: 7182
SSL/TLS: Disabled
```
You can test the connection using curl:

```
curl -H "X-ClickHouse-Key: your_token_here" \
     "http://localhost:7182/?query=SELECT%20*%20FROM%20system.tables"
```

## Available databases (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#available-databases)

The ClickHouse interface exposes four databases:

- **`<your_workspace_name>`**: Contains your Workspace's Data Sources.
- **`tinybird`**: Contains [Workspace Service Data Sources](/docs/forward/monitoring/service-datasources#service-data-sources), such as `tinybird.datasources_ops_log` and `tinybird.pipe_stats_rt`.
- **`organization`**: Contains [Organization Service Data Sources](/docs/forward/monitoring/service-datasources#organization-service-data-sources), such as `organization.workspaces` and `organization.pipe_stats_rt`.
- **`system`**: Contains a subset of system tables, including `system.tables`, `system.columns`, `system.parts`, `system.part_log`, `system.query_views_log`, and `system.query_log`.
Querying `system.query_log`, `system.query_views_log` or `system.part_log` without filters is too resource-intensive and times out. Always apply a time filter on the `event_time` column to query a reasonable time frame.

## Compatible tools (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#compatible-tools)

Tinybird's ClickHouse® interface works with BI tools and SQL clients that support ClickHouse:

### Business intelligence platforms (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#business-intelligence-platforms)

- **[Grafana](/docs/forward/guides/connect-grafana)**
- **[Hex](/docs/forward/guides/connect-hex)**
- **[Metabase](/docs/forward/guides/connect-metabase)**
- **[Superset](/docs/forward/guides/connect-superset)**
- **[Power BI](/docs/forward/guides/connect-powerbi)**
- **[Redash](/docs/forward/guides/connect-redash)**
- **[Chartbrew](/docs/forward/guides/connect-chartbrew)**
- **[Deepnote](/docs/forward/guides/connect-deepnote)**
- **[Draxlr](/docs/forward/guides/connect-draxlr)**
- **[Fabi.ai](/docs/forward/guides/connect-fabi)**
- **[Holistics](/docs/forward/guides/connect-holistics)**
- **[Luzmo](/docs/forward/guides/connect-luzmo)**
- **[Mitzu](/docs/forward/guides/connect-mitzu)**
- **[Tableau](/docs/forward/guides/connect-tableau)**

### SQL clients and IDEs (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#sql-clients-and-ides)

- **[DataGrip](/docs/forward/guides/connect-datagrip)**
- **[DBeaver](/docs/forward/guides/connect-dbeaver)**

### Application clients (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#application-clients)

- **[ClickHouse JS Client](/docs/forward/guides/connect-clickhouse-js)**
- **[ClickHouse Go Client](/docs/forward/guides/connect-clickhouse-go)**
- **[ClickHouse Python Client](/docs/forward/guides/connect-clickhouse-python)**

### Other compatible tools (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#other-compatible-tools)

Any tool that supports ClickHouse® HTTP interface can potentially work with Tinybird, including:

- Looker (via custom connection)
- Other JetBrains IDEs
- Retool and other low-code platforms

## Limitations (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#limitations)

The interface is **read-only**. You cannot perform `INSERT`, `UPDATE`, `DELETE`, or any DDL operations (`CREATE`, `ALTER`, `DROP`).

## Observability (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#observability)

Queries made through the ClickHouse® interface are logged as [Query API](/docs/api-reference/query-api) requests in the pipe statistics Data Sources. You can monitor these queries using:

- `tinybird.pipe_stats_rt` and `tinybird.pipe_stats` - Workspace-level statistics
- `organization.pipe_stats_rt` and `organization.pipe_stats` - Organization-level statistics
ClickHouse® queries appear with `pipe_id` and `pipe_name` as `'query_api'`. The actual SQL query is available in the `parameters` column under the `q` field.

For more details, see [Service Data Sources](/docs/forward/monitoring/service-datasources).

## Troubleshooting (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#troubleshooting)

**Invalid host or region**

```
Error: Connection refused
```
Verify your region and use the correct host format: `clickhouse.<region>.tinybird.co`

**Authentication failures**

```
Error: Authentication failed
```

- Check your Auth Token is valid and not expired
- Ensure the token has read permissions for the workspace
- Verify the token is entered in the password field
**SSL/TLS errors**

```
Error: SSL connection error
```

- Ensure SSL/TLS is enabled in your client
- Use port 443 with HTTPS protocol
- Check that your client supports TLS 1.2 or higher.
**Query timeouts**

```
Error: Query timeout
```

- Add appropriate WHERE clauses to filter data
- Use LIMIT to reduce result set size
- Consider querying Materialized Views instead of raw data

### Getting help (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#getting-help)

If you encounter issues not covered here:

1. Check the specific tool integration guide
2. Review your [Auth Token](https://cloud.tinybird.co/tokens) permissions
3. Test the connection with a simple query
4. Contact [Tinybird support](/docs/forward/support) for additional assistance

## Next steps (https://www.tinybird.co/docs/forward/query-data/clickhouse-interface#next-steps)

Connect your BI tool and follow the specific integration guide to set up dashboards and visualizations with your Tinybird data.

- [ClickHouse JS](/docs/forward/guides/connect-clickhouse-js)
- [ClickHouse Go](/docs/forward/guides/connect-clickhouse-go)
- [ClickHouse Python](/docs/forward/guides/connect-clickhouse-python)
- [Grafana](/docs/forward/guides/connect-grafana)
- [Metabase](/docs/forward/guides/connect-metabase)

---

URL: https://www.tinybird.co/docs/forward/query-data/api-endpoints
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "API Endpoints \| Tinybird"
theme-color: "#171612"
description: "Query Tinybird API Endpoints from applications and tools."
inkeep:version: "forward"
---

# API Endpoints (https://www.tinybird.co/docs/forward/query-data/api-endpoints#api-endpoints)

[Endpoints](/docs/forward/core-concepts/api-endpoints) publish the result of a Pipe as a REST API. Use them when an app, dashboard, agent, or external service needs a stable query contract with predictable inputs and outputs.

Use Endpoints when the query belongs to your Tinybird project and callers should only provide request parameters. Use the [Query API](/docs/forward/query-data/sql-api) when the caller owns the SQL, such as an internal tool, notebook, script, or debugging workflow.

## Call an Endpoint (https://www.tinybird.co/docs/forward/query-data/api-endpoints#call-an-endpoint)

Call an Endpoint by name and include a Token with read permissions for the Endpoint. Use a [static Token](/docs/forward/core-concepts/static-tokens) for backend services and trusted internal tools. Use a [JWT](/docs/forward/core-concepts/jwt) for browser, user-facing, or multi-tenant access where each user or tenant needs scoped access, fixed parameters, filters, expiration, or rate limits.

```
curl \
  -H "Authorization: Bearer <PIPE:READ token or JWT>" \
  "https://api.tinybird.co/v0/pipes/top_pages.json"
```
You can also pass the Token or JWT as a `token` query parameter when needed:

```
curl "https://api.tinybird.co/v0/pipes/top_pages.json?token=<PIPE:READ token or JWT>"
```

## Use query parameters (https://www.tinybird.co/docs/forward/query-data/api-endpoints#use-query-parameters)

Endpoints can use Tinybird's [templating language](/docs/forward/core-concepts/templating-language) to receive typed, validated request parameters. Use parameters for filters, date ranges, pagination, tenant IDs, sorting, and limits.

```
curl \
  -H "Authorization: Bearer <PIPE:READ token>" \
  "https://api.tinybird.co/v0/pipes/top_pages.json?start_date=2026-01-01%2000:00:00&limit=20"
```
For parameter syntax, defaults, helper functions, pagination, and advanced patterns, see [Query parameters](/docs/forward/query-data/query-parameters).

## Choose a response format (https://www.tinybird.co/docs/forward/query-data/api-endpoints#choose-a-response-format)

Choose the response format by appending the format extension to the Endpoint URL.

```
curl \
  -H "Authorization: Bearer <PIPE:READ token>" \
  "https://api.tinybird.co/v0/pipes/top_pages.ndjson"
```
Supported formats include:

- `.json`: JSON response. This is the default format.
- `.csv`: CSV response.
- `.csvwithnames`: CSV response with column names.
- `.ndjson`: Newline-delimited JSON response.
- `.parquet`: Parquet response.
- `.prometheus`: Prometheus metrics response.
See [Endpoint response formats](/docs/forward/core-concepts/api-endpoints#response-formats) and the [Pipe API reference](/docs/api-reference/pipe-api/api-endpoints#response-formats) for more details.

## Errors and retries (https://www.tinybird.co/docs/forward/query-data/api-endpoints#errors-and-retries)

Endpoints return standard HTTP success or error codes. Error responses include extra information about what went wrong, encoded as JSON.

### Error codes (https://www.tinybird.co/docs/forward/query-data/api-endpoints#error-codes)

Endpoints might return the following HTTP error codes:

| Code | Description |
| --- | --- |
| 400 | Bad request. A `HTTP400` can be returned in several scenarios and typically represents a malformed request, such as errors in your SQL queries or missing query parameters. |
| 403 | Forbidden. The auth Token doesn't have the correct scopes. |
| 404 | Not found. This usually occurs when the name of the Endpoint is wrong or hasn't been published. |
| 405 | HTTP method not allowed. The request method isn't allowed for this Endpoint. |
| 408 | Request timeout. This occurs when the query takes too long to complete. By default, this is 10 seconds. |
| 414 | Request-URI Too Large. Not all APIs have the same limit, but it's usually 2KB for GET requests. Reduce the URI length or use a POST request to avoid the limit. |
| 429 | Too many requests. Usually occurs when an Endpoint hits rate limits. |
| 499 | Connection closed. This occurs if the client closes the connection after 1 second. If this is unexpected, increase the connection timeout on your end. |
| 500 | Internal Server Error. Usually an unexpected transient service error. |
Errors when running a query are usually reported as `400 Bad request` or `500 Internal Server Error`, depending on whether the caller can fix the error.

In those cases, the API response includes an additional `X-DB-Exception-Code` HTTP header with the internal database error as a stringified number.

### Retries (https://www.tinybird.co/docs/forward/query-data/api-endpoints#retries)

When you call an Endpoint from an API Gateway, backend service, or job, handle potential errors and implement retry strategies where appropriate.

Implement automatic retries for the following errors:

- HTTP 429: Too many requests
- HTTP 500: Internal Server Error
Use exponential backoff when retrying requests that return these errors.

## Track Endpoint usage (https://www.tinybird.co/docs/forward/query-data/api-endpoints#track-endpoint-usage)

Endpoint requests are logged in Service Data Sources. Add metadata as request parameters when you want to group requests by app, customer, workflow, or agent.

```
curl \
  -H "Authorization: Bearer <PIPE:READ token>" \
  "https://api.tinybird.co/v0/pipes/top_pages.json?tenant_id=acme&app_name=dashboard"
```
See [Endpoint performance](/docs/forward/monitoring/analyze-endpoints-performance) for monitoring examples.

## Next steps (https://www.tinybird.co/docs/forward/query-data/api-endpoints#next-steps)

- Learn the [Endpoints concept](/docs/forward/core-concepts/api-endpoints).
- Add dynamic inputs with [Query parameters](/docs/forward/query-data/query-parameters).
- Use the [Query API](/docs/forward/query-data/sql-api) for caller-owned SQL.
- Review the [Pipe API reference](/docs/api-reference/pipe-api/api-endpoints).

---

URL: https://www.tinybird.co/docs/forward/pricing/saas
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "SaaS plan \| Tinybird"
theme-color: "#171612"
description: "The Tinybird SaaS plan — shared infrastructure with 4+ vCPUs, credit-based billing, and mandatory support."
inkeep:version: "forward"
---

# SaaS plan (https://www.tinybird.co/docs/forward/pricing/saas#saas-plan)

The SaaS plan runs on **shared infrastructure** with a commitment. It starts at 4 vCPUs and uses a credit-based billing system with a minimum 12-month commitment. Billing uses active minutes, QPS overages, storage overages, and related SKUs. Minimum storage is 1 TB. A support package is mandatory.

For plans on **dedicated infrastructure** with your own ClickHouse® cluster, see [Enterprise](./enterprise). For shared infrastructure without a commitment, see [Developer](./developer).

## What's included (https://www.tinybird.co/docs/forward/pricing/saas#whats-included)

SaaS plan sizes start at 4 vCPUs. For available sizes and what's included in each, refer to your contract or [contact sales](https://www.tinybird.co/contact-us).

## Billing (https://www.tinybird.co/docs/forward/pricing/saas#billing)

SaaS plans are billed based on the following resources:

| Resource | Billing | Type |
| --- | --- | --- |
| vCPUs | Number of vCPUs available. Starting at 4 vCPUs. Each vCPU size includes a number of active minutes. | Fixed. |
| Active minutes | An active minute is when any operation used a vCPU for a minute. See [Active minutes](https://www.tinybird.co/docs/forward/pricing/saas#active-vcpu-minutes--hours). | Usage based. |
| Queries per second (QPS) | Queries per second, or QPS. Each vCPU size includes a QPS allowance. See [Queries per second](https://www.tinybird.co/docs/forward/pricing/saas#queries-per-second). | Usage based when usage exceeds the QPS included in the plan. |
| Storage | Average of daily maximum usage of the compressed disk storage of all your data, in gigabytes. | Usage based when usage exceeds the minimum 1 TB of storage included in your contract. |
| Data transfer | When using [Sinks](/docs/api-reference/sink-pipes-api), usage is billed depending on the destination, which can be the same cloud provider and region as your Tinybird cluster, or a different one. | Usage based. |
| Support | Standard or Premier support package as agreed in your contract. Standard is included. Premier pricing is defined in your agreement. | Fixed. |
| LLM usage | LLM usage in AI tools within MCP and Explorations UI. Includes $200 per billing cycle. | Usage based when usage exceeds the $200 included in the plan. |
The same QPS, active minutes, and storage overage rates apply in every region. See [Pricing](https://www.tinybird.co/pricing) for current rates.

### Active vCPU minutes / hours (https://www.tinybird.co/docs/forward/pricing/saas#active-vcpu-minutes--hours)

An active minute is any minute where at least one operation used a vCPU, regardless of actual vCPU time consumed during that minute. Whether you use 1 vCPU second or 60 vCPU seconds within a minute, it counts as 1 active minute. Multiple operations executed within the same minute still count as a single active minute. When using fractioned vCPUs, an active minute is proportional to the fraction. For example, 30 seconds of 0.5 vCPU.

Plan sizes include a number of active hours, calculated as active minutes divided by 60. If you consume all your active minutes, the overage is billed at a fixed rate per minute. Usage bursts allow you to temporarily exceed the vCPU usage limit. See [vCPU burst mode](https://www.tinybird.co/docs/forward/pricing/saas#vcpu-burst-mode).

### Queries per second (https://www.tinybird.co/docs/forward/pricing/saas#queries-per-second)

Queries per second, or QPS, is the number of queries per second that your plan includes. Calls to API endpoints and queries sent to the Query API count towards your QPS allowance. The limit excludes queries from the UI.

If you exceed the QPS allowance, the platform supports traffic peaks up to a ceiling of 4x the QPS included in your plan. Requests that exceed the allowance incur a fixed charge per request. Requests beyond the ceiling are rate limited. See [QPS overages and QPS ceiling](https://www.tinybird.co/docs/forward/pricing/saas#qps-overages-and-qps-ceiling).

### QPS overages and QPS ceiling (https://www.tinybird.co/docs/forward/pricing/saas#qps-overages-and-qps-ceiling)

Once you are over your plan's QPS allowance, you can keep making requests up to the plan's QPS ceiling of 4 times your plan's QPS. Requests that exceed the allowance incur a fixed charge per request. This applies to API endpoint and Query API requests.

<!-- <figure> -->
![QPS Overages and QPS Ceiling](/docs/_next/image?url=%252Fdocs%252Fimg%252Fqps_chart.png&w=3840&q=75)

<!-- </figure> -->
While your organization is temporarily limited due to vCPU usage over the vCPU burst capacity, you don't have access to the QPS ceiling of QPS allowance × 4. Request rates are limited to your QPS allowance.

### vCPU burst mode (https://www.tinybird.co/docs/forward/pricing/saas#vcpu-burst-mode)

You can't exceed the vCPU time limit indefinitely. The limit relates to vCPU time used during a minute. Tinybird allows usage bursts of 2x the vCPU time per minute. If you reach the vCPU limit, you can use the base QPS limit of your plan size during the next 5 minutes without QPS overages.

### Max threads (https://www.tinybird.co/docs/forward/pricing/saas#max-threads)

Max threads is the maximum number of concurrent threads used to execute queries in API Endpoints and Query API calls. Your SaaS plan's thread allowance determines this limit.

While more threads can improve concurrent query processing, final performance also depends on your vCPU limit and query complexity.

### Data storage (https://www.tinybird.co/docs/forward/pricing/saas#data-storage)

You are charged for persistent data stored in ClickHouse® per Data Source. SaaS plans have a minimum of 1 TB of storage as defined in your contract. Storage beyond the included amount is billed at a fixed rate per GB. Rates aren't tied to region.

### Data transfer (https://www.tinybird.co/docs/forward/pricing/saas#data-transfer)

The data transfer SKU covers data sent out of the platform, primarily through [Sinks](/docs/forward/core-concepts/sinks). Data transfer is charged per gigabyte. The rate depends on the transfer type. Intra transfers stay within the same cloud provider and region. Inter transfers cross providers or regions.

### Support SKU (https://www.tinybird.co/docs/forward/pricing/saas#support-sku)

The Support SKU applies to SaaS plans. A support package is mandatory. Standard is included. Premier is available for SaaS S-24+. Support charges don't depend on ticket volume or platform activity. Premier pricing and final terms are defined in your customer agreement. See [Support](../support) for details.

### LLM usage (https://www.tinybird.co/docs/forward/pricing/saas#llm-usage)

LLM usage in AI tools within MCP and Explorations UI includes $200 of credits per billing cycle. Usage beyond included credits is billed at cost.

You can monitor usage through the <a href="../monitoring/service-datasources#tinybird-llm-usage">`tinybird.llm_usage`</a> Service Data Source.

## Credits and billing cycle (https://www.tinybird.co/docs/forward/pricing/saas#credits-and-billing-cycle)

SaaS plans are billed using credits. A credit is a unit of resource usage that can be consumed across Tinybird SKUs. The minimum commitment period is 12 months.

Bills are issued monthly based on credits consumed during the billing cycle (first to last day of the month). Invoices are issued when credits are purchased (for example, at contract signature or when buying additional credits).

## Monitor usage and invoices (https://www.tinybird.co/docs/forward/pricing/saas#monitor-usage-and-invoices)

You can monitor credit consumption, remaining commitment, and infrastructure usage from your organization's dashboard. You can also review monthly usage receipts.

## Support (https://www.tinybird.co/docs/forward/pricing/saas#support)

A support package is mandatory on SaaS plans. Standard is included. Premier is available for SaaS S-24+. Premier pricing and final terms are defined in your customer agreement. See [Support](../support) for full details.

## Resize and manage (https://www.tinybird.co/docs/forward/pricing/saas#resize-and-manage)

Resizing or making other changes to a SaaS plan requires [contacting sales](https://www.tinybird.co/contact-us).

## Next steps (https://www.tinybird.co/docs/forward/pricing/saas#next-steps)

- [Developer plan](./developer): Shared infrastructure without a commitment.
- [Enterprise plan](./enterprise) — Dedicated ClickHouse® cluster with no resource sharing.
- [Limits](./limits): Platform and operational limits.

---

URL: https://www.tinybird.co/docs/forward/pricing/limits
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Limits \| Tinybird"
theme-color: "#171612"
description: "Tinybird has limits on certain operations and processes to ensure the highest performance."
inkeep:version: "forward"
---

# Limits (https://www.tinybird.co/docs/forward/pricing/limits#limits)

Tinybird has limits on certain operations and processes to ensure the highest performance. All limits are applied at the organization level, meaning resources and limits are shared across all workspaces within your organization.

Each plan page documents plan-specific billing limits and SKUs:

- [Free plan](./free)
- [Developer plan](./developer)
- [SaaS plan](./saas)
- [Enterprise plan](./enterprise)

## Rate limiter (https://www.tinybird.co/docs/forward/pricing/limits#rate-limiter)

The rate limiter monitors cluster health and may temporarily limit concurrent requests to prevent instability due to memory or CPU pressure.

It activates when both conditions are met:

- The cluster is under resource pressure, either:
  - Memory usage exceeds the threshold: greater than 70% of cluster capacity on Free and Developer plans, or greater than 80% on Enterprise (dedicated) plans.
  - CPU is saturated: the 1-minute load average exceeds 3 times the number of available CPU cores.
- The percentage of `408 Timeout Exceeded` and `500 Internal Server Error` responses for a Pipe Endpoint exceeds 10% of its successful requests.
When active, the maximum concurrent requests for affected Pipe Endpoints is reduced proportionally to the observed error percentage. Under CPU pressure, reduced limits only apply to the Pipe Endpoints consuming a significant share of the cluster's CPU. Workspace administrators on dedicated infrastructure receive an email with affected Endpoints and applied concurrency limits.

Once Pipes stop receiving `408` or `500` errors, concurrency limits are gradually relaxed until removed.

While the rate limiter is active, Endpoints can return HTTP `429`. Clients should retry with exponential backoff.

## Notifications by email (https://www.tinybird.co/docs/forward/pricing/limits#email-notifications)

When reaching or temporarily exceeding your plan's limits, you receive one of the following emails.

### Free plans (https://www.tinybird.co/docs/forward/pricing/limits#free-plans)

The following table shows the limits' notifications you might receive for the Free plan.

| Limit | Warning email | Alert email |
| --- | --- | --- |
| vCPU usage (1/4 vCPU) |  | Exceeded plan's base vCPU allowance within the past hour. |
| Storage Usage | Used more than 75% of allocated storage. | Used more than 100% of allocated storage. |
| Requests per day (1,000 queries) | Used more than 75% of daily query limit. | Used more than 100% of daily query limit. |
| LLM usage ($25 LLM credits per cycle) |  | Used 100% of LLM credits in current billing cycle. |

### Developer and SaaS plans (https://www.tinybird.co/docs/forward/pricing/limits#developer-and-saas-plans)

The following table shows the limits' notifications you might receive for plans on shared infrastructure.

| Limit | Warning email | Alert email |
| --- | --- | --- |
| vCPU usage | Used more vCPU time than your plan includes, resulting in overages. This notification is sent once per billing cycle. | Exceeded your plan's vCPU time limit within the last hour. |
| Extra costs | Overage costs in vCPU time exceed 33% of your plan's base fee. | Daily email with a breakdown of extra costs when vCPU overages occur. |
| Storage Usage | Used more than 75% of allocated storage. | Used more than 100% of allocated storage. |
| LLM usage ($100 LLM credits per cycle) |  | Used 100% of LLM credits in current billing cycle. |

## Workspace limits (https://www.tinybird.co/docs/forward/pricing/limits#workspace-limits)

The following limits apply to all workspaces in an organization, regardless of the plan.

| Description | Limit |
| --- | --- |
| Number of Workspaces | 90 Workspaces. Soft limit, contact support to increase it. |
| Number of members | 90 members. Soft limit, contact support to increase it. |
| Number of branches, including `main` | Default 4. Hard limit on shared infrastructure. Soft limit on dedicated infrastructure, contact support to increase it. |
| Number of Data Sources | 100 Data Sources. Soft limit, contact support to increase it. |
| Number of Copy Pipes | Up to 200 Copy Pipes, depending on the plan. See [Copy Pipe limits](https://www.tinybird.co/docs/forward/pricing/limits#copy-pipe-limits) |
| Number of Sink Pipes | Up to 10 Pipes, depending on the plan. See [Sink Pipe limits](https://www.tinybird.co/docs/forward/pricing/limits#sink-pipe-limits). |
| Number of Tokens | 100,000 tokens. If you need more you should take a look at [JWT tokens](/docs/forward/core-concepts/jwt) |
| Number of secrets | 100 secrets. |
See [Rate limits for JWTs](/docs/forward/core-concepts/jwt#rate-limits-for-jwts) for more detail specifically on JWT limits.

## Ingestion limits (https://www.tinybird.co/docs/forward/pricing/limits#ingestion-limits)

The following ingestion limits apply to all workspaces in an organization.

| Description | Limit |
| --- | --- |
| Data Source max columns | 500 |
| Full body upload | 8 MB |
| Multipart upload - CSV and NDJSON | 500 MB |
| Multipart upload - Parquet | 50 MB |
| Max file size - Parquet - Free plan | 1 GB |
| Max file size - Parquet - Developer, SaaS, and Enterprise plans | 5 GB |
| Max file size (uncompressed) - Free plan | 10 GB |
| Max uncompressed file size - Developer, SaaS, and Enterprise plans | 32 GB |
| Kafka topics | 5 topics. Enterprise plan users can contact support to increase. |
| Max parts created at once - NDJSON/Parquet jobs and Events API | 12 parts |

### Ingestion limits (API) (https://www.tinybird.co/docs/forward/pricing/limits#ingestion-limits-api)

Tinybird throttles requests based on the capacity. So if your queries are using 100% resources you might not be able to run more queries until the running ones finish.

| Description | Limit and time window |
| --- | --- |
| Request size - Events API | 10 MB |
| Response size | 100 MB |
| Create Data Source from schema | 25 times per minute |
| Create Data Source from file or URL * | 5 times per minute |
| Append data to Data Source <a href="/docs/api-reference/datasource-api#post--v0-datasources-?">`POST /v0/datasources`</a> * | 5 times per minute |
| Append data to Data Source using <a href="/docs/api-reference/events-api#post-v0events">`POST /v0/events`</a> | 100 times per second |
| Create Data Source using <a href="/docs/api-reference/events-api#post-v0events">`POST /v0/events`</a> | 5 times per minute |
| Replace data in a Data Source * | 5 times per minute |
* The quota is shared at Workspaces level when creating, appending data, or replacing data. For example, you can't do 5 requests of each type per minute, for a total of 15 requests. You can do at most a grand total of 5 requests of those types combined. Import jobs using append mode, [S3 connector](/docs/forward/ingest-data/connectors/s3) ingests, [Datasource API](/docs/api-reference/datasource-api) posts, and [CLI datasource commands](/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-append) all use the `/v0/datasources` endpoint and are subject to the limit.

The number of rows in append requests doesn't impact the ingestion limit; each request counts as a single ingestion.

If you exceed your rate limit, your request is throttled and you receive *HTTP 429 Too Many Requests* response codes from the API. Each response contains a set of HTTP headers with your current rate limit status.

| Header Name | Description |
| --- | --- |
| `X-RateLimit-Limit` | The maximum number of requests you're permitted to make in the current limit window. |
| `X-RateLimit-Remaining` | The number of requests remaining in the current rate limit window. |
| `X-RateLimit-Reset` | The time in seconds after the current rate limit window resets. |
| `Retry-After` | The time to wait before making a another request. Only present on 429 responses. |

## Query limits (https://www.tinybird.co/docs/forward/pricing/limits#query-limits)

The following query limits apply to all workspaces in an organization, regardless of the plan.

| Description | Limit |
| --- | --- |
| SQL length | 128KB |
| Result length | 100 MB |
| Query execution time | 10 seconds |
If you exceed your rate limit, your request is throttled and you receive *HTTP 429 Too Many Requests* response codes from the API. Each response contains a set of HTTP headers with your current rate limit status.

| Header Name | Description |
| --- | --- |
| `X-RateLimit-Limit` | The maximum number of requests you're permitted to make in the current limit window. |
| `X-RateLimit-Remaining` | The number of requests remaining in the current rate limit window. |
| `X-RateLimit-Reset` | The time in seconds after the current rate limit window resets. |
| `Retry-After` | The time to wait before making a another request. Only present on 429 responses. |

### Query timeouts (https://www.tinybird.co/docs/forward/pricing/limits#query-timeouts)

If query execution time exceeds the default limit of 10 seconds, an error message appears. Long execution times hint at issues that need to be fixed in the query or the Data Source schema.

To avoid query timeouts, you can:

- Optimize your queries to remove inefficiencies and common mistakes.
- Increase the number of available threads in your plan to process queries concurrently. See [Max threads](./developer#max-threads) on the Developer plan or the equivalent section on [SaaS](./saas#max-threads) or [Free](./free#max-threads).
If you still need to increase the timeout limit, contact support. See [Get help](/docs/forward/support#get-help).

Only paid accounts can raise the timeout limit.

## Infrastructure limits (https://www.tinybird.co/docs/forward/pricing/limits#infrastructure-limits)

The following infrastructure limits apply to all plans and workspaces.

| Description | Limit |
| --- | --- |
| Concurrent connections per IP address | 250 active connections from the same IP address per region |
If you exceed the concurrent connection limit, requests are throttled and you receive *HTTP 429 Too Many Requests* response codes. This limit is enforced to ensure service availability for all users.

This limit applies per region. If you need higher connection limits, consider distributing your traffic across multiple IP addresses.

## Publishing limits (https://www.tinybird.co/docs/forward/pricing/limits#publishing-limits)

The following publishing limits apply to all workspaces in an organization, regardless of the plan.

### Materialized Views limits (https://www.tinybird.co/docs/forward/pricing/limits#materialized-views-limits)

No numerical limits. Certain operations are [inadvisable when using Materialized Views](/docs/forward/core-concepts/materialized-views#limitations).

### Copy pipe limits (https://www.tinybird.co/docs/forward/pricing/limits#copy-pipe-limits)

Copy Pipes have the following limits per Workspace, depending on your billing plan:

| Plan | Active jobs (running or queued) | Execution time | Minimum scheduled frequency | Copy Pipes |
| --- | --- | --- | --- | --- |
| Free | 1 | 20 s | Once an hour | 20 |
| Developer | 3 | 30 s | Up to every 10 minutes | 100 |
| Enterprise | 6 | 50% of the scheduling period, 30 minutes max | Up to every minute | 200 |
The 6 active Copy Pipe jobs includes:

- 2 concurrently running jobs, and
- 4 additional jobs in the queue (pending execution).
This limit ensures fair resource usage and system stability across all users. If your workflows require higher throughput, contact [Tinybird support](/docs/forward/support) so the team can evaluate your use case and scale as needed.

### Sink pipe limits (https://www.tinybird.co/docs/forward/pricing/limits#sink-pipe-limits)

Sink Pipes have the following limits, depending on your billing plan:

| Plan | Sink Pipes per Workspace | Execution time | Frequency | Memory usage per query | Active jobs (running or queued) |
| --- | --- | --- | --- | --- | --- |
| Developer | 3 | 30 s | Up to every 10 min | 10 GB | 3 |
| Enterprise | 10 | 300 s | Up to every minute | 10 GB | 6 |

## Delete limits (https://www.tinybird.co/docs/forward/pricing/limits#delete-limits)

Delete jobs have the following limits, depending on your billing plan:

| Plan | Active delete jobs per Workspace |
| --- | --- |
| Free | 1 |
| Developer | 3 |
| Enterprise | 6 |

## Next steps (https://www.tinybird.co/docs/forward/pricing/limits#next-steps)

- [Pricing plans](./): Overview of Free, Developer, SaaS, and Enterprise.

---

URL: https://www.tinybird.co/docs/forward/pricing/free
Last update: 2026-07-31T11:10:11.000Z
Content:
---
title: "Free plan \| Tinybird"
theme-color: "#171612"
description: "The Tinybird Free plan — what's included, limits, support, and how to upgrade."
inkeep:version: "forward"
---

# Free plan (https://www.tinybird.co/docs/forward/pricing/free#free-plan)

The Free plan provides a production-grade instance of Tinybird with no time limit: ingest connectors, real-time querying, and managed endpoints are all available. It runs on **shared infrastructure**.

## What's included (https://www.tinybird.co/docs/forward/pricing/free#whats-included)

| Resource | Included |
| --- | --- |
| vCPU | 0.25 vCPU (with burst up to 3x) |
| Requests | 1,000 requests per day per organization |
| Storage | 10 GB |
| Max threads | 1 |
| LLM credits | $25 per billing cycle |

## Resource limits (https://www.tinybird.co/docs/forward/pricing/free#resource-limits)

- **vCPU**: 0.25 vCPU seconds per second, with up to 3x burst. This is a hard limit. Requests that exceed the burst capacity return HTTP 429.
- **Max threads**: Queries through API Endpoints and the Query API use at most 1 thread. See [Max threads](https://www.tinybird.co/docs/forward/pricing/free#max-threads).
- **LLM usage**: Soft stop when you exhaust the $25 included credits. LLM requests stop until the next billing cycle. See [LLM usage](https://www.tinybird.co/docs/forward/pricing/free#llm-usage).
For Workspace, ingestion, query, and publishing limits, see [Limits](./limits).

### Max threads (https://www.tinybird.co/docs/forward/pricing/free#max-threads)

Max threads is the maximum number of concurrent threads that execute queries in API Endpoints and Query API calls. The Free plan limits usage to 1 thread.

### LLM usage (https://www.tinybird.co/docs/forward/pricing/free#llm-usage)

Large Language Model usage, or LLM usage, in AI tools within MCP and Explorations UI includes $25 of credits per billing cycle. When you exhaust the included credits, LLM requests stop until the next billing cycle.

You can monitor usage through the <a href="../monitoring/service-datasources#tinybird-llm-usage">`tinybird.llm_usage`</a> Service Data Source.

## Support (https://www.tinybird.co/docs/forward/pricing/free#support)

Support is available through the [Community Slack](/docs/community), monitored by the Tinybird team on a best-effort basis.

## Upgrade (https://www.tinybird.co/docs/forward/pricing/free#upgrade)

To upgrade to a Developer plan:

1. Go to **Organization settings** → **Billing**.
2. Select **Upgrade**.
3. Select your vCPU size and fill out the details.
4. Confirm the upgrade.
To upgrade to a SaaS or Enterprise plan, [contact sales](https://www.tinybird.co/contact-us).

## Next steps (https://www.tinybird.co/docs/forward/pricing/free#next-steps)

- [Developer plan](./developer) — The next step up from Free.
- [Limits](./limits): Platform and operational limits.

---

URL: https://www.tinybird.co/docs/forward/pricing/enterprise
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Enterprise plan \| Tinybird"
theme-color: "#171612"
description: "The Tinybird Enterprise plan. Dedicated ClickHouse® cluster, replica SKUs, credit-based billing, and Standard or Premier support."
inkeep:version: "forward"
---

# Enterprise plan (https://www.tinybird.co/docs/forward/pricing/enterprise#enterprise-plan)

The Enterprise plan provides **dedicated infrastructure**: your own isolated ClickHouse® cluster where you are the only customer. Your queries and ingestion run on your own replicas with no resource sharing from other customers. This gives you compute and network isolation and fine-tuned control over the infrastructure required for your workloads.

Minimum storage for Enterprise plans is 1 TB. A support package is mandatory. Billing is credit-based with a minimum 12-month commitment.

## Managing your cluster (https://www.tinybird.co/docs/forward/pricing/enterprise#managing-your-cluster)

- **Add or remove replicas**: Use **Organization settings** → **Plan & Billing** to add replicas, remove them, or change their size.
- **Traffic distribution**: Set read and write weights per replica to control how query and ingestion traffic is distributed. See [Cluster management](/docs/forward/guides/cluster-management#how-weight-distribution-works).
- **Monitoring**: Monitor cluster usage, remaining credits, and current commitment through your organization's dashboard. See [Enterprise infrastructure monitoring](/docs/forward/core-concepts/organizations#dedicated-infrastructure-monitoring).
For other adjustments to your Enterprise plan or to discuss moving to SaaS (shared infrastructure), [contact sales](https://www.tinybird.co/contact-us).

## Billing (https://www.tinybird.co/docs/forward/pricing/enterprise#billing)

Enterprise plans are billed using credits based on the following SKUs:

| SKU | What it covers | Billing type |
| --- | --- | --- |
| Dedicated ClickHouse® cluster CPUs | The total active CPUs in your dedicated ClickHouse® cluster. Billed by CPU cores at any given time. | Usage based. Rates may vary by region. |
| On-demand CPUs | Temporary compute for jobs that run outside the baseline cluster, for example populate jobs. Billed by CPU cores and duration. | Usage based. Rates may vary by region. |
| Real-time ingestion | Data ingested via HFI API, Kafka, and DynamoDB connectors. Measured as compressed bytes expressed as CPU core-hours (P95). | Usage based. Rates may vary by region and contract. |
| Workers | Batch and async operations: batch ingestion, populates, deletes, replaces, copies. Billed by peak concurrent workers per day. | Usage based. |
| Data storage | Persistent data in ClickHouse® per Data Source, including primary and quarantine data. Based on peak daily storage. | Usage based. |
| Backup storage | ClickHouse® backups retained for your organization. Based on total backup size retained per day. | Usage based. |
| Data transfer | Data transferred out of the platform via Sinks. Rate varies by transfer type (intra vs. inter cloud provider/region) and region. | Usage based. |
| Private networking | Private network connectivity for your Workspaces, such as private links. Billed as a markup on the underlying connectivity cost. | Usage based. |
| Support | Standard or Premier support as agreed in your contract. Standard is included. Not usage-based. | Fixed. |
| Enterprise authentication | SSO with SAML. Flat daily rate when turned on; no charge when off. | Fixed daily rate. |
| Private region | Private region deployment. Flat daily rate when turned on; no charge when off. | Fixed daily rate. |
| LLM usage | LLM usage in AI tools within MCP and Explorations UI. Includes $200 per billing cycle. | Usage based when usage exceeds the included credits. |
The minimum commitment period is 12 months. Bills are issued monthly based on credits consumed (first to last day of the month). Invoices are issued upon credits purchase, which can happen when signing the contract or when purchasing additional credits.

Where applicable, rates depend on your [region](/docs/api-reference#regions-and-endpoints) and contract.

### Dedicated ClickHouse® cluster CPUs (https://www.tinybird.co/docs/forward/pricing/enterprise#dedicated-clickhouse-cluster-cpus)

The Dedicated ClickHouse® cluster CPUs SKU covers the cost of running the dedicated ClickHouse® cluster that Tinybird provisions exclusively for your organization. This cluster stores your data, runs your queries, and provides isolated compute and storage according to your topology and capacity. It includes ClickHouse® nodes, coordination services when applicable, and the underlying compute resources. Pricing depends on the cluster's [region](/docs/api-reference#regions-and-endpoints).

You can upgrade or downgrade your cluster at any time, immediately affecting billing for this SKU. See [Cluster management](/docs/forward/guides/cluster-management).

Billing unit: CPU cores/hour.

### On-demand CPUs (https://www.tinybird.co/docs/forward/pricing/enterprise#on-demand-cpus)

The On-demand CPUs SKU covers temporary compute capacity for jobs that need dedicated compute instances. This currently applies to populate jobs that run on on-demand compute. These jobs run outside the baseline resources of your dedicated ClickHouse® cluster and use short-lived, single-host instances during the job. Billing is based on the CPU cores used and the duration of the on-demand instance. Rates may vary by [region](/docs/api-reference#regions-and-endpoints). See also [On-demand CPUs](./#on-demand-cpus) on the plans hub.

Billing unit: OnDemand CPU cores/hour.

### Real-time ingestion (https://www.tinybird.co/docs/forward/pricing/enterprise#real-time-ingestion)

The Real-time ingestion SKU covers real-time data ingestion into the platform. This includes data ingested using the High-Frequency Ingestion API, or HFI API, Kafka connectors, and DynamoDB connectors. This SKU doesn't include other ingestion mechanisms. Billing is based on the compute capacity required to sustain your ingestion throughput. Throughput is measured as decompressed bytes sent to the platform and expressed as CPU core-hours using P95 over time windows. Rates may vary by [region](/docs/api-reference#regions-and-endpoints) and contract.

Only data that arrives at Tinybird is considered for this SKU. After the data is written, the Dedicated ClickHouse® cluster CPUs SKU covers downstream operations such as Materialized Views and Copy Pipes.

Billing unit: MB/s/CPU/hour using P95 over 15 min windows.

### Workers (https://www.tinybird.co/docs/forward/pricing/enterprise#workers)

The Workers SKU covers batch and asynchronous operations that the platform runs on your behalf. These operations include batch ingestion from sources such as S3, CSV, or Parquet, populate jobs, deletes, replaces, copies, and other job processor operations, but exclude real-time ingestion. Billing is based on the maximum number of jobs you can run in parallel, called concurrent workers, during the billing period. For each day, the observed peak concurrency determines the number of workers charged at a fixed daily rate per worker.

Billing unit: number of concurrent workers.

### Data storage (https://www.tinybird.co/docs/forward/pricing/enterprise#data-storage)

The Data Storage SKU covers persistent data in ClickHouse® for your Workspaces. It accounts for the total amount of data on disk per Data Source, including both primary data and quarantine data. Storage is measured at the Data Source level according to peak storage usage during each billing period. Billing uses a daily storage rate according to the observed maximum storage.

Billing unit: TBs/day.

### Backup storage (https://www.tinybird.co/docs/forward/pricing/enterprise#backup-storage)

The Backup Storage SKU covers ClickHouse® backups for organizations on dedicated infrastructure. It accounts for the total size of backups retained for your organization during the billing period, including full and incremental backups, aggregated across all your dedicated clusters. Backup usage is measured at the organization level. Billing uses a daily storage rate according to the total backup size retained. For backup retention and related terms, see the [Terms and Conditions](https://www.tinybird.co/terms-and-conditions).

Billing unit: TBs/day.

### Data transfer (https://www.tinybird.co/docs/forward/pricing/enterprise#data-transfer)

The Data Transfer SKU covers data transferred out of the platform, primarily through [Sinks](/docs/forward/core-concepts/sinks) that export data to external storage. It accounts for data moved across cloud providers and regions. Intra transfers stay within the same provider and region, while inter transfers cross providers or regions. Usage is measured at the Workspace level and aggregated daily. The per-gigabyte billing rate varies by transfer type and [region](/docs/api-reference#regions-and-endpoints).

Billing unit: GBs transferred.

### Private networking (https://www.tinybird.co/docs/forward/pricing/enterprise#private-networking)

The Private Networking SKU covers activating and operating private network connectivity for your Workspaces, for example private links between services, regions, or infrastructure components that aren't exposed over the public internet. Billing is applied as a markup on the underlying connectivity cost.

Billing unit: GBs transferred.

### Support (https://www.tinybird.co/docs/forward/pricing/enterprise#support)

The Support SKU covers ongoing support services for your organization. Standard is included. Premier is available and adds a designated Customer Success engineer, prioritized SLA, on-call requests, monthly partnership meetings, and onboarding support. The applicable tier and price depend on your agreement and aren't usage-based. See [Support](../support) for full details.

### Enterprise authentication (https://www.tinybird.co/docs/forward/pricing/enterprise#enterprise-authentication)

The Enterprise authentication SKU covers activating Single Sign-On, or SSO, with SAML for your organization. This is a flat-rate SKU. When SAML is on, a fixed rate is charged regardless of usage. If SAML is off, no charges apply. Billing is a fixed daily rate for each day the feature is active.

### Private region (https://www.tinybird.co/docs/forward/pricing/enterprise#private-region)

The Private Region SKU covers activating a private region deployment for your organization. When Private Region is on, a flat rate is charged regardless of workload. If the feature is off, no charges apply. Billing is a fixed daily rate for each day the feature is active.

### LLM usage (https://www.tinybird.co/docs/forward/pricing/enterprise#llm-usage)

LLM usage in AI tools within MCP and Explorations UI includes $200 of credits per billing cycle. Usage beyond included credits is billed at cost.

## Support (https://www.tinybird.co/docs/forward/pricing/enterprise#support)

A support package is mandatory on Enterprise plans. Standard is included and provides priority email, live chat, and a dedicated customer Slack channel. Premier is available and adds a designated Customer Success engineer, prioritized SLA, on-call requests, monthly partnership meetings, and onboarding support. Your customer agreement defines your support plan, pricing, and terms. See [Support](../support) for full details.

## Next steps (https://www.tinybird.co/docs/forward/pricing/enterprise#next-steps)

- [Cluster management](/docs/forward/guides/cluster-management): Add replicas, select SKUs, set traffic weights.
- [SaaS plan](./saas) — Shared infrastructure alternative for 4+ vCPU workloads.
- [Limits](./limits): Platform and operational limits.
- [Regions and endpoints](/docs/api-reference#regions-and-endpoints): Available regions and regional pricing differences.

---

URL: https://www.tinybird.co/docs/forward/pricing/developer
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Developer plan \| Tinybird"
theme-color: "#171612"
description: "The Tinybird Developer plan — plan sizes, billing, resize, downgrade, and upgrade."
inkeep:version: "forward"
---

# Developer plan (https://www.tinybird.co/docs/forward/pricing/developer#developer-plan)

The Developer plan runs on **shared infrastructure** without a commitment. You select a plan size when signing up or changing plans. Each size has a fixed set of included resources. Developer plans include 25 GB of storage. Usage beyond this and other included amounts is billed as overage. Plans support up to 8 vCPUs. For more capacity on shared infrastructure, see [SaaS](./saas), or for a dedicated cluster, see [Enterprise](./enterprise).

Each organization is tied to a single region. You can create multiple workspaces within one organization, all covered by the same billing plan. If you need workspaces in a different region, you must create a separate organization, which requires its own plan.

## Billing (https://www.tinybird.co/docs/forward/pricing/developer#billing)

Developer plans are billed based on the following resources:

| Resource | Billing | Type |
| --- | --- | --- |
| vCPUs | Number of vCPUs available per second. More than 8 vCPUs require a SaaS or Enterprise plan. | Fixed minimum. Usage based when usage exceeds the vCPU included in the plan. |
| On-demand CPU | Temporary compute for jobs that run outside the baseline cluster, for example populate jobs. Billed by CPU cores and duration. | Usage based. Rates may vary by region. See [On-demand CPUs](./#on-demand-cpus). |
| Storage | Average of daily maximum usage of the compressed disk storage of all your data, in gigabytes. | Usage based when usage exceeds the 25 GB included in the plan. |
| Data transfer | When using [Sinks](/docs/api-reference/sink-pipes-api), usage is billed depending on the destination, which can be the same cloud provider and region as your Tinybird cluster, or a different one. | Usage based. |
| LLM usage | LLM usage in AI tools within MCP and Explorations UI. Includes $100 per billing cycle. | Usage based when usage exceeds the $100 included in the plan. |
Developer plans are billed monthly. Each invoice contains fixed costs plus usage-based costs calculated from the previous period. The billing period starts on the subscription date.

The same vCPU and storage overage rates apply in every region. See [Pricing](https://www.tinybird.co/pricing) for current rates.

### vCPU seconds and overages (https://www.tinybird.co/docs/forward/pricing/developer#vcpu-seconds)

Each plan includes a baseline **vCPU** allocation. For example, if your plan includes 1 vCPU, you can consume 1 vCPU second per second without overage.

Usage exceeding your plan allocation counts as **vCPU overage** and is billed at a fixed rate per second. This calculation includes both read and write operations.

### vCPU burst mode (https://www.tinybird.co/docs/forward/pricing/developer#vcpu-burst-mode)

You can use burst mode to temporarily exceed your base vCPU allocation:

- Developer plans S-1/4, S-1/2, S-1, and S-2 can burst up to 3× their vCPU capacity.
- Plans S-4 and higher can burst up to 2×.
Usage beyond the base allowance incurs vCPU overage charges. Requests that exceed the burst capacity return HTTP 429 from API Endpoints or the Query API.

For batch operations such as populates or copies, the operation can run until it hits a platform limit such as maximum available memory.

### Max threads (https://www.tinybird.co/docs/forward/pricing/developer#max-threads)

Max threads is the maximum number of concurrent threads used to execute queries in API Endpoints and Query API calls. The number of max threads available depends on your Developer plan size.

While more threads can improve concurrent query processing, final performance also depends on your vCPU limit and query complexity.

### Data storage (https://www.tinybird.co/docs/forward/pricing/developer#data-storage)

You are charged for persistent data stored in ClickHouse® per Data Source. Developer plans include 25 GB. Storage beyond the included amount is billed at a fixed rate per GB. Rates aren't tied to region.

### Data transfer (https://www.tinybird.co/docs/forward/pricing/developer#data-transfer)

The data transfer SKU covers data sent out of the platform, primarily through [Sinks](/docs/forward/core-concepts/sinks). Data transfer is charged per gigabyte. The rate depends on the transfer type. Intra transfers stay within the same cloud provider and region. Inter transfers cross providers or regions.

### LLM usage (https://www.tinybird.co/docs/forward/pricing/developer#llm-usage)

LLM usage in AI tools within MCP and Explorations UI includes $100 of credits per billing cycle. Usage beyond included credits is billed at cost.

You can monitor usage through the <a href="../monitoring/service-datasources#tinybird-llm-usage">`tinybird.llm_usage`</a> Service Data Source.

### Self-service (https://www.tinybird.co/docs/forward/pricing/developer#self-service)

After adding billing details, you can select and resize any Developer plan up to 8 vCPUs in self-service mode. To use more than 8 vCPUs, upgrade through sales to SaaS for shared infrastructure or Enterprise for dedicated infrastructure.

## Support (https://www.tinybird.co/docs/forward/pricing/developer#support)

Community Slack is available to everyone. Developer also includes email support. See [Support](../support) for details.

## Resize and upgrade (https://www.tinybird.co/docs/forward/pricing/developer#resize-and-upgrade)

You can resize Developer plans at any time. Increases and reductions in capacity take effect instantly.

To resize your plan:

1. Go to **Organization settings** → **Billing**.
2. Select **Change size**.
3. Select a bigger or a smaller size. Self-service plans can be up to 8 vCPUs.
4. Confirm the resize.
To upgrade to a SaaS or Enterprise plan, [contact sales](https://www.tinybird.co/contact-us).

## Downgrade (https://www.tinybird.co/docs/forward/pricing/developer#downgrade)

You can downgrade back to the Free plan by canceling your Developer plan.

To cancel your Developer plan:

1. Go to **Organization settings** → **Billing**.
2. Select the **More actions (⋯)** button and then select **Cancel plan**.
3. Review the information and confirm. Remaining costs are charged automatically upon cancellation.
After canceling your Developer plan, Free plan limits apply to your Workspaces.

## Next steps (https://www.tinybird.co/docs/forward/pricing/developer#next-steps)

- [Limits](./limits): Platform and operational limits.
- [SaaS plan](./saas): Shared infrastructure with 4+ vCPUs and a commitment.
- [Enterprise plan](./enterprise) — Dedicated ClickHouse® cluster.

---

URL: https://www.tinybird.co/docs/forward/monitoring/service-datasources
Last update: 2026-08-04T10:22:54.000Z
Content:
---
title: "Service data sources \| Tinybird"
theme-color: "#171612"
description: "In addition to the data sources you upload, Tinybird provides other \"Service data sources\" that allow you to inspect what's going on in your account."
inkeep:version: "forward"
---

# Service data sources (https://www.tinybird.co/docs/forward/monitoring/service-datasources#service-data-sources)

Tinybird provides Service data sources that you can use to inspect what's going on in your Tinybird account, diagnose issues, monitor usage, and so on.

For example, you can get real time stats about API calls or a log of every operation over your data sources. This is similar to using system tables in a database, although Service data sources contain information about the usage of the service itself.

Queries made to Service data sources are free of charge and don't count towards your usage. However, calls to API endpoints that use Service data sources do count towards API rate limits. See [Billing](../pricing).

## Considerations (https://www.tinybird.co/docs/forward/monitoring/service-datasources#considerations)

- You can't use Service data sources in materialized view queries.
- Pass dynamic query parameters to API endpoints to then query Service data sources.

## Service data sources (https://www.tinybird.co/docs/forward/monitoring/service-datasources#service-data-sources-2)

The following Service data sources are available.

### tinybird.pipe_stats_rt (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-pipe-stats-rt)

Contains information about all requests made to your [API endpoints](/docs/forward/core-concepts/api-endpoints) in real time. This data source has a TTL of 7 days. If you need to query data older than 7 days you must use the aggregated by day data available at [tinybird.pipe_stats](https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-pipe-stats).

Calls made against Service data sources are not logged and don't count towards usage limits.

| Field | Type | Description |
| --- | --- | --- |
| `start_datetime` | `DateTime` | API call start date and time. |
| `pipe_id` | `String` | Pipe ID as returned in the [Pipes API](/docs/api-reference/pipe-api). The value is `query_api` for a Query API request. |
| `pipe_name` | `String` | Pipe name as returned in the [Pipes API](/docs/api-reference/pipe-api). The value is `query_api` for a Query API request. |
| `duration` | `Float` | API call duration, in seconds. |
| `read_bytes` | `UInt64` | API call read data, in bytes. |
| `read_rows` | `UInt64` | API call rows read. |
| `result_rows` | `UInt64` | Rows returned by the API call. |
| `url` | `String` | URL (`token` param is removed for security reasons). |
| `error` | `UInt8` | `1` if query returned error, else `0`. |
| `request_id` | `String` | API call identifier returned in `x-request-id` header. Format is ULID string. |
| `token` | `String` | API call token identifier used. |
| `token_name` | `String` | API call token name used. |
| `status_code` | `Int32` | API call returned status code. |
| `method` | `String` | API call method POST or GET. |
| `parameters` | `Map(String, String)` | API call parameters used. |
| `release` | `String` | Semantic version of the release (deprecated). |
| `user_agent` | `Nullable(String)` | User Agent HTTP header from the request. |
| `resource_tags` | `Array(String)` | Tags associated with the pipe when the request was made. |
| `cpu_time` | `Float` | CPU time used by the query, in seconds. |
| `memory_usage` | `UInt64` | Memory consumption by the query, in bytes. High memory usage indicates the query may not be optimized efficiently. |

### tinybird.pipe_stats (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-pipe-stats)

Aggregates the request stats in [tinybird.pipe_stats_rt](https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-pipe-stats-rt) by day.

Calls made against Service data sources are not logged and don't count towards usage limits.

| Field | Type | Description |
| --- | --- | --- |
| `date` | `Date` | Request date and time. |
| `pipe_id` | `String` | Pipe ID as returned in the [Pipes API](/docs/api-reference/pipe-api). |
| `pipe_name` | `String` | Name of the pipe. |
| `view_count` | `UInt64` | Request count. |
| `error_count` | `UInt64` | Number of requests with error. |
| `avg_duration_state` | `AggregateFunction(avg, Float32)` | Average duration state, in seconds (see [Querying _state columns](https://www.tinybird.co/docs/forward/monitoring/service-datasources#querying-state-columns)). |
| `quantile_timing_state` | `AggregateFunction(quantilesTiming(0.9, 0.95, 0.99), Float64)` | 0.9, 0.95 and 0.99 quantiles state. Time, in milliseconds (see [Querying _state columns](https://www.tinybird.co/docs/forward/monitoring/service-datasources#querying-state-columns)). |
| `read_bytes_sum` | `UInt64` | Total bytes read. |
| `read_rows_sum` | `UInt64` | Total rows read. |
| `resource_tags` | `Array(String)` | All the tags associated with the resource when the aggregated requests were made. |

### tinybird.block_log (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-block-log)

The data source contains details about how Tinybird ingests data into your data sources. You can use this Service data source to spot problematic parts of your data.

| Field | Type | Description |
| --- | --- | --- |
| `timestamp` | `DateTime` | Date and time of the block ingestion. |
| `import_id` | `String` | Id of the import operation. |
| `job_id` | `Nullable(String)` | Id of the job that ingested the block of data, if it was ingested by URL. In this case, `import_id` and `job_id` must have the same value. |
| `request_id` | `String` | Id of the request that performed the operation. In this case, `import_id` and `job_id` must have the same value. Format is ULID string. |
| `source` | `String` | Either the URL or `stream` or `body` keywords. |
| `block_id` | `String` | Block identifier. You can cross this with the `blocks_ids` column from the [tinybird.datasources_ops_log](https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-datasources-ops-log) Service data source. |
| `status` | `String` | `done` \| `error`. |
| `datasource_id` | `String` | data source consistent id. |
| `datasource_name` | `String` | data source name when the block was ingested. |
| `start_offset` | `Nullable(Int64)` | The starting byte of the block, if the ingestion was split, where this block started. |
| `end_offset` | `Nullable(Int64)` | If split, the ending byte of the block. |
| `rows` | `Nullable(Int32)` | How many rows it ingested. |
| `parser` | `Nullable(String)` | Whether the native block parser or falling back to row by row parsing is used. |
| `quarantine_lines` | `Nullable(UInt32)` | If any, how many rows went into the quarantine data source. |
| `empty_lines` | `Nullable(UInt32)` | If any, how many empty lines were skipped. |
| `bytes` | `Nullable(UInt32)` | How many bytes the block had. |
| `processing_time` | `Nullable(Float32)` | How long it took, in seconds. |
| `processing_error` | `Nullable(String)` | Detailed message in case of error. |
When Tinybird ingests data from a URL, it splits the download in several requests, resulting in different ingestion blocks. The same happens when the data upload happens with a multipart request.

### tinybird.datasources_ops_log (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-datasources-ops-log)

Contains all operations performed to your data sources. Tinybird tracks the following operations:

| Event | Description |
| --- | --- |
| `create` | A data source is created. |
| `append` | Append operation. |
| `append-hfi` | Append operation using the [High-frequency Ingestion API](/docs/forward/ingest-data/events-api). |
| `append-kafka` | Append operation using the [Kafka Connector](/docs/forward/ingest-data/connectors/kafka). |
| `replace` | A replace operation took place in the data source. |
| `delete` | A data source is deleted. |
| `delete_data` | A delete operation took place in the data source. |
| `truncate` | A truncate operation took place in the data source. |
| `rename` | The data source was renamed. |
| `populateview-queued` | A populate operation was queued for execution. |
| `populateview` | A finished populate operation (up to 8 hours after it started). |
| `copy` | A copy operation took place in the data source. |
| `alter` | An alter operation took place in the data source. |
Materializations are logged with same `event_type` and `operation_id` as the operation that triggers them. You can track the materialization pipe with `pipe_id` and `pipe_name`.

Tinybird logs all operations with the following information in this data source:

| Field | Type | Description |
| --- | --- | --- |
| `timestamp` | `DateTime` | Date and time when the operation started. |
| `event_type` | `String` | Operation being logged. |
| `operation_id` | `String` | Groups rows affected by the same operation. Useful for checking materializations triggered by an append operation. |
| `datasource_id` | `String` | Id of your data source. The data source id is consistent after renaming operations. You should use the id when you want to track name changes. |
| `datasource_name` | `String` | Name of your data source when the operation happened. |
| `result` | `String` | `ok` \| `error` |
| `elapsed_time` | `Float32` | How much time the operation took, in seconds. |
| `error` | `Nullable(String)` | Detailed error message if the result was error. |
| `import_id` | `Nullable(String)` | Id of the import operation, if data has been ingested using one of the following operations: `create`, `append` or `replace` |
| `job_id` | `Nullable(String)` | Id of the job that performed the operation, if any. If data has been ingested, `import_id` and `job_id` must have the same value. |
| `request_id` | `String` | Id of the request that performed the operation. If data has been ingested, `import_id` and `request_id` must have the same value. Format is ULID string. |
| `rows` | `Nullable(UInt64)` | How many rows the operations affected. This depends on `event_type`: for the `append` event, how many rows got inserted; for `delete` or `truncate` events, how many rows the data source had; for `replace`, how many rows the data source has after the operation. |
| `rows_quarantine` | `Nullable(UInt64)` | How many rows went into the quarantine data source, if any. |
| `blocks_ids` | `Array(String)` | List of blocks ids used for the operation. See the [tinybird.block_log](https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-block-log) Service data source for more details. |
| `options` | `Nested(Names String, Values String)` | Tinybird stores key-value pairs with extra information for some operations. For the `replace` event, Tinybird uses the `rows_before_replace` key to track how many rows the data source had before the replacement happened, the `replace_condition` key shows what condition was used. For `append` and `replace` events, Tinybird stores the data `source`, for example the URL, or body/stream keywords. For `rename` event, `old_name` and `new_name`. For `populateview` you can find there the whole populate `job` metadata as a JSON string. For `alter` events, Tinybird stores `operations`, and dependent pipes as `dependencies` if they exist. |
| `read_bytes` | `UInt64` | Read bytes in the operation. |
| `read_rows` | `UInt64` | Read rows in the operation. |
| `written_rows` | `UInt64` | Written rows in the operation. |
| `written_bytes` | `UInt64` | Written bytes in the operation. |
| `written_rows_quarantine` | `UInt64` | Quarantined rows in the operation. |
| `written_bytes_quarantine` | `UInt64` | Quarantined bytes in the operation. |
| `pipe_id` | `String` | If present, Materialization Pipe ID as returned in the [Pipes API](/docs/api-reference/pipe-api). |
| `pipe_name` | `String` | If present, Materialization Pipe name as returned in the [Pipes API](/docs/api-reference/pipe-api). |
| `release` | `String` | Semantic version of the release (deprecated). |
| `resource_tags` | `Array(String)` | Tags associated with the pipe when the request was made. |
| `cpu_time` | `Float32` | CPU time used by the operation, in seconds. |
| `memory_usage` | `UInt64` | Memory consuptiom by the operation, in bytes. |

### tinybird.datasource_ops_stats (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-datasource-ops-stats)

Data from `datasource_ops_log`, aggregated by day.

| Field | Type | Description |
| --- | --- | --- |
| `event_date` | `Date` | Date of the event. |
| `workspace_id` | `String` | Unique identifier for the workspace. |
| `event_type` | `String` | Name of your data source. |
| `pipe_id` | `String` | Identifier of the pipe. |
| `pipe_name` | `String` | Name of the pipe. |
| `error_count` | `UInt64` | Number of requests with an error. |
| `executions` | `UInt64` | Number of executions. |
| `avg_elapsed_time_state` | `Float32` | Average time spent in elapsed state. |
| `quantiles_state` | `Float32` | 0.9, 0.95 and 0.99 quantiles state. Time in milliseconds (see [Querying _state columns](https://www.tinybird.co/docs/forward/monitoring/service-datasources#querying-state-columns)). |
| `read_bytes` | `UInt64` | Read bytes in the operation. |
| `read_rows` | `UInt64` | Read rows in the Sink operation. |
| `written_rows` | `UInt64` | Written rows in the Sink operation. |
| `read_bytes` | `UInt64` | Read bytes in the operation. |
| `written_bytes` | `UInt64` | Written bytes in the operation. |
| `written_rows_quarantine` | `UInt64` | Quarantined rows in the operation. |
| `written_bytes_quarantine` | `UInt64` | Quarantined bytes in the operation. |
| `resource_tags` | `Array(String)` | Tags associated with the pipe when the request was made. |

### tinybird.endpoint_errors (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-endpoint-errors)

It provides the last 30 days errors of your published endpoints. Tinybird logs all errors with additional information in this data source.

| Field | Type | Description |
| --- | --- | --- |
| `start_datetime` | `DateTime` | Date and time when the API call started. |
| `request_id` | `String` | The id of the request that performed the operation. Format is ULID string. |
| `pipe_id` | `String` | If present, Pipe ID as returned in the [Pipes API](/docs/api-reference/pipe-api). |
| `pipe_name` | `String` | If present, Pipe name as returned in the [Pipes API](/docs/api-reference/pipe-api). |
| `params` | `Nullable(String)` | URL query params included in the request. |
| `url` | `Nullable(String)` | URL path. |
| `status_code` | `Nullable(Int32)` | HTTP error code. |
| `error` | `Nullable(String)` | Error message. |
| `resource_tags` | `Array(String)` | Tags associated with the pipe when the request was made. |

### tinybird.kafka_ops_log (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-kafka-ops-log)

Contains all operations performed to your Kafka data sources during the last 30 days.

| Field | Type | Description |
| --- | --- | --- |
| `timestamp` | `DateTime` | Date and time when the operation took place. |
| `datasource_id` | `String` | Id of your data source. The data source id is consistent after renaming operations. You should use the id when you want to track name changes. |
| `topic` | `String` | Kafka topic. |
| `partition` | `Int16` | Partition number, or `-1` for all partitions. |
| `msg_type` | `String` | 'info' for regular messages, 'warning' for issues related to the user's Kafka cluster, deserialization or materialized views, and 'error' for other issues. |
| `lag` | `Int64` | Number of messages behind for the partition. This is the difference between the high-water mark and the last commit offset. |
| `processed_messages` | `Int32` | Messages processed for a topic and partition. |
| `processed_bytes` | `Int32` | Amount of bytes processed. |
| `committed_messages` | `Int32` | Messages ingested for a topic and partition. |
| `time_read` | `Float32` | Time taken to read messages from Kafka, in seconds. |
| `time_process` | `Float32` | Time taken to process the messages, in seconds. |
| `time_write` | `Float32` | Time taken to write messages to ClickHouse®, in seconds. |
| `msg` | `String` | Information in the case of warnings or errors. Empty otherwise. |

### tinybird.datasources_storage (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-datasources-storage)

Contains stats about your data sources storage.

Tinybird logs maximum values per hour, the same as when it calculates storage consumption.

| Field | Type | Description |
| --- | --- | --- |
| `datasource_id` | `String` | Id of your data source. The data source id is consistent after renaming operations. You should use the id when you want to track name changes. |
| `datasource_name` | `String` | Name of your data source. |
| `timestamp` | `DateTime` | When storage was tracked. By hour. |
| `bytes` | `UInt64` | Max number of bytes the data source has, not including quarantine. |
| `rows` | `UInt64` | Max number of rows the data source has, not including quarantine. |
| `bytes_quarantine` | `UInt64` | Max number of bytes the data source has in quarantine. |
| `rows_quarantine` | `UInt64` | Max number of rows the data source has in quarantine. |

### tinybird.releases_log (deprecated) (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-releases-log-deprecated)

Contains operations performed to your releases. Tinybird tracks the following operations:

| Event | Description |
| --- | --- |
| `init` | First Release is created on Git sync. |
| `override` | Release commit is overridden. `tb init --override-commit {{commit}}`. |
| `deploy` | Resources from a commit are deployed to a Release. |
| `preview` | Release status is changed to preview. |
| `promote` | Release status is changed to live. |
| `post` | Resources from a commit are deployed to the live Release. |
| `rollback` | Rollback is done a previous Release is now live. |
| `delete` | Release is deleted. |
Tinybird logs all operations with additional information in this data source.

| Field | Type | Description |
| `timestamp` | `DateTime64` | Date and time when the operation took place. |
| `event_type` | `String` | Name of your data source. |
| `semver` | `String` | Semantic version identifies a release. |
| `commit` | `String` | Git sha commit related to the operation. |
| `token` | `String` | API call token identifier used. |
| `token_name` | `String` | API call token name used. |
| `result` | `String` | `ok` \| `error` |
| `error` | `String` | Detailed error message. |

### tinybird.sinks_ops_log (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-sinks-ops-log)

Contains all operations performed to your Sink pipes.

| Field | Type | Description |
| `timestamp` | `DateTime64` | Date and time when the operation took place. |
| `service` | `LowCardinality(String)` | Type of Sink (GCS, S3, and so on). |
| `pipe_id` | `String` | The ID of the Sink pipe. |
| `pipe_name` | `String` | the name of the Sink pipe. |
| `token_name` | `String` | Token name used. |
| `result` | `LowCardinality(String)` | `ok` \| `error` |
| `error` | `Nullable(String)` | Detailed error message. |
| `elapsed_time` | `Float64` | The duration of the operation in seconds. |
| `job_id` | `Nullable(String)` | ID of the job that performed the operation, if any. |
| `read_rows` | `UInt64` | Read rows in the Sink operation. |
| `written_rows` | `UInt64` | Written rows in the Sink operation. |
| `read_bytes` | `UInt64` | Read bytes in the operation. |
| `written_bytes` | `UInt64` | Written bytes in the operation. |
| `output` | `Array(String)` | The outputs of the operation. In the case of writing to a bucket, the name of the written files. |
| `parameters` | `Map(String, String)` | The parameters used. Useful to debug the parameter query values. |
| `options` | `Map(String, String)` | Extra information. You can access the values with `options['key']` where key is one of: file_template, file_format, file_compression, bucket_path, execution_type. |
| `cpu_time` | `Float64` | The CPU time used by the sinks, in seconds. |

### tinybird.data_transfer (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-data-transfer)

Stats of data transferred per hour by a workspace.

| Field | Type | Description |
| `timestamp` | `DateTime` | Date and time data transferred is tracked. By hour. |
| `event` | `LowCardinality(String)` | Type of operation generated the data (ie. `sink`). |
| `origin_provider` | `LowCardinality(String)` | Provider data was transferred from. |
| `origin_region` | `LowCardinality(String)` | Region data was transferred from. |
| `destination_provider` | `LowCardinality(String)` | Provider data was transferred to. |
| `destination_region` | `LowCardinality(String)` | Region data was transferred to. |
| `kind` | `LowCardinality(String)` | `intra` \| `inter` depending if the data moves within or outside the region. |

### tinybird.llm_usage (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-llm-usage)

Contains LLM usage metrics from Tinybird AI features including token consumption, costs, and model usage for each request in the workspace.

| Field | Type | Description |
| --- | --- | --- |
| `start_time` | `DateTime` | Request start timestamp. |
| `end_time` | `DateTime` | Request end timestamp. |
| `organization_id` | `String` | Organization identifier. |
| `organization_name` | `String` | Organization name. |
| `workspace_id` | `String` | Workspace identifier. |
| `workspace_name` | `String` | Workspace name. |
| `user_email` | `String` | email of the user who made the request. |
| `request_id` | `String` | Unique identifier for the LLM request. |
| `prompt_tokens` | `UInt32` | Number of tokens in the prompt. |
| `completion_tokens` | `UInt32` | Number of tokens in the completion. |
| `total_tokens` | `UInt32` | Total number of tokens used. |
| `duration` | `Float32` | Request duration in seconds. |
| `cost` | `Float32` | Cost of the LLM request. |
| `origin` | `String` | Origin of the request. |
| `feature` | `String` | Tinybird AI feature used. |

### tinybird.hfi_requests (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-hfi-requests)

Contains telemetry for High-Frequency Ingestion (HFI) requests in your workspace.

| Field | Type | Description |
| --- | --- | --- |
| `event_time` | `DateTime` | Time slot for the ingestion event. |
| `workspace_id` | `String` | Workspace identifier. |
| `datasource_id` | `String` | Data Source identifier. |
| `successful_requests` | `UInt64` | Number of successful ingestion requests. |
| `error_requests` | `UInt64` | Number of failed ingestion requests. |
| `rows_total` | `UInt64` | Total number of rows received. |
| `rows_successful` | `UInt64` | Number of successfully ingested rows. |
| `rows_quarantined` | `UInt64` | Number of quarantined rows. |
| `bytes_received` | `UInt64` | Number of bytes received. |
| `bytes_decompressed` | `UInt64` | Number of bytes after decompression. |
| `ch_written_rows` **(deprecated)** | `UInt64` | **Deprecated.** |
| `ch_written_bytes` **(deprecated)** | `UInt64` | **Deprecated.** |
| `ch_cpu_time` **(deprecated)** | `Float32` | **Deprecated.** |
| `rate_limited` | `UInt8` | `1` if the request was rate-limited, `0` otherwise. |
| `ips` | `Array(String)` | Deduplicated list of source IPs seen for the aggregated bucket. |
| `tokens` | `Array(String)` | Deduplicated list of token identifiers seen for the aggregated bucket. |

### tinybird.jobs_log (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-jobs-log)

Contains all job executions performed in your workspace. Tinybird logs all jobs with extra information in this data source:

| Field | Type | Description |
| --- | --- | --- |
| `job_id` | `String` | Unique identifier for the job. |
| `job_type` | `LowCardinality(String)` | Type of job execution: `copy`, `delete`, `deployment`, `dynamodb_sync`, `gcs_sync`, `import`, `populate`, `s3_sync`, `sink`. |
| `workspace_id` | `String` | Unique identifier for the workspace. |
| `pipe_id` | `String` | Unique identifier for the pipe. |
| `pipe_name` | `String` | Name of the pipe. |
| `created_at` | `DateTime` | Timestamp when the job was created. |
| `updated_at` | `DateTime` | Timestamp when the job was last updated. |
| `started_at` | `DateTime` | Timestamp when the job execution started. |
| `status` | `LowCardinality(String)` | Current status of the job. `waiting`, `working`, `done`, `error`, `cancelled`. |
| `error` | `Nullable(String)` | Detailed error message if the result was error. |
| `job_metadata` | `JSON String` | Additional metadata related to the job execution. |
Learn more about how to track background jobs execution in the [Jobs monitoring guide](./jobs).

### tinybird.query_validator_log (https://www.tinybird.co/docs/forward/monitoring/service-datasources#tinybird-query-validator-log)

Contains information about query validation executions for Pipes and Materialized Views in your Workspace.

| Field | Type | Description |
| --- | --- | --- |
| `host` | `LowCardinality(String)` | Name of the host the query was extracted from. It might differ from the host the query was executed on. |
| `version` | `LowCardinality(String)` | The version in which the query was tested. |
| `stable_version` | `LowCardinality(String)` | The version that was running in the cluster when the query was validated. |
| `query_hash` | `UInt64` | The normalized query hash. |
| `query_last_execution` | `DateTime` | The last execution time of this query at the time when it was extracted. |
| `region` | `String` | The region in which the cluster is deployed. |
| `workspace` | `String` | The name of the Workspace, for Pipe queries. |
| `database` | `String` | The internal database name for this Workspace. |
| `pipe_name` | `String` | The name of the Pipe, for Pipe queries. |
| `error_code` | `Int16` | Error code returned by ClickHouse®. |
| `error` | `String` | Detailed error message if the query failed. |
| `query_id` | `String` | ClickHouse® query ID. For Pipe queries, this can be matched with `system.query_log`. For Materialized Views, it contains the qualified internal table name. |
| `query` | `String` | The full query. |
| `fix_suggestion` | `String` | Optionally, a suggestion on how to fix the query. |
| `run_validation` | `DateTime` | The time when this query was validated. |
Learn more about query validation in <a href="about:blank#tinybird-query-validator-log">`tinybird.query_validator_log`</a>.

## Query _state columns (https://www.tinybird.co/docs/forward/monitoring/service-datasources#query-state-columns)

Several of the Service data sources include columns suffixed with `_state`. This suffix identifies columns with values that are in an intermediate aggregated state. When reading these columns, merge the intermediate states to get the final value.

To merge intermediate states, wrap the column in the original aggregation function and apply the `-Merge` combinator.

For example, to finalize the value of the `avg_duration_state` column, you use the `avgMerge` function:

finalize the value for the avg_duration_state column
```
SELECT
  date,
  avgMerge(avg_duration_state) avg_time,
  quantilesTimingMerge(0.9, 0.95, 0.99)(quantile_timing_state) quantiles_timing_in_ms_array
FROM tinybird.pipe_stats
where pipe_id = 'PIPE_ID'
group by date
```
See [Combinators](/docs/sql-reference/functions/aggregate-functions#aggregate-function-combinators) to learn more about the `-Merge` combinator.

## Organization service data sources (https://www.tinybird.co/docs/forward/monitoring/service-datasources#organization-service-data-sources)

The following is a complete list of available Organization Service data sources:

| Field | Description |
| --- | --- |
| `organization.workspaces` | Lists all Organization workspaces and related information, including name, IDs, databases, plan, when it was created, and whether it has been soft-deleted. |
| `organization.processed_data` **(deprecated)** | **Deprecated and no longer works as of April 1, 2026.** Use `system.query_log` instead. See the [changelog](/docs/changelog/2026-03-27-march-week-four#deprecation-of-bi-and-billing-service-data-sources) for migration examples. |
| `organization.datasources_storage` | Equivalent to `tinybird.datasources_storage` but with data for all Organization workspaces. |
| `organization.pipe_stats` | Equivalent to `tinybird.pipe_stats` but with data for all Organization workspaces. |
| `organization.pipe_stats_rt` | Equivalent to `tinybird.pipe_stats_rt` but with data for all Organization workspaces. |
| `organization.datasources_ops_log` | Equivalent to `tinybird.datasources_ops_log` but with data for all Organization workspaces. |
| `organization.data_transfer` | Equivalent to `tinybird.data_transfer` but with data for all Organization workspaces. |
| `organization.jobs_log` | Equivalent to `tinybird.jobs_log` but with data for all Organization workspaces. |
| `organization.sinks_ops_log` | Equivalent to `tinybird.sinks_ops_log` but with data for all Organization workspaces. |
| `organization.endpoint_errors` | Equivalent to `tinybird.endpoint_errors` but with data for all Organization workspaces. |
| `organization.shared_infra_active_minutes` | Contains information about vCPU active minutes consumption aggregated by minute for all Organization workspaces. Used for SaaS plan billing on shared infrastructure. Deprecated soon for other shared plans. |
| `organization.shared_infra_active_seconds` | Contains the total vCPU time in seconds consumed by all operations in the organization during each natural second. Available for Developer and SaaS plans on shared infrastructure. |
| `organization.shared_infra_qps_overages` | Contains information about QPS consumption and overages aggregated by second for all Organization workspaces. Used for SaaS plan billing on shared infrastructure. |
| `organization.llm_usage` | Equivalent to `tinybird.llm_usage` but with data for all Organization workspaces. |
| `organization.backups` | Contains backup metadata for dedicated clusters in the organization, including size and status. |
| `organization.private_link` | Contains Private Link usage and cost data for observability, associated with your organization's workspaces. |
To query Organization Service data sources, go to any workspace that belongs to the Organization and use them as regular Service data sources from the Playground or within pipes. You can also make queries using your preferred method, like `tb sql`.

### metrics_logs service data source (https://www.tinybird.co/docs/forward/monitoring/service-datasources#metrics-logs-service-data-source)

The `metrics_logs` Service data source is available in all the organization's workspaces to all Organization members. New records for each of the metrics monitored are added every minute with the following schema:

The `metrics_logs` Service Data Source is only populated for Organizations with **dedicated infrastructure**. It contains hardware-level measurements (CPU, memory, disk) of dedicated cluster nodes. If your Organization is on shared infrastructure, this table is empty. For monitoring on shared infrastructure, use other Service Data Sources like `datasources_ops_log`, `pipe_stats`, and `datasources_storage`.

| Field | Type | Description |
| --- | --- | --- |
| `timestamp` | `DateTime` | Timestamp of the metric |
| `cluster` | `LowCardinality(String)` | Name of the cluster |
| `host` | `LowCardinality(String)` | Name of the host |
| `metric` | `LowCardinality(String)` | Name of the metric |
| `value` | `String` | Value of the metric |
| `description` | `LowCardinality(String)` | Description of the metric |
| `organization_id` | `String` | ID of your organization |
The available metrics are the following:

| Metric | Description |
| --- | --- |
| MemoryTracking | Total amount of memory, in bytes, allocated by the server. |
| OSMemoryTotal | The total amount of memory on the host system, in bytes. |
| InstanceType | Instance type of the host. |
| Query | Number of executing queries. |
| NumberCPU | Number of CPUs. |
| LoadAverage1 | The whole system load, averaged with exponential smoothing over 1 minute. The load represents the number of threads across all the processes (the scheduling entities of the OS kernel), that are currently running by CPU or waiting for IO, or ready to run but not being scheduled at this point of time. This number includes all the processes, not only the server. The number can be greater than the number of CPU cores, if the system is overloaded, and many processes are ready to run but waiting for CPU or IO. |
| LoadAverage15 | The whole system load, averaged with exponential smoothing over 15 minutes. The load represents the number of threads across all the processes (the scheduling entities of the OS kernel), that are currently running by CPU or waiting for IO, or ready to run but not being scheduled at this point of time. This number includes all the processes, not only the server. The number can be greater than the number of CPU cores, if the system is overloaded, and many processes are ready to run but waiting for CPU or IO. |
| CPUUsage | The ratio of time the CPU core was running OS kernel (system) code or userspace code. This is a system-wide metric, it includes all the processes on the host machine, not only the server. This includes also the time when the CPU was under-utilized due to the reasons internal to the CPU (memory loads, pipeline stalls, branch mispredictions, running another SMT core). |
| ReplicasSumQueueSize | Sum queue size (in the number of operations like get, merge) across Replicated tables. |
| ReplicasSumMergesInQueue | Sum of merge operations in the queue (still to be applied) across Replicated tables. |
| ReplicasSumInsertsInQueue | Sum of insert operations in the queue (still to be applied) across Replicated tables. |

---

URL: https://www.tinybird.co/docs/forward/monitoring/query-log
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "How to debug queries using system.query\_log \| Tinybird"
theme-color: "#171612"
description: "Learn how to use system.query_log to debug slow queries, analyze memory usage, and troubleshoot issues in your Tinybird Workspace."
inkeep:version: "forward"
---

# How to debug queries using system.query_log (https://www.tinybird.co/docs/forward/monitoring/query-log#how-to-debug-queries-using-system-query-log)

When troubleshooting performance issues or investigating query failures, you often need execution details beyond what standard monitoring provides. The `system.query_log` table is a native ClickHouse® system table that records query execution at the database engine level beneath Tinybird. It stores detailed information for every query executed in your Workspace, including execution time, memory usage, error codes, and failure reasons. For a complete description of available columns and data types, see the ClickHouse® documentation on the <a href="https://clickhouse.com/docs/operations/system-tables/query_log">`system.query_log`</a> table.

**When to use system.query_log**

Use `system.query_log` when you need to:

- Debug queries that are slower than expected
- Investigate why specific queries are failing
- Analyze memory consumption patterns
- Track which Endpoints consume the most resources
- Correlate query execution with Endpoint performance metrics
For routine monitoring of Endpoint performance, use <a href="./analyze-endpoints-performance">`tinybird.pipe_stats_rt`</a> instead.

Read on to learn how to safely query `system.query_log` and use it to solve real-world debugging challenges.

## Before you start (https://www.tinybird.co/docs/forward/monitoring/query-log#before-you-start)

The `system.query_log` table is a ClickHouse® system table that contains extensive historical data. Before querying it, understand:

- How to [analyze Endpoint performance](./analyze-endpoints-performance) using `tinybird.pipe_stats_rt`
- Basic SQL query patterns in Tinybird
The `system.query_log` table contains all historical query data. **Always filter by `event_time`** to avoid slow queries or timeouts. Queries without time filters can take hundreds of seconds or timeout completely. **Never run** `SELECT * FROM system.query_log` without filtering by time.

Safe query pattern with 1-hour lookback
```
SELECT *
FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
```
Start with a 1-hour window (typically 1-2 seconds to execute) and expand the time range only if needed.

## Understand query lifecycle (https://www.tinybird.co/docs/forward/monitoring/query-log#understand-query-lifecycle)

Every query in Tinybird generates two entries in `system.query_log`:

1. **QueryStart** - Logged when query execution begins
2. **QueryFinish** - Logged when the query completes successfully
If a query fails, the log shows a different type instead of `QueryFinish`:

- **ExceptionBeforeStart** - Query failed before execution started (usually syntax errors)
- **ExceptionWhileProcessing** - Query failed during execution (timeouts, memory exceeded, runtime errors)
Here's how the query lifecycle flows:

```
┌─────────────────┐
│  Query Begins   │
└────────┬────────┘
         │
         ▼
    QueryStart ──────────────────┐
         │                       │
         │                       │
         ▼                       ▼
┌─────────────────┐     ┌──────────────────────┐
│ Query Executes  │     │  Syntax/Init Error   │
└────────┬────────┘     └──────────┬───────────┘
         │                         │
         │                         ▼
         │              ExceptionBeforeStart
         │
    ┌────┴────┐
    │         │
    ▼         ▼
 Success   Runtime Error
    │         │
    ▼         ▼
QueryFinish   ExceptionWhileProcessing
```
This two-entry pattern helps you distinguish between queries that are still running and those that have finished or failed.

## Common debugging scenarios (https://www.tinybird.co/docs/forward/monitoring/query-log#common-debugging-scenarios)

### Find slow queries (https://www.tinybird.co/docs/forward/monitoring/query-log#find-slow-queries)

Identify queries taking longer than expected:

Queries taking more than 5 seconds
```
SELECT
    event_time,
    query_id,
    JSONExtractString(log_comment, 'pipe') as pipe_name,
    query_duration_ms / 1000.0 as duration_seconds,
    formatReadableSize(memory_usage) as peak_memory,
    substring(query, 1, 100) as query_preview
FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
    AND type = 'QueryFinish'
    AND query_duration_ms > 5000
ORDER BY query_duration_ms DESC
LIMIT 20
```
This helps you identify performance bottlenecks in your Endpoints or scheduled operations.

### Investigate query failures (https://www.tinybird.co/docs/forward/monitoring/query-log#investigate-query-failures)

Track which queries are failing and why:

Failed queries in the last hour
```
SELECT
    event_time,
    query_id,
    type,
    JSONExtractString(log_comment, 'pipe') as pipe_name,
    exception
FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
    AND type IN ('ExceptionBeforeStart', 'ExceptionWhileProcessing')
ORDER BY event_time DESC
```
The `exception` column contains the error message, helping you quickly identify syntax errors, timeouts, or resource limits.

### Analyze memory consumption (https://www.tinybird.co/docs/forward/monitoring/query-log#analyze-memory-consumption)

Find queries consuming the most memory:

Memory-intensive queries
```
SELECT
    event_time,
    query_id,
    JSONExtractString(log_comment, 'pipe') as pipe_name,
    formatReadableSize(memory_usage) as peak_memory,
    query_duration_ms / 1000.0 as duration_seconds
FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
    AND type = 'QueryFinish'
ORDER BY memory_usage DESC
LIMIT 20
```
The `memory_usage` column shows peak memory in bytes. High memory usage may indicate opportunities for query optimization.

### Track query failure rates (https://www.tinybird.co/docs/forward/monitoring/query-log#track-query-failure-rates)

Monitor how often queries are failing over time:

Query failures by hour
```
SELECT
    toStartOfHour(event_time) as hour,
    type,
    count() as failure_count
FROM system.query_log
WHERE event_time >= now() - INTERVAL 24 HOUR
    AND type IN ('ExceptionBeforeStart', 'ExceptionWhileProcessing')
GROUP BY hour, type
ORDER BY hour DESC
```

## Connect query_log with Endpoint stats (https://www.tinybird.co/docs/forward/monitoring/query-log#connect-query-log-with-endpoint-stats)

The `query_id` in `system.query_log` matches the `request_id` from `tinybird.pipe_stats_rt`. Use these fields to combine high-level request metrics with detailed query execution data.

Combine endpoint stats with query details
```
SELECT
    ps.pipe_name,
    ps.start_datetime,
    ps.duration as request_duration,
    formatReadableSize(ql.memory_usage) as query_memory,
    ql.type as execution_status
FROM tinybird.pipe_stats_rt ps
JOIN system.query_log ql ON ps.request_id = ql.query_id
WHERE ps.start_datetime >= now() - INTERVAL 1 HOUR
    AND ql.type = 'QueryFinish'
ORDER BY ps.start_datetime DESC
LIMIT 100
```
The `event_time` from `system.query_log` and `start_datetime` from `pipe_stats_rt` may differ by a few milliseconds. Always join using `query_id` and `request_id` for accurate matching.

## Extract pipe names from log_comment (https://www.tinybird.co/docs/forward/monitoring/query-log#extract-pipe-names-from-log-comment)

Endpoint queries include metadata in the `log_comment` column as JSON. Extract the pipe name to filter by specific Endpoints:

Filter by Endpoint name
```
SELECT
    event_time,
    query_id,
    JSONExtractString(log_comment, 'pipe') as pipe_name,
    JSONExtractString(log_comment, 'workspace') as workspace,
    formatReadableSize(memory_usage) as memory_usage,
    query_duration_ms / 1000.0 as duration_seconds
FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
    AND log_comment != ''
    AND JSONExtractString(log_comment, 'pipe') = 'your_pipe_name'
    AND type = 'QueryFinish'
ORDER BY event_time DESC
```
The `log_comment` JSON structure looks like:

```
{"workspace":"prod","pipe":"last_impressions__v7"}
```

## Find memory-intensive endpoints (https://www.tinybird.co/docs/forward/monitoring/query-log#find-memory-intensive-endpoints)

Identify which Endpoints consistently use the most memory:

Average memory by Endpoint
```
SELECT
    JSONExtractString(log_comment, 'pipe') as pipe_name,
    formatReadableSize(avg(memory_usage)) as avg_memory,
    formatReadableSize(max(memory_usage)) as max_memory,
    formatReadableSize(quantile(0.9)(memory_usage)) as p90_memory,
    count() as query_count
FROM system.query_log
WHERE event_time >= now() - INTERVAL 24 HOUR
    AND type = 'QueryFinish'
    AND log_comment != ''
GROUP BY pipe_name
ORDER BY avg(memory_usage) DESC
LIMIT 20
```

## Common mistakes (https://www.tinybird.co/docs/forward/monitoring/query-log#common-mistakes)

Avoid these common pitfalls when working with `system.query_log`:

### Running queries without time filters (https://www.tinybird.co/docs/forward/monitoring/query-log#running-queries-without-time-filters)

**Never do this:**

```
SELECT * FROM system.query_log
WHERE query LIKE '%my_datasource_id%'
```
This scans the entire table history and times out or takes hundreds of seconds.

**Always include time filters:**

```
SELECT * FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
  AND query LIKE '%my_datasource_id%'
```

### Selecting all columns with SELECT * (https://www.tinybird.co/docs/forward/monitoring/query-log#selecting-all-columns-with-select)

Avoid `SELECT *` in production queries. The `query` column can be large, and many columns aren't needed for most debugging tasks.

**Instead, select only what you need:**

```
SELECT
    event_time,
    query_id,
    type,
    query_duration_ms,
    memory_usage,
    exception
FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
```

### Forgetting to filter by query type (https://www.tinybird.co/docs/forward/monitoring/query-log#forgetting-to-filter-by-query-type)

When analyzing completed queries, include `type = 'QueryFinish'` to exclude `QueryStart` entries and avoid counting queries twice:

```
-- Without type filter: counts each query twice (Start + Finish)
SELECT count(*) FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR

-- With type filter: accurate count
SELECT count(*) FROM system.query_log
WHERE event_time >= now() - INTERVAL 1 HOUR
  AND type = 'QueryFinish'
```

### Using overly broad time windows (https://www.tinybird.co/docs/forward/monitoring/query-log#using-overly-broad-time-windows)

Start with 1-hour windows and expand only if needed. A 24-hour query takes significantly longer than a 1-hour query:

- **1 hour**: ~1-2 seconds
- **24 hours**: ~15-30 seconds
- **7 days**: May timeout or take minutes

## Best practices (https://www.tinybird.co/docs/forward/monitoring/query-log#best-practices)

When querying `system.query_log`, follow these guidelines:

1. **Always filter by time** - Use `event_time >= now() - INTERVAL X HOUR` in every query
2. **Filter by query type** - Add `AND type = 'QueryFinish'` if you only need completed queries
3. **Limit your results** - Use `LIMIT` to avoid processing unnecessary data
4. **Start small** - Begin with 1-hour windows, then expand as needed
5. **Create monitoring Pipes** - Turn useful debug queries into Endpoints for dashboards
Convert your most useful `system.query_log` queries into Pipes and expose them as Endpoints. This makes it easy to build real-time monitoring dashboards or set up alerts.

## Next steps (https://www.tinybird.co/docs/forward/monitoring/query-log#next-steps)

- Learn how to [analyze Endpoint performance](./analyze-endpoints-performance) using `tinybird.pipe_stats_rt`
- Monitor [Jobs](./jobs) in your Workspace

---

URL: https://www.tinybird.co/docs/forward/monitoring/organization-consumption
Last update: 2026-08-04T10:22:54.000Z
Content:
---
title: "Advanced organization consumption monitoring \| Tinybird"
theme-color: "#171612"
description: "Learn how to monitor your organization's resource consumption in detail using SQL queries"
inkeep:version: "forward"
---

# Advanced organization consumption monitoring (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#advanced-organization-consumption-monitoring)

While Tinybird provides built-in graphs and metrics in the UI for monitoring your organization's resource consumption, some use cases require more detailed information. This guide explains how to use SQL queries to monitor your consumption in detail, specifically for organizations using shared infrastructure.

This monitoring approach applies only to Developer and SaaS customers on shared infrastructure. If you're on a dedicated infrastructure plan, contact your account manager for specific monitoring solutions.

## Prerequisites (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#prerequisites)

- Your organization must be on shared infrastructure (Developer or SaaS plans)
- Basic understanding of SQL and Tinybird's Data Sources

## Understanding organization usage (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#understanding-organization-usage)

By default, Tinybird provides usage graphs in the UI that show:

- Resource consumption over the last 7 days
- A usage table displaying resources consuming the most vCPU
While these built-in visualizations are sufficient for most use cases, you might need more granular control and insight into your consumption patterns.

## Advanced vCPU monitoring (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#advanced-vcpu-monitoring)

Tinybird uses vCPU active minutes for SaaS plan billing on shared infrastructure. Developer plans bill vCPU overages using vCPU seconds instead. See <a href="/docs/forward/monitoring/service-datasources#organization-service-data-sources">`organization.shared_infra_active_seconds`</a>.

### Important notes about vCPU metrics (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#important-notes-about-vcpu-metrics)

- Materialized Views currently show 0 for direct vCPU time
- The landing Data Source includes both its own CPU time and the CPU time of its associated Materialized Views

### Detailed CPU usage analysis (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#detailed-cpu-usage-analysis)

You can analyze detailed CPU consumption across different operations using several service data sources:

#### API and SQL operations (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#api-and-sql-operations)

Using `pipe_stats` and `pipe_stats_rt` service data sources, you can monitor CPU usage for API endpoints and SQL queries:

Here's an example query that shows vCPU consumption by Pipe for the last 7 days:

Monitor vCPU consumption by Pipe for last 7 days
```
SELECT
  pipe_name,
  round(sum(cpu_time), 2) as total_cpu_seconds,
  count() as total_requests
FROM organization.pipe_stats_rt
WHERE
  start_datetime >= (now() - interval 7 day)
GROUP BY pipe_name
ORDER BY total_cpu_seconds DESC
```

#### Data Source operations (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#data-source-operations)

The `datasources_ops_log` service data source provides CPU metrics for operations on your Data Sources.

Here's an example query that shows vCPU consumption by Data Source and operation type:

Monitor Data Source operations vCPU consumption
```
SELECT
  datasource_name,
  event_type,
  round(sum(cpu_time), 2) as total_cpu_seconds,
  count() as total_operations
FROM organization.datasources_ops_log
WHERE
  timestamp >= (now() - interval 7 day)
GROUP BY
  datasource_name,
  event_type
ORDER BY total_cpu_seconds DESC
```

#### Sinks operations (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#sinks-operations)

Monitor CPU usage in your Sinks using the `sinks_ops_log` service data source:

Here's an example query that shows vCPU consumption by Sink:

Monitor Sinks operations vCPU consumption
```
SELECT
  pipe_name,
  round(sum(cpu_time), 2) as total_cpu_seconds,
  count() as total_operations
FROM organization.sinks_ops_log
WHERE
  timestamp >= (now() - interval 7 day)
GROUP BY pipe_name
ORDER BY total_cpu_seconds DESC
```

### Developer vCPU seconds (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#developer-vcpu-seconds)

For Developer plan vCPU overage billing, use the `shared_infra_active_seconds` service data source:

Monitor vCPU seconds for current day
```
SELECT * FROM organization.shared_infra_active_seconds
WHERE
  toStartOfDay(second) = today()
ORDER BY second DESC
```

### SaaS vCPU active minutes (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#saas-vcpu-active-minutes)

For SaaS plan active minutes billing, use the `shared_infra_active_minutes` service data source:

Monitor active minutes for current day
```
SELECT * FROM organization.shared_infra_active_minutes
WHERE
  toStartOfDay(minute) = today()
ORDER BY minute DESC
```

## Storage monitoring (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#storage-monitoring)

Storage consumption is a key billing metric that measures the amount of data stored in your Data Sources. You can monitor storage usage using the `datasources_storage` service data source.

Storage is billed based on two factors:

1. The maximum total storage used by your organization each day (including quarantined data)
2. The average of those daily maximums throughout your billing cycle
Storage for data in quarantine is included in your billing calculations. When monitoring storage for costs, always consider both regular and quarantined data.

### Current storage usage (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#current-storage-usage)

Here's an example query that shows current storage usage by Data Source, including both regular and quarantined data:

Monitor current storage usage by Data Source
```
SELECT
  datasource_name,
  round((bytes + bytes_quarantine)//1000000000, 2) as total_storage_gb,
  round(bytes_quarantine//1000000000, 2) as quarantine_storage_gb,
  rows + rows_quarantine as total_rows,
  rows_quarantine as quarantine_rows
FROM organization.datasources_storage
WHERE timestamp >= (now() - interval 2 hour)
ORDER BY total_storage_gb DESC
LIMIT 1 BY datasource_name
```

### Billing period storage analysis (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#billing-period-storage-analysis)

To analyze your storage consumption for billing purposes, use this query that calculates the average of daily maximum storage across your billing period:

Calculate average storage for billing period
```
SELECT
    greatest(avg(daily_max_org_storage_gb), 0) as avg_storage_gb,
    greatest(avg(daily_max_org_storage_rows), 0) as avg_storage_rows
FROM (
  SELECT
    sum(floor(max_total_bytes_by_ds/1000000000, 6)) as daily_max_org_storage_gb,
    sum(max_total_rows_by_ds) as daily_max_org_storage_rows
  FROM (
    SELECT
        toDate(timestamp) as date,
        max(bytes + bytes_quarantine) as max_total_bytes_by_ds,
        max(rows + rows_quarantine) AS max_total_rows_by_ds
    FROM organization.datasources_storage
    WHERE 1= 1
        AND date >= '2025-04-XX' -- beginning of term
        AND date <= '2025-04-xx' -- end of term
    GROUP BY date, datasource_id
  )
  GROUP BY date
)
```
Replace the date placeholders (`'2025-04-XX'`) with your actual billing period start and end dates to get accurate billing metrics.

## QPS (queries per second) monitoring (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#qps-queries-per-second-monitoring)

You can monitor your QPS consumption using two different data sources, each providing different information about your usage:

### Detailed query analysis (last 7 days) (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#detailed-query-analysis-last-7-days)

Using the `pipe_stats_rt` service data source, you can analyze detailed information about your API endpoints and SQL queries usage. This data source provides rich information about each query but is limited to the last 7 days due to TTL.

The `pipe_stats_rt` data source has a 7-day TTL (Time To Live), so historical analysis is limited to this timeframe.

Here's an example query that shows the number of requests per Pipe over the last hour:

Monitor QPS by pipe for the last hour
```
SELECT
  start_datetime,
  pipe_name,
  count() total
FROM organization.pipe_stats_rt
WHERE
  start_datetime BETWEEN (now() - interval 1 hour) AND now()
GROUP BY
  start_datetime, pipe_name
ORDER BY
  start_datetime DESC
```

### Historical QPS and overages (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#historical-qps-and-overages)

For longer-term analysis of QPS consumption and overages, you can use the `shared_infra_qps_overages` service data source. This provides aggregated QPS data and overage information per second, though with less detail about individual queries.

Here's an example query that shows daily QPS overages for the current month:

Monitor daily QPS overages for current month
```
SELECT
  toStartOfDay(start_datetime) day,
  sum(overages) total_overages
FROM organization.shared_infra_qps_overages
WHERE
  toStartOfMonth(start_datetime) = toStartOfMonth(now())
GROUP BY day
ORDER BY day DESC
```

## Data transfer monitoring (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#data-transfer-monitoring)

Data transfer metrics track the amount of data moved through Sinks in your organization. The cost varies depending on whether data is transferred within the same region (Intra) or between different regions (Inter).

### Sinks data transfer (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#sinks-data-transfer)

Monitor data transfer costs for Sinks using the `data_transfer` service data source:

Monitor Sinks data transfer by type
```
SELECT
  toStartOfDay(timestamp) as day,
  workspace_id,
  kind,
  round(sum(bytes)/1000000000, 2) as transferred_gb,
  count() as operations
FROM organization.data_transfer
WHERE
  timestamp >= (now() - interval 30 day)
  AND kind IN ('intra', 'inter')
GROUP BY
  day,
  workspace_id,
  kind
ORDER BY
  day DESC,
  kind ASC
```

## Best practices (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#best-practices)

1. **Regular Monitoring**: Set up a routine to check these metrics, especially if you're approaching your plan limits
2. **Alert Setup**: Consider setting up alerts using these queries to proactively monitor consumption
3. **Resource Optimization**: Use these metrics to identify opportunities for query optimization or resource reallocation

## Additional resources (https://www.tinybird.co/docs/forward/monitoring/organization-consumption#additional-resources)

- [Organizations](/docs/forward/core-concepts/organizations) - Learn about organization management and monitoring
- [Pricing plans](../pricing) - Compare billing models and SKUs across Free, Developer, SaaS, and Enterprise plans
- [Resource Limits](/docs/forward/pricing/limits) - Learn about the storage, QPS, and other resource limits for different plans

---

URL: https://www.tinybird.co/docs/forward/monitoring/latency
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Measure API endpoint latency \| Tinybird"
theme-color: "#171612"
description: "Latency is an essential metric to monitor in real-time applications. Learn how to measure and monitor the latency of your API endpoints in Tinybird."
inkeep:version: "forward"
---

# Measure API endpoint latency (https://www.tinybird.co/docs/forward/monitoring/latency#measure-api-endpoint-latency)

Latency is the time it takes for a request to travel from the client to the server and back; the time it takes for a request to be sent and received. Latency is usually measured in seconds or milliseconds (ms). The lower the latency, the faster the response time.

Latency is an essential metric to monitor in real-time applications. Read on to learn how latency is measured in Tinybird, and how to monitor and visualize the latency of your API endpoints when data is being retrieved.

## How latency is measured (https://www.tinybird.co/docs/forward/monitoring/latency#how-latency-is-measured)

When measuring latency in an end-to-end app, consider data ingestion, data transformation, and data retrieval. In Tinybird, latency is the time it takes to send a request and return the response to the client.

When calling an API endpoint, you can check this metric defined as `elapsed` in the `statistics` object of the response:

Statistics object within an example Tinybird API endpoint call
```
{
  "meta": [ ... ],
  "data": [ ... ],
  "rows": 10,
  "statistics": {
    "elapsed": 0.001706275,
    "rows_read": 10,
    "bytes_read": 180
  }
}
```

## Monitor latency (https://www.tinybird.co/docs/forward/monitoring/latency#monitor-latency)

To monitor the latency of your API endpoints, use the `pipe_stats_rt` and `pipe_stats` [Service data sources](./service-datasources):

- `pipe_stats_rt` consists of the real-time statistics of your API endpoints, and has a `duration` field that encapsulates the latency time in seconds.
- `pipe_stats` contains the **aggregated** statistics of your API endpoints by date, and presents a `avg_duration_state` field which is the average duration of the API endpoint by day in seconds.
Because the `avg_duration_state` field is an intermediate state, you'd need to merge it when querying the data source using something like `avgMerge`.

For details on building pipes and endpoints that monitor the performance of your API endpoints using the `pipe_stats_rt` and `pipe_stats` data sources, follow the [API endpoint performance guide](./analyze-endpoints-performance#example-2-analyzing-the-performance-of-api-endpoints-over-time).

## Visualize latency (https://www.tinybird.co/docs/forward/monitoring/latency#visualize-latency)

In your workspace, go to **Time Series** and select **Endpoint performance** to visualize the latency of your API endpoints over time.

## Next steps (https://www.tinybird.co/docs/forward/monitoring/latency#next-steps)

- Read this blog on [Monitoring global API latency](https://www.tinybird.co/blog-posts/dev-qa-global-api-latency-chronark).

---

URL: https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring
Last update: 2026-07-24T12:51:58.000Z
Content:
---
title: "Monitor Kafka to ClickHouse® Connector Lag \| Tinybird"
theme-color: "#171612"
description: "Monitor Tinybird Kafka connector lag, throughput, errors, and partition behavior with Kafka meta columns and the kafka_ops_log Service Data Source."
inkeep:version: "forward"
---

# Monitor Kafka to ClickHouse® connector and consumer lag (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#monitor-kafka-to-clickhouse-connector-and-consumer-lag)

Tinybird's Kafka connector automatically consumes data from Kafka topics and ingests it into ClickHouse® Data Sources. The connector features consumer autoscaling and multi-broker connectivity. The `tinybird.kafka_ops_log` [Service Data Source](/docs/forward/monitoring/service-datasources) provides operational logs for monitoring your integration, tracking consumer lag, analyzing performance, and optimizing your connector configuration.

Use `kafka_ops_log` to monitor consumer lag, track throughput, identify errors, and analyze performance across your Data Sources. The connector's serverless architecture automatically scales consumers based on load, and `kafka_ops_log` provides visibility into this autoscaling behavior along with error handling and debugging information.

Logs are retained for 30 days and include information about each partition, topic, and Data Source in your workspace. This supports analysis of partition assignment, rebalancing, and consumer group scaling behavior.

**New to the Kafka connector?** If you're setting up the Kafka connector for the first time, see the [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka) for setup instructions, configuration options, and troubleshooting. This monitoring guide focuses on operational monitoring once your connector is running.

## Kafka meta columns in your Data Sources (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#kafka-meta-columns-in-your-data-sources)

In addition to monitoring via `kafka_ops_log`, each Data Source connected to Kafka includes [Kafka meta columns](/docs/forward/ingest-data/connectors/kafka#kafka-meta-columns) that store metadata from Kafka messages. These columns are automatically added to your Data Source schema and can be used for analysis, debugging, and monitoring.

The available meta columns include:

- **`__topic`**: The Kafka topic the message was read from
- **`__partition`**: The partition number the message was read from
- **`__offset`**: The message offset within the partition
- **`__timestamp`**: The timestamp of the Kafka message
- **`__key`**: The message key (if present)
- **`__value`**: The raw message value (if `KAFKA_STORE_RAW_VALUE` is set to `True`)
- **`__headers`**: Kafka headers as a Map (if `KAFKA_STORE_HEADERS` is set to `True`)

### Use cases for Kafka meta columns (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#use-cases-for-kafka-meta-columns)

**Track message ordering and detect gaps**: Use `__offset` to identify missing messages or gaps in your data ingestion. Compare consecutive offsets to detect when messages are skipped.

**Analyze distribution**: Use `__partition` to understand how messages are distributed across partitions. This helps verify that your partition key design is working as expected and that messages are evenly distributed.

**Correlate with `kafka_ops_log`**: Join your Data Source with `kafka_ops_log` using `__partition` and `__offset` to correlate ingested messages with connector operations, helping you understand ingestion latency and identify bottlenecks.

**Message timestamp analysis**: Compare `__timestamp` (Kafka message timestamp) with your Data Source's ingestion timestamp to understand end-to-end latency from message production to ClickHouse ingestion.

**Debug with message keys**: Use `__key` to track specific messages or understand how keys map to partitions, which is essential for partition key design optimization.

**Leverage headers for routing and tracing**: If `KAFKA_STORE_HEADERS` is set to `True`, use `__headers` to access custom metadata, trace IDs, routing information, or other context stored in Kafka headers. This is useful for distributed tracing, A/B testing, or routing logic based on header values.

| Field | Type | Description |
| --- | --- | --- |
| `timestamp` | `DateTime` | Date and time when the operation took place. |
| `datasource_id` | `String` | ID of your Data Source. The Data Source ID is consistent after renaming operations. |
| `topic` | `String` | Kafka topic. |
| `partition` | `Int16` | Partition number, or `-1` for all partitions. Use this field to analyze partition assignment and identify which partitions are assigned to your consumer. |
| `msg_type` | `String` | `'info'` for regular messages, `'warning'` for issues related to the user's Kafka cluster, deserialization or Materialized Views, and `'error'` for other issues. |
| `lag` | `Int64` | Number of messages behind for the partition. This is the difference between the high-water mark and the last commit offset. This is the primary metric for consumer lag monitoring. |
| `processed_messages` | `Int32` | Messages processed for a topic and partition. |
| `processed_bytes` | `Int32` | Amount of bytes processed. |
| `committed_messages` | `Int32` | Messages ingested for a topic and partition. |
| `msg` | `String` | Information in the case of warnings or errors. Empty otherwise. |
The `tinybird.kafka_ops_log` Service Data Source includes operational metrics from your serverless Kafka connector. Each row represents a snapshot of the connector's state for a specific partition at a point in time, providing visibility into autoscaling consumer behavior, error handling, and debugging information.

## Monitor consumer lag (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#monitor-consumer-lag)

Consumer lag is the difference between the latest offset in a partition (high-water mark) and the last committed offset by your consumer. High lag values indicate that your consumer is falling behind the producer, which can lead to delayed data ingestion. With Tinybird's serverless connector, consumer autoscaling automatically adjusts to handle increased load, but monitoring lag helps you understand when scaling occurs and verify that autoscaling is working effectively.

Monitoring consumer lag is critical for performance tuning. When lag increases, Tinybird's autoscaling infrastructure automatically scales consumers to handle the load. The `kafka_ops_log` provides visibility into this autoscaling behavior, helping you understand partition assignment patterns and verify that your integration is performing optimally. For more details on consumer configuration, see the [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka).

### Find current consumer lag by partition (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#find-current-consumer-lag-by-partition)

Get the most recent lag measurement for each partition:

```
SELECT
    datasource_id,
    topic,
    partition,
    lag,
    timestamp
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
  AND partition >= 0
  AND msg_type = 'info'
ORDER BY timestamp DESC
LIMIT 1 BY datasource_id, topic, partition
```

### Consumer lag trend over time (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#consumer-lag-trend-over-time)

Track how lag changes over time to identify patterns and trends:

```
SELECT
    toStartOfHour(timestamp) as hour,
    datasource_id,
    topic,
    partition,
    max(lag) as max_lag,
    avg(lag) as avg_lag
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
  AND partition >= 0
  AND msg_type = 'info'
GROUP BY hour, datasource_id, topic, partition
ORDER BY hour DESC, max_lag DESC
```
Rising lag trends may indicate that your current consumer group configuration cannot keep up with the message rate. This is particularly useful for understanding how Tinybird's consumer autoscaling responds to varying load patterns.

### Find partitions with high lag (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#find-partitions-with-high-lag)

Identify partitions that need immediate attention:

```
SELECT
    datasource_id,
    topic,
    partition,
    max(lag) as current_lag,
    timestamp
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
  AND partition >= 0
  AND msg_type = 'info'
GROUP BY datasource_id, topic, partition, timestamp
HAVING current_lag > 10000
ORDER BY current_lag DESC
```
High lag on specific partitions may indicate uneven message distribution, where some partitions receive more messages than others. This is valuable information for partition key design and scaling decisions.

## Track throughput and performance (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#track-throughput-and-performance)

Throughput monitoring helps you understand how efficiently your connector is processing messages. This is essential for performance tuning and understanding how the autoscaling infrastructure responds to varying message rates.

### Messages processed per hour by topic (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#messages-processed-per-hour-by-topic)

Track message processing rates over time:

```
SELECT
    toStartOfHour(timestamp) as hour,
    datasource_id,
    topic,
    sum(processed_messages) as total_messages,
    sum(processed_bytes) as total_bytes,
    sum(committed_messages) as total_committed
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
  AND msg_type = 'info'
GROUP BY hour, datasource_id, topic
ORDER BY hour DESC
```
Use this to track throughput trends and identify peak processing periods. This data helps you understand how consumer autoscaling responds to load changes and supports capacity planning.

### Processing rate comparison across partitions (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#processing-rate-comparison-across-partitions)

Compare processing rates to identify bottlenecks:

```
SELECT
    datasource_id,
    topic,
    partition,
    sum(processed_messages) as total_processed,
    sum(committed_messages) as total_committed,
    count(*) as operation_count,
    sum(processed_messages) / count(*) as avg_messages_per_operation
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
  AND partition >= 0
  AND msg_type = 'info'
GROUP BY datasource_id, topic, partition
ORDER BY total_processed DESC
```
Uneven processing rates may indicate that you need to review your partition key design or adjust your partition assignment strategy.

### Throughput efficiency (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#throughput-efficiency)

Compare processed messages to committed messages to understand your ingestion success rate:

```
SELECT
    datasource_id,
    topic,
    sum(processed_messages) as processed,
    sum(committed_messages) as committed,
    (sum(committed_messages) * 100.0 / sum(processed_messages)) as success_rate_pct
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
  AND msg_type = 'info'
  AND processed_messages > 0
GROUP BY datasource_id, topic
ORDER BY success_rate_pct ASC
```
A low success rate indicates that messages are being processed but not successfully ingested into ClickHouse, which may point to schema issues, data quality problems, or configuration errors. The connector's quarantine feature automatically handles problematic messages, and error details in `kafka_ops_log` help you debug and resolve issues. Learn more about [Quarantine Data Sources](/docs/forward/guides/quarantine).

## Monitor errors and warnings (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#monitor-errors-and-warnings)

Errors and warnings can indicate issues with your Kafka cluster connectivity, deserialization problems, Materialized View errors, or configuration issues. The serverless connector includes comprehensive error handling with quarantine for problematic messages. Regular monitoring of `kafka_ops_log` helps you catch and resolve issues before they impact your data pipeline. For troubleshooting guidance, see the [Kafka connector troubleshooting guide](/docs/forward/ingest-data/connectors/kafka/troubleshooting).

### Recent errors and warnings (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#recent-errors-and-warnings)

View all errors and warnings from your connector operations:

```
SELECT
    timestamp,
    datasource_id,
    topic,
    partition,
    msg_type,
    msg,
    lag
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
  AND msg_type IN ('warning', 'error')
ORDER BY timestamp DESC
```
Review these messages to identify patterns and root causes. Common issues include partition rebalancing events, deserialization failures, and connectivity problems.

### Error count by Data Source and topic (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#error-count-by-data-source-and-topic)

Identify which Data Sources or topics are experiencing the most issues:

```
SELECT
    datasource_id,
    topic,
    msg_type,
    count(*) as error_count
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
  AND msg_type IN ('warning', 'error')
GROUP BY datasource_id, topic, msg_type
ORDER BY error_count DESC
```
This helps prioritize troubleshooting efforts and identify if specific topics or partitions are problematic.

### Group errors by message type (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#group-errors-by-message-type)

Identify recurring issues and patterns:

```
SELECT
    msg,
    msg_type,
    count(*) as occurrence_count,
    count(DISTINCT datasource_id) as affected_datasources
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 7 day
  AND msg_type IN ('warning', 'error')
  AND msg != ''
GROUP BY msg, msg_type
ORDER BY occurrence_count DESC
```
This analysis helps you understand common failure modes in your connector.

## Health checks (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#health-checks)

Regular health checks help you assess the overall status of your integration. These queries provide a comprehensive view of connector health, including consumer lag, throughput, and error rates. You can also set up automated [health checks](/docs/forward/monitoring/health-checks) for continuous monitoring.

### Connector health summary (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#connector-health-summary)

Get a comprehensive health overview of all your connectors:

```
SELECT
    datasource_id,
    topic,
    max(lag) as max_lag,
    sum(processed_messages) as total_processed,
    sum(committed_messages) as total_committed,
    countIf(msg_type = 'error') as error_count,
    countIf(msg_type = 'warning') as warning_count,
    max(timestamp) as last_activity
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
GROUP BY datasource_id, topic
ORDER BY max_lag DESC
```
Use this for a quick assessment of your connector performance and to identify connectors that need attention.

### Find inactive connectors (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#find-inactive-connectors)

Find Data Sources that haven't processed messages recently:

```
SELECT
    datasource_id,
    topic,
    max(timestamp) as last_activity,
    now() - max(timestamp) as time_since_last_activity
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 7 day
GROUP BY datasource_id, topic
HAVING time_since_last_activity > INTERVAL 1 hour
ORDER BY time_since_last_activity DESC
```
Inactive connectors may indicate that the consumer has stopped, the topic has no new messages, or there's a connectivity issue.

## Partition-level analysis (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#partition-level-analysis)

Partition analysis helps you understand how your partition assignment strategy is working and identify partitions that may need attention. This analysis helps with scaling decisions and optimizing your partition key design.

### Partition performance comparison (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#partition-performance-comparison)

Compare performance metrics across all partitions:

```
SELECT
    datasource_id,
    topic,
    partition,
    max(lag) as max_lag,
    avg(lag) as avg_lag,
    sum(processed_messages) as total_messages,
    countIf(msg_type = 'error') as errors,
    countIf(msg_type = 'warning') as warnings
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
  AND partition >= 0
GROUP BY datasource_id, topic, partition
ORDER BY max_lag DESC, errors DESC
```
This helps you understand if your partition assignment is balanced and whether you need to adjust your setup.

### Lag distribution analysis (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#lag-distribution-analysis)

Understand the distribution of consumer lag across partitions:

```
SELECT
    datasource_id,
    topic,
    quantile(0.5)(lag) as median_lag,
    quantile(0.95)(lag) as p95_lag,
    quantile(0.99)(lag) as p99_lag,
    max(lag) as max_lag,
    min(lag) as min_lag
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
  AND partition >= 0
  AND msg_type = 'info'
GROUP BY datasource_id, topic
ORDER BY max_lag DESC
```
This helps identify if lag is evenly distributed or concentrated in specific partitions. Uneven lag distribution may indicate that you need to review your partition key design or consider rebalancing.

## Alerting queries (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#alerting-queries)

Set up automated alerts based on these queries to catch issues early. These queries are designed to trigger alerts when critical conditions are met, helping you maintain reliable data pipelines. Combine these with [health checks](/docs/forward/monitoring/health-checks) for comprehensive monitoring.

**Connect to monitoring tools**: You can connect these monitoring queries to external alerting tools like Grafana, Datadog, PagerDuty, and Slack. Query the [ClickHouse HTTP interface](/docs/forward/query-data/clickhouse-interface) directly, or create API endpoints from these queries. For Prometheus-compatible tools, you can export queries in [Prometheus format](/docs/forward/guides/consume-api-endpoints-in-prometheus-format). Configure your monitoring tools to poll these queries periodically and trigger alerts when thresholds are exceeded.

### High consumer lag alert (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#high-consumer-lag-alert)

Alert when consumer lag exceeds a threshold:

```
SELECT
    datasource_id,
    topic,
    partition,
    lag,
    timestamp
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 5 minute
  AND partition >= 0
  AND msg_type = 'info'
  AND lag > 50000
ORDER BY timestamp DESC
LIMIT 1 BY datasource_id, topic, partition
```
High lag indicates that your consumer is falling behind and may need scaling. With Tinybird's consumer autoscaling, the infrastructure automatically handles increased load, but this alert helps you verify autoscaling effectiveness.

### Error rate alert (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#error-rate-alert)

Alert when the error rate exceeds a threshold:

```
SELECT
    datasource_id,
    topic,
    countIf(msg_type = 'error') as error_count,
    count(*) as total_operations,
    (countIf(msg_type = 'error') * 100.0 / count(*)) as error_rate_pct
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 15 minute
GROUP BY datasource_id, topic
HAVING error_rate_pct > 5
ORDER BY error_rate_pct DESC
```
High error rates indicate problems with your connector configuration, cluster connectivity, or data quality issues that need immediate attention.

### Processing stall alert (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#processing-stall-alert)

Alert when no messages are being processed:

```
SELECT
    datasource_id,
    topic,
    max(timestamp) as last_activity,
    sum(processed_messages) as messages_in_window
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 10 minute
GROUP BY datasource_id, topic
HAVING messages_in_window = 0
ORDER BY last_activity DESC
```
This indicates that your consumer may have stopped, there's a connectivity issue, or the topic has no new messages.

## Combine with other Service Data Sources (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#combine-with-other-service-data-sources)

You can join `tinybird.kafka_ops_log` with other [Service Data Sources](/docs/forward/monitoring/service-datasources) for deeper analysis. This supports comprehensive analysis that correlates connector operations with Data Source ingestion metrics. Learn more about all available [Service Data Sources](/docs/forward/monitoring/service-datasources#tinybird-kafka-ops-log).

### Correlate with Data Source operations (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#correlate-with-data-source-operations)

Join Kafka operations log with [Data Source operations log](/docs/forward/monitoring/service-datasources#tinybird-datasources-ops-log) to understand the complete pipeline from message consumption to ClickHouse ingestion:

```
SELECT
    k.timestamp,
    k.datasource_id,
    k.topic,
    k.partition,
    k.processed_messages,
    k.committed_messages,
    d.rows as rows_in_datasource,
    d.elapsed_time
FROM tinybird.kafka_ops_log k
LEFT JOIN (
    SELECT
        datasource_id,
        rows,
        elapsed_time,
        timestamp
    FROM tinybird.datasources_ops_log
    WHERE event_type = 'append-kafka'
      AND timestamp > now() - INTERVAL 1 hour
) d ON k.datasource_id = d.datasource_id
    AND toStartOfMinute(k.timestamp) = toStartOfMinute(d.timestamp)
WHERE k.timestamp > now() - INTERVAL 1 hour
  AND k.msg_type = 'info'
ORDER BY k.timestamp DESC
```
This helps you identify bottlenecks in the ingestion process and optimize your connector performance.

### Analyze ingestion using Kafka meta columns (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#analyze-ingestion-using-kafka-meta-columns)

Use the [Kafka meta columns](/docs/forward/ingest-data/connectors/kafka#kafka-meta-columns) in your Data Sources to analyze ingestion patterns. For example, track message gaps by analyzing offset sequences:

```
SELECT
    __partition,
    __topic,
    min(__offset) as min_offset,
    max(__offset) as max_offset,
    count(*) as message_count,
    max(__offset) - min(__offset) + 1 as expected_count,
    (max(__offset) - min(__offset) + 1) - count(*) as missing_messages
FROM your_kafka_datasource
WHERE timestamp > now() - INTERVAL 1 hour
GROUP BY __partition, __topic
HAVING missing_messages > 0
```
This helps identify if any messages were skipped during ingestion. You can also analyze distribution to verify your partition key design:

```
SELECT
    __partition,
    count(*) as message_count,
    count(DISTINCT __key) as unique_keys,
    min(__timestamp) as first_message,
    max(__timestamp) as last_message
FROM your_kafka_datasource
WHERE timestamp > now() - INTERVAL 24 hour
GROUP BY __partition
ORDER BY __partition
```
This shows how messages are distributed across partitions, helping you understand if your partition assignment is balanced.

## Schema evolution monitoring (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#schema-evolution-monitoring)

Monitor schema changes and their impact on ingestion:

```
SELECT
    timestamp,
    datasource_id,
    topic,
    msg,
    count(*) as occurrence_count
FROM tinybird.kafka_ops_log
WHERE msg_type = 'warning'
  AND (msg LIKE '%schema%' OR msg LIKE '%deserialization%')
  AND timestamp > now() - INTERVAL 24 hour
GROUP BY timestamp, datasource_id, topic, msg
ORDER BY occurrence_count DESC
```
This helps identify schema evolution issues and deserialization problems that may affect ingestion performance.

## Performance tuning best practices (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#performance-tuning-best-practices)

Based on monitoring data from `tinybird.kafka_ops_log`, follow these best practices:

- **Monitor consumer lag regularly**: Set up alerts for high lag values to catch issues early. With Tinybird's serverless architecture, consumer autoscaling automatically handles increased load, but monitoring lag helps you verify that autoscaling is working effectively.
- **Track error patterns**: Review error messages periodically to identify recurring issues. Common problems include partition assignment issues, deserialization failures, and connectivity problems.
- **Compare partition performance**: If one partition has significantly higher lag than others, investigate the cause. This may indicate uneven message distribution and the need to review your partition key design.
- **Monitor throughput trends**: Track processing rates over time to understand how the autoscaling infrastructure responds to varying message rates.
- **Use time windows**: Filter queries by recent time windows (for example, last hour, last 24 hours) to focus on current state and avoid analyzing stale data.
- **Analyze partition assignment**: Use partition-level metrics to understand if your partition assignment strategy is working effectively. Uneven partition loads may require rebalancing or adjustments to your setup.
- **Optimize partition key design**: If you see uneven lag distribution across partitions, consider reviewing your partition key design to ensure messages are distributed evenly.
For detailed performance optimization strategies, see the [performance optimization guide](/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization).

## Next steps (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#next-steps)

### Kafka connector documentation (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#kafka-connector-documentation)

- **[Kafka connector guide](/docs/forward/ingest-data/connectors/kafka)** - Complete setup, configuration, and troubleshooting guide
- **[Troubleshooting guide](/docs/forward/ingest-data/connectors/kafka/troubleshooting)** - Common errors and solutions
- **[Performance optimization](/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization)** - Throughput tuning and best practices
- **[Partitioning strategies](/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies)** - Partition optimization
- **[Schema management](/docs/forward/ingest-data/connectors/kafka/guides/schema-management)** - Schema evolution and data type mapping

### Monitoring and observability (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#monitoring-and-observability)

- **[Service Data Sources](/docs/forward/monitoring/service-datasources)** - Explore other monitoring options
- **[Health checks](/docs/forward/monitoring/health-checks)** - Automated monitoring setup
- **[Data Source operations](/docs/forward/monitoring/service-datasources#tinybird-datasources-ops-log)** - Monitor ingestion performance
- **[Workspace jobs](/docs/forward/monitoring/jobs)** - Additional pipeline observability

### Integrate with monitoring tools (https://www.tinybird.co/docs/forward/monitoring/kafka-clickhouse-monitoring#integrate-with-monitoring-tools)

Connect monitoring queries to Grafana, Datadog, PagerDuty, Slack, and other alerting systems by querying the [ClickHouse HTTP interface](/docs/forward/query-data/clickhouse-interface) directly. You can also create API endpoints from these queries, or export them in [Prometheus format](/docs/forward/guides/consume-api-endpoints-in-prometheus-format) for Prometheus-compatible tools.

---

URL: https://www.tinybird.co/docs/forward/monitoring/jobs
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Monitor jobs in your workspace \| Tinybird"
theme-color: "#171612"
description: "Many of the operations you can run in your workspace are executed using jobs. jobs_log provides you with an overview of all your jobs."
inkeep:version: "forward"
---

# Monitor jobs in your workspace (https://www.tinybird.co/docs/forward/monitoring/jobs#monitor-jobs-in-your-workspace)

Many operations in your Tinybird workspace, like imports, copy jobs, sinks and populates, run as background jobs within the platform. This approach ensures that the system can handle a large volume of requests efficiently without causing timeouts or delays in your workflow.

Monitoring and managing jobs, for example querying job statuses, types, and execution details, is essential for maintaining a healthy workspace. The two mechanisms for generic job monitoring are the [Jobs API](/docs/api-reference/jobs-api) and the <a href="./service-datasources#tinybird-jobs-log">`jobs_log`</a> data source.

You can also track more specific things using dedicated Service data sources, such as `datasources_ops_log` for import, replaces, or copy, or `sinks_ops_log` for sink operations, or tracking jobs across [Organizations](/docs/forward/core-concepts/organizations) with `organization.jobs_log`. See [Service data sources docs](./service-datasources).

The Jobs API and the `jobs_log` return identical information about job execution. However, the Jobs API has some limitations: It reports only on a single workspace, returns only 100 records, from the last 48 hours. If you want to monitor jobs outside these parameters, use the `jobs_log` data source.

## Track a specific job (https://www.tinybird.co/docs/forward/monitoring/jobs#track-a-specific-job)

The most elemental use case is to track a specific job. You can do this using the Jobs API or SQL queries.

### Jobs API (https://www.tinybird.co/docs/forward/monitoring/jobs#jobs-api-track-specific-job)

The Jobs API is a convenient way to programmatically check the status of a job. By sending a GET request, you can retrieve detailed information about a specific job. This method is particularly useful for integration into scripts or applications.

```
curl \
    -X GET "https://$TB_HOST/v0/jobs/{job_id}" \
    -H "Authorization: Bearer $TOKEN"
```
Replace `{job_id}` with the actual job ID.

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

### SQL queries (https://www.tinybird.co/docs/forward/monitoring/jobs#sql-queries-track-specific-job)

Alternatively, you can use SQL to query the `jobs_log` data source from directly within a Tinybird pipe. This method is ideal for users who are comfortable with SQL and prefer to run queries directly against the data, and then expose them with an endpoint or perform any other actions with it.

```
SELECT * FROM tinybird.jobs_log WHERE job_id='{job_id}'
```
Replace `{job_id}` with the desired job ID. This query retrieves all columns for the specified job, providing comprehensive details about its execution.

## Track specific job types (https://www.tinybird.co/docs/forward/monitoring/jobs#track-specific-job-types)

Track jobs by type to monitor and analyze all jobs of a certain category, such as all `copy` jobs. This can help you understand the performance and status of specific job types across your entire Workspace.

### Jobs API (https://www.tinybird.co/docs/forward/monitoring/jobs#jobs-api-track-specific-job-types)

You can fetch all jobs of a specific type by making a GET request against the Jobs API:

```
curl \
    -X GET "https://$TB_HOST/v0/jobs?kind=copy" \
    -H "Authorization: Bearer $TOKEN"
```
Replace `copy` with the type of job you want to track. Make sure you have set your Tinybird host (`$TB_HOST`) and authorization token (`$TOKEN`) correctly.

### SQL queries (https://www.tinybird.co/docs/forward/monitoring/jobs#sql-queries-track-specific-job-types)

Alternatively, you can run an SQL query to fetch all jobs of a specific type from the `jobs_log` data source:

```
SELECT * FROM tinybird.jobs_log WHERE job_type='copy'
```
Replace `copy` with the desired job type. This query retrieves all columns for jobs of the specified type.

## Track ongoing jobs (https://www.tinybird.co/docs/forward/monitoring/jobs#track-ongoing-jobs)

To keep track of jobs that are currently running, you can query the status of jobs in progress. This helps in monitoring the real-time workload and managing system performance.

### Jobs API (https://www.tinybird.co/docs/forward/monitoring/jobs#jobs-api-ongoing-jobs)

By making an HTTP GET request to the Jobs API, you can fetch all jobs that are currently in the `working` status:

```
curl \
    -X GET "https://$TB_HOST/v0/jobs?status=working" \
    -H "Authorization: Bearer $TOKEN"
```
This call retrieves jobs that are actively running. Ensure you have set your Tinybird host (`$TB_HOST`) and authorization token (`$TOKEN`) correctly.

### SQL queries (https://www.tinybird.co/docs/forward/monitoring/jobs#sql-queries-ongoing-jobs)

You can also use an SQL query to fetch currently running jobs from the `jobs_log` data source:

```
SELECT * FROM tinybird.jobs_log WHERE status='working'
```
This query retrieves all columns for jobs with the status `working`, allowing you to monitor ongoing operations.

## Track errored jobs (https://www.tinybird.co/docs/forward/monitoring/jobs#track-errored-jobs)

Tracking errored jobs is crucial for identifying and resolving issues that may arise during job execution. Jobs API or SQL queries to `jobs_log` helps you monitor jobs that errored during the execution.

### Jobs API (https://www.tinybird.co/docs/forward/monitoring/jobs#jobs-api-errored-jobs)

You can use the Jobs API to fetch details of jobs that have ended in error.

Use the following `curl` command to retrieve all jobs that have a status of `error`:

```
curl \
    -X GET "https://$TB_HOST/v0/jobs?status=error" \
    -H "Authorization: Bearer $TOKEN"
```
This call fetches a list of jobs that are currently in an errored state, providing details that can be used for further analysis or debugging. Make sure you've set your Tinybird host (`$TB_HOST`) and authorization token (`$TOKEN`) correctly.

### SQL queries (https://www.tinybird.co/docs/forward/monitoring/jobs#sql-queries-errored-jobs)

Alternatively, you can use SQL to query the `jobs_log` data source directly.

Use the following SQL query to fetch job IDs, job types, and error messages for jobs that have encountered errors in the past day:

```
SELECT job_id, job_type, error
FROM tinybird.jobs_log
WHERE
    status='error' AND
    created_at > now() - INTERVAL 1 DAY
```

### Track success rate (https://www.tinybird.co/docs/forward/monitoring/jobs#track-success-rate)

Extrapolating from errored jobs, you can also use `jobs_log` to calculate the success rate of your workspace jobs:

```
SELECT
    job_type,
    pipe_id,
    countIf(status='done') AS job_success,
    countIf(status='error') AS job_error,
    job_success / (job_success + job_error) as success_rate
FROM tinybird.jobs_log
WHERE
    created_at > now() - INTERVAL 1 DAY
GROUP BY job_type, pipe_id
```

## Get job execution metadata (https://www.tinybird.co/docs/forward/monitoring/jobs#get-job-execution-metadata)

In the `jobs_log` data source, there is a property called `job_metadata` that contains metadata related to job executions. This includes the execution type, manual or scheduled, for Copy and Sink jobs, or the count of quarantined rows for append operations, along with many other properties.

Use JSON functions within SQL queries to extract and analyze this metadata. You can get information about job executions directly from the `jobs_log` Data Source.

The following SQL query is an example of how to extract specific metadata fields from the `job_metadata` property, such as the import mode and counts of quarantined rows and invalid lines, and how to aggregate this data for analysis:

```
SELECT
    job_type,
    JSONExtractString(job_metadata, 'mode') AS import_mode,
    sum(simpleJSONExtractUInt(job_metadata, 'quarantine_rows')) AS quarantine_rows,
    sum(simpleJSONExtractUInt(job_metadata, 'invalid_lines')) AS invalid_lines
FROM tinybird.jobs_log
WHERE
    job_type='import' AND
    created_at >= toStartOfDay(now())
GROUP BY job_type, import_mode
```
You can build many other use cases with the properties in `job_metadata`.

## Advanced use cases (https://www.tinybird.co/docs/forward/monitoring/jobs#advanced-use-cases)

Beyond basic tracking, you can leverage the `jobs_log` data source for more advanced use cases, such as gathering statistics and performance metrics. This can help you optimize job scheduling and resource allocation.

### Get queue status (https://www.tinybird.co/docs/forward/monitoring/jobs#get-queue-status)

The following SQL query returns the number of jobs that are waiting to be executed, the number of jobs that are in progress, and how many of them are done already:

```
SELECT
    job_type,
    countIf(status='waiting') AS jobs_in_queue,
    countIf(status='working') AS jobs_in_progress,
    countIf(status='done') AS jobs_succeeded,
    countIf(status='error') AS jobs_errored
FROM tinybird.jobs_log
WHERE
    created_at > now() - INTERVAL 1 DAY
GROUP BY job_type
```

### Run time statistics grouped by type of job (https://www.tinybird.co/docs/forward/monitoring/jobs#run-time-statistics-grouped-by-type-of-job)

The following SQL query calculates the maximum, minimum, median, and p95 running time (in seconds) grouped by type of job over the past day. This helps in understanding the efficiency of different job types:

```
SELECT
    job_type,
    max(date_diff('s', started_at, updated_at)) as max_run_time_in_secs,
    min(date_diff('s', started_at, updated_at)) as min_run_time_in_secs,
    median(date_diff('s', started_at, updated_at)) as median_run_time_in_secs,
    quantile(0.95)(date_diff('s', started_at, updated_at)) as p95_run_time_in_secs
FROM tinybird.jobs_log
WHERE
    created_at > now() - INTERVAL 1 DAY
GROUP BY job_type
```

### Statistics on queue time by type of job (https://www.tinybird.co/docs/forward/monitoring/jobs#statistics-on-queue-time-by-type-of-job)

The following SQL query calculates the average queue time, in seconds, for a specific type of job over the past day. This can help in identifying bottlenecks in job scheduling:

```
SELECT
    job_type,
    max(date_diff('s', created_at, started_at)) as max_run_time_in_secs,
    min(date_diff('s', created_at, started_at)) as min_run_time_in_secs,
    median(date_diff('s', created_at, started_at)) as median_run_time_in_secs,
    quantile(0.95)(date_diff('s', created_at, started_at)) as p95_run_time_in_secs
FROM tinybird.jobs_log
WHERE
    created_at > now() - INTERVAL 1 DAY
GROUP BY job_type
```

### Get statistics on job completion rate (https://www.tinybird.co/docs/forward/monitoring/jobs#get-statistics-on-job-completion-rate)

The following SQL query calculates the success rate by job type, for example `copy`, and Pipe over the past day. This can help you assess workflow reliability by measuring job completion rates and finding potential issues:

```
SELECT
    job_type,
    pipe_id,
    countIf(status='done') AS job_success,
    countIf(status='error') AS job_error,
    job_success / (job_success + job_error) as success_rate
FROM tinybird.jobs_log
WHERE
    created_at > now() - INTERVAL 1 DAY
GROUP BY job_type, pipe_id
```

### Statistics on the amount of manual versus scheduled run jobs (https://www.tinybird.co/docs/forward/monitoring/jobs#statistics-on-the-amount-of-manual-versus-scheduled-run-jobs)

The following SQL query calculates the percentage of manual and scheduled jobs. Use the distribution of manually executed and scheduled jobs to identify unexpected on-demand jobs:

```
SELECT
    job_type,
    countIf(JSONExtractString(job_metadata, 'execution_type')='manual') AS job_manual,
    countIf(JSONExtractString(job_metadata, 'execution_type')='scheduled') AS job_scheduled
FROM tinybird.jobs_log
WHERE
    job_type='copy' AND
    created_at > now() - INTERVAL 1 DAY
GROUP BY job_type
```

## Next steps (https://www.tinybird.co/docs/forward/monitoring/jobs#next-steps)

- Read up on the <a href="./service-datasources#tinybird-jobs-log">`jobs_log` Service data source specification</a>.

---

URL: https://www.tinybird.co/docs/forward/monitoring/health-checks
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Health checks \| Tinybird"
theme-color: "#171612"
description: "Use the built-in Tinybird tools to monitor your data ingestion and API endpoint processes."
inkeep:version: "forward"
---

# Check the health of your data sources (https://www.tinybird.co/docs/forward/monitoring/health-checks#check-the-health-of-your-data-sources)

After you have fixed all the possible errors in your source files, matched the data source schema to your needs, and done on-the-fly transformations, you can start ingesting data periodically. Knowing the status of your ingestion processes helps you to keep your data clean and consistent.

## Data source summary (https://www.tinybird.co/docs/forward/monitoring/health-checks#data-source-summary)

Select a data source to see the size of the data source, the number of rows, the number of rows in the quarantine data source, and when it was last updated. The log contains details of the events for the data source, which appears as the results of the query.

## Service data sources for continuous monitoring (https://www.tinybird.co/docs/forward/monitoring/health-checks#service-data-sources-for-continuous-monitoring)

[Service data sources](./service-datasources) can help you with ingestion health checks. You can use them like any other data source in your Workspace, which means you can create API Endpoints to monitor your ingestion processes.

Querying the `tinybird.datasources_ops_log` directly, you can, for example, lists your ingest processes during the last week:

LISTING INGESTIONS IN THE LAST 7 DAYS
```
SELECT *
FROM tinybird.datasources_ops_log
WHERE toDate(timestamp) > now() - INTERVAL 7 DAY
ORDER BY timestamp DESC
```
This query calculates the percentage of quarantined rows for a given period of time:

CALCULATE % OF ROWS THAT WENT TO QUARANTINE
```
SELECT
  countIf(result != 'ok') / countIf(result == 'ok') * 100 percentage_failed,
  sum(rows_quarantine) / sum(rows) * 100 quarantined_rows
FROM tinybird.datasources_ops_log
```
The following query monitors the average duration of your periodic ingestion processes for a given data source:

CALCULATING AVERAGE INGEST DURATION
```
SELECT avg(elapsed_time) avg_duration
FROM tinybird.datasources_ops_log
WHERE datasource_id = 't_8417d5126ed84802aa0addce7d1664f2'
```
If you want to configure or build an external service that monitors these metrics, you need to create an Endpoint and raise an alert when passing a threshold. When you receive an alert, you can check the quarantine data source or the Operations log to see what's going on and fix your source files or ingestion processes.

## Monitoring API endpoints (https://www.tinybird.co/docs/forward/monitoring/health-checks#monitoring-api-endpoints)

You can use the `pipe_stats` and `pipe_stats_rt` [Service data sources](./service-datasources) to monitor the performance of your Endpoints.

Every request to a pipe is logged to `tinybird.pipe_stats_rt` and kept in this data source for the last week.

The following example Endpoint aggregates the statistics for each hour for the selected pipe.

PIPE_STATS_RT_BY_HR
```
SELECT
  toStartOfHour(start_datetime) as hour,
  count() as view_count,
  round(avg(duration), 2) as avg_time,
  arrayElement(quantiles(0.50)(duration),1) as quantile_50,
  arrayElement(quantiles(0.95)(duration),1) as quantile_95,
  arrayElement(quantiles(0.99)(duration),1) as quantile_99
FROM tinybird.pipe_stats_rt
WHERE pipe_id = 'PIPE_ID'
GROUP BY hour
ORDER BY hour
```
`pipe_stats` contains statistics about your pipe Endpoints' API calls aggregated per day using intermediate states.

PIPE_STATS_BY_DATE
```
SELECT
  date,
  sum(view_count) view_count,
  sum(error_count) error_count,
  avgMerge(avg_duration_state) avg_time,
  quantilesTimingMerge(0.9, 0.95, 0.99)(quantile_timing_state) quantiles_timing_in_millis_array
FROM tinybird.pipe_stats
WHERE pipe_id = 'PIPE_ID'
GROUP BY date
ORDER BY date
```
You can use these API endpoints to trigger alerts whenever statistics pass predefined thresholds. [Export API endpoint statistics in Prometheus format](/docs/forward/guides/consume-api-endpoints-in-prometheus-format) to integrate with your monitoring and alerting tools.

---

URL: https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Analyze Endpoint Performance \| Tinybird"
theme-color: "#171612"
description: "Learn more about how to measure the performance of your API Endpoints."
inkeep:version: "forward"
---

# Analyze the performance of your API Endpoints (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#analyze-the-performance-of-your-api-endpoints)

You can use the `pipe_stats` and `pipe_stats_rt` Service Data Sources to analyze the performance of your Endpoints. Read on to see several practical examples that show what you can do with these Data Sources.

## Knowing what to optimize (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#knowing-what-to-optimize)

Before you optimize, you need to know what to optimize. The `pipe_stats` and `pipe_stats_rt` Service Data Sources let you see how your Endpoints are performing, so you can find causes of overhead and improve performance.

These Service Data Sources provide performance data and consumption data for every single request. You can also filter and sort results by Tokens to see who is accessing your Endpoints and how often.

The difference between `pipe_stats_rt` and `pipe_stats` is that `pipe_stats` provides aggregate stats, like average request duration and total read bytes, per day, whereas `pipe_stats_rt` offers the same information but without aggregation. Every single request is stored in `pipe_stats_rt`. The examples in this guide use `pipe_stats_rt`, but you can use the same logic with `pipe_stats` if you need more than 7 days of lookback.

## Before you start (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#before-you-start)

You need a high-level understanding of Tinybird's [Service Data Sources](./service-datasources).

### Understand the core stats (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#understand-the-core-stats)

This guide focuses on the following fields in the `pipe_stats_rt` Service Data Source:

- `pipe_name` (String): Pipe name as returned in Pipes API.
- `duration` (Float): the duration in seconds of each specific request.
- `read_bytes` (UInt64): How much data was scanned for this particular request.
- `read_rows` (UInt64): How many rows were scanned.
- `token_name` (String): The name of the Token used in a particular request.
- `status_code` (Int32): The HTTP status code returned for this particular request.
- `memory_usage` (UInt64): Memory consumption by the query, in bytes. High memory usage indicates the query may not be optimized efficiently.
You can find the full schema for `pipe_stats_rt` in the [API docs](./service-datasources#tinybird-pipe-stats-rt).

The value of `pipe_name` is "query_api" in the event as it's a Query API request. The following section covers how to monitor query performance when using the Query API.

### Use the Query API with metadata parameters (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#use-the-query-api-with-metadata-parameters)

If you are using the [Query API](/docs/api-reference/query-api) to run queries in Tinybird you can still track query performance using the `pipe_stats_rt` Service Data Source. Add metadata related to the query as request parameters, as well as any existing parameters used in your query.

For example, when running a query against the Query API, use a parameter called `app_name` to track all queries from the "explorer" app. Here's an example using `curl`:

Using the metadata parameters with the Query API
```
curl -X POST \
    -H "Authorization: Bearer <PIPE:READ token>" \
    --data "% SELECT * FROM events LIMIT {{Int8(my_limit, 10)}}" \
    "https://<your_host>/v0/sql?my_limit=10&app_name=explorer"
```
When you run the following queries, use the `parameters` attribute to access those queries where `app_name` equals "explorer":

Simple Parameterized Query
```
SELECT *
FROM tinybird.pipe_stats_rt
WHERE parameters['app_name'] = 'explorer'
```

## Detect errors in your Endpoints (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#detect-errors-in-your-endpoints)

If you want to monitor the number of errors per Endpoint over the last hour, you can run the following query:

Errors in the last hour
```
SELECT
  pipe_name,
  status_code,
count() as error_count
FROM tinybird.pipe_stats_rt
WHERE status_code >= 400
AND start_datetime > now() - INTERVAL 1 HOUR
GROUP BY pipe_name, status_code
ORDER BY status_code desc
```
If you have errors, the query would return something like:

```
Pipe_a | 404 | 127
Pipe_b | 403 | 32
```
With one query, you can see in real time if your Endpoints are experiencing errors, and investigate further if so.

## Analyze the performance of Endpoints over time (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#analyze-the-performance-of-endpoints-over-time)

You can also use `pipe_stats_rt` to track how long API calls take using the `duration` field, and seeing how that changes over time. API performance is directly related to how much data you are reading per request, so if your Endpoint is dynamic, request duration varies.

For instance, it might be receiving start and end date parameters that alter how long a period is being read.

API Endpoint performance over time
```
SELECT
   toStartOfMinute(start_datetime) t,
   pipe_name,
   avg(duration) avg_duration,
   quantile(.95)(duration) p95_duration,
   count() requests
FROM tinybird.pipe_stats_rt
WHERE
   start_datetime >= {{DateTime(start_date_time, '2022-05-01 00:00:00', description="Start date time")}} AND
   start_datetime < {{DateTime(end_date_time, '2022-05-25 00:00:00', description="End date time")}}
GROUP BY t, pipe_name
ORDER BY t desc, pipe_name
```

## Monitor memory usage of Endpoints (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#monitor-memory-usage-of-endpoints)

Memory usage is an important metric for identifying inefficient queries. High memory consumption can indicate complex operations, large result sets, or unoptimized queries that need attention.

Here's an example of using `pipe_stats_rt` to find the Endpoints with the highest memory usage in the last 24 hours:

High memory usage endpoints last 24 hours
```
SELECT
   pipe_name,
   formatReadableSize(avg(memory_usage) as avg_memory) as avg_memory_usage,
   formatReadableSize(max(memory_usage)) as max_memory_usage,
   formatReadableSize(quantile(0.9)(memory_usage)) as p90_memory_usage,
   count() as request_count
FROM tinybird.pipe_stats_rt
WHERE
   start_datetime >= now() - INTERVAL 24 HOUR
GROUP BY pipe_name
ORDER BY avg_memory DESC
```
You can also identify endpoints that show high memory usage relative to the amount of data they process:

Memory efficiency analysis
```
SELECT
   pipe_name,
   formatReadableSize(avg(memory_usage)) as avg_memory_usage,
   formatReadableSize(avg(read_bytes)) as avg_read_bytes,
   avg(memory_usage) / avg(read_bytes) as memory_per_byte_ratio,
   count() as request_count
FROM tinybird.pipe_stats_rt
WHERE
   start_datetime >= now() - INTERVAL 24 HOUR
   AND read_bytes > 0
GROUP BY pipe_name
ORDER BY memory_per_byte_ratio DESC
```

## Find the endpoints that process the most data (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#find-the-endpoints-that-process-the-most-data)

You might want to find Endpoints that repeatedly scan large amounts of data. They are your best candidates for optimization to reduce time and spend.

Here's an example of using `pipe_stats_rt` to find the Endpoints that have processed the most data as a percentage of all processed data in the last 24 hours:

Most processed data last 24 hours
```
WITH (
   SELECT sum(read_bytes)
   FROM tinybird.pipe_stats_rt
   WHERE
   start_datetime >= now() - INTERVAL 24 HOUR
   ) as total,
sum(read_bytes) as processed_byte
SELECT
   pipe_id,
   quantile(0.9)(duration) as p90,
   formatReadableSize(processed_byte) AS processed_formatted,
   processed_byte*100/total as percentage
FROM tinybird.pipe_stats_rt
WHERE
   start_datetime >= now() - INTERVAL 24 HOUR
GROUP BY pipe_id
ORDER BY percentage DESC
```

### Include consumption of the Query API (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#include-consumption-of-the-query-api)

If you use Tinybird's Query API to query your Data Sources directly, you probably want to include in your analysis which queries are consuming more.

Whenever you use the Query API, the field `pipe_name` contain the value `query_api`. The actual query is included as part of the `q` parameter in the `url` field. You can modify the query in the previous section to extract the SQL query that's processing the data.

Using the Query API
```
WITH (
   SELECT sum(read_bytes)
   FROM tinybird.pipe_stats_rt
   WHERE
   start_datetime >= now() - INTERVAL 24 HOUR
   ) as total,
sum(read_bytes) as processed_byte
SELECT
   if(pipe_name = 'query_api', normalizeQuery(extractURLParameter(decodeURLComponent(url), 'q')),pipe_name) as pipe_name,
   quantile(0.9)(duration) as p90,
   formatReadableSize(processed_byte) AS processed_formatted,
   processed_byte*100/total as percentage
FROM tinybird.pipe_stats_rt
WHERE
   start_datetime >= now() - INTE
RVAL 24 HOUR
GROUP BY pipe_name
ORDER BY percentage DESC
```

## Monitor usage of Tokens (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#monitor-usage-of-tokens)

If you use your Endpoint with different Tokens, for example if allowing different customers to check their own data, you can track and control which Tokens are being used to access these endpoints.

The following example shows, for the last 24 hours, the number and size of requests per Token:

Token usage last 24 hours
```
SELECT
   count() requests,
   formatReadableSize(sum(read_bytes)) as total_read_bytes,
   token_name
FROM tinybird.pipe_stats_rt
WHERE
   start_datetime >= now() - INTERVAL 24 HOUR
GROUP BY token_name
ORDER BY requests DESC
```
To get this information, request the Token name (`token_name` column) or id (`token` column).

## Next steps (https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance#next-steps)

- Learn how to [monitor jobs in your Workspace](./jobs).
- Monitor the [latency of your Endpoints](./latency).

---

URL: https://www.tinybird.co/docs/forward/ingest-data/table-functions
Last update: 2026-07-31T11:10:11.000Z
Content:
---
title: "Table functions \| Tinybird"
theme-color: "#171612"
description: "Ingest data from remote databases into Tinybird."
inkeep:version: "forward"
---

# Table functions (https://www.tinybird.co/docs/forward/ingest-data/table-functions#table-functions)

Tinybird table functions allow you to read data from an existing database and schedule a regular copy pipe to orchestrate synchronization. You can load full tables or incrementally sync your data.

Tinybird supports the following table functions:

- [Apache Iceberg](/docs/forward/ingest-data/table-functions/iceberg)
- [MySQL](/docs/forward/ingest-data/table-functions/mysql)
- [PostgreSQL](/docs/forward/ingest-data/table-functions/postgresql)
- [URL](/docs/forward/ingest-data/table-functions/url)

## Prerequisites (https://www.tinybird.co/docs/forward/ingest-data/table-functions#prerequisites)

Your database needs to be open and public, exposed to the internet with publicly signed certs, so you can connect to it by passing the hostname, port, username, and password.

Store credentials as Tinybird secrets. Do not paste usernames, passwords, or access keys as plain text in pipe SQL or datafiles.

```
tb secret set DB_USERNAME <username>
tb secret set DB_PASSWORD <password>
```
Set the secret in each environment where the copy pipe runs. For example, use `tb --cloud secret set` for Cloud and `tb --branch=<branch_name> secret set` for a branch.

Reference those secrets in the table function with `tb_secret()`:

```
{{ tb_secret("DB_USERNAME") }}
{{ tb_secret("DB_PASSWORD") }}
```
In Tinybird Classic, use the [Environment Variables API](/docs/api-reference/environment-variables-api#post-v0secrets) to create the same secret values.

## Load an external table (https://www.tinybird.co/docs/forward/ingest-data/table-functions#load-an-external-table)

Create a new pipe node. Call the table function in the `FROM` and pass the connection details:

PostgreSQL table function example
```
SELECT *
FROM postgresql(
  'postgresql.example.com:5432',
  'postgres',
  'orders',
  {{ tb_secret("DB_USERNAME") }},
  {{ tb_secret("DB_PASSWORD") }}
)
```
Publish this node as a Copy Pipe. You can append only new data or replace all data.

### Using datafiles (https://www.tinybird.co/docs/forward/ingest-data/table-functions#using-datafiles)

You can also define node logic in [Pipe files](/docs/forward/dev-reference/datafiles/pipe-files). For example, this `orders_backfill` pipe reads a PostgreSQL `orders` table and replaces the target Data Source on demand:

```
NODE all_orders
SQL >

    %
    SELECT *
    FROM postgresql(
      'postgresql.example.com:5432',
      'postgres',
      'orders',
      {{ tb_secret("DB_USERNAME") }},
      {{ tb_secret("DB_PASSWORD") }}
    )

TYPE COPY
TARGET_DATASOURCE orders
COPY_SCHEDULE @on-demand
COPY_MODE replace
```

## Include filters (https://www.tinybird.co/docs/forward/ingest-data/table-functions#include-filters)

You can use a source column to filter by a value in Tinybird, for example:

Example copy pipe with PostgreSQL table function and filters
```
SELECT *
FROM postgresql(
  'postgresql.example.com:5432',
  'postgres',
  'orders',
  {{ tb_secret("DB_USERNAME") }},
  {{ tb_secret("DB_PASSWORD") }}
)
WHERE orderDate > (SELECT max(orderDate) FROM orders)
```

## Schedule runs (https://www.tinybird.co/docs/forward/ingest-data/table-functions#schedule-runs)

When publishing as a copy pipe, most users set it to run at a frequent interval using a cron expression.

You can also trigger the copy pipe on demand:

```
curl -H "Authorization: Bearer <PIPE:READ token>" \
    -X POST "https://<TB_HOST_URL>/v0/pipes/<pipe_id>/run"
```
Having on-demand pipes in your workspace is helpful, as you can run a full sync manually any time you need it. You might also use them for weekly full syncs.

## Synchronization strategies (https://www.tinybird.co/docs/forward/ingest-data/table-functions#synchronization-strategies)

When copying data from your database to Tinybird, you can use one of the following strategies:

- Use `COPY_MODE replace` to synchronize small dimensions tables, up to a few million rows, in a frequent schedule (1 to 5 minutes).
- Use `COPY_MODE append` to do incremental appends. For example, you can append events data tagged with a timestamp. Combine it with `COPY_SCHEDULE` and filters in the copy pipe SQL to sync the new events.

### Timeouts (https://www.tinybird.co/docs/forward/ingest-data/table-functions#timeouts)

When synchronizing dimensions tables with `COPY_MODE replace` and 1 minute schedule, the copy job might time out because it can't ingest the whole table in the defined schedule.

Timeouts depend on several factors:

- The timeout configured in your external database.
- The external database load.
- Network connectivity, for example when copying data from different cloud regions.
Follow these tips to avoid timeouts using incremental appends:

- Make sure to tag your data with an updated timestamp and use the column to filter the copy pipe SQL.
- Configure the Copy Pipe with an incremental append strategy and 1 minute schedule. This configuration ingests only new records from the last minute, optimizing the copy job duration.
- Create an index in the external table to speed up filtering.
- Create the target data source as a ReplacingMergeTree using a unique or primary key as the `ENGINE_SORTING_KEY`. Rows with the same `ENGINE_SORTING_KEY` are deduplicated. Remember to use the `FINAL` keyword when querying the data source to force deduplication at query time.
- Combine this approach with an hourly or daily replacement to get rid of deleted rows.

## Observability (https://www.tinybird.co/docs/forward/ingest-data/table-functions#observability)

Job executions are logged in the `datasources_ops_log` [Service Data Source](/docs/forward/monitoring/service-datasources). You can check this log directly in the Data Source view page in the UI. Filter by `datasource_id` to monitor ingestion through the table functions from the `datasources_ops_log`:

Example query to the datasources_ops_log Service data source
```
SELECT
  timestamp,
  event_type,
  result,
  error,
  job_id
FROM
  tinybird.datasources_ops_log
WHERE
  datasource_id = 't_1234'
AND
  event_type = 'copy'
ORDER BY timestamp DESC
```

## Limits (https://www.tinybird.co/docs/forward/ingest-data/table-functions#limits)

The table functions inherit all the [limits of copy pipes](/docs/forward/pricing/limits#copy-pipe-limits).

Check the [limits page](/docs/forward/pricing/limits) for limits on ingestion, queries, API Endpoints, and more.

## Billing (https://www.tinybird.co/docs/forward/ingest-data/table-functions#billing)

There are no additional or specific costs for the table function itself. Only the costs associated with Copy Pipes apply. For more information on data operations and their charges, see [Developer](/docs/forward/pricing/developer), [SaaS](/docs/forward/pricing/saas), and [Enterprise](/docs/forward/pricing/enterprise).

---

URL: https://www.tinybird.co/docs/forward/ingest-data/local-file
Last update: 2026-08-14T13:56:08.000Z
Content:
---
title: "Ingest data from files \| Tinybird"
theme-color: "#171612"
description: "Learn how to ingest data from files to Tinybird."
inkeep:version: "forward"
---

# Ingest data from files (https://www.tinybird.co/docs/forward/ingest-data/local-file#ingest-data-from-files)

You can ingest data from files to Tinybird using the [Data sources API](/docs/api-reference/datasource-api), the [tb datasource](/docs/forward/dev-reference/commands/tb-datasource) CLI command, the TypeScript SDK, or the Python SDK. [Ingestion limits](/docs/forward/pricing/limits#ingestion-limits) apply.

## Supported file types (https://www.tinybird.co/docs/forward/ingest-data/local-file#supported-file-types)

Tinybird supports these file types and compression formats at ingest time:

| File type | Method | Accepted extensions | Compression formats supported |
| --- | --- | --- | --- |
| CSV | File upload, URL | `.csv`, `.csv.gz` | `gzip` |
| NDJSON | File upload, URL, Events API | `.ndjson`, `.ndjson.gz` | `gzip` |
| Parquet | File upload, URL | `.parquet`, `.parquet.gz` | `gzip` |
| Avro | Kafka |  | `gzip` |

## Analyze the schema of a file (https://www.tinybird.co/docs/forward/ingest-data/local-file#analyze-the-schema-of-a-file)

Before you upload data from a file or create a data source, you can analyze the schema of the file. Tinybird infers column names, types, and JSONPaths. This is helpful to identify the most appropriate data types for your columns. See [Data types](/docs/sql-reference/data-types).

The following examples show how to analyze a local NDJSON file.

- Tinybird CLI
- Analyze API

```
tb datasource analyze local_file.ndjson
```

## Append data from a file (https://www.tinybird.co/docs/forward/ingest-data/local-file#append-data-from-a-file)

You can append data from a local or remote file to a data source in Tinybird Local or Tinybird Cloud.

Use `tb datasource append` in the CLI, the `/append` endpoint in the Data Sources API, `append` in the TypeScript SDK, or `append` in the Python SDK.

### Append from a local file (https://www.tinybird.co/docs/forward/ingest-data/local-file#append-from-a-local-file)

- Tinybird CLI
- cURL
- TypeScript SDK
- Python SDK

```
tb --cloud datasource append <data_source_name> local_file.csv
```

### Append from a remote file (https://www.tinybird.co/docs/forward/ingest-data/local-file#append-from-a-remote-file)

Use `POST /v1/datasources/{name}/append` with the remote file URL and its format as query parameters. Supported formats are `csv`, `ndjson`, and `parquet`. Send an empty request body.

- Tinybird CLI
- cURL
- TypeScript SDK
- Python SDK

```
tb --cloud datasource append <data_source_name> http://example_url/file.csv
```
When appending CSV files, you can improve performance by excluding the CSV Header line. However, in this case, make sure the CSV columns are ordered. If you can't guarantee the order of columns in your CSV, include the CSV header.

## Replace data from a file (https://www.tinybird.co/docs/forward/ingest-data/local-file#replace-data-from-a-file)

You can replace existing all data or a selection of data in a data source with the contents of a file. You can replace with data from local or remote files.

When using the `/replace` endpoint with an S3 URL, you must use a pre-signed URL. Replacing data is a multi-step process that passes the URL to a background worker with no access to your S3 Connector credentials. Generate a pre-signed URL programmatically using the AWS SDK or CLI before passing it to the API.

Use `tb datasource replace` in the CLI, the `/replace` endpoint in the Data Sources API, `replace` in the TypeScript SDK, or `replace` in the Python SDK.

### Replace from a local file (https://www.tinybird.co/docs/forward/ingest-data/local-file#replace-from-a-local-file)

- Tinybird CLI
- cURL
- TypeScript SDK
- Python SDK

```
tb --cloud datasource replace <data_source_name> local_file.csv
```

### Replace from a remote file (https://www.tinybird.co/docs/forward/ingest-data/local-file#replace-from-a-remote-file)

Use `POST /v1/datasources/{name}/replace` with the remote file URL and its format as query parameters. Send an empty request body.

- Tinybird CLI
- cURL
- TypeScript SDK
- Python SDK

```
tb --cloud datasource replace <data_source_name> http://example_url/file.csv
```

## Replace data based on conditions (https://www.tinybird.co/docs/forward/ingest-data/local-file#replace-data-based-on-conditions)

Instead of replacing all data, you can also replace specific partitions of data. To do this, you define an SQL condition that describes the filter that's applied. All matching rows are deleted before finally ingesting the new file. Only the rows matching the condition are ingested.

Replacements are made by partition, so make sure that the condition filters on the partition key of the data source. If the source file contains rows that don't match the filter, the rows are ignored.

Conditional replace is supported in the CLI and the Data Sources API. The TypeScript SDK and Python SDK `replace` methods don't currently support replace conditions.

### Replace from a local file with a condition (https://www.tinybird.co/docs/forward/ingest-data/local-file#replace-from-a-local-file-with-a-condition)

- Tinybird CLI
- cURL

```
tb --cloud datasource replace <data_source_name> local_file.csv --sql-condition "my_partition_key > 123"
```

### Replace from a remote file with a condition (https://www.tinybird.co/docs/forward/ingest-data/local-file#replace-from-a-remote-file-with-a-condition)

- Tinybird CLI
- cURL

```
tb --cloud datasource replace <data_source_name> http://example_url/file.csv --sql-condition "my_partition_key > 123"
```
All the dependencies of the data source are recalculated so that your data is consistent after the replacement. If you have n-level dependencies, they're also updated by this operation.

Although replacements are atomic, Tinybird can't assure data consistency if you continue appending data to any related data source at the same time the replacement takes place. The new incoming data is discarded.

---

URL: https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingestion protection \| Tinybird"
theme-color: "#171612"
description: "How Tinybird prevents ingestion overload, ensures fair resource usage, and protects your data pipelines."
inkeep:version: "forward"
---

# Ingestion protection (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#ingestion-protection)

Tinybird provides **ingestion flow control mechanisms** to prevent resource saturation and ensure fair usage across shared infrastructure. These mechanisms protect your ingestion pipelines from causing general service degradation or even data loss, maintaining overall system health and reliability.

When needed, these mechanisms may **temporarily delay ingestion** (reducing data freshness) or **temporarily reject ingestion requests**. In all cases, Tinybird monitors these events closely and sends you **notifications** so that you can take timely corrective action.

## Understanding the challenges of data ingestion (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#understanding-the-challenges-of-data-ingestion)

[ClickHouse®](https://clickhouse.com/docs/en) is not optimized for high-frequency writes. Frequent inserts can saturate system resources, but many real-time use cases benefit from near-continuous data ingestion. [Asynchronous Inserts](https://clickhouse.com/docs/optimize/asynchronous-inserts) can help in some cases, but they come with important limitations, including potential issues with Materialized Views and the risk of data duplication.

Tinybird goes beyond the base capabilities of ClickHouse® to offer a **complete data ingestion solution**. You can send high-frequency data through:

- **The Events API** [Reference](events-api)
- **Connectors** [Kafka ingestion](connectors/kafka)
To make the most of these services, it’s important to:

- Understand your ingestion patterns.
- Plan your throughput and resource requirements.
- Monitor and act when ingestion begins to saturate your current [plan’s limits](/docs/forward/pricing/limits).
Use Tinybird's **High-Frequency Ingestion (HFI)** platform and **backpressure mechanisms** to preserve system stability in dedicated and shared environments.

## How frequent ingestion works (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#how-frequent-ingestion-works)

To avoid overwhelming the ClickHouse® clusters, Tinybird gathers incoming insertions over a short period known as the **flush interval** before writing (flushing) them into the ClickHouse® tables that back your landing Data Sources.

- The **flush interval** duration depends on your workspace’s billing plan.
- A **longer flush interval** reduces cluster load but increases data freshness latency.
When ingestion flushes occur, data is written **separately for each partition** of the destination table. Therefore:

- A high number of partitions per insertion can still cause ingestion bottlenecks.
- It’s important to <a href="/docs/sql-reference/engines/mergetree">**define partitioning and insertion strategies carefully**</a> to avoid excessive pressure on the ingestion system.

## When writes fail (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#when-writes-fail)

Ingestion into a landing Data Source may fail for multiple reasons, including:

- Cluster saturation
- Network connectivity problems
- Timeout errors
- SQL execution errors
Tinybird classifies errors as either:

- **Retriable** — temporary issues that can be retried later.
- **Non-retriable** — permanent issues such as invalid data or SQL errors.

### Retriable failures (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#retriable-failures)

Data affected by retriable errors is stored in a **staging area** and automatically retried later. During this process:

- You may experience ingestion delays.
- Tinybird’s ingestion logic avoids duplicates caused for partially inserted data for up to **5 hours**. After this 5-hour window, de-duplication is not guaranteed.

### Non-retriable failures (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#non-retriable-failures)

Data affected by non-retriable errors is kept in the **quarantine table** associated with your Data Source. You can inspect these records and take corrective action manually.

**Be aware of failures**

- When using the **Events API**, make use of the `wait` parameter to [obtain synchronous confirmation of every write](events-api#write-acknowledgements)
- Access the [Services data sources](../monitoring/service-datasources) to detect problems

## Backpressure and flow control (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#backpressure-and-flow-control)

Tinybird’s **backpressure mechanisms** preserve the health of your ingestion infrastructure by isolating problematic Data Sources before they affect others.

When a particular Data Source causes excessive resource consumption, for example, due to:

- Inefficient Materialized Views
- Poor partitioning configuration
- Long-running queries
— the system automatically applies backpressure in two escalating phases.

### 1. Delayed ingestion (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#1-delayed-ingestion)

The first line of defense is **delayed ingestion**. When backpressure is activated, insertions are temporarily delayed and resumed when resource consumption returns to safe levels.

Tinybird continuously monitors:

- The number of delayed insertions
- The age of pending insertions
If these metrics increase beyond safe limits, the system escalates protection to the next phase.

#### Typical triggers for delayed ingestion: (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#typical-triggers-for-delayed-ingestion)

- High **memory usage**
- Excessive **throughput**, exceeding cluster capacity
Throughput overload can result not only from ingestion frequency but also from:

> - Large payload sizes
> - Long insert durations, for example due to complex Materialized Views
> - High partition counts
> - Your cluster’s capacity, defined by your billing plan, which determines when these limits are reached.
Whenever possible, Tinybird applies the delay **only to the affected Data Source**, minimizing the impact on your workspace or cluster.

During delayed ingestion:

- Data is accepted, but its insertion into landing tables is **postponed**.
- You’ll see **reduced data freshness** (similar to a longer flush interval).
- If the issue is transient, ingestion recovers automatically.
You receive an **email notification** whenever delayed ingestion activates. Always review these notifications and take corrective action if necessary.

### 2. Temporary ingestion restriction (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#2-temporary-ingestion-restriction)

If ingestion delays persist and accumulated data grows beyond safe thresholds, Tinybird temporarily **rate limits ingestion** for the affected Data Source (by applying a temporary rate limit of 1 request/message per second for one minute).

These restrictions can also apply when **CPU usage** exceeds plan limits.

When this occurs:

- **Events API** calls return HTTP **429** responses with details about the cause.
- Tinybird pauses **Connectors** temporarily and automatically attempts recovery.
You’ll receive an **email alert** immediately when ingestion is denied.

This temporary service denial prevents cascading issues that could otherwise impact system stability. In most cases, ingestion resumes automatically once the issue is resolved.

If the problem persists:

- Review the metrics and logs associated with your ingestion workload.
- Check for inefficient queries, heavy partitioning, or high system load.
- Contact <a href="../support">**Tinybird Support**</a> if you cannot identify the root cause.
In rare cases, ingestion protection may be triggered by issues unrelated to your ingestion, such as cloud provider disruptions. Tinybird monitoring systems detect and remediate such situations as quickly as possible.

## Best practices (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#best-practices)

To reduce the likelihood of backpressure activation:

- Design **efficient partitioning strategies**.
- Optimize **Materialized Views** and avoid long-running transformations during ingestion.
- Batch inserts where possible rather than sending individual events. Use the appropriate ingestion mechanism for your scenario. For large, infrequent writes, you can ingest [data from files](local-file) or [from S3](connectors/s3).
- Monitor ingestion metrics regularly.
- Pay attention to **flush intervals**, **notifications**, and **system alerts**.
- Choose the right **billing plan** for your ingestion needs, and consider upgrading when ingestion volume or concurrency approaches the limits of your current plan.

## Summary (https://www.tinybird.co/docs/forward/ingest-data/ingestion-protection#summary)

Tinybird’s ingestion protection mechanisms ensure:

- **Fair usage** across users and workspaces
- **Reliability** of ingestion pipelines
- **Automatic recovery** from transient issues
- **Transparent communication** through alerts and monitoring
These mechanisms combine **delayed ingestion**, **automatic retries**, and **temporary rate limiting** to keep your ingestion stable, efficient, and predictable.

**See also:**

- [Events API Reference](events-api)
- [Kafka Connector Guide](connectors/kafka)
- [Quarantine Data Sources](../core-concepts/data-sources#quarantine-data-sources)

---

URL: https://www.tinybird.co/docs/forward/ingest-data/events-api
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Send events \| Tinybird"
theme-color: "#171612"
description: "Send JSON and NDJSON data to Tinybird by calling the Events API."
inkeep:version: "forward"
---

# Send events (https://www.tinybird.co/docs/forward/ingest-data/events-api#send-events)

You send events to a [data source](/docs/forward/core-concepts/data-sources) using the Events API or the [tb datasource append](/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-append) CLI command with the `--events` flag. [Ingestion limits](/docs/forward/pricing/limits#ingestion-limits) apply.

## Send JSON events (https://www.tinybird.co/docs/forward/ingest-data/events-api#send-json-events)

You can send individual events, but microbatching several events usually gives you much higher throughput. Use NDJSON with cURL or the CLI, `ingestBatch` in the TypeScript SDK, or `ingest_batch` in the Python SDK.

- cURL
- Tinybird CLI
- TypeScript SDK
- Python SDK
Send NDJSON events
```
curl \
-H "Authorization: Bearer <import_token>" \
-d $'{"date": "2020-04-05", "city": "Chicago"}\n{"date": "2020-04-05", "city": "Madrid"}\n' \
'https://api.tinybird.co/v0/events?name=<data_source_name>'
```
NDJSON means each JSON event is separated by a newline `\n` character. The SDK batch methods accept arrays of JSON objects and handle the Events API request for you.

## Token (https://www.tinybird.co/docs/forward/ingest-data/events-api#token)

The token you use to send events must have the `DATASOURCE:APPEND` scope. This should be defined in your data source file using `TOKEN {token_name} append`. For more details, see [resource-scoped tokens](/docs/forward/core-concepts/static-tokens#resource-scoped-tokens).

## Limits (https://www.tinybird.co/docs/forward/ingest-data/events-api#limits)

The Events API delivers a default capacity of:

- Up to 100 requests per second per data source
- Up to 10 MB per request per Data Source on Free plans and up to 100 MB on Developer, SaaS, and Enterprise plans
If you reach this limit, you receive an `HTTP 413 - Request Entity Too Large` error. Split larger requests into multiple smaller ones to ensure smooth data ingestion.

### Rate limit headers (https://www.tinybird.co/docs/forward/ingest-data/events-api#rate-limit-headers)

The Events API returns the following headers with the response:

| Header Name | Description |
| --- | --- |
| `X-RateLimit-Limit` | The maximum number of requests you're permitted to make in the current limit window. |
| `X-RateLimit-Remaining` | The number of requests remaining in the current rate limit window. |
| `X-RateLimit-Reset` | The time in seconds after the current rate limit window resets. |
| `Retry-After` | The time to wait before making a another request. Only present on 429 responses. |

### Check the payload size (https://www.tinybird.co/docs/forward/ingest-data/events-api#check-the-payload-size)

To avoid hitting the request limit size, you can check your payload size before sending. For example:

- shell
- python
- js

```
echo '{"date": "2020-04-05", "city": "Chicago"}' | wc -c | awk '{print $1/1024/1024 " MB"}'
```

## Compress the data you send (https://www.tinybird.co/docs/forward/ingest-data/events-api#compress-the-data-you-send)

You can compress the data you send to the Events API using Gzip. Compressing events adds overhead to the ingestion process, which can introduce latency, although it's typically minimal.

Here is an example of sending a JSON event compressed with Gzip from the command line:

```
echo '{"timestamp":"2022-10-27T11:43:02.099Z","transaction_id":"8d1e1533-6071-4b10-9cda-b8429c1c7a67","name":"Bobby Drake","email":"bobby.drake@pressure.io","age":42,"passport_number":3847665,"flight_from":"Barcelona","flight_to":"London","extra_bags":1,"flight_class":"economy","priority_boarding":false,"meal_choice":"vegetarian","seat_number":"15D","airline":"Red Balloon"}' | gzip > body.gz

curl \
    -X POST 'https://<your_host>/v0/events?name=gzip_events_example' \
    -H "Authorization: Bearer <AUTH_TOKEN>" \
    -H "Content-Encoding: gzip" \
    --data-binary @body.gz
```

## Write operation acknowledgements (https://www.tinybird.co/docs/forward/ingest-data/events-api#write-acknowledgements)

When you send data to the Events API, you usually receive an `HTTP202` response, which indicates that the request was successful, although it doesn't confirm that the data has been committed into the underlying database. This is useful when guarantees on writes aren't strictly necessary.

Typically, it takes under two seconds to receive a response from the Events API. For example:

```
curl \
    -X POST 'https://<your_host>/v0/events?name=events_example' \
    -H "Authorization: Bearer <AUTH_TOKEN>" \
    -d $'{"timestamp":"2022-10-27T11:43:02.099Z"}'

< HTTP/2 202
< content-type: application/json
< content-length: 42
<
{"successful_rows":2,"quarantined_rows":0}
```
If your use case requires absolute guarantees that data is committed, use the `wait` parameter. The `wait` parameter is a boolean that accepts a value of `true` or `false`. A value of `false` is the default behavior, equivalent to omitting the parameter entirely.

Using `wait=true` with your request asks the Events API to wait for acknowledgement that the data you sent has been committed into the underlying database. You then receive an `HTTP200` response that confirms data has been committed.

Adding `wait=true` to your request can result in a slower response time. Use a timeout of at least 10 seconds when waiting for the response. For example:

```
curl \
    -X POST 'https://<your_host>/v0/events?name=events_example&wait=true' \
    -H "Authorization: Bearer <AUTH_TOKEN>" \
    -d $'{"timestamp":"2022-10-27T11:43:02.099Z"}'

< HTTP/2 200
< content-type: application/json
< content-length: 42
<
{"successful_rows":2,"quarantined_rows":0}
```
Log your requests and responses from and to the Events API. This helps you get visibility into any failures.

---

URL: https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Using Cloud Branches \| Tinybird"
theme-color: "#171612"
description: "Use Tinybird Cloud Branches in your development workflow."
inkeep:version: "forward"
---

# Using Cloud Branches (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#using-cloud-branches)

Use Cloud Branches to build and test changes in Tinybird Cloud before deploying them to your main Workspace. The workflow is the same one used in the quickstarts: configure the project, build in a branch, test with branch data, then deploy.

Use Cloud Branches when you need production-like data, connector validation, shared preview URLs, or pull request review. Use [Tinybird Local](/docs/forward/development-workflow/local-development) when fixture data is enough and you want a fully local loop.

For the concept model, see [Branches](/docs/forward/core-concepts/branches). Branches are not promoted to production. You validate changes in a branch, then deploy the project to your main Tinybird Cloud Workspace with `tb deploy`, an SDK deploy command, or CI/CD.

## Check your project config (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#check-your-project-config)

Your project config defines which files Tinybird builds and which development environment commands use by default.

Use `include` to point at your datafiles or SDK resource definitions. Set `dev_mode` or `devMode` to `branch` when you want `build` to target Cloud Branches by default. Set `branch_data_mode` to control the production data attached when `build` auto-creates a Cloud Branch. You can still override the target with `--branch` or `--local` on the command line.

- Tinybird CLI
- TypeScript SDK
- Python SDK
tinybird.config.json
```
{
  "dev_mode": "branch",
  "branch_data_mode": "last_partition",
  "include": [
    "tinybird"
  ]
}
```

## Typical branch workflow (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#typical-branch-workflow)

1. Set the development environment to Cloud Branches in `tinybird.config.json`.
2. Create a Git branch.
3. Build the project in a matching Tinybird Cloud Branch.
4. Edit Data Sources, Pipes, Endpoints, Tokens, or Connections.
5. Rebuild and test the branch with fixture data, production-like data, or connector data.
6. Run a deployment check against the main Cloud Workspace.
7. Merge and deploy.

```
git checkout -b add-plan-to-events
tb build
```
With `dev_mode` or `devMode` set to `branch`, `tb build` uses your Git branch name as the Tinybird Cloud Branch name. If the Cloud Branch does not exist, Tinybird creates it using `branch_data_mode` from `tinybird.config.json`. By default, that attaches the latest production partition. Run `tb build` again after editing project files.

For commands other than `build`, pass the branch name explicitly with `--branch=<git-branch-name>`.

```
Using dev_mode=branch. Running build against Tinybird branch 'add-plan-to-events'.
Running against Tinybird Cloud: Workspace production | branch add-plan-to-events
Creating branch 'add-plan-to-events' with data_mode=last_partition...
» Git branch:            add-plan-to-events
» Tinybird Cloud branch: add-plan-to-events ✓ created
» Building project...
✓ datasources/events.datasource created
✓ Build completed in 6.2s
```

## Build in a branch (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#build-in-a-branch)

Run a build before testing changes. If your project is in Git, Tinybird creates or reuses a Cloud Branch with the same name as your Git branch.

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
git checkout -b add-plan-to-events
tb build
```
When you edit project files, run build again to apply the changes to the Cloud Branch.

## Bring production-like data into a branch (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#bring-production-like-data-into-a-branch)

When `dev_mode=branch`, `tb build` auto-creates Cloud Branches with the latest production partition by default. Configure this behavior in `tinybird.config.json` with `branch_data_mode`:

tinybird.config.json
```
{
  "dev_mode": "branch",
  "branch_data_mode": "last_partition"
}
```
This attaches the latest production partition to the branch without copying the underlying data. Writes, deletes, and schema changes stay isolated in the branch.

Use production-like branch data for:

- Schema changes that need real values.
- Endpoint changes where fixture data is not enough.
- Query performance checks on realistic partition sizes.
- Review apps that need production-like responses.
Do not use branch data as a backup or long-lived replica. Branches are short-lived validation environments.

When you create a branch manually with `tb branch create`, pass `--last-partition` to attach production data. Branches created without that flag copy Workspace resources, not data:

```
tb branch create add-plan-to-events --last-partition
tb --branch=add-plan-to-events build
```

## Query and call branch resources (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#query-and-call-branch-resources)

Run SQL against the branch:

```
tb --branch=add-plan-to-events sql "SELECT plan, count() FROM events GROUP BY plan"
```
Open Tinybird UI for the branch:

```
tb --branch=add-plan-to-events open
```
The URL is `https://cloud.tinybird.co/<cloud-provider>/<region>/<workspace-name>~<branch-name>`.

Call a branch Endpoint with a branch Token:

```
curl -H "Authorization: Bearer <branch-token>" \
  "https://api.tinybird.co/v0/pipes/events_by_plan.json"
```

## Test connector data (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#test-connector-data)

Use `--with-connections` when your branch needs Kafka, S3, or GCS connector Data Sources.

```
tb build --with-connections
```

### Kafka (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#kafka)

Kafka ingestion starts stopped in branches. Start it only when you are ready to test:

```
tb --branch=add-plan-to-events datasource start my_kafka_datasource
```
Stop it when you finish:

```
tb --branch=add-plan-to-events datasource stop my_kafka_datasource
```
Each start creates a new consumer group for the branch and starts from the latest offset. This avoids collisions with production and other branches.

### S3 and GCS (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#s3-and-gcs)

Import a sample instead of syncing the whole bucket:

```
tb --branch=add-plan-to-events datasource sample my_s3_datasource --wait
```
Sample imports are separate jobs and do not affect production sync state.

## Validate before deploy (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#validate-before-deploy)

Before merging, run the checks that match the change. For schema or connector changes, include a Cloud deployment check.

```
tb build
tb test run
tb --cloud deploy --check
```
For SDK projects, use the matching SDK commands:

- TypeScript SDK
- Python SDK

```
npx tinybird build
npx tinybird deploy --check
```

## Preview branches in CI (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#preview-branches-in-ci)

Use preview branches for pull requests. They let reviewers and preview deployments call the changed Tinybird project before it is merged.

For TypeScript SDK projects:

```
npx tinybird preview --check
```
For Python SDK projects:

```
uv run tinybird preview --check
```
For datafile projects, create and build a branch explicitly:

```
tb branch create tmp_ci_my_feature --last-partition
tb --branch=tmp_ci_my_feature build
```
See [CI/CD](/docs/forward/development-workflow/cicd#preview-deployments).

## Work without Git (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#work-without-git)

If your project is not in a Git repository, pass the branch name explicitly to `build` too:

```
tb --branch=add-plan-to-events build
```

## Clean up branches (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#clean-up-branches)

Branches are meant to be temporary. Delete them after the pull request is merged or closed.

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb branch rm preview_1
```

## When to use Local instead (https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches#when-to-use-local-instead)

Use [Tinybird Local](/docs/forward/core-concepts/tinybird-local) instead of Cloud Branches when fixture data is enough, you need a fully local loop, or you want CI checks that don't modify Cloud resources.

---

URL: https://www.tinybird.co/docs/forward/development-workflow/manual-deployment
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Manual deployment \| Tinybird"
theme-color: "#171612"
description: "Deploy your Tinybird project to Tinybird Cloud using the CLI."
inkeep:version: "forward"
---

# Manual deployment (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#manual-deployment)

You can deploy your data projects to Tinybird Cloud directly from the command line using the [Tinybird CLI](/docs/forward/dev-reference/commands).

If your project was initialized with `tb init --dev-mode local` or `tb init --dev-mode branch`, running `tb deploy` without explicit environment flags targets your cloud main workspace automatically.

## What is a deployment? (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#what-is-a-deployment)

Deployments are versions of your project resources and data running on local or cloud infrastructure.

There are two types of deployments:

- **Staging deployments**: Deployments you can use to validate your changes. You access them using the `--staging` flag.
- **Live deployments**: Deployments that make your changes available to your users.
Each type can be deployed to Tinybird Local (`--local`) or Tinybird Cloud (`--cloud`).

### Deployment status (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#deployment-status)

Deployments have the following statuses:

- `In progress`: The deployment is in progress. Use `--wait` to wait for it to finish.
- `Live`: The deployment is active and has been promoted from staging.
- `Staging`: The deployment is active in staging. Use `--staging` to access it.
- `Failed`: The deployment failed. Try `tb deploy --check` to debug the issue.
- `Deleted`: The deployment was deleted as a result of creating new deployments.

## Deploy from the CLI (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#deploy-from-the-cli)

1
### Check the deployment (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#check-the-deployment)

Before creating the deployment, you can check the deployment with the `--check` flag. This runs a series of checks to ensure the deployment is ready. This is similar to a dry run.

```
tb --cloud deployment create --check
```
The `--check` flag validates external connections to S3, Kafka, GCS, and databases referenced via table functions. For local success, set connection secrets with `tb secret set` and use `tb local start --use-aws-creds` for S3 connections.

2
### Create a staging deployment (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#create-a-staging-deployment)

Create a new staging deployment in Tinybird Cloud. Pass the `--wait` flag to wait for the deployment to finish:

```
tb --cloud deployment create --wait
```
To run commands against the staging deployment, use the `--staging` flag. For example:

```
tb --staging --cloud endpoint ls
```
3
### Promote to live (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#promote-to-live)

When the staging deployment is ready, promote it to a live deployment in Tinybird Cloud:

```
tb --cloud deployment promote
```
To deploy and promote in one step, use `tb deploy`. If you prefer explicit flags, run `tb --cloud deploy`.

## Staging deployments (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#staging-deployments)

You can write data to, and read data from, a staging deployment before promoting it to live. This is useful when you've made schema changes that might be incompatible with the current live deployment, like adding new columns.

Automatic data source changes that Tinybird applies with `ALTER`, such as adding a column or changing a data source TTL, are applied only when the deployment is promoted to live. Those changes aren't available in staging deployments.

### Writing to staging deployments (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#writing-to-staging-deployments)

You can use the [Events API](/docs/forward/ingest-data/events-api) to write directly to staging deployments through the `__tb__min_deployment` parameter, which indicates the target deployment ID. For example:

```
curl \
    -H "Authorization: Bearer <import_token>" \
    -d '{"date": "2020-04-05 00:05:38", "city": "Chicago", "new_column": "value"}' \
    'https://<your_host>/v0/events?name=events_test&__tb__min_deployment=5'
```
In the example, if the ID of your current live deployment is 4 and you're creating a deployment with an ID of 5, Tinybird ingests the data into staging deployment 5 only. Follow these steps:

1. Make schema changes in a staging deployment.
2. Ingest data compatible with the new schema.
3. Validate the changes work as expected.
4. Promote the deployment to live when ready.
Without the parameter, data would be rejected if it doesn't match the schema of the current live deployment.

To get the deployment ID, run `tb deployment ls`.

### Reading from staging deployments (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#reading-from-staging-deployments)

You can query data from a staging deployment using [pipe endpoints](/docs/forward/core-concepts/api-endpoints). To access a staging endpoint, add the `__tb__deployment` parameter to your API request:

```
curl \
    -H "Authorization: Bearer <query_token>" \
    'https://<your_host>/v0/pipes/my_endpoint?__tb__deployment=5'
```
Follow these steps:

1. Test your endpoints with the new schema changes.
2. Validate query results before promoting to live.
3. Ensure your app works correctly with the updated data structure.
To get the deployment ID, run `tb deployment ls`.

### Continuous operation (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#continuous-operation)

Once the deployment is promoted to live, you can continue using the same API calls. The `__tb__min_deployment` or `__tb__deployment` parameters ensure compatibility both before and after promotion.

For more details on the Events API parameters, see the [Events API Reference](/docs/api-reference/events-api).

## On-demand compute for deployment populates (https://www.tinybird.co/docs/forward/development-workflow/manual-deployment#on-demand-compute-for-deployment-populates)

When a deployment needs to populate Data Sources, such as when creating Materialized Views or evolving schemas, Tinybird can run the populate on dedicated on-demand compute instead of your main workspace infrastructure. This keeps the populate isolated from production query and ingestion workloads.

Tinybird automatically uses on-demand compute for large deployment populates. For pricing, regional rates, and example calculations, see [On-demand CPUs](/docs/forward/pricing#on-demand-cpus).

---

URL: https://www.tinybird.co/docs/forward/development-workflow/local-development
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Local development \| Tinybird"
theme-color: "#171612"
description: "Develop your Tinybird project locally. Make changes and test them before deploying to Tinybird Cloud."
inkeep:version: "forward"
---

# Local development (https://www.tinybird.co/docs/forward/development-workflow/local-development#local-development)

Use Tinybird Local to build and test your project on your machine before deploying to Tinybird Cloud. The workflow is the same one used in the quickstarts: configure the project, build, test the Endpoint, then deploy.

Use Tinybird Local when fixture data is enough and you want a fast loop on your machine. Use [Cloud Branches](/docs/forward/development-workflow/using-cloud-branches) when you need production-like data, connector validation, shared preview URLs, or pull request review.

For installation, Docker settings, local services, API URLs, persistence, and local Tokens, see [Tinybird Local](/docs/forward/core-concepts/tinybird-local).

## Check your project config (https://www.tinybird.co/docs/forward/development-workflow/local-development#check-your-project-config)

Your project config defines which files Tinybird builds and which development environment commands use by default.

Use `include` to point at your datafiles or SDK resource definitions. Set `dev_mode` or `devMode` to `local` when you want `build` to target Tinybird Local by default. You can still override the target with `--local` or `--branch` on the command line.

- Tinybird CLI
- TypeScript SDK
- Python SDK
tinybird.config.json
```
{
  "dev_mode": "local",
  "include": [
    "tinybird"
  ]
}
```

## Typical local workflow (https://www.tinybird.co/docs/forward/development-workflow/local-development#typical-local-workflow)

1. Set the development environment to Local in `tinybird.config.json`.
2. Start Tinybird Local.
3. Check out a git branch.
4. Build the project with fixture data.
5. Edit Data Sources, Pipes, Endpoints, Tokens, or Connections.
6. Rebuild and query local resources or call local Endpoints.
7. Run a deployment check against Tinybird Cloud.
8. Merge and deploy ideally via CI/CD.

## Start Tinybird Local (https://www.tinybird.co/docs/forward/development-workflow/local-development#start-tinybird-local)

Start Tinybird Local before you build or test local resources:

```
tb local start
```
The CLI logs the status of the container and services. Do not build the project until you see the `Tinybird Local is ready!` message.

```
» Starting Tinybird Local...
* Tinybird Local container status: healthy
» Checking services...
✓ Tinybird Local container
✓ Clickhouse
✓ Redis
✓ Server
✓ Events
✓ Tinybird Local authentication
✓ Tinybird Local is ready!
```
Run Tinybird Local in the background with `--daemon`:

```
tb local start --daemon
```
If Tinybird Local becomes unhealthy, restart it:

```
tb local restart
```

## Build the project (https://www.tinybird.co/docs/forward/development-workflow/local-development#build-the-project)

Run a build before testing changes. It validates resource definitions, resolves dependencies, and loads matching fixture files.

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb build
```
If your config does not default to Local, pass the flag explicitly:

```
tb --local build
```
For SDK projects, use the matching SDK command, such as `npx tinybird build --local`.

```
» Building project...
✓ datasources/user_actions.datasource created
✓ endpoints/user_actions_line_chart.pipe created
✓ Build completed in 0.2s
```
In the `tb local start` logs, you can see whether the request is successful.

```
[API] POST /v1/build 200 67.1ms
```
When you edit project files, run build again to apply the changes to Tinybird Local.

If you want automatic rebuilds while editing local files, run `tb dev`. The examples use explicit `tb build` because it is the same command you run in CI and branch workflows.

## Test local resources (https://www.tinybird.co/docs/forward/development-workflow/local-development#test-local-resources)

Call local Endpoints with the local API host and a local Token:

```
curl -H "Authorization: Bearer <local-token>" \
  "http://localhost:7181/v0/pipes/user_actions_line_chart.json"
```
You can also run SQL against Tinybird Local:

```
tb --local sql "SELECT count() FROM user_actions"
```
For test files and fixtures, see [Test your project](/docs/forward/guides/test-your-project).

## Open Tinybird UI (https://www.tinybird.co/docs/forward/development-workflow/local-development#open-tinybird-ui)

You can also use Tinybird UI to inspect your project. Run the following command to open the Tinybird UI pointing to your local environment.

```
tb open
```
The URL is `https://cloud.tinybird.co/<cloud-provider>/<region>/<workspace-name>~local~<branch-name>`

## Deploy after validation (https://www.tinybird.co/docs/forward/development-workflow/local-development#deploy-after-validation)

When the local checks pass, validate the deployment against Tinybird Cloud before merging or releasing:

```
tb --cloud deploy --check
```
Then deploy manually or let CI/CD deploy from the main branch:

```
tb --cloud deploy
```
See [Manual deployment](/docs/forward/development-workflow/manual-deployment) and [CI/CD](/docs/forward/development-workflow/cicd).

---

URL: https://www.tinybird.co/docs/forward/development-workflow/examples
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Development workflow examples \| Tinybird"
theme-color: "#171612"
description: "Examples for iterating on Tinybird projects with the Tinybird CLI, the TypeScript SDK, the Python SDK, branches, CI, and deployments."
inkeep:version: "forward"
---

# Development workflow examples (https://www.tinybird.co/docs/forward/development-workflow/examples#development-workflow-examples)

These examples follow the order you usually use in a project: build first, test locally or in a branch, validate in CI, then deploy.

Use the command set that matches your project format.

| Task | Tinybird CLI | TypeScript SDK | Python SDK |
| --- | --- | --- | --- |
| Build or validate | `tb build` | `npx tinybird build` | `uv run tinybird build` |
| Preview a pull request | `git checkout -b ... && tb build` | `npx tinybird preview --check` | `uv run tinybird preview --check` |
| Check production deployment | `tb --cloud deploy --check` | `npx tinybird deploy --check` | `uv run tinybird deploy --check` |
| Deploy to production | `tb --cloud deploy` | `npx tinybird deploy` | `uv run tinybird deploy` |

## Local workflow (https://www.tinybird.co/docs/forward/development-workflow/examples#local-workflow)

Use Local for fast checks with fixture data. See [Tinybird Local](/docs/forward/core-concepts/tinybird-local) for installation, Docker settings, persistence, local APIs, and local Tokens. Use a [Cloud Branch](/docs/forward/development-workflow/using-cloud-branches) when you need production-like data or connector behavior.

### Build the project (https://www.tinybird.co/docs/forward/development-workflow/examples#build-the-project)

Run a build before testing changes. It catches invalid resource definitions and dependency issues.

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb build
```
For Tinybird CLI projects, `tb build` loads matching fixtures automatically. For example, `fixtures/events.ndjson` is loaded into `datasources/events.datasource`. See [Test your project](/docs/forward/guides/test-your-project#fixture-files).

### Change an Endpoint and rebuild (https://www.tinybird.co/docs/forward/development-workflow/examples#change-an-endpoint-and-rebuild)

Edit the resource, then rebuild the project.

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb local start --daemon
tb build
```
datasources/events.datasource
```
SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `user_id` String `json:$.user_id`,
    `event` LowCardinality(String) `json:$.event`,
    `plan` Nullable(String) `json:$.plan`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "timestamp"
```
endpoints/events_by_plan.pipe
```
NODE endpoint
SQL >
    SELECT
        plan,
        count() AS events
    FROM events
    WHERE timestamp >= now() - INTERVAL 7 DAY
    GROUP BY plan
    ORDER BY events DESC

TYPE ENDPOINT
```
Call the local Endpoint after the build finishes:

```
curl -H "Authorization: Bearer <local-token>" \
  "http://localhost:7181/v0/pipes/events_by_plan.json"
```

## Branch workflow (https://www.tinybird.co/docs/forward/development-workflow/examples#branch-workflow)

Branches are short-lived Cloud environments. Use them to test changes without changing production.

### Develop in a Cloud Branch (https://www.tinybird.co/docs/forward/development-workflow/examples#develop-in-a-cloud-branch)

For everyday branch development, create a Git branch and build. Tinybird creates or reuses a Cloud Branch with the same name as the Git branch.

```
git checkout -b add-plan-to-events
tb build
```
Rebuild after editing resources:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb build
```
Query the branch after the build finishes:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb --branch=add-plan-to-events sql "SELECT plan, count() FROM events GROUP BY plan"
```
When you need production-like data, `tb build` attaches the latest production partition automatically when `dev_mode=branch`. You can also create the branch explicitly with `--last-partition` before building:

```
tb branch create add-plan-to-events --last-partition
tb --branch=add-plan-to-events build
```
For connector Data Sources, create the branch with `--with-connections` and start or sample the connector when you are ready to test. See [Cloud Branches](/docs/forward/development-workflow/using-cloud-branches#test-connector-data).

## Pull requests and CI (https://www.tinybird.co/docs/forward/development-workflow/examples#pull-requests-and-ci)

Run the same checks in CI that you run locally. For pull requests, create a preview branch so reviewers and preview apps can query the changed Tinybird project.

### Preview a pull request (https://www.tinybird.co/docs/forward/development-workflow/examples#preview-a-pull-request)

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb branch create tmp_ci_my_feature --last-partition
tb --branch=tmp_ci_my_feature build
```
Use `preview` in GitHub Actions, GitLab CI, Vercel, or another CI system to create an ephemeral branch for each pull request. See [Preview deployments](/docs/forward/development-workflow/cicd#preview-deployments).

### Validate before deploying (https://www.tinybird.co/docs/forward/development-workflow/examples#validate-before-deploying)

Use deployment checks for changes that affect Data Source schemas, Materialized Views, engine settings, or connector configuration.

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb build
tb test run
tb --cloud deploy --check
```

## Data Source evolution (https://www.tinybird.co/docs/forward/development-workflow/examples#data-source-evolution)

For the migration rules behind Data Source changes, read [Evolve Data Sources](/docs/forward/guides/evolve-data-source). The following examples show the most common cases.

### Rename a column with FORWARD_QUERY (https://www.tinybird.co/docs/forward/development-workflow/examples#rename-a-column-with-forward-query)

Column renames need an explicit mapping. Tinybird sees a removed column and a new column. Without a mapping, the new column gets a default value.

Update the Data Source definition and run a deployment check before merging. With the Tinybird CLI, add `FORWARD_QUERY` to the `.datasource` file. In SDK projects, update the schema in code and review the generated deployment plan. If the plan requires a forward query, use the expression from the Data Source evolution guide as the source of truth.

- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/events.datasource
```
SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `account_id` String `json:$.account_id`,
    `event` LowCardinality(String) `json:$.event`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "timestamp"

FORWARD_QUERY >
    SELECT timestamp, user_id AS account_id, event
```
The `FORWARD_QUERY` runs against the live Data Source during deployment. It must include the `SELECT` list only. Do not include `FROM` or `WHERE`.

### Add a nullable column with ALTER (https://www.tinybird.co/docs/forward/development-workflow/examples#add-a-nullable-column-with-alter)

Adding a nullable column is usually an automatic `ALTER` during deployment. The change is applied when the deployment is promoted to live, not while it is in staging. See [Automatic ALTER operations](/docs/forward/guides/evolve-data-source#automatic-alter-operations).

- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/events.datasource
```
SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `user_id` String `json:$.user_id`,
    `event` LowCardinality(String) `json:$.event`,
    `plan` Nullable(String) `json:$.plan`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "timestamp"
```
Run a deployment check before merging the change:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb --cloud deploy --check
```

### Change a type with safe conversion (https://www.tinybird.co/docs/forward/development-workflow/examples#change-a-type-with-safe-conversion)

If existing values might not cast cleanly, make the conversion explicit. This example changes `session_id` from `String` to `UUID` and uses a default UUID when the old value is invalid.

- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/sessions.datasource
```
SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` UUID `json:$.session_id`,
    `user_id` String `json:$.user_id`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "timestamp"

FORWARD_QUERY >
    SELECT
        timestamp,
        accurateCastOrDefault(session_id, 'UUID') AS session_id,
        user_id
```
After the deployment is live and you no longer need the migration expression, remove the `FORWARD_QUERY` in a follow-up change. Leaving an old `FORWARD_QUERY` can overwrite values in a later backfill.

## Deployments (https://www.tinybird.co/docs/forward/development-workflow/examples#deployments)

After the checks pass, deploy the project or create a staging deployment for manual promotion.

### Deploy to production (https://www.tinybird.co/docs/forward/development-workflow/examples#deploy-to-production)

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb --cloud deploy
```

### Promote a staging deployment (https://www.tinybird.co/docs/forward/development-workflow/examples#promote-a-staging-deployment)

If you prefer explicit staging and promotion, create a staging deployment, inspect it, then promote it:

```
tb --cloud deployment create --wait
tb --staging --cloud endpoint ls
tb --cloud deployment promote
```
For regular production changes, put the same checks in CI/CD. See [CI/CD](/docs/forward/development-workflow/cicd).

---

URL: https://www.tinybird.co/docs/forward/development-workflow/cicd
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "CI/CD \| Tinybird"
theme-color: "#171612"
description: "Deploy your Tinybird project through CI/CD workflows."
inkeep:version: "forward"
---

# Deploying to Tinybird through CI/CD (https://www.tinybird.co/docs/forward/development-workflow/cicd#deploying-to-tinybird-through-cicd)

After you create your data project in Git, you can implement continuous integration (CI) and continuous deployment (CD) workflows to automate interaction with Tinybird.

When you initialize a project using `tb init`, Tinybird can generate CI/CD templates that you can use in GitHub and GitLab to automate testing and deployment.

The Tinybird Local container is a key part of the CI workflow. See [Tinybird Local](/docs/forward/core-concepts/tinybird-local) for more information.

## CI workflow (https://www.tinybird.co/docs/forward/development-workflow/cicd#ci-workflow)

As you expand and iterate on your data projects, you can continuously validate your changes. In the same way that you write integration and acceptance tests for source code in a software project, you can write automated tests for your API endpoints to run on each pull or merge request.

A potential CI workflow could run the following steps when you open a pull request:

1. Install Tinybird CLI and Tinybird Local: Sets up dependencies and installs the Tinybird CLI to run the required commands.
2. Build project: Checks the syntax and correctness of your datafiles.
3. Test project: Runs fixture tests, data quality tests, or both to validate changes.
4. Deployment check: Validates the deployment before creating it, similar to a dry run.
The following templates are available for GitHub and GitLab:

- GitHub
- GitLab

```
name: Tinybird - CI Workflow

on:
  workflow_dispatch:
  pull_request:
    branches:
      - main
      - master
    types: [opened, reopened, labeled, unlabeled, synchronize]

concurrency: ${{ github.workflow }}-${{ github.event.pull_request.number }}

env:
  TINYBIRD_HOST: ${{ secrets.TINYBIRD_HOST }}
  TINYBIRD_TOKEN: ${{ secrets.TINYBIRD_TOKEN }}

jobs:
  ci:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: '.'
    services:
      tinybird:
        image: tinybirdco/tinybird-local:latest
        ports:
          - 7181:7181
    steps:
      - uses: actions/checkout@v4
      - name: Install Tinybird CLI
        run: curl https://tinybird.co | sh
      - name: Build project
        run: tb --local build
      - name: Test project
        run: tb --local test run
      - name: Deployment check
        run: tb --cloud --host ${{ env.TINYBIRD_HOST }} --token ${{ env.TINYBIRD_TOKEN }} deploy --check
```

## Preview deployments (https://www.tinybird.co/docs/forward/development-workflow/cicd#preview-deployments)

You can create an ephemeral [branch](/docs/forward/core-concepts/branches) for each pull request so you can test your changes with production data before deploying. There are two ways to set up preview deployments: using the TypeScript SDK or the Tinybird CLI.

### Using the TypeScript SDK (https://www.tinybird.co/docs/forward/development-workflow/cicd#using-the-typescript-sdk)

Requires `@tinybirdco/sdk` version `0.0.53` or later.

The `tinybird preview` command creates a branch named `tmp_ci_<git_branch>`, builds your resources, and deploys them. If a branch with the same name already exists, it is deleted and recreated.

The TypeScript SDK auto-detects CI environments (Vercel, GitHub Actions, GitLab CI, CircleCI, Azure Pipelines, Bitbucket Pipelines) and resolves the correct branch token at runtime. The host is inferred from the token, so no `TINYBIRD_HOST` or additional configuration is needed.

- GitHub
- GitLab

```
name: Tinybird - Preview

on:
  pull_request:
    branches:
      - main
      - master
    types: [opened, reopened, synchronize]

concurrency: ${{ github.workflow }}-${{ github.event.pull_request.number }}

jobs:
  preview:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npx tinybird preview
        env:
          TINYBIRD_TOKEN: ${{ secrets.TINYBIRD_TOKEN }}
```

### Using the Tinybird CLI (https://www.tinybird.co/docs/forward/development-workflow/cicd#using-the-tinybird-cli)

You can also create preview branches using the `tb` CLI. This approach gives you more control over branch creation and supports connectors.

- GitHub
- GitLab

```
name: Tinybird - Preview

on:
  pull_request:
    branches:
      - main
      - master
    types: [opened, reopened, synchronize]

concurrency: ${{ github.workflow }}-${{ github.event.pull_request.number }}

env:
  TINYBIRD_HOST: ${{ secrets.TINYBIRD_HOST }}
  TINYBIRD_TOKEN: ${{ secrets.TINYBIRD_TOKEN }}

jobs:
  preview:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install Tinybird CLI
        run: curl https://tinybird.co | sh
      - name: Create branch
        run: tb --host ${{ env.TINYBIRD_HOST }} --token ${{ env.TINYBIRD_TOKEN }} branch create tmp_ci_${{ github.head_ref }} --last-partition
      - name: Build and deploy
        run: tb --host ${{ env.TINYBIRD_HOST }} --token ${{ env.TINYBIRD_TOKEN }} --branch=tmp_ci_${{ github.head_ref }} build
```

### Preview with connectors (https://www.tinybird.co/docs/forward/development-workflow/cicd#preview-with-connectors)

When your project uses Kafka, S3, or GCS connectors, preview branches can include them. The `tinybird preview` command doesn't create preview branches that ingest data from connectors. To test with connectors, create the branch manually with the `tb` CLI. See [Test with connector data](/docs/forward/core-concepts/branches#test-with-connector-data).

```
tb branch create tmp_ci_my_feature --last-partition --with-connections
```
Kafka connections are stopped by default in preview branches. Start them when you're ready to test:

```
tb --branch=tmp_ci_my_feature datasource start my_kafka_datasource
tb --branch=tmp_ci_my_feature datasource stop my_kafka_datasource
```
For S3/GCS, import sample data:

```
tb --branch=tmp_ci_my_feature datasource sample my_s3_datasource --wait
```

### Cleanup (https://www.tinybird.co/docs/forward/development-workflow/cicd#cleanup)

Preview branches are ephemeral. Delete them when the pull request is closed or merged. You can automate this with a CI workflow:

- GitHub
- GitLab

```
name: Tinybird - Cleanup Preview

on:
  pull_request:
    branches:
      - main
      - master
    types: [closed]

concurrency: ${{ github.workflow }}-${{ github.event.pull_request.number }}

jobs:
  cleanup:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npx tinybird branch delete tmp_ci_${{ github.head_ref }}
        env:
          TINYBIRD_TOKEN: ${{ secrets.TINYBIRD_TOKEN }}
```
Or delete branches manually using the TypeScript SDK or the Tinybird CLI:

```
tinybird branch delete tmp_ci_my_feature
tb branch rm tmp_ci_my_feature
```

### Frontend preview integration (https://www.tinybird.co/docs/forward/development-workflow/cicd#frontend-preview-integration)

The TypeScript SDK runtime client (`@tinybirdco/sdk` version `0.0.53` or later) detects preview environments automatically. When your frontend is deployed to a preview environment, such as a Vercel Preview Deployment, it automatically queries the Tinybird preview branch that matches the frontend's git branch, with no code changes or extra environment variables required.

```
import { Tinybird } from '@tinybirdco/sdk'

const tb = new Tinybird({ token: process.env.TINYBIRD_TOKEN })

// In preview environments (e.g., a Vercel Preview Deployment), the SDK
// automatically resolves the matching branch token.
// No additional configuration is needed.
const result = await tb.query('SELECT * FROM my_endpoint')
```
Auto-detection works with Vercel, GitHub Actions, GitLab CI, CircleCI, Azure Pipelines, and Bitbucket Pipelines.

## CD workflow (https://www.tinybird.co/docs/forward/development-workflow/cicd#cd-workflow)

Once your changes are validated by the CI pipeline, you can automate the deployment process and let Tinybird handle the migration for you with no downtime.

A potential CD workflow could run the following steps when you merge a pull request:

1. Install Tinybird CLI: Sets up dependencies and installs the Tinybird CLI to run the required commands.
2. Deploy project: Creates a staging deployment in Tinybird Cloud, migrates data, promotes to live, and removes previous deployment.
The following templates are available for GitHub and GitLab:

- GitHub
- GitLab

```
name: Tinybird - CD Workflow

on:
  push:
    branches:
      - main
      - master

concurrency: ${{ github.workflow }}-${{ github.event.ref }}

env:
  TINYBIRD_HOST: ${{ secrets.TINYBIRD_HOST }}
  TINYBIRD_TOKEN: ${{ secrets.TINYBIRD_TOKEN }}

jobs:
  cd:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install Tinybird CLI
        run: curl https://tinybird.co | sh
      - name: Deploy project
        run: tb --cloud --host ${{ env.TINYBIRD_HOST }} --token ${{ env.TINYBIRD_TOKEN }} deploy
```

## Secrets (https://www.tinybird.co/docs/forward/development-workflow/cicd#secrets)

Make sure to provide the values for the following secrets in your CI/CD settings:

- `TINYBIRD_HOST`
- `TINYBIRD_TOKEN`
Run `tb info` to get the value for the secret `TINYBIRD_HOST`. It is the `api` URL in Tinybird Cloud. For example:

```
tb info

» Tinybird Cloud:
--------------------------------------------------------------------------------------------
user: tinybird@domain.co
workspace_name: forward
workspace_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX
token: YOUR-ADMIN-TOKEN
user_token: YOUR-USER-TOKEN
api: https://api.tinybird.co
ui: https://cloud.tinybird.co/gcp/europe-west2/forward
--------------------------------------------------------------------------------------------

» Tinybird Local:
--------------------------------------------------------------------------------------------
user: tinybird@domain.co
workspace_name: forward
workspace_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX
token: YOUR-LOCAL-ADMIN-TOKEN
user_token: YOUR-LOCAL-USER-TOKEN
api: http://localhost:7181
ui: http://cloud.tinybird.co/local/7181/forward
--------------------------------------------------------------------------------------------

» Project:
---------------------------------------------------
current: /path/to/your/project
.tinyb: /path/to/your/project/.tinyb
project: /path/to/your/project
---------------------------------------------------
```
And run `tb --cloud token copy "admin <your_email>"` to get the value of `TINYBIRD_TOKEN`.

```
tb --cloud token copy "admin your_user@email.com"
Running against Tinybird Cloud: Workspace forward
** Token 'admin your_user@email.com' copied to clipboard
```
When running `tb test run`, Tinybird creates a fresh Workspace for each test run. Secrets don't persist between test runs. To avoid test failures, add a default value to your secrets. For example:

```
GCS_SERVICE_ACCOUNT_CREDENTIALS_JSON {{ tb_secret("secret_name", "default_value") }}
```
`tb_secret` replacements happen at parser time in the server. If a secret changes after a deployment, Tinybird doesn't detect it automatically and requires a new deployment.

---

URL: https://www.tinybird.co/docs/forward/development-workflow/agent-skills
Last update: 2026-07-29T12:18:04.000Z
Content:
---
title: "Tinybird Agent Skills \| Tinybird"
theme-color: "#171612"
description: "Help coding agents build, test, and deploy Tinybird projects."
inkeep:version: "forward"
---

# Tinybird Agent Skills (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#tinybird-agent-skills)

Tinybird Agent Skills give compatible AI coding agents instructions for working with Tinybird projects. They provide context for creating and reviewing project files, SQL, tests, and deployment workflows. The agent uses the skills through the tools available in your coding environment, such as the Tinybird CLI or an SDK.

The skills follow the [Agent Skills format](https://agentskills.io/) and work with compatible coding agents such as Amp, Claude Code, Codex, Cursor, Gemini CLI, and OpenCode. Their source is available in the [Tinybird Agent Skills GitHub repository](https://github.com/tinybirdco/tinybird-agent-skills).

## Install Tinybird Agent Skills (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#install-tinybird-agent-skills)

Run the following command in your project:

Install Tinybird Agent Skills
```
npx skills add tinybirdco/tinybird-agent-skills
```
Update installed skills to their latest versions:

Update installed Agent Skills
```
npx skills update
```
Once installed, a compatible coding agent can select a skill when it detects a relevant task. You can also ask the agent which skills are available.

## Available skills (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#available-skills)

### `tinybird-best-practices` (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#tinybird-best-practices)

Use this skill for general Tinybird project work. It covers project structure, Data Sources, Pipes, Endpoints, SQL and query optimization, deployments, testing, and secrets management.

### `tinybird-cli-guidelines` (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#tinybird-cli-guidelines)

Use this skill when working with datafiles or running `tb` commands. It covers Tinybird Local, builds, deployments, data operations, Tokens, secrets, mock data, and tests.

### `tinybird-typescript-sdk-guidelines` (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#tinybird-typescript-sdk-guidelines)

Use this skill with `@tinybirdco/sdk` and TypeScript projects. It covers typed Data Sources and Pipes, Tinybird clients, ingestion and queries, SDK CLI workflows, and migration from datafiles.

### `tinybird-python-sdk-guidelines` (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#tinybird-python-sdk-guidelines)

Use this skill with `tinybird-sdk` and Python projects. It covers Data Sources, Pipes, Endpoints, Connections, clients, ingestion, queries, deployment commands, and migration from datafiles.

## Example prompts (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#example-prompts)

After installing the skills, give your coding agent a concrete project task:

- "Create a Data Source for user events and add fixture data for local testing."
- "Review this Pipe and optimize its Endpoint for low latency."
- "Add tests for the Endpoints and run them with the Tinybird CLI."
- "Define a Kafka Connection and Materialized View with the Python SDK."
- "Validate this project and prepare it for deployment to Tinybird Cloud."
The coding agent applies the relevant skill guidance, then uses the tools and credentials available in your project to perform the task.

## Agent Skills and MCP (https://www.tinybird.co/docs/forward/development-workflow/agent-skills#agent-skills-and-mcp)

Agent Skills provide packaged instructions and Tinybird-specific context inside a coding workflow. Use them when an agent needs to create or review project files, write SQL, add tests, or work through build and deployment commands.

The [Tinybird MCP server](/docs/forward/query-data/mcp) connects an agent to a live Workspace. The configured Token limits its access, and the agent can inspect or query data and call published Endpoints. Use MCP when the task depends on live Workspace resources or data.

You can combine both. Agent Skills guide how the coding agent changes the project, while MCP lets the agent inspect the live Workspace or verify results through its permitted queries and Endpoints.

---

URL: https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources
Last update: 2026-06-18T10:43:45.000Z
Content:
---
title: "TypeScript SDK resources \| Tinybird"
theme-color: "#171612"
description: "Define Tinybird resources in TypeScript with the Tinybird SDK."
inkeep:version: "forward"
---

# TypeScript SDK resources (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#typescript-sdk-resources)

Use the TypeScript SDK to define Tinybird resources as code with full type inference. You can define datasources, pipes, endpoints, materialized views, copy pipes, sink pipes, and connections in TypeScript and sync them to Tinybird.

## Project layout (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#project-layout)

The SDK scaffolds the following files by default:

- `lib/tinybird.ts`
- `tinybird.config.mjs`

## Configure credentials (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#configure-credentials)

Create a `.env.local` file in your project root with the Tinybird token and host:

```
TINYBIRD_TOKEN=p.your_token_here
TINYBIRD_URL=https://api.tinybird.co
```

## Tokens (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#tokens)

Use `defineToken` to declare reusable token names, then attach them to datasources or pipes with each resource's `tokens` option.

```
import {
  defineDatasource,
  defineEndpoint,
  defineToken,
  engine,
  node,
  t,
} from "@tinybirdco/sdk";

export const eventsAppend = defineToken("events_append");
export const eventsRead = defineToken("events_read");

export const events = defineDatasource("events", {
  schema: {
    timestamp: t.dateTime(),
    payload: t.string(),
  },
  engine: engine.mergeTree({
    sortingKey: ["timestamp"],
  }),
  tokens: [
    { token: eventsAppend, scope: "APPEND" },
    { token: eventsRead, scope: "READ" },
  ],
});

export const eventsApi = defineEndpoint("events_api", {
  nodes: [
    node({
      name: "latest",
      sql: `SELECT timestamp, payload FROM events ORDER BY timestamp DESC LIMIT 100`,
    }),
  ],
  output: {
    timestamp: t.dateTime(),
    payload: t.string(),
  },
  tokens: [{ token: eventsRead, scope: "READ" }],
});
```
Token names must start with a letter or underscore and then contain only alphanumeric characters or underscores. Use `READ` or `APPEND` for datasource token bindings, and `READ` for pipe or endpoint token bindings.

## Datasources (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#datasources)

Define datasources with schemas and engines:

```
import { defineDatasource, engine, t, type InferRow } from "@tinybirdco/sdk";

export const pageViews = defineDatasource("page_views", {
  description: "Page view tracking data",
  schema: {
    timestamp: t.dateTime(),
    pathname: t.string(),
    session_id: t.string(),
    country: t.string().lowCardinality().nullable(),
  },
  engine: engine.mergeTree({
    sortingKey: ["pathname", "timestamp"],
  }),
});

export type PageViewsRow = InferRow<typeof pageViews>;
```

## Endpoints (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#endpoints)

Define API endpoints:

```
import {
  defineEndpoint,
  node,
  p,
  t,
  type InferParams,
  type InferOutputRow,
} from "@tinybirdco/sdk";

export const topPages = defineEndpoint("top_pages", {
  description: "Get the most visited pages",
  params: {
    start_date: p.dateTime(),
    end_date: p.dateTime(),
    limit: p.int32().optional(10),
  },
  nodes: [
    node({
      name: "aggregated",
      sql: `
        SELECT pathname, count() AS views
        FROM page_views
        WHERE timestamp >= {{DateTime(start_date)}}
          AND timestamp <= {{DateTime(end_date)}}
        GROUP BY pathname
        ORDER BY views DESC
        LIMIT {{Int32(limit, 10)}}
      `,
    }),
  ],
  output: {
    pathname: t.string(),
    views: t.uint64(),
  },
});

export type TopPagesParams = InferParams<typeof topPages>;
export type TopPagesOutput = InferOutputRow<typeof topPages>;
```

## Pipes (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#pipes)

Define internal pipes that are not exposed as API endpoints:

```
import { definePipe, node, p } from "@tinybirdco/sdk";

export const filteredEvents = definePipe("filtered_events", {
  description: "Filter events by date range",
  params: {
    start_date: p.dateTime(),
    end_date: p.dateTime(),
  },
  nodes: [
    node({
      name: "filtered",
      sql: `
        SELECT * FROM events
        WHERE timestamp >= {{DateTime(start_date)}}
          AND timestamp <= {{DateTime(end_date)}}
      `,
    }),
  ],
});
```

## Materialized views (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#materialized-views)

Use materialized views to populate derived datasources:

```
import {
  defineDatasource,
  defineMaterializedView,
  node,
  t,
  engine,
} from "@tinybirdco/sdk";

export const dailyStats = defineDatasource("daily_stats", {
  schema: {
    date: t.date(),
    pathname: t.string(),
    views: t.simpleAggregateFunction("sum", t.uint64()),
  },
  engine: engine.aggregatingMergeTree({
    sortingKey: ["date", "pathname"],
  }),
  jsonPaths: false,
});

export const dailyStatsMv = defineMaterializedView("daily_stats_mv", {
  datasource: dailyStats,
  nodes: [
    node({
      name: "aggregate",
      sql: `
        SELECT toDate(timestamp) AS date, pathname, count() AS views
        FROM page_views
        GROUP BY date, pathname
      `,
    }),
  ],
});
```

## Copy pipes (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#copy-pipes)

Use copy pipes for snapshots or scheduled exports:

```
import { defineCopyPipe, node } from "@tinybirdco/sdk";

export const dailySnapshot = defineCopyPipe("daily_snapshot", {
  datasource: dailyStats,
  schedule: "0 0 * * *",
  mode: "append",
  nodes: [
    node({
      name: "snapshot",
      sql: `SELECT * FROM daily_stats WHERE date = today() - 1`,
    }),
  ],
});
```

## Sink pipes (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#sink-pipes)

Use sink pipes to publish query results to Kafka or S3:

```
import { defineSinkPipe, node } from "@tinybirdco/sdk";

export const kafkaEventsSink = defineSinkPipe("kafka_events_sink", {
  sink: {
    connection: eventsKafka,
    topic: "events_export",
    schedule: "@on-demand",
  },
  nodes: [
    node({
      name: "publish",
      sql: `SELECT timestamp, payload FROM kafka_events`,
    }),
  ],
});

export const s3EventsSink = defineSinkPipe("s3_events_sink", {
  sink: {
    connection: landingS3,
    bucketUri: "s3://my-bucket/exports/",
    fileTemplate: "events_{date}",
    format: "csv",
    schedule: "@once",
  },
  nodes: [
    node({
      name: "export",
      sql: `SELECT timestamp, session_id FROM s3_landing`,
    }),
  ],
});
```

## Connections (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#connections)

Define connector connections and reference them in datasources:

```
import {
  defineGCSConnection,
  defineKafkaConnection,
  defineDynamoDBConnection,
  defineDatasource,
  engine,
  secret,
  t,
} from "@tinybirdco/sdk";

export const eventsKafka = defineKafkaConnection("events_kafka", {
  bootstrapServers: "kafka.example.com:9092",
  securityProtocol: "SASL_SSL",
  saslMechanism: "PLAIN",
  key: secret("KAFKA_KEY"),
  secret: secret("KAFKA_SECRET"),
});

export const landingGCS = defineGCSConnection("landing_gcs", {
  serviceAccountCredentialsJson: secret("GCS_SERVICE_ACCOUNT_CREDENTIALS_JSON"),
});

export const kafkaEvents = defineDatasource("kafka_events", {
  schema: {
    timestamp: t.dateTime(),
    payload: t.string(),
  },
  engine: engine.mergeTree({
    sortingKey: ["timestamp"],
  }),
  kafka: {
    connection: eventsKafka,
    topic: "events",
    groupId: "events-consumer",
  },
});

export const gcsLanding = defineDatasource("gcs_landing", {
  schema: {
    timestamp: t.dateTime(),
    session_id: t.string(),
  },
  engine: engine.mergeTree({
    sortingKey: ["timestamp"],
  }),
  gcs: {
    connection: landingGCS,
    bucketUri: "gs://my-gcs-bucket/events/*.csv",
    schedule: "@auto",
  },
});
```

## DynamoDB (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#dynamodb)

The DynamoDB connector mirrors an Amazon DynamoDB table into Tinybird: an initial backfill via a Point-in-Time Recovery export to S3, followed by continuous Change Data Capture from DynamoDB Streams. See the [DynamoDB connector](/docs/forward/ingest-data/connectors/dynamodb) page for the table requirements and IAM setup.

Defining the connection in code isn't enough. Tinybird assumes an IAM role in your own AWS account, and the role's trust policy must include a Workspace-specific external ID that only Tinybird generates. Run `tb connection create dynamodb` to get the external ID, the `Principal` account, and the policy JSON, then create the role in AWS and store its ARN as the `DYNAMODB_ROLE_ARN` secret (set it from the dashboard or with `tb secret set`). Without the correct external ID, `deploy` fails with a `403 ... external ID` error. See the [DynamoDB connector](/docs/forward/ingest-data/connectors/dynamodb) page for the full setup.

Define the connection with `defineDynamoDBConnection`. It takes a `region` and an `arn` (the IAM role Tinybird assumes to read your table, its stream, and the S3 export bucket); both are required:

```
import { defineDynamoDBConnection, secret } from "@tinybirdco/sdk";

export const eventsDynamo = defineDynamoDBConnection("events_dynamo", {
  region: "us-east-1",
  arn: secret("DYNAMODB_ROLE_ARN"),
});
```
The `arn` is read from a workspace secret. The SDK has no command for creating secrets, so set `DYNAMODB_ROLE_ARN` from the Tinybird dashboard or with `tb secret set DYNAMODB_ROLE_ARN <role-arn>` before you deploy. See the [DynamoDB connector](/docs/forward/ingest-data/connectors/dynamodb) page for the IAM role and trust policy the ARN must point to.

Each row represents a *change* to the table, so map the change record's fields with `jsonPath()` and use a `replacingMergeTree` engine with `ver` and `isDeleted` to query the current state. Attach the connection through the `dynamodb` key with the table ARN and the name of an S3 `exportBucket` you own (the bucket name only, with no `s3://` prefix):

```
import { defineDatasource, t, engine } from "@tinybirdco/sdk";
import { eventsDynamo } from "./connections";

export const dynamoEvents = defineDatasource("dynamo_events", {
  schema: {
    id: t.string().jsonPath("$.Id"),
    _record: t.string().jsonPath("$.NewImage"),
    _timestamp: t.dateTime64(3).jsonPath("$.ApproximateCreationDateTime"),
    _event_name: t.string().lowCardinality().jsonPath("$.eventName"),
    _is_deleted: t.uint8().jsonPath("$._is_deleted"),
  },
  engine: engine.replacingMergeTree({
    sortingKey: ["id"],
    ver: "_timestamp",
    isDeleted: "_is_deleted",
  }),
  dynamodb: {
    connection: eventsDynamo,
    tableArn: "arn:aws:dynamodb:us-east-1:123456789012:table/events",
    exportBucket: "my-export-bucket",
  },
});
```

## Next steps (https://www.tinybird.co/docs/forward/dev-reference/typescript-sdk-resources#next-steps)

- Review the SDK CLI: [TypeScript SDK CLI commands](/docs/forward/dev-reference/commands/typescript-sdk-cli)
- Follow the SDK quickstart: [Quick start: TypeScript SDK](/docs/forward/quickstarts/typescript-sdk)

---

URL: https://www.tinybird.co/docs/forward/dev-reference/template-functions
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Template functions \| Tinybird"
theme-color: "#171612"
description: "Template functions available in Tinybird datafiles."
inkeep:version: "forward"
---

# Template functions (https://www.tinybird.co/docs/forward/dev-reference/template-functions#template-functions)

The following template functions are available. You can use them in [datafiles](/docs/forward/dev-reference/datafiles) to accomplish different tasks. For the conceptual model of parameters, settings, secrets, and control flow, see [Templating language](/docs/forward/core-concepts/templating-language).

## defined (https://www.tinybird.co/docs/forward/dev-reference/template-functions#defined)

Checks whether a variable is defined.

defined function
```
%
SELECT
  date
FROM my_table
{% if defined(param) %}
  WHERE ...
{% end %}
```

## column (https://www.tinybird.co/docs/forward/dev-reference/template-functions#column)

Retrieves the column by its name from a variable.

column function
```
%
{% set var_1 = 'name' %}
SELECT
  {{column(var_1)}}
FROM my_table
```

## columns (https://www.tinybird.co/docs/forward/dev-reference/template-functions#columns)

Retrieves columns by their name from a variable.

columns function
```
%
{% set var_1 = 'name,age,address' %}
SELECT
  {{columns(var_1)}}
FROM my_table
```

## date_diff_in_seconds (https://www.tinybird.co/docs/forward/dev-reference/template-functions#date-diff-in-seconds)

Returns the absolute value of the difference in seconds between two `DateTime`. See [DateTime](/docs/sql-reference/data-types/datetime).

The function accepts the following parameters:

- `date_1`: the first date or DateTime.
- `date_2`: the second date or DateTime.
- `date_format`: (optional) the format of the dates. Defaults to `'%Y-%m-%d %H:%M:%S'`, so you can pass `DateTime` as `YYYY-MM-DD hh:mm:ss` when calling the function.
- `backup_date_format`: (optional) the format of the dates if the first format doesn't match. Use it when your default input format is a DateTime (`2022-12-19 18:42:22`) but you receive a date instead (`2022-12-19`).
- `none_if_error`: (optional) whether to return `None` if the dates don't match the provided formats. Defaults to `False`. Use it to provide an alternate logic in case any of the dates are specified in a different format.
An example of how to use the function:

```
date_diff_in_seconds('2022-12-19T18:42:23.521Z', '2022-12-19T18:42:23.531Z', date_format='%Y-%m-%dT%H:%M:%S.%fz')
```
The following example shows how to use the function in a datafile:

date_diff_in_seconds function
```
%
SELECT
  date, events
{% if date_diff_in_seconds(date_end, date_start, date_format="%Y-%m-%dT%H:%M:%Sz") < 3600 %}
  FROM my_table_raw
{% else %}
  FROM my_table_hourly_agg
{% end %}
  WHERE date BETWEEN
    parseDateTimeBestEffort({{String(date_start,'2023-01-11T12:24:04Z')}})
    AND
    parseDateTimeBestEffort({{String(date_end,'2023-01-11T12:24:05Z')}})
```

## date_diff_in_minutes (https://www.tinybird.co/docs/forward/dev-reference/template-functions#date-diff-in-minutes)

Same behavior as [date_diff_in_seconds](https://www.tinybird.co/docs/forward/dev-reference/template-functions#date_diff_in_seconds), but returns the difference in minutes.

## date_diff_in_hours (https://www.tinybird.co/docs/forward/dev-reference/template-functions#date-diff-in-hours)

Same behavior as [date_diff_in_seconds](https://www.tinybird.co/docs/forward/dev-reference/template-functions#date_diff_in_seconds), but returns the difference in hours.

## date_diff_in_days (https://www.tinybird.co/docs/forward/dev-reference/template-functions#date-diff-in-days)

Returns the absolute value of the difference in days between two dates or DateTime.

date_diff_in_days function
```
%
SELECT
  date
FROM my_table
{% if date_diff_in_days(date_end, date_start) < 7 %}
  WHERE ...
{% end %}
```
`date_format` is optional and defaults to `'%Y-%m-%d`, so you can pass DateTime as `YYYY-MM-DD` when calling the function.

As with `date_diff_in_seconds`, `date_diff_in_minutes`, and `date_diff_in_hours`, other [date_formats](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) are supported.

## max_threads (https://www.tinybird.co/docs/forward/dev-reference/template-functions#max-threads)

Sets the maximum number of threads a query can use.

This is useful for local development and Copy Pipes when the local ClickHouse® instance has limited threads available.

max_threads function
```
%
{{max_threads(1)}}
SELECT *
FROM events
```
See [Run Copy Pipes locally](/docs/forward/core-concepts/copy-pipes#run-copy-pipes-in-tinybird-local) for a Copy Pipe example.

## split_to_array (https://www.tinybird.co/docs/forward/dev-reference/template-functions#split-to-array)

Splits comma separated values into an array. The function accepts the following parameters:

`split_to_array(arr, default, separator=',')`

- `arr`: the value to split.
- `default`: the default value to use if `arr` is empty.
- `separator`: the separator to use. Defaults to `,`.
The following example splits `code` into an array of integers:

split_to_array function
```
%
SELECT
  arrayJoin(arrayMap(x -> toInt32(x), {{split_to_array(code, '')}})) as codes
FROM my_table
```
The following example splits `param` into an array of strings using `|` as the custom separator:

split_to_array with a custom separator function
```
%
SELECT
  {{split_to_array(String(param, 'hi, how are you|fine thanks'), separator='|')}}
```

## enumerate_with_last (https://www.tinybird.co/docs/forward/dev-reference/template-functions#enumerate-with-last)

Creates an iterable array, returning a boolean value that allows to check if the current element is the last element in the array. You can use it alongside the [split_to_array function](https://www.tinybird.co/docs/forward/dev-reference/template-functions#split_to_array).

## symbol (https://www.tinybird.co/docs/forward/dev-reference/template-functions#symbol)

Retrieves the value of a variable. The function accepts the following parameters:

`symbol(x, quote)`

For example:

enumerate_with_last function
```
%
SELECT
    {% for _last, _x in enumerate_with_last(split_to_array(attr, 'amount')) %}
        sum({{symbol(_x)}}) as {{symbol(_x)}}
        {% if not _last %}, {% end %}
    {% end %}
FROM my_table
```

## sql_and (https://www.tinybird.co/docs/forward/dev-reference/template-functions#sql-and)

Creates a list of "WHERE" clauses, along with "AND" separated filters, that checks if a field (<column>) is or isn't (<op>) in a list/tuple (<transform_type_function>).

The function accepts the following parameters:

`sql_and(<column>__<op>=<transform_type_function> [, ...] )`

- `<column>`: any column in the table.
- `<op>`: one of: `in`, `not_in`, `gt` (>), `lt` (<), `gte` (>=), `lte` (<=)
- `<transform_type_function>`: any of the transform type functions (`Array(param, 'Int8')`, `String(param)`, etc.). If one parameter isn't specified, then the filter is ignored.
For example:

sql_and function
```
%
SELECT *
FROM my_table
WHERE 1
{% if defined(param) or defined(param2_not_in) %}
    AND {{sql_and(
        param__in=Array(param, 'Int32', defined=False),
        param2__not_in=Array(param2_not_in, 'String', defined=False))}}
{% end %}
```
If this is queried with `param=1,2` and `param2_not_in=ab,bc,cd`, it translates to:

sql_and function - generated sql
```
SELECT *
FROM my_table
WHERE 1
    AND param  IN [1,2]
    AND param2 NOT IN ['ab','bc','cd']
```
If this is queried with `param=1,2` only, but `param2_not_in` isn't specified, it translates to:

sql_and function - generated sql param missing
```
SELECT *
FROM my_table
WHERE 1
    AND param  IN [1,2]
```

## Transform types functions (https://www.tinybird.co/docs/forward/dev-reference/template-functions#transform-types-functions)

The following functions validate the type of a template variable and cast it to the desired data type. They also provide a default value if no value is passed.

- `Boolean(x)`
- `DateTime64(x)`
- `DateTime(x)`
- `Date(x)`
- `Float32(x)`
- `Float64(x)`
- `Int8(x)`
- `Int16(x)`
- `Int32(x)`
- `Int64(x)`
- `Int128(x)`
- `Int256(x)`
- `UInt8(x)`
- `UInt16(x)`
- `UInt32(x)`
- `UInt64(x)`
- `UInt128(x)`
- `UInt256(x)`
- `String(x)`
- `Array(x)`
Each function accepts the following parameters:

`type(x, default, description=<description>, required=<true|false>)`

- `x`: the parameter or value.
- `default`: (optional) the default value to use if `x` is empty.
- `description`: (optional) the description of the value.
- `required`: (optional) whether the value is required.
For example, `Int32` in the following query, `lim` is the parameter to be cast to an `Int32`, `10` is the default value, and so on:

transform_type_functions
```
%
SELECT * FROM TR LIMIT {{Int32(lim, 10, description="Limit the number of rows in the response", required=False)}}
```

---

URL: https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Python SDK resources \| Tinybird"
theme-color: "#171612"
description: "Define Tinybird resources in Python with the Tinybird SDK."
inkeep:version: "forward"
---

# Python SDK resources (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#python-sdk-resources)

Use the Python SDK to define Tinybird resources as code. You can define Data Sources, Pipes, Endpoints, Materialized Views, Copy Pipes, Sink Pipes, and Connections in Python and sync them to Tinybird.

Supported Python versions for Python SDK workflows are 3.11, 3.12, and 3.13. Python 3.14 is not supported yet because these workflows rely on the Tinybird CLI.

## Project layout (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#project-layout)

A Python SDK project usually includes:

- `tinybird.config.json`
- `tb_project/resources.py`
- `tb_project/client.py`
`tinybird.config.json` points `include` at the folder or files where you define resources.

## Configure credentials (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#configure-credentials)

Create a `.env.local` file in your project root:

```
TINYBIRD_TOKEN=p.your_token_here
TINYBIRD_URL=https://api.tinybird.co
```

## Tokens (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#tokens)

Use `define_token` to declare reusable token names, then attach them to Data Sources or Pipes through each resource's `tokens` option.

```
from tinybird_sdk import define_datasource, define_endpoint, define_token, engine, node, t

events_append = define_token("events_append")
events_read = define_token("events_read")

events = define_datasource("events", {
    "schema": {
        "timestamp": t.date_time(),
        "payload": t.string(),
    },
    "engine": engine.merge_tree({"sorting_key": ["timestamp"]}),
    "tokens": [
        {"token": events_append, "scope": "APPEND"},
        {"token": events_read, "scope": "READ"},
    ],
})

events_api = define_endpoint("events_api", {
    "nodes": [
        node({
            "name": "latest",
            "sql": "SELECT timestamp, payload FROM events ORDER BY timestamp DESC LIMIT 100",
        }),
    ],
    "output": {
        "timestamp": t.date_time(),
        "payload": t.string(),
    },
    "tokens": [
        {"token": events_read, "scope": "READ"},
    ],
})
```
Token names must start with a letter or underscore and then use only alphanumeric characters or underscores. In Data Source token bindings, use `READ` or `APPEND` scopes. In Pipe or Endpoint token bindings, use `READ`.

## Datasources (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#datasources)

Define Data Sources with `define_datasource`:

```
from tinybird_sdk import define_datasource, engine, t

page_views = define_datasource("page_views", {
    "description": "Page view tracking data",
    "schema": {
        "timestamp": t.date_time(),
        "pathname": t.string(),
        "session_id": t.string(),
        "country": t.string().low_cardinality().nullable(),
    },
    "engine": engine.merge_tree({
        "sorting_key": ["pathname", "timestamp"],
    }),
})
```

### Column JSON paths (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#column-json-paths)

Use `column()` when you need custom JSONPath mappings:

```
from tinybird_sdk import column, define_datasource, engine, t

events = define_datasource("events", {
    "schema": {
        "event_id": column(t.string(), {"json_path": "$.id"}),
        "timestamp": column(t.date_time(), {"json_path": "$.ts"}),
    },
    "engine": engine.merge_tree({"sorting_key": ["event_id"]}),
})
```

## Endpoints (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#endpoints)

Define API endpoints with `define_endpoint`:

```
from tinybird_sdk import define_endpoint, node, p, t

top_pages = define_endpoint("top_pages", {
    "description": "Get the most visited pages",
    "params": {
        "start_date": p.date_time(),
        "end_date": p.date_time(),
        "limit": p.int32().optional(10),
    },
    "nodes": [
        node({
            "name": "aggregated",
            "sql": """
                SELECT pathname, count() AS views
                FROM page_views
                WHERE timestamp >= {{DateTime(start_date)}}
                  AND timestamp <= {{DateTime(end_date)}}
                GROUP BY pathname
                ORDER BY views DESC
                LIMIT {{Int32(limit, 10)}}
            """,
        }),
    ],
    "output": {
        "pathname": t.string(),
        "views": t.uint64(),
    },
})
```

## Pipes (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#pipes)

Define internal pipes with `define_pipe`:

```
from tinybird_sdk import define_pipe, node, p

filtered_events = define_pipe("filtered_events", {
    "params": {
        "start_date": p.date_time(),
        "end_date": p.date_time(),
    },
    "nodes": [
        node({
            "name": "filtered",
            "sql": """
                SELECT *
                FROM events
                WHERE timestamp >= {{DateTime(start_date)}}
                  AND timestamp <= {{DateTime(end_date)}}
            """,
        }),
    ],
})
```

## Materialized views (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#materialized-views)

Use `define_materialized_view` to populate derived Data Sources:

```
from tinybird_sdk import define_datasource, define_materialized_view, engine, node, t

daily_stats = define_datasource("daily_stats", {
    "schema": {
        "date": t.date(),
        "pathname": t.string(),
        "views": t.simple_aggregate_function("sum", t.uint64()),
    },
    "engine": engine.aggregating_merge_tree({
        "sorting_key": ["date", "pathname"],
    }),
})

daily_stats_mv = define_materialized_view("daily_stats_mv", {
    "datasource": daily_stats,
    "nodes": [
        node({
            "name": "aggregate",
            "sql": """
                SELECT toDate(timestamp) AS date, pathname, count() AS views
                FROM page_views
                GROUP BY date, pathname
            """,
        }),
    ],
})
```

## Copy pipes (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#copy-pipes)

Use `define_copy_pipe` for scheduled snapshots:

```
from tinybird_sdk import define_copy_pipe, node

daily_snapshot = define_copy_pipe("daily_snapshot", {
    "datasource": daily_stats,
    "copy_schedule": "0 0 * * *",
    "copy_mode": "append",
    "nodes": [
        node({
            "name": "snapshot",
            "sql": "SELECT * FROM daily_stats WHERE date = today() - 1",
        }),
    ],
})
```

## Sink pipes (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#sink-pipes)

Use `define_sink_pipe` to publish query results to Kafka or S3:

```
from tinybird_sdk import define_sink_pipe, node

kafka_events_sink = define_sink_pipe("kafka_events_sink", {
    "sink": {
        "connection": events_kafka,
        "topic": "events_export",
        "schedule": "@on-demand",
    },
    "nodes": [
        node({
            "name": "publish",
            "sql": "SELECT timestamp, payload FROM kafka_events",
        }),
    ],
})

s3_events_sink = define_sink_pipe("s3_events_sink", {
    "sink": {
        "connection": landing_s3,
        "bucket_uri": "s3://my-bucket/exports/",
        "file_template": "events_{date}",
        "format": "csv",
        "schedule": "@once",
    },
    "nodes": [
        node({
            "name": "export",
            "sql": "SELECT timestamp, session_id FROM events",
        }),
    ],
})
```

## Connections (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#connections)

Define connector connections with `define_kafka_connection`, `define_s3_connection`, `define_gcs_connection`, and `define_dynamodb_connection`:

```
from tinybird_sdk import (
    define_dynamodb_connection,
    define_gcs_connection,
    define_kafka_connection,
    define_s3_connection,
    secret,
)

events_kafka = define_kafka_connection("events_kafka", {
    "bootstrap_servers": "kafka.example.com:9092",
    "security_protocol": "SASL_SSL",
    "sasl_mechanism": "PLAIN",
    "key": secret("KAFKA_KEY"),
    "secret": secret("KAFKA_SECRET"),
})

landing_s3 = define_s3_connection("landing_s3", {
    "region": "eu-west-1",
    "arn": secret("AWS_ROLE_ARN"),
})

landing_gcs = define_gcs_connection("landing_gcs", {
    "service_account_credentials_json": secret("GCS_SERVICE_ACCOUNT_CREDENTIALS_JSON"),
})

events_dynamodb = define_dynamodb_connection("events_dynamodb", {
    "region": "us-east-1",
    "arn": secret("DYNAMODB_ROLE_ARN"),
})
```
A DynamoDB connection takes a `region` and an `arn`. The `arn` is the IAM role Tinybird assumes to read your table, its stream, and the S3 export bucket. Both fields are required.

## Datasource ingestion config (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#datasource-ingestion-config)

You can attach Kafka, S3, GCS, or DynamoDB ingestion directly to a Data Source:

```
from tinybird_sdk import define_datasource, engine, t

kafka_events = define_datasource("kafka_events", {
    "schema": {
        "timestamp": t.date_time(),
        "payload": t.string(),
    },
    "engine": engine.merge_tree({"sorting_key": ["timestamp"]}),
    "kafka": {
        "connection": events_kafka,
        "topic": "events",
        "group_id": "events-consumer",
    },
})
```

### DynamoDB (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#dynamodb)

The DynamoDB connector mirrors an Amazon DynamoDB table into Tinybird: an initial backfill via a Point-in-Time Recovery export to S3, followed by continuous Change Data Capture from DynamoDB Streams. See the [DynamoDB connector](/docs/forward/ingest-data/connectors/dynamodb) page for the table requirements and IAM setup.

Defining the connection in code isn't enough. Tinybird assumes an IAM role in your own AWS account, and the role's trust policy must include a Workspace-specific external ID that only Tinybird generates. Run `tinybird connection create dynamodb` to get the external ID, the `Principal` account, and the policy JSON. Then create the role in AWS and store its ARN as the `DYNAMODB_ROLE_ARN` secret. Without the correct external ID, `deploy` fails with a `403 ... external ID` error. See the [DynamoDB connector](/docs/forward/ingest-data/connectors/dynamodb) page for the full setup.

Each row in the Data Source represents a *change* to your table, not the current state. To keep its size under control, DynamoDB Data Sources use the <a href="/docs/sql-reference/engines/replacingmergetree">`ReplacingMergeTree` engine</a>. See [Querying the data](https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#querying-the-data) for considerations.

```
from tinybird_sdk import column, define_datasource, engine, t

orders = define_datasource("orders", {
    "schema": {
        "id": column(t.string(), {"json_path": "$.Item.id"}),
        "_record": column(t.string(), {"json_path": "$.NewImage"}),
        "_timestamp": column(t.date_time64(3), {"json_path": "$.ApproximateCreationDateTime"}),
        "_event_name": column(t.string().low_cardinality(), {"json_path": "$.eventName"}),
        "_is_deleted": column(t.uint8(), {"json_path": "$._is_deleted"}),
    },
    "engine": engine.replacing_merge_tree({
        "sorting_key": ["id"],
        "ver": "_timestamp",
        "is_deleted": "_is_deleted",
    }),
    "dynamodb": {
        "connection": events_dynamodb,
        "table_arn": "arn:aws:dynamodb:us-east-1:123456789012:table/orders",
        "export_bucket": "s3://my-tinybird-dynamodb-exports",
    },
})
```

#### Set the role ARN secret before deploying (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#set-the-role-arn-secret-before-deploying)

`secret("DYNAMODB_ROLE_ARN")` resolves to a [secret](/docs/forward/dev-reference/commands/tb-secret) that must already exist in the target Workspace. Set it before you deploy:

```
tinybird --cloud secret set DYNAMODB_ROLE_ARN "arn:aws:iam::123456789012:role/TinybirdRole"
tinybird --cloud secret ls
```
CLI commands such as `secret set` and `deploy` authenticate separately from the runtime client. They use the Workspace admin Token from `tinybird login` (or an explicit `--token` / `--host`), not the `TINYBIRD_TOKEN` in `.env.local`. `.env.local` feeds only the `Tinybird(...)` runtime client.

If `deploy` or `secret` reports `Forbidden: invalid authentication token`, or runs against an unexpected Workspace, the CLI is using a stale or foreign Token rather than your `.env.local`. Re-run `tinybird login`, or pass the Workspace admin Token explicitly:

```
tinybird --cloud --host https://api.tinybird.co \
  --token "<workspace-admin-token>" deploy --check
```
`secret set` and `deploy` are Workspace operations, so they need the Workspace admin Token, not an Organization or user Token.

## Runtime client (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#runtime-client)

Create a typed runtime client with `Tinybird`:

```
from tinybird_sdk import Tinybird
from .resources import page_views, top_pages

tinybird = Tinybird({
    "datasources": {"page_views": page_views},
    "pipes": {"top_pages": top_pages},
})
```
Then ingest and query:

```
tinybird.page_views.ingest({
    "timestamp": "2026-01-01 10:00:00",
    "pathname": "/home",
    "session_id": "abc123",
})

result = tinybird.top_pages.query({
    "start_date": "2026-01-01 00:00:00",
    "end_date": "2026-01-02 00:00:00",
    "limit": 5,
})
```

## Next steps (https://www.tinybird.co/docs/forward/dev-reference/python-sdk-resources#next-steps)

- Review CLI commands: [Python SDK CLI commands](/docs/forward/dev-reference/commands/python-sdk-cli)
- Follow the setup and deployment flow: [Quick start: Python SDK](/docs/forward/quickstarts/python-sdk)

---

URL: https://www.tinybird.co/docs/forward/dev-reference/list-of-errors
Last update: 2025-05-07T10:44:34.000Z
Content:
---
title: "List of API endpoint database errors \| Tinybird"
theme-color: "#171612"
description: "The following list contains all internal database errors that an API endpoint might return, and their numbers."
inkeep:version: "forward"
---

# List of internal database errors (https://www.tinybird.co/docs/forward/dev-reference/list-of-errors#list-of-internal-database-errors)

API endpoint responses have an additional HTTP header, `X-DB-Exception-Code`, where you can check the internal database error, reported as a stringified number.

The following list contains all internal database errors and their numbers:

- `UNSUPPORTED_METHOD = "1"`
- `UNSUPPORTED_PARAMETER = "2"`
- `UNEXPECTED_END_OF_FILE = "3"`
- `EXPECTED_END_OF_FILE = "4"`
- `CANNOT_PARSE_TEXT = "6"`
- `INCORRECT_NUMBER_OF_COLUMNS = "7"`
- `THERE_IS_NO_COLUMN = "8"`
- `SIZES_OF_COLUMNS_DOESNT_MATCH = "9"`
- `NOT_FOUND_COLUMN_IN_BLOCK = "10"`
- `POSITION_OUT_OF_BOUND = "11"`
- `PARAMETER_OUT_OF_BOUND = "12"`
- `SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH = "13"`
- `DUPLICATE_COLUMN = "15"`
- `NO_SUCH_COLUMN_IN_TABLE = "16"`
- `DELIMITER_IN_STRING_LITERAL_DOESNT_MATCH = "17"`
- `CANNOT_INSERT_ELEMENT_INTO_CONSTANT_COLUMN = "18"`
- `SIZE_OF_FIXED_STRING_DOESNT_MATCH = "19"`
- `NUMBER_OF_COLUMNS_DOESNT_MATCH = "20"`
- `CANNOT_READ_ALL_DATA_FROM_TAB_SEPARATED_INPUT = "21"`
- `CANNOT_PARSE_ALL_VALUE_FROM_TAB_SEPARATED_INPUT = "22"`
- `CANNOT_READ_FROM_ISTREAM = "23"`
- `CANNOT_WRITE_TO_OSTREAM = "24"`
- `CANNOT_PARSE_ESCAPE_SEQUENCE = "25"`
- `CANNOT_PARSE_QUOTED_STRING = "26"`
- `CANNOT_PARSE_INPUT_ASSERTION_FAILED = "27"`
- `CANNOT_PRINT_FLOAT_OR_DOUBLE_NUMBER = "28"`
- `CANNOT_PRINT_INTEGER = "29"`
- `CANNOT_READ_SIZE_OF_COMPRESSED_CHUNK = "30"`
- `CANNOT_READ_COMPRESSED_CHUNK = "31"`
- `ATTEMPT_TO_READ_AFTER_EOF = "32"`
- `CANNOT_READ_ALL_DATA = "33"`
- `TOO_MANY_ARGUMENTS_FOR_FUNCTION = "34"`
- `TOO_FEW_ARGUMENTS_FOR_FUNCTION = "35"`
- `BAD_ARGUMENTS = "36"`
- `UNKNOWN_ELEMENT_IN_AST = "37"`
- `CANNOT_PARSE_DATE = "38"`
- `TOO_LARGE_SIZE_COMPRESSED = "39"`
- `CHECKSUM_DOESNT_MATCH = "40"`
- `CANNOT_PARSE_DATETIME = "41"`
- `NUMBER_OF_ARGUMENTS_DOESNT_MATCH = "42"`
- `ILLEGAL_TYPE_OF_ARGUMENT = "43"`
- `ILLEGAL_COLUMN = "44"`
- `ILLEGAL_NUMBER_OF_RESULT_COLUMNS = "45"`
- `UNKNOWN_FUNCTION = "46"`
- `UNKNOWN_IDENTIFIER = "47"`
- `NOT_IMPLEMENTED = "48"`
- `LOGICAL_ERROR = "49"`
- `UNKNOWN_TYPE = "50"`
- `EMPTY_LIST_OF_COLUMNS_QUERIED = "51"`
- `COLUMN_QUERIED_MORE_THAN_ONCE = "52"`
- `TYPE_MISMATCH = "53"`
- `STORAGE_DOESNT_ALLOW_PARAMETERS = "54"`
- `STORAGE_REQUIRES_PARAMETER = "55"`
- `UNKNOWN_STORAGE = "56"`
- `TABLE_ALREADY_EXISTS = "57"`
- `TABLE_METADATA_ALREADY_EXISTS = "58"`
- `ILLEGAL_TYPE_OF_COLUMN_FOR_FILTER = "59"`
- `UNKNOWN_TABLE = "60"`
- `ONLY_FILTER_COLUMN_IN_BLOCK = "61"`
- `SYNTAX_ERROR = "62"`
- `UNKNOWN_AGGREGATE_FUNCTION = "63"`
- `CANNOT_READ_AGGREGATE_FUNCTION_FROM_TEXT = "64"`
- `CANNOT_WRITE_AGGREGATE_FUNCTION_AS_TEXT = "65"`
- `NOT_A_COLUMN = "66"`
- `ILLEGAL_KEY_OF_AGGREGATION = "67"`
- `CANNOT_GET_SIZE_OF_FIELD = "68"`
- `ARGUMENT_OUT_OF_BOUND = "69"`
- `CANNOT_CONVERT_TYPE = "70"`
- `CANNOT_WRITE_AFTER_END_OF_BUFFER = "71"`
- `CANNOT_PARSE_NUMBER = "72"`
- `UNKNOWN_FORMAT = "73"`
- `CANNOT_READ_FROM_FILE_DESCRIPTOR = "74"`
- `CANNOT_WRITE_TO_FILE_DESCRIPTOR = "75"`
- `CANNOT_OPEN_FILE = "76"`
- `CANNOT_CLOSE_FILE = "77"`
- `UNKNOWN_TYPE_OF_QUERY = "78"`
- `INCORRECT_FILE_NAME = "79"`
- `INCORRECT_QUERY = "80"`
- `UNKNOWN_DATABASE = "81"`
- `DATABASE_ALREADY_EXISTS = "82"`
- `DIRECTORY_DOESNT_EXIST = "83"`
- `DIRECTORY_ALREADY_EXISTS = "84"`
- `FORMAT_IS_NOT_SUITABLE_FOR_INPUT = "85"`
- `RECEIVED_ERROR_FROM_REMOTE_IO_SERVER = "86"`
- `CANNOT_SEEK_THROUGH_FILE = "87"`
- `CANNOT_TRUNCATE_FILE = "88"`
- `UNKNOWN_COMPRESSION_METHOD = "89"`
- `EMPTY_LIST_OF_COLUMNS_PASSED = "90"`
- `SIZES_OF_MARKS_FILES_ARE_INCONSISTENT = "91"`
- `EMPTY_DATA_PASSED = "92"`
- `UNKNOWN_AGGREGATED_DATA_VARIANT = "93"`
- `CANNOT_MERGE_DIFFERENT_AGGREGATED_DATA_VARIANTS = "94"`
- `CANNOT_READ_FROM_SOCKET = "95"`
- `CANNOT_WRITE_TO_SOCKET = "96"`
- `CANNOT_READ_ALL_DATA_FROM_CHUNKED_INPUT = "97"`
- `CANNOT_WRITE_TO_EMPTY_BLOCK_OUTPUT_STREAM = "98"`
- `UNKNOWN_PACKET_FROM_CLIENT = "99"`
- `UNKNOWN_PACKET_FROM_SERVER = "100"`
- `UNEXPECTED_PACKET_FROM_CLIENT = "101"`
- `UNEXPECTED_PACKET_FROM_SERVER = "102"`
- `RECEIVED_DATA_FOR_WRONG_QUERY_ID = "103"`
- `TOO_SMALL_BUFFER_SIZE = "104"`
- `CANNOT_READ_HISTORY = "105"`
- `CANNOT_APPEND_HISTORY = "106"`
- `FILE_DOESNT_EXIST = "107"`
- `NO_DATA_TO_INSERT = "108"`
- `CANNOT_BLOCK_SIGNAL = "109"`
- `CANNOT_UNBLOCK_SIGNAL = "110"`
- `CANNOT_MANIPULATE_SIGSET = "111"`
- `CANNOT_WAIT_FOR_SIGNAL = "112"`
- `THERE_IS_NO_SESSION = "113"`
- `CANNOT_CLOCK_GETTIME = "114"`
- `UNKNOWN_SETTING = "115"`
- `THERE_IS_NO_DEFAULT_VALUE = "116"`
- `INCORRECT_DATA = "117"`
- `ENGINE_REQUIRED = "119"`
- `CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE = "120"`
- `UNSUPPORTED_JOIN_KEYS = "121"`
- `INCOMPATIBLE_COLUMNS = "122"`
- `UNKNOWN_TYPE_OF_AST_NODE = "123"`
- `INCORRECT_ELEMENT_OF_SET = "124"`
- `INCORRECT_RESULT_OF_SCALAR_SUBQUERY = "125"`
- `CANNOT_GET_RETURN_TYPE = "126"`
- `ILLEGAL_INDEX = "127"`
- `TOO_LARGE_ARRAY_SIZE = "128"`
- `FUNCTION_IS_SPECIAL = "129"`
- `CANNOT_READ_ARRAY_FROM_TEXT = "130"`
- `TOO_LARGE_STRING_SIZE = "131"`
- `AGGREGATE_FUNCTION_DOESNT_ALLOW_PARAMETERS = "133"`
- `PARAMETERS_TO_AGGREGATE_FUNCTIONS_MUST_BE_LITERALS = "134"`
- `ZERO_ARRAY_OR_TUPLE_INDEX = "135"`
- `UNKNOWN_ELEMENT_IN_CONFIG = "137"`
- `EXCESSIVE_ELEMENT_IN_CONFIG = "138"`
- `NO_ELEMENTS_IN_CONFIG = "139"`
- `ALL_REQUESTED_COLUMNS_ARE_MISSING = "140"`
- `SAMPLING_NOT_SUPPORTED = "141"`
- `NOT_FOUND_NODE = "142"`
- `FOUND_MORE_THAN_ONE_NODE = "143"`
- `FIRST_DATE_IS_BIGGER_THAN_LAST_DATE = "144"`
- `UNKNOWN_OVERFLOW_MODE = "145"`
- `QUERY_SECTION_DOESNT_MAKE_SENSE = "146"`
- `NOT_FOUND_FUNCTION_ELEMENT_FOR_AGGREGATE = "147"`
- `NOT_FOUND_RELATION_ELEMENT_FOR_CONDITION = "148"`
- `NOT_FOUND_RHS_ELEMENT_FOR_CONDITION = "149"`
- `EMPTY_LIST_OF_ATTRIBUTES_PASSED = "150"`
- `INDEX_OF_COLUMN_IN_SORT_CLAUSE_IS_OUT_OF_RANGE = "151"`
- `UNKNOWN_DIRECTION_OF_SORTING = "152"`
- `ILLEGAL_DIVISION = "153"`
- `AGGREGATE_FUNCTION_NOT_APPLICABLE = "154"`
- `UNKNOWN_RELATION = "155"`
- `DICTIONARIES_WAS_NOT_LOADED = "156"`
- `ILLEGAL_OVERFLOW_MODE = "157"`
- `TOO_MANY_ROWS = "158"`
- `TIMEOUT_EXCEEDED = "159"`
- `TOO_SLOW = "160"`
- `TOO_MANY_COLUMNS = "161"`
- `TOO_DEEP_SUBQUERIES = "162"`
- `TOO_DEEP_PIPELINE = "163"`
- `READONLY = "164"`
- `TOO_MANY_TEMPORARY_COLUMNS = "165"`
- `TOO_MANY_TEMPORARY_NON_CONST_COLUMNS = "166"`
- `TOO_DEEP_AST = "167"`
- `TOO_BIG_AST = "168"`
- `BAD_TYPE_OF_FIELD = "169"`
- `BAD_GET = "170"`
- `CANNOT_CREATE_DIRECTORY = "172"`
- `CANNOT_ALLOCATE_MEMORY = "173"`
- `CYCLIC_ALIASES = "174"`
- `CHUNK_NOT_FOUND = "176"`
- `DUPLICATE_CHUNK_NAME = "177"`
- `MULTIPLE_ALIASES_FOR_EXPRESSION = "178"`
- `MULTIPLE_EXPRESSIONS_FOR_ALIAS = "179"`
- `THERE_IS_NO_PROFILE = "180"`
- `ILLEGAL_FINAL = "181"`
- `ILLEGAL_PREWHERE = "182"`
- `UNEXPECTED_EXPRESSION = "183"`
- `ILLEGAL_AGGREGATION = "184"`
- `UNSUPPORTED_MYISAM_BLOCK_TYPE = "185"`
- `UNSUPPORTED_COLLATION_LOCALE = "186"`
- `COLLATION_COMPARISON_FAILED = "187"`
- `UNKNOWN_ACTION = "188"`
- `TABLE_MUST_NOT_BE_CREATED_MANUALLY = "189"`
- `SIZES_OF_ARRAYS_DOESNT_MATCH = "190"`
- `SET_SIZE_LIMIT_EXCEEDED = "191"`
- `UNKNOWN_USER = "192"`
- `WRONG_PASSWORD = "193"`
- `REQUIRED_PASSWORD = "194"`
- `IP_ADDRESS_NOT_ALLOWED = "195"`
- `UNKNOWN_ADDRESS_PATTERN_TYPE = "196"`
- `SERVER_REVISION_IS_TOO_OLD = "197"`
- `DNS_ERROR = "198"`
- `UNKNOWN_QUOTA = "199"`
- `QUOTA_DOESNT_ALLOW_KEYS = "200"`
- `QUOTA_EXCEEDED = "201"`
- `TOO_MANY_SIMULTANEOUS_QUERIES = "202"`
- `NO_FREE_CONNECTION = "203"`
- `CANNOT_FSYNC = "204"`
- `NESTED_TYPE_TOO_DEEP = "205"`
- `ALIAS_REQUIRED = "206"`
- `AMBIGUOUS_IDENTIFIER = "207"`
- `EMPTY_NESTED_TABLE = "208"`
- `SOCKET_TIMEOUT = "209"`
- `NETWORK_ERROR = "210"`
- `EMPTY_QUERY = "211"`
- `UNKNOWN_LOAD_BALANCING = "212"`
- `UNKNOWN_TOTALS_MODE = "213"`
- `CANNOT_STATVFS = "214"`
- `NOT_AN_AGGREGATE = "215"`
- `QUERY_WITH_SAME_ID_IS_ALREADY_RUNNING = "216"`
- `CLIENT_HAS_CONNECTED_TO_WRONG_PORT = "217"`
- `TABLE_IS_DROPPED = "218"`
- `DATABASE_NOT_EMPTY = "219"`
- `DUPLICATE_INTERSERVER_IO_ENDPOINT = "220"`
- `NO_SUCH_INTERSERVER_IO_ENDPOINT = "221"`
- `ADDING_REPLICA_TO_NON_EMPTY_TABLE = "222"`
- `UNEXPECTED_AST_STRUCTURE = "223"`
- `REPLICA_IS_ALREADY_ACTIVE = "224"`
- `NO_ZOOKEEPER = "225"`
- `NO_FILE_IN_DATA_PART = "226"`
- `UNEXPECTED_FILE_IN_DATA_PART = "227"`
- `BAD_SIZE_OF_FILE_IN_DATA_PART = "228"`
- `QUERY_IS_TOO_LARGE = "229"`
- `NOT_FOUND_EXPECTED_DATA_PART = "230"`
- `TOO_MANY_UNEXPECTED_DATA_PARTS = "231"`
- `NO_SUCH_DATA_PART = "232"`
- `BAD_DATA_PART_NAME = "233"`
- `NO_REPLICA_HAS_PART = "234"`
- `DUPLICATE_DATA_PART = "235"`
- `ABORTED = "236"`
- `NO_REPLICA_NAME_GIVEN = "237"`
- `FORMAT_VERSION_TOO_OLD = "238"`
- `CANNOT_MUNMAP = "239"`
- `CANNOT_MREMAP = "240"`
- `MEMORY_LIMIT_EXCEEDED = "241"`
- `TABLE_IS_READ_ONLY = "242"`
- `NOT_ENOUGH_SPACE = "243"`
- `UNEXPECTED_ZOOKEEPER_ERROR = "244"`
- `CORRUPTED_DATA = "246"`
- `INCORRECT_MARK = "247"`
- `INVALID_PARTITION_VALUE = "248"`
- `NOT_ENOUGH_BLOCK_NUMBERS = "250"`
- `NO_SUCH_REPLICA = "251"`
- `TOO_MANY_PARTS = "252"`
- `REPLICA_IS_ALREADY_EXIST = "253"`
- `NO_ACTIVE_REPLICAS = "254"`
- `TOO_MANY_RETRIES_TO_FETCH_PARTS = "255"`
- `PARTITION_ALREADY_EXISTS = "256"`
- `PARTITION_DOESNT_EXIST = "257"`
- `UNION_ALL_RESULT_STRUCTURES_MISMATCH = "258"`
- `CLIENT_OUTPUT_FORMAT_SPECIFIED = "260"`
- `UNKNOWN_BLOCK_INFO_FIELD = "261"`
- `BAD_COLLATION = "262"`
- `CANNOT_COMPILE_CODE = "263"`
- `INCOMPATIBLE_TYPE_OF_JOIN = "264"`
- `NO_AVAILABLE_REPLICA = "265"`
- `MISMATCH_REPLICAS_DATA_SOURCES = "266"`
- `STORAGE_DOESNT_SUPPORT_PARALLEL_REPLICAS = "267"`
- `CPUID_ERROR = "268"`
- `INFINITE_LOOP = "269"`
- `CANNOT_COMPRESS = "270"`
- `CANNOT_DECOMPRESS = "271"`
- `CANNOT_IO_SUBMIT = "272"`
- `CANNOT_IO_GETEVENTS = "273"`
- `AIO_READ_ERROR = "274"`
- `AIO_WRITE_ERROR = "275"`
- `INDEX_NOT_USED = "277"`
- `ALL_CONNECTION_TRIES_FAILED = "279"`
- `NO_AVAILABLE_DATA = "280"`
- `DICTIONARY_IS_EMPTY = "281"`
- `INCORRECT_INDEX = "282"`
- `UNKNOWN_DISTRIBUTED_PRODUCT_MODE = "283"`
- `WRONG_GLOBAL_SUBQUERY = "284"`
- `TOO_FEW_LIVE_REPLICAS = "285"`
- `UNSATISFIED_QUORUM_FOR_PREVIOUS_WRITE = "286"`
- `UNKNOWN_FORMAT_VERSION = "287"`
- `DISTRIBUTED_IN_JOIN_SUBQUERY_DENIED = "288"`
- `REPLICA_IS_NOT_IN_QUORUM = "289"`
- `LIMIT_EXCEEDED = "290"`
- `DATABASE_ACCESS_DENIED = "291"`
- `MONGODB_CANNOT_AUTHENTICATE = "293"`
- `INVALID_BLOCK_EXTRA_INFO = "294"`
- `RECEIVED_EMPTY_DATA = "295"`
- `NO_REMOTE_SHARD_FOUND = "296"`
- `SHARD_HAS_NO_CONNECTIONS = "297"`
- `CANNOT_PIPE = "298"`
- `CANNOT_FORK = "299"`
- `CANNOT_DLSYM = "300"`
- `CANNOT_CREATE_CHILD_PROCESS = "301"`
- `CHILD_WAS_NOT_EXITED_NORMALLY = "302"`
- `CANNOT_SELECT = "303"`
- `CANNOT_WAITPID = "304"`
- `TABLE_WAS_NOT_DROPPED = "305"`
- `TOO_DEEP_RECURSION = "306"`
- `TOO_MANY_BYTES = "307"`
- `UNEXPECTED_NODE_IN_ZOOKEEPER = "308"`
- `FUNCTION_CANNOT_HAVE_PARAMETERS = "309"`
- `INVALID_SHARD_WEIGHT = "317"`
- `INVALID_CONFIG_PARAMETER = "318"`
- `UNKNOWN_STATUS_OF_INSERT = "319"`
- `VALUE_IS_OUT_OF_RANGE_OF_DATA_TYPE = "321"`
- `BARRIER_TIMEOUT = "335"`
- `UNKNOWN_DATABASE_ENGINE = "336"`
- `DDL_GUARD_IS_ACTIVE = "337"`
- `UNFINISHED = "341"`
- `METADATA_MISMATCH = "342"`
- `SUPPORT_IS_DISABLED = "344"`
- `TABLE_DIFFERS_TOO_MUCH = "345"`
- `CANNOT_CONVERT_CHARSET = "346"`
- `CANNOT_LOAD_CONFIG = "347"`
- `CANNOT_INSERT_NULL_IN_ORDINARY_COLUMN = "349"`
- `INCOMPATIBLE_SOURCE_TABLES = "350"`
- `AMBIGUOUS_TABLE_NAME = "351"`
- `AMBIGUOUS_COLUMN_NAME = "352"`
- `INDEX_OF_POSITIONAL_ARGUMENT_IS_OUT_OF_RANGE = "353"`
- `ZLIB_INFLATE_FAILED = "354"`
- `ZLIB_DEFLATE_FAILED = "355"`
- `BAD_LAMBDA = "356"`
- `RESERVED_IDENTIFIER_NAME = "357"`
- `INTO_OUTFILE_NOT_ALLOWED = "358"`
- `TABLE_SIZE_EXCEEDS_MAX_DROP_SIZE_LIMIT = "359"`
- `CANNOT_CREATE_CHARSET_CONVERTER = "360"`
- `SEEK_POSITION_OUT_OF_BOUND = "361"`
- `CURRENT_WRITE_BUFFER_IS_EXHAUSTED = "362"`
- `CANNOT_CREATE_IO_BUFFER = "363"`
- `RECEIVED_ERROR_TOO_MANY_REQUESTS = "364"`
- `SIZES_OF_NESTED_COLUMNS_ARE_INCONSISTENT = "366"`
- `TOO_MANY_FETCHES = "367"`
- `ALL_REPLICAS_ARE_STALE = "369"`
- `DATA_TYPE_CANNOT_BE_USED_IN_TABLES = "370"`
- `INCONSISTENT_CLUSTER_DEFINITION = "371"`
- `SESSION_NOT_FOUND = "372"`
- `SESSION_IS_LOCKED = "373"`
- `INVALID_SESSION_TIMEOUT = "374"`
- `CANNOT_DLOPEN = "375"`
- `CANNOT_PARSE_UUID = "376"`
- `ILLEGAL_SYNTAX_FOR_DATA_TYPE = "377"`
- `DATA_TYPE_CANNOT_HAVE_ARGUMENTS = "378"`
- `UNKNOWN_STATUS_OF_DISTRIBUTED_DDL_TASK = "379"`
- `CANNOT_KILL = "380"`
- `HTTP_LENGTH_REQUIRED = "381"`
- `CANNOT_LOAD_CATBOOST_MODEL = "382"`
- `CANNOT_APPLY_CATBOOST_MODEL = "383"`
- `PART_IS_TEMPORARILY_LOCKED = "384"`
- `MULTIPLE_STREAMS_REQUIRED = "385"`
- `NO_COMMON_TYPE = "386"`
- `DICTIONARY_ALREADY_EXISTS = "387"`
- `CANNOT_ASSIGN_OPTIMIZE = "388"`
- `INSERT_WAS_DEDUPLICATED = "389"`
- `CANNOT_GET_CREATE_TABLE_QUERY = "390"`
- `EXTERNAL_LIBRARY_ERROR = "391"`
- `QUERY_IS_PROHIBITED = "392"`
- `THERE_IS_NO_QUERY = "393"`
- `QUERY_WAS_CANCELLED = "394"`
- `FUNCTION_THROW_IF_VALUE_IS_NON_ZERO = "395"`
- `TOO_MANY_ROWS_OR_BYTES = "396"`
- `QUERY_IS_NOT_SUPPORTED_IN_MATERIALIZED_VIEW = "397"`
- `UNKNOWN_MUTATION_COMMAND = "398"`
- `FORMAT_IS_NOT_SUITABLE_FOR_OUTPUT = "399"`
- `CANNOT_STAT = "400"`
- `FEATURE_IS_NOT_ENABLED_AT_BUILD_TIME = "401"`
- `CANNOT_IOSETUP = "402"`
- `INVALID_JOIN_ON_EXPRESSION = "403"`
- `BAD_ODBC_CONNECTION_STRING = "404"`
- `PARTITION_SIZE_EXCEEDS_MAX_DROP_SIZE_LIMIT = "405"`
- `TOP_AND_LIMIT_TOGETHER = "406"`
- `DECIMAL_OVERFLOW = "407"`
- `BAD_REQUEST_PARAMETER = "408"`
- `EXTERNAL_EXECUTABLE_NOT_FOUND = "409"`
- `EXTERNAL_SERVER_IS_NOT_RESPONDING = "410"`
- `PTHREAD_ERROR = "411"`
- `NETLINK_ERROR = "412"`
- `CANNOT_SET_SIGNAL_HANDLER = "413"`
- `ALL_REPLICAS_LOST = "415"`
- `REPLICA_STATUS_CHANGED = "416"`
- `EXPECTED_ALL_OR_ANY = "417"`
- `UNKNOWN_JOIN = "418"`
- `MULTIPLE_ASSIGNMENTS_TO_COLUMN = "419"`
- `CANNOT_UPDATE_COLUMN = "420"`
- `CANNOT_ADD_DIFFERENT_AGGREGATE_STATES = "421"`
- `UNSUPPORTED_URI_SCHEME = "422"`
- `CANNOT_GETTIMEOFDAY = "423"`
- `CANNOT_LINK = "424"`
- `SYSTEM_ERROR = "425"`
- `CANNOT_COMPILE_REGEXP = "427"`
- `UNKNOWN_LOG_LEVEL = "428"`
- `FAILED_TO_GETPWUID = "429"`
- `MISMATCHING_USERS_FOR_PROCESS_AND_DATA = "430"`
- `ILLEGAL_SYNTAX_FOR_CODEC_TYPE = "431"`
- `UNKNOWN_CODEC = "432"`
- `ILLEGAL_CODEC_PARAMETER = "433"`
- `CANNOT_PARSE_PROTOBUF_SCHEMA = "434"`
- `NO_COLUMN_SERIALIZED_TO_REQUIRED_PROTOBUF_FIELD = "435"`
- `PROTOBUF_BAD_CAST = "436"`
- `PROTOBUF_FIELD_NOT_REPEATED = "437"`
- `DATA_TYPE_CANNOT_BE_PROMOTED = "438"`
- `CANNOT_SCHEDULE_TASK = "439"`
- `INVALID_LIMIT_EXPRESSION = "440"`
- `CANNOT_PARSE_DOMAIN_VALUE_FROM_STRING = "441"`
- `BAD_DATABASE_FOR_TEMPORARY_TABLE = "442"`
- `NO_COLUMNS_SERIALIZED_TO_PROTOBUF_FIELDS = "443"`
- `UNKNOWN_PROTOBUF_FORMAT = "444"`
- `CANNOT_MPROTECT = "445"`
- `FUNCTION_NOT_ALLOWED = "446"`
- `HYPERSCAN_CANNOT_SCAN_TEXT = "447"`
- `BROTLI_READ_FAILED = "448"`
- `BROTLI_WRITE_FAILED = "449"`
- `BAD_TTL_EXPRESSION = "450"`
- `BAD_TTL_FILE = "451"`
- `SETTING_CONSTRAINT_VIOLATION = "452"`
- `MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES = "453"`
- `OPENSSL_ERROR = "454"`
- `SUSPICIOUS_TYPE_FOR_LOW_CARDINALITY = "455"`
- `UNKNOWN_QUERY_PARAMETER = "456"`
- `BAD_QUERY_PARAMETER = "457"`
- `CANNOT_UNLINK = "458"`
- `CANNOT_SET_THREAD_PRIORITY = "459"`
- `CANNOT_CREATE_TIMER = "460"`
- `CANNOT_SET_TIMER_PERIOD = "461"`
- `CANNOT_DELETE_TIMER = "462"`
- `CANNOT_FCNTL = "463"`
- `CANNOT_PARSE_ELF = "464"`
- `CANNOT_PARSE_DWARF = "465"`
- `INSECURE_PATH = "466"`
- `CANNOT_PARSE_BOOL = "467"`
- `CANNOT_PTHREAD_ATTR = "468"`
- `VIOLATED_CONSTRAINT = "469"`
- `QUERY_IS_NOT_SUPPORTED_IN_LIVE_VIEW = "470"`
- `INVALID_SETTING_VALUE = "471"`
- `READONLY_SETTING = "472"`
- `DEADLOCK_AVOIDED = "473"`
- `INVALID_TEMPLATE_FORMAT = "474"`
- `INVALID_WITH_FILL_EXPRESSION = "475"`
- `WITH_TIES_WITHOUT_ORDER_BY = "476"`
- `INVALID_USAGE_OF_INPUT = "477"`
- `UNKNOWN_POLICY = "478"`
- `UNKNOWN_DISK = "479"`
- `UNKNOWN_PROTOCOL = "480"`
- `PATH_ACCESS_DENIED = "481"`
- `DICTIONARY_ACCESS_DENIED = "482"`
- `TOO_MANY_REDIRECTS = "483"`
- `INTERNAL_REDIS_ERROR = "484"`
- `SCALAR_ALREADY_EXISTS = "485"`
- `CANNOT_GET_CREATE_DICTIONARY_QUERY = "487"`
- `UNKNOWN_DICTIONARY = "488"`
- `INCORRECT_DICTIONARY_DEFINITION = "489"`
- `CANNOT_FORMAT_DATETIME = "490"`
- `UNACCEPTABLE_URL = "491"`
- `ACCESS_ENTITY_NOT_FOUND = "492"`
- `ACCESS_ENTITY_ALREADY_EXISTS = "493"`
- `ACCESS_ENTITY_FOUND_DUPLICATES = "494"`
- `ACCESS_STORAGE_READONLY = "495"`
- `QUOTA_REQUIRES_CLIENT_KEY = "496"`
- `ACCESS_DENIED = "497"`
- `LIMIT_BY_WITH_TIES_IS_NOT_SUPPORTED = "498"`
- S3_ERROR = "499"
- `AZURE_BLOB_STORAGE_ERROR = "500"`
- `CANNOT_CREATE_DATABASE = "501"`
- `CANNOT_SIGQUEUE = "502"`
- `AGGREGATE_FUNCTION_THROW = "503"`
- `FILE_ALREADY_EXISTS = "504"`
- `CANNOT_DELETE_DIRECTORY = "505"`
- `UNEXPECTED_ERROR_CODE = "506"`
- `UNABLE_TO_SKIP_UNUSED_SHARDS = "507"`
- `UNKNOWN_ACCESS_TYPE = "508"`
- `INVALID_GRANT = "509"`
- `CACHE_DICTIONARY_UPDATE_FAIL = "510"`
- `UNKNOWN_ROLE = "511"`
- `SET_NON_GRANTED_ROLE = "512"`
- `UNKNOWN_PART_TYPE = "513"`
- `ACCESS_STORAGE_FOR_INSERTION_NOT_FOUND = "514"`
- `INCORRECT_ACCESS_ENTITY_DEFINITION = "515"`
- `AUTHENTICATION_FAILED = "516"`
- `CANNOT_ASSIGN_ALTER = "517"`
- `CANNOT_COMMIT_OFFSET = "518"`
- `NO_REMOTE_SHARD_AVAILABLE = "519"`
- `CANNOT_DETACH_DICTIONARY_AS_TABLE = "520"`
- `ATOMIC_RENAME_FAIL = "521"`
- `UNKNOWN_ROW_POLICY = "523"`
- `ALTER_OF_COLUMN_IS_FORBIDDEN = "524"`
- `INCORRECT_DISK_INDEX = "525"`
- `NO_SUITABLE_FUNCTION_IMPLEMENTATION = "527"`
- `CASSANDRA_INTERNAL_ERROR = "528"`
- `NOT_A_LEADER = "529"`
- `CANNOT_CONNECT_RABBITMQ = "530"`
- `CANNOT_FSTAT = "531"`
- `LDAP_ERROR = "532"`
- `INCONSISTENT_RESERVATIONS = "533"`
- `NO_RESERVATIONS_PROVIDED = "534"`
- `UNKNOWN_RAID_TYPE = "535"`
- `CANNOT_RESTORE_FROM_FIELD_DUMP = "536"`
- `ILLEGAL_MYSQL_VARIABLE = "537"`
- `MYSQL_SYNTAX_ERROR = "538"`
- `CANNOT_BIND_RABBITMQ_EXCHANGE = "539"`
- `CANNOT_DECLARE_RABBITMQ_EXCHANGE = "540"`
- `CANNOT_CREATE_RABBITMQ_QUEUE_BINDING = "541"`
- `CANNOT_REMOVE_RABBITMQ_EXCHANGE = "542"`
- `UNKNOWN_MYSQL_DATATYPES_SUPPORT_LEVEL = "543"`
- `ROW_AND_ROWS_TOGETHER = "544"`
- `FIRST_AND_NEXT_TOGETHER = "545"`
- `NO_ROW_DELIMITER = "546"`
- `INVALID_RAID_TYPE = "547"`
- `UNKNOWN_VOLUME = "548"`
- `DATA_TYPE_CANNOT_BE_USED_IN_KEY = "549"`
- `CONDITIONAL_TREE_PARENT_NOT_FOUND = "550"`
- `ILLEGAL_PROJECTION_MANIPULATOR = "551"`
- `UNRECOGNIZED_ARGUMENTS = "552"`
- `LZMA_STREAM_ENCODER_FAILED = "553"`
- `LZMA_STREAM_DECODER_FAILED = "554"`
- `ROCKSDB_ERROR = "555"`
- `SYNC_MYSQL_USER_ACCESS_ERROR = "556"`
- `UNKNOWN_UNION = "557"`
- `EXPECTED_ALL_OR_DISTINCT = "558"`
- `INVALID_GRPC_QUERY_INFO = "559"`
- `ZSTD_ENCODER_FAILED = "560"`
- `ZSTD_DECODER_FAILED = "561"`
- `TLD_LIST_NOT_FOUND = "562"`
- `CANNOT_READ_MAP_FROM_TEXT = "563"`
- `INTERSERVER_SCHEME_DOESNT_MATCH = "564"`
- `TOO_MANY_PARTITIONS = "565"`
- `CANNOT_RMDIR = "566"`
- `DUPLICATED_PART_UUIDS = "567"`
- `RAFT_ERROR = "568"`
- `MULTIPLE_COLUMNS_SERIALIZED_TO_SAME_PROTOBUF_FIELD = "569"`
- `DATA_TYPE_INCOMPATIBLE_WITH_PROTOBUF_FIELD = "570"`
- `DATABASE_REPLICATION_FAILED = "571"`
- `TOO_MANY_QUERY_PLAN_OPTIMIZATIONS = "572"`
- `EPOLL_ERROR = "573"`
- `DISTRIBUTED_TOO_MANY_PENDING_BYTES = "574"`
- `UNKNOWN_SNAPSHOT = "575"`
- `KERBEROS_ERROR = "576"`
- `INVALID_SHARD_ID = "577"`
- `INVALID_FORMAT_INSERT_QUERY_WITH_DATA = "578"`
- `INCORRECT_PART_TYPE = "579"`
- `CANNOT_SET_ROUNDING_MODE = "580"`
- `TOO_LARGE_DISTRIBUTED_DEPTH = "581"`
- `NO_SUCH_PROJECTION_IN_TABLE = "582"`
- `ILLEGAL_PROJECTION = "583"`
- `PROJECTION_NOT_USED = "584"`
- `CANNOT_PARSE_YAML = "585"`
- `CANNOT_CREATE_FILE = "586"`
- `CONCURRENT_ACCESS_NOT_SUPPORTED = "587"`
- `DISTRIBUTED_BROKEN_BATCH_INFO = "588"`
- `DISTRIBUTED_BROKEN_BATCH_FILES = "589"`
- `CANNOT_SYSCONF = "590"`
- `SQLITE_ENGINE_ERROR = "591"`
- `DATA_ENCRYPTION_ERROR = "592"`
- `ZERO_COPY_REPLICATION_ERROR = "593"`
- BZIP2_STREAM_DECODER_FAILED = "594"
- BZIP2_STREAM_ENCODER_FAILED = "595"
- `INTERSECT_OR_EXCEPT_RESULT_STRUCTURES_MISMATCH = "596"`
- `NO_SUCH_ERROR_CODE = "597"`
- `BACKUP_ALREADY_EXISTS = "598"`
- `BACKUP_NOT_FOUND = "599"`
- `BACKUP_VERSION_NOT_SUPPORTED = "600"`
- `BACKUP_DAMAGED = "601"`
- `NO_BASE_BACKUP = "602"`
- `WRONG_BASE_BACKUP = "603"`
- `BACKUP_ENTRY_ALREADY_EXISTS = "604"`
- `BACKUP_ENTRY_NOT_FOUND = "605"`
- `BACKUP_IS_EMPTY = "606"`
- `CANNOT_RESTORE_DATABASE = "607"`
- `CANNOT_RESTORE_TABLE = "608"`
- `FUNCTION_ALREADY_EXISTS = "609"`
- `CANNOT_DROP_FUNCTION = "610"`
- `CANNOT_CREATE_RECURSIVE_FUNCTION = "611"`
- `OBJECT_ALREADY_STORED_ON_DISK = "612"`
- `OBJECT_WAS_NOT_STORED_ON_DISK = "613"`
- `POSTGRESQL_CONNECTION_FAILURE = "614"`
- `CANNOT_ADVISE = "615"`
- `UNKNOWN_READ_METHOD = "616"`
- LZ4_ENCODER_FAILED = "617"
- LZ4_DECODER_FAILED = "618"
- `POSTGRESQL_REPLICATION_INTERNAL_ERROR = "619"`
- `QUERY_NOT_ALLOWED = "620"`
- `CANNOT_NORMALIZE_STRING = "621"`
- `CANNOT_PARSE_CAPN_PROTO_SCHEMA = "622"`
- `CAPN_PROTO_BAD_CAST = "623"`
- `BAD_FILE_TYPE = "624"`
- `IO_SETUP_ERROR = "625"`
- `CANNOT_SKIP_UNKNOWN_FIELD = "626"`
- `BACKUP_ENGINE_NOT_FOUND = "627"`
- `OFFSET_FETCH_WITHOUT_ORDER_BY = "628"`
- `HTTP_RANGE_NOT_SATISFIABLE = "629"`
- `HAVE_DEPENDENT_OBJECTS = "630"`
- `UNKNOWN_FILE_SIZE = "631"`
- `UNEXPECTED_DATA_AFTER_PARSED_VALUE = "632"`
- `QUERY_IS_NOT_SUPPORTED_IN_WINDOW_VIEW = "633"`
- `MONGODB_ERROR = "634"`
- `CANNOT_POLL = "635"`
- `CANNOT_EXTRACT_TABLE_STRUCTURE = "636"`
- `INVALID_TABLE_OVERRIDE = "637"`
- `SNAPPY_UNCOMPRESS_FAILED = "638"`
- `SNAPPY_COMPRESS_FAILED = "639"`
- `NO_HIVEMETASTORE = "640"`
- `CANNOT_APPEND_TO_FILE = "641"`
- `CANNOT_PACK_ARCHIVE = "642"`
- `CANNOT_UNPACK_ARCHIVE = "643"`
- `REMOTE_FS_OBJECT_CACHE_ERROR = "644"`
- `NUMBER_OF_DIMENSIONS_MISMATCHED = "645"`
- `CANNOT_BACKUP_DATABASE = "646"`
- `CANNOT_BACKUP_TABLE = "647"`
- `WRONG_DDL_RENAMING_SETTINGS = "648"`
- `INVALID_TRANSACTION = "649"`
- `SERIALIZATION_ERROR = "650"`
- `CAPN_PROTO_BAD_TYPE = "651"`
- `ONLY_NULLS_WHILE_READING_SCHEMA = "652"`
- `CANNOT_PARSE_BACKUP_SETTINGS = "653"`
- `WRONG_BACKUP_SETTINGS = "654"`
- `FAILED_TO_SYNC_BACKUP_OR_RESTORE = "655"`
- `MEILISEARCH_EXCEPTION = "656"`
- `UNSUPPORTED_MEILISEARCH_TYPE = "657"`
- `MEILISEARCH_MISSING_SOME_COLUMNS = "658"`
- `UNKNOWN_STATUS_OF_TRANSACTION = "659"`
- `HDFS_ERROR = "660"`
- `CANNOT_SEND_SIGNAL = "661"`
- `FS_METADATA_ERROR = "662"`
- `INCONSISTENT_METADATA_FOR_BACKUP = "663"`
- `ACCESS_STORAGE_DOESNT_ALLOW_BACKUP = "664"`
- `CANNOT_CONNECT_NATS = "665"`
- `NOT_INITIALIZED = "667"`
- `INVALID_STATE = "668"`
- `NAMED_COLLECTION_DOESNT_EXIST = "669"`
- `NAMED_COLLECTION_ALREADY_EXISTS = "670"`
- `NAMED_COLLECTION_IS_IMMUTABLE = "671"`
- `INVALID_SCHEDULER_NODE = "672"`
- `RESOURCE_ACCESS_DENIED = "673"`
- `RESOURCE_NOT_FOUND = "674"`
- CANNOT_PARSE_IPV4 = "675"
- CANNOT_PARSE_IPV6 = "676"
- `THREAD_WAS_CANCELED = "677"`
- `IO_URING_INIT_FAILED = "678"`
- `IO_URING_SUBMIT_ERROR = "679"`
- `MIXED_ACCESS_PARAMETER_TYPES = "690"`
- `UNKNOWN_ELEMENT_OF_ENUM = "691"`
- `TOO_MANY_MUTATIONS = "692"`
- `AWS_ERROR = "693"`
- `ASYNC_LOAD_CYCLE = "694"`
- `ASYNC_LOAD_FAILED = "695"`
- `ASYNC_LOAD_CANCELED = "696"`
- `CANNOT_RESTORE_TO_NONENCRYPTED_DISK = "697"`
- `INVALID_REDIS_STORAGE_TYPE = "698"`
- `INVALID_REDIS_TABLE_STRUCTURE = "699"`
- `USER_SESSION_LIMIT_EXCEEDED = "700"`
- `CLUSTER_DOESNT_EXIST = "701"`
- `CLIENT_INFO_DOES_NOT_MATCH = "702"`
- `INVALID_IDENTIFIER = "703"`
- `QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS = "704"`
- `TABLE_NOT_EMPTY = "705"`
- `LIBSSH_ERROR = "706"`
- `GCP_ERROR = "707"`
- `ILLEGAL_STATISTICS = "708"`
- `CANNOT_GET_REPLICATED_DATABASE_SNAPSHOT = "709"`
- `FAULT_INJECTED = "710"`
- `FILECACHE_ACCESS_DENIED = "711"`
- `TOO_MANY_MATERIALIZED_VIEWS = "712"`
- `BROKEN_PROJECTION = "713"`
- `UNEXPECTED_CLUSTER = "714"`
- `CANNOT_DETECT_FORMAT = "715"`
- `CANNOT_FORGET_PARTITION = "716"`
- `EXPERIMENTAL_FEATURE_ERROR = "717"`
- `TOO_SLOW_PARSING = "718"`
- `QUERY_CACHE_USED_WITH_SYSTEM_TABLE = "719"`
- `USER_EXPIRED = "720"`
- `DEPRECATED_FUNCTION = "721"`
- `ASYNC_LOAD_WAIT_FAILED = "722"`
- `PARQUET_EXCEPTION = "723"`
- `TOO_MANY_TABLES = "724"`
- `TOO_MANY_DATABASES = "725"`
- `DISTRIBUTED_CACHE_ERROR = "900"`
- `CANNOT_USE_DISTRIBUTED_CACHE = "901"`
- `KEEPER_EXCEPTION = "999"`
- `POCO_EXCEPTION = "1000"`
- `STD_EXCEPTION = "1001"`
- `UNKNOWN_EXCEPTION = "1002"`

---

URL: https://www.tinybird.co/docs/forward/dev-reference/datafiles
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Datafiles \| Tinybird"
theme-color: "#171612"
description: "Datafiles describe your Tinybird resources, like data sources, pipes, and so on. They're the source code of your project."
inkeep:version: "forward"
---

# Datafiles (https://www.tinybird.co/docs/forward/dev-reference/datafiles#datafiles)

Datafiles describe your Tinybird resources, like data sources, pipes, and so on. They're the source code of your project.

You can use datafiles to manage your projects as source code and take advantage of version control. Tinybird CLI helps you produce and push datafiles to the Tinybird platform.

## Data projects (https://www.tinybird.co/docs/forward/dev-reference/datafiles#data-projects)

A data project is a collection of datafiles that describe your Tinybird resources.

The minimal data project includes:

- A .datasource file, which describes the source and schema of your data.
- A pipe, typically a .endpoint file, which describes how to process and serve the data.
For example:

- Data source
- Pipe

```
SCHEMA >
    `day` Date `json:$.day`,
    `station` String `json:$.station`,
    `abslevel` Nullable(Float32) `json:$.abslevel`,
    `percentagevolume` Float32 `json:$.percentagevolume`,
    `volume` Float32 `json:$.volume`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "station"
ENGINE_SORTING_KEY "day, station"
```

## Folder structure (https://www.tinybird.co/docs/forward/dev-reference/datafiles#folder-structure)

Tinybird projects are organized in a folder structure that helps you organize the source for each resource type.

The datafiles that you generate when you run `tb init` are organized in the following folders:

- `connections`: Contains [connection files](/docs/forward/dev-reference/datafiles/connection-files).
- `copies`: Contains the [copy pipes](/docs/forward/dev-reference/datafiles/pipe-files#copy-pipes).
- `datasources`: Contains the [data sources](/docs/forward/dev-reference/datafiles/datasource-files).
- `endpoints`: Contains the [endpoint pipes](/docs/forward/dev-reference/datafiles/pipe-files#endpoint-pipes).
- `fixtures`: Contains test fixtures and test data.
- `infra`: Contains the infrastructure files. See [tb infra](/docs/forward/dev-reference/commands/tb-infra).
- `materializations`: Contains [materialized views](/docs/forward/dev-reference/datafiles/pipe-files#materialized-pipes).
- `pipes`: Contains non-published [pipes](/docs/forward/dev-reference/datafiles/pipe-files).
- `sinks`: Contains the [sink pipes](/docs/forward/dev-reference/datafiles/pipe-files#sink-pipes).
- `tests`: Contains the test suites.
The following example shows a typical `tinybird` project folder structure that includes subfolders for supported types:

Example file structure
```
.
├── .tinyb
├── tinybird.config.json
├── connections
├── copies
├── datasources
│   └── user_actions.datasource
├── endpoints
│   ├── user_actions_line_chart.pipe
│   └── user_actions_total_widget.pipe
├── fixtures
│   ├── user_actions.prompt
│   └── user_actions_d1046873.ndjson
├── infra
│   └── aws
│       ├── config.json
│       ├── k8s.yaml
│       └── main.tf
├── materializations
├── pipes
├── sinks
└── tests
    └── user_actions_line_chart.yaml
```
The `.tinyb` file contains local authentication context, and `tinybird.config.json` contains project settings like the included project path and default development mode. See [Config files](/docs/forward/dev-reference/config-files).

## Types of datafiles (https://www.tinybird.co/docs/forward/dev-reference/datafiles#types-of-datafiles)

Tinybird uses the following types of datafiles:

- Datasource files (.datasource) represent data sources. See [Datasource files](/docs/forward/dev-reference/datafiles/datasource-files).
- Pipe files (.pipe) represent pipes of various types. See [Pipe files](/docs/forward/dev-reference/datafiles/pipe-files).

## Syntactic conventions (https://www.tinybird.co/docs/forward/dev-reference/datafiles#syntactic-conventions)

Datafiles follow the same syntactic conventions.

### Casing (https://www.tinybird.co/docs/forward/dev-reference/datafiles#casing)

Instructions always appear at the beginning of a line in upper case. For example:

Basic syntax
```
COMMAND value
ANOTHER_INSTR "Value with multiple words"
```

### Multiple lines (https://www.tinybird.co/docs/forward/dev-reference/datafiles#multiple-lines)

Instructions can span multiples lines. For example:

Multiline syntax
```
SCHEMA >
    `d` DateTime,
    `total` Int32,
    `from_novoa` Int16
```

### Comments (https://www.tinybird.co/docs/forward/dev-reference/datafiles#comments)

You can add comments using the `#` character. For example:

```
# This is a comment
COMMAND value
```

## Next steps (https://www.tinybird.co/docs/forward/dev-reference/datafiles#next-steps)

- Learn about [datasource files](/docs/forward/dev-reference/datafiles/datasource-files).
- Learn about [connection files](/docs/forward/dev-reference/datafiles/connection-files).
- Learn about [pipe files](/docs/forward/dev-reference/datafiles/pipe-files).

---

URL: https://www.tinybird.co/docs/forward/dev-reference/config-files
Last update: 2026-06-23T20:48:23.000Z
Content:
---
title: "Config files \| Tinybird"
theme-color: "#171612"
description: "Tinybird project configuration files, including tinybird.config.json and .tinyb."
inkeep:version: "forward"
---

# Config files (https://www.tinybird.co/docs/forward/dev-reference/config-files#config-files)

Tinybird projects use configuration files for two different jobs:

- `tinybird.config.json` defines project settings, such as which files to include and whether development runs in Cloud Branches or Tinybird Local.
- `.tinyb` stores local authentication and Workspace context created by `tb login`.
Commit `tinybird.config.json` to version control. Don't commit `.tinyb` because it can contain authentication tokens.

## tinybird.config.json (https://www.tinybird.co/docs/forward/dev-reference/config-files#tinybird-config-json)

Use `tinybird.config.json` to tell Tinybird where your project resources live. The development mode controls the default build target. Use `branch` for Cloud Branches or `local` for Tinybird Local.

- Tinybird CLI
- TypeScript SDK
- Python SDK
tinybird.config.json
```
{
  "dev_mode": "branch",
  "branch_data_mode": "last_partition",
  "folder": "tinybird",
  "include": [
    "tinybird"
  ]
}
```

### Config keys (https://www.tinybird.co/docs/forward/dev-reference/config-files#config-keys)

| Key | Description |
| --- | --- |
| `include` | Paths to datafiles or SDK resource definitions that Tinybird builds and deploys. |
| `folder` | Project folder for datafiles. `tb init` writes this value to `tinybird.config.json`. When omitted, Tinybird infers the folder from `include`. |
| `dev_mode` / `devMode` | Default build target: `branch` for Cloud Branches, `local` for Tinybird Local, or `manual` to require explicit `--cloud`, `--local`, or `--branch` flags. |
| `branch_data_mode` | Data to attach when `tb build` auto-creates a Cloud Branch with `dev_mode=branch`. Only `last_partition` is supported. Defaults to `last_partition`. The legacy key `branch_data_on_create` is no longer supported. |
| `token` / `baseUrl` | SDK-only. Authentication token and API host, often loaded from environment variables. |
For development mode setup, see [Local development](/docs/forward/development-workflow/local-development) and [Using Cloud Branches](/docs/forward/development-workflow/using-cloud-branches).

## .tinyb (https://www.tinybird.co/docs/forward/dev-reference/config-files#tinyb)

The `.tinyb` file contains the Tinybird project configuration, including the authentication token obtained by running `tb login`.

Running commands requires a valid `.tinyb` file in the root of your project. If you don't have one, create one by running [tb login](/docs/forward/dev-reference/commands/tb-login).

### Location (https://www.tinybird.co/docs/forward/dev-reference/config-files#location)

You can place the `.tinyb` file in the root of your app repository or keep it together with your Tinybird datafiles. Tinybird looks for the `.tinyb` file in all parent folders until it reaches the home directory.

If your Tinybird files and folders are located in a subdirectory, such as `tinybird/`, add a `cwd` field to specify the relative path to that directory.

### File structure (https://www.tinybird.co/docs/forward/dev-reference/config-files#file-structure)

The following is a sample `.tinyb` file:

.tinyb
```
{
  "host": "<tinybird-host>",
  "id": "<workspace-id>",
  "name": "<workspace-name>",
  "scope": "user",
  "token": "<authentication-token>",
  "tokens": {
    "<tinybird-host>": "<authentication-token>",
    "<another-tinybird-host>": "<authentication-token>"
  },
  "cwd": "<optional-path-to-your-tinybird-folder>",
  "user_email": "<user-email>",
  "user_id": "<user-id>",
  "user_token": "<authentication-token>",
  "version": "<tinybird-version>"
}
```

---

URL: https://www.tinybird.co/docs/forward/dev-reference/common-error-patterns
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Common error patterns \| Tinybird"
theme-color: "#171612"
description: "Learn about common error patterns in Tinybird and how to resolve them."
inkeep:version: "forward"
---

# Common error patterns (https://www.tinybird.co/docs/forward/dev-reference/common-error-patterns#common-error-patterns)

This guide explains common error patterns you might encounter when working with Tinybird and provides solutions to resolve them. Check here for the [full list of errors](./list-of-errors).

### MEMORY_LIMIT_EXCEEDED (241) (https://www.tinybird.co/docs/forward/dev-reference/common-error-patterns#memory-limit-exceeded-241)

This error occurs when a query requires more memory than the available limit.

**Common causes:**

- Large result sets
- Complex joins or query time computations
- Insufficient sorting key optimization
**Solutions:**

- Optimize your sorting key to better match query patterns
- Add appropriate indexes
- Break down complex queries into smaller parts
- Precompute functions, such as [JSONExtract()](/docs/sql-reference/functions/json-functions#jsonextract-functions), and materialize them in an intermediate data source.
- Use `LIMIT` clauses when appropriate

### TIMEOUT_EXCEEDED (159) (https://www.tinybird.co/docs/forward/dev-reference/common-error-patterns#timeout-exceeded-159)

This error indicates that a query took longer than the default limit of 10 seconds to execute.

**Common causes:**

- Complex queries without proper optimization
- Large data scans
- Inefficient sorting keys
**Solutions:**

- Review and optimize your sorting key (do they align with the query filters?)
- Add appropriate indexes
- Consider materializing frequently used queries
- Break down complex queries

### TOO_MANY_PARTS (252) (https://www.tinybird.co/docs/forward/dev-reference/common-error-patterns#too-many-parts-252)

This error occurs when you have too many data parts in a partition.

**Common causes:**

- Writing to more than 30 partitions in a single ingest.
- Partition Key is too granular for regular ingests, such as a unique ID or a full-precision timestamp
- Partition Key is too granular for large historic backfills, such as a month
**Solutions:**

- Break your inserts into smaller chunks
- Reduce partition granularity

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands
Last update: 2026-06-18T09:36:38.000Z
Content:
---
title: "Tinybird CLI command reference \| Tinybird"
theme-color: "#171612"
description: "The Tinybird CLI allows you to use all the Tinybird functionality directly from the command line. Get to know the command reference."
inkeep:version: "forward"
---

# CLI command reference (https://www.tinybird.co/docs/forward/dev-reference/commands#cli-command-reference)

Recent CLI versions removed AI-assisted command generation and introduced `tb init` for project setup.

- `tb init` replaces `tb create` for scaffolding projects.
- `tb mock` has been removed. Create fixture files manually (or asking a codign agent) in `fixtures/`.
- `tb test create` has been removed. Create YAML tests manually (or asking a coding agent), then use `tb test run` and `tb test update`.
- `tb --prompt` and interactive `tb` mode are no longer available.
The following list shows all available commands in the Tinybird command-line interface, their options, and their arguments.

For SDK-specific command references, see:

- TypeScript SDK CLI (`npx tinybird`): [TypeScript SDK CLI commands](/docs/forward/dev-reference/commands/typescript-sdk-cli)
- Python SDK CLI (`tinybird`): [Python SDK CLI commands](/docs/forward/dev-reference/commands/python-sdk-cli)
- [Global options](./commands/global-options)
- [tb](./commands/tb)
- [tb branch](./commands/tb-branch)
- [tb build](./commands/tb-build)
- [tb connection](./commands/tb-connection)
- [tb copy](./commands/tb-copy)
- [tb datasource](./commands/tb-datasource)
- [tb deploy](./commands/tb-deploy)
- [tb deployment](./commands/tb-deployment)
- [tb dev](./commands/tb-dev)
- [tb endpoint](./commands/tb-endpoint)
- [tb fmt](./commands/tb-fmt)
- [tb info](./commands/tb-info)
- [tb init](./commands/tb-init)
- [tb infra](./commands/tb-infra)
- [tb job](./commands/tb-job)
- [tb local](./commands/tb-local)
- [tb login](./commands/tb-login)
- [tb logout](./commands/tb-logout)
- [tb materialization](./commands/tb-materialization)
- [tb open](./commands/tb-open)
- [tb pipe](./commands/tb-pipe)
- [tb pull](./commands/tb-pull)
- [tb secret](./commands/tb-secret)
- [tb sink](./commands/tb-sink)
- [tb sql](./commands/tb-sql)
- [tb test](./commands/tb-test)
- [tb token](./commands/tb-token)
- [tb update](./commands/tb-update)
- [tb workspace](./commands/tb-workspace)

---

URL: https://www.tinybird.co/docs/forward/guides/test-your-project
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Test your project \| Tinybird"
theme-color: "#171612"
description: "Set up local checks, fixtures, ingest API tests, and test suites for your Tinybird project."
inkeep:version: "forward"
---

# Test your project (https://www.tinybird.co/docs/forward/guides/test-your-project#test-your-project)

Test your data project before deploying to Tinybird Cloud. Use this guide to validate deployments, load fixture data, call local ingest APIs, and run YAML test suites.

You can validate and test with these workflows:

- Check the deployment before creating it.
- Create fixture files in your project.
- Ingest sample data through local APIs.
- Run and update YAML tests with `tb test`.

## Check deployment (https://www.tinybird.co/docs/forward/guides/test-your-project#check-deployment)

After finishing development, run `tb deploy --check` to validate the deployment before creating it.

```
tb deploy --check
```

## Connection checks (https://www.tinybird.co/docs/forward/guides/test-your-project#connection-checks)

When you run `tb deploy --check` (local) or `tb --cloud deploy --check` (cloud), Tinybird validates [external connections](/docs/forward/core-concepts/connections) to S3, Kafka, GCS, and databases referenced via [table functions](/docs/forward/ingest-data/table-functions).

For local checks:

- Use [tb secret set](/docs/forward/dev-reference/commands/tb-secret) to store connection secrets.
- If you use [S3 locally](/docs/forward/ingest-data/connectors/s3#local-environment), start Tinybird Local with `tb local start --use-aws-creds`.

## Fixture files (https://www.tinybird.co/docs/forward/guides/test-your-project#fixture-files)

Fixtures are sample files stored in the `fixtures/` folder.

```
my-app/
├─ datasources/
│  ├─ user_actions.datasource
│  └─ ...
├─ fixtures/
│  ├─ user_actions.ndjson
│  └─ ...
```
Create fixtures manually. For example:

```
cat > fixtures/user_actions.ndjson <<'EOF'
{"timestamp":"2026-01-01 00:00:00","action":"CLICKED","user_id":"u_1"}
{"timestamp":"2026-01-01 00:01:00","action":"VIEWED","user_id":"u_2"}
EOF
```
When fixture files match the data source name (for example, `fixtures/user_actions.ndjson` for `datasources/user_actions.datasource`), `tb build` loads them automatically.

If you use custom fixture filenames, append them manually:

```
tb datasource append user_actions --file fixtures/custom_user_actions.ndjson
```

## Call the ingest APIs (https://www.tinybird.co/docs/forward/guides/test-your-project#call-the-ingest-apis)

You can also test by calling local ingest APIs:

- Send events to the local [Events API](/docs/api-reference/events-api).
- Ingest files through the [Data Sources API](/docs/api-reference/datasource-api).
Get a token with `tb token ls`, then call your local endpoint:

- cURL (Events API)
- Local file (Data sources API)
- Remote file (Data sources API)

```
curl \
      -X POST 'http://localhost:7181/v0/events?name=<your_datasource>' \
      -H "Authorization: Bearer <your_token>" \
      -d $'<your_data>'
```

## Create a test suite (https://www.tinybird.co/docs/forward/guides/test-your-project#create-a-test-suite)

Create test YAML files manually in the `tests/` folder.

Example `tests/user_action_insights_widget.yaml`:

```
- name: user_action_insights_widget_clicked
  description: Returns user actions filtered by CLICKED
  expected_http_status: 200
  parameters: action=CLICKED
  expected_result: |
    {"action":"CLICKED", "user_id":"u_1"}
```
Then run tests:

```
tb test run
```
To refresh expected results for a test file or resource:

```
tb test update user_action_insights_widget
```
Copy Pipes write into target data sources instead of returning rows directly. To test them:

1. Create a YAML test file for the copy pipe in `tests/`.
2. Run `tb copy run <copy_pipe>` to populate the target data source.
3. Run `tb test update <copy_pipe>` to capture expected results.

## Next steps (https://www.tinybird.co/docs/forward/guides/test-your-project#next-steps)

- Make changes to your data sources and test them. See [Evolve data sources](/docs/forward/guides/evolve-data-source).
- Learn more about [deployments](/docs/forward/core-concepts/deployments).
- Learn about datafiles, like `.datasource` and `.pipe` files. See [Datafiles](/docs/forward/dev-reference/datafiles).

---

URL: https://www.tinybird.co/docs/forward/guides/sorting-key-pagination
Last update: 2026-08-10T12:50:33.000Z
Content:
---
title: "Paginate Endpoint results with a cursor \| Tinybird"
theme-color: "#171612"
description: "Use cursor pagination to page through Endpoint results."
inkeep:version: "forward"
---

# Paginate Endpoint results with a cursor (https://www.tinybird.co/docs/forward/guides/sorting-key-pagination#paginate-endpoint-results-with-a-cursor)

Cursor pagination, also called keyset pagination, uses values from the last row of one page to retrieve the next page. Use it to process a large result set sequentially without the increasing query cost of a large `OFFSET`.

This guide creates a Data Source and an API Endpoint that paginate tenant activity by timestamp and activity ID.

## Before you start (https://www.tinybird.co/docs/forward/guides/sorting-key-pagination#before-you-start)

Before you create the resources, ensure you have:

- A Tinybird project
- The Tinybird CLI installed
- A dataset with columns that define a stable, unique order

## Step 1: Create the Data Source (https://www.tinybird.co/docs/forward/guides/sorting-key-pagination#step-1-create-the-data-source)

Create a `tenant_activity_log.datasource` file:

datasources/tenant_activity_log.datasource
```
DESCRIPTION >
    Fake tenant activity data for testing keyset pagination

SCHEMA >
    `tenant_id` String `json:$.tenant_id`,
    `occurred_at` DateTime64(3) `json:$.occurred_at`,
    `activity_id` UInt64 `json:$.activity_id`,
    `activity_type` LowCardinality(String) `json:$.activity_type`,
    `details` String `json:$.details`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "tenant_id, occurred_at, activity_id"
```
The sorting key matches how the Endpoint filters and orders the data:

- `tenant_id` identifies one tenant and appears first in the sorting key.
- `occurred_at` orders the tenant's activity chronologically.
- `activity_id` provides a unique tie-breaker when multiple activities have the same timestamp.

## Step 2: Create the API Endpoint (https://www.tinybird.co/docs/forward/guides/sorting-key-pagination#step-2-create-the-api-endpoint)

Create a `tenant_activity_page.pipe` file:

pipes/tenant_activity_page.pipe
```
TOKEN tenant_activity_page_read READ

NODE paginated_activity
SQL >
    %
    SELECT
        tenant_id,
        occurred_at,
        activity_id,
        activity_type,
        details
    FROM tenant_activity_log
    WHERE tenant_id = {{String(tenant_id, required=True)}}

        AND (occurred_at, activity_id) < (
            {{DateTime64(cursor_occurred_at)}},
            {{UInt64(cursor_activity_id)}}
        )

    ORDER BY
        occurred_at DESC,
        activity_id DESC

    LIMIT {{UInt16(page_size, 5)}}

TYPE ENDPOINT
```

---

URL: https://www.tinybird.co/docs/forward/guides/share-endpoint-documentation
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Share API endpoints documentation \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to share your Tinybird API endpoint documentation with development teams."
inkeep:version: "forward"
---

# Share Tinybird API endpoint documentation (https://www.tinybird.co/docs/forward/guides/share-endpoint-documentation#share-tinybird-api-endpoint-documentation)

Learn how to share your Tinybird API endpoint documentation with development teams.

## The Tinybird API endpoint page (https://www.tinybird.co/docs/forward/guides/share-endpoint-documentation#the-tinybird-api-endpoint-page)

When you publish an API endpoint, Tinybird generates a documentation page for you that is ready to share and compatible with OpenAPI 3.0. It contains your API endpoint description, information about the dynamic parameters you can use when querying this endpoint, and code snippets for quickly integrating your API in third-party applications.

To share your published API Endpoint, select **Create Chart** > **Share this API endpoint**:

## Use Static Tokens to define API endpoint subsets (https://www.tinybird.co/docs/forward/guides/share-endpoint-documentation#use-static-tokens-to-define-api-endpoint-subsets)

Tinybird authentication is based on [Tokens](/docs/forward/core-concepts/tokens), which contain different scopes for specific resources. For example, use a Token to read from one or more Endpoints or to get write permissions for a particular Data Source.

In the URLs generated for sharing a public Endpoint page, a Token parameter appears after the Endpoint ID. This page is accessible only if the Token provided in the URL has read permissions for it:

```
https://api.tinybird.co/endpoint/t_bdcad2252e794c6573e21e7e?token=<token_with_permissions>
```
For security, Tinybird automatically generates a read-only Token when sharing a public Endpoint page for the first time. If you don't explicitly use it, your Admin Token is never exposed.

### The API endpoints list page (https://www.tinybird.co/docs/forward/guides/share-endpoint-documentation#the-api-endpoints-list-page)

You can also render the Endpoint information for a given Token.

```
https://app.tinybird.co/<provider>/<region>/endpoints?token=<your_token>
```
Enter the previous URL, with your Token and the provider and region where the API endpoint is published, into the browser to retrieve a list that shows all API endpoints that the token can read from.

When integrating your Endpoint in your apps, manage dedicated Tokens. Create a Token for every app environment so you can track requests to your Endpoints by app and choose which Endpoints are accessible to them.

Once you do that, you can share auto-generated documentation with ease, without compromising your data privacy and security.

API endpoint docs pages include a read token by default. In the "Share this API endpoint" modal, you can also see public URLs for every token with read permissions for your pipe.

## Browse your API docs (https://www.tinybird.co/docs/forward/guides/share-endpoint-documentation#browse-your-api-docs)

All endpoint documentation is compatible with OpenAPI 3.0 and accessible through the API. If you use a token with permissions for more than one API endpoint, the OpenAPI documentation contains information about all the API endpoints at once.

---

URL: https://www.tinybird.co/docs/forward/guides/replace-and-delete-data
Last update: 2026-06-29T08:18:10.000Z
Content:
---
title: "Replace and delete data \| Tinybird"
theme-color: "#171612"
description: "Update & delete operations are common in transactional databases over operational data, but sometimes you also need to make these changes on your analytical data in Tinybird."
inkeep:version: "forward"
---

# Replace and delete data in your Tinybird Data Sources (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#replace-and-delete-data-in-your-tinybird-data-sources)

Update and delete operations are common in transactional databases over operational data, but sometimes you also need to make these changes on your analytical data in Tinybird.

Sometimes, you need to delete or replace some of your data in Tinybird. Perhaps there was a bug in your app, a transient error in your operational database, or simply an evolution of requirements due to product or regulatory changes.

It's **not safe** to replace data in the partitions where you are actively ingesting data. You may lose the data inserted during the process.

Tinybird works well with append-only workloads but also fully supports replacing and deleting data. It abstracts away the tricky complexities of data replication, partition management and mutations rewriting, allowing you to focus on your data engineering flows and not the internals of real-time analytical databases.

This guide shows you using different examples, how to selectively delete or update data in Tinybird using the REST API. You can then adapt these processes for your own needs.

All API operations on this page require a Token with the `ADMIN` [scope](/docs/forward/core-concepts/static-tokens#other-tokens). In the code snippets, replace `<token_with_ADMIN_scope>` by your actual token.

## Delete data selectively (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#delete-data-selectively)

To delete data that's within a condition, send a POST request to the [Data Sources /delete API](/docs/api-reference/datasource-api#post--v0-datasources-(.+)-delete), providing the name of one of your Data Sources in Tinybird and a `delete_condition` parameter, which is an SQL expression filter.

Delete operations don't automatically cascade to downstream Materialized Views. You may need to perform separate delete operations on Materialized Views.

ClickHouse®'s lightweight delete feature is now available in beta. Use **synchronous** mode to wait for the deletion to complete, or **asynchronous** mode to poll a job and monitor its progress. See [Lightweight delete](https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#lightweight-delete) or the [Data Sources API reference](/docs/api-reference/datasource-api#delete-rows-from-a-data-source).

Imagine you have a Data Source called `events` and you want to remove all the transactions for November 2019. You'd send a POST request like this:

- Tinybird CLI
- API
Delete data selectively
```
tb datasource delete events --sql-condition "toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'"
```
Once you make the request, you can see that the `POST` request to the delete API Endpoint is asynchronous. It returns a [job response](/docs/api-reference/jobs-api#jobs-api-getting-information-about-jobs), indicating an ID for the job, the status of the job, the `delete_condition`, and some other metadata. Although the delete operation runs asynchronously, the operation waits synchronously for all the mutations to be rewritten and delete the data replicas. Queries reading data either see the state before the operation or after it's complete.

```
{
  "id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
  "job_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
  "job_url": "https://<your_host>/v0/jobs/64e5f541-xxxx-xxxx-xxxx-00524051861b",
  "job": {
    "kind": "delete_data",
    "id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
    "job_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
    "status": "waiting",
    "created_at": "2023-04-11 13:52:32.423207",
    "updated_at": "2023-04-11 13:52:32.423213",
    "started_at": null,
    "is_cancellable": true,
    "datasource": {
      "id": "t_c45d5ae6781b41278fcee365f5bxxxxx",
      "name": "shopping_data"
    },
    "delete_condition": "event = 'search'"
  },
  "status": "waiting",
  "delete_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b"
}
```
You can periodically poll the `job_url` with the given ID to check the status of the deletion process. When the status is `done` your job deleted the data matching the SQL expression filter and all your Pipes and API Endpoints continue running with the remaining data in the Data Source.

### Lightweight delete (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#lightweight-delete)

The lightweight delete endpoint is currently in beta.

The `POST /v1/datasources/{name}/delete` endpoint uses ClickHouse®'s lightweight delete algorithm: matched rows are flagged in place, and queries filter them out through that mask at read time.

It takes a `delete_condition` (the SQL `WHERE` clause that matches the rows to delete), plus an optional `partition` parameter to scope the operation. You control how the call behaves with the `wait` parameter: synchronous by default, asynchronous when you pass `wait=false`. The full parameter reference is in the [Data Sources API reference](/docs/api-reference/datasource-api#delete-rows-from-a-data-source).

#### Synchronous (default) (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#synchronous-default)

A synchronous call holds the connection open until the underlying mutation finishes. The response includes a `rows_affected` property and a partition-level `mutation` block in the response body. ClickHouse® doesn't expose the affected row count for lightweight deletes, so Tinybird computes `rows_affected` by counting the rows that match `delete_condition` immediately before issuing the delete. Treat it as a best-effort number, not a strict guarantee. Reach for this mode when you want an inline acknowledgement of the deletion. Typical use cases include cleanup scripts, GDPR erasure requests, and one-off fixes.

- Tinybird CLI
- API
Synchronous lightweight delete
```
tb datasource delete events \
  --sql-condition "toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'" \
  --lightweight-delete
```
Synchronous response
```
{
  "rows_affected": 1287,
  "query_id": "5ad8c2e0-xxxx-xxxx-xxxx-7a3a6b2db1f0",
  "delete_condition": "toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'",
  "mutation": {
    "is_done": true,
    "partition_ids": ["2019-11-01", "2019-11-02"],
    "partitions_changed": 2,
    "partitions_done": 2,
    "partitions_in_progress": 0
  }
}
```
If the mutation hasn't finished within `wait_max_seconds` (default `60`), the response is still `200 OK` but with `mutation.is_done = false`. Query the `system.mutations` table to track the remaining progress until the delete completes.

#### Asynchronous (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#asynchronous)

With `wait=false`, the call returns immediately with a job envelope. Reach for this mode when the delete is large enough that you'd rather not hold the connection open, or when you already have a worker polling jobs.

- Tinybird CLI
- API
Asynchronous lightweight delete
```
tb datasource delete events \
  --sql-condition "toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'" \
  --lightweight-delete \
  --no-wait
```
Asynchronous response
```
{
  "id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
  "job_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
  "job_url": "https://<your_host>/v0/jobs/64e5f541-xxxx-xxxx-xxxx-00524051861b",
  "job": {
    "kind": "delete_data",
    "delete_strategy": "lightweight",
    "id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
    "status": "waiting",
    "datasource": {
      "id": "t_c45d5ae6781b41278fcee365f5bxxxxx",
      "name": "events"
    },
    "delete_condition": "toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'"
  },
  "status": "waiting",
  "delete_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b"
}
```
Poll the `job_url` through the [Jobs API](/docs/api-reference/jobs-api#jobs-api-getting-information-about-jobs). Once `status` is `done`, the job payload carries the same `rows_affected` and `mutation` fields as the synchronous response. Asynchronous lightweight deletes share the delete-job concurrency limit with the v0 endpoint.

The lightweight delete endpoint doesn't support deleting from Quarantine Data Sources or using subqueries inside `delete_condition`. Use the `POST /v0/datasources/{name}/delete` endpoint for those cases.

### Truncate a Data Source (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#truncate-a-data-source)

Sometimes you want to delete all data contained in a Data Source. You can perform this action from the UI and API.

Using the API, the [truncate](/docs/api-reference/datasource-api#post--v0-datasources-(.+)-truncate) endpoint deletes all rows in a Data Source as shown in this example:

- Tinybird CLI
- API
Truncate a Data Source
```
tb datasource truncate <your_datasource>
```
You can also truncate a Data Source directly from the UI:

<!-- <figure> -->
![](/docs/_next/image?url=%252Fdocs%252Fimg%252Freplacing-and-deleting-data-1-forward.png&w=3840&q=75)

<!-- <figcaption> -->
Deleting available via CLI or API, but truncating it to delete all the data is available via the UI.
<!-- </figcaption> -->

<!-- </figure> -->

## Replace data selectively (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#replace-data-selectively)

The ability to update data is often not the top priority when designing analytical databases, but there are always scenarios where you need to update or replace your analytical data. For example, you might have reconciliation processes over your transactions that affect your original data. Or maybe your ingestion process was simply faulty, and you ingested inaccurate data for a period of time.

In Tinybird, you can specify a condition to replace only part of the data during the ingestion process. For instance, if you want to reingest a CSV with the data for November 2019 and update your Data Source accordingly. To update the data, you pass the `replace_condition` parameter with the `toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'` condition.

When using `mode=replace` with an S3 URL via the API, you must use a pre-signed URL. Unlike `mode=append`, `mode=replace` is a multi-step process that passes the URL to a background worker with no access to your S3 Connector credentials. Generate a pre-signed URL programmatically using the AWS SDK or CLI before passing it to the API.

- Tinybird CLI
- API
Replace data selectively
```
tb datasource replace events \
https://storage.googleapis.com/tinybird-assets/datasets/guides/events_1M_november2019_1.csv \
--sql-condition "toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'"
```
The response to the previous API call looks like this:

Response after replacing data
```
{
    "id": "a83fcb35-8d01-47b9-842c-a288d87679d0",
    "job_id": "a83fcb35-8d01-47b9-842c-a288d87679d0",
    "job_url": "https://<your_host>/v0/jobs/a83fcb35-8d01-47b9-842c-a288d87679d0",
    "job": {
        "kind": "import",
        "id": "a83fcb35-8d01-47b9-842c-a288d87679d0",
        "job_id": "a83fcb35-8d01-47b9-842c-a288d87679d0",
        "import_id": "a83fcb35-8d01-47b9-842c-a288d87679d0",
        "status": "waiting",
        "statistics": null,
        "datasource": { ... },
        "quarantine_rows": 0,
        "invalid_lines": 0
    },
    "status": "waiting",
    "import_id": "a83fcb35-8d01-47b9-842c-a288d87679d0"
}
```
As in the case of the selective deletion, selective replacement also runs as an asynchronous request, so [check the status of the job](/docs/api-reference/jobs-api#jobs-api-getting-information-about-jobs) periodically. You can see the status of the job by using the `job_url` returned in the previous response.

### About the replace condition (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#about-the-replace-condition)

Conditional replaces apply over partitions and the match condition selects partitions needed for the operation. The records remaining after the match condition determine the partitions involved.

Always include the partition key in the replace condition to maintain consistency.

The replace condition filters the new data that's appended, meaning it excludes rows not matching the condition. The condition is also applied for the selected partitions in the Data Source, removing rows that don't match the condition in these partitions. Rows that don't match the condition and may be present in other partitions remain.

See the [example](https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#example) that follows for a better understanding of selectively replacing data in a Data Source.

### Linked Materialized Views (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#linked-materialized-views)

If you have several connected Materialized Views, then selective replaces proceed in a cascading fashion. For example, if Data Source A materializes data to Data Source B and from there to Data Source C, then when you replace data in Data Source A, Data Sources B and C automatically update accordingly. All three Data Sources need to have compatible partition keys since replaces processed by partition.

Remember: The provided Token must have the `ADMIN` [scope](/docs/forward/core-concepts/static-tokens#other-tokens).

### Example (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#example)

For this example, consider this Data Source:

<!-- <figure> -->
![](/docs/_next/image?url=%252Fdocs%252Fimg%252Freplacing-example-1-forward.png&w=3840&q=75)

<!-- </figure> -->
characters.datasource
```
SCHEMA >
    `age` Int32 `json:$.age`,
    `name` String `json:$.name`,
    `profession` String `json:$.profession`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "name"
ENGINE_PARTITION_KEY "profession"
```
Its partition key is `ENGINE_PARTITION_KEY "profession"`. If you wanted to replace the last two rows with new data, you can send this request with the replace condition `replace_condition=(profession='Jedi')`:

- Tinybird CLI
- API
Replace with partition in condition
```
echo "50,Mace Windu,Jedi" > jedi.csv
tb datasource replace characters jedi.csv --sql-condition "profession='Jedi'"
```
Since the replace condition column matches the partition key, the result is:

<!-- <figure> -->
![](/docs/_next/image?url=%252Fdocs%252Fimg%252Freplacing-example-2-forward.png&w=3840&q=75)

<!-- </figure> -->
However, consider what happens if you create the Data Source with `ENGINE_PARTITION_KEY "name"`:

characters.datasource
```
SCHEMA >
    `age` Int32 `json:$.age`,
    `name` String `json:$.name`,
    `profession` String `json:$.profession`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "name"
ENGINE_PARTITION_KEY "name"
```
If you were to run the same replace request, the result probably doesn't make sense:

<!-- <figure> -->
![](/docs/_next/image?url=%252Fdocs%252Fimg%252Freplacing-example-3-forward.png&w=3840&q=75)

<!-- </figure> -->
Why were the existed rows not removed? Because the `replace` process uses the payload rows to identify which partitions to work on. The Data Source is now partitioned by name and not profession, so the process didn't delete the other "Jedi" rows. They're in different partitions because they have different names.

The rule of thumb is this: **Always make sure the replace condition uses the partition key as the filter field**.

## Replace a Data Source completely (https://www.tinybird.co/docs/forward/guides/replace-and-delete-data#replace-a-data-source-completely)

To replace a complete Data Source, make an API call similar to the previous example, without providing a `replace_condition`:

- Tinybird CLI
- API
Replace Data Source completely
```
tb datasource replace events https://storage.googleapis.com/tinybird-assets/datasets/guides/events_1M_november2019_1.csv
```
The example request is replacing a Data Source with the data found in a given URL pointing to a CSV file.

Schemas must be identical. When replacing data either selectively or entirely, the schema of the new inbound data must match that of the original Data Source. Rows not containing the same schema go to quarantine.

---

URL: https://www.tinybird.co/docs/forward/guides/reliable-scheduling-with-trigger
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Reliable scheduling with Trigger.dev \| Tinybird"
theme-color: "#171612"
description: "Learn how to create complex, reliable scheduling with Trigger.dev"
inkeep:version: "forward"
---

# Reliable scheduling with Trigger.dev (https://www.tinybird.co/docs/forward/guides/reliable-scheduling-with-trigger#reliable-scheduling-with-trigger-dev)

[Trigger.dev](https://trigger.dev/) is an open source background job platform. With Trigger.dev you can easily create, schedule, and manage background jobs using code.

Read on to learn how to create complex, reliable scheduling with Trigger.dev.

## Before you start (https://www.tinybird.co/docs/forward/guides/reliable-scheduling-with-trigger#before-you-start)

Before you start, ensure:

- You have a [Trigger.dev account](https://trigger.dev/).
- You have a [Tinybird workspace](https://www.tinybird.co/).

## Create your first trigger task (https://www.tinybird.co/docs/forward/guides/reliable-scheduling-with-trigger#create-your-first-trigger-task)

The [tinybird-trigger-tasks package](https://www.npmjs.com/package/@sdairs/tinybird-trigger-tasks) implements tasks for Tinybird copy pipes and the Query API. You can [find the source code in the @sdairs/tinybird-trigger repo](https://github.com/sdairs/tinybird-trigger).

1. Create a working directory, and run `npx trigger.dev@latest init` to connect the project to Trigger.dev.
2. Inside the `trigger` directory, install the npm package with `npm install @sdairs/tinybird-trigger-tasks`.
3. Create a new file called `myTask.ts` and add the following code:

```
import { task } from "@trigger.dev/sdk/v3";
import { tinybirdCopyTask } from "@sdairs/tinybird-trigger-tasks";

export const exampleExecutor = task({
    id: "example-executor",
    run: async (payload, { ctx }) => {
        console.log("Example executor task is running");

        // Run a copy job
        const copyResult = await tinybirdCopyTask.triggerAndWait({ pipeId: <COPY_PIPE_ID> });
        console.log(copyResult);

    },
});
```

1. Create a new pipe using the following SQL:

```
SELECT number + 1 AS value
FROM numbers(100)
```

1. Name the pipe `my_copy` and deploy the changes.
2. Update `myTask.ts`, replacing `<COPY_PIPE_ID>` with the name of your pipe, `my_copy` in this case.
3. Create a `.env` file in your directory root.
4. Go to your Tinybird workspace and copy the Admin Token, then add it to the `.env` file as follows:

```
TINYBIRD_TOKEN=p.eyJ...
```

1. Run `npx trigger.dev@latest dev` to push the task to Trigger.dev.
2. Go to your Trigger.dev dashboard, and perform a test run to trigger the task and the copy pipe.
3. Go to your Tinybird workspace and check the copy pipe results.

## See also (https://www.tinybird.co/docs/forward/guides/reliable-scheduling-with-trigger#see-also)

- [Trigger.dev quick start](https://trigger.dev/docs/quick-start)
- [tinybird-trigger repo](https://github.com/sdairs/tinybird-trigger)
- [YouTube: Using Trigger.dev with Tinybird for code-first background job execution](https://www.youtube.com/watch?v=0TcQfcMrGNw)

---

URL: https://www.tinybird.co/docs/forward/guides/redpanda-setup
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Redpanda setup guide \| Tinybird"
theme-color: "#171612"
description: "Guide to setting up Tinybird's Kafka connector with Redpanda, including authentication and network configuration."
inkeep:version: "forward"
---

# Redpanda setup guide (https://www.tinybird.co/docs/forward/guides/redpanda-setup#redpanda-setup-guide)

This guide walks you through setting up Tinybird's Kafka connector with Redpanda, a Kafka-compatible streaming data platform.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/redpanda-setup#prerequisites)

- A Redpanda cluster (self-hosted or Redpanda Cloud)
- Redpanda credentials (if authentication is turned on)
- Network access to your Redpanda cluster
- A Tinybird workspace

## Redpanda compatibility (https://www.tinybird.co/docs/forward/guides/redpanda-setup#redpanda-compatibility)

Redpanda is fully compatible with the Apache Kafka protocol, so you can use Tinybird's Kafka connector with Redpanda without any special configuration. The connector works with Redpanda the same way it works with Apache Kafka.

## Step 1: Get your Redpanda connection details (https://www.tinybird.co/docs/forward/guides/redpanda-setup#step-1-get-your-redpanda-connection-details)

### Bootstrap servers (https://www.tinybird.co/docs/forward/guides/redpanda-setup#bootstrap-servers)

1. For **Redpanda Cloud**: Get the bootstrap server from your Redpanda Cloud console
2. For **self-hosted Redpanda**: Use your Redpanda broker addresses
The bootstrap server format is: `<host>:<port>`

Common ports:

- `9092` for PLAINTEXT
- `9093` for SASL_SSL
- `9094` for TLS

### Authentication (https://www.tinybird.co/docs/forward/guides/redpanda-setup#authentication)

Redpanda supports multiple authentication methods:

- **SASL/PLAIN** - Username and password
- **SASL/SCRAM-SHA-256** or **SCRAM-SHA-512** - SCRAM authentication
- **TLS/mTLS** - Certificate-based authentication
- **PLAINTEXT** - No authentication (local development only)

## Step 2: Create the Kafka connection (https://www.tinybird.co/docs/forward/guides/redpanda-setup#step-2-create-the-kafka-connection)

You can create the Kafka connection using the CLI wizard or by manually creating a connection file.

### Option 1: Use the CLI wizard (recommended) (https://www.tinybird.co/docs/forward/guides/redpanda-setup#option-1-use-the-cli-wizard-recommended)

Run the following command to create a connection interactively:

```
tb connection create kafka
```
The wizard prompts you for:

1. Connection name
2. Bootstrap server
3. Kafka key (username for SASL/PLAIN or SCRAM)
4. Kafka secret (password for SASL/PLAIN or SCRAM)
After the wizard completes, you can edit the generated `.connection` file to add additional settings like `KAFKA_SASL_MECHANISM` if needed (defaults to PLAIN).

### Option 2: Manually create a connection file (https://www.tinybird.co/docs/forward/guides/redpanda-setup#option-2-manually-create-a-connection-file)

#### SASL/PLAIN authentication (https://www.tinybird.co/docs/forward/guides/redpanda-setup#saslplain-authentication)

connections/redpanda.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS <REDPANDA_BOOTSTRAP_SERVER>
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("REDPANDA_USERNAME") }}
KAFKA_SECRET {{ tb_secret("REDPANDA_PASSWORD") }}
```
Set the secrets:

```
tb [--cloud] secret set REDPANDA_USERNAME <YOUR_USERNAME>
tb [--cloud] secret set REDPANDA_PASSWORD <YOUR_PASSWORD>
```

### SASL/SCRAM authentication (https://www.tinybird.co/docs/forward/guides/redpanda-setup#saslscram-authentication)

connections/redpanda_scram.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS <REDPANDA_BOOTSTRAP_SERVER>
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM SCRAM-SHA-256
KAFKA_KEY {{ tb_secret("REDPANDA_USERNAME") }}
KAFKA_SECRET {{ tb_secret("REDPANDA_PASSWORD") }}
```

### PLAINTEXT (local development only) (https://www.tinybird.co/docs/forward/guides/redpanda-setup#plaintext-local-development-only)

connections/redpanda_local.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS localhost:9092
KAFKA_SECURITY_PROTOCOL PLAINTEXT
```
PLAINTEXT connections should only be used for local development. Production Redpanda clusters should use SASL_SSL.

## Step 3: SSL/TLS certificate (if required) (https://www.tinybird.co/docs/forward/guides/redpanda-setup#step-3-ssltls-certificate-if-required)

If your Redpanda cluster uses self-signed certificates or a private CA, provide the CA certificate:

```
tb [--cloud] secret set --multiline REDPANDA_SSL_CA_PEM
```
Add to your connection file:

```
KAFKA_SSL_CA_PEM >
  {{ tb_secret("REDPANDA_SSL_CA_PEM") }}
```

## Step 4: Create the Kafka Data Source (https://www.tinybird.co/docs/forward/guides/redpanda-setup#step-4-create-the-kafka-data-source)

Now that your connection is configured, create a Kafka Data Source. See [Create a Kafka data source](/docs/forward/ingest-data/connectors/kafka#create-a-kafka-data-source) in the main Kafka connector guide for detailed instructions on:

- Using `tb datasource create --kafka` for a guided setup
- Manually creating `.datasource` files
- Defining schemas with JSONPath expressions
- Configuring Kafka-specific settings

## Step 5: Test the connection (https://www.tinybird.co/docs/forward/guides/redpanda-setup#step-5-test-the-connection)

Test your connection and preview data:

```
tb connection data redpanda
```
This command prompts you to select a topic and consumer group ID, then returns preview data. This verifies that Tinybird can connect to your Redpanda cluster, authenticate, and consume messages.

## Redpanda Cloud specific setup (https://www.tinybird.co/docs/forward/guides/redpanda-setup#redpanda-cloud-specific-setup)

If you're using Redpanda Cloud:

1. **Get connection details** from the Redpanda Cloud console
2. **Use SASL/PLAIN** authentication with your Redpanda Cloud credentials
3. **Use the provided bootstrap server** address
4. **No SSL certificate needed** - Redpanda Cloud uses public CA certificates

## Common Redpanda issues (https://www.tinybird.co/docs/forward/guides/redpanda-setup#common-redpanda-issues)

### Issue: Authentication failed (https://www.tinybird.co/docs/forward/guides/redpanda-setup#issue-authentication-failed)

**Solutions:**

1. Verify username and password are correct
2. Check SASL mechanism matches your Redpanda configuration
3. Ensure the user has read permissions on the topic
4. Verify security protocol matches (SASL_SSL vs PLAINTEXT)

### Issue: Connection timeout (https://www.tinybird.co/docs/forward/guides/redpanda-setup#issue-connection-timeout)

**Solutions:**

1. Verify bootstrap server address is correct
2. Check network connectivity to Redpanda cluster
3. Verify firewall rules allow outbound connections
4. For self-hosted Redpanda, ensure the cluster is accessible

### Issue: SSL certificate validation (https://www.tinybird.co/docs/forward/guides/redpanda-setup#issue-ssl-certificate-validation)

**Solutions:**

1. Provide CA certificate if using self-signed certificates
2. Verify certificate is in PEM format
3. Check certificate hasn't expired

## Redpanda vs Kafka differences (https://www.tinybird.co/docs/forward/guides/redpanda-setup#redpanda-vs-kafka-differences)

While Redpanda is Kafka-compatible, there are some differences to be aware of:

1. **Performance**: Redpanda is optimized for lower latency and higher throughput
2. **Resource usage**: Redpanda typically uses less CPU and memory
3. **Configuration**: Some Kafka-specific configurations may not apply
4. **Schema Registry**: Redpanda doesn't include Schema Registry by default (use Confluent Schema Registry separately if needed)

## Best practices (https://www.tinybird.co/docs/forward/guides/redpanda-setup#best-practices)

1. **Use SASL_SSL** for production Redpanda clusters
2. **Monitor consumer lag** using `kafka_ops_log`
3. **Use unique consumer group IDs** for each Data Source
4. **Set up alerts** for high lag or errors
5. **Test connections** before deploying to production

## Related documentation (https://www.tinybird.co/docs/forward/guides/redpanda-setup#related-documentation)

- [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka) - Main setup and configuration guide
- [Monitor Kafka connectors](/docs/forward/monitoring/kafka-clickhouse-monitoring) - Set up monitoring and alerts
- [Troubleshooting guide](/docs/forward/ingest-data/connectors/kafka/troubleshooting) - Common issues and solutions

---

URL: https://www.tinybird.co/docs/forward/guides/quarantine
Last update: 2026-08-05T14:53:06.000Z
Content:
---
title: "Quarantine Data Sources \| Tinybird"
theme-color: "#171612"
description: "Quarantine data sources store data that doesn't fit the schema."
inkeep:version: "forward"
---

# Quarantine Data Sources (https://www.tinybird.co/docs/forward/guides/quarantine#quarantine-data-sources)

Every data source in your workspace has an associated quarantine data source that stores data that doesn't fit the schema. If you send rows that don't match the data source schema, they're automatically sent to the quarantine table so that the ingest process doesn't fail.

By convention, quarantine data sources follow the naming pattern `{datasource_name}_quarantine`. You can review quarantined rows at any time or perform operations on them using Pipes. This is a useful source of information when fixing issues in the origin source or applying changes during ingest.

Data in quarantine data sources is retained for 1 month, after which it's automatically deleted.

## Review quarantined data (https://www.tinybird.co/docs/forward/guides/quarantine#review-quarantined-data)

To check your quarantine data sources, run the `tb sql` command. For example:

```
tb sql "select * from <datasource_name>_quarantine limit 10"
```
A sample output of the `tb sql` command is the following:

```
──────────────────────────────────────────────────────────────────
c__error_column: ['abslevel']
c__error: ["value '' on column 'abslevel' is not Float32"]
c__import_id: 01JKQPWT8GVXAN5GJ1VBD4XM27
day: 2014-07-30
station: Embassament de Siurana (Cornudella de Montsant)
volume: 11.57
insertion_date: 2025-02-10 10:36:20
──────────────────────────────────────────────────────────────────
```
The quarantine data source schema contains the columns of the original row and the following columns with information about the issues that caused the quarantine:

- `c__error_column` Array(String) contains an array of all the columns that contain an invalid value.
- `c__error` Array(String) contains an array of all the errors that caused the ingestion to fail and led to the row being stored in the Quarantine Data Source. Use this column with `c__error_column` to identify which columns have problems and their specific errors.
- `c__import_id` Nullable(String) contains the job's identifier in case the column was imported through a job.
- `insertion_date` (DateTime) contains the timestamp in which the ingestion was done.

## Fixing quarantined data example (https://www.tinybird.co/docs/forward/guides/quarantine#fixing-quarantined-data-example)

Using the Electric Vehicle Population Data example:

```
tb datasource create --url "https://data.wa.gov/api/views/f6w7-q2d2/rows.csv?accessType=DOWNLOAD" --name rows
tb build
tb datasource append rows "https://data.wa.gov/api/views/f6w7-q2d2/rows.csv?accessType=DOWNLOAD"
```
You should get the following quarantine error: `Error appending fixtures for 'rows': There was an error with file contents: 564 rows in quarantine.`

Inspecting the `rows_quarantine` data source:

```
tb sql "SELECT DISTINCT c__error FROM rows_quarantine"

# ────────────────────────────────────────────────────────────────────────────────────────────
# c__error: ["value '' on column 'postal_code' is not Int64", "value '' on column 'legislative_district' is not Int16", # "value '' on column 'c_2020_census_tract' is not Int64"]
# ────────────────────────────────────────────────────────────────────────────────────────────
# c__error: ["value '' on column 'electric_range' is not Int32", "value '' on column 'base_msrp' is not Int64"]
# ────────────────────────────────────────────────────────────────────────────────────────────
# c__error: ["value '' on column 'legislative_district' is not Int16"]
# ────────────────────────────────────────────────────────────────────────────────────────────
```
The problem is that some columns should be `Nullable` or have a `DEFAULT` value. Add a `DEFAULT` value of 0 for them.

Edit the `datasources/rows.datasource` file.

datasources/rows.datasource
```
DESCRIPTION >
    Generated from https://data.wa.gov/api/views/f6w7-q2d2/rows.csv?accessType=DOWNLOAD

SCHEMA >
    `vin__1_10_` String,
    `county` String,
    `city` String,
    `state` String,
    `postal_code` Int64 DEFAULT 0,
    `model_year` Int32,
    `make` String,
    `model` String,
    `electric_vehicle_type` String,
    `clean_alternative_fuel_vehicle__cafv__eligibility` String,
    `electric_range` Int32 DEFAULT 0,
    `base_msrp` Int64 DEFAULT 0,
    `legislative_district` Int16 DEFAULT 0,
    `dol_vehicle_id` Int64,
    `vehicle_location` String,
    `electric_utility` String,
    `c_2020_census_tract` Int64 DEFAULT 0
```
The dev server rebuilds the edited resources.

```
tb build
tb datasource append rows "https://data.wa.gov/api/views/f6w7-q2d2/rows.csv?accessType=DOWNLOAD"
```
No errors now, you're good to continue developing.

## Recover data from quarantine (https://www.tinybird.co/docs/forward/guides/quarantine#recover-data-from-quarantine)

If you have data in quarantine, the first thing to do is understand what caused the error and fix it. See [Fixing quarantined data example](https://www.tinybird.co/docs/forward/guides/quarantine#fixing-quarantined-data-example).

Once the schema issues are fixed, you can recover the quarantined data back to your main data source. The approach involves deploying the schema fix and a Copy Pipe that reads from quarantine at the same time, then triggering the copy to recover the data.

### Step 1: Fix the Data Source schema (https://www.tinybird.co/docs/forward/guides/quarantine#step-1-fix-the-data-source-schema)

Edit the Data Source definition so the quarantined rows become valid. Most quarantine issues come down to a wrong data type or a missing column, which you fix by changing the type, making the column nullable, adding a `DEFAULT` value, or adding and dropping columns.

Make only the minimum changes required to fix the schema, and leave anything unrelated for a later deployment.

Continuing the previous example, `datasources/rows.datasource` now has `DEFAULT 0` on the columns that were quarantining rows. See [Evolve data sources](/docs/forward/guides/evolve-data-source) for how Tinybird applies each type of change.

### Step 2: Add a Copy Pipe that reads from quarantine (https://www.tinybird.co/docs/forward/guides/quarantine#step-2-add-a-copy-pipe-that-reads-from-quarantine)

Add a Copy Pipe that selects the quarantined rows, casts them to the fixed schema, and appends them to the Data Source. Because quarantine columns are `Nullable(String)`, cast every column to its target type, and don't select the `c__error`, `c__error_column`, `c__import_id`, and `insertion_date` columns.

The Copy Pipe has no schedule, so it only runs when you trigger it.

- Tinybird CLI
- TypeScript SDK
- Python SDK
pipes/recover_rows_quarantine.pipe
```
DESCRIPTION Recovers quarantined rows into the rows data source

NODE recover
SQL >
    SELECT
        coalesce(vin__1_10_, '') AS vin__1_10_,
        coalesce(county, '') AS county,
        coalesce(city, '') AS city,
        coalesce(state, '') AS state,
        coalesce(toInt64OrNull(postal_code), 0) AS postal_code,
        coalesce(toInt32OrNull(model_year), 0) AS model_year,
        coalesce(make, '') AS make,
        coalesce(model, '') AS model,
        coalesce(electric_vehicle_type, '') AS electric_vehicle_type,
        coalesce(clean_alternative_fuel_vehicle__cafv__eligibility, '') AS clean_alternative_fuel_vehicle__cafv__eligibility,
        coalesce(toInt32OrNull(electric_range), 0) AS electric_range,
        coalesce(toInt64OrNull(base_msrp), 0) AS base_msrp,
        coalesce(toInt16OrNull(legislative_district), 0) AS legislative_district,
        coalesce(toInt64OrNull(dol_vehicle_id), 0) AS dol_vehicle_id,
        coalesce(vehicle_location, '') AS vehicle_location,
        coalesce(electric_utility, '') AS electric_utility,
        coalesce(toInt64OrNull(c_2020_census_tract), 0) AS c_2020_census_tract
    FROM rows_quarantine

TYPE COPY
TARGET_DATASOURCE rows
COPY_SCHEDULE @on-demand
COPY_MODE append
```

### Step 3: Deploy the schema fix and the Copy Pipe together (https://www.tinybird.co/docs/forward/guides/quarantine#step-3-deploy-the-schema-fix-and-the-copy-pipe-together)

Validate the deployment first:

```
tb --cloud deploy --check
```
Then deploy. The schema fix and the Copy Pipe go live in the same deployment:

```
tb --cloud deploy
```
Wait until the deployment is live before you run the copy. Changes applied with `ALTER` aren't available in staging deployments, so a copy triggered before promotion doesn't write to the fixed schema. `tb deploy` promotes the deployment for you.

### Step 4: Trigger the copy (https://www.tinybird.co/docs/forward/guides/quarantine#step-4-trigger-the-copy)

Run the Copy Pipe and wait for it to finish:

```
tb --cloud copy run recover_rows_quarantine --wait
```
Check that the rows landed in the Data Source:

```
tb --cloud sql "SELECT count() FROM rows"
```
For large recoveries, run the copy on a dedicated instance to avoid competing with your production workloads:

```
tb --cloud copy run recover_rows_quarantine --wait --on-demand-compute
```
See [on-demand compute for copy jobs](/docs/forward/core-concepts/copy-pipes#on-demand-compute-for-copy-jobs).

### Step 5: Remove the Copy Pipe (https://www.tinybird.co/docs/forward/guides/quarantine#step-5-remove-the-copy-pipe)

Once the rows are recovered, delete the Copy Pipe and deploy again:

```
tb --cloud deploy
```
This deployment doesn't change the Data Source schema, so it doesn't affect the Quarantine Data Source or its remaining rows.

---

URL: https://www.tinybird.co/docs/forward/guides/postgres-cdc-with-redpanda-connect
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Postgres CDC with Redpanda Connect \| Tinybird"
theme-color: "#171612"
description: "Learn how to ingest data from a Postgres database using Redpanda Connect and the Postgres CDC input."
inkeep:version: "forward"
---

# PostgreSQL CDC with Redpanda Connect (https://www.tinybird.co/docs/forward/guides/postgres-cdc-with-redpanda-connect#postgresql-cdc-with-redpanda-connect)

[Redpanda Connect](https://www.redpanda.com/connect) is an ecosystem of high-performance streaming connectors that serves as a simplified and powerful alternative to Kafka Connect.

Tinybird is the ideal complement to Postgres for handling OLAP workloads. The following guide shows you how to use Redpanda Connect to ingest data from a Postgres database into Tinybird.

## Before you start (https://www.tinybird.co/docs/forward/guides/postgres-cdc-with-redpanda-connect#before-you-start)

Before you connect Postgres to Redpanda, ensure:

- You have a Redpanda cluster and Redpanda Connect installed with version 4.43.0 or higher. The following instructions use Redpanda Serverless, but you can use Redpanda Cloud Dedicated or self-hosted.
- You have a PostgreSQL database with logical replication enabled.

## Connect Postgres to Redpanda (https://www.tinybird.co/docs/forward/guides/postgres-cdc-with-redpanda-connect#connect-postgres-to-redpanda)

1. In the Redpanda Cloud console, select **Connect** from the navigation menu, then select **Create pipeline**.
2. Add the pipeline configuration. You need the following information:

- Postgres connection string (`dsn`)
- Redpanda brokers (`seed_brokers`)
- SASL mechanism (`mechanism`)
- Username (`username`)
- Password (`password`)
Use the following YAML template:

```
input:
  label: "postgres_cdc"
  postgres_cdc:
    dsn: <<postgresql://user:pass@host:port/db>>
    include_transaction_markers: false
    slot_name: test_slot_native_decoder
    snapshot_batch_size: 100000
    stream_snapshot: true
    temporary_slot: true
    schema: public
    tables:
      - <<Table name>>

output:
 redpanda:
   seed_brokers:
     - ${REDPANDA_BROKERS}
   topic: <<Topic name>>
   tls:
     enabled: false
   sasl:
     - mechanism: SCRAM-SHA-512
       password: <<Password>>
       username: <<Username>>
```
See the Redpanda Connect docs for more information on the <a href="https://docs.redpanda.com/redpanda-connect/components/outputs/redpanda/">`redpanda` output</a> and <a href="https://docs.redpanda.com/redpanda-connect/components/inputs/postgres_cdc/">`postgres_cdc` input</a>.

1. Start the Redpanda Connect pipeline
Select **Create** to save and create the pipeline. This takes you back to the pipeline screen, where you can find your new pipeline. Open the new pipeline to view the logs and confirm that the pipeline is running.

Select the Topics page from the navigation menu and confirm that the topic exists and that messages are being produced.

1. Connect Redpanda to Tinybird using the Kafka Connector
Create a new data source with the Kafka Connector. See [Kafka](/docs/forward/ingest-data/connectors/kafka) for more information.

Redpanda Connect continuosly consumes changes from Postgres and pushes them to your Redpanda topic. Tinybird consumes the changes from Redpanda in real time, making them available to query with minimal latency.

## See also (https://www.tinybird.co/docs/forward/guides/postgres-cdc-with-redpanda-connect#see-also)

- <a href="https://docs.redpanda.com/redpanda-connect/components/outputs/redpanda/">`redpanda` output</a>
- <a href="https://docs.redpanda.com/redpanda-connect/components/inputs/postgres_cdc/">`postgres_cdc` input</a>

---

URL: https://www.tinybird.co/docs/forward/guides/multitenant-real-time-apis-with-clerk-and-tinybird
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Multi-Tenant Real-Time APIs With Clerk \| Tinybird"
theme-color: "#171612"
description: "In this guide, you'll learn how to build a multi-tenant real-time API with Clerk and Tinybird."
inkeep:version: "forward"
---

# Multi-tenant real-time APIs with Clerk and Tinybird (https://www.tinybird.co/docs/forward/guides/multitenant-real-time-apis-with-clerk-and-tinybird#multi-tenant-real-time-apis-with-clerk-and-tinybird)

Learn how to build a multi-tenant real-time API with Clerk and Tinybird.

You can view the [live demo](https://clerk-tinybird.vercel.app/) or browse the [GitHub repo (clerk-tinybird)](https://github.com/tinybirdco/clerk-tinybird).

## Prerequisites (https://www.tinybird.co/docs/forward/guides/multitenant-real-time-apis-with-clerk-and-tinybird#prerequisites)

This guide assumes that you have a Tinybird account, and you are familiar with creating a Tinybird workspace and pushing resources to it.

You need a working familiarity with Clerk and Next.js.

## JWT template (https://www.tinybird.co/docs/forward/guides/multitenant-real-time-apis-with-clerk-and-tinybird#jwt-template)

Create a JWT template in Clerk, and use the generated JWT to access Tinybird pipe endpoints.

In Clerk go to `Configure` > `JWT Templates` and choose Tinybird.

<!-- <figure> -->
![](/docs/_next/image?url=%252Fdocs%252Fassets%252Fguides%252Fclerk%252Fclerk-jwt-tinybird.png&w=3840&q=75)

<!-- <figcaption> -->
Clerk JWT tokens Tinybird template
<!-- </figcaption> -->

<!-- </figure> -->
Modify the Tinybird JWT template with these claims:

```
{
	"name": "frontend_jwt",
	"limits": {
		"rps": 10
	},
	"scopes": [
		{
			"type": "PIPES:READ",
			"resource": "<YOUR-TINYBIRD-PIPE-NAME>",
			"fixed_params": {
				"org": "{{org.slug}}",
				"user": "{{user.id}}"
			}
		}
	],
	"workspace_id": "<YOUR-TINYBIRD-WORKSPACE-ID>"
}
```

- Use your Tinybird admin token as signking key.
- Add as many scopes as needed, use fixed params to filter your Tinybird API endpoints.
- Configure `fixed_params` to match the parameter names and values in your Tinybird API endpoints.
From your app, request a token from Clerk using the `tinybird` template, where `tinybird` is the name you gave the template.

```
const authentication = await auth()
  const { userId, sessionId, getToken } = authentication
  const token = await getToken({ template: "tinybird" })

  fetch('https://api.tinybird.co/v0/pipes/your_pipe.json', {
  headers: {
    Authorization: `Bearer ${token}`
  }
})
```
Use this [demo project](https://www.tinybird.co/templates/clerk-jwt) to for a fully working example.

---

URL: https://www.tinybird.co/docs/forward/guides/monitoring-examples
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Monitoring examples \| Tinybird"
theme-color: "#171612"
description: "Practical examples for monitoring Tinybird workspaces, queries, jobs, and API Endpoints."
inkeep:version: "forward"
---

# Monitoring examples (https://www.tinybird.co/docs/forward/guides/monitoring-examples#monitoring-examples)

Use these examples to monitor Tinybird workspaces, queries, jobs, and Endpoints.

- Debug slow queries and failures with [system.query_log](/docs/forward/monitoring/query-log).
- Monitor Endpoint performance with [Endpoint performance](/docs/forward/monitoring/analyze-endpoints-performance).
- Track background jobs with [Workspace jobs](/docs/forward/monitoring/jobs).
- Monitor ingestion and platform operations with [Service Data Sources](/docs/forward/monitoring/service-datasources).
- Set up continuous checks with [Health checks](/docs/forward/monitoring/health-checks).

---

URL: https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Migrate From Shared to Dedicated Infra \| Tinybird"
theme-color: "#171612"
description: "Migrate your Tinybird data from shared to dedicated infrastructure without disrupting live services."
inkeep:version: "forward"
---

# Migrate from shared to dedicated infrastructure (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#migrate-from-shared-to-dedicated-infrastructure)

This guide covers migrating data from shared to dedicated infrastructure without disrupting live services.

## Pre-migration checklist (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#pre-migration-checklist)

Before starting, verify the following:

- **Workspace setup**: Your dedicated Workspace is set up and accessible. You can create it yourself if:
  - The creating user's email matches your company domain, for example, `@example.com`
  - The Workspace is created in the region of your dedicated cluster
- **Access rights**: Your team can access both the old and new Workspaces
- **Downtime planning**: The process minimizes disruption but can fail. Plan accordingly
- **Customer Success**: Coordinate with your Customer Success Data Engineer so they can assist if needed

## Migration overview (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#migration-overview)

1. Replicate the project in the new Workspace
2. Recreate access Tokens
3. Export and load dimensional tables; duplicate any periodic update processes
4. Duplicate real-time ingestion, then backfill historical data for fact/event tables
5. Point your app to the new Endpoints and Tokens
6. Stop ingestion to the old Workspace

## Step 1: Replicate the Project (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#step-1-replicate-the-project)

- If your project resources (`.datasource`, `.pipe`, `.connection` files) are already in a Git repository, clone the repo locally and proceed to the deploy step
- If your resources are **not** in a Git repo, pull them from the existing Workspace first:

```
tb pull
```

- If you use **Connectors** (Kafka, S3, DynamoDB, Snowflake, etc.), ensure your `.connection` files use the same connection names as your current Workspace
- This is a good opportunity to clean up old versions and unnecessary resources before deploying
- Log in to the new Workspace, validate, then deploy:

```
tb login
tb deploy --check
tb deploy
```

## Step 2: Recreate Access Tokens (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#step-2-recreate-access-tokens)

Tokens cannot be migrated. Recreate them in the new Workspace with the same permissions. Use these Tokens to configure ingestion and consumption later.

See [Tokens](/docs/forward/core-concepts/tokens) and the [Tokens API reference](/docs/api-reference/token-api) for details.

If you use a Connector and can reingest all data from the original source, skip Steps 3 and 4.

## Step 3: Export dimensional tables (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#step-3-export-dimensional-tables)

Dimensional tables are smaller and updated periodically, making them simpler to migrate.

### If you have a periodic update process (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#if-you-have-a-periodic-update-process)

- Identify the Token used for updates in the source Workspace
- Create a matching Token in the destination Workspace
- Duplicate the update process, pointing to the new Workspace URL and Token
- **Same region, same cloud provider**: only the Token changes
- **Different region or cloud provider**: update both Token and URL
- Keep both processes running until migration is fully validated

### If you don't have a periodic update process (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#if-you-dont-have-a-periodic-update-process)

Export via Sinks, then ingest into the new Workspace. Repeat for every dimensional Data Source.

**Export the data:**

- Create a Pipe selecting all columns explicitly (avoid `SELECT *`)
- Create an on-demand Sink to write results to S3
- Split output files to stay within [ingestion limits](/docs/forward/ingest-data)
**Ingest the data** — choose one approach:

#### Option A: Direct URL ingestion (few files) (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#option-a-direct-url-ingestion-few-files)

```
curl \
  -H "Authorization: Bearer <DATASOURCES:CREATE token>" \
  -X POST "https://api.tinybird.co/v0/datasources" \
  -d "name=stocks" \
  -d url='https://.../data.csv'
```

#### Option B: S3 connector (many files) (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#option-b-s3-connector-many-files)

- Configure an auxiliary Data Source to ingest from S3 on-demand with the filename pattern from your export
- Configure bucket access policies
- Monitor ingestion:

```
SELECT * FROM tinybird.sinks_ops_log ORDER BY timestamp DESC
```

- Copy data to the destination Data Source using a copy Pipe (select columns explicitly, match types)

## Step 4: Migrate fact and event tables (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#step-4-migrate-fact-and-event-tables)

Fact tables require maintaining ingestion continuity with no gaps or duplicates.

### 4a. Duplicate ingestion (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#4a-duplicate-ingestion)

Duplicate your ingestion process to also write to the new Workspace. Keep the original ingestion to the shared Workspace running. This creates two parallel active ingests: one continues writing to the shared Workspace, and one writes to the dedicated Workspace.

To configure the duplicate ingestion:

- **Same region, same cloud provider**: change only the Token
- **Different region or cloud provider**: change Token and URL. See [API base URLs](/docs/api-reference#regions-and-endpoints).

### 4b. Validate ingestion (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#4b-validate-ingestion)

Check via service Data Sources or the UI. Verify no data is in quarantine:

```
SELECT * FROM tinybird.datasources_ops_log
WHERE event_type = 'append'
ORDER BY timestamp DESC
```

### 4c. Identify the ingestion start point (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#4c-identify-the-ingestion-start-point)

Query the Data Source for the first ingested event (typically using a timestamp field). Use this timestamp as the cutoff for backfill.

### 4d. Export historical data (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#4d-export-historical-data)

- Create a Pipe querying the fact table up to the ingestion start point
- Create an on-demand Sink to export to S3
- Use **Parquet format** for large tables
- Partition into files < 1 GB
- Monitor the export:

```
SELECT * FROM tinybird.sinks_ops_log ORDER BY timestamp DESC
```

### 4e. Ingest backfill data (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#4e-ingest-backfill-data)

- Use the S3 connector to ingest into an auxiliary Data Source
- Copy to the destination Data Source using a copy Pipe (select columns explicitly, match types including `LowCardinality`, nullability, etc.)
- Monitor and validate as in Step 3

### 4f. Validate Endpoints (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#4f-validate-endpoints)

Confirm all Endpoints return accurate data. Materialized Views execute after the backfill copy, so data should be ready for consumption.

## Step 5: Switch over (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#step-5-switch-over)

- Point your applications to the new Workspace Endpoints
  - **Same region, same cloud provider**: change only the access Token
  - **Different region or cloud provider**: update both Token and URL
- Use the Tokens created in Step 2 with matching permissions
- Stop ingestion to the old Workspace once everything is confirmed working

## FAQ (https://www.tinybird.co/docs/forward/guides/migrate-from-shared-to-dedicated#faq)

**Memory limit or timeout errors occur during migration**

Contact Tinybird through your support or community Slack channel.

---

URL: https://www.tinybird.co/docs/forward/guides/migrate-from-postgres
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Migrate from PostgreSQL \| Tinybird"
theme-color: "#171612"
description: "Choose the right PostgreSQL migration path for Tinybird. Compare PostgreSQL table functions, CDC streams, and application writes."
inkeep:version: "forward"
---

# Migrate from PostgreSQL (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#migrate-from-postgresql)

PostgreSQL is usually where app data starts. Tinybird is where you move analytical workloads when PostgreSQL is doing too much: user-facing dashboards, high-concurrency API Endpoints, event analytics, operational reporting, or real-time aggregations over large tables.

There is no single migration path. The right choice depends on freshness requirements, table size, update patterns, operational constraints, and whether you already run Kafka-compatible infrastructure.

## Choose a migration path (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#choose-a-migration-path)

| Path | Use it for | Tradeoffs | Start here |
| --- | --- | --- | --- |
| PostgreSQL table function with Copy Pipes | Small tables, dimension tables, periodic imports, one-off backfills, prototypes. | Simple to operate. Pulls from PostgreSQL on a schedule, so freshness is bounded by the schedule. Large tables need chunking and indexed filters. Deletes need an explicit strategy. | [PostgreSQL table function](/docs/forward/ingest-data/table-functions/postgresql) |
| Change data capture to Kafka, then Tinybird Kafka connector | Real-time integrations, large event streams, production sync, low-latency analytics, updates and deletes. | More moving parts. Requires logical replication and a CDC tool such as Redpanda Connect, Debezium, Confluent, Estuary, or a managed equivalent. Best option when data must stay fresh. | [Postgres CDC with Redpanda Connect](/docs/forward/guides/postgres-cdc-with-redpanda-connect) |
| Application writes to Tinybird | New events generated by your app, migrations where PostgreSQL should stop being the analytics source. | Requires app changes. Avoids polling PostgreSQL and usually gives the cleanest event model for analytics. | [Events API](/docs/forward/ingest-data/events-api) |
| File export and batch import | Large historical backfills, archive imports, or migrations where database access is restricted. | Operationally simple, but not a live sync. You manage export, storage, and import cadence. | [Files](/docs/forward/ingest-data/local-file) |
Most migrations use more than one path:

1. Backfill historical data with the PostgreSQL table function or exported files.
2. Keep new data fresh with CDC or app writes.
3. Model updates and deletes in Tinybird with a `ReplacingMergeTree`, soft deletes, or a final materialized Data Source.
4. Move analytical queries from PostgreSQL to Tinybird Pipes and API Endpoints.

## Use the PostgreSQL table function for periodic imports (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#use-the-postgresql-table-function-for-periodic-imports)

Use the `postgresql()` table function when you want Tinybird to read directly from PostgreSQL and copy the result into a Data Source.

This is the simplest path when:

- The source table is small enough to copy within Copy Pipe limits.
- You can tolerate freshness measured in minutes or hours.
- The table is mostly append-only.
- The table has an indexed timestamp or monotonically increasing column for incremental filters.
- You are importing dimension tables that can be replaced on each run.
It is not the best default for high-volume transactional tables with constant updates and deletes. For those, use CDC.

### Example periodic import (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#example-periodic-import)

Create a target Data Source and a scheduled Copy Pipe that reads recent PostgreSQL rows.

- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/orders.datasource
```
SCHEMA >
    `order_id` UInt64,
    `customer_id` UInt64,
    `status` String,
    `amount` Float64,
    `updated_at` DateTime

ENGINE "ReplacingMergeTree(updated_at)"
ENGINE_SORTING_KEY "order_id"
```
pipes/orders_from_postgres.pipe
```
NODE updated_orders
SQL >
    %
    SELECT
        order_id,
        customer_id,
        status,
        amount,
        updated_at
    FROM postgresql(
        'postgres.example.com:5432',
        'app',
        'orders',
        {{ tb_secret("PG_USERNAME") }},
        {{ tb_secret("PG_PASSWORD") }}
    )
    WHERE updated_at >= now() - INTERVAL 10 MINUTE

TYPE copy
TARGET_DATASOURCE orders
COPY_MODE append
COPY_SCHEDULE */5 * * * *
```
Set credentials as secrets in each environment where the Copy Pipe runs:

```
tb --cloud secret set PG_USERNAME <username>
tb --cloud secret set PG_PASSWORD <password>
```
Deploy the project:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb --cloud deploy
```
For the full syntax, SDK examples, type mapping, and scheduling details, see [PostgreSQL table function](/docs/forward/ingest-data/table-functions/postgresql).

### Table function tradeoffs (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#table-function-tradeoffs)

The table function runs a query against PostgreSQL every time the Copy Pipe runs. Treat it like any other production query on your primary database.

- Add an index on the column used in the `WHERE` clause.
- Push filters down to PostgreSQL. Do not copy a large table every minute.
- Keep schedule frequency realistic for the source table size.
- Use `COPY_MODE replace` for small dimension tables.
- Use `COPY_MODE append` for event-like tables.
- Use `ReplacingMergeTree(updated_at)` when rows can be updated.
- Keep a strategy for deletes. Scheduled append jobs do not remove rows that disappeared from PostgreSQL.
For large historical imports, split the backfill into chunks by time or ID range. Each chunk should finish within PostgreSQL statement timeouts and Tinybird Copy Pipe limits.

## Use CDC for real-time PostgreSQL integration (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#use-cdc-for-real-time-postgresql-integration)

Use change data capture when PostgreSQL remains the source of truth and Tinybird needs to stay current with inserts, updates, and deletes.

The usual architecture is:

The CDC tool reads PostgreSQL logical replication or WAL changes and writes events to a Kafka-compatible topic. Tinybird consumes that topic with the [Kafka connector](/docs/forward/ingest-data/connectors/kafka).

Use CDC when:

- You need seconds-level freshness.
- You need to capture updates and deletes.
- The source table is large or changes continuously.
- You already use Kafka, Redpanda, Confluent, Debezium, Estuary, or another CDC platform.
- You want PostgreSQL query load to come from replication rather than repeated analytical scans.
CDC adds infrastructure. You need to operate or configure the CDC capture tool, Kafka-compatible storage, schemas, offsets, retries, and monitoring. The benefit is lower database impact and better freshness for production analytics.

### CDC event modeling in Tinybird (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#cdc-event-modeling-in-tinybird)

CDC events are not always the same shape as your final analytical table. Most tools emit metadata such as operation type, source timestamp, transaction position, and before or after values.

Common patterns in Tinybird:

- Store raw CDC events in a Kafka Data Source.
- Use Materialized Views to extract the current row shape into a query-friendly Data Source.
- Use `ReplacingMergeTree` with a version column for upserts.
- Represent deletes with a deleted flag or tombstone rows, then filter them from serving Pipes.
- Keep raw CDC events for replay and debugging.
For a practical setup with Redpanda Connect, see [Postgres CDC with Redpanda Connect](/docs/forward/guides/postgres-cdc-with-redpanda-connect). For the Tinybird side of the stream, see [Kafka connector](/docs/forward/ingest-data/connectors/kafka), [Confluent Cloud setup](/docs/forward/guides/confluent-cloud-setup), [AWS MSK setup](/docs/forward/guides/aws-msk-setup), and [Redpanda setup](/docs/forward/guides/redpanda-setup).

## Use app writes for new analytical events (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#use-app-writes-for-new-analytical-events)

If you control the app code and the data is event-like, consider writing new events directly to Tinybird instead of writing them to PostgreSQL first and extracting them later.

This works well for:

- Clickstream events.
- Product usage events.
- Logs and telemetry.
- Audit events.
- Analytics-only events that do not need PostgreSQL transactions.
The tradeoff is that you change the app write path. The benefit is a cleaner analytics model and less load on PostgreSQL.

Use the [Events API](/docs/forward/ingest-data/events-api), [TypeScript SDK quickstart](/docs/forward/quickstarts/typescript-sdk), or [Python SDK quickstart](/docs/forward/quickstarts/python-sdk) to write events directly.

## Backfill historical data (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#backfill-historical-data)

Even CDC migrations usually need a backfill. You have two common options:

- Use the PostgreSQL table function and run Copy Pipes over time or ID ranges.
- Export PostgreSQL data to files and import them into Tinybird.
For table function backfills, index the chunking column and check that PostgreSQL uses the index:

```
CREATE INDEX idx_orders_updated_at ON orders (updated_at);
VACUUM ANALYZE orders;

EXPLAIN
SELECT *
FROM orders
WHERE updated_at >= '2026-01-01 00:00:00'
  AND updated_at < '2026-01-02 00:00:00';
```
Then run one Copy Pipe job per chunk. Start with small chunks, measure runtime, then increase the interval if both PostgreSQL and Tinybird have headroom.

When using CDC, confirm how your CDC tool handles initial snapshots. Some tools can snapshot existing rows and then continue streaming changes from the same consistent point. If your tool does that reliably, you may not need a separate table function backfill.

## Decide how to handle updates and deletes (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#decide-how-to-handle-updates-and-deletes)

PostgreSQL tables often have mutable rows. Tinybird Data Sources are optimized for analytical reads and append-heavy ingestion, so model mutability explicitly.

Use one of these patterns:

| Source behavior | Tinybird pattern | Notes |
| --- | --- | --- |
| Append-only events | `MergeTree` ordered by query filters | Best fit. Keep events immutable. |
| Updates by primary key | `ReplacingMergeTree(version_column)` sorted by primary key | Query with `FINAL` only when needed. Prefer materializing the latest state for serving paths. |
| Deletes | Soft delete flag or tombstone event | Filter deleted rows in serving Pipes, or materialize active rows into a final Data Source. |
| Late-arriving changes | Version column from source timestamp, WAL position, or CDC event timestamp | Make the version monotonic for each primary key. |
See [Deduplication strategies](/docs/forward/guides/deduplication-strategies) and [Materialized Views](/docs/forward/core-concepts/materialized-views) for the modeling details.

## Migration checklist (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#migration-checklist)

1. Identify the PostgreSQL tables used by analytical queries.
2. Classify each table as append-only, mutable dimension, or transactional current state.
3. Pick an ingestion path per table: table function, CDC, app writes, or file import.
4. Create Tinybird Data Sources with sorting keys based on your query filters.
5. Backfill historical data.
6. Start the live sync path.
7. Build Pipes and Endpoints for app queries.
8. Compare counts and key aggregates between PostgreSQL and Tinybird.
9. Move read traffic to Tinybird.
10. Remove old analytical queries from PostgreSQL when the new path is stable.

## See also (https://www.tinybird.co/docs/forward/guides/migrate-from-postgres#see-also)

- [PostgreSQL table function](/docs/forward/ingest-data/table-functions/postgresql)
- [Postgres CDC with Redpanda Connect](/docs/forward/guides/postgres-cdc-with-redpanda-connect)
- [Kafka connector](/docs/forward/ingest-data/connectors/kafka)
- [Events API](/docs/forward/ingest-data/events-api)
- [Evolve Data Sources](/docs/forward/guides/evolve-data-source)

---

URL: https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Migrate From Self-Hosted ClickHouse® \| Tinybird"
theme-color: "#171612"
description: "Learn how to migrate from self-hosted ClickHouse to Tinybird, from understanding the benefits, translating your SQL, deployment options, and integrating with your app."
inkeep:version: "forward"
---

# Migrate from ClickHouse self-hosted to Tinybird (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#migrate-from-clickhouse-self-hosted-to-tinybird)

If you're running ClickHouse self-hosted and looking for a more streamlined, production-ready analytics platform, use this guide to migrate to Tinybird. Learn how to translate your existing ClickHouse concepts, take advantage of Tinybird's unique features, and integrate the platform into your app architecture.

## Why migrate from ClickHouse self-hosted to Tinybird? (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#why-migrate-from-clickhouse-self-hosted-to-tinybird)

While ClickHouse is an excellent OLAP database, self-hosting comes with significant operational overhead. You're responsible for:

- **Infrastructure management**: Cluster operations, sharding, replication, and scaling
- **Security**: Authentication, authorization, and network security
- **Monitoring**: Performance tuning, query optimization, and observability
- **Maintenance**: Updates, backups, disaster recovery, and capacity planning
- **Development workflow**: Building APIs, handling authentication, and creating deployment pipelines
Tinybird abstracts away these operational complexities while providing additional capabilities that transform ClickHouse from a database into a complete analytics platform.

## Benefits of Tinybird over ClickHouse self-hosted (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#benefits-of-tinybird-over-clickhouse-self-hosted)

Tinybird offers several key advantages over managing ClickHouse yourself:

### Fully managed infrastructure (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#fully-managed-infrastructure)

Tinybird handles all the operational complexity of running ClickHouse at scale:

- **Automatic scaling**: Infrastructure scales transparently based on your workload
- **Zero cluster operations**: No need to manage shards, replicas, or cluster topology
- **Built-in monitoring**: Comprehensive observability with performance metrics and alerting
- **Managed updates**: Automatic updates and patches without downtime

### API-first architecture (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#api-first-architecture)

Transform SQL queries into production-grade REST APIs:

- **Instant API endpoints**: Turn any SQL query into a scalable HTTP API with authentication
- **Parameterized queries**: Build dynamic APIs with template variables and type validation
- **Built-in authentication**: Token-based and JWT authentication with fine-grained permissions
- **Rate limiting**: Built-in protection against abuse and cost control

### Real-time data ingestion (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#real-time-data-ingestion)

Simplified data ingestion with multiple options:

- **Events API**: High-throughput streaming ingestion for real-time events via HTTP
- **Native connectors**: Managed integrations for Kafka, S3, GCS, and more
- **Data Sources API**: Batch ingestion for CSV, Parquet, and NDJSON files
- **Schema evolution**: Handle schema changes without downtime

### Development workflow (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#development-workflow)

Modern development practices built in:

- **Local development**: Full local environment with Tinybird Local container
- **Version control**: Git-based workflow for all data assets
- **CI/CD integration**: Deploy changes through your existing pipelines
- **Testing and validation**: Built-in deployment validation and rollback capabilities

### BI and tool integration (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#bi-and-tool-integration)

Connect to your existing analytics tools:

- **ClickHouse HTTP interface**: Use existing ClickHouse drivers and tools
- **PostgreSQL compatibility**: Connect BI tools like Grafana, Metabase, and Tableau
- **Query API**: Direct SQL execution for existing applications

## ClickHouse to Tinybird concept translation (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#clickhouse-to-tinybird-concept-translation)

Tinybird has an opinionated approach to structuring analytics projects using two primary file types: `.datasource` files (tables) and `.pipe` files (queries, views, and materializations).

### Database → Workspace (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#database-workspace)

In ClickHouse, you organize tables within databases. In Tinybird:

- **Organization**: Top-level container for all your projects
- **Workspace**: Equivalent to a database, contains all your data sources and pipes
Learn more about [Tinybird Core Concepts](/docs/forward/core-concepts).

### Table → Data Source (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#table-data-source)

ClickHouse tables become Tinybird data sources, defined in `.datasource` files. Data sources in Tinybird define table schemas, engines, sorting keys, and partition keys. They may also contain references to [connectors](/docs/forward/core-concepts/connections) from which they receive data.

**ClickHouse `CREATE TABLE` statement:**

ClickHouse table creation
```
CREATE TABLE user_events
(
    user_id UInt32 ,
    event_type String,
    timestamp DateTime,
    properties JSON
)
ENGINE = MergeTree
ORDER BY (user_id, timestamp)
PARTITION BY toYYYYMM(timestamp)
```
**Tinybird `datasources/user_events.datasource`:**

user_events.datasource
```
DESCRIPTION >
    User events tracking with properties for analytics

SCHEMA >
    `user_id` UInt32 `json:$.user_id`,
    `event_type` String `json:$.event_type`,
    `timestamp` DateTime `json:$.timestamp`,
    `properties` String `json:$.properties`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "user_id, timestamp"

# optional connector configuration
KAFKA_CONNECTION_NAME kafka_connection # The name of the .connection file
KAFKA_TOPIC topic_name
KAFKA_GROUP_ID {{ tb_secret("KAFKA_GROUP_ID") }}
```
Key differences:

- **JSONPath expressions**: Map JSON fields to columns automatically
- **Declarative schema**: Define structure without SQL DDL
- **Built-in description**: Document your data sources inline
- **Source connector definitions**: Define where the data comes from
Learn more about [data sources](/docs/forward/core-concepts/data-sources).

### Views and parametrized queries → Pipes (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#views-and-parametrized-queries-pipes)

ClickHouse views become Tinybird pipes, which can be used for to define API endpoints, materializations, internal queries, scheduled copies, sinks, or as building blocks for other pipes.

**ClickHouse parametrized view:**

ClickHouse parametrized view
```
CREATE VIEW user_activity_summary
AS SELECT
    user_id,
    event_type,
    count() AS event_count,
    min(timestamp) AS first_event,
    max(timestamp) AS last_event
FROM user_events
WHERE timestamp >= {start_date:DateTime}
  AND timestamp <= {end_date:DateTime}
  AND user_id = {target_user:UInt32}
GROUP BY user_id, event_type
ORDER BY event_count DESC
```
**Tinybird `pipes/user_activity_summary.pipe`:**

user_activity_summary.pipe
```
DESCRIPTION >
    User activity summary with date range and user filtering

NODE activity_summary
SQL >
    %
    SELECT
        user_id,
        event_type,
        count() AS event_count,
        min(timestamp) AS first_event,
        max(timestamp) AS last_event
    FROM user_events
    WHERE timestamp >= {{DateTime(start_date)}}
      AND timestamp <= {{DateTime(end_date)}}
      AND user_id = {{UInt32(target_user)}}
    GROUP BY user_id, event_type
    ORDER BY event_count DESC
```
Key differences:

- **Template variables**: Use `%` token at the beginning of a query and `{{Type(variable)}}` syntax for parameters.
- **Advanced logic**: Support for `{% %}` [advanced templating functions](/docs/forward/query-data/query-parameters#advanced-templating-using-dynamic-parameters)
- **Node structure**: Organize complex queries into multiple nodes
- **Type safety**: Automatic type validation for parameters
Learn more about [pipes](/docs/forward/core-concepts/pipes).

### Materialized Views → Materialized Views (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#materialized-views-materialized-views)

ClickHouse incremental materialized views translate to Tinybird materialized views with target data sources.

**ClickHouse materialized view:**

ClickHouse materialized view
```
-- Target table
CREATE TABLE user_metrics_hourly
(
    hour DateTime,
    user_id UInt32,
    event_count AggregateFunction(count),
    unique_events AggregateFunction(uniq, String)
)
ENGINE = AggregatingMergeTree()
ORDER BY (hour, user_id);

-- Materialized view
CREATE MATERIALIZED VIEW user_metrics_hourly_mv
TO user_metrics_hourly
AS SELECT
    toStartOfHour(timestamp) AS hour,
    user_id,
    countState() AS event_count,
    uniqState(event_type) AS unique_events
FROM user_events
GROUP BY hour, user_id;
```
**Tinybird implementation:**

Target data source (`datasources/user_metrics_hourly.datasource`):

user_metrics_hourly.datasource
```
DESCRIPTION >
    Hourly aggregated user metrics with state functions

SCHEMA >
    `hour` DateTime,
    `user_id` UInt32,
    `event_count` AggregateFunction(count, UInt64),
    `unique_events` AggregateFunction(uniq, String)

ENGINE "AggregatingMergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(hour)"
ENGINE_SORTING_KEY "hour, user_id"
```
Materialized view (`materializations/user_metrics_hourly.pipe`):

user_metrics_hourly.pipe
```
DESCRIPTION >
    Incremental materialization of user metrics

NODE hourly_aggregation
SQL >
    SELECT
        toStartOfHour(timestamp) AS hour,
        user_id,
        countState() AS event_count,
        uniqState(event_type) AS unique_events
    FROM user_events
    GROUP BY hour, user_id

TYPE MATERIALIZED
DATASOURCE user_metrics_hourly
```
Learn more about [materialized views](/docs/forward/core-concepts/materialized-views).

### Refreshable Materialized Views → Copy Pipes (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#refreshable-materialized-views-copy-pipes)

ClickHouse refreshable materialized views become Tinybird copy pipes with scheduling.

**ClickHouse refreshable materialized view:**

ClickHouse refreshable materialized view
```
CREATE MATERIALIZED VIEW daily_user_summary
REFRESH EVERY 1 HOUR
ENGINE = MergeTree()
ORDER BY (date, user_id)
AS SELECT
    toDate(timestamp) AS date,
    user_id,
    count() AS total_events,
    countDistinct(event_type) AS unique_event_types
FROM user_events
GROUP BY date, user_id;
```
**Tinybird implementation:**

Target Data Source (`datasources/daily_user_summary.datasource`):

daily_user_summary.datasource
```
DESCRIPTION >
    Daily user activity summary

SCHEMA >
    `date` Date,
    `user_id` UInt32,
    `total_events` UInt64,
    `unique_event_types` UInt64

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(date)"
ENGINE_SORTING_KEY "date, user_id"
```
Copy Pipe (`copies/daily_user_summary.pipe`):

daily_user_summary.pipe
```
DESCRIPTION >
    Hourly copy job for daily user summaries

NODE daily_aggregation
SQL >
    SELECT
        toDate(timestamp) AS date,
        user_id,
        count() AS total_events,
        countDistinct(event_type) AS unique_event_types
    FROM user_events
    GROUP BY date, user_id

TYPE COPY
TARGET_DATASOURCE daily_user_summary
COPY_SCHEDULE "0 * * * *"  -- Every hour
```
Learn more about [copy pipes](/docs/forward/core-concepts/copy-pipes).

### Schema evolution (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#schema-evolution)

Evolving your schema in Tinybird is handled through deployments. Edit your `.datasource` or `.pipe` files and deploy changes:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
# Validate changes before deploying
tb deploy --check

# Deploy changes
tb deploy
```
Tinybird automatically handles backfills and schema migrations based on your changes.

Learn more about [Schema Evolution](/docs/forward/guides/evolve-data-source) and [Deployments](/docs/forward/core-concepts/deployments).

## Migrate data from ClickHouse (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#migrate-data-from-clickhouse)

After you translate schemas into Tinybird Data Sources, backfill historical data from ClickHouse. The usual path is:

1. Export each ClickHouse table, or each partition of a large table, to Parquet.
2. Create the target Data Source in Tinybird.
3. Import the Parquet files with `tb datasource append`, the Data Sources API, or an S3 connector.
4. Validate row counts and key aggregates before moving read traffic.
Prefer Parquet for backfills. It preserves column types better than CSV, compresses well, and avoids line-based parsing overhead for large tables.

### Export ClickHouse data to Parquet (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#export-clickhouse-data-to-parquet)

For small and medium tables, export the full table:

Export a ClickHouse table to Parquet
```
clickhouse-client \
  --host <CLICKHOUSE_HOST> \
  --user <CLICKHOUSE_USER> \
  --password <CLICKHOUSE_PASSWORD> \
  --query "SELECT * FROM analytics.user_events FORMAT Parquet" \
  > user_events.parquet
```
For large tables, export in chunks. Use the same partitioning or time column you use for queries:

Export one month of data
```
clickhouse-client \
  --host <CLICKHOUSE_HOST> \
  --user <CLICKHOUSE_USER> \
  --password <CLICKHOUSE_PASSWORD> \
  --query "
    SELECT *
    FROM analytics.user_events
    WHERE timestamp >= '2026-01-01 00:00:00'
      AND timestamp < '2026-02-01 00:00:00'
    FORMAT Parquet
  " \
  > user_events_2026_01.parquet
```
Chunking keeps each import job smaller, makes retries easier, and supports comparing counts per interval.

### Import local Parquet files (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#import-local-parquet-files)

Create the Tinybird Data Source first, then append each exported file:

Import a Parquet backfill file
```
tb --cloud datasource append user_events user_events_2026_01.parquet
```
You can also use the Data Sources API directly:

Import Parquet with the Data Sources API
```
curl \
  -H "Authorization: Bearer <DATASOURCES:APPEND token>" \
  -X POST "https://<your_host>/v0/datasources?mode=append&name=user_events" \
  -F parquet=@user_events_2026_01.parquet
```
Use this path for one-off migrations and controlled backfills from your local machine or migration runner. See [Ingest data from files](/docs/forward/ingest-data/local-file).

### Import from S3 (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#import-from-s3)

For larger migrations, export Parquet files to S3 and ingest them from there. This avoids moving large files through a laptop and gives you a durable staging area for retries.

Use one of these options:

- Append individual S3 files with `tb datasource append` or the Data Sources API using a URL.
- Configure the [S3 Connector](/docs/forward/ingest-data/connectors/s3) when files continue landing in S3 and Tinybird should ingest them automatically.
For a one-off backfill with signed URLs:

Import a Parquet file from S3
```
tb --cloud datasource append user_events "https://<bucket>.s3.amazonaws.com/backfills/user_events_2026_01.parquet?<signed_query_string>"
```
For recurring imports, define an S3 Connection and Data Source, then deploy it. Tinybird detects matching files in the bucket and ingests them into the Data Source. See [S3 connector](/docs/forward/ingest-data/connectors/s3).

### Validate the backfill (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#validate-the-backfill)

Compare counts and aggregates between ClickHouse and Tinybird for each chunk:

ClickHouse validation query
```
SELECT
  toYYYYMM(timestamp) AS month,
  count() AS rows,
  uniqExact(user_id) AS users
FROM analytics.user_events
GROUP BY month
ORDER BY month
```
Tinybird validation query
```
tb --cloud sql "
  SELECT
    toYYYYMM(timestamp) AS month,
    count() AS rows,
    uniqExact(user_id) AS users
  FROM user_events
  GROUP BY month
  ORDER BY month
"
```
When counts diverge, validate one chunk at a time. Check filters, time zones, nullable columns, and whether the source table contains duplicate or replacing rows that need `FINAL` in the ClickHouse export query.

## Integrate Tinybird into your app (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#integrate-tinybird-into-your-app)

### Data ingestion services (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#data-ingestion-services)

Tinybird provides multiple ways to ingest data. You do not need to set up ingestion infrastructure or use `INSERT` statements. Tinybird offers several native ingestion connectors and APIs.

#### Events API (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#events-api)

Best for real-time streaming data, and can be used to send data directly from the client to Tinybird. Send JSON or NDJSON events via HTTP POST:

Events API ingestion
```
curl -X POST "https://<your_host>/v0/events?name=user_events" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"user_id": 123, "event_type": "page_view", "timestamp": "2024-01-15 10:30:00", "properties": "{\"page\": \"/dashboard\"}"}'
```
Learn more about the [Events API](/docs/api-reference/events-api).

#### Data Sources API (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#data-sources-api)

For batch ingestion of CSV, Parquet, or NDJSON files:

Batch file ingestion
```
curl -X POST "https://<your_host>/v0/datasources?mode=append&name=user_events" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -F "csv=@user_events.csv"
```
Learn more about the [Data Sources API](/docs/api-reference/datasource-api).

#### Connectors (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#connectors)

Managed integrations for popular data sources:

- **Kafka**: Real-time streaming with automatic schema detection
- **S3/GCS**: Automated file ingestion from cloud storage
Connectors handle authentication, schema mapping, and error handling automatically.

Learn more about [Connectors](/docs/forward/core-concepts/connections).

#### Table functions (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#table-functions)

Tinybird supports ClickHouse table functions for specialized data ingestion and transformation:

- **`url()`**: Load data directly from HTTP/HTTPS URLs
- **iceberg()**: Import data from Apache Iceberg tables
- **postgres()**: Import data from PostgreSQL databases
- **mysql()**: Import data from MySQL databases
Example using postgres() to import data from PostgreSQL:

pipes/import_from_postgres.pipe
```
DESCRIPTION >
    Import user data from PostgreSQL database

NODE postgres_import
SQL >
    SELECT
        user_id,
        email,
        created_at,
        last_login
    FROM postgres(
        'postgresql://username:password@hostname:5432/database',
        'users'
    )
    WHERE created_at >= yesterday()

TYPE COPY
TARGET_DATASOURCE user_data
COPY_SCHEDULE "0 */6 * * *"  -- Every 6 hours
```
This example creates a scheduled copy pipe that imports user data from a PostgreSQL table every 6 hours.

Learn more about [table functions](/docs/forward/ingest-data/table-functions).

### Query development (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#query-development)

Tinybird offers flexible options for querying data, from direct SQL to API endpoints.

#### Pipes vs Query API (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#pipes-vs-query-api)

**Pipes** are the default approach for production applications:

- **Type-safe parameters**: Automatic validation and conversion
- **Versioning**: Track changes and roll back if needed
- **Performance optimization**: Automatic query optimization and caching
- **Security**: Fine-grained access control with tokens
**Query API** for direct SQL execution:

- **Migration path**: Easy transition from existing ClickHouse applications
- **Ad-hoc analysis**: Quick data exploration and debugging
- **BI tools**: Direct integration with SQL-based tools
Example API endpoint from a pipe:

endpoints/user_activity.pipe
```
TOKEN analytics_api READ

DESCRIPTION >
    User activity API endpoint with authentication

NODE user_metrics
SQL >
    %
    SELECT
        user_id,
        event_type,
        count() AS event_count,
        max(timestamp) AS last_activity
    FROM user_events
    WHERE user_id = {{UInt32(user_id)}}
      AND timestamp >= {{DateTime(start_date, '2024-01-01 00:00:00')}}
    GROUP BY user_id, event_type
    ORDER BY event_count DESC

TYPE ENDPOINT
```
Access the endpoint:

```
curl "https://<your_host>/v0/pipes/user_activity.json?user_id=123&start_date=2024-01-01" \
  -H "Authorization: Bearer READ_TOKEN"
```
Learn more about [API Endpoints](/docs/forward/core-concepts/api-endpoints).

### BI integration using ClickHouse interface (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#bi-integration-using-clickhouse-interface)

Connect your existing BI tools using the [ClickHouse HTTP interface](/docs/forward/query-data/clickhouse-interface)

## Deployment options (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#deployment-options)

Tinybird offers flexible deployment options to meet your infrastructure and compliance requirements.

### Tinybird Cloud hosting (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#tinybird-cloud-hosting)

**Fully managed service** with global regions:

- **No infrastructure management**: Tinybird handles all operations
- **Global availability**: Multiple regions across AWS and GCP
- **Automatic scaling**: Infrastructure scales as needed based on selected plan
- **Enterprise features**: SSO, audit logs, advanced security controls

### Tinybird self-managed regions (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#tinybird-self-managed-regions)

**Bring Your Own Cloud** deployment for enhanced control:

- **Data sovereignty**: Keep data in your cloud environment
- **Network isolation**: Deploy within your VPC/VNet
- **Compliance**: Meet specific regulatory requirements
- **Custom configurations**: Tailored performance optimizations
**Benefits of self-managed regions:**

- Complete control over infrastructure and data location
- Integration with existing cloud commitments and reserved instances
- Direct connection to internal data sources without network egress
- Custom security policies and network configurations
**Deployment process:**

1. Register a self-managed region with Tinybird Cloud
2. Deploy Tinybird infrastructure in your cloud environment
3. Connect and manage through Tinybird CLI with `--region` flag
Self-managed regions are currently in beta. Contact Tinybird support for deployment assistance and requirements.

Learn more about [self-managed Regions](/docs/forward/core-concepts/self-managed).

## Open source projects using Tinybird (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#open-source-projects-using-tinybird)

Many successful open source projects have adopted Tinybird for their analytics needs:

- **[Ghost](https://ghost.org/)**: Publishing platform using Tinybird for member analytics and engagement metrics
- **[Papermark](https://www.papermark.io/)**: Document sharing platform with real-time analytics on document views and interactions
- **[Inbox Zero](https://www.getinboxzero.com/)**: Email management tool using Tinybird for email analytics and automation data
- **[LocalStack](https://localstack.cloud/)**: Cloud development platform using Tinybird for usage analytics and performance monitoring
- **[OpenStatus](https://www.openstatus.dev/)**: Website monitoring platform built on Tinybird for real-time uptime and performance analytics
- **[Dub](https://dub.co/)**: Link management platform using Tinybird for click analytics and link performance tracking
Each leverages Tinybird's real-time capabilities, API-first architecture, and managed infrastructure to focus on their core product instead of analytics infrastructure, while offering the flexibility between hosted and self-hosted deployments..

## Migration strategy (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#migration-strategy)

To successfully migrate from ClickHouse self-hosted to Tinybird:

### 1. Assessment and planning (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#1-assessment-and-planning)

- **Inventory your current setup**: Document tables, views, materialized views, and queries
- **Identify integration points**: Map how your app currently interacts with ClickHouse
- **Plan data migration**: Choose between full migration or gradual transition

### 2. Local development setup (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#2-local-development-setup)

```
# Install Tinybird CLI
curl https://tinybird.co | sh

# Authenticate
tb login

# Start local development environment
tb local start
```

### 3. Schema translation (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#3-schema-translation)

- Convert ClickHouse tables to `.datasource` files
- Transform views and materialized views to `.pipe` files
- Set up materialized views and copy pipes for aggregations

### 4. Data migration (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#4-data-migration)

- Export historical ClickHouse data to Parquet and import it into Tinybird. See [Migrate data from ClickHouse](https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#migrate-data-from-clickhouse).
- Use the Data Sources API or `tb datasource append` for one-off bulk imports.
- Use the S3 connector for large backfills or recurring file imports.
- Set up real-time ingestion with the Events API or connectors.
- Validate data consistency between systems.

### 5. Application integration (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#5-application-integration)

- Replace direct ClickHouse queries with Tinybird API endpoints or requests to the Query API
- Update authentication to use Tinybird tokens
- Configure BI tools to use Tinybird's ClickHouse interface

### 6. Production deployment (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#6-production-deployment)

- Deploy to Tinybird Cloud or set up self-managed region
- Configure monitoring and alerting via Tinybird service data sources
- Gradually shift traffic from ClickHouse to Tinybird

## Next steps (https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse#next-steps)

Ready to start your migration? Here are some immediate next steps:

1. **[Sign up for a free Tinybird account](https://cloud.tinybird.co/signup)** and explore the platform
2. **[Follow the quick start guide](/docs/forward/quickstarts/cli)** to understand core concepts
3. **[Install the CLI](/docs/forward/quickstarts/cli#install-the-cli)** and set up local development
4. **[Join the Tinybird community](https://www.tinybird.co/community)** for migration support and best practices
For complex migrations or enterprise requirements, [contact the Tinybird team](https://www.tinybird.co/contact-us) for personalized assistance and migration planning.

---

URL: https://www.tinybird.co/docs/forward/guides/migrate-from-classic
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Migrate from Tinybird Classic \| Tinybird"
theme-color: "#171612"
description: "Migrate your project from Tinybird Classic to Tinybird Forward."
inkeep:version: "forward"
---

# Migrate from Tinybird Classic (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#migrate-from-tinybird-classic)

Tinybird Forward introduces a new way of working with your data projects, with changes to APIs and CLI that may be incompatible with Tinybird Classic. If you're starting a new project, see the [Get started](/docs/forward) guide.

Migration upgrades your existing workspace to Forward in place. Your tables and data remain in the workspace and are available in Forward after migration. `tb --cloud pull` downloads your project definitions as Forward-compatible datafiles. It does not download table contents. `tb migrate-to-forward` then updates the workspace’s project definition and deploys it using Forward.

## Why migrate to Forward (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#why-migrate-to-forward)

Forward is a significant evolution of the Tinybird developer experience. Migrating unlocks the following benefits:

- **Deployments**: Project iteration is easier. You ship a new version of the project, and Tinybird handles all backfills and table exchanges automatically. No more `tb push datasource_v2` and copying data from `datasource_v1`.
- **Git-native deployments and CI/CD**: Every change ships through your repository. Use the built-in [CI/CD workflows](/docs/forward/development-workflow/cicd) for automated validation, [preview deployments](/docs/forward/core-concepts/deployments), and safe rollouts to production.
- **Local development**: Build and iterate against [Tinybird Local](/docs/forward/core-concepts/tinybird-local) with full parity to Tinybird Cloud. Test changes on your machine before they ever reach a shared environment. And easier to verify in your CI with the local image.
- **Faster, simpler testing**: [Fixture tests](/docs/forward/guides/test-your-project) make it easier to define realistic scenarios and catch regressions before deploy, replacing the heavier regression and data quality test workflows from Classic.
- **Cleaner project structure**: First-class [connection files](/docs/forward/dev-reference/datafiles/connection-files), [secrets](/docs/forward/dev-reference/commands/tb-secret), and [generic pipes](/docs/forward/core-concepts/pipes#create-generic-pipes) replace ad-hoc include files and per-environment glue, so projects stay portable and reviewable in code.
- **Modern, consistent CLI**: A redesigned `tb` command provides a unified workflow for local development, cloud deployments, and testing, with clearer commands and better feedback.
- **AI friendly**: Use the agent skills `npx skills add tinybirdco/tinybird-agent-skills` to teach Tinybird best practices to your coding agent.
- **Active development**: New features land in Forward first. Classic continues to receive critical fixes, but the roadmap (new connectors, deployment improvements, agent integrations) is Forward-only.

## Considerations before migrating (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#considerations-before-migrating)

Before migrating your workspace from Tinybird Classic, understand these key differences in Forward:

- Development is file based. You edit `.pipe` and `.datasource` files in a Cloud Branch or locally using [Tinybird Local](/docs/forward/core-concepts/tinybird-local). Check [Development Workflow](/docs/forward/development-workflow) for more details.
- The following features have limitations or require changes:

| Feature | Status | Solution/Alternative |
| --- | --- | --- |
| BI Connector | Not supported | Use the [ClickHouse® HTTP Interface](/docs/forward/query-data/clickhouse-interface) instead. Most BI tools support ClickHouse® HTTP connections. |
| Include files | Not supported | Use [tb secret](/docs/forward/dev-reference/commands/tb-secret) for connector credentials and [generic pipes](/docs/forward/core-concepts/pipes#create-generic-pipes) to reuse query logic. See [Replace include files](https://www.tinybird.co/docs/forward/guides/migrate-from-classic#replace-include-files) for migration steps. |
| CI/CD workflows | Different commands | Forward uses different CLI commands. See [CI/CD](/docs/forward/development-workflow/cicd) for details. |
| Testing strategy | Different approach | Regression tests and data quality tests are not supported in Forward. Fixture tests have been enhanced for easier test creation and management. See [Test your project](/docs/forward/guides/test-your-project) for details. |
| Resource-scoped tokens with `:sql_filter` | Not supported | Remove all tokens using the `:sql_filter` suffix, for example, `DATASOURCES:READ:datasource_name:sql_filter`, before migrating. Use [JWTs](/docs/forward/core-concepts/jwt) instead. |
If these changes work for your use case, continue reading to learn how to migrate.

Migration is permanent and cannot be reversed. After deploying with Forward, you cannot switch your workspace back to Classic.

## Prepare to migrate (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#prepare-to-migrate)

Both migration paths start from the same setup: install the Forward CLI, authenticate, and pull your project.

1
### Install the Tinybird Forward CLI (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#install-the-tinybird-forward-cli)

Run the following command to install the Tinybird Forward CLI and the Tinybird Local container:

- For macOS and Linux
- For Windows

```
curl https://tinybird.co | sh
```
See [Tinybird Local](/docs/forward/core-concepts/tinybird-local#install-tinybird) for more information.

The migration requires Forward CLI version 4.6.4 or later. Check your version with `tb --version` and run `tb update` if it's lower.

**Managing CLI Versions**: Having both Tinybird Classic and Forward CLIs installed can cause version conflicts since both use the `tb` command. To avoid these conflicts, consider:

1. Using the uv Python package manager to keep both CLIs completely isolated (recommended):

```
# For Classic CLI
uvx --from tinybird-cli@latest tb

# For Forward CLI
uvx --from tinybird@latest tb
```

1. Creating aliases in your shell configuration:

```
# Add to .bashrc or .zshrc
alias tb-classic="path/to/classic/tb"
alias tb-forward="path/to/forward/tb"
```

1. Using separate virtual environments for each CLI version.
This ensures you use the correct CLI version for each operation during migration.

The following steps use the uv Python package.

2
### Authenticate to your workspace (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#authenticate-to-your-workspace)

Authenticate to your workspace using the Classic CLI:

```
uvx --from tinybird-cli@latest tb auth --interactive
```
Follow the prompts to complete authentication.

3
### Pull your project (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#pull-your-project)

Pull your datafiles from Tinybird Cloud with the Forward CLI:

```
tb --cloud pull
```
For connector-backed projects, this writes a `.connection` file per connector under `connections/` and the data sources under `datasources/`, and for S3, GCS, Kafka, and DynamoDB it stores the credentials as [secrets](/docs/forward/dev-reference/commands/tb-secret) referenced with `tb_secret(...)`. If a secret of the same name already exists, the pull reuses it and prints a warning so you can confirm the value matches before deploying.

## Migrate to Forward (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#migrate-to-forward)

`tb migrate-to-forward` completes the migration end to end. It works for projects with and without connectors (S3, GCS, Kafka, DynamoDB). If it reports changes it can't reconcile, contact [Tinybird support](/docs/forward/support).

1
### Audit the pulled connector files (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#audit-the-pulled-connector-files)

If your project doesn't use connectors, skip to the next step.

Open each pulled `.connection` file and confirm it has the connector type, credentials reference, and region. The exact fields depend on the connector:

- S3
- GCS
- Kafka
- DynamoDB
An access-key connection references the secret key with `tb_secret(...)` and keeps the access key ID and region inline:

connections/s3_events.connection
```
TYPE s3
S3_ACCESS_KEY AKIAEXAMPLE
S3_SECRET {{ tb_secret("s3_secret_s3_events") }}
S3_REGION eu-west-1
```
An IAM-role connection references the role ARN with `tb_secret(...)`:

connections/s3_events.connection
```
TYPE s3
S3_ARN {{ tb_secret("s3_role_arn_s3_events") }}
S3_REGION eu-west-3
```
Then open each connector-backed `.datasource` file and confirm it references the connection by name and keeps the import settings. For DynamoDB, confirm it keeps the `ReplacingMergeTree` settings that updates and deletes rely on:

datasources/orders.datasource
```
ENGINE "ReplacingMergeTree"
ENGINE_SORTING_KEY "<partition_key>, <sort_key>"
ENGINE_VER "_timestamp"
ENGINE_IS_DELETED "_is_deleted"

IMPORT_CONNECTION_NAME 'ddb_orders'
IMPORT_TABLE_ARN 'arn:aws:dynamodb:us-east-1:123456789012:table/orders'
IMPORT_EXPORT_BUCKET 'my-dynamodb-export-bucket'
```
For DynamoDB, use only the S3 bucket name in `IMPORT_EXPORT_BUCKET`, not an `s3://` URI. Treat `IMPORT_TABLE_ARN` and `IMPORT_EXPORT_BUCKET` as immutable: the first Forward deployment should preserve the existing DynamoDB binding, not change it.

2
### Run the migration (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#run-the-migration)

Run the migration command:

```
tb migrate-to-forward
```
This command deletes branches and releases, runs a deploy check, generates a `tinybird.config.json` file, switches the workspace to Forward, and deploys it. After it finishes, query a few endpoints and confirm ingestion continues.

If you have CI/CD configured, push the changes to your repository to complete the migration. See [CI/CD](/docs/forward/development-workflow/cicd) for details on the CI workflow, CD workflow, and creating preview deployments.

If your workspace uses AWS PrivateLink for private networking, the migration doesn't change it. Private networking is configured at the workspace and region level, not in the datafiles you migrate, and the workspace keeps the same region. Connectors reach AWS through the same IAM role and endpoints before and after migration.

The migration is complete. Your project continues working as expected. You do not need to change your Tokens, Endpoint URLs, or anything else.

## Common migration errors (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#common-migration-errors)

Common errors and changes include (but are not limited to):

### Replace include files (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#replace-include-files)

Include files are not supported in Forward. The fix depends on your use of include files:

- If you use include files to store secrets, use [tb secret](/docs/forward/dev-reference/commands/tb-secret) to set secrets in your local and cloud environments.
- If you use include files to reuse query logic, you can create [generic pipes](/docs/forward/core-concepts/pipes#create-generic-pipes) and reference them in your endpoint pipes. For example:
reusable_filters.pipe
```
NODE apply_params
SQL >
    %
    SELECT * FROM my_datasource
    WHERE
        tenant_id = {{ String(tenant) }}
        AND date BETWEEN {{ Date(start_date) }} AND {{ Date(end_date) }}
```
my_endpoint.pipe
```
NODE endpoint
SQL >
    %
    SELECT * FROM reusable_filters

TYPE endpoint
```

## Next steps (https://www.tinybird.co/docs/forward/guides/migrate-from-classic#next-steps)

- Learn about working with Forward in the [Forward documentation](/docs/forward).

---

URL: https://www.tinybird.co/docs/forward/guides/lambda-architecture
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Build a lambda architecture in Tinybird \| Tinybird"
theme-color: "#171612"
description: "In this guide, you'll learn a useful alternative processing pattern for when the typical Tinybird flow doesn't fit."
inkeep:version: "forward"
---

# Build a lambda architecture in Tinybird (https://www.tinybird.co/docs/forward/guides/lambda-architecture#build-a-lambda-architecture-in-tinybird)

Use this alternative processing pattern when the typical Tinybird flow doesn't fit.

The typical flow is Data Source --> incremental transformation through Materialized Views --> API Endpoint publication. When Materialized Views don't fit your processing requirements, use Copy Pipes to create intermediate Data Sources that keep your Endpoints performant.

## The ideal Tinybird flow (https://www.tinybird.co/docs/forward/guides/lambda-architecture#the-ideal-tinybird-flow)

You ingest data (usually streamed in, but can also be in batch), transform it using SQL, and serve the results of the queries via [parameterizable](/docs/forward/query-data/query-parameters) Endpoints. Tinybird provides freshness, low latency, and high concurrency: Your data is ready to be queried as soon as it arrives.

<!-- <figure> -->
![Data flow with Data Source and Endpoint](/docs/_next/image?url=%252Fdocs%252Fimg%252Fguides-lambda-1.png&w=3840&q=75)

<!-- <figcaption> -->
Data flow with Data Source and Endpoint
<!-- </figcaption> -->

<!-- </figure> -->
Sometimes, transforming the data at query time isn't ideal. Some operations - doing aggregations, or extracting fields from JSON - are better if done at ingest time, then you can query that prepared data. [Materialized Views](/docs/forward/core-concepts/materialized-views) are perfect for this kind of situation. They're triggered at ingest time and create intermediate tables (data sources in Tinybird lingo) to keep your Endpoints performance super efficient.

<!-- <figure> -->
![Data flow with Data Source, MV, and Endpoint](/docs/_next/image?url=%252Fdocs%252Fimg%252Fguides-lambda-2.png&w=3840&q=75)

<!-- <figcaption> -->
Data flow with Data Source, MV, and Endpoint
<!-- </figcaption> -->

<!-- </figure> -->
The best practice for this approach is usually having a Materialized View (MV) per use case:

<!-- <figure> -->
![Materialized Views for different use cases](/docs/_next/image?url=%252Fdocs%252Fimg%252Fguides-lambda-3.png&w=3840&q=75)

<!-- <figcaption> -->
Materialized Views for different use cases
<!-- </figcaption> -->

<!-- </figure> -->
If your use case fits in these first two paragraphs, stop reading. No need to over-engineer it.

## When the ideal flow isn't enough (https://www.tinybird.co/docs/forward/guides/lambda-architecture#when-the-ideal-flow-isnt-enough)

There are some cases where you may need intermediate Data Sources (tables) and Materialized Views don't fit.

- Most common: Things like Window Functions where you need to check the whole table to make calculations.
- Fairly common: Needing an Aggregation MV over a deduplication table (ReplacingMergeTree).
- Scenarios where Materialized Views fit but aren't super efficient (hey `uniqState`).
- And lastly, one of the hardest problems in syncing OLTP and OLAP databases: Change data capture (CDC).
Want to know more about *why* Materialized Views don't work in these cases? [Read the docs.](/docs/forward/core-concepts/materialized-views#limitations)

For example, consider the *Aggregation Materialized Views over deduplication DS* scenario.

Deduplication in Tinybird happens asynchronously, during merges, which you can't force in Tinybird. That's why you always have to add `FINAL` or the `-Merge` combinator when querying.

Materialized Views only see the block of data being processed. When materializing an aggregation, the Materialized View processes any new row, whether it has a new or duplicate ID. That's why this pattern fails.

<!-- <figure> -->
![](/docs/_next/image?url=%252Fdocs%252Fimg%252Fguides-lambda-4.png&w=3840&q=75)

<!-- <figcaption> -->
Aggregating MV over deduplication DS doesn't work as expected
<!-- </figcaption> -->

<!-- </figure> -->

## Solution: Use an alternative architecture with Copy Pipes (https://www.tinybird.co/docs/forward/guides/lambda-architecture#solution-use-an-alternative-architecture-with-copy-pipes)

Tinybird has another kind of Pipe that helps here: [Copy Pipes](/docs/forward/core-concepts/copy-pipes).

At a high level, they're a helpful `INSERT INTO SELECT`, and they can be set to execute following a cron expression. You write your query, and (either on a recurring basis or on demand), the Copy Pipe appends the result in a different table.

So, in this example, you can have a clean, deduplicated snapshot of your data, with the correct Sorting Keys, and can use it to materialize:

<!-- <figure> -->
![](/docs/_next/image?url=%252Fdocs%252Fimg%252Fguides-lambda-5.png&w=3840&q=75)

<!-- <figcaption> -->
Copy Pipes to the rescue
<!-- </figcaption> -->

<!-- </figure> -->

## Avoid loss of freshness (https://www.tinybird.co/docs/forward/guides/lambda-architecture#avoid-loss-of-freshness)

*"But if you recreate the snapshot every hour, day, or another interval, aren't you losing freshness?"* Yes. That's when the lambda architecture comes into play:

<!-- <figure> -->
![](/docs/_next/image?url=%252Fdocs%252Fimg%252Fguides-lambda-6.png&w=3840&q=75)

<!-- <figcaption> -->
Lambda/Kappa Architecture
<!-- </figcaption> -->

<!-- </figure> -->
Combine the already-prepared data with the same operations over the fresh data being ingested at that moment. This provides higher performance despite complex logic over both fresh and old data.

## Next steps (https://www.tinybird.co/docs/forward/guides/lambda-architecture#next-steps)

- Read more about [Copy Pipes](/docs/forward/core-concepts/copy-pipes).
- Read more about [Materialized Views](/docs/forward/core-concepts/materialized-views).

---

URL: https://www.tinybird.co/docs/forward/guides/integrate-with-open-source
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Add Tinybird to your Open Source Project \| Tinybird"
theme-color: "#171612"
description: "Tips and best practices for adding Tinybird to your open source project and provide your users flexible deployment options."
inkeep:version: "forward"
---

# Add Tinybird to your open source project (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#add-tinybird-to-your-open-source-project)

Tinybird is a managed ClickHouse® service with flexible deployment options used by many open source developers to integrate ClickHouse®-based analytics features into their open source projects.

This guide walks you through best practices for adding Tinybird into your open source project.

[Apply to the Tinybird Open Source Program](https://www.tinybird.co/open-source-program) to get plan credits, swag, and support integrating Tinybird into your open source project.

## Including Tinybird in your open source repository (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#including-tinybird-in-your-open-source-repository)

Tinybird resources are represented as plaintext files that can be versioned alongside your app code. When you initialize a Tinybird project with `tb init`, it creates a default directory structure with folders for different resource types.

### Default project structure (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#default-project-structure)

Tinybird generates the following folder structure:

```
├── .tinyb                    # Project configuration and authentication
├── connections/              # Connection files
├── datasources/              # Data source definitions (.datasource)
│   └── user_actions.datasource
├── endpoints/                # API endpoint pipes (.pipe)
│   ├── daily_metrics.pipe
│   └── user_stats.pipe
├── pipes/                    # Processing pipes (.pipe)
├── materializations/         # Materialized views
├── fixtures/                 # Test data files
│   └── sample_data.ndjson
│   └── sample_data.sql
└── tests/                    # Test suites
    └── daily_metrics.yaml
```
You can include these Tinybird resources as a part of your open source distribution so that self-hosters and contributors can deploy the Tinybird project using their preferred deployment model.

Refer to the [reference implementations](https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#reference-implementations) for example approaches to organizing Tinybird resources within your open source codebase.

### Documenting Tinybird (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#documenting-tinybird)

Include a `TINYBIRD.md` or a `README.md` within your `/tinybird` folder to explain Tinybird resources to contributors. The `tb init` command can generate starter project files and a structure that you can build on.

## Development workflow for contributors and self-hosters (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#development-workflow-for-contributors-and-self-hosters)

Tinybird projects follow standard Git-based development workflows. Contributors can pull repositories, run [Tinybird Local](/docs/forward/core-concepts/tinybird-local) for development, and use agent skills in their coding assistants to understand and evolve Tinybird resources.

```
npx skills add tinybirdco/tinybird-agent-skills
```
As with any software codebase, the typical workflow is: clone → install → build → develop → test → deploy.

### CI/CD and testing (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#cicd-and-testing)

Tinybird automatically generates CI/CD templates for GitHub and GitLab to automate testing and deployment.

You can also use Tinybird's testing framework to generate unit tests and data quality tests to detect breaking changes during CI/CD.

Learn more:

- [CI/CD workflows](/docs/forward/development-workflow/cicd)
- [Test your project](/docs/forward/guides/test-your-project)

## Select your query interface (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#select-your-query-interface)

Developers often choose Tinybird because of the simplified abstractions it provides over ClickHouse®, including its concept of pipes that can be instantly deployed as API endpoints.

Endpoint pipes are the most common way to query data in Tinybird, however Tinybird offers three different query interfaces, each with different trade-offs for open source projects.

### API Endpoints (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#api-endpoints)

Deploy pipes as hosted API endpoints with type-safe parameters and built-in monitoring. This provides the highest level of abstraction and separation of concerns, as query logic - including filter parameters - is maintained entirely within Tinybird pipe files.

An example endpoint pipe configuration:

```
TOKEN analytics_read READ

DESCRIPTION >
    User activity metrics for the last 30 days

NODE user_activity
SQL >
    %
    SELECT
        user_id,
        event_type,
        count() AS event_count,
        max(timestamp) AS last_activity
    FROM user_events
    WHERE user_id = {{UInt32(user_id)}}
      AND timestamp >= {{DateTime(start_date, '2024-01-01 00:00:00')}}
    GROUP BY user_id, event_type
    ORDER BY event_count DESC

TYPE ENDPOINT
```

```
// Calling the endpoint
const response = await fetch(
  `https://api.tinybird.co/v0/pipes/user_activity.json?user_id=123&start_date=2024-01-01`,
  {
    headers: {
      'Authorization': 'Bearer tb_your_read_token'
    }
  }
);
const data = await response.json();
```
Learn more:

- [Endpoints](/docs/forward/core-concepts/api-endpoints)
- [Pipes](/docs/forward/core-concepts/pipes)

### Query API (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#query-api)

Use the Query API to execute arbitrary SQL queries directly against your Tinybird Data Sources and Pipes. With this approach, queries are defined directly within your app code and passed to the Tinybird API. This provides some portability for open source projects that want to give self-hosters the option to port the analytics deployment to open source ClickHouse®, but it still relies on Tinybird APIs.

```
// Direct SQL query
const queryData = {
  q: `
    SELECT
      event_type,
      count() as total_events,
      uniq(user_id) as unique_users
    FROM user_events
    WHERE timestamp >= today() - INTERVAL 7 DAY
    GROUP BY event_type
    ORDER BY total_events DESC
    FORMAT JSON
  `
};

const response = await fetch('https://api.tinybird.co/v0/sql', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer tb_your_token',
    'Content-Type': 'application/x-www-form-urlencoded'
  },
  body: new URLSearchParams(queryData)
});
```
Learn more: [Query API](/docs/api-reference/query-api)

### ClickHouse® HTTP interface (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#clickhouse-http-interface)

You can connect to the underlying ClickHouse® databases in your Tinybird Workspace using the native ClickHouse® HTTP interface. Query Tinybird with existing ORMs, language clients, and BI tools. This approach provides maximum portability and perhaps the easiest migration path to open source ClickHouse®, though it eliminates the convenient query abstractions that Tinybird provides.

```
Protocol: HTTPS
Host: clickhouse.<region>.tinybird.co  # (adjust for your region)
Port: 443
Username: <optional - workspace_name>
Password: <your_read_token>
```

```
// Using @clickhouse/client (HTTP interface)
import { createClient } from '@clickhouse/client'

const client = createClient({
  host: 'https://clickhouse.<region>.tinybird.co',
  username: 'optional_workspace_name', // optional
  password: 'tb_your_read_token'
})

const resultSet = await client.query({
  query: 'SELECT * FROM user_events LIMIT 10',
  format: 'JSON'
})

const data = await resultSet.json()
console.log(data)
```

### Choosing the right query interface (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#choosing-the-right-query-interface)

Each query interface offers different tradeoffs: Endpoints provide standardization but may be less portable for those who want to self-host ClickHouse®. The Query API offers a middle ground with familiar SQL syntax defined in the app code while still using Tinybird APIs, and the ClickHouse® HTTP Interface maximizes portability through a raw connection to the underlying databases.

Many projects use a combination based on their specific needs - for example, using ClickHouse® Interface for BI tools, Query API for ad hoc queries, and Endpoints for common query patterns in user-facing analytics.

**Ingestion APIs**: All data ingestion into Tinybird uses Tinybird-specific APIs (Events API, Data Sources API, Connectors). If you plan to support ClickHouse® open source as an alternative, your project must implement separate ingestion patterns for pure ClickHouse® deployments.

## Choose your deployment model (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#choose-your-deployment-model)

Open source projects integrating Tinybird have several deployment options. The most common pattern for open source projects:

- Use **Tinybird Cloud** to power analytics for the hosted or commercial service.
- Offer **self-hosting users** the flexibility to choose between Tinybird Cloud, [Tinybird Self-Managed Regions](/docs/forward/core-concepts/self-managed), or, perhaps, open source ClickHouse®.
Learn more: [Self-managed](/docs/forward/core-concepts/self-managed)

### Considerations for open source ClickHouse® deployments (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#considerations-for-open-source-clickhouse-deployments)

If your [query interface](https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#select-your-query-interface) choice supports it, self-hosting users can potentially deploy your project with open source ClickHouse® as their analytics database.

Note, however, there are some requirements you must consider:

- **Alternative ingestion patterns** - ClickHouse®-native ingestion instead of Tinybird APIs
- **Custom deployment logic** - Handling ClickHouse® cluster setup, scaling, and maintenance
- **Limited features** - No access to Tinybird's pipes, endpoints, or management tools
- **Additional complexity** - Users manage ClickHouse® operations, security, and monitoring
- **Zero-copy support** - Open source ClickHouse® currently doesn't support zero-copy replication, which can impact storage cost and performance for self-hosters.
Some open source projects might offer flexibility to self-hosters. Tinybird's ClickHouse® HTTP interface can help you design this flexibility into your open source project with these considerations in mind.

Read the [guide on migrating from ClickHouse® to Tinybird](/docs/forward/guides/migrate-from-clickhouse) to better understand the mapping of concepts between Tinybird and ClickHouse®.

## Reference implementations (https://www.tinybird.co/docs/forward/guides/integrate-with-open-source#reference-implementations)

The following open source projects demonstrate different approaches to integrating Tinybird and can be used as reference implementations:

- [Ghost](https://github.com/TryGhost/Ghost): a popular open source publishing platform that uses Tinybird for content and audience analytics.
- [Dub](https://github.com/dubinc/dub): an open source link attribution platform that uses Tinybird to capture link click events and build real-time analytics for link creators.
- [OpenStatus](https://github.com/openstatusHQ/openstatus): an open source synthetic monitoring platform that uses Tinybird to store and analyze uptime monitoring data.
- [Inbox Zero](https://github.com/elie222/inbox-zero): an open source email management tool that uses Tinybird to analyze email patterns and monitor LLM calls.
- [Papermark](https://github.com/mfts/papermark): an open source document sharing platform that uses Tinybird to track document views, engagement metrics, and sharing analytics.

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-with-estuary
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingest with Estuary \| Tinybird"
theme-color: "#171612"
description: "In this guide, you'll learn how to use Estuary to push data streams to Tinybird."
inkeep:version: "forward"
---

# Ingest with Estuary (https://www.tinybird.co/docs/forward/guides/ingest-with-estuary#ingest-with-estuary)

Use Estuary to push data streams to Tinybird.

[Estuary](https://estuary.dev/) is a real-time ETL tool that allows you capture data from a range of source, and push it to a range of destinations. Using Estuary's Dekaf, you can connect Tinybird to Estuary as if it was a Kafka broker - meaning you can use Tinybird's native Kafka Connector to consume data from Estuary.

[Read more about Estuary Dekaf.](https://docs.estuary.dev/reference/Connectors/materialization-connectors/Dekaf/)

## Prerequisites (https://www.tinybird.co/docs/forward/guides/ingest-with-estuary#prerequisites)

- An Estuary account and collection
- A Tinybird account and workspace

## Connect to Estuary (https://www.tinybird.co/docs/forward/guides/ingest-with-estuary#connect-to-estuary)

In Estuary, create a new Dekaf materialization to use for the Tinybird connection.

You can create it from the [Estuary destinations tab](https://dashboard.estuary.dev/materializations/create). You have all the details on the [Tinybird Dekaf Estuary docs page.](https://docs.estuary.dev/reference/Connectors/materialization-connectors/Dekaf/tinybird/)

In your Tinybird workspace, create a new data source and use the [Kafka Connector](/docs/forward/ingest-data/connectors/kafka).

To configure the connection details, use the following settings (these can also be found in the [Estuary Dekaf docs](https://docs.estuary.dev/guides/dekaf_reading_collections_from_kafka/#connection-details)).

- Bootstrap servers: `dekaf.estuary-data.com`
- SASL Mechanism: `PLAIN`
- SASL Username: Your materialization task name, such as `YOUR-ORG/YOUR-PREFIX/YOUR-MATERIALIZATION`
- SASL Password: Auth token provided when you the Dekaf materialization was created on Estuary
Tick the `Decode Avro messages with Schema Register` box, and use the following settings:

- URL: `https://dekaf.estuary-data.com`
- Username: The same Materialization name from the preceding step, `YOUR-ORG/YOUR-PREFIX/YOUR-MATERIALIZATION`
- Password: The same Auth token created on the Dekaf materialization from the preceding step
Select **Next** and you see a list of topics. These topics are the collections you have in Estuary. Select the collection you want to ingest into Tinybird, and select **Next**.

Configure your consumer group as needed.

Finally, you see a preview of the data source schema. Feel free to make any modifications as required, then select **Create data source**.

This completes the connection with Estuary, and new data from the Estuary collection arrives in your Tinybird data source in real-time.

If you need *support for deletions*, check the [configuring support for deletions](https://docs.estuary.dev/reference/Connectors/materialization-connectors/Dekaf/tinybird) section on Estuary docs.

## Handling updates and deletes (https://www.tinybird.co/docs/forward/guides/ingest-with-estuary#handling-updates-and-deletes)

When capturing change data that includes updates and deletes, you need to deduplicate the data in Tinybird to maintain the latest state.

There are several strategies to deduplicate data in your data source, but with Estuary, the recommended approach is to use a [ReplacingMergeTree engine](/docs/sql-reference/engines/replacingmergetree) with appropriate settings and the `FINAL` modifier.

Do not build materialized views with an AggregatingMergeTree on top of a ReplacingMergeTree. The target data source always contains duplicates due to the incremental nature of materialized views.

## Learn more (https://www.tinybird.co/docs/forward/guides/ingest-with-estuary#learn-more)

For a complete step-by-step tutorial on setting up CDC with PostgreSQL, Estuary Flow, and Tinybird, see the [From CDC to real-time analytics with Tinybird and Estuary](https://www.tinybird.co/blog-posts/cdc-real-time-analytics-estuary-dekaf) blog post.

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-vercel-logdrains
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Send Vercel log drains to Tinybird \| Tinybird"
theme-color: "#171612"
description: "Learn how to send Vercel events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---

# Send Vercel log drains to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-vercel-logdrains#send-vercel-log-drains-to-tinybird)

[Vercel](https://vercel.com/) is a platform for building and deploying web applications. By integrating Vercel with Tinybird, you can analyze your Vercel events in real time and enrich it with other data sources.

Some common use cases for sending Vercel Log Drains to Tinybird include:

1. Analyze logs from your applications.
2. Monitor logs from your applications.
3. Create custom analytical dashboards.
4. Build an alert system based on logging patterns.
Read on to learn how to send logs from Vercel to Tinybird.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-vercel-logdrains#before-you-start)

Before you connect Vercel Log Drains to Tinybird, ensure:

- You have a Vercel account.
- You have a Tinybird workspace.

## Connect Vercel to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-vercel-logdrains#connect-vercel-to-tinybird)

1. Choose your team scope on the dashboard, and go to **Team Settings** > **Log Drains**.
2. Select the **Projects** to send logs to Tinybird.
3. Select **Sources** you want to send logs to Tinybird.
4. Select **NDJSON** as Delivery Format.
5. Select **Environments** and **Sampling Rate**.
6. In your Tinybird project, create a data source called `vercel_logs`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/vercel_logs.datasource):

```
SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.type` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
The proxy column is a JSON string. Use the [JSONExtract](/docs/sql-reference/functions/json-functions) functions to extract the data you need in your pipes.

1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in Vercel, paste the Events API URL in your Log Drains Endpoint. Use the query parameter `name` to match the name of the data source you created in Tinybird.
Log Drains webhook needs to be verified by Vercel. You can do this by adding the `x-vercel-verify` parameter to the request.

```
https://<your_host>/v0/events?name=vercel_logs&x-vercel-verify=<your-x-vercel-verify-token>
```

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

1. Select **Custom Headers**, add `Authorization` with the value `Bearer <your-tinybird-token>` and select **Add**.
2. Select **Verify** and optionally use **Test Log Drain** from Vercel to check data gets to the `vercel_logs` data source in Tinybird.
3. You're done. Any of the Vercel Log Drains you selected is automatically sent to Tinybird through the [Events API](/docs/forward/ingest-data/events-api).
Check the status of the from the **Log** tab in the Tinybird `vercel_logs` data source.

## Vercel Logs Explorer template (https://www.tinybird.co/docs/forward/guides/ingest-vercel-logdrains#vercel-logs-explorer-template)

Use the [Vercel Logs Explorer Template](https://www.tinybird.co/templates/vercel-log-drains) to bootstrap a multi-tenant, user-facing logs explorer for your Vercel account. You can fork it and make it your own.

## See also (https://www.tinybird.co/docs/forward/guides/ingest-vercel-logdrains#see-also)

- [Events API](/docs/forward/ingest-data/events-api)
- [Vercel Log Drains](https://vercel.com/docs/log-drains/log-drains-reference)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-vercel-ai-sdk
Last update: 2026-07-24T12:51:58.000Z
Content:
---
title: "Instrument LLM Calls With Vercel AI SDK \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to instrument LLM calls with the Vercel AI SDK and Tinybird."
inkeep:version: "forward"
---

# Instrument LLM calls from Vercel AI SDK (https://www.tinybird.co/docs/forward/guides/ingest-vercel-ai-sdk#instrument-llm-calls-from-vercel-ai-sdk)

[Vercel AI SDK](https://sdk.vercel.ai/) is a powerful tool for building AI applications. It's a popular choice for many developers and organizations.

To start instrumenting LLM calls with the Vercel AI SDK and Tinybird, first create a data source with this schema:

```
SCHEMA >
    `model` LowCardinality(String) `json:$.model` DEFAULT 'unknown',
    `messages` Array(Map(String, String)) `json:$.messages[:]` DEFAULT [],
    `user` String `json:$.user` DEFAULT 'unknown',
    `start_time` DateTime `json:$.start_time` DEFAULT now(),
    `end_time` DateTime `json:$.end_time` DEFAULT now(),
    `id` String `json:$.id` DEFAULT '',
    `stream` Bool `json:$.stream` DEFAULT false,
    `call_type` LowCardinality(String) `json:$.call_type` DEFAULT 'unknown',
    `provider` LowCardinality(String) `json:$.provider` DEFAULT 'unknown',
    `api_key` String `json:$.api_key` DEFAULT '',
    `log_event_type` LowCardinality(String) `json:$.log_event_type` DEFAULT 'unknown',
    `llm_api_duration_ms` Float32 `json:$.llm_api_duration_ms` DEFAULT 0,
    `cache_hit` Bool `json:$.cache_hit` DEFAULT false,
    `response_status` LowCardinality(String) `json:$.standard_logging_object_status` DEFAULT 'unknown',
    `response_time` Float32 `json:$.standard_logging_object_response_time` DEFAULT 0,
    `proxy_metadata` String `json:$.proxy_metadata` DEFAULT '',
    `organization` String `json:$.proxy_metadata.organization` DEFAULT '',
    `environment` String `json:$.proxy_metadata.environment` DEFAULT '',
    `project` String `json:$.proxy_metadata.project` DEFAULT '',
    `chat_id` String `json:$.proxy_metadata.chat_id` DEFAULT '',
    `response` String `json:$.response` DEFAULT '',
    `response_id` String `json:$.response.id`,
    `response_object` String `json:$.response.object` DEFAULT 'unknown',
    `response_choices` Array(String) `json:$.response.choices[:]` DEFAULT [],
    `completion_tokens` UInt16 `json:$.response.usage.completion_tokens` DEFAULT 0,
    `prompt_tokens` UInt16 `json:$.response.usage.prompt_tokens` DEFAULT 0,
    `total_tokens` UInt16 `json:$.response.usage.total_tokens` DEFAULT 0,
    `cost` Float32 `json:$.cost` DEFAULT 0,
    `exception` String `json:$.exception` DEFAULT '',
    `traceback` String `json:$.traceback` DEFAULT '',
    `duration` Float32 `json:$.duration` DEFAULT 0

ENGINE MergeTree
ENGINE_SORTING_KEY start_time, organization, project, model
ENGINE_PARTITION_KEY toYYYYMM(start_time)
```
Use a wrapper around the LLM provider you use, this is an example using OpenAI:

```
const openai = createOpenAI({ apiKey: apiKey });
const wrappedOpenAI = wrapModelWithTinybird(
    openai('gpt-3.5-turbo'),
    process.env.NEXT_PUBLIC_TINYBIRD_API_URL!,
    process.env.TINYBIRD_TOKEN!,
    {
    event: 'search_filter',
    environment: process.env.NODE_ENV,
    project: 'ai-analytics',
    organization: 'your-org',
    }
);
```
Implement the wrapper in your app:

```
import type { LanguageModelV1 } from '@ai-sdk/provider';

type TinybirdConfig = {
  event?: string;
  organization?: string;
  project?: string;
  environment?: string;
  user?: string;
  chatId?: string;
};

export function wrapModelWithTinybird(
  model: LanguageModelV1,
  tinybirdHost: string,
  tinybirdToken: string,
  config: TinybirdConfig = {}
) {
  const originalDoGenerate = model.doGenerate;
  const originalDoStream = model.doStream;

  const logToTinybird = async (
    messageId: string,
    startTime: Date,
    status: 'success' | 'error',
    // eslint-disable-next-line @typescript-eslint/no-explicit-any
    args: any[],
    result?: { text?: string; usage?: { promptTokens?: number; completionTokens?: number } },
    error?: Error
  ) => {
    const endTime = new Date();
    const duration = endTime.getTime() - startTime.getTime();

    const event = {
      start_time: startTime.toISOString(),
      end_time: endTime.toISOString(),
      message_id: messageId,
      model: model.modelId || 'unknown',
      provider: 'openai',
      duration,
      llm_api_duration_ms: duration,
      response: status === 'success' ? {
        id: messageId,
        object: 'chat.completion',
        usage: {
          prompt_tokens: result?.usage?.promptTokens || 0,
          completion_tokens: result?.usage?.completionTokens || 0,
          total_tokens: (result?.usage?.promptTokens || 0) + (result?.usage?.completionTokens || 0),
        },
        choices: [{ message: { content: result?.text ?? '' } }],
      } : undefined,
      messages: args[0]?.prompt ? [{ role: 'user', content: args[0].prompt }].map(m => ({
        role: String(m.role),
        content: String(m.content)
      })) : [],
      proxy_metadata: {
        organization: config.organization || '',
        project: config.project || '',
        environment: config.environment || '',
        chat_id: config.chatId || '',
      },
      user: config.user || 'unknown',
      standard_logging_object_status: status,
      standard_logging_object_response_time: duration,
      log_event_type: config.event || 'chat_completion',
      id: messageId,
      call_type: 'completion',
      cache_hit: false,
      ...(status === 'error' && {
        exception: error?.message || 'Unknown error',
        traceback: error?.stack || '',
      }),
    };

    // Send to Tinybird
    fetch(`${tinybirdHost}/v0/events?name=llm_events`, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Authorization': `Bearer ${tinybirdToken}`,
      },
      body: JSON.stringify(event),
    }).catch(console.error);
  };

  model.doGenerate = async function (...args) {
    const startTime = new Date();
    const messageId = crypto.randomUUID();

    try {
      const result = await originalDoGenerate.apply(this, args);
      await logToTinybird(messageId, startTime, 'success', args, result);
      return result;
    } catch (error) {
      await logToTinybird(messageId, startTime, 'error', args, undefined, error as Error);
      throw error;
    }
  };

  model.doStream = async function (...args) {
    const startTime = new Date();
    const messageId = crypto.randomUUID();

    try {
      const result = await originalDoStream.apply(this, args);
      await logToTinybird(messageId, startTime, 'success', args, { text: '', usage: { promptTokens: 0, completionTokens: 0 } });
      return result;
    } catch (error) {
      await logToTinybird(messageId, startTime, 'error', args, undefined, error as Error);
      throw error;
    }
  };

  return model;
}
```

## AI analytics template (https://www.tinybird.co/docs/forward/guides/ingest-vercel-ai-sdk#ai-analytics-template)

Use the [LLM tracker template](https://github.com/tinybirdco/llm-performance-tracker) to bootstrap a multi-tenant, user-facing AI analytics dashboard and LLM cost calculator for your AI models. You can fork it and make it your own.

## See also (https://www.tinybird.co/docs/forward/guides/ingest-vercel-ai-sdk#see-also)

- [Vercel AI SDK docs](https://sdk.vercel.ai/)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-litellm
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Send LiteLLM Events to Tinybird \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to send LiteLLM events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---

# Send LiteLLM events to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-litellm#send-litellm-events-to-tinybird)

[LiteLLM](https://www.litellm.ai/) is an LLM gateway that provides AI models access, fallbacks, and spend tracking across 100+ LLMs. It's a popular choice for many developers and organizations.

LiteLLM is open source and can be self-hosted.

To start sending LiteLLM events to Tinybird, first create a data source with this schema:

```
SCHEMA >
    `model` LowCardinality(String) `json:$.model` DEFAULT 'unknown',
    `messages` Array(Map(String, String)) `json:$.messages[:]` DEFAULT [],
    `user` String `json:$.user` DEFAULT 'unknown',
    `start_time` DateTime `json:$.start_time` DEFAULT now(),
    `end_time` DateTime `json:$.end_time` DEFAULT now(),
    `id` String `json:$.id` DEFAULT '',
    `stream` Bool `json:$.stream` DEFAULT false,
    `call_type` LowCardinality(String) `json:$.call_type` DEFAULT 'unknown',
    `provider` LowCardinality(String) `json:$.provider` DEFAULT 'unknown',
    `api_key` String `json:$.api_key` DEFAULT '',
    `log_event_type` LowCardinality(String) `json:$.log_event_type` DEFAULT 'unknown',
    `llm_api_duration_ms` Float32 `json:$.llm_api_duration_ms` DEFAULT 0,
    `cache_hit` Bool `json:$.cache_hit` DEFAULT false,
    `response_status` LowCardinality(String) `json:$.standard_logging_object_status` DEFAULT 'unknown',
    `response_time` Float32 `json:$.standard_logging_object_response_time` DEFAULT 0,
    `proxy_metadata` String `json:$.proxy_metadata` DEFAULT '',
    `organization` String `json:$.proxy_metadata.organization` DEFAULT '',
    `environment` String `json:$.proxy_metadata.environment` DEFAULT '',
    `project` String `json:$.proxy_metadata.project` DEFAULT '',
    `chat_id` String `json:$.proxy_metadata.chat_id` DEFAULT '',
    `response` String `json:$.response` DEFAULT '',
    `response_id` String `json:$.response.id`,
    `response_object` String `json:$.response.object` DEFAULT 'unknown',
    `response_choices` Array(String) `json:$.response.choices[:]` DEFAULT [],
    `completion_tokens` UInt16 `json:$.response.usage.completion_tokens` DEFAULT 0,
    `prompt_tokens` UInt16 `json:$.response.usage.prompt_tokens` DEFAULT 0,
    `total_tokens` UInt16 `json:$.response.usage.total_tokens` DEFAULT 0,
    `cost` Float32 `json:$.cost` DEFAULT 0,
    `exception` String `json:$.exception` DEFAULT '',
    `traceback` String `json:$.traceback` DEFAULT '',
    `duration` Float32 `json:$.duration` DEFAULT 0

ENGINE MergeTree
ENGINE_SORTING_KEY start_time, organization, project, model
ENGINE_PARTITION_KEY toYYYYMM(start_time)
```
Install the Tinybird AI Python SDK:

```
pip install tinybird-python-sdk[ai]
```
Finally, use the following handler in your app:

```
import litellm
from litellm import acompletion
from tb.litellm.handler import TinybirdLitellmAsyncHandler

customHandler = TinybirdLitellmAsyncHandler(
    api_url="https://api.us-east.aws.tinybird.co",
    tinybird_token=os.getenv("TINYBIRD_TOKEN"),
    datasource_name="litellm"
)

litellm.callbacks = [customHandler]

response = await acompletion(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}],
    stream=True
)
```

## AI analytics template (https://www.tinybird.co/docs/forward/guides/ingest-litellm#ai-analytics-template)

Use the [LLM tracker template](https://github.com/tinybirdco/llm-performance-tracker) to bootstrap a multi-tenant, user-facing AI analytics dashboard and LLM cost calculator for your AI models. You can fork it and make it your own.

## See also (https://www.tinybird.co/docs/forward/guides/ingest-litellm#see-also)

- [Events API](/docs/forward/ingest-data/events-api)
- [LiteLLM docs](https://docs.litellm.ai/)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-vercel
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send Vercel Webhooks to Tinybird \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to send Vercel events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---

# Send Vercel events to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-vercel#send-vercel-events-to-tinybird)

[Vercel](https://vercel.com/) is a platform for building and deploying web applications. By integrating Vercel with Tinybird, you can analyze your Vercel events in real time and enrich it with other data sources.

Some common use cases for sending Vercel events to Tinybird include:

1. Tracking deployments, projects, integrations and domains status and errors.
2. Creating custom analytical dashboards.
3. Monitoring attacks.
Read on to learn how to send data from Vercel to Tinybird.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-vercel#before-you-start)

Before you connect Vercel webhooks to Tinybird, ensure:

- You have a Vercel account.
- You have a Tinybird workspace.

## Connect Vercel to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-vercel#connect-vercel-to-tinybird)

1. Choose your team scope on the dashboard, and go to **Settings** > **Webhooks**.
2. Select the Webhooks and Projects you want to send to Tinybird.
3. In Tinybird, create a data source, called `vercel`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/vercel.datasource):

```
SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.type` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Vercel webhooks in a single column. You can later retrieve various events and their metadata as needed in your pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).

1. In Tinybird, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in Vercel, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name` to match the name of the data source you created in Tinybird. For example:

```
https://<your_host>/v0/events?name=vercel&token=<your user token>
```

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

1. You're done. Any of the Vercel events you selected is automatically sent to Tinybird through the [Events API](/docs/forward/ingest-data/events-api).
You can check the status of the integration from the **Log** tab in the Tinybird `vercel` data source.

## See also (https://www.tinybird.co/docs/forward/guides/ingest-from-vercel#see-also)

- [Events API](/docs/forward/ingest-data/events-api)
- [Vercel webhooks](https://vercel.com/docs/observability/webhooks-overview)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-vector
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingest data using Vector.dev \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to use the Events API as a Vector sink."
inkeep:version: "forward"
---

# Ingest data using Vector.dev (https://www.tinybird.co/docs/forward/guides/ingest-from-vector#ingest-data-using-vector-dev)

[Vector.dev](https://vector.dev/) is an open source tool created by DataDog for collecting, transforming, and shipping logs, metrics, and traces.

Some common use cases for using Vector.dev as a Tinybird sink include:

1. Ingesting data from a number of [Vector.dev sources](https://vector.dev/components/) to Tinybird.
2. Enriching other data sources with real-time Vector metrics.
3. Aggregate logs and metrics from Vector.dev to Tinybird.
4. Transform and redact sensitive data before ingesting it into Tinybird.
Read on to learn how to use the Events API as a Vector.dev sink.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-vector#before-you-start)

Before you connect Vector.dev to Tinybird, ensure:

- You have installed Vector.dev.
- You have a Tinybird workspace.

## Use the Events API as a Vector.dev sink (https://www.tinybird.co/docs/forward/guides/ingest-from-vector#use-the-events-api-as-a-vector-dev-sink)

To push events to Tinybird from Vector.dev, you need to configure Vector to use the Events API as a sink.

You can use the following example Vector configuration to push events, in this case Docker logs, to Tinybird:

```
sources:
  docker_logs:
    type: "docker_logs"

transforms:
  remap_docker_logs:
    inputs:
      - "docker_logs"
    type: "remap"
    source: |
      . = parse_json!(.log)

sinks:
  push_docker_logs_to_tinybird:
    inputs:
      - "remap_docker_logs"
    type: "http"
    uri: "$TINYBIRD_HOST/v0/events?name=docker"
    auth:
      strategy: "bearer"
      token: "$TINYBIRD_TOKEN"
    encoding:
      codec: "json"
    framing:
        method: "newline_delimited"
```
The previous snippet uses the `docker_logs` source to collect Docker logs, and the `remap_docker_logs` transform to parse the logs as JSON.

The `push_docker_logs_to_tinybird` sink uses the Events API to push the transformed logs to Tinybird in NDJSON (`newline_delimited`) format.

You can customize the `$TINYBIRD_HOST` and `$TINYBIRD_TOKEN` environment variables to use your Tinybird workspace.

Learn more about other sources you can use to ingest data into Tinybird in the [Vector.dev documentation](https://vector.dev/docs/reference/configuration/sources/).

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-stripe
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Send Stripe Events to Tinybird \| Tinybird"
theme-color: "#171612"
description: "Learn how to send Stripe events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---

# Send Stripe events to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-stripe#send-stripe-events-to-tinybird)

[Stripe](https://stripe.com/) is a platform for payments and financial services, and it provides a way to send events to Tinybird using webhooks.

Some common use cases for sending Stripe events to Tinybird include:

1. Monitor Stripe events.
2. Run analytical workflows based on Stripe events.
3. Create custom dashboards based on Stripe events.
4. Create alerts and notifications based on Stripe events.
5. Join Stripe events with other data sources to enrich your user data.
Read on to learn how to send events from Stripe to Tinybird.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-stripe#before-you-start)

Before you connect Stripe to Tinybird, ensure:

- You have a Stripe account.
- You have a Tinybird workspace.

## Connect Stripe to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-stripe#connect-stripe-to-tinybird)

Stripe provides a variety of [webhook event types](https://docs.stripe.com/api/events/object) that you can use to send events to Tinybird.

This guide covers the base case for sending Stripe events to Tinybird.

1. In Stripe, go to [Webhooks](https://dashboard.stripe.com/webhooks)
2. Select **Add endpoint**.
3. In your Tinybird project, create a data source called `stripe`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/stripe.datasource):

```
SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.type` DEFAULT 'unknown',
  `event` String `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Store the JSON-formatted data you receive from Stripe as a `String` in a single column. You can later parse the JSON string to retrieve various events and their metadata as needed in your Pipes.

1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in Stripe, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name` to match the name of the data source you created in Tinybird.

```
https://<your_host>/v0/events?name=stripe&format=json&token=<your user token>
```
Make sure to use the `format=json` query parameter.

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

1. Select **Select events** and select the events you want to send to Tinybird.
2. Save and you're done.
Check the status of the integration by selecting the webhook in Stripe or from the **Log** tab in the Tinybird `stripe` data source.

## See also (https://www.tinybird.co/docs/forward/guides/ingest-from-stripe#see-also)

- [Stripe Webhooks](https://docs.stripe.com/webhooks)
- [Stripe Events](https://docs.stripe.com/api/events/object)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-incremental-updates
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingest Snowflake Data Incrementally \| Tinybird"
theme-color: "#171612"
description: "Learn how to ingest data from Snowflake doing incremental appends, so you can keep last transactional data sources in sync with Tinybird."
inkeep:version: "forward"
---

# Ingest from Snowflake using incremental updates (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-incremental-updates#ingest-from-snowflake-using-incremental-updates)

Read on to learn how to incrementally load data from a Snowflake table into Tinybird, using Amazon S3 as an intermediary staging area.

An incremental loading strategy ensures that only new or updated rows are transferred.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-incremental-updates#before-you-start)

Before you begin, ensure the following:

- Snowflake Account: A Snowflake instance with the data you want to load.
- Amazon S3 Bucket: Access to an S3 bucket for staging data, along with appropriate permissions (write from Snowflake and read from Tinybird).
- Tinybird Account: A Tinybird workspace with an appropriate data source set up.
- Snowflake Permissions: Ensure the Snowflake user has privileges to:
  - Access the target table.
  - Create and manage stages.
  - Unload data into S3.
- AWS Credentials: Ensure Snowflake can use AWS credentials (IAM Role or Access Key/Secret Key pair) to write to the S3 bucket.
1
## Create the unload task in Snowflake (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-incremental-updates#create-the-unload-task-in-snowflake)

Follow these steps to create the unload task in Snowflake:

1. Grant the required permissions in AWS IAM Console
Make sure the S3 bucket allows Snowflake to write files by setting up an appropriate IAM role or policy. You can use this template to create the policy and attach it to the AWS role:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["s3:PutObject", "s3:AbortMultipartUpload"],
            "Resource": "arn:aws:s3:::your-bucket-name/path/*"
        }
    ]
}
```
Replace `your-bucket-name/path/*` with your bucket name, and optionally the path you want to grant access to.

1. Create the storage integration
Run the following SQL statement to create the storage integration:

```
/* Create the S3 integration.
 */
CREATE or replace STORAGE INTEGRATION tinybird_integration
  TYPE = EXTERNAL_STAGE
  STORAGE_PROVIDER = 'S3'
  ENABLED = TRUE
  STORAGE_AWS_ROLE_ARN = '<arn_role>'
  STORAGE_ALLOWED_LOCATIONS = ('*');

-- describe integration tinybird_integration;
```
Replace `<arn_role>` with the ARN of the role created in the previous step.

1. Create the file format
Run the following SQL statement to create the file format:

```
/* Create the file format for the output files generated.
 */
CREATE OR REPLACE FILE FORMAT csv_unload_format
  TYPE = 'CSV';
```

1. Create the stage
Run the following SQL statement to create the stage:

```
/* And finally the stage we'll use to unload the data to.
 */
CREATE or replace STAGE tinybird_stage
  STORAGE_INTEGRATION = tinybird_integration
  URL = 's3://your-bucket-name/path/'
  FILE_FORMAT = csv_unload_format;
```
Replace `your-bucket-name` and `path` with your S3 bucket details.

2
## Create the unload task (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-incremental-updates#create-the-unload-task)

Run the following SQL statement to create the scheduled task that unloads the new records since the last successful execution to the S3 bucket:

```
/* Create the scheduled task that unloads the new records since
 * last successful execution to the S3 bucket.
 *
 * Note how it reads the timestamp of the last successful execution,
 * and leaves a one hour margin.
 *
 * Orders need to be deduplicated later in Tinybird.
 */
CREATE or replace TASK export_order_deltas
    WAREHOUSE = compute_wh
    SCHEDULE = 'USING CRON 05 * * * * UTC'
AS
BEGIN
   LET sql := 'COPY INTO @tinybird_stage/orders/orders_<ts> from (
    select
        O_ORDERKEY, O_CUSTKEY, O_ORDERSTATUS, O_TOTALPRICE, O_ORDERDATE,
        O_ORDERPRIORITY, O_CLERK
    from tinybird.samples.orders_incremental
    where o_orderdate >= (
        SELECT coalesce(timestampadd(hour,-1,max(QUERY_START_TIME)),\'1970-01-01\')
        FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(TASK_NAME=>\'export_order_deltas\'))
        where state = \'SUCCEEDED\'
        ORDER BY SCHEDULED_TIME
    )) max_file_size=1000000000';

   sql := REPLACE(sql, '<ts>', TO_VARCHAR(CONVERT_TIMEZONE('UTC',current_timestamp()), 'YYYY_MM_DD__hh24_mi_ss'));

   EXECUTE IMMEDIATE (sql);

   RETURN sql;
END;
```
3
## Configure the ingestion in Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-incremental-updates#configure-the-ingestion-in-tinybird)

Create the S3 connection. See [S3 Connector](/docs/forward/ingest-data/connectors/s3).

You can use the following schema for the data source:

tinybird/datasources/s3_landing_ds.datasource - data source with S3 connection
```
SCHEMA >
    `O_ORDERKEY` Int64,
    `O_CUSTKEY` Int64,
    `O_ORDERSTATUS` String,
    `O_TOTALPRICE` Float32,
    `O_ORDERDATE` DateTime64(3),
    `O_ORDERPRIORITY` String,
    `O_CLERK` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYear(O_ORDERDATE)"
ENGINE_SORTING_KEY "O_ORDERDATE, O_ORDERPRIORITY, O_CLERK"

IMPORT_CONNECTION_NAME 'tinybird-tb-s3'
IMPORT_BUCKET_URI 's3://tinybird-tb/snowflake/csv/orders/*.csv.gz'
IMPORT_SCHEDULE '@auto'
```
Deploy the data source. The new files Snowflake writes to the bucket are automatically ingested by Tinybird in a few seconds.

4
## Handle duplicates in Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-incremental-updates#handle-duplicates-in-tinybird)

Use a materialized view to handle duplicates in Tinybird. For example:

tinybird/materializations/mat_s3_data.pipe - pipe to materialize data
```
NODE mat_s3_data_0
SQL >

    SELECT *
    FROM landing_ds

TYPE materialized
DATASOURCE deduplicate_rmt_mv
```
If you only need to work with the latest snapshot of the data, include a `file_ingested_at` field in the materializing pipe.

This is important when your incremental loads from Snowflake don’t indicate deleted records. Since a new ingest doesn’t overwrite a deleted record from a previous ingest, the record persists in the dataset. By filtering downstream on the `file_ingested_at` field, you can exclude these stale records and isolate only the most recent ingest. For example:

tinybird/materializations/mat_s3_data_latest.pipe - pipe to materialize data with timestamp
```
NODE mat_s3_data_latest_0
SQL >

    SELECT *, toDateTime(now()) as file_ingested_at
    FROM landing_ds

TYPE materialized
DATASOURCE deduplicate_rmt_mv
```
In the target Data Source, the `ReplacingMergeTree` and `ENGINE_VER` options [deduplicate records](/sql-reference/engines/replacingmergetree) with the same sorting key value.

tinybird/datasources/deduplicate_rmt_mv.datasource - Replacing Merge Tree to deduplicate data
```
SCHEMA >
    `O_ORDERKEY` Int64,
    `O_CUSTKEY` Int64,
    `O_ORDERSTATUS` String,
    `O_TOTALPRICE` Float32,
    `O_ORDERDATE` DateTime64(3),
    `O_ORDERPRIORITY` String,
    `O_CLERK` String,
    `file_ingested_at` DateTime

ENGINE "ReplacingMergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(O_ORDERDATE)"
ENGINE_SORTING_KEY "O_ORDERKEY"
ENGINE_VER "file_ingested_at"
```
Finally, create a pipe to query the deduplicated data source. Filter on the maximum timestamp to get the latest snapshot, [rounding the timestamp](/sql-reference/functions/date-time-functions#tostartofday) based on your import schedule. This ensures that you query the most recent snapshot with no duplicates.

tinybird/endpoints/get_snapshot.pipe - query for latest snapshot
```
NODE get_snapshot_0
SQL >

    WITH (SELECT max(toStartOfDay(file_ingested_at)) FROM deduplicate_rmt_mv) AS latest_file_version
    SELECT *
    FROM deduplicate_rmt_mv
    FINAL
    WHERE toStartOfDay(file_ingested_at) = latest_file_version

TYPE endpoint
```
Remember to use `final` when querying a ReplacingMergeTree.

5
## Next steps (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-incremental-updates#next-steps)

See the following resources:

- [Ingest from Snowflake using Azure Blob Storage](./ingest-from-snowflake-using-azure-blob-storage)
- [Ingest from Snowflake using AWS S3](./ingest-from-snowflake-using-aws-s3)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingest Snowflake via Azure Blob Storage \| Tinybird"
theme-color: "#171612"
description: "Learn how to send data from Snowflake to Tinybird using Azure Blob Storage."
inkeep:version: "forward"
---

# Ingest from Snowflake using Azure Blob Storage (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage#ingest-from-snowflake-using-azure-blob-storage)

Read on to learn how to send data from Snowflake to Tinybird, for example when you need to periodically run full replaces of a table or do a one-off ingest.

This process relies on [unloading](https://docs.snowflake.com/en/user-guide/data-unload-overview), or bulk exporting, data as gzipped CSVs and then ingesting them using the Data Sources API.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage#prerequisites)

To follow these steps you need a Tinybird account and access to Snowflake and permissions to create SAS Tokens for Azure Blob Storage.

1
## Unload the Snowflake table (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage#unload-the-snowflake-table)

Use Snowflake to [unload](https://docs.snowflake.com/en/user-guide/data-unload-overview) query results to flat files in an external storage service. For example:

```
COPY INTO 'azure://myaccount.blob.core.windows.net/unload/'
  FROM mytable
  CREDENTIALS = ( AZURE_SAS_TOKEN='****' )
  FILE_FORMAT = ( TYPE = CSV  COMPRESSION = GZIP )
  HEADER = FALSE;
```
The most basic implementation is [unloading directly](https://docs.snowflake.com/en/sql-reference/sql/copy-into-location#unloading-data-from-a-table-directly-to-files-in-an-external-location), but for production use cases consider adding a [named stage](https://docs.snowflake.com/en/user-guide/data-unload-azure#unloading-data-into-an-external-stage) as suggested in the Snowflake docs. Stages give you fine-grained control to access rights.

2
## Create a SAS token for the file (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage#create-a-sas-token-for-the-file)

Using [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli), generate a [shared access signature (SAS) token](https://learn.microsoft.com/en-us/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens?tabs=blobs) so Tinybird can read the file:

```
az storage blob generate-sas \
    --account-name myaccount \
    --account-key '****' \
    --container-name unload \
    --name data.csv.gz \
    --permissions r \
    --expiry <expiry-ts> \
    --https-only \
    --output tsv \
    --full-uri

> 'https://myaccount.blob.core.windows.net/unload/data.csv.gz?se=2024-05-31T10%3A57%3A41Z&sp=r&spr=https&sv=2022-11-02&sr=b&sig=PMC%2E9ZvOFtKATczsBQgFSsH1%2BNkuJvO9dDPkTpxXH0g%5D'
```
You can use the same behavior in S3 and GCS to generate presigned URLs.

3
## Ingest into Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage#ingest-into-tinybird)

Take the generated URL and make a call to Tinybird. You need a [token](/docs/administration/auth-tokens) with `DATASOURCES:CREATE` permissions:

```
curl \
-H "Authorization: Bearer <DATASOURCES:CREATE token>" \
-X POST "https://<your_host>/v0/datasources?name=my_datasource_name" \
-d url='https://myaccount.blob.core.windows.net/unload/data.csv.gz?se=2024-05-31T10%3A57%3A41Z&sp=r&spr=https&sv=2022-11-02&sr=b&sig=PMC%2E9ZvOFtKATczsBQgFSsH1%2BNkuJvO9dDPkTpxXH0g%5D'
```
You now have your Snowflake Table in Tinybird.

## Automation (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage#automation)

To adapt to production scenarios, like having to append data on a timely basis or replacing data that has been updated in Snowflake, you might need to define scheduled actions to move the data.

## Limits (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage#limits)

Because you're using the data sources API, its [limits](/docs/api-reference#limits) apply.

You might need to adjust your [COPY INTO <location>](https://docs.snowflake.com/en/sql-reference/sql/copy-into-location) expression adding `PARTITION` or `MAX_FILE_SIZE = 5000000000`. For example:

```
COPY INTO 'azure://myaccount.blob.core.windows.net/unload/'
  FROM mytable
  CREDENTIALS=( AZURE_SAS_TOKEN='****')
  FILE_FORMAT = ( TYPE = CSV  COMPRESSION = GZIP )
  HEADER = FALSE
  MAX_FILE_SIZE = 5000000000;
```

## Next steps (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-azure-blob-storage#next-steps)

See the following resources:

- [Ingest from Snowflake using AWS S3](./ingest-from-snowflake-using-aws-s3)
- [Ingest from Snowflake using incremental updates](./ingest-from-snowflake-using-incremental-updates)
- [GCS Connector](/docs/forward/ingest-data/connectors/gcs)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-aws-s3
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Ingest from Snowflake using AWS S3 \| Tinybird"
theme-color: "#171612"
description: "Learn how to send data from Snowflake to Tinybird using AWS S3."
inkeep:version: "forward"
---

# Ingest data from Snowflake using AWS S3 (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-aws-s3#ingest-data-from-snowflake-using-aws-s3)

Read on to learn how to send data from Snowflake to Tinybird, for example when you need to periodically run full replaces of a table or do a one-off ingest.

This process relies on [unloading](https://docs.snowflake.com/en/user-guide/data-unload-overview), or bulk exporting, data as gzipped CSVs and then ingesting them using the Data Sources API. Data is then ingested using the [S3 Connector](/docs/forward/ingest-data/connectors/s3).

## Prerequisites (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-aws-s3#prerequisites)

To follow these steps you need a Tinybird account and access to Snowflake and AWS S3.

1
## Unload the Snowflake table (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-aws-s3#unload-the-snowflake-table)

The first step consists in unloading the Snowflake table to a gzipped CSV file.

1. Grant the required permissions in AWS IAM Console
Make sure the S3 bucket allows Snowflake to write files by setting up an appropriate IAM role or policy. You can use this template to create the policy and attach it to the AWS role:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["s3:PutObject", "s3:AbortMultipartUpload"],
            "Resource": "arn:aws:s3:::your-bucket-name/path/*"
        }
    ]
}
```
Replace `your-bucket-name/path/*` with your bucket name, and optionally the path you want to grant access to. Attach the policy to the role you want use to unload the data from Snowflake.

1. Create the storage integration
Run the following SQL statement to create the storage integration:

```
/* Create the S3 integration.
 */
CREATE or replace STORAGE INTEGRATION tinybird_integration
  TYPE = EXTERNAL_STAGE
  STORAGE_PROVIDER = 'S3'
  ENABLED = TRUE
  STORAGE_AWS_ROLE_ARN = '<arn_role>'
  STORAGE_ALLOWED_LOCATIONS = ('*');

-- describe integration tinybird_integration;
```
Replace `<arn_role>` with the ARN of the role created in the previous step.

1. Create the file format
Run the following SQL statement to create the file format:

```
/* Create the file format for the output files generated.
 */
CREATE OR REPLACE FILE FORMAT csv_unload_format
  TYPE = 'CSV';
```

1. Create the stage
Run the following SQL statement to create the stage:

```
/* And finally the stage we'll use to unload the data to.
 */
CREATE or replace STAGE tinybird_stage
  STORAGE_INTEGRATION = tinybird_integration
  URL = 's3://your-bucket-name/path/'
  FILE_FORMAT = csv_unload_format;
```
Replace `your-bucket-name` and `path` with your S3 bucket details.

1. Unload the data
Run the following SQL statement to unload the data:

```
COPY INTO @tinybird_stage/orders/ from (
    select
        O_ORDERKEY, O_CUSTKEY, O_ORDERSTATUS, O_TOTALPRICE, O_ORDERDATE,
        O_ORDERPRIORITY, O_CLERK
    from my_database.my_schema.orders
)
```
To automate the unloading, you can create a Snowflake task that runs the `COPY INTO` on a schedule. For example:

```
CREATE or replace TASK export_order_deltas
    WAREHOUSE = compute_wh
    SCHEDULE = 'USING CRON 05 * * * * UTC'
AS
COPY INTO @tinybird_stage/orders from (
    select
        O_ORDERKEY, O_CUSTKEY, O_ORDERSTATUS, O_TOTALPRICE, O_ORDERDATE,
        O_ORDERPRIORITY, O_CLERK
    from my_database.my_schema.orders
) max_file_size=1000000000
```
2
## Ingest data into Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-aws-s3#ingest-data-into-tinybird)

Before ingesting your Snowflake data from the S3 bucket, you need to create the S3 connection. See [S3 Connector](/docs/forward/ingest-data/connectors/s3).

You can use the following schema for the data source:

```
SCHEMA >
    `O_ORDERKEY` Int64,
    `O_CUSTKEY` Int64,
    `O_ORDERSTATUS` String,
    `O_TOTALPRICE` Float32,
    `O_ORDERDATE` DateTime64(3),
    `O_ORDERPRIORITY` String,
    `O_CLERK` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYear(O_ORDERDATE)"
ENGINE_SORTING_KEY "O_ORDERDATE, O_ORDERPRIORITY, O_CLERK"

IMPORT_CONNECTION_NAME 'tb-s3'
IMPORT_BUCKET_URI 's3://tb/snowflake/csv/orders/*.csv.gz'
IMPORT_SCHEDULE '@auto'
```
Deploy the data source. The new files Snowflake writes to the bucket are automatically ingested by Tinybird.

Each file is appended to the Tinybird data source. As records might be duplicated, consider using a materialized view to consolidate a stateful set of your Snowflake table.

## Limits (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-aws-s3#limits)

Because you're using the data sources API, its [limits](/docs/api-reference#limits) apply.

You might need to adjust your [COPY INTO <location>](https://docs.snowflake.com/en/sql-reference/sql/copy-into-location) expression adding `PARTITION` or `MAX_FILE_SIZE = 5000000000`.

## Next steps (https://www.tinybird.co/docs/forward/guides/ingest-from-snowflake-using-aws-s3#next-steps)

See the following resources:

- [Ingest from Snowflake using Azure Blob Storage](./ingest-from-snowflake-using-azure-blob-storage)
- [Ingest from Snowflake using incremental updates](./ingest-from-snowflake-using-incremental-updates)
- [GCS Connector](/docs/forward/ingest-data/connectors/gcs)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-sentry
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send Sentry Webhooks to Tinybird \| Tinybird"
theme-color: "#171612"
description: "Learn how to send Sentry events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---

# Send Sentry events to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-sentry#send-sentry-events-to-tinybird)

[Sentry](https://sentry.io/) is a platform for monitoring and alerting on errors in your applications. By integrating Sentry with Tinybird, you can analyze your Sentry events in real time and enrich it with other data sources.

Some common use cases for sending Sentry events to Tinybird include:

1. Analyze errors from your applications.
2. Detect patterns in your error data.
3. Build an alert system based on error patterns.
4. Build custom analytical dashboards.
Read on to learn how to send logs from Sentry to Tinybird.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-sentry#before-you-start)

Before you connect Sentry to Tinybird, ensure:

- You have a Sentry account.
- You have a Tinybird workspace.

## Connect Sentry to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-sentry#connect-sentry-to-tinybird)

1. In Sentry, go to **Settings** > **Developer Settings** > **Custom Integrations**.
2. Select **Create New Integration**.
3. In your Tinybird project, create a data source called `sentry`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/sentry.datasource):

```
SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.action` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Sentry in a single column. You can later retrieve various events and their metadata as needed in your pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).

1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in Sentry, paste the Events API URL in your Custom Integration. Use the query parameter `name` to match the name of the data source you created in Tinybird.

```
https://<your_host>/v0/events?name=sentry&token=<your user token>
```

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

1. Select **Alert Rule Action**.
2. In the **Permissions** box **Issue and Event** > **Read**.
3. Check all webhooks and **Save Changes**.
4. If you also want to send alerts to Tinybird, select **Alerts** from the left menu, select an alert, and select **Edit Rule**. You can select **Send Notifications via** your previously created Custom Integration.
5. You can then select **Send Test Notification** to check the connection.
6. You're done. Any of the Sentry events you selected are automatically sent to Tinybird through the [Events API](/docs/forward/ingest-data/events-api).
Check the status of the integration from the **Log** tab in the Tinybird `sentry` data source.

## See also (https://www.tinybird.co/docs/forward/guides/ingest-from-sentry#see-also)

- [Sentry Webhooks](https://docs.sentry.io/organization/integrations/integration-platform/webhooks/)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Stream from RudderStack \| Tinybird"
theme-color: "#171612"
description: "In this guide, you'll learn two different methods to send events from RudderStack to Tinybird."
inkeep:version: "forward"
---

# Stream from RudderStack (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#stream-from-rudderstack)

This guide shows you two methods to send events from RudderStack to Tinybird.

To better understand the behavior of their customers, companies need to unify timestamped data coming from a wide variety of products and platforms. Typical events to track would be 'sign up', 'login', 'page view' or 'item purchased'. A customer data platform can be used to capture complete customer data like this from wherever your customers interact with your brand. It defines events, collects them from different platforms and products, and routes them to where they need to be consumed.

[RudderStack](https://www.rudderstack.com/) is an open source customer data pipeline tool. It collects, processes and routes data from your websites, apps, cloud tools, and data warehouse. By using Tinybird's event ingestion Endpoint for [high-frequency ingestion](/docs/forward/ingest-data/events-api) as a Webhook in RudderStack, you can stream customer data in real time to Data Sources.

## Option 1: A separate data source for each event type (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#option-1-a-separate-data-source-for-each-event-type)

This is the preferred approach. It sends each type of event to a corresponding data source.

The advantages of this method are:

- Your data is well organized from the start.
- Different event types can have different attributes (columns in their data source).
- Whenever new attributes are added to an event type, a prompt asks you to add new columns.
- New event types get a new Data Source.
Start by getting a Token to allow RudderStack to write to Tinybird.

### Get a Tinybird Token (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#get-a-tinybird-token)

Use your **Workspace admin token**, which has full access to your workspace. You can find it in **Tokens** in the sidebar, or list your tokens with the CLI:

```
tb token ls
```

### Create a RudderStack destination (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#create-a-rudderstack-destination)

In RudderStack, select **Destinations** in the side panel and then **New destination**.

Select **Webhook**:

1. Give the destination a descriptive name.
2. Connect your sources, you can test with the RudderStack Sample HTTP Source.
3. Input the following Connection Settings:

- Webhook URL: *< https://<your_host> /v0/events>*
- URL Method: *POST*
- Headers Key: *Authorization*
- Headers Value: *Bearer TINYBIRD_AUTH_TOKEN*
On the next page, select **Create new transformation**.

You can code a function in the box to apply to events when this transformation is active using the following example snippet. In this function, you can dynamically append the target data source to the target URL of the Webhook. Give your transformation a descriptive name and a helpful description.

Transformation code
```
export function transformEvent(event, metadata){
    event.appendPath="?name=rudderstack_"+event.event.toLowerCase().replace(/[\s\.]/g, '_')
    return event;
}
```
This example snippet uses the prefix `*rudderstack\_*` followed by the name of the event in lower case, with its words separated by an underscore. For instance, a "Product purchased" event would go to a data source named `rudderstack_product_purchased`.

Save the transformation. Your destination has been created successfully.

### Test ingestion (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#test-ingestion)

In RudderStack, select **Sources** > **RudderStack Sample HTTP** > **Live events** > **Send test event** and paste the provided curl command into your terminal. The event appears on the screen and is sent to Tinybird.

## Option 2: All events in the same data source (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#option-2-all-events-in-the-same-data-source)

This alternative approach sends all events into a single Data Source and then splits them using Tinybird. By pre-configuring the Data Source, Tinybird ingests any events that RudderStack sends with the full JSON object as a String in a single column. This is useful for complex JSON objects, but using JSONExtract to parse data from the JSON object after ingestion affects performance.

Tinybird detects new columns from parsing the data and asks whether you want to save them. You can adjust the inferred data types before saving any new columns. Use Pipes to filter the Data Source by different events.

### Preconfigure a data source (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#preconfigure-a-data-source)

Create a new file in your local project, named `rudderstack_events.datasource`, for example. Add an `APPEND` token and configure the schema:

rudderstack_events.datasource
```
TOKEN rudderstack_append APPEND

SCHEMA >
    `value` String `json:$`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "value"
```
Deploy the changes using `tb deploy`. Tinybird creates the `rudderstack_append` token automatically during deployment. Retrieve its value with:

```
tb token ls
```
This pre-configured Data Source is required only if you need a column containing the full JSON object as a String. Otherwise, skip this step and let Tinybird infer the columns and data types when you send the first event. Then select which columns to save and adjust their data types.

### Create a RudderStack destination (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#create-a-rudderstack-destination)

In RudderStack, select **Destinations** in the side panel and then **New destination**.

Select **Webhook**:

1. Give the destination a descriptive name.
2. Connect your sources, you can test with the RudderStack Sample HTTP Source.
3. Input the following Connection Settings:

- Webhook URL: *< https://<your_host> /v0/events?name=rudderstack_events>*
- URL Method: *POST*
- Headers Key: *Authorization*
- Headers Value: *Bearer TINYBIRD_AUTH_TOKEN*
Select **No transformation needed** and save. Your destination has been created successfully.

### Test ingestion (https://www.tinybird.co/docs/forward/guides/ingest-from-rudderstack#test-ingestion)

Select **Sources** > **RudderStack Sample HTTP** > **Live events** > **Send test event** and paste the provided curl command into your terminal. The event appears on the screen and is sent to Tinybird.

The `value` column contains the full JSON object. You can also parse the data into columns. When viewing the new columns, select which ones to save and adjust their data types.

Whenever Tinybird detects new columns in the stream of events, it asks whether you want to save them.

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-resend
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send Resend webhooks to Tinybird \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to send data from Resend to Tinybird."
inkeep:version: "forward"
---

# Send Resend webhooks to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-resend#send-resend-webhooks-to-tinybird)

With [Resend](https://resend.com/) you can send and receive emails programmatically. By integrating Resend with Tinybird, you can analyze your email data in real time.

Some common use cases for sending Resend webhooks to Tinybird include:

1. Tracking email opens and clicks.
2. Monitoring delivery rates and bounces.
3. Analyzing user engagement patterns.
4. Creating custom dashboards for email performance.
5. Enriching other data sources with real-time email metrics.
Read on to learn how to send data from Resend to Tinybird.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-resend#before-you-start)

Before you connect Resend to Tinybird, ensure:

- You have a Resend account.
- You have a Tinybird workspace.

## Connect Resend to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-resend#connect-resend-to-tinybird)

1. Open the Resend UI and go to the Webhooks page.
2. Select **Add Webhook**.
3. In your Tinybird project, create a data source called `resend`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/resend.datasource):

```
SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.type` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Resend in a single column. You can later retrieve various events and their metadata as needed in your pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).

1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in Resend, paste the Events API URL in your Webhook URL. Use the query parameter `name` to match the name of the data source you created in Tinybird. For example:

```
https://<your_host>/v0/events?name=resend&token=<your user token>
```

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

1. Select the checkboxes for the Resend events you want to send to Tinybird, and select **Add**.
2. You're done. Sending emails to Resend now pushes events to Tinybird through the [Events API](/docs/forward/ingest-data/events-api).

## See also (https://www.tinybird.co/docs/forward/guides/ingest-from-resend#see-also)

- [Resend event types](https://resend.com/docs/dashboard/webhooks/event-types)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-pagerduty
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send PagerDuty events to Tinybird \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to send PagerDuty events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---

# Send PagerDuty events to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-pagerduty#send-pagerduty-events-to-tinybird)

[PagerDuty](https://www.pagerduty.com/) is a platform for incident management and alerting. By integrating PagerDuty with Tinybird, you can analyze your incident data in real time and enrich it with other data sources.

Some common use cases for sending PagerDuty events to Tinybird include:

1. Monitoring and alerting on incidents.
2. Creating custom dashboards for incident analysis.
3. Incident logs.
Read on to learn how to send events from PagerDuty to Tinybird.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-pagerduty#before-you-start)

Before you connect PagerDuty to Tinybird, ensure:

- You have an PagerDuty account.
- You have a Tinybird workspace.

## Connect PagerDuty to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-pagerduty#connect-pagerduty-to-tinybird)

1. From the PagerDuty dashboard, select **Integrations** > **Developer Tools** > **Webhooks**.
2. Select **New Webhook**.
3. In your Tinybird project, create a data source called `pagerduty`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/pagerduty.datasource):

```
SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.event.event_type` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from PagerDuty in a single column. You can later retrieve various events and their metadata as needed in your pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).

1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in PagerDuty, paste the Events API URL in your Webhook URL. Use the query parameter `name` to match the name of the data source you created in Tinybird. For example:

```
https://<your_host>/v0/events?name=pagerduty
```

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

1. Select **Add custom header** and add 'Authorization' as **Name** and paste the token you created in Tinybird as **Value**.

```
Bearer <your user token>
```

1. Select all event subcriptions and **Add webhook**
2. You're done. Any of the PagerDuty events is automatically sent to Tinybird through the [Events API](/docs/forward/ingest-data/events-api).
You can check the status of the integration by testing the Webhook integration in PagerDuty or from the **Log** tab in the Tinybird `pagerduty` data source.

## See also (https://www.tinybird.co/docs/forward/guides/ingest-from-pagerduty#see-also)

- [PagerDuty webhooks](https://support.pagerduty.com/docs/webhooks)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-orb
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send Orb events to Tinybird \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to send Orb events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---

# Send Orb events to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-orb#send-orb-events-to-tinybird)

[Orb](https://withorb.com/) is a developer-focused platform to manage your subscription billing and revenue operations. By integrating Orb with Tinybird, you can analyze your subscription billing data in real time and enrich it with other data sources.

Some common use cases for sending Orb events to Tinybird include:

1. Tracking and monitoring subscriptions.
2. Monitoring user churn.
3. Creating custom dashboards for subscription analysis.
4. Subscriptions logs.
Read on to learn how to send events from Orb to Tinybird.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-orb#before-you-start)

Before you connect Orb to Tinybird, ensure:

- You have an Orb account.
- You have a Tinybird workspace.

## Connect Orb to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-orb#connect-orb-to-tinybird)

1. From the Orb dashboard, select **Developers** > **Webhooks**.
2. Select **Add Endpoint**.
3. In your Tinybird project, create a data source called `orb`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/orb.datasource):

```
SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.type` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Orb in a single column. You can later retrieve various events and their metadata as needed in your pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).

1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in Orb, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name` to match the name of the data source you created in Tinybird. For example:

```
https://<your_host>/v0/events?name=orb&token=<your user token>
```

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

1. Select **Send test request** to test the connection and check the data gets to the `orb` data source in Tinybird.
2. You're done. Any of the Orb events is automatically sent to Tinybird through the [Events API](/docs/forward/ingest-data/events-api).
You can check the status of the integration by clicking on the Webhook endpoint in Orb or from the **Log** tab in the Tinybird `orb` data source.

## See also (https://www.tinybird.co/docs/forward/guides/ingest-from-orb#see-also)

- [Orb webhooks](https://docs.withorb.com/guides/integrations-and-exports/webhooks)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingest from OpenTelemetry \| Tinybird"
theme-color: "#171612"
description: "Learn how to ingest metrics, traces, and logs from OpenTelemetry into Tinybird using the Tinybird OpenTelemetry Collector distribution."
inkeep:version: "forward"
---

# Ingest from OpenTelemetry (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#ingest-from-opentelemetry)

[OpenTelemetry](https://opentelemetry.io/) is an open source observability framework for collecting, processing, and exporting telemetry data (metrics, traces, and logs) from your apps and infrastructure.

By integrating OpenTelemetry with Tinybird, you can analyze observability data in real time, build dashboards, and enrich it with other data sources.

Some common use cases for sending OpenTelemetry data to Tinybird include:

1. Centralizing metrics, traces, and logs for unified analytics.
2. Building custom dashboards and alerts on top of observability data.
3. Enriching telemetry with business or app data.
Read on to learn how to send data from OpenTelemetry to Tinybird.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#before-you-start)

Before you connect OpenTelemetry to Tinybird, ensure:

- You have a Tinybird workspace.
- You have a Tinybird Token with **append** permissions to the target Data Sources.
- You are running a release version of the OpenTelemetry Collector higher than release v0.131.0.

- [GitHub Releases](https://github.com/open-telemetry/opentelemetry-collector-contrib/releases)
- [Docker Hub](https://hub.docker.com/r/otel/opentelemetry-collector-contrib)
The Tinybird OpenTelemetry Exporter is officially available in the OpenTelemetry Collector Contrib repository as of release v0.131.0. You can use the official OpenTelemetry Collector distributions that include the Tinybird exporter by default.

## Use the Tinybird OpenTelemetry project template (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#use-the-tinybird-opentelemetry-project-template)

To get started quickly, you can use the [Tinybird OpenTelemetry project template](https://www.tinybird.co/templates/opentelemetry). This template provides ready-to-use Data Sources and Pipes for storing and analyzing your telemetry data in Tinybird.

```
# authenticate and select your Workspace
tb login

# deploy the template
tb --cloud deploy --template https://github.com/tinybirdco/tinybird-otel-template/tree/main/
```

## Recommended OpenTelemetry Collector configuration (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#recommended-opentelemetry-collector-configuration)

The following example configures the Tinybird OpenTelemetry Collector to export metrics, traces, and logs to Tinybird:

config.yaml
```
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s
    send_batch_size: 8192

exporters:
  tinybird:
    endpoint: ${OTEL_TINYBIRD_API_HOST}         # Your Events API endpoint, e.g. https://api.us-east.aws.tinybird.co
    token: ${OTEL_TINYBIRD_TOKEN}               # Token with append permissions
    sending_queue:
      enabled: true
      queue_size: 104857600                # Total memory buffer in bytes (100 MB)
      sizer: bytes
      batch:
        flush_timeout: 5s                  # Max wait time before flushing
        min_size: 1024000                  # Min batch size: 1 MB
        max_size: 8388608                  # Max batch size: 8 MB (Events API limit is 10 MB)
    retry_on_failure:
      enabled: true
    metrics:
      sum:
        datasource: otel_metrics_sum
      histogram:
        datasource: otel_metrics_histogram
      exponential_histogram:
        datasource: otel_metrics_exponential_histogram
      gauge:
        datasource: otel_metrics_gauge
    logs:
      datasource: otel_logs
    traces:
      datasource: otel_traces

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [tinybird]

    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [tinybird]

    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [tinybird]
```

### Environment variables (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#environment-variables)

- `OTEL_TINYBIRD_API_HOST`: The API host for your Tinybird Workspace, for example `{% user("apiHost", "https://api.tinybird.co") %}`.
- `OTEL_TINYBIRD_TOKEN`: A Tinybird token with **append** permissions to the target Data Sources (`otel_metrics`, `otel_traces`, `otel_logs`).
You can create a token in the Tinybird UI under **Tokens**. Make sure it has the required append permissions for the Data Sources you want to ingest into.

### Run the collector (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#run-the-collector)

#### Using the binary (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#using-the-binary)

Precompiled binaries for Linux and macOS are available for both `amd64` and `arm64` architectures.

Download them from the [GitHub Releases page](https://github.com/tinybirdco/opentelemetry-collector-contrib/releases).

```
./otelcontribcol_linux_amd64 --config config.yaml
```

#### Using Docker (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#using-docker)

```
docker run \
  --platform linux/amd64 \
  -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml \
  -p 4317:4317 \
  -p 4318:4318 \
  -e OTEL_TINYBIRD_API_HOST="${OTEL_TINYBIRD_API_HOST}" \
  -e OTEL_TINYBIRD_TOKEN="${OTEL_TINYBIRD_TOKEN}" \
  tinybirdco/opentelemetry-collector-contrib:v0.128.0
```
Use the `--platform linux/amd64` flag to ensure compatibility when running on ARM-based systems like Apple Silicon Macs.

### Troubleshooting (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#troubleshooting)

To troubleshoot your setup, edit your `config.yaml` to enable detailed logging from the Collector. You can also check the [Tinybird Service Data Sources](/docs/forward/monitoring/service-datasources) to confirm data is arriving in your Workspace.

To enable detailed logging, define the `debug` exporter and add it to your service pipelines:

config.yaml
```
#(...)
exporters:
  debug:
    verbosity: detailed
exporters:
  tinybird:
    endpoint: ${OTEL_TINYBIRD_API_HOST}

#(...)
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug,tinybird]
#(...)
```
You can send mock data with [otelgen](https://github.com/krzko/otelgen).

## Next steps (https://www.tinybird.co/docs/forward/guides/ingest-from-opentelemetry#next-steps)

- Explore and customize the [Tinybird OpenTelemetry project template](https://github.com/tinybirdco/tinybird-otel-template) to fit your needs.
- Run the official [opentelemetry-demo](https://github.com/tinybirdco/opentelemetry-demo) locally.
- Connect to [Grafana](/forward/guides/connect-grafana), HyperDX and other [visualization tools](/forward/guides).
For more details on the available configuration options, see the [Tinybird OpenTelemetry Collector documentation](https://github.com/tinybirdco/opentelemetry-collector-contrib).

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-mongodb
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingest data from MongoDB \| Tinybird"
theme-color: "#171612"
description: "In this guide, you'll learn how to ingest data into Tinybird from MongoDB."
inkeep:version: "forward"
---

# Connect MongoDB to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-mongodb#connect-mongodb-to-tinybird)

This guide shows you how to ingest data into Tinybird from MongoDB.

Use these components:

- MongoDB Atlas as the source MongoDB database.
- Confluent Cloud's MongoDB Atlas Source connector to capture change events from MongoDB Atlas and push to Kafka
- Tinybird Confluent Cloud connector to ingest the data from Kafka
This guide uses Confluent Cloud as a managed Kafka service, and MongoDB Atlas as a managed MongoDB service. You can use any Kafka service and MongoDB instance, but the setup steps may vary.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/ingest-from-mongodb#prerequisites)

This guide assumes you have:

- An existing Tinybird account & workspace
- An existing Confluent Cloud account
- An existing MongoDB Atlas account & collection

## 1. Create a Confluent Cloud source (https://www.tinybird.co/docs/forward/guides/ingest-from-mongodb#1-create-a-confluent-cloud-source)

[Create a new MongoDB Atlas Source in Confluent Cloud](https://docs.confluent.io/cloud/current/connectors/cc-mongo-db-source.html#get-started-with-the-mongodb-atlas-source-connector-for-ccloud). Use the following template to configure the Source:

```
{
  "name": "<CONNECTOR NAME>",
  "config": {
    "name": "<CONNECTOR NAME>",

    "connection.host": "<MONGO HOST>",
    "connection.user": "<MONGO USER>",
    "connection.password": "<MONGO PASS>",
    "database": "<MONGO DATABASE>",
    "collection": "<MONGO COLLECTION>",

    "cloud.provider": "<CLOUD PROVIDER>",
    "cloud.environment": "<CLOUD ENV>",
    "kafka.region": "<KAFKA REGION>",
    "kafka.auth.mode": "KAFKA_API_KEY",
    "kafka.api.key": "<KAFKA KEY>",
    "kafka.api.secret": "<KAFKA SECRET>",
    "kafka.endpoint": "<KAFKA ENDPOINT>",

    "topic.prefix": "<KAFKA TOPIC PREFIX>",
    "errors.deadletterqueue.topic.name": "<KAFKA DEADLETTER TOPIC>",

    "startup.mode": "copy_existing",
    "copy.existing": "true",
    "copy.existing.max.threads": "1",
    "copy.existing.queue.size": "16000",

    "poll.await.time.ms": "5000",
    "poll.max.batch.size": "1000",
    "heartbeat.interval.ms": "10000",
    "errors.tolerance": "all",
    "max.batch.size": "100",

    "connector.class": "MongoDbAtlasSource",
    "output.data.format": "JSON",
    "output.json.format": "SimplifiedJson",
    "json.output.decimal.format": "NUMERIC",
    "change.stream.full.document": "updateLookup",
    "change.stream.full.document.before.change": "whenAvailable",
    "tasks.max": "1"
  }
}
```
When the source is created, a new Kafka topic appears in your Confluent Cloud account. This topic contains the change events from your MongoDB collection.

## 2. Create Tinybird data source (https://www.tinybird.co/docs/forward/guides/ingest-from-mongodb#2-create-tinybird-data-source)

Create a new Kafka data source in your Tinybird project. See [Kafka](/docs/forward/ingest-data/connectors/kafka) for more information.

The data source should have the following schema:

```
SCHEMA >
    `_id` String `json:$.documentKey._id` DEFAULT JSONExtractString(__value, '_id._id'),
    `operation_type` LowCardinality(String) `json:$.operationType`,
    `database` LowCardinality(String) `json:$.ns.db`,
    `collection` LowCardinality(String) `json:$.ns.coll`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(__timestamp)"
ENGINE_SORTING_KEY "__timestamp, _id"

KAFKA_CONNECTION_NAME '<CONNECTION NAME>'
KAFKA_TOPIC '<KAFKA TOPIC>'
KAFKA_GROUP_ID '<KAFKA CONSUMER GROUP ID>'
KAFKA_AUTO_OFFSET_RESET 'earliest'
KAFKA_STORE_RAW_VALUE 'True'
KAFKA_STORE_HEADERS 'False'
KAFKA_STORE_BINARY_HEADERS 'True'
KAFKA_TARGET_PARTITIONS 'auto'
KAFKA_KEY_AVRO_DESERIALIZATION ''
```
Deploy the Data Source before proceeding to the next step.

## 3. Validate the data source (https://www.tinybird.co/docs/forward/guides/ingest-from-mongodb#3-validate-the-data-source)

Go to Tinybird Cloud and validate that a data source has been created.

As changes occur in MongoDB, you should see the data being ingested into Tinybird. This is an append log of all changes, so multiple records appear for the same document as it's updated.

## 4. Deduplicate with ReplacingMergeTree (https://www.tinybird.co/docs/forward/guides/ingest-from-mongodb#4-deduplicate-with-replacingmergetree)

Tinybird creates a new data source using the ReplacingMergeTree engine to store the deduplicated data, and a pipe to process the data from the original data source and write to the new data source.

First, create a new data source to store the deduplicated data.

Create a new file called `deduped_ds.datasource` and add the following content:

```
SCHEMA >
    `fullDocument` String,
    `_id` String,
    `database` LowCardinality(String),
    `collection` LowCardinality(String),
    `k_timestamp` DateTime,
    `is_deleted` UInt8

ENGINE "ReplacingMergeTree"
ENGINE_SORTING_KEY "_id"
ENGINE_VER "k_timestamp"
ENGINE_IS_DELETED "is_deleted"
```
Then, create a new file called `dedupe_mongo.pipe` and add the following content:

```
NODE mv
SQL >

    SELECT
        JSONExtractRaw(__value, 'fullDocument') as fullDocument,
        _id,
        database,
        collection,
        __timestamp as k_timestamp,
        if(operation_type = 'delete', 1, 0) as is_deleted
    FROM <ORIGINAL DATASOURCE NAME>

TYPE materialized
DATASOURCE <DESTINATION DATASOURCE NAME>
```
Deploy the changes. As new data arrives via Kafka, it's processed automatically through the materialized view, writing it into the `ReplacingMergeTree` data source.

Query this new data source to access the deduplicated data:

```
SELECT * FROM deduped_ds FINAL
```

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-mailgun
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send Mailgun Events to Tinybird \| Tinybird"
theme-color: "#171612"
description: "Learn how to send Mailgun events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---

# Send Mailgun events to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-mailgun#send-mailgun-events-to-tinybird)

[Mailgun](https://www.mailgun.com/) is a platform for sending email, and it provides a way to send events to Tinybird using webhooks.

Read on to learn how to send events from Mailgun to Tinybird.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-mailgun#before-you-start)

Before you connect Mailgun to Tinybird, ensure:

- You have a Mailgun account.
- You have a Tinybird workspace.

## Connect Mailgun to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-mailgun#connect-mailgun-to-tinybird)

Mailgun provides a variety of [webhook event types](https://mailgun-docs.redoc.ly/docs/mailgun/user-manual/events/#event-structure) that you can use to send events to Tinybird.

This guide covers the base case for sending Mailgun events to Tinybird.

1. In Mailgun, go to **Send** > **Sending** > **Webhooks**.
2. Select **Domain** and **Add webhook**.
3. In your Tinybird project, create a data source called `mailgun`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/mailgun.datasource):

```
SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.event-data.event` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Mailgun in a single column. You can later retrieve various events and their metadata as needed in your pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).

1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in Mailgun, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name` to match the name of the data source you created in Tinybird.

```
https://<your_host>/v0/events?name=mailgun&format=json&token=<your user token>
```
Make sure to use the `format=json` query parameter.

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

1. Select **Event type** and choose the event you want to send to Tinybird. You can use the same Tinybird data source for multiple events.
2. Select **Create webhook** abd you're done.
Check the status of the integration from the **Log** tab in the Tinybird `mailgun` data source.

## See also (https://www.tinybird.co/docs/forward/guides/ingest-from-mailgun#see-also)

- [Mailgun Events](https://mailgun-docs.redoc.ly/docs/mailgun/user-manual/events/#event-structure)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-knock
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send Knock Events to Tinybird \| Tinybird"
theme-color: "#171612"
description: "Learn how to send Knock events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---

# Send Knock events to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-knock#send-knock-events-to-tinybird)

[Knock](https://knock.app/) is a platform for notifications and alerts, and it provides a way to send events to Tinybird using webhooks.

Some common use cases for sending Knock events to Tinybird include:

1. Monitor Knock message events.
2. Run analytical workflows based on Knock events.
3. Create custom dashboards based on Knock events.
4. Create alerts and notifications based on Knock events.
5. Join Knock message events with other data sources to enrich your user data.
Read on to learn how to send events from Knock to Tinybird.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-knock#before-you-start)

Before you connect Knock to Tinybird, ensure:

- You have a Knock account.
- You have a Tinybird workspace.

## Connect Knock to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-knock#connect-knock-to-tinybird)

Knock provides a variety of [webhook event types](https://docs.knock.app/developer-tools/outbound-webhooks/event-types#message-events) that you can use to send events to Tinybird.

This guide covers the base case for sending Knock Message events to Tinybird.

1. In Knock, go to your repository **Developers** > **Webhooks**.
2. Select **Create webhook**.
3. Webhooks payloads vary depending on the event type. You can check here the list of [Knock events](https://docs.knock.app/developer-tools/outbound-webhooks/event-types#message-events).
For this guide, select events related to `message`.

1. In your Tinybird project, create a data source called `knock`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/knock.datasource):

```
SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.type` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Knock in a single column. You can later retrieve various events and their metadata as needed in your pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).

1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in Knock, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name` to match the name of the data source you created in Tinybird.

```
https://<your_host>/v0/events?name=knock&token=<your user token>
```

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

1. Select **Save webhook**.
2. You're done.
Check the status of the integration from the `Logs` tab in the Knock webhook or from the **Log** tab in the Tinybird `knock` data source.

## See also (https://www.tinybird.co/docs/forward/guides/ingest-from-knock#see-also)

- [Knock Webhooks](https://docs.knock.app/developer-tools/outbound-webhooks/overview)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-google-pubsub
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingest from Google Pub/Sub \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to send data from Google Pub/Sub to Tinybird."
inkeep:version: "forward"
---

# Stream from Google Pub/Sub (https://www.tinybird.co/docs/forward/guides/ingest-from-google-pubsub#stream-from-google-pubsub)

This guide shows you how to send data from Google Pub/Sub to Tinybird.

## Overview (https://www.tinybird.co/docs/forward/guides/ingest-from-google-pubsub#overview)

[Google Pub/Sub](https://cloud.google.com/pubsub) is often used as a messaging middleware that decouples event stream sources from the end destination. Pub/Sub streams are usually consumed by Google's DataFlow which can send events on to destinations such as BigQuery, BigTable, or Google Cloud Storage.

This DataFlow pattern works with Tinybird too, however, Pub/Sub also has a feature called [Push subscriptions](https://cloud.google.com/pubsub/docs/push) which can forward messages directly from Pub/Sub to Tinybird. The following guide steps use the subscription approach.

## Push messages from Pub/Sub to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-google-pubsub#push-messages-from-pubsub-to-tinybird)

### 1. Create a Pub/Sub topic (https://www.tinybird.co/docs/forward/guides/ingest-from-google-pubsub#1-create-a-pubsub-topic)

Start by creating a topic in Google Pub/Sub following the [Google Pub/Sub documentation](https://cloud.google.com/pubsub/docs/admin#create_a_topic).

### 2. Create a push subscription (https://www.tinybird.co/docs/forward/guides/ingest-from-google-pubsub#2-create-a-push-subscription)

Next, [create a Push subscription in Pub/Sub](https://cloud.google.com/pubsub/docs/create-push-subscription).

Set the **Delivery Type** to **Push**.

In the **Endpoint URL** field, ue the following snippet (which uses the [Tinybird Events API](/docs/forward/ingest-data/events-api)) and pass your own Token, which you can find in your workspace > Tokens:

Endpoint URL
```
https://<your_host>/v0/events?wait=true&name=<Data Source name>&token=<Static token>
```

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

If you are sending a single-line JSON payload through Pub/Sub, select the **Enable payload unwrapping** option. This means that data isn't base64 encoded before sending it to Tinybird. If you are sending any other format through Pub/Sub, leave this option unselected and follow the decoding steps at the end of this guide.

Set **Retry policy** to **Retry after exponential backoff delay**. Set the **Minimum backoff** to **1** and **Maximum backoff** to **60**.

You don't need to create the Data Source in advance. Tinybird creates it automatically. This snippet also includes the `wait=true` parameter, which is explained in the [Events API docs](/docs/forward/ingest-data/events-api#wait-for-acknowledgement).

### 3. Send sample messages (https://www.tinybird.co/docs/forward/guides/ingest-from-google-pubsub#3-send-sample-messages)

Generate and send some sample messages to test your connection. If you don't have your own messages to test, use [this script](https://gist.github.com/alejandromav/dec8e092ef62d879e6821da06f6459c2).

### 4. Check the data source (https://www.tinybird.co/docs/forward/guides/ingest-from-google-pubsub#4-check-the-data-source)

Pub/Sub starts pushing data to Tinybird. Check the Tinybird UI to confirm that the Data Source has been created and events are arriving.

### Optional: Decode the payload (https://www.tinybird.co/docs/forward/guides/ingest-from-google-pubsub#optional-decode-the-payload)

If you selected the **Enable payload unwrapping** option, there is nothing else to do.

However, if you aren't sending single-line JSON payloads (NDJSON, JOSNL) through Pub/Sub, continue to base64 encode data before sending it to Tinybird. When the data arrives in Tinybird, decode it using the `base64Decode` function:

```
SELECT
    message_message_id as message_id,
    message_publish_time,
    base64Decode(message_data) as message_data
  FROM events_demo
```

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-gitlab
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send GitLab Events to Tinybird \| Tinybird"
theme-color: "#171612"
description: "Learn how to send GitLab events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---

# Send GitLab events to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-gitlab#send-gitlab-events-to-tinybird)

[GitLab](https://gitlab.com/) is a platform for building and deploying web applications. By integrating GitLab with Tinybird, you can analyze your GitLab events in real time and enrich it with other data sources.

Some common use cases for sending GitLab events to Tinybird include:

1. Analyze GitLab issues and merge requests.
2. Analyze GitLab push events.
3. Analyze and monitor GitLab pipeline.
4. Analyze custom DORA metrics.
Use this data to build a more complete picture of your GitLab events and improve your DevOps processes.

Read on to learn how to send events from GitLab to Tinybird.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-gitlab#before-you-start)

Before you connect GitLab to Tinybird, ensure:

- You have a GitLab account.
- You have a Tinybird workspace.

## Connect GitLab to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-gitlab#connect-gitlab-to-tinybird)

1. In GitLab, go to **Settings** > **Webhooks**.
2. Select **Add new webhook**.
3. Webhooks payloads vary depending on the event type. You can check here the list of [GitLab events](https://docs.gitlab.com/ee/user/project/integrations/webhook_events.html).
Select **Issues Events**.

1. In your Tinybird project, create a data source called `gitlab`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/gitlab.datasource):

```
SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.object_kind` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from GitLab in a single column. You can later retrieve various events and their metadata as needed in your pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).

1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in GitLab, paste the Events API URL in your Webhook URL. Use the query parameter `name` to match the name of the data source you created in Tinybird.

```
https://<your_host>/v0/events?name=gitlab
```

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

1. Select **Add custom header** and add 'Authorization' as **Header name** and paste the token you created in Tinybird as **Header value**.

```
Bearer <your user token>
```

1. You're done. You can select **Test** to check if the webhook is working.
Check the status of the integration from the **Log** tab in the Tinybird `gitlab` data source.

## See also (https://www.tinybird.co/docs/forward/guides/ingest-from-gitlab#see-also)

- [GitLab Webhooks](https://docs.gitlab.com/ee/user/project/integrations/webhook_events.html)
- [Tinybird data sources](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-github
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send GitHub Events to Tinybird \| Tinybird"
theme-color: "#171612"
description: "Learn how to send GitHub events to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---

# Send GitHub events to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-github#send-github-events-to-tinybird)

[GitHub](https://github.com/) is a platform for building and deploying web applications. By integrating GitHub with Tinybird, you can analyze your GitHub events in real time and enrich it with other data sources.

Some common use cases for sending GitHub events to Tinybird include:

1. Analyze GitHub issues and pull requests.
2. Analyze GitHub push events.
3. Analyze and monitor GitHub pipeline.
4. Analyze custom DORA metrics.
Use this data to build a more complete picture of your GitHub events and improve your DevOps processes.

Read on to learn how to send events from GitHub to Tinybird.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-github#before-you-start)

Before you connect GitHub to Tinybird, ensure:

- You have a GitHub account.
- You have a Tinybird workspace.

## Connect GitHub to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-github#connect-github-to-tinybird)

GitHub provides more than 70 webhooks that you can use to send events to Tinybird at Organization, repository, or app level.

This guide covers the base case for sending GitHub events from a repository to Tinybird.

1. In GitHub, go to your repository **Settings** > **Webhooks**.
2. Select **Add webhook**.
3. Webhooks payloads vary depending on the event type. You can check here the list of [GitHub events](https://docs.github.com/en/webhooks/webhook-events-and-payloads).
Select **Send me everything** to configure GitHub to send all events.

1. In your Tinybird project, create a data source called `github`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/github.datasource):

```
SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.type` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from GitHub in a single column. You can later retrieve various events and their metadata as needed in your pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).

1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in GitHub, paste the Events API URL in your Webhook URL. Use the query parameter `name` to match the name of the data source you created in Tinybird.

```
https://<your_host>/v0/events?name=github&token=<your user token>
```

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

1. Select **application/json** as the content type.
2. You're done.
Check the status of the integration from the `Recent deliveries` in the GitHub webhooks panel or from the **Log** tab in the Tinybird `github` data source.

## See also (https://www.tinybird.co/docs/forward/guides/ingest-from-github#see-also)

- [GitHub Webhook events and payloads](https://docs.github.com/en/webhooks/webhook-events-and-payloads)
- [GitHub Webhooks](https://docs.github.com/en/webhooks/using-webhooks/creating-webhooks)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-dynamodb-single-table-design
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Working with DynamoDB Single-Table Design \| Tinybird"
theme-color: "#171612"
description: "In this guide, you'll learn how to work with data that follows DynamoDB Single-Table Design."
inkeep:version: "forward"
---

# Work with DynamoDB single-table design (https://www.tinybird.co/docs/forward/guides/ingest-from-dynamodb-single-table-design#work-with-dynamodb-single-table-design)

Single-Table Design is a common pattern [recommended by AWS](https://aws.amazon.com/blogs/compute/creating-a-single-table-design-with-amazon-dynamodb/) in which different table schemas are stored in the same table. Single-table design makes it easier to support many-to-many relationships and avoid the need for JOINs, which DynamoDB doesn't support.

Single-Table Design is a good pattern for DynamoDB, but it's not optimal for analytics. To achieve higher performance in Tinybird, normalize data from DynamoDB into multiple tables that support the access patterns of your analytical queries.

The normalization process is achieved entirely within Tinybird by ingesting the raw DynamoDB data into a landing data source and then creating materialized views to extract items into separate tables.

This guide assumes you're familiar with DynamoDB, Tinybird, creating DynamoDB data sources in Tinybird, and materialized views.

## Example DynamoDB table (https://www.tinybird.co/docs/forward/guides/ingest-from-dynamodb-single-table-design#example-dynamodb-table)

For example, if Tinybird metadata were stored in DynamoDB using Single-Table Design, the table might look like this:

- **Partition Key**: `Org#Org_name`, example values: **Org#AWS** or **Org#Tinybird**.
- **Sort Key**: `Item_type#Id`, example values: **USER#1** or **WS#2**.
- **Attributes**: the information stored for each kind of item, like user email or workspace cores.

## Create the DynamoDB data source (https://www.tinybird.co/docs/forward/guides/ingest-from-dynamodb-single-table-design#create-the-dynamodb-data-source)

Use the DynamoDB Connector to ingest your DynamoDB table into a data source. The DynamoDB Connector is only documented for Classic. For new Forward projects, prefer the [S3 connector](/docs/forward/ingest-data/connectors/s3), [Kafka connector](/docs/forward/ingest-data/connectors/kafka), or another ingestion path that matches how you export or stream DynamoDB changes.

Rather than defining all columns in this landing data source, set only the Partition Key (PK) and Sort Key (SK) columns. The rest of the attributes are stored in the `_record` column as JSON. You don't need to define the `_record` column in the schema, as it's created automatically.

```
SCHEMA >
   `PK` String `json:$.Org#Org_name`,
   `SK` String `json:$.Item_type#Id`

ENGINE "ReplacingMergeTree"
ENGINE_SORTING_KEY "PK, SK"
ENGINE_VER "_timestamp"
ENGINE_IS_DELETED "_is_deleted"

IMPORT_SERVICE 'dynamodb'
IMPORT_CONNECTION_NAME <your_connection_name>
IMPORT_TABLE_ARN <your_table_arn>
IMPORT_EXPORT_BUCKET <your_dynamodb_export_bucket>
```
The following image shows how data looks. The DynamoDB Connector creates some additional rows, such as `_timestamp`, that aren't in the .datasource file:

<!-- <figure> -->
![DynamoDB Table storing users and worskpaces information](/docs/_next/image?url=%252Fdocs%252Fimg%252Fguides-ddb-std-2.png&w=3840&q=75)

<!-- <figcaption> -->
DynamoDB Table storing users and worskpaces information
<!-- </figcaption> -->

<!-- </figure> -->

## Use a pipe to filter and extract items (https://www.tinybird.co/docs/forward/guides/ingest-from-dynamodb-single-table-design#use-a-pipe-to-filter-and-extract-items)

Data is now be available in your landing data source. However, you need to use the `JSONExtract` function to access attributes from the `_record` column. To optimize performance, use Materialized Views to extract and store item types in separate data sources with their own schemas.

Create a pipe, use the PK and SK columns as needed to filter for a particular item type, and parse the attributes from the JSON in `_record` column.

The example table has User and workspace items, requiring a total of two materialized views, one for each item type.

<!-- <figure> -->
![Workspace Data Flow showing std connection, landing DS and users and workspaces materialized views](/docs/_next/image?url=%252Fdocs%252Fimg%252Fguides-ddb-std-4.png&w=3840&q=75)

<!-- <figcaption> -->
Two materialized views from landing DS
<!-- </figcaption> -->

<!-- </figure> -->
To extract the workspace items, the pipe uses the SK to filter for workspace items, and parses the attributes from the JSON in `_record` column. For example:

```
SELECT
  toLowCardinality(splitByChar('#', PK)[2]) org,
  toUInt32(splitByChar('#', SK)[2]) workspace_id,
  JSONExtractString(_record,'ws_name') ws_name,
  toUInt16(JSONExtractUInt(_record,'cores')) cores,
  JSONExtractUInt(_record,'storage_tb') storage_tb,
  _record,
  _old_record,
  _timestamp,
  _is_deleted
FROM dynamodb_ds_std
WHERE splitByChar('#', SK)[1] = 'WS'
```

## Create the materialized views (https://www.tinybird.co/docs/forward/guides/ingest-from-dynamodb-single-table-design#create-the-materialized-views)

Create a materialized view from the pipe to store the extracted data in a new data source.

The materialized view must use the ReplacingMergeTree engine to handle the deduplication of rows, supporting updates and deletes from DynamoDB. Use the following engine settings and configure them as needed for your table:

- `ENGINE "ReplacingMergeTree"`: the ReplacingMergeTree engine is used to deduplicate rows.
- `ENGINE_SORTING_KEY "key1, key2"`: the columns used to identify unique items, can be one or more columns, typically the part of the PK and SK that isn't idetifying Item type.
- `ENGINE_VER "_timestamp"`: the column used to identify the most recent row for each key.
- `ENGINE_IS_DELETED "_is_deleted"`: the column used to identify if a row has been deleted.
For example, the materialized view for the workspace items uses the following schema and engine settings:

```
SCHEMA >
    `org` LowCardinality(String),
    `workspace_id` UInt32,
    `ws_name` String,
    `cores` UInt16,
    `storage_tb` UInt64,
    `_record` String,
    `_old_record` Nullable(String),
    `_timestamp` DateTime64(3),
    `_is_deleted` UInt8

ENGINE "ReplacingMergeTree"
ENGINE_SORTING_KEY "org, workspace_id"
ENGINE_VER "_timestamp"
ENGINE_IS_DELETED "_is_deleted"
```
Repeat the same process for each item type.

<!-- <figure> -->
![Materialized View for extracting Users attributes](/docs/_next/image?url=%252Fdocs%252Fimg%252Fguides-ddb-std-3.png&w=3840&q=75)

<!-- <figcaption> -->
Materialized View for extracting Users attributes
<!-- </figcaption> -->

<!-- </figure> -->
You have now your data sources with the extracted columns ready to be queried.

## Review performance gains (https://www.tinybird.co/docs/forward/guides/ingest-from-dynamodb-single-table-design#review-performance-gains)

This process offers significant performance gains over querying the landing data source. To demonstrate this, you can use a Playground to compare the performance of querying the raw data vs the extracted data.

For the example table, the following queries aggregate the total number of users, workspaces, cores, and storage per organization using the unoptimized raw data and the optimized extracted data. The query over raw data took 335 ms, while the query over the extracted data took 144 ms, for a 2.3x improvement.

```
NODE users_stats
SQL >
    SELECT org, count() total_users
    FROM ddb_users_mv FINAL
    GROUP BY org

NODE ws_stats
SQL >
    SELECT org, count() total_workspaces, sum(cores) total_cores, sum(storage_tb) total_storage_tb
    FROM ddb_workspaces_mv FINAL
    GROUP BY org

NODE users_stats_raw
SQL >
    SELECT
      toLowCardinality(splitByChar('#', PK)[2]) org,
      count() total_users
    FROM dynamodb_ds_std FINAL
    WHERE splitByChar('#', SK)[1] = 'USER'
    GROUP BY org

NODE ws_stats_raw
SQL >
    SELECT
      toLowCardinality(splitByChar('#', PK)[2]) org,
      count() total_ws,
      sum(toUInt16(JSONExtractUInt(_record,'cores'))) total_cores,
      sum(JSONExtractUInt(_record,'storage_tb')) total_storage_tb
    FROM dynamodb_ds_std FINAL
    WHERE splitByChar('#', SK)[1] = 'WS'
    GROUP BY org

NODE org_stats
SQL >
    SELECT * FROM users_stats JOIN ws_stats using org

NODE org_stats_raw
SQL >
    SELECT * FROM users_stats_raw JOIN ws_stats_raw using org
```
This is how the outcome looks in Tinybird:

<!-- <figure> -->
![Comparison of same query](/docs/_next/image?url=%252Fdocs%252Fimg%252Fguides-ddb-std-5.png&w=3840&q=75)

<!-- <figcaption> -->
Same info, faster and more efficient from materialized views
<!-- </figcaption> -->

<!-- </figure> -->

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-dub
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send Dub webhooks to Tinybird \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to send data from Dub to Tinybird."
inkeep:version: "forward"
---

# Send Dub webhooks to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-dub#send-dub-webhooks-to-tinybird)

With [Dub](https://dub.co/), you can shorten any link and get powerful [conversion analytics](https://dub.co/analytics). By integrating Dub with Tinybird, you can analyze your events and usage data in real time.

Some common use cases for sending Dub webhooks to Tinybird include:

1. Tracking link clicks.
2. Monitoring link performance.
3. Analyzing user engagement patterns.
4. Creating custom dashboards for link performance.
5. Enriching other data sources with real-time link metrics.
Read on to learn how to send data from Dub to Tinybird.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-dub#before-you-start)

Before you connect Dub to Tinybird, ensure:

- You have a Dub account.
- You have a Tinybird workspace.

## Connect Dub to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-dub#connect-dub-to-tinybird)

1. Open the Dub UI and go to the **Settings** > **Webhooks** page.
2. Select **Create Webhook**.
3. In your Tinybird project, create a data source called `dub`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/dub.datasource):

```
SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.event` DEFAULT 'unknown',
  `event` JSON(max_dynamic_types=2, max_dynamic_paths=16) `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Dub in a single column. You can later retrieve various events and their metadata as needed in your pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).

1. From Tinybird Cloud, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in Dub, paste the Events API URL as your webhook URL. Use the query parameter `name` to match the name of the data source you created in Tinybird. For example:

```
https://<your_host>/v0/events?name=dub&token=<your user token>
```

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

1. Select the checkboxes for the Dub events you want to send to Tinybird, and select **Create webhook**.
2. You're done. Dub now pushes events to Tinybird through the [Events API](/docs/forward/ingest-data/events-api).

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-csv-files
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingest CSV files \| Tinybird"
theme-color: "#171612"
description: "In this guide, you'll learn how to ingest data into Tinybird using CSV (comma-separated values) files."
inkeep:version: "forward"
---

# Ingest CSV files (https://www.tinybird.co/docs/forward/guides/ingest-from-csv-files#ingest-csv-files)

CSV (comma-separated values) is one of the most widely used formats out there. However, it's used in different ways; some people don't use commas, and other people use escape values differently, or are unsure about using headers.

The Tinybird platform handles many scenarios. If your data doesn't comply with format and syntax best practices, Tinybird still attempts to understand and ingest your file. Following certain best practices can speed CSV processing by up to 10x.

## Syntax best practices (https://www.tinybird.co/docs/forward/guides/ingest-from-csv-files#syntax-best-practices)

By default, Tinybird processes your CSV file assuming the file follows the most common standard ([RFC4180](https://datatracker.ietf.org/doc/html/rfc4180#section-2)). Key points:

- Separate values with commas.
- Each record is a line (with CRLF as the line break). The last line may or may not have a line break.
- First line as a header is optional (though not using one is faster in Tinybird.)
- Double quotes are optional but using them means you can escape values (for example, if your content has commas or line breaks).
Example: Instead of using the backslash `\` as an escape character, like this:

```
1234567890,0,0,0,0,2021-01-01 10:00:00,"{\"authorId\":\"123456\",\"handle\":\"aaa\"}"
```
Use two double quotes:

More performant
```
1234567890,0,0,0,0,2021-01-01 10:00:00,"{""authorId"":""123456"",""handle"":""aaa""}"
```

- Fields containing line breaks, double quotes, and commas should be enclosed in double quotes.
- Double quotes can also be escaped by using another double quote (""aaa"",""b""""bb"",""ccc"")
In addition to the previous points, it's also recommended to:

1. Format `DateTime` columns as `YYYY-MM-DD HH:MM:SS` and `Date` columns as `YYYY-MM-DD`.
2. Send the encoding in the `charset` part of the `content-type` header, if it's different to UTF-8. The expectation is UTF-8, so it should look like this `Content-Type: text/html; charset=utf-8`.
3. You can set values as `null` in different ways, for example, *""[]""*, *""""* (empty space), *N* and *"N"*.
4. If you use a delimiter other than a comma, explicitly define it with the API parameter *``dialect_delimiter``.*
5. If you use an escape character other than a ", explicitly define it with the API parameter *``dialect_escapechar``.*
6. If you have no option but to use a different line break character, explicitly define it with the API parameter `dialect_new_line`.
For more information, check the [Data Sources API docs](/docs/api-reference/datasource-api).

## Append data (https://www.tinybird.co/docs/forward/guides/ingest-from-csv-files#append-data)

Once the Data Source schema has been created, you can optimize performance by omitting the header. Keep the data in the same order.

However, you can still ingest data if the header contains all the names present in the Data Source schema, even when the columns follow a different order from the initial creation.

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-clerk
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send Clerk webhooks to Tinybird \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to send data from Clerk to Tinybird."
inkeep:version: "forward"
---

# Send Clerk webhooks to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-clerk#send-clerk-webhooks-to-tinybird)

[Clerk](https://clerk.com/) is a developer-focused user management platform to handle user authentication with many prebuilt UI components. By integrating Clerk with Tinybird, you can analyze your user authentication data in real time and enrich it with other data sources.

Some common use cases for sending Clerk webhooks to Tinybird include:

1. Tracking net user and organization growth.
2. Monitoring user churn.
3. Identifying common auth errors.
4. Creating custom dashboards for auth analysis.
5. Enriching other data sources with real-time auth metrics.
Read on to learn how to send data from Clerk to Tinybird.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-from-clerk#before-you-start)

Before you connect Clerk to Tinybird, ensure:

- You have a Clerk account.
- You have a Tinybird workspace.

## Connect Clerk to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-clerk#connect-clerk-to-tinybird)

1. From the Clerk UI, select **Configure** > **Webhooks**.
2. Select **Add Endpoint**.
3. In your Tinybird project, create a data source called `clerk`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/clerk.datasource):

```
SCHEMA >
    `event_time` DateTime64(3) `json:$.tinybirdIngestTime` DEFAULT now(),
    `event_type` String `json:$.type` DEFAULT 'unknown',
    `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Clerk in a single column. You can later retrieve various events and their metadata as needed in your pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).

1. Back in Clerk, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name` to match the name of the data source you created in Tinybird, for example:

```
https://<your_host>/v0/events?name=clerk
```

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

1. From Tinybird Cloud, copy a token with privileges to write to the data source you created. You can use the admin token or create one with the required scope.
2. Return to the Clerk Webhooks page, and update the URL to add a new search parameter `token` with the token you copied. The final URL looks like the following:

```
https://<your_host>/v0/events?name=clerk&token=p.eyXXXXX
```

1. Select the checkboxes for the Clerk events you want to send to Tinybird, and select **Create**.
2. You're done. Any of the Clerk events you selected is automatically sent to Tinybird through the [Events API](/docs/forward/ingest-data/events-api). You can test the integration from the **Testing** tab in the Clerk Webhooks UI.

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingest BigQuery Data via Google Cloud Storage \| Tinybird"
theme-color: "#171612"
description: "Learn how to send data from BigQuery to Tinybird using Google Cloud Storage."
inkeep:version: "forward"
---

# Ingest data from BigQuery using Google Cloud Storage (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#ingest-data-from-bigquery-using-google-cloud-storage)

Read on to learn how to send data from BigQuery to Tinybird, for example when you need to periodically run full replaces of a table or do a one-off ingest.

This process relies on [BigQuery's exporting capabilities](https://cloud.google.com/bigquery/docs/exporting-data#sql), or bulk exporting data as CSV, NDJSON, or Parquet files and then ingesting them using the [GCS Connector](/docs/forward/ingest-data/connectors/gcs).

## Prerequisites (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#prerequisites)

You need a Tinybird account and access to BigQuery and Google Cloud Storage.

1
## Export the BigQuery table (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#export-the-bigquery-table)

The first step consists in exporting the BigQuery table, or query result set, to a GCS bucket.

### 1. Grant the required permissions in Google Cloud Console (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#1-grant-the-required-permissions-in-google-cloud-console)

Make sure the GCS bucket allows BigQuery to write files. The service account used by BigQuery needs the `roles/storage.objectCreator` role on the bucket. You can grant this permission in the Google Cloud Console:

1. Navigate to your GCS bucket in the Cloud Console
2. Go to the **Permissions** tab
3. Select **Grant Access**
4. Add the BigQuery service account (format: `service-<project-number>@gcp-sa-bigquery.iam.gserviceaccount.com`) with the **Storage Object Creator** role

### 2. Export the data (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#2-export-the-data)

Run the following SQL statement to export the data:

```
EXPORT DATA OPTIONS(
  uri = 'gs://your-bucket-name/path/orders/*.csv',
  format = 'CSV',
  compression = 'GZIP',
  overwrite = true,
  header = true,
  field_delimiter = ','
) AS (
  SELECT
    order_id,
    customer_id,
    order_status,
    total_price,
    order_date,
    order_priority,
    clerk
  FROM `your-project.your_dataset.orders`
  ORDER BY order_date
);
```
Replace `your-bucket-name`, `path`, and the table reference with your actual values.

**CSV format limitations**: CSV does not support nested or repeated fields (arrays, structs). If your BigQuery table contains arrays or nested data types, you must use JSON or Parquet format instead. Attempting to export arrays or nested structures to CSV results in an error.

### 3. Export arrays and nested data (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#3-export-arrays-and-nested-data)

For tables with arrays or nested structures, use JSON format:

```
EXPORT DATA OPTIONS(
  uri = 'gs://your-bucket-name/path/orders/*.json',
  format = 'JSON',
  compression = 'GZIP',
  overwrite = true
) AS (
  SELECT
    order_id,
    customer_id,
    items,  -- Array field
    shipping_address,  -- Nested struct
    order_date
  FROM `your-project.your_dataset.orders`
  ORDER BY order_date
);
```
**JSON format considerations**: When exporting to JSON format, BigQuery encodes `INT64` values as JSON strings to maintain precision. You may need to handle this in your Tinybird schema by parsing these strings as integers.

### 4. Export using Parquet format (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#4-export-using-parquet-format)

Parquet format also supports nested and repeated data, and can be more efficient for large datasets:

```
EXPORT DATA OPTIONS(
  uri = 'gs://your-bucket-name/path/orders/*.parquet',
  format = 'PARQUET',
  compression = 'GZIP',
  overwrite = true
) AS (
  SELECT
    order_id,
    customer_id,
    items,
    shipping_address,
    order_date
  FROM `your-project.your_dataset.orders`
  ORDER BY order_date
);
```

### 5. Automate the export (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#5-automate-the-export)

To automate the export, you can create a BigQuery Scheduled Query:

1. In the BigQuery Console, select **Scheduled queries**
2. Select **Create scheduled query**
3. Enter your `EXPORT DATA` statement
4. Set the schedule (for example, daily or hourly)
5. Configure the destination dataset and table if needed
Follow the [Google Cloud documentation for Scheduled Queries](https://cloud.google.com/bigquery/docs/scheduling-queries) for more details.

2
## Ingest data into Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#ingest-data-into-tinybird)

Before ingesting your BigQuery data from the GCS bucket, you need to create the GCS connection. For more details and advanced use cases, see the [GCS Connector](/docs/forward/ingest-data/connectors/gcs) documentation.

### Supported file formats (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#supported-file-formats)

The GCS Connector supports the following formats exported from BigQuery:

| File Type | Accepted Extensions | Supported Compression | Notes |
| --- | --- | --- | --- |
| CSV | `.csv`, `.csv.gz` | `gzip` | Does not support arrays or nested data. Use for simple, flat schemas. |
| NDJSON | `.ndjson`, `.ndjson.gz`, `.jsonl`, `.jsonl.gz` | `gzip` | Supports arrays and nested data. Each line must be a valid JSON object. |
| Parquet | `.parquet`, `.parquet.gz` | `snappy`, `gzip`, `lzo`, `brotli`, `lz4`, `zstd` | Supports arrays and nested data. More efficient for large datasets. |
**NDJSON format**: JSON files must follow the **Newline Delimited JSON (NDJSON)** format. Each line must be a valid JSON object and must end with a `\n` character. BigQuery's `EXPORT DATA` with `format='JSON'` produces NDJSON format.

### Create the data source (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#create-the-data-source)

You can use the following schema for the data source:

```
SCHEMA >
    `order_id` Int64,
    `customer_id` Int64,
    `order_status` String,
    `total_price` Float32,
    `order_date` DateTime,
    `order_priority` String,
    `clerk` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(order_date)"
ENGINE_SORTING_KEY "order_date, order_priority"

IMPORT_CONNECTION_NAME 'tb-gcs'
IMPORT_BUCKET_URI 'gs://your-bucket-name/path/orders/*.csv.gz'
IMPORT_SCHEDULE '@on-demand'
```
For JSON exports with arrays, handle the nested structure:

```
SCHEMA >
    `order_id` Int64 `json:$.order_id`,
    `customer_id` Int64 `json:$.customer_id`,
    `items` Array(String) `json:$.items[:]`,
    `shipping_address` String `json:$.shipping_address`,
    `order_date` DateTime `json:$.order_date`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(order_date)"
ENGINE_SORTING_KEY "order_date"

IMPORT_CONNECTION_NAME 'tb-gcs'
IMPORT_BUCKET_URI 'gs://your-bucket-name/path/orders/*.json.gz'
IMPORT_SCHEDULE '@on-demand'
```
**Handling BigQuery arrays and nested data**: When exporting arrays or nested structures from BigQuery to JSON, they are stored as JSON strings in Tinybird. You can parse them in your Pipes using JSON functions like `JSONExtractArrayRaw()` or `JSONExtractString()` to work with the nested data.

Deploy the Data Source. Since the GCS Connector uses `@on-demand` mode, manually sync data when new files are available.

### Sync data (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#sync-data)

After BigQuery exports new files to GCS, trigger a sync in Tinybird:

#### Using the API (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#using-the-api)

```
curl -X POST "https://<your_host>/v0/datasources/<datasource_name>/scheduling/runs" \
  -H "Authorization: Bearer <your-tinybird-token>"
```

#### Using the CLI (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#using-the-cli)

```
tb datasource sync <datasource_name>
```
Each file is appended to the Tinybird data source. As records might be duplicated if BigQuery re-exports the same data, consider using a materialized view to consolidate a stateful set of your BigQuery table.

## Data type considerations (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#data-type-considerations)

When exporting from BigQuery to Tinybird, be aware of these data type quirks:

**INT64 in JSON**: BigQuery exports `INT64` values as JSON strings in JSON format to preserve precision. In your Tinybird schema, you may need to parse these strings. For example, if BigQuery exports `"1234567890123456789"` as a string, use `toInt64(column_name)` in your Pipes.

## Limits (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#limits)

Because you're using the GCS Connector, its [limits](/docs/forward/ingest-data/connectors/gcs#limitations) apply.

The GCS Connector has its own limitations. To stay within your Tinybird plan's limits, you might need to [limit the size of the exported files](https://docs.cloud.google.com/bigquery/docs/exporting-data#limit_the_exported_file_size). BigQuery automatically splits large exports into multiple files of up to 1 GB when you use a wildcard in the URI, for example, `gs://bucket/path/*.csv`.

## Next steps (https://www.tinybird.co/docs/forward/guides/ingest-from-bigquery-using-google-cloud-storage#next-steps)

See the following resources:

- [GCS Connector](/docs/forward/ingest-data/connectors/gcs)
- [Ingest from Snowflake using AWS S3](./ingest-from-snowflake-using-aws-s3)
- [Ingest from Snowflake using Azure Blob Storage](./ingest-from-snowflake-using-azure-blob-storage)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-from-aws-kinesis
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Stream from AWS Kinesis \| Tinybird"
theme-color: "#171612"
description: "In this guide, you'll learn how to send data from AWS Kinesis to Tinybird."
inkeep:version: "forward"
---

# Stream from AWS Kinesis (https://www.tinybird.co/docs/forward/guides/ingest-from-aws-kinesis#stream-from-aws-kinesis)

This guide shows you how to send data from AWS Kinesis to Tinybird.

If you have a [Kinesis Data Stream](https://aws.amazon.com/kinesis/data-streams/) that you want to send to Tinybird, use [Kinesis Firehose](https://aws.amazon.com/kinesis/data-firehose/) to integrate it with Tinybird.

## 1. Push messages from Kinesis to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-aws-kinesis#1-push-messages-from-kinesis-to-tinybird)

### Create a token with the right scope (https://www.tinybird.co/docs/forward/guides/ingest-from-aws-kinesis#create-a-token-with-the-right-scope)

In your workspace, create a Token with the `Create new data sources or append data to existing ones` scope.

### Create a new data stream (https://www.tinybird.co/docs/forward/guides/ingest-from-aws-kinesis#create-a-new-data-stream)

Start by creating a new data stream in AWS Kinesis. See the [AWS documentation](https://docs.aws.amazon.com/streams/latest/dev/working-with-streams.html) for more information.

<!-- <figure> -->
![](/docs/_next/image?url=%252Fdocs%252Fimg%252Fingest-from-aws-kinesis-2.png&w=3840&q=75)

<!-- <figcaption> -->
Create a Kinesis Data Stream
<!-- </figcaption> -->

<!-- </figure> -->

### Create a Firehose delivery stream (https://www.tinybird.co/docs/forward/guides/ingest-from-aws-kinesis#create-a-firehose-delivery-stream)

Next, [create a Kinesis Data Firehose delivery stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html).

Set the **Source** to **Amazon Kinesis Data Streams** and the **Destination** to **HTTP Endpoint**.

In the **Destination Settings**, set **HTTP Endpoint URL** to point to the [Tinybird Events API](/docs/forward/ingest-data/events-api).

```
https://<your_host>/v0/events?name=<your_datasource_name>&wait=true&token=<your_token_with_DS_rights>
```
This example is for workspaces in the `GCP` --> `europe-west3` region. If necessary, replace with the [correct region for your workspace](/docs/api-reference#regions-and-endpoints). Additionally, note the `wait=true` parameter. Learn more about it [in the Events API docs](/docs/forward/ingest-data/events-api#wait-for-acknowledgement).

You don't need to create the Data Source in advance. Tinybird creates it automatically.

### Send sample messages and check that they arrive to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-from-aws-kinesis#send-sample-messages-and-check-that-they-arrive-to-tinybird)

If you don't have an active data stream, follow [this python script](https://gist.github.com/GnzJgo/f1a80186a301cd8770a946d02343bafd) to generate dummy data.

Back in Tinybird, you should see 3 columns filled with data in your data source. `timestamp` and `requestId` are self explanatory, and your messages are in `records\_\data`:

<!-- <figure> -->
![](/docs/_next/image?url=%252Fdocs%252Fimg%252Fingest-from-aws-kinesis-3.png&w=3840&q=75)

<!-- <figcaption> -->
Firehose data source
<!-- </figcaption> -->

<!-- </figure> -->

## 2. Decode message data (https://www.tinybird.co/docs/forward/guides/ingest-from-aws-kinesis#2-decode-message-data)

### Decode message data (https://www.tinybird.co/docs/forward/guides/ingest-from-aws-kinesis#decode-message-data)

The `records\_\data` column contains an array of encoded messages.

To get one row per array element, use the ARRAY JOIN clause. Also decode the messages with the base64Decode() function.

Now that the raw JSON is in a column, you can use [JSONExtract functions](/docs/sql-reference/functions/json-functions) to extract the desired fields:

Decoding messages
```
NODE decode_messages
SQL >
   SELECT
       base64Decode(encoded_m) message,
       fromUnixTimestamp64Milli(timestamp) kinesis_ts
   FROM firehose
   ARRAY JOIN records__data as encoded_m

NODE extract_message_fields
SQL >
   SELECT
       kinesis_ts,
       toDateTime64(JSONExtractString(message, 'datetime'), 3) datetime,
       JSONExtractString(message, 'event') event,
       JSONExtractString(message, 'product') product
   FROM decode_messages
```

<!-- <figure> -->
![](/docs/_next/image?url=%252Fdocs%252Fimg%252Fingest-from-aws-kinesis-4.png&w=3840&q=75)

<!-- <figcaption> -->
Decoding messages
<!-- </figcaption> -->

<!-- </figure> -->

## Recommended settings (https://www.tinybird.co/docs/forward/guides/ingest-from-aws-kinesis#recommended-settings)

When configuring AWS Kinesis as a data source, use the following settings:

- Set `wait=true` when calling the Events API. See [the Events API docs](/docs/forward/ingest-data/events-api#wait-for-acknowledgement) for more information.
- Set the buffer size lower than 10 Mb in Kinesis.
- Set 128 shards as the maximum in Kinesis.

## Performance optimizations (https://www.tinybird.co/docs/forward/guides/ingest-from-aws-kinesis#performance-optimizations)

Persist the decoded and unrolled result in a different data source. You can do it with a materialized view: A combination of a pipe and a data source that leaves the transformed data into the destination data source as soon as new data arrives to the Firehose data source.

Don't store what you don't need. In this example, some of the extra columns could be skipped. [Add a TTL](/docs/forward/dev-reference/datafiles/datasource-files) to the Firehose data source to prevent keeping more data than you need.

Another alternative is to create the Firehose data source with a Null Engine. This way, data ingested there can be transformed and fill the destination data source without being persisted in the data source with the Null Engine.

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-aws-elb-logs
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Ingest AWS ELB logs \| Tinybird"
theme-color: "#171612"
description: "Learn how to import AWS Elastic Load Balancer (ELB) logs into Tinybird using the Data Sources API."
inkeep:version: "forward"
---

# Ingest AWS ELB logs (https://www.tinybird.co/docs/forward/guides/ingest-aws-elb-logs#ingest-aws-elb-logs)

AWS Elastic Load Balancer (ELB) logs record detailed information about requests sent to your load balancers. These logs are useful for monitoring traffic, troubleshooting, and analyzing app performance.

## What are AWS ELB logs? (https://www.tinybird.co/docs/forward/guides/ingest-aws-elb-logs#what-are-aws-elb-logs)

AWS ELB logs capture information such as the time a request was received, the client's IP address, request paths, backend responses, and more. Each log entry is a single line in a space-delimited text format.

Here's an example of an ELB log entry:

Sample AWS ELB log entry
```
2023-06-01T12:00:00.000000Z my-elb 192.0.2.1:12345 203.0.113.1:80 0.000022 0.001048 0.00002 200 200 0 57 "GET http://www.example.com:80/ HTTP/1.1" "curl/7.46.0" - -
```
For a full description of each field, see the [AWS documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/access-log-collection.html#access-log-entry-format).

## File format (https://www.tinybird.co/docs/forward/guides/ingest-aws-elb-logs#file-format)

AWS ELB logs are typically exported to Amazon S3 as compressed files with the `.log.gz` extension. Each file contains multiple log entries, one per line.

- **Format**: Space-delimited text
- **Compression**: Gzip (`.gz`)

## Importing ELB logs into Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-aws-elb-logs#importing-elb-logs-into-tinybird)

You can import ELB logs into Tinybird using the [Data Sources API](/docs/api-reference/datasource-api) with `format=csv` and a custom delimiter. Although ELB logs are not comma-separated, you can specify the space character as the delimiter.

### Step 1: Prepare your Data Source schema (https://www.tinybird.co/docs/forward/guides/ingest-aws-elb-logs#step-1-prepare-your-data-source-schema)

Define a schema that matches the ELB log fields. For example:

ELB log Data Source schema
```
SCHEMA >
  `timestamp` DateTime,
  `elb` String,
  `client_port` String,
  `backend_port` String,
  `request_processing_time` Float32,
  `backend_processing_time` Float32,
  `response_processing_time` Float32,
  `elb_status_code` UInt16,
  `backend_status_code` UInt16,
  `received_bytes` UInt64,
  `sent_bytes` UInt64,
  `request` String,
  `user_agent` String,
  `ssl_cipher` String,
  `ssl_protocol` String
ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"
```
Adjust the schema to match your log format and needs.

### Step 2: Upload and ingest the logs (https://www.tinybird.co/docs/forward/guides/ingest-aws-elb-logs#step-2-upload-and-ingest-the-logs)

You can send your `.log.gz` files directly to Tinybird. Tinybird automatically decompresses and parses the file as CSV, as long as you set `format=csv` in the request. By default, Tinybird expects comma-separated values. Since ELB logs are space-delimited, set the delimiter explicitly with the `dialect_delimiter` parameter:

Ingest ELB logs with space delimiter
```
curl \
  -H "Authorization: Bearer <your_auth_token>" \
  -X POST "https://<your_host>/v0/datasources?name=logs&format=csv&dialect_delimiter=%20" \
  -F "csv=@file.log.gz"
```
If your ELB logs contain quoted fields with spaces, consider preprocessing the logs to use a different delimiter or to quote fields consistently.

## Next steps (https://www.tinybird.co/docs/forward/guides/ingest-aws-elb-logs#next-steps)

- [Data Sources API reference](/docs/api-reference/datasource-api)
- [AWS ELB access log documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/access-log-collection.html)

---

URL: https://www.tinybird.co/docs/forward/guides/ingest-auth0-logs
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Send Auth0 Log Streams to Tinybird \| Tinybird"
theme-color: "#171612"
description: "In this guide you'll learn how to send Auth0 Log Streams to Tinybird using webhooks and the Events API."
inkeep:version: "forward"
---

# Send Auth0 log streams to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-auth0-logs#send-auth0-log-streams-to-tinybird)

[Auth0](https://auth0.com/) is a developer-focused user management platform to handle user authentication with many prebuilt UI components. By integrating Auth0 with Tinybird, you can analyze your user authentication data in real time and enrich it with other data sources.

Some common use cases for sending Auth0 logs to Tinybird include:

1. Tracking net user and organization growth.
2. Monitoring user churn.
3. Identifying common auth errors.
4. Creating custom dashboards for auth analysis.
5. User authentication audit logs.
Read on to learn how to send data from Auth0 Logs Streams to Tinybird.

## Before you start (https://www.tinybird.co/docs/forward/guides/ingest-auth0-logs#before-you-start)

Before you connect Auth0 Logs Streams to Tinybird, ensure:

- You have an Auth0 account.
- You have a Tinybird workspace.

## Connect Auth0 to Tinybird (https://www.tinybird.co/docs/forward/guides/ingest-auth0-logs#connect-auth0-to-tinybird)

1. From the Auth0 dashboard, select **Monitoring** > **Streams**.
2. Select **Create Stream**.
3. In Tinybird, create a data source, called `auth0`. You can follow this [schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/auth0.datasource):

```
SCHEMA >
    `event_time` DateTime64(3) `json:$.tinybirdIngestTime` DEFAULT now(),
    `event_type` String `json:$.data.type` DEFAULT 'unknown',
    `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time"
```
Using the [JSON Data Type](/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Auth0 Logs Streams in a single column. You can later retrieve various events and their metadata as needed in your pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird in the [Community Slack](/docs/community) or through [Support](/docs/forward/support).

1. In Tinybird, copy a token with privileges to append to the data source you created. You can use the admin token or create one with the required scope.
2. Back in Auth0, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name` to match the name of the data source you created in Tinybird. For example:

```
https://<your_host>/v0/events?name=auth0&token=<your user token>
```

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

Content Type is `application/json` and Content Format is `JSON Lines`.

1. Select the any event category to filter, like `All`, and a date in case you want to perform some backfilling. Then select **Save**.
2. You're done. Any of the Auth0 Log Streams events you selected is automatically sent to Tinybird through the [Events API](/docs/forward/ingest-data/events-api).
You can check the status of the integration from the **Health** tab in the created webhook or from the **Log** tab in the Tinybird `auth0` data source.

## Auth0 Logs Explorer template (https://www.tinybird.co/docs/forward/guides/ingest-auth0-logs#auth0-logs-explorer-template)

Use the [Auth0 Logs Explorer Template](https://github.com/tinybirdco/auth0-logs-explorer-template) to bootstrap a multi-tenant, user-facing logs explorer for your Auth0 account. You can fork it and make it your own.

## See also (https://www.tinybird.co/docs/forward/guides/ingest-auth0-logs#see-also)

- [Events API](/docs/forward/ingest-data/events-api)
- [Auth0 Logs Streams](https://auth0.com/docs/customize/log-streams/custom-log-streams)

---

URL: https://www.tinybird.co/docs/forward/guides/gdpr-compliant-data-deletion
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "GDPR-compliant data deletion \| Tinybird"
theme-color: "#171612"
description: "Learn how to delete user data in Tinybird to maintain GDPR compliance using various methods and best practices."
inkeep:version: "forward"
---

# GDPR-compliant data deletion in Tinybird (https://www.tinybird.co/docs/forward/guides/gdpr-compliant-data-deletion#gdpr-compliant-data-deletion-in-tinybird)

Tinybird provides several methods for deleting user data to maintain GDPR compliance. The best approach depends on your data volume and deletion frequency.

## Methods for data deletion (https://www.tinybird.co/docs/forward/guides/gdpr-compliant-data-deletion#methods-for-data-deletion)

### Use the CLI (https://www.tinybird.co/docs/forward/guides/gdpr-compliant-data-deletion#use-the-cli)

For smaller datasets or infrequent deletions, you can use the <a href="/docs/forward/dev-reference/commands/tb-datasource">`tb datasource delete`</a> command:

```
tb datasource delete [datasource_name] --sql-condition "user_id = 'user_to_delete'"
```
This method is suitable for datasets with up to a few million rows and deletion frequencies of up to 1000 operations per week.

### Use the Delete API Endpoint (https://www.tinybird.co/docs/forward/guides/gdpr-compliant-data-deletion#use-the-delete-api-endpoint)

For more frequent deletions, you can use the [Delete API Endpoint](/docs/api-reference/datasource-api#post--v0-datasources-(.+)-delete):

```
POST /v0/datasources/(.+)/delete
```
This method is more efficient for regular deletion operations.

### Implement a deletion queue (https://www.tinybird.co/docs/forward/guides/gdpr-compliant-data-deletion#implement-a-deletion-queue)

For large-scale applications with frequent deletion requests:

- Create a `delete_user` table to queue deletion requests.
- Set up a scheduled job, such as a weekly job, to process the queue and delete user data from all relevant Data Sources.
- Use partitioning on large Data Sources to optimize deletion operations.

## Best practices (https://www.tinybird.co/docs/forward/guides/gdpr-compliant-data-deletion#best-practices)

- Ensure deletions cascade to all relevant Data Sources and Materialized Views.
- For Data Sources with millions of rows, consider partitioning to improve deletion performance.
- Monitor the performance impact of deletion operations and adjust your strategy as your data grows.
- Implement TTL on Data Sources where appropriate to automatically remove old data.

## Considerations (https://www.tinybird.co/docs/forward/guides/gdpr-compliant-data-deletion#considerations)

- Deleting data using `ALTER TABLE ... DELETE WHERE` can have performance implications on large datasets.
- Balance between immediate deletion for GDPR compliance and system performance.
- Regularly review and optimize your deletion strategy as your data volume grows.

---

URL: https://www.tinybird.co/docs/forward/guides/evolve-data-source
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Evolve data sources \| Tinybird"
theme-color: "#171612"
description: "Evolve your data sources in Tinybird."
inkeep:version: "forward"
---

# Evolve data sources (https://www.tinybird.co/docs/forward/guides/evolve-data-source#evolve-data-sources)

After you've deployed your project, you can evolve your data sources. For example, you might need to add a new column, change the data type of a column, or change the sorting key. Tinybird handles the data migration.

## Types of changes (https://www.tinybird.co/docs/forward/guides/evolve-data-source#types-of-changes)

You can evolve your data sources by editing one or more of the following:

- Landing Data Source schema.
- Landing Data Source engine settings.
- Materialized Data Source schema.
- Materialized Data Source engine settings.
- Materialized View query logic. See <a href="/docs/forward/core-concepts/materialized-views#altering-materialized-views">Altering Materialized Views</a>.

## How Tinybird evolves data sources (https://www.tinybird.co/docs/forward/guides/evolve-data-source#how-tinybird-evolves-data-sources)

Tinybird uses one of two methods to evolve a data source during deployment:

- Rewrite and backfill. Tinybird creates a new version of the Data Source with the updated schema or settings and backfills data from the live deployment using one of the following methods:
  - **Materialized View query**: Re-runs the Materialized View query that writes to the Data Source to repopulate it from the upstream Data Source.
  - **Forward query**: Uses a <a href="about:blank#forward-query">`FORWARD_QUERY`</a> instruction you provide to transform and repopulate the data.
- `ALTER`. Tinybird applies the change in place without rebuilding and repopulating the data source.
Tinybird automatically chooses the method based on the changes in your `.datasource` files.

### Automatic ALTER operations (https://www.tinybird.co/docs/forward/guides/evolve-data-source#automatic-alter-operations)

The currently supported automatic `ALTER` operations are:

- Adding a new column to a data source.
- Removing a column from a data source.
- Making a column nullable.
- Adding a data source TTL.
- Modifying a data source TTL.
- Removing a data source TTL.
- Adding an index to a data source.
- Dropping an index from a data source.
Data Source changes evolved with `ALTER` are applied only when the deployment is promoted to live. They aren't applied to the staging deployment, so those changes aren't visible while testing in staging.

## Landing data source schema (https://www.tinybird.co/docs/forward/guides/evolve-data-source#landing-data-source-schema)

When you make changes to the schema of a landing data source, such as adding or editing columns or changing a data type, you can follow these steps:

1. Check that Tinybird Local is running and your Workspace has the project deployed. If not, run `tb local start && tb deploy`.
2. Edit the Data Source definition to add the changes. In Tinybird CLI projects, update the `.datasource` file. In SDK projects, update the `defineDatasource` or `define_datasource` resource. See [SCHEMA instructions](/docs/forward/dev-reference/datafiles/datasource-files#schema), [TypeScript SDK resources](/docs/forward/dev-reference/typescript-sdk-resources#datasources), and [Python SDK resources](/docs/forward/dev-reference/python-sdk-resources#datasources).
3. For <a href="about:blank#automatic-alter-operations">automatic `ALTER` operations</a>, Tinybird applies the change automatically. For changes that require a rewrite and backfill, add a [forward query](https://www.tinybird.co/docs/forward/guides/evolve-data-source#forward-query) instruction to tell Tinybird how to migrate your data.
4. Run a deployment check to validate the deployment before creating it. This is a good way of catching potential breaking changes.
5. Deploy and promote your changes in Tinybird Cloud.
If the change requires a rewrite, Tinybird Cloud automatically populates the new table following the updated schema.

If a deployment fails, Tinybird automatically discards the staging deployment and maintains the live version.

If the change can be applied with `ALTER`, Tinybird uses that method instead of rewriting and backfilling.

- You don't need to add a `FORWARD_QUERY`.
- Tinybird doesn't rebuild and repopulate the data source.
- The changes become available only after promotion, not in staging.
If you add a `FORWARD_QUERY` to a data source that could otherwise be evolved with `ALTER`, Tinybird performs a full rewrite and backfill instead. This is useful when you want to validate the schema change in staging before promoting, since `ALTER` changes aren't applied in staging.

If a `FORWARD_QUERY` in your `.datasource` file can overwrite existing column values during a backfill, for example by using default values for columns that already contain data, Tinybird shows a warning. This can happen when a `FORWARD_QUERY` from a previous deployment is still present in the file. Remove the `FORWARD_QUERY` after promotion if it's no longer needed to avoid unexpected data overwrites in future deployments.

### Forward query (https://www.tinybird.co/docs/forward/guides/evolve-data-source#forward-query)

If you make changes to a .datasource file that are incompatible with the live version, Tinybird needs a forward query to transform data from the live schema to the new one.
For most changes, Tinybird auto-generates the forward query. Run `tb build` to generate the query and copy it into your `.datasource` file. Auto-generated forward queries handle the following changes:

- Add or remove columns.
- Change column type, nullable, or codec.
- Change column comment, default expression, or column-level TTL.
- Change sorting key, partition key, sampling key, or primary key.
- Change engine type, engine settings, or engine arguments.
- Add, remove, or modify indexes.
For changes where the auto-generated mapping can't express the transformation, you must provide an explicit `FORWARD_QUERY`. For example:

- **Incompatible type conversions**: Changing a `String` containing `"hello"` to `Int64`. Use a custom query to handle the transformation with functions like `coalesce`, `accurateCastOrDefault`, or a default value.
- **Column renames**: Tinybird sees a removed column and a new column. Without a `FORWARD_QUERY`, the old column's data is lost and the new column gets a default value.
- **Split or merge columns**: For example, splitting `full_name` into `first_name` and `last_name`.
- **Computed columns from existing data**: For example, a new `total` column that should be `price * quantity` instead of a default value.
The `FORWARD_QUERY` instruction is a `SELECT` query executed on the live data source. The query must include the column selection part of the query, for example `SELECT a, b, c` or `SELECT * except 'guid', toUUID(guid) AS guid`. The `FROM` and `WHERE` clauses aren't supported.

The following is an example of a forward query that changes the `session_id` column from a `String` to a `UUID` type:

- Tinybird CLI
- TypeScript SDK
- Python SDK
tinybird/datasources/forward-query.datasource - Data Source with a FORWARD_QUERY declaration
```
DESCRIPTION >
    Analytics events landing data source

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` UUID `json:$.session_id`,
    `action` String `json:$.action`,
    `version` String `json:$.version`,
    `payload` String `json:$.payload`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"
ENGINE_TTL "timestamp + toIntervalDay(60)"

FORWARD_QUERY >
    SELECT timestamp, CAST(session_id, 'UUID') AS session_id, action, version, payload
```
Tinybird runs a backfill to migrate the data to the new schema. These backfills are logged in `datasources_ops_log` with the `event_type` set to `deployment_backfill`.

For large-scale populates during data migrations, Tinybird automatically uses on-demand compute to run the populate operations on dedicated instances, avoiding resource contention with your production workloads. See [on-demand compute for deployment populates](/docs/forward/core-concepts/deployments#on-demand-compute-for-deployment-populates) and [On-demand CPUs pricing](/docs/forward/pricing#on-demand-cpus).

If the existing data is incompatible with the schema change, the staging deployment fails and is discarded. For example, if you change a data type from `String` to `UUID`, but the existing data contains invalid values like `'abc'`, the deployment fails with this error:

```
» tb --cloud deploy
...

✓ Deployment submitted successfully
Deployment failed
* Error on datasource '<datasource_name>': Error migrating data: Populate job <job_id> failed, status: error
Rolling back deployment
Previous deployment is already live
Removing current deployment
Discard process successfully started
Discard process successfully completed
```
If you're willing to accept data loss or default values for incompatible records, you can make the deployment succeed by using the [accurateCastOrDefault](/docs/sql-reference/functions/type-conversion-functions#accuratecastordefaultx-t-default-value) function in your forward query:

```
FORWARD_QUERY >
    SELECT timestamp, accurateCastOrDefault(session_id, 'UUID') as session_id, action, version, payload
```
After changes have been deployed and promoted, if you want to deploy other changes that don't affect that data source, you can safely remove the forward query.

## Landing data source engine settings (https://www.tinybird.co/docs/forward/guides/evolve-data-source#landing-data-source-engine-settings)

When you make changes to the engine settings of a landing data source, such as changing the sorting or partition key, you can follow these steps:

1. In Tinybird Local, be sure you have your project ready. If not, run `tb deploy`.
2. Edit the Data Source definition to add the changes. In Tinybird CLI projects, update the `.datasource` file. In SDK projects, update the `defineDatasource` or `define_datasource` resource. No forward query is required. See [engine settings](/docs/forward/dev-reference/datafiles/datasource-files#engine-settings), [TypeScript SDK resources](/docs/forward/dev-reference/typescript-sdk-resources#datasources), and [Python SDK resources](/docs/forward/dev-reference/python-sdk-resources#datasources).
3. Run a deployment check to validate the deployment before creating it. This is a good way of catching potential breaking changes.
4. Deploy and promote your changes in Tinybird Cloud.
If the change requires a rewrite, Tinybird Cloud automatically populates the new table following the new settings.

If the only engine change is adding, modifying, or removing `ENGINE_TTL`, Tinybird applies it with `ALTER`:

- You don't need to add a `FORWARD_QUERY`.
- Tinybird doesn't rebuild and repopulate the data source.
- The TTL change is applied only after promotion, not in staging.
Other engine changes, such as changing the sorting key or partition key, still use the regular evolution flow and create a rewritten copy of the data source.

## Materialized data sources (https://www.tinybird.co/docs/forward/guides/evolve-data-source#materialized-data-sources)

When editing materialized data sources, you need to consider the settings of the landing data sources that feed into them, especially the TTL (Time To Live) settings.

[Forward queries](https://www.tinybird.co/docs/forward/guides/evolve-data-source#forward-query) are essential when evolving materialized data sources, both schema and engine settings, to retain historical data.

You can't use `ALTER` to change the source of a Materialized View. To do so, you must drop the Materialized View and recreate it with the new source. To prevent data loss during this process, add a `SELECT *` `FORWARD_QUERY` to the destination `.datasource` file.

If your landing Data Source has a shorter TTL than your materialized Data Source, a warning appears when you deploy your changes. Add a forward query to prevent data loss. If you accept loss of historical data, add the `--allow-destructive-operations` flag to your deployment command.

For example, consider this scenario:

- Landing data source has a 7-day TTL.
- Materialized data source has no TTL (keeps data indefinitely).
- You want to change the data type of a column in the materialized data source.
Without a forward query, recalculating the materialized data source would only process the last 7 days of data due to the landing source's TTL, causing you to lose historical data beyond that period. To retain all historical data, use a forward query to transform the data from the live schema to the new one.

Here's an example materialized data source that uses a forward query to transform the data type of the `visits` column from `AggregateFunction(count, UInt16)` to `AggregateFunction(count, UInt64)`:

- Tinybird CLI
- TypeScript SDK
- Python SDK
tinybird/datasources/daily_page_visits.datasource
```
DESCRIPTION >
    Materialized data source for daily page visits aggregation

SCHEMA >
    `date` Date,
    `page_url` String,
    `visits` AggregateFunction(count, UInt64)

ENGINE "AggregatingMergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(date)"
ENGINE_SORTING_KEY "date, page_url"

FORWARD_QUERY >
    SELECT date, page_url, CAST(visits, 'AggregateFunction(count, UInt64)') AS visits
```
Omitting the forward query instruction fully recalculates the materialized data source.

You can omit the forward query when:

- Landing data source has a longer TTL than the materialized data source, or no TTL.
- Making non-backward compatible changes, like adding a new group by column.
- Accepting loss of historical data.

### Materialized data source engine settings (https://www.tinybird.co/docs/forward/guides/evolve-data-source#materialized-data-source-engine-settings)

You can safely change engine settings on a Materialized Data Source, including the engine type, without risk of data loss. You don't need to create a new Data Source.

When you change engine settings on a Materialized Data Source, Tinybird backfills it by re-running the Materialized View query by default. You can see the backfill type when you run `tb --cloud deploy --check`.

If you don't want to reprocess all historical data through the Materialized View query, add a `SELECT *` `FORWARD_QUERY` to the `.datasource` file. Tinybird copies data from the live deployment instead of re-running the Materialized View. See [Unchanged downstream Data Sources](https://www.tinybird.co/docs/forward/guides/evolve-data-source#unchanged-downstream-data-sources).

## Backfill strategies (https://www.tinybird.co/docs/forward/guides/evolve-data-source#backfill-strategies)

When you deploy changes, Tinybird automatically decides whether affected Data Sources can be evolved with `ALTER` or need to be backfilled. The backfill strategy depends on the type of Data Source and whether it has a forward query.

### Preview the backfill strategy (https://www.tinybird.co/docs/forward/guides/evolve-data-source#preview-the-backfill-strategy)

Run a deployment check to see the deployment strategy before creating a deployment. The output shows which Data Sources evolve with `ALTER`, which ones are rewritten, and whether forward queries or Materialized View queries backfill rewritten Data Sources.

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb deploy --check
```
You can then adjust the backfill strategy by adding or removing forward queries before deploying.

### Landing Data Sources (https://www.tinybird.co/docs/forward/guides/evolve-data-source#landing-data-sources)

If a landing Data Source has no Materialized Views writing to it, Tinybird backfills it using a `FORWARD_QUERY`:

- If the schema is fully compatible, for example you're only changing the sorting key, Tinybird automatically applies a `SELECT *` `FORWARD_QUERY`.
- If the columns aren't identical, you must provide a `FORWARD_QUERY` instruction to tell Tinybird how to transform the data.

### Data Sources with Materialized Views (https://www.tinybird.co/docs/forward/guides/evolve-data-source#data-sources-with-materialized-views)

If a Data Source is the destination of one or more Materialized Views, Tinybird determines the backfill strategy as follows:

- If the `.datasource` file has a `FORWARD_QUERY` instruction, Tinybird uses it for the backfill. You can force this behavior by adding a forward query.
- If there's no `FORWARD_QUERY`, Tinybird backfills the Data Source by re-running the Materialized View queries that write to it.
- If the `.datasource` file has a `BACKFILL skip` instruction, Tinybird skips the backfill entirely.
Use `BACKFILL skip` to create a new Data Source without backfilling it with historical data from the Materialized View query. This instruction is configured in `.datasource` files and isn't exposed as a TypeScript SDK or Python SDK resource option. Add it at the end of the `.datasource` file:

tinybird/datasources/daily_page_visits.datasource
```
DESCRIPTION >
    Materialized data source

SCHEMA >
    `date` Date,
    `page_url` String,
    `visits` UInt64

ENGINE "SummingMergeTree"
ENGINE_SORTING_KEY "date, page_url"

BACKFILL skip
```
`BACKFILL skip` only applies when the Data Source is first created and has no effect if it already exists.

### Unchanged downstream Data Sources (https://www.tinybird.co/docs/forward/guides/evolve-data-source#unchanged-downstream-data-sources)

Data Sources that haven't changed but are downstream of other changes in your project are also backfilled:

- By default, they're backfilled using Materialized View queries.
- If they have an explicit `FORWARD_QUERY`, Tinybird uses it instead. Since these Data Sources haven't changed, the forward query must be `SELECT *`. This query selects all data from the current live deployment and repopulates it in the Data Source. If you're adding a new Materialized View upstream of an existing Data Source, add an explicit `SELECT *` `FORWARD_QUERY` to the downstream Data Source to avoid re-running the Materialized View query for the backfill.

### Materialized View backfill warnings (https://www.tinybird.co/docs/forward/guides/evolve-data-source#materialized-view-backfill-warnings)

When Tinybird detects potential issues with Materialized View query backfills, it outputs a warning. For example:

- The Materialized View reads from a [Null engine](/docs/sql-reference/engines/null) Data Source, which doesn't store data.
- The source Data Source has a shorter TTL than the Materialized View target, which can result in data loss.
Tinybird can't detect all potential issues. For example, filters in the Materialized View query might cause data loss depending on your use case. Use `tb deploy --check` to review the backfill strategy and add forward queries where needed.

## Next steps (https://www.tinybird.co/docs/forward/guides/evolve-data-source#next-steps)

- Learn more about [deployments](/docs/forward/core-concepts/deployments).
- Learn about datafiles, like .datasource and .pipe files. See [Datafiles](/docs/forward/dev-reference/datafiles).

---

URL: https://www.tinybird.co/docs/forward/guides/deduplication-strategies
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Deduplicate data in your data source \| Tinybird"
theme-color: "#171612"
description: "Learn several strategies for deduplicating data in Tinybird."
inkeep:version: "forward"
---

# Deduplicate data in your data source (https://www.tinybird.co/docs/forward/guides/deduplication-strategies#deduplicate-data-in-your-data-source)

Sometimes you might need to deduplicate data, for example to receive updates or data from a transactional database through CDC. You might want to retrieve only the latest data point, or keep a historic record of the evolution of the attributes of an object over time. Because Tinybird doesn't enforce uniqueness for primary keys when inserting rows, you need to follow different strategies to deduplicate data with minimal side effects.

## Deduplication strategies (https://www.tinybird.co/docs/forward/guides/deduplication-strategies#deduplication-strategies)

You can use one of the following strategies to deduplicate your data.

| Method | When to use |
| --- | --- |
| [Deduplicate at query time](https://www.tinybird.co/docs/forward/guides/deduplication-strategies#deduplicate-at-query-time) | Deduplicate data at query time if you are still prototyping or the data source is small. |
| [Use ReplacingMergeTree](https://www.tinybird.co/docs/forward/guides/deduplication-strategies#use-the-replacingmergetree-engine) | Use `ReplacingMergeTree` or `AggregatingMergeTree` for greater performance. |
| [Snapshot based deduplication](https://www.tinybird.co/docs/forward/guides/deduplication-strategies#snapshot-based-deduplication) | If data freshness isn't required, generate periodic snapshots of the data and take advantage of subsequent materialized views for rollups. |
| [Hybrid approach using Lambda architecture](https://www.tinybird.co/docs/forward/guides/deduplication-strategies#hybrid-approach-using-lambda-architecture) | When you need to overcome engine approach limitations while preserving freshness, combine approaches in a Lambda architecture. |
| Using argMax with null values |  |
For dimensional and small tables, a periodical full replace is usually the best option.

## Example case (https://www.tinybird.co/docs/forward/guides/deduplication-strategies#example-case)

Consider a dataset from a social media analytics company that wants to track some data content over time. You receive an event with the latest info for each post, identified by `post_id`. The three fields, `views`, `likes`, `tags`, vary from event to event. For example:

post.ndjson
```
{ "timestamp": "2024-07-02T02:22:17", "post_id": 956, "views": 856875, "likes": 2321, "tags": "Sports" }
```

## Deduplicate at query time (https://www.tinybird.co/docs/forward/guides/deduplication-strategies#deduplicate-at-query-time)

Imagine you're only interested in the latest value of views for each post. In that case, you can deduplicate data on `post_id` and get the latest value with these strategies:

- Get the max date for each post in a subquery and then filter by its results.
- Group data by `post_id` and use the `argMax` function.
- Use the `LIMIT BY` clause.
Select `Subquery`, `argMax`, or `LIMIT BY` to see the example queries for each.

- Subquery
- argMax
- LIMIT BY
Deduplicating data on post_id using Subquery
```
SELECT *
FROM posts_info
WHERE (post_id, timestamp) IN
(
    SELECT
        post_id,
        max(timestamp)
    FROM posts_info
    GROUP BY post_id
)
```
Depending on your data and how you define the sorting keys in your data sources to store it on disk, one approach is faster than the others.

In general, deduplicating at query time is fine if the size of your data is small. If you have lots of data, use a specific Engine that takes care of deduplication for you.

## Use the ReplacingMergeTree engine (https://www.tinybird.co/docs/forward/guides/deduplication-strategies#use-the-replacingmergetree-engine)

If you've lots of data and you're interested in the latest insertion for each unique key, use the [ReplacingMergeTree](/docs/sql-reference/engines/replacingmergetree) engine with the following options: `ENGINE_SORTING_KEY`, `ENGINE_VER`, and `ENGINE_IS_DELETED`.

- Rows with the same `ENGINE_SORTING_KEY` are deduplicated. You can select one or more columns.
- If you specify a type for `ENGINE_VER`, the row with the highest `ENGINE_VER` for each unique `ENGINE_SORTING_KEY` is kept, for example a timestamp.
- `ENGINE_IS_DELETED` is only active if you use `ENGINE_VER`. This column determines whether the row represents the state or is to be deleted; `1` is a deleted row, `0` is a state row. The type must be `UInt8`.
- You can omit `ENGINE_VER`, so that the last inserted row for each unique `ENGINE_SORTING_KEY` is kept.
Do not build Materialized Views with an AggregatingMergeTree on top of a ReplacingMergeTree. The target Data Source always contains duplicates due to the incremental nature of Materialized Views.

### Define a data source (https://www.tinybird.co/docs/forward/guides/deduplication-strategies#define-a-data-source)

Define a data source like the following:

post_views_rmt.datasource
```
DESCRIPTION >
    data source to save post info. ReplacingMergeTree Engine.

SCHEMA >
    `post_id` Int32 `json:$.post_id`,
    `views` Int32 `json:$.views`,
    `likes` Int32 `json:$.likes`,
    `tag` String `json:$.tag`,
    `timestamp` DateTime `json:$.timestamp`,
    `_is_deleted` UInt8 `json:$._is_deleted`

ENGINE "ReplacingMergeTree"
ENGINE_PARTITION_KEY ""
ENGINE_SORTING_KEY "post_id"
ENGINE_VER "timestamp"
ENGINE_IS_DELETED "_is_deleted"
```
ReplacingMergeTree deduplicates during a merge, and merges can't be controlled. Consider adding the `FINAL` clause, or an alternative deduplication method, to apply the merge at query time. Note also that rows are masked, not removed, when using `FINAL`.

- FINAL
- Subquery
- argMax
- LIMIT BY
Deduplicating data on post_id using FINAL
```
SELECT *
FROM posts_info_rmt FINAL
```
You can define the `posts_info_rmt` as the landing data source, the one you send events to, or as a materialized view from `posts_info`. You can also create a data source with an `AggregatingMergeTree` Engine using `maxState(ts)` and `argMaxState(field,ts)`.

## Snapshot based deduplication (https://www.tinybird.co/docs/forward/guides/deduplication-strategies#snapshot-based-deduplication)

Use [Copy Pipes](/docs/forward/core-concepts/copy-pipes) to take a query result and write it to a new data source in the following situations:

- You need other Sorting Keys that might change with updates.
- You need to do rollups and want to use materialized views.
- Response times are too long with a `ReplacingMergeTree`.
The following is an example snapshot:

post_generate_snapshot.pipe
```
NODE gen_snapshot
SQL >

    SELECT
        post_id,
        argMax(views, timestamp) views,
        argMax(likes, timestamp) likes,
        argMax(tag, timestamp) tag,
        max(timestamp) as ts,
        toStartOfMinute(now()) - INTERVAL 1 MINUTE as snapshot_ts
    FROM posts_info
    WHERE timestamp <= toStartOfMinute(now()) - INTERVAL 1 MINUTE
    GROUP BY post_id

TYPE COPY
TARGET_DATASOURCE post_snapshot
COPY_MODE replace
COPY_SCHEDULE 0 * * * *
```
Because the `TARGET_DATASOURCE` engine is a MergeTree, you can use fields that you expect to be updated as sorting keys in the ReplacingMergeTree.

post_snapshot.datasource
```
SCHEMA >
    `post_id` Int32,
    `views` Int32,
    `likes` Int32,
    `tag` String,
    `ts` DateTime,
    `snapshot_ts` DateTime

ENGINE "MergeTree"
ENGINE_PARTITION_KEY ""
ENGINE_SORTING_KEY "tag, post_id"
```

## Hybrid approach using Lambda architecture (https://www.tinybird.co/docs/forward/guides/deduplication-strategies#hybrid-approach-using-lambda-architecture)

Snapshots might decrease data freshness, and running Copy Pipes too frequently might be more expensive than materializations. A way to mitigate these issues is to combine batch and real-time processing, reading the latest snapshot and incorporating the changes that happened since then.

This pattern is described in the [Lambda architecture](/docs/forward/guides/lambda-architecture) guide.

Using the `post_snapshot` data source created before, the real-time Pipe would be like the following:

latest_values.pipe
```
NODE get_latest_changes
SQL >

    SELECT
        max(timestamp) last_ts,
        post_id,
        argMax(views, timestamp) views,
        argMax(likes, timestamp) likes,
        argMax(tag, timestamp) tag
    FROM posts_info_rmt
    WHERE timestamp > (SELECT max(snapshot_ts) FROM post_snapshot)
    GROUP BY post_id

NODE get_snapshot
SQL >

    SELECT
        last_ts,
        post_id,
        views,
        likes,
        tag
    FROM post_snapshot
    WHERE snapshot_ts = (SELECT max(snapshot_ts) FROM post_snapshot)
    AND post_id NOT IN (SELECT post_id FROM get_latest_changes)

NODE combine_both
SQL >

    SELECT * FROM get_snapshot
    UNION ALL
    SELECT * FROM get_latest_changes
```

## A note on using argMax with null values (https://www.tinybird.co/docs/forward/guides/deduplication-strategies#a-note-on-using-argmax-with-null-values)

Here are the definitions for `argMax` functions:

- `argMaxState`: used to pipe a constantly updating max state into a materialized view.
- `argMaxMerge`: used to query a max state value out of a materialized view.
**When dealing with null values, `argMax` functions might not behave as you expect.** In raw queries, you might see a null value for the most recent record. However, when data is piped into an AggregatingMergeTree materialized view using `argMaxState` and later queried using `argMaxMerge`, the result can be different.

Returning to the social media analytics example, imagine you want to track most recent time when a post was flagged.

The raw query would be as follows:

get_latest_flagged_at_raw.pipe
```
# This returns `null` if the most recent record's `flaggedAt` value is null.

NODE get_latest_flagged_at_raw
SQL >

    SELECT flaggedAt
    FROM posts
    WHERE post_id = 'abc123'
    ORDER BY timestamp DESC
    LIMIT 1
```
First, data is aggregated into a materialized view with a pipe that uses `argMaxState`:

get_latest_flagged_at_mat.pipe
```
NODE get_latest_flagged_at
SQL >

    SELECT argMaxState(flaggedAt, timestamp)
    FROM posts

TYPE materialized
DATASOURCE post_analytics_mv
```
Later, the aggregated state is merged with `argMaxMerge` when the materialized view is queried:

get_latest_flagged_at_mat.pipe
```
NODE get_absolute_latest_flagged_at
SQL >

    SELECT argMaxMerge(flaggedAt)
    FROM post_analytics_mv
    WHERE post_id = 'abc123'
```
Although you might expect a `null` result because the raw query returned `null`, the merging process prefers any non‑null value over a null value—even if its associated timestamp is lower. The result is the most recent non‑null `flaggedAt` value.

#### Why this happens and workaround (https://www.tinybird.co/docs/forward/guides/deduplication-strategies#why-this-happens-and-workaround)

During the merge process, the materialized view combines key‑value candidate states. A key-value candidate in this example would be `timestamp` and `flaggedAt`. If one candidate has a non‑null `flaggedAt` and another has a null value for `flaggedAt`, the non‑null value "wins" regardless of its timestamp. This behavior is inherent to the merging logic in `argMaxMerge`. It explains why in an argMaxMerge query, you might not get the absolute max.

To prevent the merging behavior from overriding a null with a non‑null candidate, you need to handle null values explicitly before they enter the materialized view. One common approach is to transform null values to a known default—often the Unix epoch `1970-01-01 00:00:00`. However, be aware that such conversions produce "fake" values which must be recognized in subsequent processing.

Assume your raw data includes a flaggedAt column, which may contain nulls. You can pre-process the data during aggregation as follows:

get_latest_flagged_at_no_nulls.pipe
```
NODE get_latest_flagged_at_no_nulls
SQL >

    SELECT
        argMaxState(
            CASE
                WHEN flaggedAt IS NULL
                THEN toDateTime('1970-01-01 00:00:00')
                ELSE flaggedAt
            END,
            timestamp
        )
    FROM posts
```
Here, a CASE expression is used to convert any null flaggedAt values into the default datetime 1970-01-01 00:00:00. This ensures that during the merge, the aggregation logic processes these explicit default values rather than implicitly "overriding" nulls.

Since `1970-01-01 00:00:00` is used as a default placeholder, ensure that any downstream logic differentiates between genuine datetime values and these default values.

## Next steps (https://www.tinybird.co/docs/forward/guides/deduplication-strategies#next-steps)

- Read the [materialized views docs](/docs/forward/core-concepts/materialized-views).
- Read the [Lambda architecture guide](/docs/forward/guides/lambda-architecture).
- Publish your data with [API endpoints](/docs/forward/core-concepts/api-endpoints).

---

URL: https://www.tinybird.co/docs/forward/guides/copy-and-export-data
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Copy and export data \| Tinybird"
theme-color: "#171612"
description: "Learn when to move data inside Tinybird and when to export results to external systems."
inkeep:version: "forward"
---

# Copy and export data (https://www.tinybird.co/docs/forward/guides/copy-and-export-data#copy-and-export-data)

Use this guide to choose between moving data inside Tinybird and exporting query results to external systems.

- To move data between Tinybird Data Sources, use [Copy Pipes](/docs/forward/core-concepts/copy-pipes).
- To export query results to external systems, use [Sinks](/docs/forward/core-concepts/sinks).
- For destination-specific setup, see [Kafka Sink](/docs/forward/copy-export-data/kafka-sink), [S3 Sink](/docs/forward/copy-export-data/s3-sink), and [GCS Sink](/docs/forward/copy-export-data/gcs-sink).
See [Copy and export data](/docs/forward/copy-export-data) for the full workflow.

---

URL: https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Consume APIs in a Next.js frontend with JWTs \| Tinybird"
theme-color: "#171612"
description: "In this guide, you'll learn how to generate self-signed JWTs from your backend, and call Tinybird APIs directly from your frontend, using Next.js."
inkeep:version: "forward"
---

# Consume APIs in a Next.js frontend with JWTs (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#consume-apis-in-a-next-js-frontend-with-jwts)

This guide shows you how to generate self-signed JWTs from your backend and call Tinybird APIs directly from your frontend using Next.js.

JWTs are signed tokens that allow you to securely authorize and share data between your app and Tinybird.

You can view the [live demo](https://guide-nextjs-jwt-auth.vercel.app/) or browse the [GitHub repo (guide-nextjs-jwt-auth)](https://github.com/tinybirdco/guide-nextjs-jwt-auth).

## Prerequisites (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#prerequisites)

This guide assumes that you have a Tinybird account, and you are familiar with creating a Tinybird workspace and pushing resources to it.

Make sure you understand the concept of Tinybird's [Static Tokens](/docs/forward/core-concepts/tokens#what-should-i-use-tokens-for).

You need a working familiarity with JWTs, JavaScript, and Next.js.

## Run the demo (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#run-the-demo)

These steps cover running the GitHub demo locally. [Skip to the next section](https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#understand-the-code) for a breakdown of the code.

### 1. Clone the GitHub repo (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#1-clone-the-github-repo)

Clone the [GitHub repo (guide-nextjs-jwt-auth)](https://github.com/tinybirdco/guide-nextjs-jwt-auth) to your local machine.

### 2. Push Tinybird resources (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#2-push-tinybird-resources)

The repo includes two sample Tinybird resources:

- `events.datasource`: The data source for incoming events.
- `top_airlines.pipe`: An API endpoint giving a list of top 10 airlines by booking volume.

### 3. Generate some fake data (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#3-generate-some-fake-data)

Use [Mockingbird](https://tbrd.co/mockingbird-nextjs-jwt-demo) to generate fake data for the `events` data source.

Using this link ^ provides a pre-configured schema, but you must enter your Workspace admin Token and Host. When configured, find and select `Start Generating!`.

In the Tinybird UI, confirm that the `events` data source is successfully receiving data.

### 4. Install dependencies (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#4-install-dependencies)

Navigate to the cloned repo and install the dependencies with `npm install`.

### 5. Configure .env (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#5-configure-env)

First create a new file `.env.local`

```
cp .env.example .env.local
```
Copy your [Tinybird host](/docs/api-reference#regions-and-endpoints) and admin Token (used as the `TINYBIRD_SIGNING_TOKEN`) to the `.env.local` file:

```
TINYBIRD_SIGNING_TOKEN="TINYBIRD_SIGNING_TOKEN>" # Use your Admin Token as the signing Token
TINYBIRD_WORKSPACE="YOUR_WORKSPACE_ID"  # The UUID of your workspace
NEXT_PUBLIC_TINYBIRD_HOST="YOUR_TINYBIRD_API_REGION e.g. https://api.tinybird.co" # Your regional API host
```

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

### Run the demo app (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#run-the-demo-app)

Run it locally:

```
npm run dev
```
Then open `localhost:3000` with your browser.

## Understand the code (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#understand-the-code)

This section breaks down the key parts of code from the example.

### .env (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#env)

<a href="https://github.com/tinybirdco/guide-nextjs-jwt-auth/blob/main/.env.example">The `.env` file</a> contains the environment variables used in the app.

.env file
```
TINYBIRD_SIGNING_TOKEN="YOUR SIGNING TOKEN"
TINYBIRD_WORKSPACE="YOUR WORKSPACE ID"
NEXT_PUBLIC_TINYBIRD_HOST="YOUR API HOST e.g. https://api.tinybird.co"
```

#### TINYBIRD_SIGNING_TOKEN (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#tinybird-signing-token)

`TINYBIRD_SIGNING_TOKEN` is the Token used to sign JWTs. **You must use your Admin Token**. It is a shared secret between your app and Tinybird. Your app uses this Token to sign JWTs, and Tinybird uses it to verify the JWTs. Keep it secret because exposing it could allow unauthorized access to your Tinybird resources. Store it in an environment variable instead of hardcoding it in your app.

#### TINYBIRD_WORKSPACE (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#tinybird-workspace)

`TINYBIRD_WORKSPACE` is the ID of your workspace. It is used to identify the workspace that the JWT is generated for. The workspace ID is included inside the JWT payload. workspace IDs are UUIDs and can be found using the CLI `tb workspace current` command or from the Tinybird UI.

#### NEXT_PUBLIC_TINYBIRD_HOST (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#next-public-tinybird-host)

`NEXT_PUBLIC_TINYBIRD_HOST` is the base URL of the Tinybird API. It is used to construct the URL for the Tinybird API endpoints. You must use the correct URL for [your Tinybird region](/docs/api-reference#regions-and-endpoints). The `NEXT_PUBLIC_` prefix is required for Next.js to expose the variable to the client side.

### token.ts (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#token-ts)

<a href="https://github.com/tinybirdco/guide-nextjs-jwt-auth/blob/main/server/token.ts">The `token.ts` file</a> contains the logic to generate and sign JWTs. It uses the `jsonwebtoken` library to create the Token.

token.ts
```
"use server";

import jwt from "jsonwebtoken";

const TINYBIRD_SIGNING_TOKEN = process.env.TINYBIRD_SIGNING_TOKEN ?? "";
const WORKSPACE_ID = process.env.TINYBIRD_WORKSPACE ?? "";
const PIPE_ID = "top_airlines";

export async function generateJWT() {
  const next10minutes = new Date();
  next10minutes.setTime(next10minutes.getTime() + 1000 * 60 * 10);

  const payload = {
    workspace_id: WORKSPACE_ID,
    name: "my_demo_jwt",
    exp: Math.floor(next10minutes.getTime() / 1000),
    scopes: [
      {
        type: "PIPES:READ",
        resource: PIPE_ID,
      },
    ],
  };

  return jwt.sign(payload, TINYBIRD_SIGNING_TOKEN, {noTimestamp: true});
}
```
This code runs on the backend to generate JWTs without exposing secrets to the user.

It pulls in the `TINYBIRD_SIGNING_TOKEN` and `WORKSPACE_ID` from the environment variables.

As this example only exposes a single API endpoint (`top_airlines.pipe`), the `PIPE_ID` is hardcoded to its deployed ID. If you had multiple API endpoints, you would need to create an item in the `scopes` array for each one.

The `generateJWT` function handles creation of the JWT. A JWT has various [required fields](/docs/forward/core-concepts/tokens#jwt-payload).

The `exp` field sets the expiration time of the JWT in the form a UTC timestamp. In this case, it's set to 10 minutes in the future. You can adjust this value to suit your needs.

The `name` field is a human-readable name for the JWT. This value is only used for logging.

The `scopes` field defines what the JWT can access. This is an array, which allows you create one JWT that can access multiple API endpoints. In this case, you only have one API endpoint. Under `scopes`, the `type` field is always `PIPES:READ` for reading data from a pipe. The `resource` field is the ID or name of the pipe you want to access. If required, you can also add `fixed_parameters` here to supply parameters to the API endpoint.

Finally, the payload is signed using the `jsonwebtoken` library and the `TINYBIRD_SIGNING_TOKEN`.

### useFetch.tsx (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#usefetch-tsx)

<a href="https://github.com/tinybirdco/guide-nextjs-jwt-auth/blob/main/hooks/useFetch.tsx">The `useFetch.tsx` file</a> contains a custom React hook that fetches data from the Tinybird API using a JWT. It also handles refreshing the token if it expires.

useFetch.tsx
```
import { generateJWT } from "@/server/token";
import { useState } from "react";

export function useFetcher() {
  const [token, setToken] = useState("");

  const refreshToken = async () => {
    const newToken = await generateJWT();
    setToken(newToken);
    return newToken;
  };

  return async (url: string) => {
    let currentToken = token;
    if (!currentToken) {
      currentToken = await refreshToken();
    }
    const response = await fetch(url + "?token=" + currentToken);

    if (response.status === 200) {
      return response.json();
    }
    if (response.status === 403) {
      const newToken = await refreshToken();
      return fetch(url + "?token=" + newToken).then((res) => res.json());
    }
  };
}
```
This code runs on the client side and is used to fetch data from the Tinybird API.

It uses the `generateJWT` function from the <a href="about:blank#token-ts">`token.ts` file</a> to get a JWT. The JWT is stored in the `token` state.

Most importantly, it uses the standard `fetch` API to make requests to the Tinybird API. The JWT is passed as a `token` query parameter in the URL.

If the request returns a `403` status code, the hook calls `refreshToken` to get a new JWT and retries the request. However, this is a simple implementation, and a request might fail with a `403` status code for other reasons. For example, the JWT might be invalid or the API Endpoint might have been removed.

### page.tsx (https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#page-tsx)

<a href="https://github.com/tinybirdco/guide-nextjs-jwt-auth/blob/main/app/page.tsx">The `page.tsx` file</a> contains the main logic for the Next.js page. It is responsible for initiating the call to the Tinybird API endpoints and rendering the data into a chart.

page.tsx
```
"use client";

import { BarChart, Card, Subtitle, Text, Title } from "@tremor/react";
import useSWR from "swr";
import { getEndpointUrl } from "@/utils";
import { useFetcher } from "@/hooks/useFetch";

const REFRESH_INTERVAL_IN_MILLISECONDS = 5000; // five seconds

export default function Dashboard() {
  const endpointUrl = getEndpointUrl();
  const fetcher = useFetcher();

  let top_airline, latency, errorMessage;

  const { data } = useSWR(endpointUrl, fetcher, {
    refreshInterval: REFRESH_INTERVAL_IN_MILLISECONDS,
    onError: (error) => (errorMessage = error),
  });

  if (!data) return;

  if (data?.error) {
    errorMessage = data.error;
    return;
  }

  top_airline = data.data;
  latency = data.statistics?.elapsed;

  return (
    <Card>
      <Title>Top airlines by bookings</Title>
      <Subtitle>Ranked from highest to lowest</Subtitle>
      {top_airline && (
        <BarChart
          className="mt-6"
          data={top_airline}
          index="airline"
          categories={["bookings"]}
          colors={["blue", "red"]}
          yAxisWidth={48}
          showXAxis={true}
        />
      )}
      {latency && <Text>Latency: {latency * 1000} ms</Text>}
      {errorMessage && (
        <div className="mt-4 text-red-600">
          <p>
            Oops, something happens: <strong>{errorMessage}</strong>
          </p>
          <p className="text-sm">Check your console for more information</p>
        </div>
      )}
    </Card>
  );
}
```
It uses [SWR](https://swr.vercel.app/) and the `useFetcher` hook from [useFetch.tsx](https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs#usefetch-tsx) to fetch data from the Tinybird API.

When the API endpoint returns data, it's rendered as bar chart using the `BarChart` component from the <a href="https://www.tremor.so/">`@tremor/react` library</a>.

---

URL: https://www.tinybird.co/docs/forward/guides/consume-apis-in-a-notebook
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Consume APIs in a Notebook \| Tinybird"
theme-color: "#171612"
description: "Notebooks are a great resource for exploring data and generating plots. In this guide, you'll learn how to consume Tinybird APIs in a colab notebook."
inkeep:version: "forward"
---

# Consume APIs in a Notebook (https://www.tinybird.co/docs/forward/guides/consume-apis-in-a-notebook#consume-apis-in-a-notebook)

Notebooks are a great resource for exploring data and generating plots. This guide shows you how to consume Tinybird APIs in a Colab Notebook.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/consume-apis-in-a-notebook#prerequisites)

This [Colab notebook](https://github.com/tinybirdco/examples/blob/master/notebook/consume_from_apis.ipynb) uses a data source of updates to Wikipedia to show how to consume data from queries. There are two options: Using the [Query API](/docs/api-reference/query-api), and using API endpoints using the [Pipes API](/docs/api-reference/pipe-api) and parameters. The full code for every example in this guide can be found in the notebook.

This guide assumes some familiarity with Python.

## Setup (https://www.tinybird.co/docs/forward/guides/consume-apis-in-a-notebook#setup)

Follow the setup steps in the [notebook file](https://github.com/tinybirdco/examples/blob/master/notebook/consume_from_apis.ipynb) and use the linked CSV file of Wikipedia updates to create a new data source in your workspace.

For less than 100 MB of data, you can fetch all the data. For calls with than 100 MB of data, you need to do it sequentially, with not more than 100 MB per API call. The solution is to get batches using data source sorting keys. Selecting the data by columns used in the sorting key keeps it fast. In this example, the data source is sorted on the `timestamp` column, so you can use batches of a fixed amount of time. In general, time is a good way to batch.

The functions `fetch_table_streaming_query` and `fetch_table_streaming_endpoint` in the notebook work as generators. They should always be used in a `for` loop or as the input for another generator.

You should process each batch as it arrives and discard unwanted fetched data. Only fetch the data you need in the processing. The idea here isn't to recreate a data source in the notebook, but to process each batch as it arrives and write less data to your DataFrame.

## Fetch data with the Query API (https://www.tinybird.co/docs/forward/guides/consume-apis-in-a-notebook#fetch-data-with-the-query-api)

This guide uses the [requests library for Python](https://pypi.org/project/requests/). The SQL query pulls in an hour less of data than the full data source. A DataFrame is created from the text part of the response.

DataFrame from the query API
```
table_name = 'wiki'
host = 'api.tinybird.co'
format = 'CSVWithNames'
time_column = 'toDateTime(timestamp)'
date_end = 'toDateTime(1644754546)'

s = requests.Session()
s.headers['Authorization'] = f'Bearer {token}'

URL = f'https://{host}/v0/sql'
sql = f'select * from {table_name} where {time_column} <= {date_end}'
params = {'q': sql + f" FORMAT {format}"}

r = s.get(f"{URL}?{urlencode(params)}")
df = pd.read_csv(StringIO(r.text))
```

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

## Fetch data from an API endpoint & parameters (https://www.tinybird.co/docs/forward/guides/consume-apis-in-a-notebook#fetch-data-from-an-api-endpoint-parameters)

This Endpoint node in the pipe `endpoint_wiki` selects from the data source within a range of dates, using the parameters for `date_start` and `date_end`.

Endpoint wiki
```
%
SELECT * FROM wiki
WHERE timestamp BETWEEN
toInt64(toDateTime({{String(date_start, '2022-02-13 10:30:00')}}))
AND
toInt64(toDateTime({{String(date_end, '2022-02-13 11:00:00')}}))
```
These parameters are passed in the call to the API endpoint to select only the data within the range. A DataFrame is created from the text part of the response.

Dataframe from API endpoint
```
host = 'api.tinybird.co'
api_endpoint = 'endpoint_wiki'
format = 'csv'

date_start = '2022-02-13 10:30:00'
date_end = '2022-02-13 11:30:00'

s = requests.Session()
s.headers['Authorization'] = f'Bearer {token}'

URL = f'https://{host}/v0/pipes/{api_endpoint}.{format}'
params = {'date_start': date_start,
         'date_end': date_end
         }

r = s.get(f"{URL}?{urlencode(params)}")
df = pd.read_csv(StringIO(r.text))
```

The API host in the following examples must match your Workspace's region. See the full list of [regions and hosts](/docs/api-reference#regions-and-endpoints)

## Fetch batches of data using the Query API (https://www.tinybird.co/docs/forward/guides/consume-apis-in-a-notebook#fetch-batches-of-data-using-the-query-api)

The function `fetch_table_streaming_query` in the notebook accepts more complex queries than a date range. Here you choose what you filter and sort by. This example reads in batches of 5 minutes to create a small DataFrame, which should then be processed, with the results of the processing appended to the final DataFrame.

<!-- <figure> -->
![](/docs/_next/image?url=%252Fdocs%252Fimg%252Fconsume-apis-in-a-notebook-1.png&w=3840&q=75)

<!-- <figcaption> -->
5-minute batches of data using the index
<!-- </figcaption> -->

<!-- </figure> -->
DataFrames from batches returned by the Query API
```
tinybird_stream = fetch_table_streaming_query(token,
                                       'wiki',
                                       60*5,
                                       1644747337,
                                       1644758146,
                                       sorting='timestamp',
                                       filters="type IN ['edit','new']",
                                       time_column="timestamp",
                                       host='api.tinybird.co')

df_all=pd.DataFrame()
for x in tinybird_stream:
   df_batch = pd.read_csv(StringIO(x))
   # TO DO: process batch and discard fetched data
   df_proc=process_dataframe(df_batch)
   df_all = df_all.append(df_proc) # Careful: appending dfs means keeping a lot of data in memory
```

## Fetch batches of data from an API endpoint and parameters (https://www.tinybird.co/docs/forward/guides/consume-apis-in-a-notebook#fetch-batches-of-data-from-an-api-endpoint-and-parameters)

The function `fetch_table_streaming_endpoint` in the notebook sends a call to the API with parameters for the `batch size`, `start` and `end` dates, and, optionally, filters on the `bot` and `server_name` columns. This example reads in batches of 5 minutes to create a small DataFrame, which should then be processed, with the results of the processing appended to the final DataFrame.

‍The API endpoint `wiki_stream_example` first selects data for the range of dates, then for the batch, and then applies the filters on column values.

API endpoint wiki_stream_example
```
%
SELECT * from wiki
--DATE RANGE
WHERE timestamp BETWEEN toUInt64(toDateTime({{String(date_start, '2022-02-13 10:30:00', description="start")}}))
AND toUInt64(toDateTime({{String(date_end, '2022-02-13 10:35:00', description="end")}}))
--BATCH BEGIN
AND timestamp BETWEEN toUInt64(toDateTime({{String(date_start, '2022-02-13 10:30:00', description="start")}})
              + interval {{Int16(batch_no, 1, description="batch number")}}
              * {{Int16(batch_size, 10, description="size of the batch")}} second)
--BATCH END
AND toUInt64(toDateTime({{String(date_start, '2022-02-13 10:30:00', description="start")}})
              + interval ({{Int16(batch_no, 1, description="batch number")}} + 1)
              * {{Int16(batch_size, 10, description="size of the batch")}} second)
--FILTERS
{% if defined(bot) %}
 AND bot = {{String(bot, description="is a bot")}}
{% end %}
{% if defined(server_name) %}
 AND server_name = {{String(server_name, description="server")}}
{% end %}
```
These parameters are passed in the call to the API endpoint to select only the data for the batch. A DataFrame is created from the text part of the response.

DataFrames from batches from the API endpoint
```
tinybird_stream = fetch_table_streaming_endpoint(token,
                                                 'csv',
                                                 60*5,
                                                 '2022-02-13 10:15:00',
                                                 '2022-02-13 13:15:00',
                                                 bot = False,
                                                 server_name='en.wikipedia.org'
                                                )

df_all=pd.DataFrame()
for x in tinybird_stream:
    df_batch = pd.read_csv(StringIO(x))
    # TO DO: process batch and discard fetched data
    df_proc=process_dataframe(df_batch)
    df_all = df_all.append(df_proc) # Careful: appending dfs means keeping a lot of data in memory
```

---

URL: https://www.tinybird.co/docs/forward/guides/consume-api-endpoints-in-prometheus-format
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Consume API endpoints in Prometheus format \| Tinybird"
theme-color: "#171612"
description: "Export pipe endpoints in Prometheus format to integrate Tinybird data into your monitoring stack."
inkeep:version: "forward"
---

# Consume API endpoints in Prometheus format (https://www.tinybird.co/docs/forward/guides/consume-api-endpoints-in-prometheus-format#consume-api-endpoints-in-prometheus-format)

Prometheus is a powerful open source monitoring and alerting toolkit widely used for metrics collection and visualization. You can export pipe endpoints in Prometheus format to integrate your Tinybird data into your monitoring stack.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/consume-api-endpoints-in-prometheus-format#prerequisites)

This guide assumes you have a Tinybird workspace with an active data source, pipes, and at least one API endpoint.

## Structure data for Prometheus (https://www.tinybird.co/docs/forward/guides/consume-api-endpoints-in-prometheus-format#structure-data-for-prometheus)

To export the pipe output in Prometheus format, data must conform to the following structure:

### Mandatory columns (https://www.tinybird.co/docs/forward/guides/consume-api-endpoints-in-prometheus-format#mandatory-columns)

- `name (String)`: The name of the metric.
- `value (Number)`: The numeric value for the metric.

### Optional columns (https://www.tinybird.co/docs/forward/guides/consume-api-endpoints-in-prometheus-format#optional-columns)

- `help (String)`: A description of the metric.
- `timestamp (Number)`: A Unix timestamp for the metric.
- `type (String)`: Defines the metric type (`counter`, `gauge`, `histogram`, `summary`, `untyped`, or empty).
- `labels (Map(String, String))`: A set of key-value pairs providing metric dimensions.

### Add token (https://www.tinybird.co/docs/forward/guides/consume-api-endpoints-in-prometheus-format#add-token)

Add the right authentication token to your API endpoint. If you are monitoring your Tinybird's Organization metrics, you would need to use `ORG_DATASOURCES:READ` scope [to query Organization level data sources](/forward/core-concepts/static-tokens#user-created-tokens).

### Example (https://www.tinybird.co/docs/forward/guides/consume-api-endpoints-in-prometheus-format#example)

Here’s an example of a Tinybird pipe query that outputs two metrics, `http_request_count` and `http_request_duration_seconds`, in the same query. Both metrics include labels for `method` and `status_code`.

```
SELECT
    -- Metric 1: http_request_count
    'http_request_count' AS name,
    toFloat64(count(*)) AS value,
    'Total number of HTTP requests' AS help,
    'counter' AS type,
    map('method', method, 'status_code', status_code) AS labels
FROM
    http_requests
GROUP BY
    method, status_code

UNION ALL

SELECT
    -- Metric 2: http_request_duration_seconds
    'http_request_duration_seconds' AS name,
    avg(request_time) AS value,
    'Average HTTP request duration in seconds' AS help,
    'gauge' AS type,
    map('method', method, 'status_code', status_code) AS labels
FROM
    http_requests
GROUP BY
    method, status_code

ORDER BY
    name
```

## Export Tinybird pipe endpoint in Prometheus format (https://www.tinybird.co/docs/forward/guides/consume-api-endpoints-in-prometheus-format#export-tinybird-pipe-endpoint-in-prometheus-format)

Export pipe data in Prometheus format by appending .prometheus to your API endpoint URI. For example:

```
https://api.tinybird.co/v0/pipes/your_pipe_name.prometheus
```
The following is an example Prometheus output:

```
# HELP http_request_count Total number of HTTP requests
# TYPE http_request_count counter
http_request_count{method="PUT",status_code="203"} 1
http_request_count{method="PATCH",status_code="203"} 1
http_request_count{method="DELETE",status_code="201"} 4
http_request_count{method="POST",status_code="203"} 1
http_request_count{method="OPTIONS",status_code="203"} 1
http_request_count{method="PATCH",status_code="204"} 1
http_request_count{method="PUT",status_code="204"} 1
http_request_count{method="HEAD",status_code="203"} 1
http_request_count{method="GET",status_code="201"} 4
http_request_count{method="POST",status_code="204"} 1
http_request_count{method="GET",status_code="203"} 1
http_request_count{method="POST",status_code="201"} 4
http_request_count{method="DELETE",status_code="204"} 1
http_request_count{method="OPTIONS",status_code="201"} 4
http_request_count{method="GET",status_code="204"} 1
http_request_count{method="PATCH",status_code="201"} 4
http_request_count{method="PUT",status_code="201"} 4
http_request_count{method="DELETE",status_code="203"} 1
http_request_count{method="HEAD",status_code="201"} 4

# HELP http_request_duration_seconds Average HTTP request duration in seconds
# TYPE http_request_duration_seconds gauge
http_request_duration_seconds{method="GET",status_code="200"} 75.01
http_request_duration_seconds{method="DELETE",status_code="201"} 11.01
http_request_duration_seconds{method="POST",status_code="202"} 102.00999999999999
http_request_duration_seconds{method="HEAD",status_code="204"} 169.01
http_request_duration_seconds{method="PATCH",status_code="204"} 169.01
http_request_duration_seconds{method="PUT",status_code="204"} 169.01
http_request_duration_seconds{method="HEAD",status_code="202"} 102.00999999999999
http_request_duration_seconds{method="OPTIONS",status_code="202"} 102.00999999999999
http_request_duration_seconds{method="DELETE",status_code="200"} 75.01
http_request_duration_seconds{method="OPTIONS",status_code="204"} 169.01
http_request_duration_seconds{method="GET",status_code="201"} 11.01
http_request_duration_seconds{method="PATCH",status_code="202"} 102.00999999999999
http_request_duration_seconds{method="PUT",status_code="202"} 102.00999999999999
http_request_duration_seconds{method="POST",status_code="204"} 169.01
http_request_duration_seconds{method="DELETE",status_code="202"} 102.00999999999999
http_request_duration_seconds{method="PUT",status_code="200"} 75.01
http_request_duration_seconds{method="POST",status_code="200"} 75.01
http_request_duration_seconds{method="PATCH",status_code="200"} 75.01
http_request_duration_seconds{method="POST",status_code="201"} 11.01
http_request_duration_seconds{method="DELETE",status_code="204"} 169.01
http_request_duration_seconds{method="OPTIONS",status_code="201"} 11.01
http_request_duration_seconds{method="GET",status_code="204"} 169.01
http_request_duration_seconds{method="PATCH",status_code="201"} 11.01
http_request_duration_seconds{method="PUT",status_code="201"} 11.01
http_request_duration_seconds{method="GET",status_code="202"} 102.00999999999999
http_request_duration_seconds{method="HEAD",status_code="201"} 11.01
```

## Integrate endpoints in Prometheus-compatible tools (https://www.tinybird.co/docs/forward/guides/consume-api-endpoints-in-prometheus-format#integrate-endpoints-in-prometheus-compatible-tools)

Now that you’ve structured and exported your pipe data in Prometheus format, you can integrate it into monitoring and observability tools.

Prometheus is widely supported by various visualization and alerting platforms, making it easy to use your Tinybird data with tools like Grafana, Datadog, and more. See the documentation for these tools to integrate them with the Prometheus endpoints from Tinybird.

## Monitor your Tinybird Organization with Grafana and Datadog (https://www.tinybird.co/docs/forward/guides/consume-api-endpoints-in-prometheus-format#monitor-your-tinybird-organization-with-grafana-and-datadog)

Check the [Tinybird Organization metrics](https://github.com/tinybirdco/tinybird-org-metrics-exporter) repository for a working example of how to consume the Prometheus endpoints in Grafana and Datadog to monitor your Tinybird Organization.

---

URL: https://www.tinybird.co/docs/forward/guides/connect-tableau
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Connect Tableau to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect Tableau to Tinybird using the ClickHouse data source, enabling collaborative analytics and data science workflows."
inkeep:version: "forward"
---

# Connect Tableau to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-tableau#connect-tableau-to-tinybird)

Connect Tableau to Tinybird using the ClickHouse® JDBC driver to take advantage of Tinybird's ClickHouse® HTTP protocol compatibility. Use Tinybird data in Tableau.

The ClickHouse® connection to Tinybird is read-only. You can use it to analyze and visualize data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-tableau#prerequisites)

- A Tinybird workspace with data sources
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).

## Add Tinybird as a data source (https://www.tinybird.co/docs/forward/guides/connect-tableau#add-tinybird-as-a-data-source)

1. Download the [ClickHouse® JDBC Driver](https://github.com/ClickHouse/clickhouse-java/releases/), use the latest `clickhouse-jdbc-x.y.z-all-dependencies.jar` version.
2. Save the ClickHouse JDBC driver in any of these folders depending on your OS:

- macOS: ~/Library/Tableau/Drivers
- Windows: C:\Program Files\Tableau\Drivers

1. In Tableau Desktop, select **Connect > More** and type `ClickHouse`. Select `ClickHouse by ClickHouse` and follow the wizard.
2. After installing the connector, select **Connect > To a Server > ClickHouse JDBC**
3. **Configure the connection** with these settings:

### Connection configuration (https://www.tinybird.co/docs/forward/guides/connect-tableau#connection-configuration)

```
Server: clickhouse.tinybird.co
Port: 443
Username: <WORKSPACE_NAME>  # Your workspace name
Password: <TOKEN>  # Your Tinybird auth token
SSL: Yes
```
See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

Select **Sign In** to see your data sources

<!-- <figure> -->
![Tableau ClickHouse® connection configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Ftableau-clickhouse-connection.png&w=3840&q=75)

<!-- <figcaption> -->
Tableau ClickHouse® connection configuration
<!-- </figcaption> -->

<!-- </figure> -->

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-tableau#learn-more)

- [ClickHouse Interface overview](/docs/forward/query-data/clickhouse-interface)
- Tinybird Auth Token management

---

URL: https://www.tinybird.co/docs/forward/guides/connect-superset
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Connect Superset to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect Apache Superset to Tinybird using the ClickHouse database driver, enabling modern data exploration and visualization."
inkeep:version: "forward"
---

# Connect Superset to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-superset#connect-superset-to-tinybird)

Apache Superset can connect to Tinybird using the ClickHouse® database driver, taking advantage of Tinybird's ClickHouse® HTTP protocol compatibility. Create interactive dashboards, explore data, and build advanced visualizations with your Tinybird data.

The ClickHouse® connection to Tinybird is read-only. You can use it to visualize and analyze data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-superset#prerequisites)

- Apache Superset instance (self-hosted or cloud)
- ClickHouse® database driver installed (`clickhouse-driver` Python package)
- A Tinybird workspace with data sources
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).

## Add Tinybird as a database (https://www.tinybird.co/docs/forward/guides/connect-superset#add-tinybird-as-a-database)

1. **Access Admin Interface**: In Superset, select **Settings** > **Database Connections**
2. **Add Database**: Select the **+ Database** button
3. **Select ClickHouse®**: Choose **ClickHouse®** from the database type dropdown

### Connection configuration (https://www.tinybird.co/docs/forward/guides/connect-superset#connection-configuration)

```
Host: clickhouse.tinybird.co
Port: 443
Database Name: <WORKSPACE_NAME>
Display Name: Tinybird prod # Choose a descriptive name
```
See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

### Authentication settings (https://www.tinybird.co/docs/forward/guides/connect-superset#authentication-settings)

```
Username: <WORKSPACE_NAME>  # not used for auth, helps you identify the connection
Password:    # Your Tinybird Auth Token
SSL: yes
```

1. **Save the database**: Select **Finish** to add the database to Superset

<!-- <figure> -->
![Superset ClickHouse® connection configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Fsuperset-clickhouse-connection.png&w=3840&q=75)

<!-- <figcaption> -->
Superset ClickHouse® connection configuration
<!-- </figcaption> -->

<!-- </figure> -->

#### Alternative SQLAlchemy URI format (https://www.tinybird.co/docs/forward/guides/connect-superset#alternative-sqlalchemy-uri-format)

```
clickhousedb://<WORKSPACE_NAME>:<TOKEN>@clickhouse.<REGION>.tinybird.co:443/<WORKSPACE_NAME>
```

## Test the connection (https://www.tinybird.co/docs/forward/guides/connect-superset#test-the-connection)

Once configured, test your connection by creating a simple query:

1. Select **+** to create new SQL query
2. Select your newly created database
3. Write a simple query to test the connection
If the connection is working, you should see the result.

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-superset#learn-more)

- [ClickHouse Interface overview](/docs/forward/query-data/clickhouse-interface)
- [Tinybird Auth Token management](/docs/forward/core-concepts/tokens)

---

URL: https://www.tinybird.co/docs/forward/guides/connect-redash
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Connect Redash to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect Redash to Tinybird using the ClickHouse HTTP interface."
inkeep:version: "forward"
---

# Connect Redash to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-redash#connect-redash-to-tinybird)

Redash can connect to Tinybird using the ClickHouse® Data Source, taking advantage of Tinybird's ClickHouse® HTTP protocol compatibility.

The ClickHouse® connection to Tinybird is read-only. You can use it to query and analyze data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-redash#prerequisites)

- [Redash](https://redash.io/) installed
- A Tinybird workspace with data sources
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).

## Create a new connection (https://www.tinybird.co/docs/forward/guides/connect-redash#create-a-new-connection)

1. Select **Data Sources > New Data Source**.
2. Select **ClickHouse®** from the data sources list.
3. Fill in the ClickHouse database credentials.

### Connection settings (https://www.tinybird.co/docs/forward/guides/connect-redash#connection-settings)

```
Name: Tinybird
URL: https://clickhouse.tinybird.co:443
Password: <TOKEN>  # Your Tinybird auth token
User: <WORKSPACE_NAME>  # Your workspace name
Database Name: <WORKSPACE_NAME>  # Your workspace name
```
See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-redash#learn-more)

- [ClickHouse Interface overview](/docs/forward/query-data/clickhouse-interface)
- Tinybird Auth Token management

---

URL: https://www.tinybird.co/docs/forward/guides/connect-powerbi
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Connect Microsoft Power BI to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect Microsoft Power BI to Tinybird using the ClickHouse® HTTP interface."
inkeep:version: "forward"
---

# Connect Microsoft Power BI to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-powerbi#connect-microsoft-power-bi-to-tinybird)

Microsoft Power BI can connect to Tinybird using the ClickHouse® Data Source, taking advantage of Tinybird's ClickHouse® HTTP protocol compatibility.

The ClickHouse® connection to Tinybird is read-only. You can use it to query and analyze data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-powerbi#prerequisites)

- [Microsoft Power BI](https://www.microsoft.com/en-us/download/details.aspx?id=58494)
- A Tinybird workspace with data sources
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).

## Install the ODBC driver (https://www.tinybird.co/docs/forward/guides/connect-powerbi#install-the-odbc-driver)

Download the most recent [ClickHouse® ODBC release](https://github.com/ClickHouse/clickhouse-odbc/releases).

Execute the .msi installer and follow the wizard.

## Create a new connection (https://www.tinybird.co/docs/forward/guides/connect-powerbi#create-a-new-connection)

1. On Power BI Desktop select **Get Data**

<!-- <figure> -->
![ClickHouse® connector configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Fpowerbi01.png&w=3840&q=75)

<!-- <figcaption> -->
ClickHouse® connector configuration
<!-- </figcaption> -->

<!-- </figure> -->

1. Select **ClickHouse®**.

<!-- <figure> -->
![ClickHouse® connector configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Fpowerbi02.png&w=3840&q=75)

<!-- <figcaption> -->
ClickHouse® connector configuration
<!-- </figcaption> -->

<!-- </figure> -->

1. Fill in the ClickHouse® database credentials.

### Connection settings (https://www.tinybird.co/docs/forward/guides/connect-powerbi#connection-settings)

```
Host: clickhouse.tinybird.co
Port: 443  # For HTTPS connections
Database Name: <WORKSPACE_NAME>  # Your workspace name
User name: <WORKSPACE_NAME>  # Your workspace name
Password: <TOKEN>  # Your Tinybird auth token
```
Use **DirectQuery** for querying Tinybird directly.

You should see the workspaces and tables in the Navigator view.

See the list of [ClickHouse® hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

## Troubleshooting (https://www.tinybird.co/docs/forward/guides/connect-powerbi#troubleshooting)

Failed to verify certificate chain: CA not trusted

<!-- <figure> -->
![ClickHouse® connector configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Fpowerbi03.png&w=3840&q=75)

<!-- <figcaption> -->
ClickHouse® connector configuration
<!-- </figcaption> -->

<!-- </figure> -->
You need to install the root and intermediate certificates in your local Windows machine (see [issue](https://github.com/ClickHouse/clickhouse-odbc/issues/443#issuecomment-2507243488))

Follow this steps:

Open the ClickHouse® host URL in your browser:

```
clickhouse.tinybird.co
```
Export certificates from the browser:

<!-- <figure> -->
![ClickHouse® connector configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Fpowerbi04.png&w=3840&q=75)

<!-- <figcaption> -->
ClickHouse® connector configuration
<!-- </figcaption> -->

<!-- </figure> -->

<!-- <figure> -->
![ClickHouse® connector configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Fpowerbi05.png&w=3840&q=75)

<!-- <figcaption> -->
ClickHouse® connector configuration
<!-- </figcaption> -->

<!-- </figure> -->

<!-- <figure> -->
![ClickHouse® connector configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Fpowerbi06.png&w=3840&q=75)

<!-- <figcaption> -->
ClickHouse® connector configuration
<!-- </figcaption> -->

<!-- </figure> -->

<!-- <figure> -->
![ClickHouse® connector configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Fpowerbi07.png&w=3840&q=75)

<!-- <figcaption> -->
ClickHouse® connector configuration
<!-- </figcaption> -->

<!-- </figure> -->
Install certificates in the local store:

- Win → write **mmc** → Run as administrator.
- File → Add/Remove Snap-in… → Certificates → Computer account → Local computer.
- Trusted Root Certification Authorities → Certificates → open the context menu → All Tasks → Import… → select the root .cer file → Place in this store.
- Intermediate Certification Authorities → Certificates → import the intermediate .cer.
Close, save and restart Power BI Desktop

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-powerbi#learn-more)

- [ClickHouse® Interface overview](/docs/forward/query-data/clickhouse-interface)
- Tinybird Auth Token management

---

URL: https://www.tinybird.co/docs/forward/guides/connect-mitzu
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Connect Mitzu to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect Mitzu to Tinybird using the ClickHouse HTTP interface."
inkeep:version: "forward"
---

# Connect Mitzu to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-mitzu#connect-mitzu-to-tinybird)

Mitzu can connect to Tinybird using the ClickHouse® Data Source, taking advantage of Tinybird's ClickHouse® HTTP protocol compatibility.

The ClickHouse® connection to Tinybird is read-only. You can use it to query and analyze data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-mitzu#prerequisites)

- A [Mitzu](https://mitzu.io/) account
- A Tinybird workspace with data sources
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).

## Create a new connection (https://www.tinybird.co/docs/forward/guides/connect-mitzu#create-a-new-connection)

1. Select **Connect Mitzu with your data warehouse**.
2. In the databases list select **ClickHouse®**.
3. Fill in the ClickHouse database credentials.

### Connection settings (https://www.tinybird.co/docs/forward/guides/connect-mitzu#connection-settings)

```
Host: https://clickhouse.tinybird.co:443/<WORKSPACE_NAME>
Username / Key: <WORKSPACE_NAME>  # Your workspace name
Password / Token: <TOKEN>  # Your Tinybird auth token
```
Then create a **New Dataset** selecting the previously created connection.

See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-mitzu#learn-more)

- [ClickHouse Interface overview](/docs/forward/query-data/clickhouse-interface)
- Tinybird Auth Token management

---

URL: https://www.tinybird.co/docs/forward/guides/connect-metabase
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Connect Metabase to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect Metabase to Tinybird using the ClickHouse® database driver, enabling business intelligence and data visualization."
inkeep:version: "forward"
---

# Connect Metabase to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-metabase#connect-metabase-to-tinybird)

Metabase can connect to Tinybird using the ClickHouse® database driver, taking advantage of Tinybird's ClickHouse® HTTP protocol compatibility. Create dashboards, explore data, and build business intelligence reports with your Tinybird data.

The ClickHouse® connection to Tinybird is read-only. You can use it to visualize and analyze data from your Tinybird data sources, but you cannot modify data through this connection.

Tinybird does not support Metabase's Roles or Impersonation features. You can access the data, but the Tinybird Token used to configure the Connection controls the permissions.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-metabase#prerequisites)

- A Tinybird workspace with data sources
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).
- ClickHouse® driver installed in Metabase (available by default in recent versions)

## Add Tinybird as a database (https://www.tinybird.co/docs/forward/guides/connect-metabase#add-tinybird-as-a-database)

1. **Access Admin Panel**: In Metabase, select the **gear icon** and select **Admin settings**
2. **Navigate to Databases**: Select **Databases**
3. **Add a database**: Select **Add database** button
4. **Select ClickHouse®**: Choose **ClickHouse®** from the database type dropdown

### Database configuration (https://www.tinybird.co/docs/forward/guides/connect-metabase#database-configuration)

Configure the connection with these settings:

```
Display name: Tinybird Production  # Choose a descriptive name
Host: clickhouse.tinybird.co
Port: 443  # Use 443 for HTTPS
Databases: <WORKSPACE_NAME> tinybird organization  # Your workspace name and service datasources
```
See the list of [ClickHouse® hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

### Authentication (https://www.tinybird.co/docs/forward/guides/connect-metabase#authentication)

```
Username: <WORKSPACE_NAME>  # not used for auth, helps you identify the connection
Password: <TOKEN>  # Your Tinybird Auth Token
```

### Advanced options (https://www.tinybird.co/docs/forward/guides/connect-metabase#advanced-options)

```
Use a secure connection (SSL): enabled
```

1. **Save the database**: Select **Save** to add the database to Metabase

<!-- <figure> -->
![Metabase ClickHouse® connection configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Fmetabase-clickhouse-connection.png&w=3840&q=75)

<!-- <figcaption> -->
Metabase ClickHouse® connection configuration
<!-- </figcaption> -->

<!-- </figure> -->

## Test the connection (https://www.tinybird.co/docs/forward/guides/connect-metabase#test-the-connection)

Once configured, test your connection by creating a simple query:

1. Select **+ New** to create new SQL query
2. Select your newly created database
3. Write a simple query to test the connection
If the connection is working, you should see the result.

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-metabase#learn-more)

- [ClickHouse® Interface overview](/docs/forward/query-data/clickhouse-interface)
- [Tinybird Auth Token management](/docs/forward/core-concepts/tokens)

---

URL: https://www.tinybird.co/docs/forward/guides/connect-luzmo
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Connect Luzmo to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect Luzmo to Tinybird using the ClickHouse HTTP interface."
inkeep:version: "forward"
---

# Connect Luzmo to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-luzmo#connect-luzmo-to-tinybird)

Luzmo can connect to Tinybird using the ClickHouse® Data Source, taking advantage of Tinybird's ClickHouse® HTTP protocol compatibility.

The ClickHouse® connection to Tinybird is read-only. You can use it to query and analyze data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-luzmo#prerequisites)

- A [Luzmo](https://luzmo.com/) account
- A Tinybird workspace with data sources
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).

## Create a new connection (https://www.tinybird.co/docs/forward/guides/connect-luzmo#create-a-new-connection)

1. Select **Connections > New connection**.
2. In the **Databases** select **ClickHouse®**.
3. Fill in the ClickHouse database credentials.

### Connection settings (https://www.tinybird.co/docs/forward/guides/connect-luzmo#connection-settings)

```
Host: https://clickhouse.tinybird.co:443/<WORKSPACE_NAME>
Username / Key: <WORKSPACE_NAME>  # Your workspace name
Password / Token: <TOKEN>  # Your Tinybird auth token
```
Then create a **New Dataset** selecting the previously created connection.

See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-luzmo#learn-more)

- [ClickHouse Interface overview](/docs/forward/query-data/clickhouse-interface)
- Tinybird Auth Token management

---

URL: https://www.tinybird.co/docs/forward/guides/connect-holistics
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Connect Holistics to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect Holistics to Tinybird using the ClickHouse HTTP interface."
inkeep:version: "forward"
---

# Connect Holistics to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-holistics#connect-holistics-to-tinybird)

Holistics can connect to Tinybird using the ClickHouse® Data Source, taking advantage of Tinybird's ClickHouse® HTTP protocol compatibility.

The ClickHouse® connection to Tinybird is read-only. You can use it to query and analyze data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-holistics#prerequisites)

- A [Holistics](https://holistics.io/) account
- A Tinybird workspace with data sources
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).

## Create a new connection (https://www.tinybird.co/docs/forward/guides/connect-holistics#create-a-new-connection)

1. Select **Connect your SQL database**.
2. In the databases list select **ClickHouse®**.
3. Fill in the ClickHouse database credentials.

### Connection settings (https://www.tinybird.co/docs/forward/guides/connect-holistics#connection-settings)

```
Display name: Tinybird
Host: clickhouse.tinybird.co
Port: 443
Database Name: <WORKSPACE_NAME>  # Your workspace name
Username: <WORKSPACE_NAME>  # Your workspace name
Password: <TOKEN>  # Your Tinybird auth token
Require SSL: Yes
```
Then create a **New Dataset** selecting the previously created connection.

See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-holistics#learn-more)

- [ClickHouse Interface overview](/docs/forward/query-data/clickhouse-interface)
- Tinybird Auth Token management

---

URL: https://www.tinybird.co/docs/forward/guides/connect-hex
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Connect Hex to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect Hex to Tinybird using the ClickHouse data source, enabling collaborative analytics and data science workflows."
inkeep:version: "forward"
---

# Connect Hex to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-hex#connect-hex-to-tinybird)

Hex can connect to Tinybird using the ClickHouse® Data Source, taking advantage of Tinybird's ClickHouse® HTTP protocol compatibility. Use Tinybird data in Hex's collaborative analytics platform for notebooks, dashboards, and data apps.

The ClickHouse® connection to Tinybird is read-only. You can use it to analyze and visualize data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-hex#prerequisites)

- A Tinybird workspace with data sources
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).

## Add Tinybird as a data source (https://www.tinybird.co/docs/forward/guides/connect-hex#add-tinybird-as-a-data-source)

1. **Navigate to Data browser**: In your Hex project, go to **Data browser** in the sidebar.
2. **Add a new connection**: In the **Warehouse** tab select **Add data connection**
3. **Select ClickHouse®**: Choose **ClickHouse®** from the list of available data connection
4. **Configure the connection** with these settings:

### Connection configuration (https://www.tinybird.co/docs/forward/guides/connect-hex#connection-configuration)

```
Name: Tinybird Production  # Choose a descriptive name
Host: clickhouse.tinybird.co
Port: 443
```
See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

### Authentication settings (https://www.tinybird.co/docs/forward/guides/connect-hex#authentication-settings)

```
Username: <WORKSPACE_NAME>  # not used for auth, helps you identify the connection
Password: <TOKEN>  # Your Tinybird Auth Token
Enables SSL for ClickHouse: yes
```

1. **Save the connection**: Select **Create connection** to save the data source

<!-- <figure> -->
![Hex ClickHouse® connection configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Fhex-clickhouse-connection.png&w=3840&q=75)

<!-- <figcaption> -->
Hex ClickHouse® connection configuration
<!-- </figcaption> -->

<!-- </figure> -->

## Test the connection (https://www.tinybird.co/docs/forward/guides/connect-hex#test-the-connection)

Once configured, test your connection by creating a simple query:

1. Add SQL query cell to your Notebook
2. Select your newly created data connection
3. Write a simple query to test the connection
If the connection is working, you should see the result.

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-hex#learn-more)

- [ClickHouse Interface overview](/docs/forward/query-data/clickhouse-interface)
- [Tinybird Auth Token management](/docs/forward/core-concepts/tokens)

---

URL: https://www.tinybird.co/docs/forward/guides/connect-grafana
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Connect Grafana to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect Grafana to Tinybird using Altinity and ClickHouse plugins, enabling the visualization of Tinybird data within Grafana."
inkeep:version: "forward"
---

# Connect Grafana to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-grafana#connect-grafana-to-tinybird)

Grafana can connect to Tinybird using ClickHouse® plugins, taking advantage of Tinybird's ClickHouse® HTTP protocol compatibility.

This guide covers how to set up Grafana with Tinybird using two popular ClickHouse® plugins:

- **[ClickHouse® plugin](https://grafana.com/grafana/plugins/grafana-clickhouse-datasource/)**
- **[Altinity plugin](https://grafana.com/grafana/plugins/vertamedia-clickhouse-datasource/)**
The ClickHouse® connection to Tinybird is read-only. You can use it to visualize data from your Tinybird data sources, but you cannot modify data through this connection.

Remember you can expose metrics like API Endpoints in Prometheus format. See an [example](https://www.tinybird.co/templates/tinybird-org-metrics).
And if you want to connect regular Endpoints and not Data Sources you can use Infinity data source plugin for Grafana.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-grafana#prerequisites)

- A Tinybird workspace
- A Grafana instance (local or cloud)
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).

## Install the ClickHouse® plugin (https://www.tinybird.co/docs/forward/guides/connect-grafana#install-the-clickhouse-plugin)

Install one of the ClickHouse® plugins in your Grafana instance:

1. **ClickHouse® plugin** - The official ClickHouse® plugin for Grafana
2. **Altinity plugin** - developed by Vertamedia, maintaned by Altinity since 2020
Select one based on your needs and install it through the Grafana plugin catalog or using grafana-cli.

## Configure the ClickHouse® plugin (https://www.tinybird.co/docs/forward/guides/connect-grafana#configure-the-clickhouse-plugin)

If you're using the official ClickHouse® plugin:

1. Go to **Configuration** > **Data Sources** in Grafana
2. Select **Add data source** and select **ClickHouse®**
3. Configure the connection with these settings:

```
Server address: clickhouse.tinybird.co # your host with 'clickhouse' subdomain instead of 'api', like clickhouse.eu-west-1.aws.tinybird.co
Server port: 443
Protocol: HTTP
Secure Connection: yes
```
See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

1. Add your authentication credentials:

```
Username: <WORKSPACE_NAME> # not used for auth, helps you identify the connection
Password: <TOKEN> # your Tinybird Auth Token
```

1. Select **Save & Test** to verify the connection

<!-- <figure> -->
![ClickHouse® plugin configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Fgrafana-plugin-1.png&w=3840&q=75)

<!-- <figcaption> -->
ClickHouse® plugin configuration
<!-- </figcaption> -->

<!-- </figure> -->

## Configure the ClickHouse® plugin for OpenTelemetry (https://www.tinybird.co/docs/forward/guides/connect-grafana#configure-the-clickhouse-plugin-for-opentelemetry)

Use the [OpenTelemetry template](https://tinybird.co/templates/opentelemetry) to build an end to end observability solution with OpenTelemetry and Tinybird.

This section assumes you have the OpenTelemetry template deployed in a Tinybird Workspace.

Once you've configured the ClickHouse plugin, edit the Grafana Data Source. The `Logs` and `Traces` configurations appear. Select the `Use OTel` checkboxes and set your Tinybird Workspace name as `Default database`.

<!-- <figure> -->
![ClickHouse® plugin configuration for OpenTelemetry](/docs/_next/image?url=%252Fdocs%252Fimg%252Fgrafana-plugin-3.png&w=3840&q=75)

<!-- <figcaption> -->
ClickHouse® plugin configuration for OpenTelemetry
<!-- </figcaption> -->

<!-- </figure> -->
From the `Dashboards` tab, you can import the `Simple ClickHouse OTel dashboard` as a starting point to analyze your OpenTelemetry logs and traces.

<!-- <figure> -->
![ClickHouse® plugin configuration for OpenTelemetry](/docs/_next/image?url=%252Fdocs%252Fimg%252Fgrafana-plugin-4.png&w=3840&q=75)

<!-- <figcaption> -->
ClickHouse® OTel dashboard
<!-- </figcaption> -->

<!-- </figure> -->
You can add and edit visualization panels by writing new SQL queries to analyze your logs, metrics and traces.

<!-- <figure> -->
![ClickHouse® plugin configuration for OpenTelemetry](/docs/_next/image?url=%252Fdocs%252Fimg%252Fgrafana-plugin-6.png&w=3840&q=75)

<!-- <figcaption> -->
ClickHouse® Simple OTel dashboard
<!-- </figcaption> -->

<!-- </figure> -->
Use the `Explore` feature in Grafana to filter and analyze your OpenTelemetry logs:

<!-- <figure> -->
![ClickHouse® plugin configuration for OpenTelemetry](/docs/_next/image?url=%252Fdocs%252Fimg%252Fgrafana-plugin-5.png&w=3840&q=75)

<!-- <figcaption> -->
OpenTelemetry logs explorer
<!-- </figcaption> -->

<!-- </figure> -->

### Provisioning the ClickHouse data source (https://www.tinybird.co/docs/forward/guides/connect-grafana#provisioning-the-clickhouse-data-source)

You can also provision the ClickHouse data source using a YAML file, which is useful for managing your Grafana configuration as code.

This snippet uses the following environment variables, which you must set before applying it:

- `TINYBIRD_WORKSPACE_NAME`: Your workspace name
- `TINYBIRD_TOKEN`: A scoped or admin token
- `TINYBIRD_CLICKHOUSE_HOST`: Your ClickHouse HTTP interface host
See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

```
apiVersion: 1

datasources:
  - name: Tinybird OTel Cloud
    uid: tinybird-otel-cloud
    type: grafana-clickhouse-datasource
    access: proxy
    editable: true
    isDefault: false
    jsonData:
      defaultDatabase: ${TINYBIRD_WORKSPACE_NAME}
      host: ${TINYBIRD_CLICKHOUSE_HOST}
      logs:
        contextColumns: []
        defaultDatabase: ${TINYBIRD_WORKSPACE_NAME}
        defaultTable: otel_logs
        otelEnabled: true
        otelVersion: latest
        selectContextColumns: true
      pdcInjected: false
      port: 443
      protocol: http
      secure: true
      tlsSkipVerify: false
      traces:
        defaultDatabase: ${TINYBIRD_WORKSPACE_NAME}
        defaultTable: otel_traces
        durationUnit: nanoseconds
        otelEnabled: true
        otelVersion: latest
      useOtel: true
      username: otel
      version: 4.10.2
    secureJsonData:
      password: ${TINYBIRD_TOKEN}
```

## Configure the Altinity plugin (https://www.tinybird.co/docs/forward/guides/connect-grafana#configure-the-altinity-plugin)

If you're using the Altinity plugin:

1. Go to **Configuration** > **Data Sources** in Grafana
2. Select **Add data source** and select **Altinity plugin for ClickHouse®**
3. Configure the connection with these settings:

```
URL: https://clickhouse.tinybird.co
Access: Server
```
See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

1. Select Basic auth and add credentials:

```
User: <WORKSPACE_NAME> # not used for auth, helps you identify the connection
Password: <TOKEN> # your Tinybird Auth Token
```

1. Select **Save & Test** to verify the connection

<!-- <figure> -->
![Altinity plugin configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Fgrafana-plugin-2.png&w=3840&q=75)

<!-- <figcaption> -->
Altinity plugin configuration
<!-- </figcaption> -->

<!-- </figure> -->

## Test the connection (https://www.tinybird.co/docs/forward/guides/connect-grafana#test-the-connection)

Once configured, test your connection by creating a simple query:

1. Create a new dashboard and add a panel
2. Select your newly created data source
3. Write a simple query to test the connection
If the connection is working, you should see the result.

## Databases and tables (https://www.tinybird.co/docs/forward/guides/connect-grafana#databases-and-tables)

The SQL API exposes four databases: organization, system, tinybird, and workspace name:

- organization: for the Organization data sources, like `organization.workspaces`, `organization.pipe_stats_rt`
- system: for `system.tables`, `system.columns`
- tinybird: for the Workspace Service data sources like `tinybird.datasources_ops_log`, `tinybird.pipe_stats_rt`
- your workspace name: for your Workspace data sources.

## Next steps (https://www.tinybird.co/docs/forward/guides/connect-grafana#next-steps)

- Explore Grafana's visualization options with your Tinybird data
- Set up alerts based on your real-time data
- Create dashboards that combine data from multiple sources
- Use Grafana variables to make your dashboards interactive

## Troubleshooting (https://www.tinybird.co/docs/forward/guides/connect-grafana#troubleshooting)

**Connection failed**: Verify that your Auth Token has read permissions and that the host URL is correct for your region.

**Query timeout**: For large datasets, consider using Materialized Views or adding appropriate filters to your queries.

**Authentication errors**: Double-check that your Auth Token is correct and that you've entered it into the Password field.

**Write operations fail**: Remember that the ClickHouse® connection to Tinybird is read-only. You cannot use it to insert, update, or delete data.

---

URL: https://www.tinybird.co/docs/forward/guides/connect-fabi
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Connect Fabi.ai to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect Fabi.ai to Tinybird using the ClickHouse HTTP interface."
inkeep:version: "forward"
---

# Connect Fabi.ai to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-fabi#connect-fabi-ai-to-tinybird)

Fabi.ai can connect to Tinybird using the ClickHouse® Data Source, taking advantage of Tinybird's ClickHouse® HTTP protocol compatibility.

The ClickHouse® connection to Tinybird is read-only. You can use it to query and analyze data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-fabi#prerequisites)

- A [Fabi.ai](https://fabi.ai/) account
- A Tinybird workspace with data sources
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).

## Create a new connection (https://www.tinybird.co/docs/forward/guides/connect-fabi#create-a-new-connection)

1. Select **Connect data warehouse**.
2. In the drop-down, select **ClickHouse®** from the SQL databases list.
3. Fill in the ClickHouse database credentials.

### Connection settings (https://www.tinybird.co/docs/forward/guides/connect-fabi#connection-settings)

```
Host: clickhouse.tinybird.co
Port: 443  # For HTTPS connections
Database Name: <WORKSPACE_NAME>  # Your workspace name
Username: <WORKSPACE_NAME>  # Your workspace name
Password: <TOKEN>  # Your Tinybird auth token
```
See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-fabi#learn-more)

- [ClickHouse Interface overview](/docs/forward/query-data/clickhouse-interface)
- Tinybird Auth Token management

---

URL: https://www.tinybird.co/docs/forward/guides/connect-draxlr
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Connect Draxlr to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect Draxlr to Tinybird using the ClickHouse HTTP interface."
inkeep:version: "forward"
---

# Connect Draxlr to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-draxlr#connect-draxlr-to-tinybird)

Draxlr can connect to Tinybird using the ClickHouse® Data Source, taking advantage of Tinybird's ClickHouse® HTTP protocol compatibility.

The ClickHouse® connection to Tinybird is read-only. You can use it to query and analyze data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-draxlr#prerequisites)

- A [Draxlr](https://www.draxlr.com/) account
- A Tinybird workspace with data sources
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).

## Create a new connection (https://www.tinybird.co/docs/forward/guides/connect-draxlr#create-a-new-connection)

1. Select **Connect Database**.
2. In the connection wizard, select **ClickHouse®** from the SQL databases list.
3. Fill in the ClickHouse database credentials.

### Connection settings (https://www.tinybird.co/docs/forward/guides/connect-draxlr#connection-settings)

```
Connection name: Tinybird
Host: clickhouse.tinybird.co
Port: 443  # For HTTPS connections
User: <WORKSPACE_NAME>  # Your workspace name
Password: <TOKEN>  # Your Tinybird auth token
Database: <WORKSPACE_NAME>  # Your workspace name
Use TSL: Yes
```
See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-draxlr#learn-more)

- [ClickHouse Interface overview](/docs/forward/query-data/clickhouse-interface)
- Tinybird Auth Token management

---

URL: https://www.tinybird.co/docs/forward/guides/connect-deepnote
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Connect Deepnote to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect Deepnote to Tinybird using the ClickHouse HTTP interface."
inkeep:version: "forward"
---

# Connect Deepnote to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-deepnote#connect-deepnote-to-tinybird)

Deepnote can connect to Tinybird using the ClickHouse® Data Source, taking advantage of Tinybird's ClickHouse® HTTP protocol compatibility.

The ClickHouse® connection to Tinybird is read-only. You can use it to query and analyze data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-deepnote#prerequisites)

- A [Deepnote](https://deepnote.com/) account
- A Tinybird workspace with data sources
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).

## Create a new connection (https://www.tinybird.co/docs/forward/guides/connect-deepnote#create-a-new-connection)

1. Select **Integrations > Add integration**.
2. In the **Create new integration** view, select **ClickHouse®** from the Data warehouses & lakes list.
3. Fill in the ClickHouse database credentials.

### Connection settings (https://www.tinybird.co/docs/forward/guides/connect-deepnote#connection-settings)

```
Integration name: Tinybird
Host name: clickhouse.tinybird.co
Port: 443  # For HTTPS connections
Database: <WORKSPACE_NAME>  # Your workspace name
Username: <WORKSPACE_NAME>  # Your workspace name
Password: <TOKEN>  # Your Tinybird auth token
Enforce SSL encryption: yes
```
See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-deepnote#learn-more)

- [ClickHouse Interface overview](/docs/forward/query-data/clickhouse-interface)
- Tinybird Auth Token management

---

URL: https://www.tinybird.co/docs/forward/guides/connect-dbeaver
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Connect DBeaver to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect DBeaver to Tinybird using the ClickHouse JDBC driver, enabling SQL development and database management."
inkeep:version: "forward"
---

# Connect DBeaver to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-dbeaver#connect-dbeaver-to-tinybird)

DBeaver can connect to Tinybird using the ClickHouse® JDBC driver, taking advantage of Tinybird's ClickHouse® HTTP protocol compatibility. Use DBeaver's powerful SQL development and database management features with your Tinybird data.

The ClickHouse® connection to Tinybird is read-only. You can use it to query and analyze data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-dbeaver#prerequisites)

- DBeaver Community or Enterprise Edition
- A Tinybird workspace with data sources
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).

## Create a new connection (https://www.tinybird.co/docs/forward/guides/connect-dbeaver#create-a-new-connection)

1. Select the **+** icon and select **New Connection**.
2. In the connection wizard, select **ClickHouse®** from the database list.
3. Select **Next** to proceed to the connection settings.

### Connection settings (https://www.tinybird.co/docs/forward/guides/connect-dbeaver#connection-settings)

Configure the main connection parameters:

```
Host: clickhouse.tinybird.co
Port: 443  # For HTTPS connections
Database: <WORKSPACE_NAME>  # Your workspace name
Connection name: Tinybird production
```
See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

### Authentication (https://www.tinybird.co/docs/forward/guides/connect-dbeaver#authentication)

```
User name: <WORKSPACE_NAME>  # Not used for authentication, for identification only
User password: <TOKEN>  # Your Tinybird Auth Token
Save credentials for all users with access: yes
```

### SSL configuration (https://www.tinybird.co/docs/forward/guides/connect-dbeaver#ssl-configuration)

1. Select the **SSL** tab.
2. Check **Enable SSL**. No certificates are required.

### Test and save (https://www.tinybird.co/docs/forward/guides/connect-dbeaver#test-and-save)

1. **Test Connection**: Select **Test Connection** to verify the setup
2. **Create connection**: Select **Create** to create the connection

<!-- <figure> -->
![DBeaver ClickHouse® connection configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Fdbeaver-clickhouse-connection-1.png&w=3840&q=75)

<!-- <figcaption> -->
DBeaver ClickHouse® connection configuration
<!-- </figcaption> -->

<!-- </figure> -->

<!-- <figure> -->
![DBeaver ClickHouse® connection SSL configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Fdbeaver-clickhouse-connection-2.png&w=3840&q=75)

<!-- <figcaption> -->
DBeaver ClickHouse® connection SSL configuration
<!-- </figcaption> -->

<!-- </figure> -->

## Test the connection (https://www.tinybird.co/docs/forward/guides/connect-dbeaver#test-the-connection)

Once configured, test your connection by creating a simple query:

1. Select your newly created data connection and SQL Editor
2. In the **Database Authentication** dialog, select **Login** and leave the SSL fields empty.
3. Write a simple query to test the connection
If the connection is working, you should see the result.

## Troubleshooting (https://www.tinybird.co/docs/forward/guides/connect-dbeaver#troubleshooting)

**Transport error**

```
SQL Error [22000]: Code: 0. DB::Exception: <Unreadable error message> (transport error: 400)
```
Make sure you enabled SSL, in some DBeaver distributions you have to manually change `ssl` to `true` in the Driver properties.

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-dbeaver#learn-more)

- [ClickHouse Interface overview](/docs/forward/query-data/clickhouse-interface)
- [Tinybird Auth Token management](/docs/forward/core-concepts/tokens)

---

URL: https://www.tinybird.co/docs/forward/guides/connect-datagrip
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Connect DataGrip to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect DataGrip to Tinybird using the ClickHouse database driver, enabling advanced SQL development and database management."
inkeep:version: "forward"
---

# Connect DataGrip to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-datagrip#connect-datagrip-to-tinybird)

Connect DataGrip to Tinybird using the ClickHouse® database driver to take advantage of Tinybird's ClickHouse® HTTP protocol compatibility. Use DataGrip's advanced SQL development features, including intelligent code completion, on-the-fly error detection, and powerful refactoring capabilities with your Tinybird data.

The ClickHouse® connection to Tinybird is read-only. You can use it to query and analyze data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-datagrip#prerequisites)

- DataGrip IDE (part of JetBrains IDEs)
- A Tinybird workspace with data sources
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).

## Create a new data source (https://www.tinybird.co/docs/forward/guides/connect-datagrip#create-a-new-data-source)

1. Open DataGrip and select the **+** icon in the Database Explorer
2. Select **Data Source** > **ClickHouse**
3. If prompted, install or update the ClickHouse driver

### General settings (https://www.tinybird.co/docs/forward/guides/connect-datagrip#general-settings)

Configure the main connection parameters in the **General** tab:

```
Host: clickhouse.tinybird.co
Port: 443  # For HTTPS connections
User: <WORKSPACE_NAME>  # Not used for authentication, for identification only
Password: <TOKEN>  # Your Tinybird Auth Token
```
See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

<!-- <figure> -->
![DataGrip configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Fdata-grip-1.png&w=3840&q=75)

<!-- <figcaption> -->
DataGrip configuration
<!-- </figcaption> -->

<!-- </figure> -->

### SSH/SSL settings (https://www.tinybird.co/docs/forward/guides/connect-datagrip#sshssl-settings)

1. Select the **SSH/SSL** tab
2. Check **Use SSL**
3. Leave all other SSL settings at their defaults (no custom certificates required)

<!-- <figure> -->
![DataGrip configuration](/docs/_next/image?url=%252Fdocs%252Fimg%252Fdata-grip-2.png&w=3840&q=75)

<!-- <figcaption> -->
DataGrip configuration
<!-- </figcaption> -->

<!-- </figure> -->

### Test and save (https://www.tinybird.co/docs/forward/guides/connect-datagrip#test-and-save)

1. **Test Connection**: Select **Test Connection** to verify the setup
2. **Apply**: Select **Apply** and then **OK** to save the connection

## Explore your data (https://www.tinybird.co/docs/forward/guides/connect-datagrip#explore-your-data)

Once connected, you can:

1. **Browse databases**: Expand your data source to see available databases:

  - Your workspace database (`<WORKSPACE_NAME>`)
  - Service data sources (`tinybird`, `organization`)
  - System tables (`system`)
2. **View tables and schemas**: Navigate through your Data Sources and examine their structure
3. **Execute queries**: Open the context menu for your Data Source and select **New** > **Query Console** to start writing SQL.

## Test the connection (https://www.tinybird.co/docs/forward/guides/connect-datagrip#test-the-connection)

Create a simple query to verify your connection:

1. Open a new Query Console for your Tinybird data source
2. Write a simple query to test the connection:

```
SELECT *
FROM system.tables
WHERE database = '<WORKSPACE_NAME>'
LIMIT 10
```
Replace `<WORKSPACE_NAME>` with your actual workspace name. If the connection is working, you should see a list of your Data Sources.

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-datagrip#learn-more)

- [ClickHouse Interface overview](/docs/forward/query-data/clickhouse-interface)
- [Tinybird Auth Token management](/docs/forward/core-concepts/tokens)
- [DataGrip documentation](https://www.jetbrains.com/help/datagrip/)

---

URL: https://www.tinybird.co/docs/forward/guides/connect-clickhouse-python
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Connect ClickHouse® Python Client to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect the official ClickHouse Python client to Tinybird using the ClickHouse HTTP interface, enabling programmatic data access in Python applications."
inkeep:version: "forward"
---

# Connect ClickHouse® Python client to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-python#connect-clickhouse-python-client-to-tinybird)

The official ClickHouse Python client can connect to Tinybird using the ClickHouse® HTTP protocol compatibility. Use it to query your Tinybird Data Sources programmatically from Python apps with excellent performance and integration with the Python data ecosystem.

The ClickHouse® connection to Tinybird is read-only. You can use it to query and analyze data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-python#prerequisites)

- Python 3.8 or later
- A Tinybird workspace with data sources
- A Tinybird Auth Token with read permissions for the workspace data sources

## Installation (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-python#installation)

Install the official ClickHouse Python client:

```
pip install clickhouse-connect
```

## Configuration (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-python#configuration)

Create a client instance with the following configuration:

```
import clickhouse_connect

client = clickhouse_connect.get_client(
    interface='https',
    host='clickhouse.tinybird.co',
    username='<WORKSPACE_NAME>',  # Optional, for identification
    password='<TOKEN>',  # Your Tinybird Auth Token
    port=443,
)
```
See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

## Test the connection (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-python#test-the-connection)

Create a simple query to verify your connection:

```
def test_connection():
    try:
        result = client.query('SELECT database, name, engine FROM system.tables LIMIT 5')
        print("Connection successful!")

        # Print column names
        print(" | ".join(result.column_names))

        # Print data rows
        for row in result.result_rows:
            print(" | ".join(str(cell) for cell in row))

    except Exception as error:
        print(f"Connection failed: {error}")

test_connection()
```

## Query your data (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-python#query-your-data)

Once connected, you can query your Tinybird data sources:

```
# Query a specific data source
def query_data_source():
    result = client.query('SELECT * FROM your_data_source_name LIMIT 10')
    return result.result_rows

# Query with parameters
def query_with_params(start_date, limit):
    result = client.query(
        """
        SELECT timestamp, user_id, event_name
        FROM events
        WHERE timestamp >= %(start_date)s
        LIMIT %(limit)s
        """,
        parameters={
            'start_date': start_date,
            'limit': limit
        }
    )
    return result.result_rows

# Convert to pandas DataFrame
def query_to_dataframe():
    import pandas as pd

    result = client.query('SELECT * FROM your_data_source_name LIMIT 100')

    # Create DataFrame from result
    df = pd.DataFrame(result.result_rows, columns=result.column_names)
    return df
```

## Observability (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-python#observability)

You can also explore Tinybird's observability data from internal service data sources:

```
# View recent API endpoint usage and performance
def query_recent_usage():
    result = client.query(
        """
        SELECT
            start_datetime,
            pipe_name,
            duration,
            result_rows,
            read_bytes,
            status_code
        FROM tinybird.pipe_stats_rt
        WHERE start_datetime >= now() - INTERVAL 1 DAY
        ORDER BY start_datetime DESC
        LIMIT 10
        """
    )

    print("Recent API endpoint usage:")
    for row in result.result_rows:
        start_time, pipe_name, duration, result_rows, read_bytes, status_code = row
        duration_ms = duration * 1000
        print(f"[{start_time}] {pipe_name} - {duration_ms:.2f}ms, {result_rows} rows, status {status_code}")

# Analyze endpoint performance metrics
def query_performance_metrics():
    result = client.query(
        """
        SELECT
            pipe_name,
            count() as request_count,
            avg(duration) as avg_duration_ms,
            avg(result_rows) as avg_result_rows,
            sum(read_bytes) as total_bytes_read
        FROM tinybird.pipe_stats_rt
        WHERE start_datetime >= now() - INTERVAL 1 HOUR
        GROUP BY pipe_name
        ORDER BY request_count DESC
        """
    )

    print("\nEndpoint performance metrics (last hour):")
    for row in result.result_rows:
        pipe_name, request_count, avg_duration, avg_result_rows, total_bytes_read = row
        avg_duration_ms = avg_duration * 1000
        print(f"{pipe_name}: {request_count} requests, {avg_duration_ms:.2f}ms avg, {avg_result_rows:.0f} avg rows, {total_bytes_read:,} bytes")

# Convert to pandas DataFrame for analysis
def query_to_dataframe():
    import pandas as pd

    result = client.query(
        """
        SELECT
            start_datetime,
            pipe_name,
            duration,
            result_rows,
            read_bytes,
            status_code
        FROM tinybird.pipe_stats_rt
        WHERE start_datetime >= now() - INTERVAL 1 DAY
        ORDER BY start_datetime DESC
        """
    )

    df = pd.DataFrame(result.result_rows, columns=result.column_names)
    return df
```

## Working with data types (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-python#working-with-data-types)

Handle various ClickHouse data types:

```
from datetime import datetime
from typing import Any, List, Dict

def handle_query_results():
    result = client.query(
        """
        SELECT
            toUInt32(42) as number,
            'hello world' as text,
            now() as timestamp,
            [1, 2, 3] as array_col,
            map('key1', 'value1', 'key2', 'value2') as map_col
        """
    )

    for row in result.result_rows:
        number, text, timestamp, array_col, map_col = row
        print(f"Number: {number} (type: {type(number)})")
        print(f"Text: {text} (type: {type(text)})")
        print(f"Timestamp: {timestamp} (type: {type(timestamp)})")
        print(f"Array: {array_col} (type: {type(array_col)})")
        print(f"Map: {map_col} (type: {type(map_col)})")
```

## Error handling (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-python#error-handling)

Handle connection and query errors appropriately:

```
from clickhouse_connect.driver.exceptions import ClickHouseError

def safe_query(query, parameters=None):
    try:
        result = client.query(query, parameters=parameters)
        return result
    except ClickHouseError as error:
        print(f"ClickHouse error: {error}")
        print(f"Error code: {error.code}")
        raise
    except Exception as error:
        print(f"Connection error: {error}")
        raise

# Example usage
try:
    result = safe_query('SELECT COUNT(*) FROM your_data_source')
    count = result.result_rows[0][0]
    print(f"Total rows: {count:,}")
except Exception:
    print("Query failed")
```

## Advanced usage (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-python#advanced-usage)

### Streaming large results (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-python#streaming-large-results)

For large datasets, use streaming to avoid memory issues:

```
def stream_large_query():
    # Stream results for large datasets
    result = client.query_row_block_stream(
        'SELECT * FROM large_data_source',
        settings={'max_block_size': 10000}
    )

    for block in result:
        # Process each block of rows
        for row in block:
            process_row(row)

def process_row(row):
    # Your row processing logic here
    pass
```

### Custom settings (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-python#custom-settings)

Pass ClickHouse settings for query optimization:

```
# Query with custom settings
result = client.query(
    'SELECT * FROM system.numbers LIMIT 5',
    settings={
        'max_result_rows': 5,
        'max_execution_time': 10
    }
)
```

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-python#learn-more)

- [ClickHouse Interface overview](/docs/forward/query-data/clickhouse-interface)
- [Tinybird Auth Token management](/docs/forward/core-concepts/tokens)
- [ClickHouse Python Client documentation](https://github.com/ClickHouse/clickhouse-connect)

---

URL: https://www.tinybird.co/docs/forward/guides/connect-clickhouse-js
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Connect ClickHouse® JS Client to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect the official ClickHouse JavaScript client to Tinybird using the ClickHouse HTTP interface, enabling programmatic data access in Node.js applications."
inkeep:version: "forward"
---

# Connect ClickHouse® JS client to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-js#connect-clickhouse-js-client-to-tinybird)

The official ClickHouse JavaScript client can connect to Tinybird using the ClickHouse® HTTP protocol compatibility. Use it to query your Tinybird Data Sources programmatically from Node.js apps with full type safety and modern JavaScript features.

The ClickHouse® connection to Tinybird is read-only. You can use it to query and analyze data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-js#prerequisites)

- Node.js 16.0 or later
- A Tinybird workspace with data sources
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).

## Installation (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-js#installation)

Install the official ClickHouse JavaScript client:

```
npm install @clickhouse/client
```

## Configuration (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-js#configuration)

Create a client instance with the following configuration:

```
import { createClient } from '@clickhouse/client'

const client = createClient({
  url: 'https://clickhouse.tinybird.co',
  username: '<WORKSPACE_NAME>', // Optional, for identification
  password: '<TOKEN>', // Your Tinybird Auth Token
})
```
See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

## Test the connection (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-js#test-the-connection)

Create a simple query to verify your connection:

```
async function testConnection() {
  try {
    const resultSet = await client.query({
      query: 'SELECT database, name, engine FROM system.tables LIMIT 5',
    })

    const data = await resultSet.json()
    console.table(data.data)
  } catch (error) {
    console.error('Connection failed:', error)
  }
}

testConnection()
```

## Query your data (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-js#query-your-data)

Once connected, you can query your Tinybird data sources:

```
// Query a specific data source
async function queryDataSource() {
  const resultSet = await client.query({
    query: 'SELECT * FROM your_data_source_name LIMIT 10',
  })

  const data = await resultSet.json()
  return data.data
}

// Query with parameters
async function queryWithParams() {
  const resultSet = await client.query({
    query: `
      SELECT timestamp, user_id, event_name
      FROM events
      WHERE timestamp >= {start_date:DateTime}
      LIMIT {limit:UInt32}
    `,
    query_params: {
      start_date: '2024-01-01 00:00:00',
      limit: 100,
    },
  })

  return await resultSet.json()
}
```

## Observability (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-js#observability)

You can also explore Tinybird's observability data from internal service data sources:

```
// View recent API endpoint usage and performance
const recentUsage = await client.query({
  query: `
    SELECT
      start_datetime,
      pipe_name,
      duration,
      result_rows,
      read_bytes,
      status_code
    FROM tinybird.pipe_stats_rt
    WHERE start_datetime >= now() - INTERVAL 1 DAY
    ORDER BY start_datetime DESC
    LIMIT 10
  `
})

// Analyze endpoint performance metrics
const performanceMetrics = await client.query({
  query: `
    SELECT
      pipe_name,
      count() as request_count,
      avg(duration) as avg_duration_ms,
      avg(result_rows) as avg_result_rows,
      sum(read_bytes) as total_bytes_read
    FROM tinybird.pipe_stats_rt
    WHERE start_datetime >= now() - INTERVAL 1 HOUR
    GROUP BY pipe_name
    ORDER BY request_count DESC
  `
})

const usage = await recentUsage.json()
const metrics = await performanceMetrics.json()
console.table(usage.data)
console.table(metrics.data)
```

## Error handling (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-js#error-handling)

Handle connection and query errors appropriately:

```
import { ClickHouseError } from '@clickhouse/client'

async function safeQuery(query) {
  try {
    const resultSet = await client.query({ query })
    return await resultSet.json()
  } catch (error) {
    if (error instanceof ClickHouseError) {
      console.error('ClickHouse error:', error.message)
      console.error('Code:', error.code)
    } else {
      console.error('Connection error:', error.message)
    }
    throw error
  }
}
```

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-js#learn-more)

- [ClickHouse Interface overview](/docs/forward/query-data/clickhouse-interface)
- [Tinybird Auth Token management](/docs/forward/core-concepts/tokens)
- [ClickHouse JavaScript Client documentation](https://github.com/ClickHouse/clickhouse-js)

---

URL: https://www.tinybird.co/docs/forward/guides/connect-clickhouse-go
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Connect ClickHouse® Go Client to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect the official ClickHouse Go client to Tinybird using the ClickHouse HTTP interface, enabling programmatic data access in Go applications."
inkeep:version: "forward"
---

# Connect ClickHouse® Go client to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-go#connect-clickhouse-go-client-to-tinybird)

The official ClickHouse Go client can connect to Tinybird using the ClickHouse® HTTP protocol compatibility. Use it to query your Tinybird Data Sources programmatically from Go apps with high performance and type safety.

The ClickHouse® connection to Tinybird is read-only. You can use it to query and analyze data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-go#prerequisites)

- Go 1.19 or later
- A Tinybird workspace with data sources
- A Tinybird Auth Token with read permissions for the workspace data sources

## Installation (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-go#installation)

Add the ClickHouse Go client to your project:

```
go mod init your-project
go get github.com/ClickHouse/clickhouse-go/v2
```

## Configuration (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-go#configuration)

Create a connection with the following configuration:

```
package main

import (
    "context"
    "crypto/tls"
    "log"

    "github.com/ClickHouse/clickhouse-go/v2"
    "github.com/ClickHouse/clickhouse-go/v2/lib/driver"
)

func connect() (driver.Conn, error) {
    conn, err := clickhouse.Open(&clickhouse.Options{
        Addr:     []string{"clickhouse.tinybird.co:443"},
        Protocol: clickhouse.HTTP, // Use HTTP protocol
        Auth: clickhouse.Auth{
            Database: "default",
            Username: "<WORKSPACE_NAME>", // Optional, for identification
            Password: "<TOKEN>", // Your Tinybird Auth Token
        },
        TLS: &tls.Config{
            InsecureSkipVerify: false, // Set to true for testing only
        },
        ClientInfo: clickhouse.ClientInfo{
            Products: []struct {
                Name    string
                Version string
            }{
                {Name: "your-app-name", Version: "1.0.0"},
            },
        },
    })
    if err != nil {
        return nil, err
    }

    // Test the connection
    if err := conn.Ping(context.Background()); err != nil {
        return nil, err
    }

    return conn, nil
}
```
See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

## Test the connection (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-go#test-the-connection)

Create a simple query to verify your connection:

```
func main() {
    conn, err := connect()
    if err != nil {
        log.Fatal(err)
    }
    defer conn.Close()

    ctx := context.Background()

    // Test query
    rows, err := conn.Query(ctx, "SELECT database, name, engine FROM system.tables LIMIT 5")
    if err != nil {
        log.Fatal(err)
    }
    defer rows.Close()

    for rows.Next() {
        var database, name, engine string
        if err := rows.Scan(&database, &name, &engine); err != nil {
            log.Fatal(err)
        }
        fmt.Printf("Database: %s, Table: %s, Engine: %s\n", database, name, engine)
    }
}
```

## Query your data (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-go#query-your-data)

Once connected, you can query your Tinybird data sources:

```
// Query a specific data source
func queryDataSource(conn driver.Conn) error {
    ctx := context.Background()

    rows, err := conn.Query(ctx, "SELECT * FROM your_data_source_name LIMIT 10")
    if err != nil {
        return err
    }
    defer rows.Close()

    // Process results
    for rows.Next() {
        var col1 string
        var col2 int64
        // Scan based on your data source schema
        if err := rows.Scan(&col1, &col2); err != nil {
            return err
        }
        fmt.Printf("Col1: %s, Col2: %d\n", col1, col2)
    }

    return rows.Err()
}

// Query with parameters using named parameters
func queryWithParams(conn driver.Conn, startDate string, limit int) error {
    ctx := context.Background()

    query := `
        SELECT timestamp, user_id, event_name
        FROM events
        WHERE timestamp >= @start_date
        LIMIT @limit
    `

    rows, err := conn.Query(ctx, query,
        clickhouse.Named("start_date", startDate),
        clickhouse.Named("limit", limit),
    )
    if err != nil {
        return err
    }
    defer rows.Close()

    // Process results...
    return nil
}
```

## Observability (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-go#observability)

You can also explore Tinybird's observability data from internal service data sources:

```
// View recent API endpoint usage and performance
func queryRecentUsage(conn driver.Conn) error {
    ctx := context.Background()

    query := `
        SELECT
            start_datetime,
            pipe_name,
            duration,
            result_rows,
            read_bytes,
            status_code
        FROM tinybird.pipe_stats_rt
        WHERE start_datetime >= now() - INTERVAL 1 DAY
        ORDER BY start_datetime DESC
        LIMIT 10
    `

    rows, err := conn.Query(ctx, query)
    if err != nil {
        return err
    }
    defer rows.Close()

    fmt.Println("Recent API endpoint usage:")
    for rows.Next() {
        var startTime time.Time
        var pipeName string
        var duration float64
        var resultRows, readBytes, statusCode uint64

        if err := rows.Scan(&startTime, &pipeName, &duration, &resultRows, &readBytes, &statusCode); err != nil {
            return err
        }

        fmt.Printf("[%s] %s - %.2fms, %d rows, status %d\n",
            startTime.Format("2006-01-02 15:04:05"), pipeName, duration*1000, resultRows, statusCode)
    }

    return rows.Err()
}

// Analyze endpoint performance metrics
func queryPerformanceMetrics(conn driver.Conn) error {
    ctx := context.Background()

    query := `
        SELECT
            pipe_name,
            count() as request_count,
            avg(duration) as avg_duration_ms,
            avg(result_rows) as avg_result_rows,
            sum(read_bytes) as total_bytes_read
        FROM tinybird.pipe_stats_rt
        WHERE start_datetime >= now() - INTERVAL 1 HOUR
        GROUP BY pipe_name
        ORDER BY request_count DESC
    `

    rows, err := conn.Query(ctx, query)
    if err != nil {
        return err
    }
    defer rows.Close()

    fmt.Println("\nEndpoint performance metrics (last hour):")
    for rows.Next() {
        var pipeName string
        var requestCount uint64
        var avgDuration, avgResultRows float64
        var totalBytesRead uint64

        if err := rows.Scan(&pipeName, &requestCount, &avgDuration, &avgResultRows, &totalBytesRead); err != nil {
            return err
        }

        fmt.Printf("%s: %d requests, %.2fms avg, %.0f avg rows, %d bytes\n",
            pipeName, requestCount, avgDuration*1000, avgResultRows, totalBytesRead)
    }

    return rows.Err()
}
```

## Error handling (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-go#error-handling)

Handle ClickHouse-specific errors:

```
import (
    "github.com/ClickHouse/clickhouse-go/v2"
)

func handleQueryError(err error) {
    if exception, ok := err.(*clickhouse.Exception); ok {
        log.Printf("ClickHouse exception [%d]: %s", exception.Code, exception.Message)
        if exception.StackTrace != "" {
            log.Printf("Stack trace: %s", exception.StackTrace)
        }
    } else {
        log.Printf("Connection error: %v", err)
    }
}
```

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-clickhouse-go#learn-more)

- [ClickHouse Interface overview](/docs/forward/query-data/clickhouse-interface)
- Tinybird Auth Token management
- [ClickHouse Go Client documentation](https://github.com/ClickHouse/clickhouse-go)

---

URL: https://www.tinybird.co/docs/forward/guides/connect-chartbrew
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Connect Chartbrew to Tinybird \| Tinybird"
theme-color: "#171612"
description: "This guide covers the steps to connect Chartbrew to Tinybird using the ClickHouse HTTP interface."
inkeep:version: "forward"
---

# Connect Chartbrew to Tinybird (https://www.tinybird.co/docs/forward/guides/connect-chartbrew#connect-chartbrew-to-tinybird)

Chartbrew can connect to Tinybird using the ClickHouse® Data Source, taking advantage of Tinybird's ClickHouse® HTTP protocol compatibility.

The ClickHouse® connection to Tinybird is read-only. You can use it to query and analyze data from your Tinybird data sources, but you cannot modify data through this connection.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/connect-chartbrew#prerequisites)

- A [Chartbrew](https://chartbrew.com/) account
- A Tinybird workspace with data sources
- A Tinybird Auth Token with scopes `WORKSPACE:READ_ALL` and optionally `ORG_DATASOURCES:READ`. See [how to create it](/docs/forward/query-data/clickhouse-interface#auth-token-requirements).

## Create a new connection (https://www.tinybird.co/docs/forward/guides/connect-chartbrew#create-a-new-connection)

1. Select **Connections > Create your first connection**.
2. In the connection wizard, select **ClickHouse®** from the SQL databases list.
3. Fill in the ClickHouse database credentials.

### Connection settings (https://www.tinybird.co/docs/forward/guides/connect-chartbrew#connection-settings)

```
Connection name: Tinybird
Host: clickhouse.tinybird.co
Port: 443  # For HTTPS connections
Database username: <WORKSPACE_NAME>  # Your workspace name
Database password: <TOKEN>  # Your Tinybird auth token
Database name: <WORKSPACE_NAME>  # Your workspace name
Enable SSL: Yes
SSL Mode: Require
```
See the list of [ClickHouse hosts](/docs/forward/query-data/clickhouse-interface#clickhouse-interface-hosts) to find the correct one for your region.

## Learn more (https://www.tinybird.co/docs/forward/guides/connect-chartbrew#learn-more)

- [ClickHouse Interface overview](/docs/forward/query-data/clickhouse-interface)
- Tinybird Auth Token management

---

URL: https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Confluent Cloud setup guide \| Tinybird"
theme-color: "#171612"
description: "Complete guide to setting up Tinybird's Kafka connector with Confluent Cloud, including authentication, Schema Registry, and Private Link configuration."
inkeep:version: "forward"
---

# Confluent Cloud setup guide (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#confluent-cloud-setup-guide)

This guide walks you through setting up Tinybird's Kafka connector with Confluent Cloud, including authentication, network configuration, and Schema Registry integration.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#prerequisites)

- A Confluent Cloud account with an active cluster
- API keys for your Confluent Cloud cluster
- Access to your Confluent Cloud console
- A Tinybird workspace

## Step 1: Get your Confluent Cloud connection details (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#step-1-get-your-confluent-cloud-connection-details)

### Bootstrap servers (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#bootstrap-servers)

1. Log in to the [Confluent Cloud console](https://confluent.cloud/)
2. Navigate to your cluster
3. Go to **Cluster settings** > **Bootstrap server**
4. Copy the bootstrap server address (for example, `pkc-xxxxx.us-east-1.aws.confluent.cloud:9092`)

### API keys (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#api-keys)

1. In Confluent Cloud, go to **API Keys**
2. Select **Add key** and select **Global access** or **Resource-specific**
3. Copy the **API Key** and **API Secret**
4. Save these securely - you need them for the Tinybird connection

## Step 2: Create the Kafka connection (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#step-2-create-the-kafka-connection)

You can create the Kafka connection using the CLI wizard or by manually creating a connection file.

### Option 1: Use the CLI wizard (recommended) (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#option-1-use-the-cli-wizard-recommended)

Run the following command to create a connection interactively:

```
tb connection create kafka
```
The wizard prompts you for:

1. Connection name
2. Bootstrap server
3. Kafka key (API Key)
4. Kafka secret (API Secret)
After the wizard completes, you can edit the generated `.connection` file to add additional settings like `KAFKA_SCHEMA_REGISTRY_URL` if needed.

### Option 2: Manually create a connection file (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#option-2-manually-create-a-connection-file)

Create a `.connection` file in your Tinybird project:

connections/confluent_cloud.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS <YOUR_BOOTSTRAP_SERVER>
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("CONFLUENT_API_KEY") }}
KAFKA_SECRET {{ tb_secret("CONFLUENT_API_SECRET") }}
```
Set the secrets:

```
tb [--cloud] secret set CONFLUENT_API_KEY <YOUR_API_KEY>
tb [--cloud] secret set CONFLUENT_API_SECRET <YOUR_API_SECRET>
```

## Step 3: Configure Schema Registry (optional) (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#step-3-configure-schema-registry-optional)

If you're using Avro or JSON Schema with Schema Registry:

1. In Confluent Cloud, go to **Schema Registry**
2. Get the Schema Registry endpoint URL
3. Create API keys for Schema Registry (separate from cluster API keys)
4. Add to your connection file:
connections/confluent_cloud_with_registry.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS <YOUR_BOOTSTRAP_SERVER>
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("CONFLUENT_API_KEY") }}
KAFKA_SECRET {{ tb_secret("CONFLUENT_API_SECRET") }}
KAFKA_SCHEMA_REGISTRY_URL https://<REGISTRY_API_KEY>:<REGISTRY_API_SECRET>@<REGISTRY_ENDPOINT>
```
Set the Schema Registry credentials:

```
tb [--cloud] secret set CONFLUENT_REGISTRY_API_KEY <YOUR_REGISTRY_API_KEY>
tb [--cloud] secret set CONFLUENT_REGISTRY_API_SECRET <YOUR_REGISTRY_API_SECRET>
```
Then use in the connection:

```
KAFKA_SCHEMA_REGISTRY_URL https://{{ tb_secret("CONFLUENT_REGISTRY_API_KEY") }}:{{ tb_secret("CONFLUENT_REGISTRY_API_SECRET") }}@<REGISTRY_ENDPOINT>
```

## Step 4: Network configuration (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#step-4-network-configuration)

### Standard setup (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#standard-setup)

For most Confluent Cloud clusters, no additional network configuration is needed. Tinybird connects directly to Confluent Cloud's public endpoints.

### Private Link (Enterprise only) (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#private-link-enterprise-only)

If you're using Confluent Cloud Private Link:

1. Ensure your Confluent Cloud cluster has Private Link turned on
2. Contact Tinybird support to set up Private Link connectivity
3. Use the Private Link endpoint as your bootstrap server
For Private Link setup, contact [Tinybird support](/docs/forward/support) with:

- Your Confluent Cloud cluster details
- Private Link endpoint information
- Your Tinybird organization name

## Step 5: Create the Kafka Data Source (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#step-5-create-the-kafka-data-source)

Now that your connection is configured, create a Kafka Data Source. See [Create a Kafka data source](/docs/forward/ingest-data/connectors/kafka#create-a-kafka-data-source) in the main Kafka connector guide for detailed instructions on:

- Using `tb datasource create --kafka` for a guided setup
- Manually creating `.datasource` files
- Defining schemas with JSONPath expressions
- Configuring Kafka-specific settings

## Step 6: Test the connection (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#step-6-test-the-connection)

Test your connection and preview data:

```
tb connection data confluent_cloud
```
This command prompts you to select a topic and consumer group ID, then returns preview data. This verifies that Tinybird can connect to your Confluent Cloud cluster, authenticate, and consume messages.

## Common Confluent Cloud issues (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#common-confluent-cloud-issues)

### Issue: Authentication failed (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#issue-authentication-failed)

**Symptoms:**

- "Authentication failed" errors in `kafka_ops_log`
- Connection check fails
**Solutions:**

1. Verify API keys are correct and active in Confluent Cloud
2. Check API key has the correct permissions (read access to topics)
3. Ensure you're using `PLAIN` as the SASL mechanism
4. Verify the API key hasn't been deleted or rotated

### Issue: Schema Registry connection failed (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#issue-schema-registry-connection-failed)

**Symptoms:**

- "Schema Registry connection failed" errors
- Avro messages not being ingested
**Solutions:**

1. Verify Schema Registry endpoint URL is correct
2. Check Schema Registry API keys are separate from cluster API keys
3. Ensure Schema Registry API keys have read permissions
4. Verify the schema exists in Schema Registry for your topic

### Issue: Network connectivity (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#issue-network-connectivity)

**Symptoms:**

- Connection timeout errors
- Broker unreachable
**Solutions:**

1. Verify bootstrap server address is correct
2. Check if your Confluent Cloud cluster allows public access
3. For Private Link, ensure the connection is properly configured
4. Verify firewall rules allow outbound connections

## Best practices (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#best-practices)

1. **Use separate API keys for different environments** (dev, staging, prod)
2. **Rotate API keys regularly** for security
3. **Use resource-specific API keys** instead of global access when possible
4. **Monitor API key usage** in Confluent Cloud console
5. **Set up alerts** for authentication failures

## Related documentation (https://www.tinybird.co/docs/forward/guides/confluent-cloud-setup#related-documentation)

- [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka) - Main setup and configuration guide
- [Monitor Kafka connectors](/docs/forward/monitoring/kafka-clickhouse-monitoring) - Set up monitoring and alerts
- [Troubleshooting guide](/docs/forward/ingest-data/connectors/kafka/troubleshooting) - Common issues and solutions

---

URL: https://www.tinybird.co/docs/forward/guides/cluster-management
Last update: 2026-07-31T11:10:11.000Z
Content:
---
title: "Cluster management \| Tinybird"
theme-color: "#171612"
description: "Enterprise customers on dedicated infrastructure can scale and distribute load in their cluster by adding, removing, or resizing replicas"
inkeep:version: "forward"
---

# Cluster management (https://www.tinybird.co/docs/forward/guides/cluster-management#cluster-management)

If you're on an Enterprise plan with dedicated infrastructure, you can manage your ClickHouse® cluster replicas directly from the **Plan and Billing** page. Use this interface to horizontally scale your cluster by adding or removing replicas and controlling the distribution of read and write workloads across them. For the available replica SKUs and plan details, see [Enterprise](/docs/forward/pricing/enterprise).

## Access cluster management (https://www.tinybird.co/docs/forward/guides/cluster-management#access-cluster-management)

To manage your cluster replicas:

1. Navigate to **Organization settings** from the sidebar.
2. Go to the **Plan & Billing** section.
This opens the cluster management interface where you can view your current replicas and their configurations.

### Understanding replica configuration (https://www.tinybird.co/docs/forward/guides/cluster-management#understanding-replica-configuration)

Each replica in your cluster has the following configuration:

| Setting | Description |
| --- | --- |
| Read weight | Controls the proportion of query traffic this replica receives. Valid range: `0-65535`. |
| Write weight | Controls whether this replica handles data ingestion. Valid range: `0-65535`. |
| Copy job weight | Controls whether this replica handles copy jobs. Default and recommended value for all replicas: `1`. Valid range: `0-65535`. |
| CPUs | Number of virtual CPUs allocated to this replica. |
| Memory | Amount of memory allocated to this replica in gigabytes. |

## Managing a cluster (https://www.tinybird.co/docs/forward/guides/cluster-management#managing-a-cluster)

Cluster operations like adding, removing or rebalancing replica weights may take a few minutes to complete. During this time, no other operations can be performed on your cluster. However, the cluster continues to operate normally while operations are in progress.

## Managing a cluster via API (https://www.tinybird.co/docs/forward/guides/cluster-management#managing-a-cluster-via-api)

The following operations can also be performed programmatically using the [Organizations API](/docs/api-reference/organizations-api).

API calls that modify replicas require an `old_weights` field to prevent conflicts from concurrent modifications. You must first `GET` the current configuration from the `/v0/organizations/<organization_id>/clusters-configuration` endpoint and use the returned weights when you add, delete, or rebalance replicas.

### Add a replica (https://www.tinybird.co/docs/forward/guides/cluster-management#add-a-replica)

To add a new replica to your cluster:

1. In the cluster management interface, select **Add new replica**.
2. Configure the Read and Write weights
3. Select the CPU and memory configuration from the dropdown.
4. Review the cost impact shown in the configuration panel.
5. Select **Add replica**.
To add a replica programmatically, use the following endpoint from the [Organizations API](/docs/api-reference/organizations-api):

```
curl -X POST \
    -H "Authorization: Bearer <user_token>" \
    -H "Content-Type: application/json" \
    "https://<your_host>/v0/organizations/<organization_id>/clusters/<cluster_id>/replicas" \
     -d '{
        "replica_size": "4-16",
        "existing_replicas": {
            "old_weights": {
                "copyjob": {"<replica-1-name>": 1, "<replica-2-name>": 1},
                "writer": {"<replica-1-name>": 50, "<replica-2-name>": 50},
                "reader": {"<replica-1-name>": 50, "<replica-2-name>": 50}
            },
            "new_weights": {
                "copyjob": {"<replica-1-name>": 1, "<replica-2-name>": 1},
                "writer": {"<replica-1-name>": 30,  "<replica-2-name>": 40},
                "reader": {"<replica-1-name>": 40,  "<replica-2-name>": 40}
            }
        },
        "new_replica": {
            "copyjob": 1,
            "writer": 30,
            "reader": 20
        }
    }'
```

#### Cost implications (https://www.tinybird.co/docs/forward/guides/cluster-management#cost-implications)

Adding replicas increases your monthly credit consumption. The exact cost depends on the CPU and memory configuration you select. Cost estimates are displayed in the interface before you add a replica.

### Remove a replica (https://www.tinybird.co/docs/forward/guides/cluster-management#remove-a-replica)

To remove a replica from your cluster:

1. In the cluster management interface, locate the replica you want to remove.
2. Select the **×** icon next to the replica.
3. Adjust the replica weights to redistribute traffic across remaining replicas.
4. Confirm the removal by selecting **Delete replica**.
Removing a replica may affect traffic distribution and cluster stability, so make sure you reassign its weights to other replicas to maintain performance and that the remaining replicas can handle the load.

### Resize your cluster (https://www.tinybird.co/docs/forward/guides/cluster-management#resize-your-cluster)

To resize your cluster, replace existing replicas with new ones of the desired size. You do this through the replica management workflow:

1. [Add new replicas](https://www.tinybird.co/docs/forward/guides/cluster-management#add-a-replica) with the target CPU and memory configuration.
2. [Rebalance traffic](https://www.tinybird.co/docs/forward/guides/cluster-management#rebalance-traffic-across-replicas) by setting read and write weights to route traffic to the new replicas.
3. [Remove the old replicas](https://www.tinybird.co/docs/forward/guides/cluster-management#remove-a-replica) once all traffic has been rerouted.
Before removing the larger replicas, verify that the new replicas can handle your workload. Monitor query performance and ingestion throughput after rerouting traffic.

#### Downtime considerations (https://www.tinybird.co/docs/forward/guides/cluster-management#downtime-considerations)

Resizing through replica management is a zero-downtime operation. Your cluster continues to serve traffic throughout the process because the old replicas remain active until you explicitly remove them.

#### Cost implications (https://www.tinybird.co/docs/forward/guides/cluster-management#cost-implications)

During the transition, you're billed for both old and new replicas. To minimize costs, complete the migration promptly and remove the old replicas as soon as traffic is fully rerouted. Review the cost estimates in the cluster management interface before adding new replicas.

### Rebalance traffic across replicas (https://www.tinybird.co/docs/forward/guides/cluster-management#rebalance-traffic-across-replicas)

You can control how your workload is distributed across replicas by adjusting their weights:

1. Modify the Read weights to control query traffic distribution.
2. Adjust Write weights to control which replicas handle data ingestion.
3. Select **Save changes** to apply the new configuration.

## How weight distribution works (https://www.tinybird.co/docs/forward/guides/cluster-management#how-weight-distribution-works)

Weights determine the proportional distribution of traffic across replicas using a weighted round-robin approach:

- **Read weights**: Query traffic is distributed proportionally based on each replica's read weight relative to the total of all read weights. At least one replica must have a non-zero read weight. If a replica has a read weight of 0, it doesn't receive any query traffic.
- **Write weights**: Ingestion traffic from the Events API and Kafka is distributed proportionally based on each replica's write weight. At least one replica must have a non-zero write weight. If a replica has a write weight of 0, it doesn't receive any ingestion traffic.
- **Copy job weights**: Copy job execution is distributed proportionally based on each replica's weight. The recommended value for all replicas is 1.
- **Valid range**: Each weight must be between 0 and 65535.
- **Required minimums**: At least one replica must have a non-zero read weight, and at least one replica must have a non-zero write weight.
Proportional distribution of write traffic applies only to the Events API and Kafka. It is not guaranteed for other ingestion sources, such as file imports or Copy Pipes.

### Weight distribution examples (https://www.tinybird.co/docs/forward/guides/cluster-management#weight-distribution-examples)

#### Read traffic distribution (https://www.tinybird.co/docs/forward/guides/cluster-management#read-traffic-distribution)

- **Example 1**: equal distribution of query traffic across replicas (useful when all replicas have the same size)
  - `replica-a [R: 1]`, replica receives 50% of query traffic
  - `replica-b [R: 1]`, replica receives 50% of query traffic
- **Example 2**: non-equal distribution of query traffic across replicas (useful when replicas have different sizes)
  - `replica-a [R: 1]`, replica receives 33% of query traffic (1/3)
  - `replica-b [R: 2]`, replica receives 67% of query traffic, or 2/3
- **Example 3**: non-equal distribution of query traffic across replicas (useful when replicas have different sizes)
  - `replica-a [R: 10]`, replica receives 17% of query traffic (1/6)
  - `replica-b [R: 20]`, replica receives 33% of query traffic, or 2/6
  - `replica-c [R: 30]`, replica receives 50% of query traffic, or 3/6

#### Write traffic distribution (https://www.tinybird.co/docs/forward/guides/cluster-management#write-traffic-distribution)

- **Example 1**: single writer replica
  - `replica-a [W: 1]`, replica receives 100% of ingestion traffic
  - `replica-b [W: 0]`, replica receives 0% of ingestion traffic
  - `replica-c [W: 0]`, replica receives 0% of ingestion traffic
- **Example 2**: equal distribution of ingestion traffic across replicas
  - `replica-a [W: 1]`, replica receives 50% of ingestion traffic
  - `replica-b [W: 1]`, replica receives 50% of ingestion traffic
- **Example 3**: unequal distribution of ingestion traffic across replicas
  - `replica-a [W: 1]`, replica receives 33% of ingestion traffic, or 1/3
  - `replica-b [W: 2]`, replica receives 67% of ingestion traffic, or 2/3

#### Example cluster traffic distribution (https://www.tinybird.co/docs/forward/guides/cluster-management#example-cluster-traffic-distribution)

| Replica name | Read weight | Write weight | Explanation |
| --- | --- | --- | --- |
| replica-a | 0 | 1 | Receive no query traffic and 100% of ingestion traffic |
| replica-b | 1 | 0 | Receive 17% of query traffic and no ingestion traffic |
| replica-c | 1 | 0 | Receive 17% of query traffic and no ingestion traffic |
| replica-d | 2 | 0 | Receive 33% of query traffic and no ingestion traffic |
| replica-e | 2 | 0 | Receive 33% of query traffic and no ingestion traffic |

## Copy job weight (https://www.tinybird.co/docs/forward/guides/cluster-management#copy-job-weight)

`copyjob` controls where copy job workloads (triggered by Copy Pipes) run across replicas.

Set `copyjob` to `1` on all replicas by default.

This ensures copy job execution remains available on all replicas and avoids unintentionally excluding replicas from processing.

---

URL: https://www.tinybird.co/docs/forward/guides/aws-msk-setup
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "AWS MSK setup guide \| Tinybird"
theme-color: "#171612"
description: "Complete guide to setting up Tinybird's Kafka connector with AWS MSK, including IAM authentication, security groups, VPC configuration, and PrivateLink setup."
inkeep:version: "forward"
---

# AWS MSK setup guide (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#aws-msk-setup-guide)

This guide walks you through setting up Tinybird's Kafka connector with Amazon MSK (Managed Streaming for Apache Kafka), including IAM authentication, network configuration, and security group setup.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#prerequisites)

- An AWS account with an MSK cluster
- AWS IAM permissions to create roles and policies
- Access to AWS Console
- A Tinybird workspace

## Step 1: Get your MSK cluster details (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#step-1-get-your-msk-cluster-details)

### Bootstrap servers (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#bootstrap-servers)

1. In AWS Console, navigate to **Amazon MSK**
2. Select your cluster
3. Go to **Properties** tab
4. Copy the **Bootstrap broker string** (for example, `b-1.example-cluster.abc123.c2.kafka.us-east-1.amazonaws.com:9098,b-2.example-cluster.abc123.c2.kafka.us-east-1.amazonaws.com:9098`)
**Important:** Use the port that matches your authentication method:

- Port 9098 for SASL/IAM (OAUTHBEARER)
- Port 9096 for SASL/SCRAM
- Port 9094 for TLS
- Port 9092 for plaintext (not recommended for production)

### Cluster ARN (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#cluster-arn)

1. In the cluster **Properties** tab
2. Copy the **Cluster ARN** (for example, `arn:aws:kafka:us-east-1:123456789012:cluster/example-cluster/abc123-def456-789`)
You need this for the IAM policy configuration.

## Step 2: Create the Kafka connection (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#step-2-create-the-kafka-connection)

You can create the Kafka connection using the CLI wizard or by manually creating a Connection file.

### Option 1: Use the CLI wizard (recommended) (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#option-1-use-the-cli-wizard-recommended)

Run the following command to create a connection interactively:

```
tb connection create kafka
```
The wizard prompts you for:

1. Connection name
2. Bootstrap server
3. Kafka key (for SASL/SCRAM) or IAM role ARN (for IAM authentication)
4. Kafka secret (for SASL/SCRAM) or external ID (for IAM authentication)
After the wizard completes, edit the generated Connection file to add AWS MSK-specific settings like `KAFKA_SASL_OAUTHBEARER_METHOD AWS` and `AWS_ROLE_ARN` for IAM authentication.

### Option 2: Manually create a Connection file (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#option-2-manually-create-a-connection-file)

For AWS MSK with IAM authentication, manually create the Connection file with the specific IAM settings. See [Step 5: Create the Kafka connection (manual method)](https://www.tinybird.co/docs/forward/guides/aws-msk-setup#step-5-create-the-kafka-connection-manual-method) after completing IAM and security group setup.

## Step 3: Set up IAM authentication (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#step-3-set-up-iam-authentication)

MSK supports IAM authentication using OAUTHBEARER. You need to create an IAM role that Tinybird can assume.

### Create IAM role (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#create-iam-role)

1. In AWS Console, go to **IAM** > **Roles**
2. Select **Create role**
3. Select **AWS account** as the trusted entity type
4. For **Account ID**, use Tinybird's AWS account ID (contact support for the correct ID for your region)
5. Check **Require external ID** and enter a unique external ID (save this for later)
6. Select **Next**

### Create access policy (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#create-access-policy)

Create a policy that grants access to your MSK cluster:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:DescribeCluster"
            ],
            "Resource": "arn:aws:kafka:<REGION>:<ACCOUNT_ID>:cluster/<CLUSTER_NAME>/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:ReadData"
            ],
            "Resource": "arn:aws:kafka:<REGION>:<ACCOUNT_ID>:topic/<CLUSTER_NAME>/*/<TOPIC_NAME>"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": "arn:aws:kafka:<REGION>:<ACCOUNT_ID>:group/<CLUSTER_NAME>/*/<GROUP_ID>"
        }
    ]
}
```
Replace:

- `<REGION>`: Your AWS region (for example, `us-east-1`)
- `<ACCOUNT_ID>`: Your AWS account ID
- `<CLUSTER_NAME>`: Your MSK cluster name
- `<TOPIC_NAME>`: Your Kafka topic name (or `*` for all topics)
- `<GROUP_ID>`: Your consumer group ID (or `*` for all groups)
**Alternative:** Use Tinybird's API to generate the policy:

```
curl "https://<your_host>/v0/integrations/kafka/policies/read-access-policy?msk_cluster_arn=<CLUSTER_ARN>&topics=<TOPIC_NAME>&groups=<GROUP_ID>"
```

### Create trust policy (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#create-trust-policy)

The trust policy allows Tinybird to assume the role:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Principal": {
                "AWS": "arn:aws:iam::<TINYBIRD_ACCOUNT_ID>:root"
            },
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "<EXTERNAL_ID>"
                }
            }
        }
    ]
}
```
Replace:

- `<TINYBIRD_ACCOUNT_ID>`: Tinybird's AWS account ID (contact support for your region)
- `<EXTERNAL_ID>`: The external ID you set when creating the role
**Alternative:** Use Tinybird's API:

```
curl "https://<your_host>/v0/integrations/kafka/policies/trust-policy?external_id_seed=<CONNECTION_NAME>"
```

### Attach policies to role (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#attach-policies-to-role)

1. Attach the access policy to your IAM role
2. Set the trust policy on the role
3. Copy the **Role ARN** (for example, `arn:aws:iam::123456789012:role/msk-tinybird-role`)

## Step 4: Configure security groups (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#step-4-configure-security-groups)

### MSK security group (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#msk-security-group)

1. In AWS Console, go to **EC2** > **Security Groups**
2. Find the security group used by your MSK cluster
3. Add an inbound rule:
  - **Type:** Custom TCP
  - **Port:** 9098 (or the port matching your authentication)
  - **Source:** 0.0.0.0/0
Tinybird's egress IP addresses change dynamically, so you can't use a fixed IP allowlist. IAM authentication with role assumption and external ID provides the security boundary for your connection.

If your security policy requires restricting network access, use a [PrivateLink setup](https://www.tinybird.co/docs/forward/guides/aws-msk-setup#privatelink-setup-enterprise-only) instead.

## Step 5: Create the Kafka connection (manual method) (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#step-5-create-the-kafka-connection-manual-method)

If you didn't use the CLI wizard in Step 2, create a Connection file manually:

connections/aws_msk.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS <BOOTSTRAP_BROKER_STRING>
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM OAUTHBEARER
KAFKA_SASL_OAUTHBEARER_METHOD AWS
KAFKA_SASL_OAUTHBEARER_AWS_REGION <REGION>
KAFKA_SASL_OAUTHBEARER_AWS_ROLE_ARN {{ tb_secret("AWS_ROLE_ARN") }}
KAFKA_SASL_OAUTHBEARER_AWS_EXTERNAL_ID <EXTERNAL_ID>
```
Set the role ARN secret:

```
tb [--cloud] secret set AWS_ROLE_ARN <YOUR_ROLE_ARN>
```
Replace:

- `<BOOTSTRAP_BROKER_STRING>`: The bootstrap broker string from Step 1
- `<EXTERNAL_ID>`: The external ID you set in the IAM role in Step 3
- `<REGION>`: Your AWS region (for example, `us-east-1`)

## Step 6: VPC and network configuration (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#step-6-vpc-and-network-configuration)

### Standard setup (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#standard-setup)

For most MSK clusters, Tinybird connects via the public endpoint. Ensure:

1. Your MSK cluster has public access turned on
2. Security groups allow inbound connections from Tinybird
3. Network ACLs don't block the connection

### PrivateLink setup (Enterprise only) (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#privatelink-setup-enterprise-only)

For PrivateLink connectivity:

1. Ensure your MSK cluster supports PrivateLink
2. Contact Tinybird support to set up PrivateLink endpoint
3. Use the PrivateLink endpoint as your bootstrap server
Contact [Tinybird support](/docs/forward/support) with:

- Your MSK cluster details
- VPC and subnet information
- Your Tinybird organization name

## Step 7: Create the Kafka Data Source (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#step-7-create-the-kafka-data-source)

Now that your connection is configured, create a Kafka Data Source. See [Create a Kafka data source](/docs/forward/ingest-data/connectors/kafka#create-a-kafka-data-source) in the main Kafka connector guide for detailed instructions on:

- Using `tb datasource create --kafka` for a guided setup
- Manually creating Data Source files
- Defining schemas with JSONPath expressions
- Configuring Kafka-specific settings

## Step 8: Test the connection (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#step-8-test-the-connection)

Test your connection and preview data:

```
tb connection data aws_msk
```
This command prompts you to select a topic and consumer group ID, then returns preview data. This verifies that Tinybird can:

1. Assume the IAM role
2. Connect to your MSK cluster
3. Authenticate using OAUTHBEARER
4. Consume messages from the topic

## Common AWS MSK issues (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#common-aws-msk-issues)

### Issue: IAM authentication failed (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#issue-iam-authentication-failed)

**Symptoms:**

- "Authentication failed" errors
- "Unable to assume role" errors
**Solutions:**

1. Verify the IAM role ARN is correct
2. Check the trust policy allows Tinybird's AWS account
3. Verify the external ID matches between connection and trust policy
4. Ensure the IAM role has the correct access policy attached
5. Verify the region in your Connection file matches your MSK cluster region
6. Use AWS CloudTrail to see detailed error messages for IAM authentication failures

### Issue: Security group blocking connection (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#issue-security-group-blocking-connection)

**Symptoms:**

- Connection timeout errors
- Broker unreachable
**Solutions:**

1. Verify security group allows inbound traffic on the correct port
2. Check the security group allows `0.0.0.0/0` on the correct port, as Tinybird IPs change dynamically
3. Verify network ACLs don't block the connection
4. For PrivateLink, ensure VPC endpoint is configured correctly

### Issue: Network connectivity (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#issue-network-connectivity)

**Symptoms:**

- Connection timeout
- Unable to reach bootstrap servers
**Solutions:**

1. Verify bootstrap server address is correct
2. Check if MSK cluster has public access turned on
3. Verify DNS resolution for MSK cluster endpoints
4. For PrivateLink, ensure endpoint is active and accessible

## Best practices (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#best-practices)

1. **Use least privilege IAM policies** - Only grant access to specific topics and groups
2. **Use unique external IDs** for each connection
3. **Monitor IAM role usage** in CloudTrail
4. **Rotate IAM roles periodically** for security
5. **Use separate roles** for different environments (dev, staging, prod)
6. **Turn on VPC flow logs** to monitor network traffic
7. **Set up CloudWatch alarms** for MSK cluster health

## Related documentation (https://www.tinybird.co/docs/forward/guides/aws-msk-setup#related-documentation)

- [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka) - Main setup and configuration guide
- [Monitor Kafka connectors](/docs/forward/monitoring/kafka-clickhouse-monitoring) - Set up monitoring and alerts
- [Troubleshooting guide](/docs/forward/ingest-data/connectors/kafka/troubleshooting) - Common issues and solutions

---

URL: https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Advanced Template Functions for APIs \| Tinybird"
theme-color: "#171612"
description: "Learn more about creating dynamic API endpoint using advanced templates."
inkeep:version: "forward"
---

# Advanced template functions for dynamic API endpoint (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#advanced-template-functions-for-dynamic-api-endpoint)

Learn how to use [template functions](/docs/forward/dev-reference/template-functions) to create dynamic API endpoint with Tinybird.

## Prerequisites (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#prerequisites)

Make sure you're familiar with template functions and [query parameters](/docs/forward/query-data/query-parameters).

## Example data (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#example-data)

This guide uses the eCommerce events data enriched with products. The data looks like the following:

Events and products data
```
SELECT *, price, city, day FROM events_mat
ANY LEFT JOIN products_join_sku ON product_id = sku
```

## Tips and tricks (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#tips-and-tricks)

When the complexity of pipes and API endpoints grows, developing them and knowing what's going-on to debug problems can become challenging. Here are some useful tricks for using Tinybird's product:

### WHERE 1=1 (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#where-11)

When you filter by different criteria, given by dynamic parameters that can be omitted, use a `WHERE` clause. If none of the parameters are present, add a `WHERE` statement with a dummy condition (like `1=1`) that's always true. Then add the other filter statements dynamically if the parameters are defined, as shown in the [defined](https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#defined) example of this guide.

### Use the set function (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#use-the-set-function)

Use the `set` function in the previous snippet to set the value of a parameter in a node and check the query output for different parameter values. Otherwise, you'd have to publish an API endpoint and make requests to it with different parameters.

Using `set`, you don't have to exit the Tinybird UI while creating an API endpoint and the whole process is faster, without needing to go back and forth between your browser or IDE and Postman or cURL.

Another example of its usage:

Using set to try out different parameter values
```
%
{% set select_cols = 'date,user_id,event,city' %}
SELECT
  {{columns(select_cols)}}
FROM events_mat
```
You can use more than one `set` statement. Put each one on a separate line at the beginning of a node.

`set` is also a way to set defaults for parameters. If you used `set` statements to test your API endpoint while developing, remember to remove them before publishing your code, because if not, the `set` function overrides any incoming parameter.

### Default argument (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#default-argument)

Another way to set default values for parameters is using the `default` argument that most Tinybird template functions accept. The previous code could be rewritten as follows:

Using the default argument
```
%
SELECT
  {{columns(select_cols, 'date,user_id,event,city')}}
FROM events_mat
```
Keep in mind that defining the same parameter in more than one place in your code in different ways can lead to inconsistent behavior. Here's a solution to avoid that:

### Using WITH statements to avoid duplicating code (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#using-with-statements-to-avoid-duplicating-code)

If you plan to use the same dynamic parameters more than once in a node of a Pipe, define them in one place to avoid duplicating code. This also makes it clearer which parameters appear in the node. Add one or more statements at the beginning of a node using the `WITH` clause.

The WITH clause supports CTEs. These are preprocessed before executing the query, and can only return one row. This is different to other databases such as Postgres. For example:

DRY with the with clause
```
%
{% set terms='orchid' %}
WITH {{split_to_array(terms, '1,2,3')}} AS needles
SELECT
  *,
  joinGet(products_join_sku, 'color', product_id) color,
  joinGet(products_join_sku, 'title', product_id) title
FROM events
WHERE
  multiMatchAny(lower(color), needles)
  OR multiMatchAny(lower(title), needles)
```

### Documenting your API endpoints (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#documenting-your-api-endpoints)

Tinybird creates auto-generated documentation for all your published API endpoints, taking the information from the dynamic parameters found in the pipe. It's best practice to set default values and descriptions for every parameter in one place (also because some functions don't accept a description, for example). This is typically done in the final node, with `WITH` statements at the beginning. See how to do it in the [last section](https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#putting-it-all-together) of this guide.

### Hidden parameters (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#hidden-parameters)

If you use some functions like `enumerate_with_last` in the [enumarate with last example](https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#enumerate-with-last), you might end up with some variables (called `x`, `last` in that code snippet) that Tinybird interprets as if they were parameters that you can set. They appear in the auto-generated documentation page. To avoid that, add a leading underscore to their name, renaming `x` to `_x` and `last` to `_last`.

## Advanced functions (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#advanced-functions)

The following are practical examples of advanced template functions usage so that it's easier for you to understand how to use them.

### defined (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#defined)

Use the `defined` function to check whether a query string parameter exists in the request URL.

Imagine you want to filter events with a price within a minimum or a maximum price, set by two dynamic parameters that could be omitted. A way to define the API endpoint would be like this:

filter by price
```
%
{% set min_price=20 %}
{% set max_price=50 %}

SELECT *, price
FROM events_mat
WHERE 1 = 1
{% if defined(min_price) %}
  AND price >= {{Float32(min_price)}}
{% end %}
{% if defined(max_price) %}
  AND price <= {{Float32(max_price)}}
{% end %}
```
To see the effect of having a parameter not defined, use `set` to set its value to `None` like this:

filter by price, price not defined
```
%
{% set min_price=None %}
{% set max_price=None %}

SELECT *, price
FROM events_mat
WHERE 1 = 1
{% if defined(min_price) %}
  AND price >= {{Float32(min_price)}}
{% end %}
{% if defined(max_price) %}
  AND price <= {{Float32(max_price)}}
{% end %}
```
It's also possible to provide smart defaults to avoid needing to use the `defined` function at all:

filter by price with default values
```
%
SELECT *, price
FROM events_mat_cols
WHERE price >= {{Float32(min_price, 0)}}
  AND price <= {{Float32(max_price, 999999999)}}
```

### Array(variable_name, 'type', [default]) (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#arrayvariable-name-type-default)

Transforms a comma-separated list of values into a Tuple. You can provide a default value for it or not:

```
%
SELECT
    {{Array(code, 'UInt32', default='13412,1234123,4123')}} AS codes_1,
    {{Array(code, 'UInt32', '13412,1234123,4123')}} AS codes_2,
    {{Array(code, 'UInt32')}} AS codes_3
```
To filter events whose type belongs to the ones provided in a dynamic parameter, separated by commas, you'd define the API endpoint like this:

Filter by list of elements
```
%
SELECT *
FROM events
WHERE event IN {{Array(event_types, 'String', default='buy,view')}}
```
And then the URL of the API endpoint would be something like `{% user("apiHost") %}/v0/pipes/your_pipe_name.json?event_types=buy,view`

### sql_and (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#sql-and)

Use `sql_and` to create a filter dynamically with `AND` operators and several expressions, based on whether the template's dynamic parameters are present in the request URL.

It's not possible to use Tinybird functions inside the `{{ }}` brackets in templates. `sql_and` can only be used with the `{column_name}__{operand}` syntax. This function does the same as the previous query: filtering a column by the values that are present in a tuple generated by `Array(...)` if `operand` is `in`, are greater than (with the `gt` operand), or less than (with the `lt` operand). The following example demonstrates this behavior:

- Endpoint template code
- Generated SQL
SQL_AND AND COLUMN__IN
```
%
SELECT
  *,
  joinGet(products_join_sku, 'section_id', product_id) section_id
FROM events
WHERE {{sql_and(event__in=Array(event_types, 'String', default='buy,view'),
                section_id__in=Array(sections, 'Int16', default='1,2'))}}
```
You don't have to provide default values. If you set the `defined` argument of `Array` to `False`, no SQL expression is generated when that parameter isn't provided. The following code snippet shows this behavior:

- Endpoint template code
- Generated SQL
defined=False
```
%
SELECT
  *,
  joinGet(products_join_sku, 'section_id', product_id) section_id
FROM events
WHERE {{sql_and(event__in=Array(event_types, 'String', default='buy,view'),
                section_id__in=Array(sections, 'Int16', defined=False))}}
```

### split_to_array(name, [default]) (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#split-to-arrayname-default)

This works similarly to `Array`, but it returns an Array of Strings instead of a tuple. Cast the result to the type you want. The following example shows the similar behavior:

array and split_to_array
```
%
SELECT
    {{Array(code, 'UInt32', default='1,2,3')}},
    {{split_to_array(code, '1,2,3')}},
    arrayMap(x->toInt32(x), {{split_to_array(code, '1,2,3')}}),
    1 in {{Array(code, 'UInt32', default='1,2,3')}},
    '1' in {{split_to_array(code, '1,2,3')}}
```
One thing that you want to keep in mind is that you can't pass non-constant values (arrays, for example) to operations that require them. For example, this would fail:

using a non-constant expression where one is required
```
%
SELECT
    1 IN arrayMap(x->toInt32(x), {{split_to_array(code, '1,2,3')}})
```
If you find an error like this, use a Tuple instead. Remember that `{{Array(...)}}` returns a tuple. The following version works:

Use a tuple instead
```
%
SELECT
    1 IN {{Array(code, 'Int32', default='1,2,3')}}
```
`split_to_array` is often used with [enumerate_with_last](https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#enumerate-with-last).

### column and columns (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#column-and-columns)

They let you select one or several columns from a data source or pipe, given their name. You can also provide a default value.

columns
```
%
SELECT {{columns(cols, 'date,user_id,event')}}
FROM events
```
column
```
%
SELECT date, {{column(user, 'user_id')}}
FROM events
```

### enumerate_with_last (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#enumerate-with-last)

Creates an iterable array, returning a Boolean value that allows checking if the current element is the last element in the array. Its most common usage is to select several columns, or compute some function over them. See an example of `columns` and `enumerate_with_last` here:

- Endpoint template code
- Generated SQL
enumerate_with_last + columns
```
%
SELECT
    {% if defined(group_by) %}
        {{columns(group_by)}},
    {% end %}
    sum(price) AS revenue,
    {% for last, x in enumerate_with_last(split_to_array(count_unique_vals_columns, 'section_id,city')) %}
        uniq({{symbol(x)}}) as {{symbol(x)}}
        {% if not last %},{% end %}
    {% end %}
  FROM events_enriched
{% if defined(group_by) %}
    GROUP BY
         {{columns(group_by)}}

    ORDER BY
        {{columns(group_by)}}
{% end %}
```
If you use the `defined` function around a parameter, don't give it a default value. If the parameter isn't provided, that line doesn't run.

### error and custom_error (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#error-and-custom-error)

They let you return customized error responses. With `error` you can customize the error message:

error
```
%
{% if not defined(event_types) %}
  {{error('You need to provide a value for event_types')}}
{% end %}
SELECT
  *,
  joinGet(products_join_sku, 'section_id', product_id) section_id
FROM events
WHERE event IN {{Array(event_types, 'String')}}
```
error response using error
```
{"error": "You need to provide a value for event_types"}
```
And with `custom_error` you can also customize the response code:

custom_error
```
%
{% if not defined(event_types) %}
  {{custom_error({'error': 'You need to provide a value for event_types', 'code': 400})}}
{% end %}
SELECT
  *,
  joinGet(products_join_sku, 'section_id', product_id) section_id
FROM events
WHERE event IN {{Array(event_types, 'String')}}
```
error response using custom_error
```
{"error": "You need to provide a value for event_types", "code": 400}
```
**Note:** Place `error` and `custom_error` at the start of a node. Use this order:

1. `set` lines, to give some parameter a default value (optional)
2. Parameter validation functions: `error` and `custom_error` definitions
3. The SQL query itself

## Putting it all together (https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions#putting-it-all-together)

You've created a pipe where you use most of these advanced techniques to filter ecommerce events.

This is its code:

advanced_dynamic_endpoints.pipe
```
NODE events_enriched
SQL >

    SELECT
        *,
        price,
        city,
        day
    FROM events_mat_cols
    ANY LEFT JOIN products_join_sku ON product_id = sku

NODE filter_by_price
SQL >

    %
        SELECT * FROM events_enriched
        WHERE 1 = 1
        {% if defined(min_price) %}
          AND price >= {{Float32(min_price)}}
        {% end %}
        {% if defined(max_price) %}
          AND price <= {{Float32(max_price)}}
        {% end %}

NODE filter_by_event_type_and_section_id
SQL >

    %
    SELECT
          *
        FROM filter_by_price
        {% if defined(event_types) or defined(section_ids) %} ...
            WHERE {{sql_and(event__in=Array(event_types, 'String', defined=False, enum=['remove_item_from_cart','view','search','buy','add_item_to_cart']),
                            section_id__in=Array(section_ids, 'Int32', defined=False))}}
        {% end %}

NODE filter_by_title_or_color
SQL >

    %
    SELECT *
    FROM filter_by_event_type_and_section_id
    {% if defined(search_terms) %}
    WHERE
      multiMatchAny(lower(color), {{split_to_array(search_terms)}})
      OR multiMatchAny(lower(title), {{split_to_array(search_terms)}})
    {% end %}

NODE group_by_or_not
SQL >

    %
        SELECT
            {% if defined(group_by) %}
              {{columns(group_by)}},
              sum(price) AS revenue,
              {% for _last, _x in enumerate_with_last(split_to_array(count_unique_vals_columns)) %}
                  uniq({{symbol(_x)}}) as {{symbol(_x)}}
                  {% if not _last %},{% end %}
              {% end %}
            {% else %}
              *
            {% end %}
        FROM filter_by_title_or_color
        {% if defined(group_by) %}
            GROUP BY {{columns(group_by)}}
            ORDER BY {{columns(group_by)}}
        {% end %}

NODE pagination
SQL >

    %
    WITH
      {{Array(group_by,
      'String',
      '',
               description='Comma-separated name of columns. If defined, group by and order the results by these columns. The sum of revenue will be returned')}},
      {{Array(count_unique_vals_columns, 'String', '',
               description='Comma-separated name of columns. If both group_by and count_unique_vals_columns are defined, the number of unique values in the columns given in count_unique_vals_columns will be returned as well')}},
       {{Array(search_terms, 'String', '',
              description='Comma-separated list of search terms present in the color or title of products')}},
       {{Array(event_types, 'String', '',
               description="Comma-separated list of event name types", enum=['remove_item_from_cart','view','search','buy','add_item_to_cart'])}},
       {{Array(section_ids, 'String', '',
               description="Comma-separated list of section IDs. The minimum value for an ID is 0 and the max is 50.")}}
    SELECT * FROM group_by_or_not
    LIMIT {{Int32(page_size, 100)}}
    OFFSET {{Int32(page, 0) * Int32(page_size, 100)}}
```
To replicate it in your account, copy the previous code to a new endpoint file called `advanced_dynamic_endpoints.pipe` locally and deploy your changes.

---

URL: https://www.tinybird.co/docs/forward/guides/add-self-managed-region-manually
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Add a self-managed region manually \| Tinybird"
theme-color: "#171612"
description: "Create your own Tinybird Cloud region in the cloud service provider of your choice."
inkeep:version: "forward"
---

# Add a self-managed region manually (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-manually#add-a-self-managed-region-manually)

To add a self-managed region manually, you need to:

1. Create a region in Tinybird Cloud.
2. Deploy the Tinybird Local container on your cloud provider.
Self-managed regions are currently in beta. Use them for development environments only; avoid production workloads. Features, requirements, and implementation details might change as Tinybird continues to improve this capability.

## Resource requirements (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-manually#resource-requirements)

When planning your self-managed region deployment, consider the following resource requirements:

- Compute: At least 4 vCPUs and 16 GB RAM for development environments. Production environments may require more resources depending on your workload.
- Storage: Allocate at least 100 GB for the ClickHouse® volume and 10 GB for the Redis volume. Scale according to your data volume needs.
  - Required paths in the container are `/redis-data` and `/var/lib/clickhouse`
- Network: Ensure your deployment has sufficient bandwidth for your expected query and ingestion rates.
  - You also need a publicly accessible HTTPS URL. For example, `https://tinybird.example.com`.
- Init parameters: Provide the following parameters to the deployed container. These parameters can be provided in the `tb infra add` command or automatically generated by `tb infra init`:
  - `TB_INFRA_TOKEN`
  - `TB_INFRA_WORKSPACE`
  - `TB_INFRA_ORGANIZATION`
  - `TB_INFRA_USER`

## Create a region manually (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-manually#create-a-region-manually)

To add a new self-managed region to Tinybird Cloud manually, follow these steps:

1
### Log into Tinybird Cloud (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-manually#log-into-tinybird-cloud)

Log into Tinybird Cloud using the `tb login` command.

```
tb login
```
2
### Add the new region (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-manually#add-the-new-region)

Before you deploy the `tblocal` container on your cloud provider, add a self-managed region using `tb infra add`:

```
tb infra add

Running against Tinybird Cloud: Workspace example_workspace
Enter name: example
Enter host: https://tinybird.example.com
» Adding infrastructure 'example' in Tinybird...
✓ Infrastructure 'example' added
» Required environment variables:
TB_INFRA_TOKEN=example_token
TB_INFRA_WORKSPACE=example_workspace
TB_INFRA_ORGANIZATION=example_organization
TB_INFRA_USER=user@example.com
```
The host is optional, as you might not known it yet. You can update it later using `tb infra update`. Copy the environment variables and their values to use in the next step.

If you're an organization admin, you can also list, add, update, or remove infrastructure regions in Tinybird Cloud. Go to **Settings**, **Managed regions** in the Tinybird Cloud.

3
### Deploy Tinybird Local (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-manually#deploy-tinybird-local)

Deploy the [Tinybird Local](/docs/forward/core-concepts/tinybird-local) container on your cloud provider. At the end of the deployment, the `tblocal` container must be publicly accessible on an HTTP URL, which is the host of the self-managed region.

Make sure to expose and set the following environment variables to the `tblocal` container using the values provided in the previous step:

- `TB_INFRA_TOKEN`
- `TB_INFRA_WORKSPACE`
- `TB_INFRA_ORGANIZATION`
- `TB_INFRA_USER`
4
### Log into your instance (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-manually#log-into-your-instance)

Navigate to your Tinybird project and run `tb login --host <host>`, where `<host>` is the host of your self-managed region that you set up in the previous step.

```
tb login --host https://<host>
```
5
### Use your instance (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-manually#use-your-instance)

After you're logged in, you can run commands against it using the `--cloud` flag.

```
tb --cloud workspace ls
```

## Examples (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-manually#examples)

The following examples show how to add a self-managed region manually in different cloud providers.

- [Minimal setup on a dedicated VM](https://gist.github.com/alejandromav/f4b61d77580ad9596a0a92c6a2df7c19)

## Troubleshooting (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-manually#troubleshooting)

If you encounter issues with your self-managed region:

- Check the container logs for error messages.
- Verify that all required environment variables are set correctly.
- Ensure your network configuration allows the necessary connections.
- Confirm that your persistent volumes are properly mounted and have sufficient space.
- For connectivity issues, verify that your HTTPS endpoint is properly configured.
For additional help, contact Tinybird support with details about your deployment.

---

URL: https://www.tinybird.co/docs/forward/guides/add-self-managed-region-cli
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Use the CLI to add a self-managed region \| Tinybird"
theme-color: "#171612"
description: "Create your own Tinybird Cloud region in the cloud service provider of your choice."
inkeep:version: "forward"
---

# Use the CLI to add a self-managed region (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-cli#use-the-cli-to-add-a-self-managed-region)

The [tb infra init](/docs/forward/dev-reference/commands/tb-infra) command helps you set up the requirements for a self-managed region and deploy the [Tinybird Local](/docs/forward/core-concepts/tinybird-local) container on your cloud provider.

When you run `tb infra init`, the command:

1. Registers a new self-managed region with Tinybird Cloud.
2. Generates infrastructure-as-code files for your chosen cloud provider: A certificate in AWS Certificate Manager and the necessary DNS records in Route 53 for validating the certificate.
3. Generates a Kubernetes configuration file to deploy the container on your EKS cluster with the correct environment variables. The command prompts you for the Kubernetes context to use, the namespace to create the deployment in, and the storage class to use.
4. Optionally, it applies these configurations to create the necessary resources.
This automated approach significantly reduces the manual steps required to set up a self-managed region, especially on AWS where it handles certificate management, DNS configuration, and Kubernetes deployment.

Self-managed regions are currently in beta. Use them for development environments only; avoid production workloads. Features, requirements, and implementation details might change as Tinybird continues to improve this capability.

## Supported clouds (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-cli#supported-clouds)

The following cloud service providers are supported:

- AWS
- GCP (Coming soon)
- Azure (Coming soon)

## Use tb infra init (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-cli#use-tb-infra-init)

Follow these steps to create a self-managed region using `tb infra init`.

1
### Check the requirements (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-cli#check-the-requirements)

To run the `tb infra init` command, you need to have the following.

- AWS
The following tools are required to deploy Tinybird Local on AWS:

- [Terraform CLI](https://developer.hashicorp.com/terraform/install) to create the Kubernetes cluster.
- [Kubectl CLI](https://kubernetes.io/docs/tasks/tools/) to manage your Kubernetes installation.
- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) with credentials configured.
Before initiating deployment, you need to set up the following in AWS:

- A zone and domain name in [Route53 zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/CreatingHostedZone.html).

  - Write down the hosted zone name and the zone ID for your domain.
- An [EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html) with the following components:

  - [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/) installed.
  - [external-dns](https://github.com/kubernetes-sigs/external-dns) configured.
  - Both components need sufficient permissions to manage resources.
  - Optionally, set a storage class in the EKS cluster with high IOPS and throughput.
2
### Log into Tinybird Cloud (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-cli#log-into-tinybird-cloud)

Log into Tinybird Cloud using the `tb login` command.

```
tb login
```
3
### Run tb infra init (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-cli#run-tb-infra-init)

Run `tb infra init` to set up the requirements for a self-managed region and deploy Tinybird Local on your cloud provider.

```
tb infra init
```
The command requires the following information:

- AWS
When prompted, enter the following information from the first step:

- AWS region. For example, `us-east-1`.
- DNS zone name. For example, `example.com`.
- DNS record name. For example, `tinybird`
- The Kubernetes context to apply the changes on.
- Kubernetes namespace. For example, `default`.
- EKS storage class. For example, `gp3-encrypted`.
Review the changes before applying them. To generate the files without applying them, use the `--skip-apply` flag.

After the deployment is complete, `tb infra` shows the URL to access your Tinybird Local instance.

4
### Log into your instance (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-cli#log-into-your-instance)

Navigate to your Tinybird project and run `tb login --host <host>`, where `<host>` is the host of your self-managed region that you set up in the previous step.

```
tb login --host https://<host>
```
5
### Use your instance (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-cli#use-your-instance)

After you're logged in, you can run commands against it using the `--cloud` flag.

```
tb --cloud workspace ls
```

## Troubleshooting (https://www.tinybird.co/docs/forward/guides/add-self-managed-region-cli#troubleshooting)

If you encounter issues with your self-managed region:

- Check the container logs for error messages.
- Verify that all required environment variables are set correctly.
- Ensure your network configuration allows the necessary connections.
- Confirm that your persistent volumes are properly mounted and have sufficient space.
- For connectivity issues, verify that your HTTPS endpoint is properly configured.
For additional help, contact Tinybird support with details about your deployment.

---

URL: https://www.tinybird.co/docs/forward/copy-export-data/s3-sink
Last update: 2026-07-31T11:10:11.000Z
Content:
---
title: "S3 Sink \| Tinybird"
theme-color: "#171612"
description: "Offload data to S3 on a batch-based schedule using Tinybird's fully managed S3 Sink Connector."
inkeep:version: "forward"
---

# S3 Sink (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#s3-sink)

Set up an S3 Sink to export data from Tinybird to any S3 bucket in CSV, NDJSON, or Parquet format. Offload data on a batch-based schedule using Tinybird's fully managed Connector.

Setting up the S3 Sink requires:

1. Configuring AWS [permissions](https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#aws-permissions) using [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html).
2. Creating a connection file in Tinybird.
3. Creating a Sink pipe that uses this connection.
The S3 Sink feature is available for Developer and Enterprise plans. See [Plans](/docs/forward/pricing).

## Environment considerations (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#environment-considerations)

Before setting up the S3 Sink, understand how it works in different environments.

### Cloud environment (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#cloud-environment)

In the Tinybird Cloud environment, Tinybird uses its own AWS account to assume the IAM role you create and write to your S3 bucket.

### Local environment (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#local-environment)

When using the S3 Sink in the Tinybird Local environment, which runs in a container, you need to pass your local AWS credentials to the container. These credentials must have the [permissions described in the AWS permissions section](https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#aws-permissions), including access to S3 operations like `PutObject`, `ListBucket`, etc. This allows Tinybird Local to assume the IAM role you specify in your connection.

To pass your AWS credentials, use the `--use-aws-creds` flag when starting Tinybird Local:

```
tb local start --use-aws-creds
» Starting Tinybird Local...
✓ AWS credentials found and will be passed to Tinybird Local (region: us-east-1)
* Waiting for Tinybird Local to be ready...
✓ Tinybird Local is ready!
```
If you're using a specific AWS profile, you can specify it using the `AWS_PROFILE` environment variable:

```
AWS_PROFILE=my-profile tb local start --use-aws-creds
```
When using the S3 Sink in the `--local` environment, scheduled sink operations are not supported. You can only run on-demand sinks using `tb sink run <pipe_name>`. For scheduled sink operations, use the Cloud environment.

## AWS permissions (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#aws-permissions)

The S3 Sink requires an IAM Role with specific permissions to write objects to your Amazon S3 bucket:

- `s3:GetObject`
- `s3:PutObject`
- `s3:PutObjectAcl`
- `s3:ListBucket`
- `s3:GetBucketLocation`
**One IAM role can access multiple buckets**: Update the access policy to include multiple buckets by adding their ARNs to the `Resource` array. Reuse the same IAM role across multiple S3 Sink Connections for different buckets.

You need to create both an access policy and a trust policy in AWS:

- AWS Access Policy
- AWS Trust Policy

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::{bucket-name}/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::{bucket-name}"
        }
    ]
}
```

## Set up the sink (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#set-up-the-sink)

1
### Create an S3 connection (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#create-an-s3-connection)

You can create an S3 connection in Tinybird using either the guided CLI process or by manually creating a connection file.

#### Option 1: Use the guided CLI process (recommended) (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#option-1-use-the-guided-cli-process-recommended)

The Tinybird CLI provides a guided process that helps you set up the required AWS permissions and creates the connection file automatically:

```
tb connection create s3
```
When prompted, you need to:

1. Enter a name for your connection.
2. Specify whether you use this Connection for sinking or ingesting data.
3. Enter the S3 bucket name.
4. Enter the AWS region where your bucket is located.
5. Copy the displayed AWS IAM policy to your clipboard. You need it to set up permissions in AWS.
6. Copy the displayed AWS IAM role trust policy for your Local environment, then enter the ARN of the role you create.
7. Copy the displayed AWS IAM role trust policy for your Cloud environment, then enter the ARN of the role you create.
8. Store the ARN values securely using [tb secret](/docs/forward/dev-reference/commands/tb-secret) to use different roles for each environment.

#### Option 2: Define the Connection manually (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#option-2-define-the-connection-manually)

You can also define the S3 Connection manually in your project. The recommended authentication method is an IAM role with secrets for the role ARN.

- Tinybird CLI
- TypeScript SDK
- Python SDK
connections/s3sample.connection
```
TYPE s3
S3_REGION "<S3_REGION>"
S3_ARN {{ tb_secret("AWS_ROLE_ARN") }}
```
When creating your Connection manually, set up the required AWS IAM role with appropriate permissions. See the [AWS permissions](https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#aws-permissions) section for details on the required access policy and trust policy configurations.

Use IAM role authentication instead of HMAC authentication. It uses temporary credentials and follows AWS security best practices.

See [Connection files](/docs/forward/dev-reference/datafiles/connection-files) for more details on how to create a connection file and manage secrets.

Create separate connections for each environment: Local and Cloud.

For example, you can create:

- `my-s3-local` for your Local environment
- `my-s3-cloud` for your Cloud environment
2
### Create a Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#create-a-sink-pipe)

To create a Sink pipe, filter the data you want to export to your bucket in the SQL section as in any other pipe. Then, specify the pipe as a Sink and add the needed configuration.

- Tinybird CLI
- TypeScript SDK
- Python SDK
s3_export.pipe
```
NODE node_0

SQL >
    SELECT *
    FROM events
    WHERE status = 'processed'

TYPE sink
EXPORT_CONNECTION_NAME "s3sample"
EXPORT_BUCKET_URI "s3://tinybird-sinks"
EXPORT_FILE_TEMPLATE "daily_prices" # Supports partitioning
EXPORT_SCHEDULE "*/5 * * * *"
EXPORT_FORMAT "csv" # Optional
EXPORT_COMPRESSION "gz" # Optional
EXPORT_STRATEGY "create_new" # Optional
```
3
### Deploy the Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#deploy-the-sink-pipe)

After defining your S3 data source and connection, test it by running a deploy check:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb --cloud deploy --check
```
This runs the connection locally and checks if the connection is valid. To see the connection details, run `tb --cloud connection ls`.

When ready, push the datafile to your Workspace using `tb deploy` to create the Sink pipe:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb --cloud deploy
```
This creates the Sink pipe in your workspace and makes it available for execution.

## Connection file settings (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#connection-file-settings)

The S3 Connector uses the following settings in `.connection` files:

| Instruction | Required | Description |
| --- | --- | --- |
| `S3_REGION` | Yes | Region of the S3 bucket. |
| `S3_ARN` | No* | ARN of the IAM role with the required permissions. Required for IAM Role authentication. |
| `S3_ACCESS_KEY` | No* | AWS access key for HMAC authentication. Store as a [Tinybird secret](/docs/forward/dev-reference/commands/tb-secret). |
| `S3_SECRET` | No* | AWS secret key for HMAC authentication. Store as a [Tinybird secret](/docs/forward/dev-reference/commands/tb-secret). |
*Either `S3_ARN` (for IAM Role authentication) or both `S3_ACCESS_KEY` and `S3_SECRET` (for HMAC authentication) are required.

## Pipe settings (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#pipe-settings)

The S3 Sink pipe uses the following settings in .pipe files:

| Key | Type | Description |
| --- | --- | --- |
| `EXPORT_CONNECTION_NAME` | string | Required. The connection name to the destination service. This the connection created in Step 1. |
| `EXPORT_BUCKET_URI` | string | Required. The path to the destination bucket. Example: `s3://tinybird-export` |
| `EXPORT_FILE_TEMPLATE` | string | Required. The target filename. Can use parameters to dynamically name and partition the files. See [File template](https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#file-template). Example: `daily_prices_{customer_id}` |
| `EXPORT_SCHEDULE` | string | Required. A crontab expression that sets the frequency of the Sink operation or the @on-demand string. |
| `EXPORT_FORMAT` | string | Optional. The output format of the file. Values: CSV, NDJSON, Parquet. Default value: CSV |
| `EXPORT_COMPRESSION` | string | Optional. Accepted values: `none`, `gz` for gzip, `br` for brotli, `xz` for LZMA, `zst` for zstd. Default: `none` |
| `EXPORT_STRATEGY` | string | Optional. Defines how to handle existing files. Values: `create_new` (default), `replace`. See [Write strategies](https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#write-strategies). |

### Supported regions (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#supported-regions)

The Tinybird S3 Sink feature only supports exporting data to the following AWS regions:

- `us-east-*`
- `us-west-*`
- `eu-central-*`
- `eu-west-*`
- `eu-south-*`
- `eu-north-*`

### Scheduling considerations (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#scheduling-considerations)

The schedule applied to a Sink pipe doesn't guarantee that the underlying job executes immediately at the configured time. The job is placed into a job queue when the configured time elapses. It is possible that, if the queue is busy, the job could be delayed and executed after the scheduled time.

To reduce the chances of a busy queue affecting your Sink pipe execution schedule, distribute the jobs over a wider period of time rather than grouping them close together.

### Write strategies (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#write-strategies)

The `EXPORT_STRATEGY` parameter determines how Tinybird handles existing files in your S3 bucket:

- **`create_new`** (default): Creates new files without overwriting existing ones. If a file with the same name already exists, Tinybird appends a suffix to make the filename unique.
- **`replace`**: Overwrites existing files with the same name. Use this when you want to replace previous exports entirely.

### Query parameters (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#query-parameters)

You can add [query parameters](/docs/forward/query-data/query-parameters) to your Sink pipes, the same way you do in API Endpoints or Copy pipes.

- For on-demand executions, you can set parameters when you trigger the Sink pipe to whatever values you wish.
- For scheduled executions, the Sink pipe uses the default parameter values.

## Execute the Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#execute-the-sink-pipe)

### On-demand execution (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#on-demand-execution)

You can trigger your Sink pipe manually using:

```
tb sink run <pipe_name>
```
When triggering a Sink pipe you have the option of overriding several of its settings, like format or compression. Refer to the [Sink pipes API spec](/docs/api-reference/sink-pipes-api) for the full list of parameters.

### Scheduled execution (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#scheduled-execution)

If you configured a schedule with `EXPORT_SCHEDULE`, the Sink pipe runs automatically according to the cron expression.

Once the Sink pipe is triggered, it creates a standard Tinybird job that can be followed via the `v0/jobs` API or using `tb job ls --kind=sink`.

## File template (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#file-template)

Partition the result into different files to organize your data and get smaller files. The file template defines the partitioning based on the values of columns in the result set.

### Partition by column (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#partition-by-column)

Add a template variable like `{COLUMN_NAME}` to the filename. For instance, consider the following query schema and result for an export:

| customer_id | invoice_id | amount |
| --- | --- | --- |
| ACME | INV20230608 | 23.45 |
| ACME | 12345INV | 12.3 |
| GLOBEX | INV-ABC-789 | 35.34 |
| OSCORP | INVOICE2023-06-08 | 57 |
| ACME | INV-XYZ-98765 | 23.16 |
| OSCORP | INV210608-001 | 62.23 |
| GLOBEX | 987INV654 | 36.23 |
With the given file template `invoice_summary_{customer_id}.csv` you'd get 3 files:

`invoice_summary_ACME.csv`

| customer_id | invoice_id | amount |
| --- | --- | --- |
| ACME | INV20230608 | 23.45 |
| ACME | 12345INV | 12.3 |
| ACME | INV-XYZ-98765 | 23.16 |
`invoice_summary_OSCORP.csv`

| customer_id | invoice_id | amount |
| --- | --- | --- |
| OSCORP | INVOICE2023-06-08 | 57 |
| OSCORP | INV210608-001 | 62.23 |
`invoice_summary_GLOBEX.csv`

| customer_id | invoice_id | amount |
| --- | --- | --- |
| GLOBEX | INV-ABC-789 | 35.34 |
| GLOBEX | 987INV654 | 36.23 |

### Values format (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#values-format)

For DateTime columns, partitioning by the column can create as many files as seconds because each second is a different value. In an hour, that's potentially 3600 files.

To help partition in a sensible way, you can add a format string to the column name using the following placeholders:

| Placeholder | Description | Example |
| --- | --- | --- |
| %Y | Year | 2023 |
| %m | Month as an integer number (01-12) | 06 |
| %d | Day of the month, zero-padded (01-31) | 07 |
| %H | Hour in 24 h format from 00 through 23 | 14 |
| %i | Minute (00-59) | 45 |
For instance, for a result like this:

| timestamp | invoice_id | amount |
| --- | --- | --- |
| 2023-07-07 09:07:05 | INV20230608 | 23.45 |
| 2023-07-07 09:07:01 | 12345INV | 12.3 |
| 2023-07-07 09:06:45 | INV-ABC-789 | 35.34 |
| 2023-07-07 09:05:35 | INVOICE2023-06-08 | 57 |
| 2023-07-06 23:14:05 | INV-XYZ-98765 | 23.16 |
| 2023-07-06 23:14:02 | INV210608-001 | 62.23 |
| 2023-07-06 23:10:55 | 987INV654 | 36.23 |
All 7 events have different times in the column timestamp. Using a file template like `invoices_{timestamp}` would create 7 different files.

If you were interested in writing one file per hour, you could use a file template like `invoices_{timestamp, '%Y%m%d-%H'}`. You'd then get only two files for that dataset:

`invoices_20230707-09.csv`

| timestamp | invoice_id | amount |
| --- | --- | --- |
| 2023-07-07 09:07:05 | INV20230608 | 23.45 |
| 2023-07-07 09:07:01 | 12345INV | 12.3 |
| 2023-07-07 09:06:45 | INV-ABC-789 | 35.34 |
| 2023-07-07 09:05:35 | INVOICE2023-06-08 | 57 |
`invoices_20230706-23.csv`

| timestamp | invoice_id | amount |
| --- | --- | --- |
| 2023-07-06 23:14:05 | INV-XYZ-98765 | 23.16 |
| 2023-07-06 23:14:02 | INV210608-001 | 62.23 |
| 2023-07-06 23:10:55 | 987INV654 | 36.23 |

### By number of files (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#by-number-of-files)

You also have the option to write the result into X files. Instead of using a column name, use an integer between brackets.

Example: `invoice_summary.{8}.csv`

This is convenient to reduce the file size of the result, especially when the files are meant to be consumed by other services, like Snowflake where uploading big files is discouraged.

The results are written in random order. This means that the final result rows would be written in X files, but you can't count the specific order of the result.

There are a maximum of 16 files.

### Combining different partitions (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#combining-different-partitions)

It's possible to add more than one partitioning parameter in the file template. This is useful, for instance, when you do a daily dump of data, but want to export one file per hour.

Setting the file template as `invoices/dt={timestamp, '%Y-%m-%d'}/H{timestamp, '%H}.csv` would create the following file structure in different days and executions:

```
Invoices
├── dt=2023-07-07
│   └── H23.csv
│   └── H22.csv
│   └── H21.csv
│   └── ...
├── dt=2023-07-06
│   └── H23.csv
│   └── H22.csv
```
You can also mix column names and number of files. For instance, setting the file template as `invoices/{customer_id}/dump_{4}.csv` would create the following file structure in different days and executions:

```
Invoices
├── ACME
│   └── dump_0.csv
│   └── dump_1.csv
│   └── dump_2.csv
│   └── dump_3.csv
├── OSCORP
│   └── dump_0.csv
│   └── dump_1.csv
│   └── dump_2.csv
│   └── dump_3.csv
```
Avoid excessive partitioning. The write process creates as many files as combinations of the partitioning column values for a given result set.

## Supported file types (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#supported-file-types)

The S3 Sink supports exporting data in the following file formats:

| File type | Accepted extensions | Compression formats supported |
| --- | --- | --- |
| CSV | `.csv`, `.csv.gz` | `gzip` |
| NDJSON | `.ndjson`, `.ndjson.gz`, `.jsonl`, `.jsonl.gz`, `.json`, `.json.gz` | `gzip` |
| Parquet | `.parquet`, `.parquet.gz` | `snappy`, `gzip`, `lzo`, `brotli`, `lz4`, `zstd` |
You can optionally configure the export format using the `EXPORT_FORMAT` parameter (defaults to CSV) and compression using the `EXPORT_COMPRESSION` parameter in your Sink pipe configuration.

## Observability (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#observability)

Sink pipes operations are logged in the [tinybird.jobs_log](/docs/forward/monitoring/service-datasources#tinybird-jobs-log) Service Data Source. You can filter by `job_type = 'sink'` to see only Sink pipe executions.

For more detailed Sink-specific information, you can also use [tinybird.sinks_ops_log](/docs/forward/monitoring/service-datasources#tinybird-sinks-ops-log).

Data Transfer incurred by Sink pipes is tracked in [tinybird.data_transfer](/docs/forward/monitoring/service-datasources#tinybird-data-transfer) Service Data Source.

## Limits & quotas (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#limits-quotas)

Check the [limits page](/docs/forward/pricing/limits) for limits on ingestion, queries, API Endpoints, and more.

## Billing (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#billing)

Tinybird bills Sink pipes based on Data Transfer. When a Sink pipe executes, it uses your plan's included compute resources to run the query, then writes the result to a bucket as Data Transfer. If Tinybird compresses the resulting files, it accounts for the compressed size.

### Data transfer (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#data-transfer)

Data Transfer depends on your environment. There are two scenarios:

- The destination bucket is in the **same** cloud provider and region as your Tinybird Workspace: $0.01 / GB
- The destination bucket is in a **different** cloud provider or region as your Tinybird Workspace: $0.10 / GB

## Next steps (https://www.tinybird.co/docs/forward/copy-export-data/s3-sink#next-steps)

- Get familiar with the [Service Data Source](/docs/forward/monitoring/service-datasources) and see what's going on in your account
- Deep dive on Tinybird's [pipes concept](/docs/forward/core-concepts/pipes)

---

URL: https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Kafka Sink \| Tinybird"
theme-color: "#171612"
description: "Push events to Kafka on a batch-based schedule using Tinybird's fully managed Kafka Sink Connector."
inkeep:version: "forward"
---

# Kafka Sink (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#kafka-sink)

Set up a Kafka Sink to export your data from Tinybird to any Kafka topic. Push events to Kafka on a batch-based schedule using Tinybird's fully managed Connector.

Setting up the Kafka Sink requires:

1. Creating a connection file in Tinybird with your Kafka configuration.
2. Creating a Sink pipe that uses this connection.
Kafka Sinks are available on the Developer and Enterprise plans. See [Plans](/docs/forward/pricing).

## Environment considerations (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#environment-considerations)

Before setting up the Kafka Sink, understand how it works in different environments.

### Cloud environment (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#cloud-environment)

In the Tinybird Cloud environment, Tinybird connects directly to your Kafka cluster using the connection credentials you provide.

### Local environment (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#local-environment)

When using the Kafka Sink in the Tinybird Local environment, the connection is made from within the container to your Kafka cluster. Ensure your Kafka cluster is accessible from the container network.

In the `--local` environment, you can only run on-demand Kafka Sinks using `tb sink run <pipe_name>`. Use the Cloud environment for scheduled Sink operations.

## Set up the sink (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#set-up-the-sink)

1
### Create a Kafka connection (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#create-a-kafka-connection)

You can create a Kafka connection in Tinybird using either the guided CLI process or by manually creating a connection file.

#### Option 1: Use the guided CLI process (recommended) (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#option-1-use-the-guided-cli-process-recommended)

The Tinybird CLI provides a guided process that helps you set up the Kafka connection:

```
tb connection create kafka
```
When prompted, provide:

1. Enter a name for your connection.
2. Provide the Kafka bootstrap servers (comma-separated list).
3. Configure authentication settings (SASL/SSL if required).
4. Optionally configure additional Kafka client properties.

#### Option 2: Define the Connection manually (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#option-2-define-the-connection-manually)

You can also define the Kafka Connection manually in your project. Store credentials as secrets instead of writing them in plain text.

- Tinybird CLI
- TypeScript SDK
- Python SDK
connections/kafka_sample.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS "bootstrap_servers:port"
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("KAFKA_KEY") }}
KAFKA_SECRET {{ tb_secret("KAFKA_SECRET") }}
```
See [Connection files](/docs/forward/dev-reference/datafiles/connection-files) for more details on how to create a connection file and manage secrets.

You need to create separate Connections for each environment you're working with, Local and Cloud.

For example, you can create:

- `kafka-local` for your Local environment
- `kafka-cloud` for your Cloud environment
2
### Create a Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#create-a-sink-pipe)

To create a Sink pipe, filter the data you want to export to your Kafka topic in the SQL section as in any other pipe. Then, specify the pipe as a Sink and add the needed configuration.

- Tinybird CLI
- TypeScript SDK
- Python SDK
kafka_export.pipe
```
NODE node_0

SQL >
    SELECT
        customer_id,
        event_type,
        status,
        amount
    FROM events
    WHERE status = 'completed'

TYPE sink
EXPORT_CONNECTION_NAME "kafka_connection"
EXPORT_KAFKA_TOPIC "events_topic"
EXPORT_SCHEDULE "*/5 * * * *"
```
3
### Deploy the Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#deploy-the-sink-pipe)

After defining your Sink pipe and connection, test the Kafka connection:

```
tb connection data <connection_name>
```
This command tests the Kafka connection configuration by attempting to consume from a topic (it's designed for the Kafka source connector). While it validates connection details, for Kafka Sinks, the connection is used to produce data to Kafka, not consume from it. To see the connection details, run `tb --cloud connection ls`.

When ready, push the datafile to your Workspace using `tb deploy` to create the Sink pipe:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb --cloud deploy
```
This creates the Sink pipe in your workspace and makes it available for execution.

## Connection file settings (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#connection-file-settings)

The Kafka connector uses the following settings in `.connection` files:

| Instruction | Required | Description |
| --- | --- | --- |
| `KAFKA_BOOTSTRAP_SERVERS` | Yes | Comma-separated list of one or more Kafka brokers, including Port numbers. |
| `KAFKA_KEY` | Yes | Key used to authenticate with Kafka. Sometimes called Key, Client Key, or Username depending on the Kafka distribution. |
| `KAFKA_SECRET` | Yes | Secret used to authenticate with Kafka. Sometimes called Secret, Secret Key, or Password depending on the Kafka distribution. |
| `KAFKA_SECURITY_PROTOCOL` | No | Security protocol for the connection. Accepted values are `PLAINTEXT` and `SASL_SSL`. Default value is `SASL_SSL`. |
| `KAFKA_SASL_MECHANISM` | No | SASL mechanism to use for authentication. Supported values are `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`. Default value is `PLAIN`. |
| `KAFKA_SCHEMA_REGISTRY_URL` | No | URL of the Kafka schema registry. Used for `avro` and `json_with_schema` deserialization of keys and values. If the registry requires Basic Auth, include the username and password in the URL user information component. |
| `KAFKA_SSL_CA_PEM` | No | Content of the CA certificate in PEM format for SSL connections. |

## Pipe file settings (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#pipe-file-settings)

The Kafka Sink pipe uses the following settings in .pipe files:

| Key | Type | Description |
| --- | --- | --- |
| `EXPORT_CONNECTION_NAME` | string | Required. The connection name to the destination service. This is the connection created in Step 1. |
| `EXPORT_KAFKA_TOPIC` | string | Required. The Kafka topic where events are published. |
| `EXPORT_SCHEDULE` | string | Required. A crontab expression that sets the frequency of the Sink operation or the @on-demand string. |

### Scheduling considerations (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#scheduling-considerations)

The schedule applied to a Sink pipe doesn't guarantee that the underlying job executes immediately at the configured time. The job is placed into a job queue when the configured time elapses. It is possible that, if the queue is busy, the job could be delayed and executed after the scheduled time.

To reduce the chances of a busy queue affecting your Sink pipe execution schedule, distribute the jobs over a wider period of time rather than grouping them close together.

### Query parameters (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#query-parameters)

You can add [query parameters](/docs/forward/query-data/query-parameters) to your Sink pipes, the same way you do in API Endpoints or Copy pipes.

- For on-demand executions, you can set parameters when you trigger the Sink pipe to whatever values you wish.
- For scheduled executions, the default values for the parameters are used when the Sink pipe runs.

## Execute the Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#execute-the-sink-pipe)

### On-demand execution (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#on-demand-execution)

You can trigger your Sink pipe manually using:

```
tb sink run <pipe_name>
```
When triggering a Sink pipe you have the option of overriding several of its settings, like topic or format. Refer to the [Sink pipes API spec](/docs/api-reference/sink-pipes-api) for the full list of parameters.

### Scheduled execution (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#scheduled-execution)

If you configured a schedule with `EXPORT_SCHEDULE`, the Sink pipe runs automatically according to the cron expression.

Once the Sink pipe is triggered, it creates a standard Tinybird job that can be followed via the `v0/jobs` API or using `tb job ls --kind=sink`.

## Observability (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#observability)

Sink pipes operations are logged in the [tinybird.jobs_log](/docs/forward/monitoring/service-datasources#tinybird-jobs-log) Service Data Source. You can filter by `job_type = 'sink'` to see only Sink pipe executions.

For more detailed Sink-specific information, you can also use [tinybird.sinks_ops_log](/docs/forward/monitoring/service-datasources#tinybird-sinks-ops-log).

## Limits & quotas (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#limits-quotas)

Check the [limits page](/docs/forward/pricing/limits) for limits on ingestion, queries, API Endpoints, and more.

## Billing (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#billing)

When a Sink pipe executes, it uses your plan's included compute resources to run the query, then publishes the result to Kafka.

### Enterprise customers (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#enterprise-customers)

Tinybird includes Data Transfer allowances for Enterprise customers. Contact your Customer Success team or [Tinybird support](/docs/forward/support) to discuss your specific requirements.

## Performance (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#performance)

### Throughput considerations (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#throughput-considerations)

Kafka Sink performance depends on:

- Query result size
- Network latency to Kafka cluster
- Kafka cluster capacity
- Message serialization overhead
**Monitor Sink performance:**

```
SELECT
    pipe_name,
    job_type,
    status,
    elapsed_time,
    rows_written,
    bytes_written
FROM tinybird.jobs_log
WHERE job_type = 'sink'
  AND timestamp > now() - INTERVAL 24 hour
ORDER BY timestamp DESC
```

### Optimization tips (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#optimization-tips)

1. **Batch size**: Larger batches improve throughput but increase latency
2. **Schedule frequency**: Balance between freshness and resource usage
3. **Query optimization**: Optimize your Sink pipe queries for performance
4. **Message format**: Use efficient serialization (JSON is typically fastest)

## Use cases and examples (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#use-cases-and-examples)

### Real-time event streaming (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#real-time-event-streaming)

Stream processed events to downstream systems:

event_stream.pipe
```
NODE node_0

SQL >
    SELECT
        user_id,
        event_type,
        timestamp,
        metadata
    FROM processed_events
    WHERE timestamp > now() - INTERVAL 1 hour
      AND status = 'processed'

TYPE sink
EXPORT_CONNECTION_NAME "kafka_connection"
EXPORT_KAFKA_TOPIC "downstream_events"
EXPORT_SCHEDULE "*/5 * * * *"
```

### Data synchronization (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#data-synchronization)

Sync data changes to external systems:

user_sync.pipe
```
NODE node_0

SQL >
    SELECT
        user_id,
        email,
        updated_at,
        status
    FROM users
    WHERE updated_at > now() - INTERVAL 1 hour

TYPE sink
EXPORT_CONNECTION_NAME "kafka_connection"
EXPORT_KAFKA_TOPIC "user_updates"
EXPORT_SCHEDULE "0 * * * *"
```

### Analytics data export (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#analytics-data-export)

Export aggregated analytics to Kafka for further processing:

analytics_export.pipe
```
NODE node_0

SQL >
    SELECT
        toStartOfHour(timestamp) as hour,
        event_type,
        count() as event_count,
        uniq(user_id) as unique_users
    FROM events
    WHERE timestamp > now() - INTERVAL 1 hour
    GROUP BY hour, event_type

TYPE sink
EXPORT_CONNECTION_NAME "kafka_connection"
EXPORT_KAFKA_TOPIC "hourly_analytics"
EXPORT_SCHEDULE "0 * * * *"
```

## Troubleshooting (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#troubleshooting)

### Issue: Sink not executing (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#issue-sink-not-executing)

**Symptoms:**

- No jobs in `jobs_log`
- Scheduled Sink not running
**Solutions:**

1. Verify schedule syntax is correct (cron format)
2. Check Sink pipe is deployed: `tb pipe ls`
3. Test connection: `tb connection data <connection_name>` (validates connection details; note this command is for the Kafka source connector and tests consumption, while Sinks produce data)
4. Check for errors in `jobs_log`

### Issue: Messages not appearing in Kafka (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#issue-messages-not-appearing-in-kafka)

**Symptoms:**

- Sink executes successfully
- No messages in Kafka topic
**Solutions:**

1. Verify topic name is correct
2. Check Kafka connection credentials
3. Verify topic exists in Kafka cluster
4. Check Kafka producer permissions
5. Review `sinks_ops_log` for errors

### Issue: Slow Sink execution (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#issue-slow-sink-execution)

**Symptoms:**

- Long execution times
- Timeouts
**Solutions:**

1. Optimize Sink pipe query
2. Reduce query result size
3. Add filters to limit data
4. Check network latency to Kafka
5. Monitor Kafka cluster performance

### Monitoring Sink operations (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#monitoring-sink-operations)

**Check Sink execution status:**

```
SELECT
    timestamp,
    pipe_name,
    status,
    elapsed_time,
    rows_written,
    error
FROM tinybird.jobs_log
WHERE job_type = 'sink'
  AND timestamp > now() - INTERVAL 24 hour
ORDER BY timestamp DESC
```
**Check Sink-specific metrics:**

```
SELECT
    timestamp,
    pipe_name,
    topic,
    messages_sent,
    bytes_sent,
    error
FROM tinybird.sinks_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
ORDER BY timestamp DESC
```

## Next steps (https://www.tinybird.co/docs/forward/copy-export-data/kafka-sink#next-steps)

- Get familiar with the [Service Data Source](/docs/forward/monitoring/service-datasources) and see what's going on in your account
- Deep dive on Tinybird's [pipes concept](/docs/forward/core-concepts/pipes)
- Review [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka) for Kafka setup
- Learn about [monitoring Sink operations](/docs/forward/monitoring/service-datasources#tinybird-sinks-ops-log)

---

URL: https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink
Last update: 2026-07-31T11:10:11.000Z
Content:
---
title: "GCS Sink \| Tinybird"
theme-color: "#171612"
description: "Offload data to Google Cloud Storage on a batch-based schedule using Tinybird's fully managed GCS Sink Connector."
inkeep:version: "forward"
---

# GCS Sink (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#gcs-sink)

Set up a GCS Sink to export data from Tinybird to any GCS bucket in CSV, NDJSON, or Parquet format. Offload data on a batch-based schedule using Tinybird's fully managed Connector.

Setting up the GCS Sink requires:

1. Configuring a [Service Account](https://cloud.google.com/iam/docs/service-accounts-create) with these [permissions](https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#gcs-permissions) in GCP.
2. Creating a connection file in Tinybird.
3. Creating a Sink pipe that uses this connection
The GCS Sink feature is available for Developer and Enterprise plans. See [Plans](/docs/forward/pricing).

## Environment considerations (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#environment-considerations)

Before setting up the GCS Sink, understand how it works in different environments.

### Cloud environment (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#cloud-environment)

In the Tinybird Cloud environment, Tinybird uses the Service Account credentials you provide to write to your GCS bucket.

### Local environment (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#local-environment)

When using the GCS Sink in the Tinybird Local environment, which runs in a container, you need to pass your local GCP credentials to the container. These credentials must have the [permissions described in the GCS permissions section](https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#gcs-permissions), including access to GCS operations like `storage.objects.create`, `storage.objects.get`, etc.

When using the GCS Sink in the `--local` environment, scheduled sink operations are not supported. You can only run on-demand sinks using `tb sink run <pipe_name>`. For scheduled sink operations, use the Cloud environment.

## Set up the sink (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#set-up-the-sink)

1
### Create a GCS connection (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#create-a-gcs-connection)

You can create a GCS connection in Tinybird using either the guided CLI process or by manually creating a connection file.

#### Option 1: Use the guided CLI process (recommended) (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#option-1-use-the-guided-cli-process-recommended)

The Tinybird CLI provides a guided process that helps you set up the required GCP permissions and creates the connection file automatically:

```
tb connection create gcs
```
When prompted, you need to:

1. Enter a name for your connection.
2. Enter the GCS bucket name.
3. Provide the service account credentials (JSON key file).
4. Store the credentials securely using [tb secret](/docs/forward/dev-reference/commands/tb-secret) to use different credentials for each environment.

#### Option 2: Define the Connection manually (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#option-2-define-the-connection-manually)

You can also define the GCS Connection manually in your project. Use a GCP Service Account with the credentials stored as a secret:

- Tinybird CLI
- TypeScript SDK
- Python SDK
connections/gcs_sample.connection
```
TYPE gcs
GCS_SERVICE_ACCOUNT_CREDENTIALS_JSON {{ tb_secret("GCS_KEY") }}
```
When creating your Connection manually, set up the required GCP Service Account with appropriate permissions. See the [GCS permissions](https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#gcs-permissions) section for details on the required role configurations.

Use Service Account authentication instead of HMAC authentication. It provides better integration with GCP IAM and more granular permission control.

See [Connection files](/docs/forward/dev-reference/datafiles/connection-files) for more details on how to create a connection file and manage secrets.

Create separate connections for each environment: Local and Cloud.

For example, you can create:

- `my-gcs-local` for your Local environment
- `my-gcs-cloud` for your Cloud environment
2
### Create a Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#create-a-sink-pipe)

To create a Sink pipe, filter the data you want to export to your bucket in the SQL section as in any other pipe. Then, specify the pipe as a Sink and add the needed configuration.

- Tinybird CLI
- TypeScript SDK
- Python SDK
gcs_export.pipe
```
NODE node_0

SQL >
    SELECT *
    FROM events
    WHERE status = 'processed'

TYPE sink
EXPORT_CONNECTION_NAME "gcs_sample"
EXPORT_BUCKET_URI "gs://tinybird-sinks"
EXPORT_FILE_TEMPLATE "daily_prices" # Supports partitioning
EXPORT_SCHEDULE "*/5 * * * *"
EXPORT_FORMAT "csv" # Optional
EXPORT_COMPRESSION "gz" # Optional
EXPORT_STRATEGY "create_new" # Optional
```
3
### Deploy the Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#deploy-the-sink-pipe)

After defining your GCS data source and connection, test it by running a deploy check:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb --cloud deploy --check
```
This runs the connection locally and checks if the connection is valid. To see the connection details, run `tb --cloud connection ls`.

When ready, push the datafile to your Workspace using `tb deploy` to create the Sink pipe:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb --cloud deploy
```
This creates the Sink pipe in your workspace and makes it available for execution.

## Connection file settings (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#connection-file-settings)

The GCS connector uses the following settings in `.connection` files:

| Instruction | Required | Description |
| --- | --- | --- |
| `GCS_SERVICE_ACCOUNT_CREDENTIALS_JSON` | No* | Service Account Key in JSON format for Service Account authentication. Store it using [Tinybird Secrets](/docs/forward/dev-reference/commands/tb-secret). |
| `GCS_ACCESS_ID` | No* | HMAC access ID for HMAC authentication. Store as a [Tinybird secret](/docs/forward/dev-reference/commands/tb-secret). |
| `GCS_SECRET` | No* | HMAC secret key for HMAC authentication. Store as a [Tinybird secret](/docs/forward/dev-reference/commands/tb-secret). |
*Either `GCS_SERVICE_ACCOUNT_CREDENTIALS_JSON` (for Service Account authentication) or both `GCS_ACCESS_ID` and `GCS_SECRET` (for HMAC authentication) are required.

## Pipe file settings (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#pipe-file-settings)

The GCS Sink pipe uses the following settings in `.pipe` files:

| Key | Type | Description |
| --- | --- | --- |
| `EXPORT_CONNECTION_NAME` | string | Required. The connection name to the destination service. This is the connection created in Step 1. |
| `EXPORT_BUCKET_URI` | string | Required. The path to the destination bucket. Example: `gs://tinybird-export` |
| `EXPORT_FILE_TEMPLATE` | string | Required. The target filename. Can use parameters to dynamically name and partition the files. See [File template](https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#file-template). Example: `daily_prices_{customer_id}` |
| `EXPORT_SCHEDULE` | string | Required. A crontab expression that sets the frequency of the Sink operation or the @on-demand string. |
| `EXPORT_FORMAT` | string | Optional. The output format of the file. Values: CSV, NDJSON, Parquet. Default value: CSV |
| `EXPORT_COMPRESSION` | string | Optional. Accepted values: `none`, `gz` for gzip, `br` for brotli, `xz` for LZMA, `zst` for zstd. Default: `none` |
| `EXPORT_STRATEGY` | string | Optional. Defines how to handle existing files. Values: `create_new` (default), `replace`. See [Write strategies](https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#write-strategies). |

### Supported regions (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#supported-regions)

The Tinybird GCS Sink feature only supports exporting data to the following GCP regions:

- `us-*`
- `eu-*`
- `us-central-*`
- `us-east-*`
- `us-west-*`
- `europe-west-*`
- `northamerica-northeast-*`

### Scheduling considerations (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#scheduling-considerations)

The schedule applied to a Sink pipe doesn't guarantee that the underlying job executes immediately at the configured time. The job is placed into a job queue when the configured time elapses. It is possible that, if the queue is busy, the job could be delayed and executed after the scheduled time.

To reduce the chances of a busy queue affecting your Sink pipe execution schedule, distribute the jobs over a wider period of time rather than grouping them close together.

### Write strategies (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#write-strategies)

The `EXPORT_STRATEGY` parameter determines how Tinybird handles existing files in your GCS bucket:

- **`create_new`** (default): Creates new files without overwriting existing ones. If a file with the same name already exists, Tinybird appends a suffix to make the filename unique.
- **`replace`**: Overwrites existing files with the same name. Use this when you want to replace previous exports entirely.

### Query parameters (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#query-parameters)

You can add [query parameters](/docs/forward/query-data/query-parameters) to your Sink pipes, the same way you do in API Endpoints or Copy pipes.

- For on-demand executions, you can set parameters when you trigger the Sink pipe to whatever values you wish.
- For scheduled executions, the Sink pipe uses the default parameter values.

## Execute the Sink pipe (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#execute-the-sink-pipe)

### On-demand execution (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#on-demand-execution)

You can trigger your Sink pipe manually using:

```
tb sink run <pipe_name>
```
When triggering a Sink pipe you have the option of overriding several of its settings, like format or compression. Refer to the [Sink pipes API spec](/docs/api-reference/sink-pipes-api) for the full list of parameters.

### Scheduled execution (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#scheduled-execution)

If you configured a schedule with `EXPORT_SCHEDULE`, the Sink pipe runs automatically according to the cron expression.

Once the Sink pipe is triggered, it creates a standard Tinybird job that can be followed via the `v0/jobs` API or using `tb job ls --kind=sink`.

## File template (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#file-template)

Partition the result into different files to organize your data and get smaller files. The file template defines the partitioning based on the values of columns in the result set.

### Partition by column (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#partition-by-column)

Add a template variable like `{COLUMN_NAME}` to the filename. For instance, consider the following query schema and result for an export:

| customer_id | invoice_id | amount |
| --- | --- | --- |
| ACME | INV20230608 | 23.45 |
| ACME | 12345INV | 12.3 |
| GLOBEX | INV-ABC-789 | 35.34 |
| OSCORP | INVOICE2023-06-08 | 57 |
| ACME | INV-XYZ-98765 | 23.16 |
| OSCORP | INV210608-001 | 62.23 |
| GLOBEX | 987INV654 | 36.23 |
With the given file template `invoice_summary_{customer_id}.csv` you'd get 3 files:

`invoice_summary_ACME.csv`

| customer_id | invoice_id | amount |
| --- | --- | --- |
| ACME | INV20230608 | 23.45 |
| ACME | 12345INV | 12.3 |
| ACME | INV-XYZ-98765 | 23.16 |
`invoice_summary_OSCORP.csv`

| customer_id | invoice_id | amount |
| --- | --- | --- |
| OSCORP | INVOICE2023-06-08 | 57 |
| OSCORP | INV210608-001 | 62.23 |
`invoice_summary_GLOBEX.csv`

| customer_id | invoice_id | amount |
| --- | --- | --- |
| GLOBEX | INV-ABC-789 | 35.34 |
| GLOBEX | 987INV654 | 36.23 |

### Values format (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#values-format)

For DateTime columns, partitioning by the column can create as many files as seconds because each second is a different value. In an hour, that's potentially 3600 files.

To help partition in a sensible way, you can add a format string to the column name using the following placeholders:

| Placeholder | Description | Example |
| --- | --- | --- |
| %Y | Year | 2023 |
| %m | Month as an integer number (01-12) | 06 |
| %d | Day of the month, zero-padded (01-31) | 07 |
| %H | Hour in 24 h format from 00 through 23 | 14 |
| %i | Minute (00-59) | 45 |
For instance, for a result like this:

| timestamp | invoice_id | amount |
| --- | --- | --- |
| 2023-07-07 09:07:05 | INV20230608 | 23.45 |
| 2023-07-07 09:07:01 | 12345INV | 12.3 |
| 2023-07-07 09:06:45 | INV-ABC-789 | 35.34 |
| 2023-07-07 09:05:35 | INVOICE2023-06-08 | 57 |
| 2023-07-06 23:14:05 | INV-XYZ-98765 | 23.16 |
| 2023-07-06 23:14:02 | INV210608-001 | 62.23 |
| 2023-07-06 23:10:55 | 987INV654 | 36.23 |
All 7 events have different times in the column timestamp. Using a file template like `invoices_{timestamp}` would create 7 different files.

If you were interested in writing one file per hour, you could use a file template like `invoices_{timestamp, '%Y%m%d-%H'}`. You'd then get only two files for that dataset:

`invoices_20230707-09.csv`

| timestamp | invoice_id | amount |
| --- | --- | --- |
| 2023-07-07 09:07:05 | INV20230608 | 23.45 |
| 2023-07-07 09:07:01 | 12345INV | 12.3 |
| 2023-07-07 09:06:45 | INV-ABC-789 | 35.34 |
| 2023-07-07 09:05:35 | INVOICE2023-06-08 | 57 |
`invoices_20230706-23.csv`

| timestamp | invoice_id | amount |
| --- | --- | --- |
| 2023-07-06 23:14:05 | INV-XYZ-98765 | 23.16 |
| 2023-07-06 23:14:02 | INV210608-001 | 62.23 |
| 2023-07-06 23:10:55 | 987INV654 | 36.23 |

### By number of files (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#by-number-of-files)

You also have the option to write the result into X files. Instead of using a column name, use an integer between brackets.

Example: `invoice_summary.{8}.csv`

This is convenient to reduce the file size of the result, especially when the files are meant to be consumed by other services where uploading big files is discouraged.

The results are written in random order. This means that the final result rows would be written in X files, but you can't count the specific order of the result.

There are a maximum of 16 files.

### Combining different partitions (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#combining-different-partitions)

It's possible to add more than one partitioning parameter in the file template. This is useful, for instance, when you do a daily dump of data, but want to export one file per hour.

Setting the file template as `invoices/dt={timestamp, '%Y-%m-%d'}/H{timestamp, '%H}.csv` would create the following file structure in different days and executions:

```
Invoices
├── dt=2023-07-07
│   └── H23.csv
│   └── H22.csv
│   └── H21.csv
│   └── ...
├── dt=2023-07-06
│   └── H23.csv
│   └── H22.csv
```
You can also mix column names and number of files. For instance, setting the file template as `invoices/{customer_id}/dump_{4}.csv` would create the following file structure in different days and executions:

```
Invoices
├── ACME
│   └── dump_0.csv
│   └── dump_1.csv
│   └── dump_2.csv
│   └── dump_3.csv
├── OSCORP
│   └── dump_0.csv
│   └── dump_1.csv
│   └── dump_2.csv
│   └── dump_3.csv
```
Avoid excessive partitioning. The write process creates as many files as combinations of the partitioning column values for a given result set.

## Supported file types (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#supported-file-types)

The GCS Sink supports exporting data in the following file formats:

| File type | Accepted extensions | Compression formats supported |
| --- | --- | --- |
| CSV | `.csv`, `.csv.gz` | `gzip` |
| NDJSON | `.ndjson`, `.ndjson.gz`, `.jsonl`, `.jsonl.gz`, `.json`, `.json.gz` | `gzip` |
| Parquet | `.parquet`, `.parquet.gz` | `snappy`, `gzip`, `lzo`, `brotli`, `lz4`, `zstd` |
You can optionally configure the export format using the `EXPORT_FORMAT` parameter (defaults to CSV) and compression using the `EXPORT_COMPRESSION` parameter in your Sink pipe configuration.

## GCS permissions (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#gcs-permissions)

The GCS Connector requires a Service Account with specific permissions to access objects in your GCS bucket. Following the principle of least privilege, the minimum required roles are:

- **Storage Object Creator**, `roles/storage.objectCreator` - Allows users to create objects
- **Storage Object Viewer** (`roles/storage.objectViewer`) - Grants access to view objects and their metadata, and list objects in a bucket
- **Storage Bucket Viewer** (`roles/storage.legacyBucketReader`) - Allows users to list objects in a bucket and view bucket metadata
Create a Service Account in GCP:

1. In the GCP console, create or use an existing Service Account.
2. Assign the following roles to the service account for the specific bucket or project:
  - `roles/storage.legacyBucketReader`
  - `roles/storage.objectCreator`
  - `roles/storage.objectViewer`
3. Generate a JSON key file and download it.
4. Store the key as a Tinybird secret.
Alternatively, you can use the broader **Storage Object Admin** (`roles/storage.objectAdmin`) role which includes all the necessary permissions, but it grants additional permissions beyond what's required for the sink operation.

## Observability (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#observability)

Sink pipes operations are logged in the [tinybird.jobs_log](/docs/forward/monitoring/service-datasources#tinybird-jobs-log) Service Data Source. You can filter by `job_type = 'sink'` to see only Sink pipe executions.

For more detailed Sink-specific information, you can also use [tinybird.sinks_ops_log](/docs/forward/monitoring/service-datasources#tinybird-sinks-ops-log).

Data Transfer incurred by Sink pipes is tracked in [tinybird.data_transfer](/docs/forward/monitoring/service-datasources#tinybird-data-transfer) Service Data Source.

## Limits & quotas (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#limits-quotas)

Check the [limits page](/docs/forward/pricing/limits) for limits on ingestion, queries, API Endpoints, and more.

## Billing (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#billing)

Tinybird bills Sink pipes based on Data Transfer. When a Sink pipe executes, it uses your plan's included compute resources to run the query, then writes the result to a bucket as Data Transfer. If Tinybird compresses the resulting files, it accounts for the compressed size.

### Data transfer (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#data-transfer)

Data Transfer depends on your environment. There are two scenarios:

- The destination bucket is in the **same** cloud provider and region as your Tinybird Workspace: $0.01 / GB
- The destination bucket is in a **different** cloud provider or region as your Tinybird Workspace: $0.10 / GB
You must include the **Storage Bucket Viewer** permission in your Service Account configuration. This permission allows Tinybird to determine your bucket's region and apply the correct billing rate. Without this permission, Tinybird can't detect the bucket region and charges the higher cross-region rate of $0.10 / GB regardless of your bucket's actual location.

## Next steps (https://www.tinybird.co/docs/forward/copy-export-data/gcs-sink#next-steps)

- Get familiar with the [Service Data Source](/docs/forward/monitoring/service-datasources) and see what's going on in your account
- Deep dive on Tinybird's [pipes concept](/docs/forward/core-concepts/pipes)

---

URL: https://www.tinybird.co/docs/forward/core-concepts/workspaces
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Workspaces \| Tinybird"
theme-color: "#171612"
description: "Workspaces contain your Tinybird project resources, data, tokens, branches, and deployments."
inkeep:version: "forward"
---

# Workspaces (https://www.tinybird.co/docs/forward/core-concepts/workspaces#workspaces)

A Workspace is the cloud environment for a Tinybird project. It contains your resources, data, tokens, branches, deployments, service data sources, and member permissions.

Use Workspaces for long-lived project boundaries: production, staging, separate products, or separate customers when you need hard isolation. Use [Branches](/docs/forward/core-concepts/branches) for short-lived development and preview environments inside a Workspace.

Every Workspace belongs to an [Organization](/docs/forward/core-concepts/organizations). Organization settings control billing, plan limits, member policy, and the regions or clusters where Workspaces can run.

## What belongs to a Workspace (https://www.tinybird.co/docs/forward/core-concepts/workspaces#what-belongs-to-a-workspace)

A Workspace contains the resources you define and deploy:

- [Data Sources](/docs/forward/core-concepts/data-sources)
- [Connections](/docs/forward/core-concepts/connections)
- [Pipes](/docs/forward/core-concepts/pipes)
- [API Endpoints](/docs/forward/core-concepts/api-endpoints)
- [Materialized Views](/docs/forward/core-concepts/materialized-views)
- [Copy Pipes](/docs/forward/core-concepts/copy-pipes)
- [Sinks](/docs/forward/core-concepts/sinks)
- [Tokens](/docs/forward/core-concepts/tokens)
Workspace state also includes ingested data, secrets, service data sources, branch metadata, deployments, and member roles.

You define resources in code, test them locally or in a branch, then deploy to the workspace.

You can build Forward projects with:

- [TypeScript SDK](/docs/forward/quickstarts/typescript-sdk)
- [Python SDK](/docs/forward/quickstarts/python-sdk)
- [Tinybird CLI datafiles](/docs/forward/quickstarts/cli)

## Create a workspace (https://www.tinybird.co/docs/forward/core-concepts/workspaces#create-a-workspace)

Create new workspaces from Tinybird Cloud:

1. Open [Tinybird Cloud](https://cloud.tinybird.co/).
2. Use the workspace switcher or the **Organization settings** > **Workspaces** page.
3. Select **Create workspace**.
4. Choose the organization, region, or cluster if your organization has more than one option.
5. Choose how you want to start: TypeScript SDK, Python SDK, CLI datafiles, or the UI.
New workspaces can show SDK onboarding directly in the UI. For example, the TypeScript SDK path gives you project setup instructions, credentials, and an agent-ready prompt you can copy into your coding agent.

You can also create a workspace with the Tinybird CLI:

```
tb workspace create <workspace_name>
```
If your account has access to multiple organizations, pass the organization ID:

```
tb workspace create <workspace_name> --organization-id <organization_id>
```

## Manage workspace members (https://www.tinybird.co/docs/forward/core-concepts/workspaces#manage-workspace-members)

In Tinybird Cloud, open the workspace and go to **Settings** > **Members**.

From there you can:

- Invite users by email.
- Assign a workspace role.
- Resend or copy invite links.
- Remove members from the workspace.
Organization admins can also manage workspace membership from **Organization settings** > **Members** without being a member of every workspace.

## Workspace roles (https://www.tinybird.co/docs/forward/core-concepts/workspaces#workspace-roles)

Members always have a workspace role. You can change a member's role at any time.

| Role | Manage resources | Manage users | Create a branch |
| --- | --- | --- | --- |
| `Admin` | Yes | Yes | Yes |
| `Guest` | Yes | No | Yes |
| `Viewer` | No | No | Yes |

## Connect local tooling to a workspace (https://www.tinybird.co/docs/forward/core-concepts/workspaces#connect-local-tooling-to-a-workspace)

After creating a workspace, connect your local project to it.

For CLI datafile projects, authenticate with `tb login`:

```
tb login
```
This writes workspace credentials to `.tinyb`.

For TypeScript SDK projects, authenticate with the SDK CLI:

```
tinybird login
```
This writes workspace credentials to `.env.local`.

To see the workspace your CLI session points to, run:

```
tb workspace current
```
To list the workspaces you can access:

```
tb workspace ls
```
To switch your CLI session to another workspace:

```
tb workspace use <workspace_name_or_id>
```

## Workspace ID (https://www.tinybird.co/docs/forward/core-concepts/workspaces#workspace-id)

Each workspace has a unique ID. You need it for some API calls, JWT configuration, and support requests.

Find it in Tinybird Cloud from the workspace list, or run:

```
tb workspace current
```
Example output:

```
** Current workspace:
--------------------------------------------------------------------------------------------
| name                   | id                                   | role  | plan   | current |
--------------------------------------------------------------------------------------------
| analytics_prod         | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | admin | Custom | True    |
--------------------------------------------------------------------------------------------
```

## Delete a workspace (https://www.tinybird.co/docs/forward/core-concepts/workspaces#delete-a-workspace)

Deleting a workspace deletes all resources and data in it, including Data Sources, ingested data, Pipes, published endpoints, tokens, branches, and deployments.

Deleted workspaces can't be recovered.

You can delete a workspace from Tinybird Cloud, or with the CLI if you are a workspace admin:

```
tb workspace delete <workspace_name_or_id>
```
Use `--yes` to skip the confirmation prompt in automation:

```
tb workspace delete <workspace_name_or_id> --yes
```

---

URL: https://www.tinybird.co/docs/forward/core-concepts/tokens
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Authentication \| Tinybird"
theme-color: "#171612"
description: "Learn how to authenticate your requests to Tinybird."
inkeep:version: "forward"
---

# Tokens (https://www.tinybird.co/docs/forward/core-concepts/tokens#tokens)

Tinybird uses tokens to authenticate CLI and API requests. Tokens protect access to your resources. Any operation to manage your resources using the CLI or REST API requires a valid token with the necessary permissions.

There are two types of tokens:

- [Static Tokens](/docs/forward/core-concepts/static-tokens): Use them to perform operations on your account, like importing data, creating Data Sources, or publishing APIs using the CLI or REST API. Use them to read data as well, and be mindful of their permanent nature.
- [JSON Web Tokens](/docs/forward/core-concepts/jwt): Use them to read from published API Endpoints that expose your data to an app when you want to implement filtering per user through fixed parameters (RBAC) or apply rate limiting for different end users of Tinybird Endpoints.

## Where tokens are stored and used (https://www.tinybird.co/docs/forward/core-concepts/tokens#where-tokens-are-stored-and-used)

How you store and pass tokens depends on the environment:

- **Tinybird CLI**: `tb login` stores your CLI credentials in the local `.tinyb` file.
- **SDK projects**: the TypeScript and Python SDK CLIs read `TINYBIRD_TOKEN` and `TINYBIRD_URL` from `.env.local` during local development.
- **CI/CD**: store `TINYBIRD_TOKEN` and `TINYBIRD_URL` as secret environment variables in your CI provider.
- **API calls**: pass the token in the `Authorization` header.
SDK projects commonly use a `.env.local` file during development:

.env.local
```
TINYBIRD_TOKEN=p.your_token_here
TINYBIRD_URL=https://api.tinybird.co
```
In CI/CD, set the same values as protected secrets and use them when deploying:

CI/CD deploy
```
export TINYBIRD_TOKEN="$TINYBIRD_TOKEN"
export TINYBIRD_URL="$TINYBIRD_URL"
tb deploy
```
For direct API calls, pass the token as a bearer token:

API request with a token
```
curl \
  -H "Authorization: Bearer <token>" \
  "https://api.tinybird.co/v0/pipes/<endpoint>.json"
```

---

URL: https://www.tinybird.co/docs/forward/core-concepts/tinybird-local
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Tinybird Local \| Tinybird"
theme-color: "#171612"
description: "Run Tinybird on your machine for local development, testing, and CI."
inkeep:version: "forward"
---

# Tinybird Local (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#tinybird-local)

Tinybird Local is a containerized Tinybird runtime for development, testing, and CI. It runs on your machine and gives you the same project model you use in Tinybird Cloud: Data Sources, Pipes, Endpoints, Tokens, APIs, and local services.

Use Tinybird Local when you want a fast, isolated environment with fixture data. Use [Cloud Branches](/docs/forward/development-workflow/using-cloud-branches) when you need a shared Cloud environment, production-like data, or connector validation before deployment.

## Prerequisites (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#prerequisites)

To install and run Tinybird Local, you need:

- A [Tinybird account](https://cloud.tinybird.co/).
- Linux or macOS.
- A container runtime, such as Docker or OrbStack.
OrbStack containers do not require extra configuration. Docker users should check the following settings:

- **Apple Silicon**: Enable Rosetta in **Docker Desktop > Settings > General > Use Rosetta for x86/amd64 emulation on Apple Silicon**. Without Rosetta, Docker falls back to QEMU emulation, which can be slower and less stable.
- **Memory**: Set Docker memory to at least 8 GB in **Docker Desktop > Settings > Advanced > Memory**.

## Install Tinybird (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#install-tinybird)

Install the Tinybird CLI and Tinybird Local:

- For macOS and Linux
- For Windows

```
curl https://tinybird.co | sh
```
Authenticate with Tinybird Cloud:

```
tb login
```
This opens a browser window where you can sign in to Tinybird Cloud. If you need a specific region, see [Regions and endpoints](/docs/api-reference#regions-and-endpoints) and pass the host to `tb login`.

## Start Tinybird Local (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#start-tinybird-local)

Start Tinybird Local with the CLI:

```
tb local start
```
Run it in the background with `--daemon`:

```
tb local start --daemon
```
The CLI waits until the local services are ready. Do not start building your project until you see `Tinybird Local is ready!`.

Stop Tinybird Local with `Ctrl+C` if it is running in the foreground, or with:

```
tb local stop
```
If Tinybird Local becomes unhealthy, restart it:

```
tb local restart
```

## Services (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#services)

Tinybird Local starts the services needed to run a Tinybird project locally:

- ClickHouse®: Stores data and runs queries.
- Redis: Stores user and Workspace metadata.
- Kafka consumer: Ingests data from Kafka topics into Data Sources.
- Events API server: Ingests HTTP events into Data Sources.
- Tinybird server: Exposes Tinybird APIs, such as authentication and Endpoints.
- Project server: Exposes your project to Tinybird UI.
- MCP server: Exposes your Workspace as tools for agents.
The local API base URL is:

```
http://localhost:7181/v0/
```
For example, you can write to a local Events API Data Source:

```
curl \
  -X POST 'http://localhost:7181/v0/events?name=<your_datasource>' \
  -H "Authorization: Bearer <your_token>" \
  -d $'<your_data>'
```

## Persist local data (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#persist-local-data)

By default, local state is tied to the Tinybird Local container. To persist data between development sessions, pass a custom volume path:

```
tb local start --volumes-path <your/path>
```
`tb local stop` does not remove local data. `tb local restart` recreates the local runtime. To return to an earlier local state, restart with a previous volume path:

```
tb local restart --volumes-path ./tb_previous_snapshot
```

## Run with Docker directly (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#run-with-docker-directly)

The CLI is the recommended way to start Tinybird Local. If you need to run the container directly, use Docker:

```
docker run --platform linux/amd64 -p 7181:7181 --name tinybird-local -d tinybirdco/tinybird-local:latest
```
To connect the `tb` CLI to a manually started container, create a `.tinyb` file in your project directory:

```
{
  "name": "<your_workspace_name>"
}
```

## Run with Docker Compose (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#run-with-docker-compose)

Use Docker Compose when you want explicit volume mounts for ClickHouse® and metadata storage:

docker-compose.yaml
```
version: "3.8"
services:
  tinybird-local:
    image: tinybirdco/tinybird-local:latest
    container_name: tinybird-local
    platform: linux/amd64
    ports:
      - "7181:7181"
    volumes:
      - ./data/data:/var/lib/clickhouse
      - ./data/metadata:/redis-data
    restart: unless-stopped
```
Start it:

```
docker compose up -d
```
The `./data/data` directory stores ClickHouse® data. The `./data/metadata` directory stores internal metadata.

## Local Tokens (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#local-tokens)

Tinybird Local can generate default user and Workspace Tokens for development and tests:

```
tb local generate-tokens
```
Pass generated Token values when starting Tinybird Local:

```
tb local start --user-token=<USER_TOKEN> --workspace-token=<WORKSPACE_TOKEN>
```
See [Tinybird Local Tokens](/docs/forward/core-concepts/static-tokens#tinybird-local-tokens).

## Next steps (https://www.tinybird.co/docs/forward/core-concepts/tinybird-local#next-steps)

- [Local development](/docs/forward/development-workflow/local-development)
- [Cloud Branches](/docs/forward/development-workflow/using-cloud-branches)
- [Test your project](/docs/forward/guides/test-your-project)
- [Tinybird CLI commands](/docs/forward/dev-reference/commands)

---

URL: https://www.tinybird.co/docs/forward/core-concepts/templating-language
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Templating language \| Tinybird"
theme-color: "#171612"
description: "Learn how Tinybird's templating language works in Pipes, Connections, and Data Sources."
inkeep:version: "forward"
---

# Templating language (https://www.tinybird.co/docs/forward/core-concepts/templating-language#templating-language)

Use Tinybird's templating language to make project definitions dynamic while keeping them safe and deployable as code. Use it in SQL nodes to define Query Parameters, in connector settings to reference secrets, and in datafiles to reuse values across environments.

Templates are rendered before Tinybird runs the SQL or creates the resource. The rendered result must be valid SQL or a valid datafile setting.

## Where templates are used (https://www.tinybird.co/docs/forward/core-concepts/templating-language#where-templates-are-used)

You use templates in different places depending on the resource:

- **API Endpoints, SQL API queries, Copy Pipes, and Sink Pipes**: Use typed parameters in SQL, such as `{{DateTime(start_date)}}` or `{{String(country, 'US')}}`.
- **Connection files and connector Data Sources**: Use secrets in settings, such as `{{ tb_secret("KAFKA_KEY") }}`.
- **Conditional SQL**: Use control flow with `{% if %}`, `{% else %}`, `{% for %}`, and `defined()` to include SQL only when a value exists.
In `.pipe` files, start dynamic SQL with `%` so Tinybird parses template parameters:

Parameterized SQL node
```
%
SELECT pathname, count() AS views
FROM page_views
WHERE timestamp >= {{DateTime(start_date)}}
  AND timestamp < {{DateTime(end_date)}}
  AND country = {{String(country, 'US')}}
GROUP BY pathname
ORDER BY views DESC
LIMIT {{Int32(limit, 10)}}
```
In SDK projects, declare parameters in the `params` object and use the same SQL template syntax inside SQL strings. The `%` prefix is not required:

TypeScript SDK endpoint parameters
```
import { defineEndpoint, node, p } from "@tinybirdco/sdk";

export const topPages = defineEndpoint("top_pages", {
  params: {
    start_date: p.dateTime(),
    end_date: p.dateTime(),
    country: p.string().optional("US"),
    limit: p.int32().optional(10),
  },
  nodes: [
    node({
      name: "top_pages",
      sql: `
        SELECT pathname, count() AS views
        FROM page_views
        WHERE timestamp >= {{DateTime(start_date)}}
          AND timestamp < {{DateTime(end_date)}}
          AND country = {{String(country, 'US')}}
        GROUP BY pathname
        ORDER BY views DESC
        LIMIT {{Int32(limit, 10)}}
      `,
    }),
  ],
});
```

## Parameters (https://www.tinybird.co/docs/forward/core-concepts/templating-language#parameters)

Parameters are values supplied when an API Endpoint, SQL API query, on-demand Copy Pipe, or on-demand Sink Pipe runs. Define parameters with type functions so Tinybird validates and casts incoming values before injecting them into SQL.

The basic pattern is:

```
{{<type>(<name>, <default>, description=<description>, required=<true|false>)}}
```
For example:

```
{{Int32(limit, 10, description="Maximum number of rows", required=False)}}
```
Use defaults for scheduled resources and for optional filters. If a parameter has no default, use `defined()` before referencing it:

Optional filter
```
%
SELECT *
FROM events
WHERE 1
{% if defined(country) %}
  AND country = {{String(country)}}
{% end %}
```
Common parameter functions include `String`, `Boolean`, `Date`, `DateTime`, `DateTime64`, `Float32`, `Float64`, integer types such as `Int32` and `UInt64`, and `Array`. Use `column()` when you need to parameterize a column name:

```
ORDER BY {{column(order_by, 'timestamp')}}
```
For the full list of functions and arguments, see [Template functions](/docs/forward/dev-reference/template-functions).

Do not concatenate untyped user input into SQL. Use typed template functions so Tinybird validates values and renders safe SQL literals.

## Settings (https://www.tinybird.co/docs/forward/core-concepts/templating-language#settings)

Settings are datafile instructions that configure resources. They include values like `ENGINE_SORTING_KEY`, `KAFKA_BOOTSTRAP_SERVERS`, `IMPORT_BUCKET_URI`, and `EXPORT_SCHEDULE`.

Settings can use template functions when the value must vary by environment or must stay out of source control:

Connection settings with secrets
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS {{ tb_secret("KAFKA_BOOTSTRAP_SERVERS", "localhost:9092") }}
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_KEY {{ tb_secret("KAFKA_KEY") }}
KAFKA_SECRET {{ tb_secret("KAFKA_SECRET") }}
```
The rendered setting must still be valid for the resource type. For available settings, see the relevant datafile reference:

- [Datasource files](/docs/forward/dev-reference/datafiles/datasource-files)
- [Connection files](/docs/forward/dev-reference/datafiles/connection-files)
- [Pipe files](/docs/forward/dev-reference/datafiles/pipe-files)

## Secrets (https://www.tinybird.co/docs/forward/core-concepts/templating-language#secrets)

Secrets store credentials and environment-specific values outside your source files. In datafiles, read them with `tb_secret()`:

```
S3_ARN {{ tb_secret("S3_ARN") }}
```
In TypeScript and Python SDK projects, use `secret()`:

```
import { secret } from "@tinybirdco/sdk";

const arn = secret("S3_ARN");
```

```
from tinybird_sdk import secret

arn = secret("S3_ARN")
```
Set secrets in each environment where the resource runs:

```
tb secret set KAFKA_KEY <local-value>
tb --cloud secret set KAFKA_KEY <cloud-value>
tb --branch=my_branch secret set KAFKA_KEY <branch-value>
```
You can provide a default value as the second argument to `tb_secret()`. Defaults are useful for local development, but Cloud resources should use real secret values:

```
KAFKA_BOOTSTRAP_SERVERS {{ tb_secret("KAFKA_BOOTSTRAP_SERVERS", "localhost:9092") }}
```
Secrets are replaced when resources are deployed. If you change a secret, redeploy the project for the affected resources to use the new value.

See [tb secret](/docs/forward/dev-reference/commands/tb-secret) to set, list, and remove secrets.

## Control flow (https://www.tinybird.co/docs/forward/core-concepts/templating-language#control-flow)

Tinybird templates support Python-style control flow for conditional SQL. Use `{% if %}`, `{% elif %}`, `{% else %}`, `{% for %}`, and `{% end %}` with escaped Markdoc syntax in Markdown examples.

Conditional SQL
```
%
SELECT *
FROM events
WHERE 1
{% if defined(plan) %}
  AND plan = {{String(plan)}}
{% end %}
{% if defined(min_amount) %}
  AND amount >= {{Float64(min_amount)}}
{% end %}
```
Use `WHERE 1` before optional `AND` clauses to avoid invalid SQL when no optional filters are rendered.

## Helper functions (https://www.tinybird.co/docs/forward/core-concepts/templating-language#helper-functions)

In addition to typed parameters and control flow, Tinybird includes helper functions for common dynamic SQL patterns:

- `defined(param)`: Check if a parameter exists before rendering SQL that uses it.
- `column(param, default)`: Render a validated column name, often for dynamic `ORDER BY` or `GROUP BY` clauses.
- `columns(param)`: Render multiple column names from a parameter.
- `sql_and(...)`: Build optional `AND` filters from parameterized comparisons, such as `in`, `not_in`, `gt`, `lt`, `gte`, and `lte`.
- `split_to_array(param, default, separator=',')`: Split a parameter into an array.
- `date_diff_in_seconds()`, `date_diff_in_minutes()`, `date_diff_in_hours()`, and `date_diff_in_days()`: Compare dates in template control flow, for example to choose between raw and aggregated Data Sources based on the requested time range.
- `JSON(param, default)`: Parse a JSON parameter so you can iterate over structured filters.
- `max_threads(n)`: Set the maximum number of threads for a query. This is useful for Copy Pipes in local environments with limited resources.
- `error()` and `custom_error()`: Stop rendering and return a custom error when a parameter is invalid or missing.
For example, use `sql_and()` when you need to add several optional filters without manually assembling every `AND` clause:

Optional filters with sql_and
```
%
SELECT *
FROM events
WHERE 1
{% if defined(plan) or defined(countries) %}
  AND {{sql_and(
    plan__in=Array(plan, 'String', defined=False),
    country__in=Array(countries, 'String', defined=False)
  )}}
{% end %}
```
See [Template functions](/docs/forward/dev-reference/template-functions) for the full helper reference and examples.

## Scope and limitations (https://www.tinybird.co/docs/forward/core-concepts/templating-language#scope-and-limitations)

- Query parameters are available in Endpoints, SQL API queries, and in on-demand Copy Pipes and Sink Pipes.
- Scheduled Copy Pipes and Sink Pipes can use parameters only when those parameters have defaults, because there is no request-time value.
- Materialized Views don't support query parameters.
- Parameters with the same name cascade through dependent Pipes.
- Fixed parameters from JWTs override URL parameters with the same name.

## Next steps (https://www.tinybird.co/docs/forward/core-concepts/templating-language#next-steps)

- Add request-time filters with [Query parameters](/docs/forward/query-data/query-parameters).
- Publish parameterized [Endpoints](/docs/forward/core-concepts/api-endpoints).
- Store credentials with [tb secret](/docs/forward/dev-reference/commands/tb-secret).
- Review every available [Template function](/docs/forward/dev-reference/template-functions).

---

URL: https://www.tinybird.co/docs/forward/core-concepts/storage-architecture
Last update: 2026-07-31T11:10:11.000Z
Content:
---
title: "Storage Architecture \| Tinybird"
theme-color: "#171612"
description: "How Tinybird Cloud stores and queries data using object storage, local SSD caching, and zero-copy replication."
inkeep:version: "forward"
---

# Tinybird Cloud storage architecture (https://www.tinybird.co/docs/forward/core-concepts/storage-architecture#tinybird-cloud-storage-architecture)

Tinybird Cloud uses compute-storage separation. Tinybird stores all data in object storage, using S3 on AWS and GCS on GCP. Each ClickHouse® replica has a local SSD cache for fast reads. Replicas share data in object storage via zero-copy replication.

## Storage (https://www.tinybird.co/docs/forward/core-concepts/storage-architecture#storage)

Tinybird writes all data to **object storage** (S3 or GCS, depending on your [region](/docs/api-reference#regions-and-endpoints)), and object storage is the source of truth for durability and replication. Writes always go there first; the local SSD cache is only a read-path optimization.

## Local SSD cache (https://www.tinybird.co/docs/forward/core-concepts/storage-architecture#local-ssd-cache)

Each replica has a local SSD that caches data parts fetched from object storage. When the database plans a new query, it checks the local cache first. If the part is cached, it reads at SSD speed. If not, it fetches from object storage and caches the part for future reads.

The cache lives on fast local SSD, with capacity determined by cloud provider and machine type. ClickHouse® manages eviction: when cache pressure increases, older or less frequently accessed parts are removed first.

To protect cache stability, Tinybird applies limits to how much data a single query can write into cache, so large scans don't flush useful working sets. There is also no TTL-based expiration: ClickHouse® data parts are immutable, so cached parts stay valid until they are superseded by merges and naturally evicted.

## Ingestion (https://www.tinybird.co/docs/forward/core-concepts/storage-architecture#ingestion)

Data enters the ClickHouse® cluster through two paths. **Streaming ingest** (Events API, Kafka) goes through the **Gatherer**, which batches and buffers incoming events before writing to ClickHouse®. This reduces small-part fragmentation in object storage and improves merge behavior. **Batch ingest** (S3/GCS connectors, files via the datasources API) writes directly to the cluster.

In both paths, persisted writes end up in object storage.

## Zero-copy replication (https://www.tinybird.co/docs/forward/core-concepts/storage-architecture#zero-copy-replication)

Replicas share the same data in object storage rather than each storing its own copy. This means that adding a replica doesn't require copying data. The new replica syncs metadata and starts caching data locally when reading.

Tinybird maintains zero-copy replication in its [ClickHouse® fork](https://www.tinybird.co/blog/why-we-maintain-a-clickhouse-fork-at-tinybird).

## Query path (https://www.tinybird.co/docs/forward/core-concepts/storage-architecture#query-path)

When a query reaches Tinybird, ClickHouse® determines which data parts it needs, checks the **local SSD cache** first, and fetches any misses from **object storage**, caching those parts locally for subsequent reads.

## Infrastructure options (https://www.tinybird.co/docs/forward/core-concepts/storage-architecture#infrastructure-options)

|  | Shared | Dedicated |
| --- | --- | --- |
| Cluster | Shared with other customers | Exclusive cluster |
| Replica management | Managed by Tinybird | [Self-serve](/docs/forward/guides/cluster-management) |
| Cache sizing | Automatic | Configurable via replica size |
| Plans | Free, Developer, SaaS | Enterprise |
See [Developer](/docs/forward/pricing/developer), [SaaS](/docs/forward/pricing/saas), and [Enterprise](/docs/forward/pricing/enterprise) for plan and billing details.

## Related (https://www.tinybird.co/docs/forward/core-concepts/storage-architecture#related)

- [Core concepts](/docs/forward/core-concepts)
- [Pricing plans](/docs/forward/pricing)
- [Enterprise](/docs/forward/pricing/enterprise)
- [Cluster management](/docs/forward/guides/cluster-management)
- [Why Tinybird maintains a ClickHouse® fork](https://www.tinybird.co/blog/why-we-maintain-a-clickhouse-fork-at-tinybird)
- [How Tinybird's storage architecture works: S3, local caching, and zero-copy replication](https://www.tinybird.co/blog/tinybird-architecture)

---

URL: https://www.tinybird.co/docs/forward/core-concepts/static-tokens
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Static tokens \| Tinybird"
theme-color: "#171612"
description: "Static tokens are permanent and long-term."
inkeep:version: "forward"
---

# Static tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#static-tokens)

Static Tokens are permanent and long-term. Tinybird stores them without an expiration date or time. They remain valid until you delete or refresh them. They're useful for backend-to-backend integrations, where you call Tinybird as another service.

## Default tokens (created by Tinybird) (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#default-tokens-created-by-tinybird)

All workspaces come with a set of default tokens:

| Token name | Description |
| --- | --- |
| `Workspace admin token` | The Workspace token. This token is workspace-bound and enables any operation over it. Note: only workspace admins have access to this token. |
| `Admin <your-email> token` | The CLI token. This token is managed by Tinybird for you and the CLI uses it to authenticate via 'tb login' (stores it locally in the `.tinyb` file). |
| `User token` | Required only for certain operations through the API (like creating Workspaces). The system asks you for it if required. |
Learn how to [list existing Tokens](https://www.tinybird.co/docs/forward/core-concepts/static-tokens#list-existing-tokens).

## User created tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#user-created-tokens)

Users can create additional tokens with different authorization scopes. This allow you to grant granular access to resources or to create tokens for CI/CD or for other purposes.

There are two types of static tokens:

- **[Resource-scoped tokens](https://www.tinybird.co/docs/forward/core-concepts/static-tokens#resource-scoped-tokens):** grant specific permissions on specific resources, such as reading from a given endpoint or appending to a given data source. Created in *.pipe* and *.datasource* files and managed via deployments.
- **[Workspace and Org. level tokens](https://www.tinybird.co/docs/forward/core-concepts/static-tokens#other-tokens):** tokens with workspace or organization-wide scopes: `WORKSPACE:READ_ALL`, `WORKSPACE:DEPLOY`, `ADMIN`, `TOKENS` or `ORG_DATASOURCES:READ`. Created and managed via the CLI or API.

### Resource-scoped tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#resource-scoped-tokens)

When you create a resource-scoped token, you can define which resources can be accessed by that token, and which methods can be used to access them.

In datafile projects, resource-scoped tokens are managed using the `TOKEN` directive with the structure `TOKEN <token_name> <scope>`. In SDK projects, declare reusable token names and attach them to Data Sources, Pipes, or Endpoints with each resource's `tokens` option.

- Tinybird CLI
- TypeScript SDK
- Python SDK
events.datasource
```
TOKEN events_append APPEND
TOKEN events_read READ

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `payload` String `json:$.payload`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "timestamp"
```
events_api.pipe
```
TOKEN events_read READ

NODE latest
SQL >
    SELECT timestamp, payload FROM events ORDER BY timestamp DESC LIMIT 100

TYPE ENDPOINT
```
Deployments create and update resource-scoped Tokens. Tinybird tracks which ones to create or destroy based on all the Tokens defined within the datafiles in your project. Find deployment-generated Tokens in the Workspace UI or by running `tb (--cloud) token ls`.

The following scopes are available for resource-scoped tokens:

| Token Scope (API) | Token Scope (CLI) | Description |
| --- | --- | --- |
| `DATASOURCES:READ:datasource_name` | `TOKEN <token_name> READ` in `.datasource` files | Grants the token read permissions on the specified data sources |
| `DATASOURCES:APPEND:datasource_name` | `TOKEN <token_name> APPEND` in `.datasource` files | Grants the token permission to append data to the specified data source. |
| `PIPES:READ:pipe_name` | `TOKEN <token_name> APPEND` in `.pipe` files | Grants the token read permissions for the specified pipe. |
When adding the `DATASOURCES:READ` scope to a token, it automatically grants read permissions to the [quarantine data source](/docs/forward/guides/quarantine) associated with it.

SQL filters (`:sql_filter` suffix) are not supported in Tinybird Forward. Use fixed parameters in JWTs for row-level security instead.

### Other tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#other-tokens)

These are operational tokens that are not tied to specific resources. Run the following command in the CLI:

```
tb token create static new_admin_token --scope <scope>
```
The following scopes are available for general tokens:

| Value | Description |
| --- | --- |
| `TOKENS` | Grants the token permission to create, delete or refresh tokens. |
| `ADMIN` | Grants full access to the workspace. Use sparingly. |
| `WORKSPACE:READ_ALL` | Grants read access to all workspace resources: datasources, pipes, `tinybird.*` service data sources, and `system.*` tables. Particularly useful for BI Tools. |
| `WORKSPACE:DEPLOY` | Grants permission to run deploy-related operations in the workspace. |
| `ORG_DATASOURCES:READ` | Grants the token read access to organization service datasources. |

## List existing tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#list-existing-tokens)

You can review your existing tokens using:

- **CLI**: Run `tb token ls` to list all tokens in your workspace. See [tb token](/docs/forward/dev-reference/commands/tb-token) for reference.
- **UI**: Navigate to the "Tokens" section in the sidebar of your Tinybird workspace.

## Refresh a static token (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#refresh-a-static-token)

To refresh a token, run the `tb token refresh` command:

```
tb token refresh my_static_token
```
See [tb token](/docs/forward/dev-reference/commands/tb-token) for more information.

## Delete a static token (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#delete-a-static-token)

### Resource-scoped tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#resource-scoped-tokens)

Deployments update resource-scoped Tokens. Tinybird tracks which ones to destroy based on all the Tokens defined within the datafiles in your project.

To remove a resource-scoped Token, **delete it from the datafiles and deploy the project.** Tinybird applies the changes automatically.

### Other tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#other-tokens)

To delete [other tokens](/docs/forward/core-concepts/static-tokens#other-tokens) that are not tied to specific resources, run the following command:

```
tb token rm <token_name>
```
See [tb token](/docs/forward/dev-reference/commands/tb-token) for more information.

## Tinybird Local tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#tinybird-local-tokens)

When working with [Tinybird Local](/docs/forward/core-concepts/tinybird-local), you can authenticate by running `tb login`:

```
tb login
```
The command opens a browser window where you can sign in. See [tb login](/docs/forward/dev-reference/commands/tb-login).

Credentials are stored in the `.tinyb` file. See [.tinyb file](/docs/forward/dev-reference/datafiles/tinyb-file).

Tinybird Local also supports generating default user and workspace tokens for development and testing. This is especially useful in CI/CD pipelines, tests, or automated setups where dynamically [fetching tokens](/docs/api-reference/token-api) adds unnecessary complexity.

### Generate Tinybird Local tokens (https://www.tinybird.co/docs/forward/core-concepts/static-tokens#generate-tinybird-local-tokens)

You can generate valid Tinybird Local tokens using the following [command](/docs/forward/dev-reference/commands/tb-local):

```
tb local generate-tokens
```
This command outputs valid tokens for both a user token and a workspace token. You can then export them as environment variables:

```
TB_LOCAL_WORKSPACE_TOKEN=$(tb --output=json local generate-tokens | jq -r '.workspace_token')
TB_LOCAL_USER_TOKEN=$(tb --output=json local generate-tokens | jq -r '.user_token')
tb local start
```
Alternatively, you can pass the generated token values in the arguments to `tb local start`. The `generate-tokens` command prints the tokens to your console; you must copy these values to use in the start command:

```
tb local generate-tokens
tb local start --user-token=<USER_TOKEN> --workspace-token=<WORKSPACE_TOKEN>
```
Once Tinybird Local has started, you can reference `$TB_LOCAL_USER_TOKEN` and `$TB_LOCAL_WORKSPACE_TOKEN` as environment variables in your API calls or scripts from that shell session.

---

URL: https://www.tinybird.co/docs/forward/core-concepts/sinks
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Sinks \| Tinybird"
theme-color: "#171612"
description: "Sinks are the destinations for your data. They are the places where you can store your data after it has been transformed."
inkeep:version: "forward"
---

# Sinks (https://www.tinybird.co/docs/forward/core-concepts/sinks#sinks)

Tinybird Sinks allow you to export data from your Tinybird Workspace to external systems on a scheduled or on-demand basis. Sinks are built on top of [Pipes](/docs/forward/core-concepts/pipes) and provide a fully managed way to push data to various destinations.

## Available Sinks (https://www.tinybird.co/docs/forward/core-concepts/sinks#available-sinks)

Tinybird supports the following Sink destinations:

- [Kafka Sink](/docs/forward/copy-export-data/kafka-sink)
- [S3 Sink](/docs/forward/copy-export-data/s3-sink)
- [GCS Sink](/docs/forward/copy-export-data/gcs-sink)
Sinks are available on the Developer and Enterprise plans. See [Plans](/docs/forward/pricing).

## Create a Sink (https://www.tinybird.co/docs/forward/core-concepts/sinks#create-a-sink)

Sinks are defined with a connection and a pipe that exports query results to that connection. Use `TYPE SINK` in `.pipe` files, `defineSinkPipe` in the TypeScript SDK, or `define_sink_pipe` in the Python SDK.

The following example creates an S3 Sink that exports processed events to an S3 bucket on a schedule.

- Tinybird CLI
- TypeScript SDK
- Python SDK
connections/landing_s3.connection
```
TYPE s3
S3_REGION "us-east-1"
S3_ARN {{ tb_secret("AWS_ROLE_ARN") }}
```
datasources/events.datasource
```
SCHEMA >
    `timestamp` DateTime,
    `session_id` String,
    `status` String

ENGINE "MergeTree"
ENGINE_SORTING_KEY "timestamp"
```
pipes/events_export_sink.pipe
```
NODE export
SQL >
    SELECT timestamp, session_id
    FROM events
    WHERE status = 'processed'

TYPE SINK
EXPORT_CONNECTION_NAME "landing_s3"
EXPORT_BUCKET_URI "s3://my-bucket/exports/"
EXPORT_FILE_TEMPLATE "events_{date}"
EXPORT_FORMAT "csv"
EXPORT_SCHEDULE "0 * * * *"
```
See all syntax options in the [Sink pipes reference](/docs/forward/dev-reference/datafiles/pipe-files#sink-pipes), [TypeScript SDK reference](/docs/forward/dev-reference/typescript-sdk-resources#sink-pipes), and [Python SDK reference](/docs/forward/dev-reference/python-sdk-resources#sink-pipes).

## Key features (https://www.tinybird.co/docs/forward/core-concepts/sinks#key-features)

- Fully Managed: Sinks require no additional tooling or infrastructure management.
- Scheduled or On-Demand: Run exports on a defined schedule using cron expressions or trigger them manually when needed.
- Query Parameters: Support for parameterized queries allows flexible data filtering and transformation.
- Observability: Monitor Sink operations and data transfer through Service Data Sources.

## Common use cases (https://www.tinybird.co/docs/forward/core-concepts/sinks#common-use-cases)

Sinks enable various data integration scenarios:

- Regular data exports to clients or partner systems.
- Feeding data lakes and data warehouses.
- Real-time data synchronization with external systems.
- Event-driven architectures and data pipelines.

---

URL: https://www.tinybird.co/docs/forward/core-concepts/shared-vs-dedicated-infrastructure
Last update: 2026-07-31T11:10:11.000Z
Content:
---
title: "Shared vs dedicated infrastructure \| Tinybird"
theme-color: "#171612"
description: "Understand the main differences between shared and dedicated Tinybird infrastructure."
inkeep:version: "forward"
---

# Shared vs dedicated infrastructure (https://www.tinybird.co/docs/forward/core-concepts/shared-vs-dedicated-infrastructure#shared-vs-dedicated-infrastructure)

Tinybird runs Workspaces on either shared infrastructure or dedicated infrastructure. Both give you managed ClickHouse®, ingestion, APIs, and Tinybird tooling. The difference is how much infrastructure isolation and operational control you need.

This maps to the familiar multi-tenant versus single-tenant trade-off: shared infrastructure is multi-tenant, while dedicated infrastructure gives your Organization isolated compute.

## Shared infrastructure (https://www.tinybird.co/docs/forward/core-concepts/shared-vs-dedicated-infrastructure#shared-infrastructure)

On shared infrastructure, your Workspace runs on infrastructure shared with other Tinybird customers. Tinybird manages the cluster topology, ClickHouse® version, upgrades, and operational limits.

Shared infrastructure is a good fit when you want a managed experience with less operational overhead:

- You co-live on shared clusters managed by Tinybird.
- You resize vertically by choosing a larger plan size, up to the limits of shared infrastructure.
- You don't manage replicas, traffic weights, or cluster topology.
- Tinybird manages ClickHouse® versions and upgrade timing.
- Limits and safeguards protect shared resources from noisy-neighbor effects.
For billing details, see [Developer](/docs/forward/pricing/developer), [SaaS](/docs/forward/pricing/saas), and [Free](/docs/forward/pricing/free).

## Dedicated infrastructure (https://www.tinybird.co/docs/forward/core-concepts/shared-vs-dedicated-infrastructure#dedicated-infrastructure)

On dedicated infrastructure, your organization gets an isolated ClickHouse® cluster. Your ingestion and query workloads run on replicas dedicated to you.

Dedicated infrastructure is a good fit when you need more control, isolation, or scale:

- Your workload runs on dedicated replicas, not shared compute.
- You can add, remove, resize, and rebalance replicas.
- You can control how read, write, and copy workloads are distributed across replicas.
- You have more control over ClickHouse® versioning and when upgrades happen.
- You can tune topology and capacity for workload isolation, high throughput, or stricter production requirements.
For billing details, see [Enterprise](/docs/forward/pricing/enterprise). For operational controls, see [Cluster management](/docs/forward/guides/cluster-management).

## Which one should you use? (https://www.tinybird.co/docs/forward/core-concepts/shared-vs-dedicated-infrastructure#which-one-should-you-use)

Start with shared infrastructure if you want the simplest managed path and your workload fits within shared plan limits. Move to dedicated infrastructure when you need stronger isolation, higher scale, custom replica topology, or more control over operational changes like upgrades.

---

URL: https://www.tinybird.co/docs/forward/core-concepts/self-managed
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Self-managed \| Tinybird"
theme-color: "#171612"
description: "Run Tinybird infrastructure in your own cloud environment with self-managed regions."
inkeep:version: "forward"
---

# Self-managed (https://www.tinybird.co/docs/forward/core-concepts/self-managed#self-managed)

By default, Tinybird runs on [Tinybird managed regions](/docs/api-reference#regions-and-endpoints). Self-managed regions let you run Tinybird infrastructure in your own cloud environment while Tinybird Cloud manages the control plane. Use them when you need control over data residency, compliance requirements, or infrastructure costs.

Self-managed regions are currently in beta. Use them for development environments only. Avoid production workloads. Features, requirements, and implementation details might change as Tinybird improves this capability.

## Key benefits (https://www.tinybird.co/docs/forward/core-concepts/self-managed#key-benefits)

Key benefits of self-managed regions include:

- Data sovereignty and compliance: Keep your data in specific geographic locations to meet regulatory requirements.
- Integration: Connect directly to your existing data sources and internal systems without data leaving your network.
- Infrastructure control: Tune hardware resources for your workload patterns.
- Cost efficiency: Avoid egress fees and use your existing cloud commitments and reserved instances.

## Limitations (https://www.tinybird.co/docs/forward/core-concepts/self-managed#limitations)

Self-managed regions have some limitations compared to Tinybird managed regions:

- You are responsible for infrastructure maintenance, scaling, and security.
- Some advanced features may require additional configuration.
- Performance tuning and optimization are your responsibility.
- Upgrades must be managed manually by redeploying with newer container versions.

## How self-managed regions work (https://www.tinybird.co/docs/forward/core-concepts/self-managed#how-self-managed-regions-work)

When you set up a self-managed region:

1. You register a region with Tinybird Cloud.
2. You deploy the Tinybird Local container in your cloud environment.
3. You interact with your self-managed region directly using the Tinybird CLI with the `--cloud` flag.
This approach keeps your data and infrastructure in your cloud environment while Tinybird Cloud manages the region metadata.

## Add a self-managed region (https://www.tinybird.co/docs/forward/core-concepts/self-managed#add-a-self-managed-region)

There are two main ways to create a self-managed region:

- **Manually**: See [Create a self-managed region manually](/docs/forward/guides/add-self-managed-region-manually).
- **Using the CLI**: See [Use the CLI to add a self-managed region](/docs/forward/guides/add-self-managed-region-cli).

## Supported clouds (https://www.tinybird.co/docs/forward/core-concepts/self-managed#supported-clouds)

The following cloud service providers are supported:

- AWS
- GCP (Coming soon)
- Azure (Coming soon)

You can also create a self-managed region by deploying the container on the following Platform as a Service (PaaS) providers:

- Fly.io
- Zeet
- Google Cloud Run
- CloudFoundry
- Azure Container Instances
- DigitalOcean App Platform
When deploying Tinybird Local on a PaaS, make sure to expose the following environment variables to the container:

- `TB_INFRA_TOKEN`
- `TB_INFRA_WORKSPACE`
- `TB_INFRA_ORGANIZATION`
- `TB_INFRA_USER`
See [tb infra](/docs/forward/dev-reference/commands/tb-infra) for more information.

---

URL: https://www.tinybird.co/docs/forward/core-concepts/pipes
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Pipes \| Tinybird"
theme-color: "#171612"
description: "Pipes help you to bring your SQL queries together to achieve a purpose, like publishing an endpoint."
inkeep:version: "forward"
---

# Pipes (https://www.tinybird.co/docs/forward/core-concepts/pipes#pipes)

A pipe is a collection of one or more SQL queries. Each query is called a node.

Use Pipes to build features over your data. For example, you can write SQL that joins, aggregates, or otherwise transforms your data. Pipes may be internal, which means your project references them, or they can have outputs such as API Endpoints, Sinks, Materialized Views, and Copy Jobs.

## Nodes (https://www.tinybird.co/docs/forward/core-concepts/pipes#nodes)

A node is a container for a single SQL `SELECT` statement. Nodes live within pipes, and you can have many sequential nodes inside the same pipe. They allow you to break your query logic down into multiple smaller queries. You can then chain nodes together to build the logic incrementally.

A query in a node can read data from a data source, other nodes inside the same pipe, or from endpoint nodes in other pipes. Each node can be developed and tested individually. This makes it much easier to build complex query logic in Tinybird as you avoid creating large monolithic queries with many subqueries.

Pipes are defined with `.pipe` files in regular projects, in `.ts` files with the TypeScript SDK, or in `.py` files with the Python SDK.

SQL nodes can use Tinybird's [templating language](/docs/forward/core-concepts/templating-language) for typed parameters, conditional SQL, and environment-specific values.

- Tinybird CLI
- TypeScript SDK
- Python SDK
pipes/events_by_path.pipe
```
DESCRIPTION >
    Count events by path for a date range

NODE recent_events
SQL >
    %
    SELECT timestamp, pathname
    FROM events
    WHERE timestamp >= {{DateTime(start_date)}}
      AND timestamp <= {{DateTime(end_date)}}

NODE aggregation
SQL >
    SELECT pathname, count() AS events
    FROM recent_events
    GROUP BY pathname
    ORDER BY events DESC
```
See all syntax options in the [Pipe files reference](/docs/forward/dev-reference/datafiles/pipe-files), [TypeScript SDK reference](/docs/forward/dev-reference/typescript-sdk-resources#pipes), and [Python SDK reference](/docs/forward/dev-reference/python-sdk-resources#pipes).

## Create generic pipes (https://www.tinybird.co/docs/forward/core-concepts/pipes#create-generic-pipes)

Generic Pipes are internal Pipes that you use from other Pipes or Endpoints. The preceding examples create generic Pipes because they don't declare an output type such as Endpoint, Materialized View, Copy Job, or Sink.

Use generic pipes to split complex SQL into reusable steps or to replace include-file patterns from Classic projects. To publish the result as an API endpoint, see [Endpoints](/docs/forward/core-concepts/api-endpoints).

## Next steps (https://www.tinybird.co/docs/forward/core-concepts/pipes#next-steps)

- Learn about [Endpoints](/docs/forward/core-concepts/api-endpoints).
- Learn about [Copy pipes](/docs/forward/core-concepts/copy-pipes).
- Learn about [Materialized views](/docs/forward/core-concepts/materialized-views).

---

URL: https://www.tinybird.co/docs/forward/core-concepts/organizations
Last update: 2026-08-04T10:22:54.000Z
Content:
---
title: "Organizations \| Tinybird"
theme-color: "#171612"
description: "Tinybird Organizations group workspaces, members, billing, limits, and infrastructure settings."
inkeep:version: "forward"
---

# Organizations (https://www.tinybird.co/docs/forward/core-concepts/organizations#organizations)

Organizations group Workspaces, members, billing, limits, and infrastructure settings in one region. Every [Workspace](/docs/forward/core-concepts/workspaces) belongs to an Organization.

Use Organizations to manage access across Workspaces, monitor usage, review billing, and control dedicated infrastructure settings when your plan includes them. See [Plans](/docs/forward/pricing).

The **Organization settings** section consists of the following areas:

- Overview
- Cluster (dedicated infrastructure only)
- Observability
- Logs

- Workspaces
- Members
- Billing (organization admins only)
- Managed regions

## Users and organizations (https://www.tinybird.co/docs/forward/core-concepts/organizations#users-and-organizations)

Users can be members of one or more Organizations. A user can only be the admin of one Organization.

## Access the organization settings (https://www.tinybird.co/docs/forward/core-concepts/organizations#access-the-organization-settings)

To access **Organization settings**, select the organization name in the page header. You can also select **Organization settings** from any **Settings** page.

## Overview (https://www.tinybird.co/docs/forward/core-concepts/organizations#overview)

The **Overview** page is the Organization landing page. It gives you a snapshot of your Organization's health and usage at a glance.

The page is organized in three sections:

- **Recently visited workspaces**: shows the Workspaces you visited most recently. Up to 4 Workspaces are shown. Select **View all workspaces** to see all Workspaces in the Organization.
- **Usage metrics**: 7-day usage cards with maximum aggregation for CPU, API vCPU, Memory, and other metrics depending on your plan.
- **Errors**: a 7-day errors chart with the 5 most recent error log entries. Select the chart to go to the [Logs](https://www.tinybird.co/docs/forward/core-concepts/organizations#logs) page filtered by errors.
The **Usage metrics** section varies by plan:

- **Free, Developer, and SaaS**: 7-day usage cards with maximum values.
- **Enterprise**: per-cluster Load and Memory indicators with one row per replica. You can toggle between **Now**, **6 h avg**, and **24 h avg** views.

## Cluster (https://www.tinybird.co/docs/forward/core-concepts/organizations#cluster)

The **Cluster** section is only available for Organizations with dedicated infrastructure.

The **Cluster** page is the central place to monitor and manage your dedicated cluster. It includes:

- **Load and memory charts** showing cluster resource usage over time.
- **Replica information** with the status and metrics for each replica in your cluster.
- **Manage Cluster** flow to adjust your cluster configuration. Only Organization admins can perform this action.
For replica operations, see [Cluster management](/docs/forward/guides/cluster-management).

## Observability (https://www.tinybird.co/docs/forward/core-concepts/organizations#observability)

The **Observability** page shows details about your resource usage.

### Charts view (https://www.tinybird.co/docs/forward/core-concepts/organizations#charts-view)

The following charts are available depending on your plan:

| Plan | Charts |
| --- | --- |
| Free | Max/Average vCPU time, Max/Average API vCPU time, Max/Average memory, Storage, Accumulated daily requests, Total errors |
| Developer | Total errors, Max/Average vCPU time, Max/Average API vCPU time, Max/Average memory, Requests, Storage, Ingested rows, Quarantine rows, Data Transfer |
| SaaS | Max/Average vCPU time, Max/Average QPS, Max/Average memory, Requests, Storage, Data Transfer, Total errors |
| Enterprise (Dedicated infrastructure) | Max load, Max memory, Requests, Storage, Data Transfer, Total errors |
You can select between average and maximum values for all usage charts, as well as the period, from **Last hour** to **Last 7 days**.

### Table view (https://www.tinybird.co/docs/forward/core-concepts/organizations#table-view)

Use the view toggle to switch between charts and table view.

The table view shows resource usage by Workspace or resource. You can filter by resource name by typing in the text box.

## Logs (https://www.tinybird.co/docs/forward/core-concepts/organizations#logs)

The **Logs** page is a cross-workspace logs explorer. It surfaces logs from all Workspaces in your Organization in one place, with the same filters and layout as the per-workspace logs page.

The page includes:

- A **chart** at the top showing log volume over time. Drag to zoom in on a time range.
- A **filter sidebar** with time range presets and a custom range picker, plus filters for errors, status codes, log types, sources, Workspace, and a text search.
- A **paginated table** showing log entries. Select a row to open a side panel with full log details.
- A **real-time toggle** that refreshes the view every 5 seconds.
With the **Workspace filter**, you can quickly narrow the results to Workspaces that have logs in the current time range, ranked by log count.

Filter state is reflected in the URL, so you can share links to a specific view with colleagues.

## Workspaces (https://www.tinybird.co/docs/forward/core-concepts/organizations#workspaces)

The **Workspaces** page shows details of all your Workspaces, including their name, members, and resources used by each Workspace.

New Workspaces created by a user with an email domain linked to, or matching an Organization are automatically added to that Organization.

## Members (https://www.tinybird.co/docs/forward/core-concepts/organizations#members)

The **Members** page shows details of your organization members, the workspaces they belong to, and their roles. From this page you can manage existing members, their permissions, or invite new members.

The table shows the following information:

- email
- Workspaces and roles
To view the detail of a member’s workspaces and roles, select the arrow next to the Workspace count. A menu shows all the workspaces that user is part of, plus their role in each Workspace.

To change a user’s role or remove them from a Workspace, hover over the Workspace name and follow the arrow. Select a new role from **Admin** or **Viewer**, or remove them from the Workspace. You don't need to be a user in that Workspace to make changes to its users.

### Add or remove members (https://www.tinybird.co/docs/forward/core-concepts/organizations#add-or-remove-members)

To add a user, select **Add member**.

To remove a user from the Organization, select **Remove member** in the menu.

All Organization members can access the **Members** page. Members can only see users from Workspaces they belong to and cannot manage other members. Only Organization administrators can invite or remove members and change roles.

### Add an organization admin (https://www.tinybird.co/docs/forward/core-concepts/organizations#add-an-organization-admin)

To add another user as an organization admin, follow these steps:

1. Navigate to the **Organization settings** page.
2. Go to the **Members** section.
3. Locate the user you want to make an admin.
4. Select the **More actions (⋯)** icon.
5. Select **Organization admin** in the menu.
This grants organization admin access to the selected users.

## Billing (https://www.tinybird.co/docs/forward/core-concepts/organizations#billing)

Only organization administrators can access the **Billing** section.

The **Billing** page contains a summary of the credits balance for your plan, with links to billing details and a summary of your plan. See [Billing](/docs/forward/pricing).

From the **Billing** page you can upgrade or resize your plan, or cancel it if you wish to downgrade to Free.

If you're on a Developer plan, the usage diagram shows the total expenditure by monthly invoice.

## Managed regions (https://www.tinybird.co/docs/forward/core-concepts/organizations#managed-regions)

The **Managed regions** page shows the self-managed regions where your Organization has Workspaces. See [Self-managed](/docs/forward/core-concepts/self-managed).

### Add a region (https://www.tinybird.co/docs/forward/core-concepts/organizations#add-a-region)

To add a region, select **Add region**.

### Remove a region (https://www.tinybird.co/docs/forward/core-concepts/organizations#remove-a-region)

To remove a region, select the **More actions (⋯)** and then select **Delete**.

---

URL: https://www.tinybird.co/docs/forward/core-concepts/materialized-views
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Materialized views \| Tinybird"
theme-color: "#171612"
description: "Materialized views process your data at ingest time to increase the performance of your queries and endpoints."
inkeep:version: "forward"
---

# Materialized views (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#materialized-views)

A materialized view is the continuous, streaming result of a pipe saved as a new data source. As new data is ingested into the origin data source, the transformed results from the pipe are continually inserted in the new materialized view, which you can query as any other data source.

Processing data at ingest time reduces latency and cost-per-query, and can significantly improve the performance of your queries. If your Endpoint, MCP, or regular query usually needs calculations such as counts, sums, averages, or arrays, or transformations like string manipulations or joins, materializations are the best optimization. Materialize the operation and use the resulting Materialized View as a Data Source for the published query.

Typical use cases of Materialized views include:

- Improving the speed of a query that's taking too much time to run.
- Aggregating, sorting, or filtering data at ingest time.
- Simplifying query development by automating common filters and aggregations.
- Changing an existing schema for a different use case.

## Create materialized views (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#create-materialized-views)

Materialized views are defined with a target Data Source and a pipe that writes into it. Use the `TYPE MATERIALIZED` directive in `.pipe` files, `defineMaterializedView` in the TypeScript SDK, or `define_materialized_view` in the Python SDK.

The following example starts with a `page_views` Data Source and creates a `daily_page_views` Data Source populated by a materialized view.

When you define a Materialized View target Data Source with an SDK, turn off JSON paths on the target. The target receives rows from the Materialized View, not raw JSON events from the Events API.

- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/page_views.datasource
```
SCHEMA >
    `timestamp` DateTime,
    `pathname` String,
    `session_id` String

ENGINE "MergeTree"
ENGINE_SORTING_KEY "pathname, timestamp"
```
datasources/daily_page_views.datasource
```
SCHEMA >
    `date` Date,
    `pathname` String,
    `views` SimpleAggregateFunction(sum, UInt64)

ENGINE "AggregatingMergeTree"
ENGINE_SORTING_KEY "date, pathname"
```
pipes/daily_page_views_mv.pipe
```
NODE aggregate
SQL >
    SELECT
        toDate(timestamp) AS date,
        pathname,
        count() AS views
    FROM page_views
    GROUP BY date, pathname

TYPE materialized
DATASOURCE daily_page_views
```
See all syntax options in the [Materialized pipes reference](/docs/forward/dev-reference/datafiles/pipe-files#materialized-pipes), [TypeScript SDK reference](/docs/forward/dev-reference/typescript-sdk-resources#materialized-views), and [Python SDK reference](/docs/forward/dev-reference/python-sdk-resources#materialized-views).

Once you have the origin and destination data sources and the materialized pipe defined, deploy the project to apply the changes. Materialized views are populated automatically during deployment process.

The order of columns in your transformation Pipe's SELECT statement doesn't need to match the order defined in the destination Data Source schema. Tinybird maps columns by name, not by position. As long as the column names and data types are compatible, the materialization works correctly.

## Populating materialized views (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#populating-materialized-views)

Materialized Views destinations receive data on each insert on the origin data source. A populate is the process to move the data that was already in the origin data source to the destination data source.

Materialized views are populated automatically when you deploy your project. The deployment process handles the initial population and subsequent data migrations.

When you first deploy a project with a Materialized View, Tinybird creates the view and populates it with all existing data from the origin Data Source. Tinybird's deployment system manages this process automatically.

## Changing a materialized view (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#changing-a-materialized-view)

If you need to make changes to a Materialized View after deployment (such as changing its schema or engine settings), follow the Data Source evolution process. This might require using a [forward query](/docs/forward/guides/evolve-data-source#forward-query) to ensure historical data is properly migrated.

Learn more about:

- [Deployments in Tinybird](/docs/forward/core-concepts/deployments) to understand how deployments manage state changes
- [Evolving data sources](/docs/forward/guides/evolve-data-source#materialized-data-sources) for details on making changes to materialized views after deployment

### Altering materialized views (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#altering-materialized-views)

When you need to update the query logic of an existing materialized view without recreating the entire table, you can use the `DEPLOYMENT_METHOD alter` directive. This approach uses `ALTER TABLE ... MODIFY QUERY` instead of dropping and recreating the materialized view, which minimizes data movement and improves deployment performance.

#### When to use `ALTER` (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#when-to-use)

Use the `DEPLOYMENT_METHOD alter` directive when you want to:

- Update the query logic of an existing materialized view
- Preserve existing data while modifying the transformation
- Minimize deployment time and resource usage
- Avoid data movement in production environments with large datasets

#### Syntax (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#syntax)

Add the `DEPLOYMENT_METHOD alter` directive to your materialized pipe file:

my_transformation.pipe
```
NODE transformation_node
SQL >
    SELECT
        id,
        local_date,
        name,
        sum(count) as total_count
    FROM
        my_origin
    WHERE
        local_date >= today() - interval 90 day  -- Updated filter from 30 to 90 days
    GROUP BY
        id,
        local_date,
        name

TYPE materialized
DATASOURCE my_destination
DEPLOYMENT_METHOD alter
```

#### Limitations of `ALTER` usage (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#limitations-of-usage)

- **Existing materialized views only**: Can only be used on materialized views that already exist in the target environment
- **Query changes only**: The directive only works when there are actual changes to the SQL query
- **No target changes**: Cannot change the target data source (`DATASOURCE` directive)
- **Deployment limitations**: Staging ingestion and reads are not available during ALTER deployments

## Limitations (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#limitations)

As explained, materialized views work as insert triggers, which means a delete or truncate operation on your original data source doesn't affect the related materialized views. But be sure to understand [materialized views iterations](/docs/forward/guides/evolve-data-source#materialized-data-sources) before deleting data or setting TTLs to the origin data source.

As transformation and ingestion in the materialization is done on each block of inserted data in the original data source, some operations such as `GROUP BY`, `ORDER BY`, `DISTINCT` and `LIMIT` might need a specific `engine`, such as `AggregatingMergeTree` or `SummingMergeTree`, which can handle data aggregations.

The data source resulting from a materialization generated using `JOIN` is automatically updated **only** if and when a new operation is performed over the data source in the `FROM`.

You can't create Materialized Views that depend on the `UNION` of several Data Sources.

## Error messages (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#error-messages)

### The performance of this query is not compatible with realtime ingestion (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#the-performance-of-this-query-is-not-compatible-with-realtime-ingestion)

When you deploy a Materialized View, Tinybird validates that the transformation query runs fast enough to keep up with real-time ingestion. The validation works as follows:

1. Tinybird replaces the source Data Source in your query with a subquery that reads a limited batch of rows (2,000 by default).
2. It executes the batch query with tight resource constraints: a 1-second time limit and 2 threads.
3. If the query times out, Tinybird retries once. If it times out again, deployment fails with this error.
This check is a proactive guardrail. Rather than letting a slow Materialized View degrade your ingestion pipeline at runtime, Tinybird rejects it at deploy time.

**Common causes and fixes:**

- **Complex JOINs**: Joins against large Data Sources are expensive to run per batch. Pre-aggregate the joined Data Source or restructure your query.
- **Heavy aggregations or subqueries**: Simplify the transformation logic. If the query is complex, break it into multiple chained Materialized Views.
- **Expensive functions**: Some ClickHouse® functions are slow at scale, such as complex regular expressions or array operations on large arrays. Use simpler alternatives where possible.
- **Too many columns**: Selecting more columns than you need increases the work per batch. Select only the columns you need instead of `SELECT *`.

## Next steps (https://www.tinybird.co/docs/forward/core-concepts/materialized-views#next-steps)

- Learn about [Copy pipes](/docs/forward/core-concepts/copy-pipes).
- Publish your data with [API Endpoints](/docs/forward/core-concepts/api-endpoints).

---

URL: https://www.tinybird.co/docs/forward/core-concepts/jwt
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "JSON Web tokens (JWTs) \| Tinybird"
theme-color: "#171612"
description: "JWTs are signed tokens that allow you to securely authorize and share data between your application and Tinybird."
inkeep:version: "forward"
---

# JSON Web Tokens (JWTs) (https://www.tinybird.co/docs/forward/core-concepts/jwt#json-web-tokens-jwts)

JWTs are signed tokens that allow you to securely and independently authorize and consume data from Tinybird.

Unlike static tokens, **JWTs are not stored in Tinybird**. You create them inside your app and sign them with a shared secret between your app and Tinybird. Tinybird validates the signature of the JWT, using the shared secret, to ensure it's authentic.

JWTs are not defined as Tinybird datafiles or SDK resources. If you use the TypeScript or Python SDKs for your Tinybird project, generate JWTs in your app backend. You can self-sign them with a JWT library and a signing key such as `TINYBIRD_SIGNING_KEY`, or use the SDK token helpers to ask Tinybird to create the JWT through the Token API.

A great use case for JWTs is when you want to allow your app to call Tinybird API endpoints directly from the browser without proxying through your backend.

The typical pattern looks like this:

1. A user starts a session in your app.
2. The frontend requests a JWT from your backend.
3. Your backend generates a new JWT, signed with the Tinybird shared secret, and returns to the frontend.
4. The frontend uses the JWT to call the Tinybird API endpoints directly.

## JWT payload (https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-payload)

The payload of a JWT is a JSON object that contains the following fields:

| Key | Example Value | Required | Description |
| --- | --- | --- | --- |
| workspace_id | workspaces_id | Yes | The UUID of your Tinybird workspace, found in the workspace list. |
| name | frontend_jwt | Yes | Used to identify the token in the `tinybird.pipe_stats_rt` table, useful for analytics. Doesn't need to be unique. |
| exp | 123123123123 | Yes | The Unix timestamp (UTC) showing the expiry date & time. After a token has expired, Tinybird returns a 403 HTTP status code. |
| scopes | [{"type": "PIPES:READ", "resource": "requests_per_day", "fixed_params": {"org_id": "testing"}}] | Yes | Used to pass data to Tinybird, including the Tinybird scope, resources and fixed parameters. |
| scopes.type | PIPES:READ or DATASOURCES:READ | Yes | The type of scope, for example `PIPES:READ`. See [JWT scopes](https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-scopes) for supported scopes. |
| scopes.resource | t_b9427fe2bcd543d1a8923d18c094e8c1 or top_airlines | Yes | The ID or name of the pipe that the scope applies to, like which API endpoint the token can access. |
| scopes.fixed_params | {"org_id": "testing"} | No | Valid for scope `PIPES:READ`. Pass arbitrary fixed values to the API endpoint. These values can be accessed by pipe templates to supply dynamic values at query time. |
| scopes.filter | "org_id = 'testing'" | No | Valid for scope `DATASOURCES:READ`. Passes a WHERE filter that Tinybird appends to the specified scope resource. |
| limits | {"rps": 10} | No | You can limit the number of requests per second the JWT can perform. See [JWT rate limit](https://www.tinybird.co/docs/forward/core-concepts/jwt#rate-limits-for-jwts). |
Check out the [JWT example](https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-example) to see what a complete payload looks like.

## JWT algorithm (https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-algorithm)

Tinybird always uses HS256 as the algorithm for JWTs and doesn't read the `alg` field in the JWT header. You can skip the `alg` field in the header.

## JWT scopes (https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-scopes)

| Value | Description |
| --- | --- |
| `PIPES:READ:pipe_name` | Gives your token read permissions for the specified pipe. Use `fixed_params` to filter by the pipe parameters. |
| `DATASOURCES:READ:datasource_name` | Gives your token read permissions for the specified datasource. Use `filter` to filter by the data source columns. |

## JWT expiration (https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-expiration)

JWTs can have an expiration time that gives each token a finite lifespan.

Setting the `exp` field in the JWT payload is mandatory, and not setting it results in a 403 HTTP status code from Tinybird when requesting the API endpoint.

Tinybird validates that a JWT hasn't expired before allowing access to the API endpoint.

If a token has expired, Tinybird returns a 403 HTTP status code.

## JWT fixed parameters (https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-fixed-parameters)

Fixed parameters allow you to pass arbitrary values to the API endpoint. These values can be accessed by pipe templates to supply dynamic values at query time. See [Templating language](/docs/forward/core-concepts/templating-language) to learn how parameters are validated and rendered in SQL.

Consider the following API Endpoint that accepts a parameter called `org` that filters by the `org_id` column:

example.pipe
```
SELECT fieldA, fieldB FROM my_ds WHERE org_id = '{{ String(org) }}'
TYPE ENDPOINT
```
The following JWT payload passes a parameter called `org` with the value `test_org` to the API endpoint:

Example fixed parameters
```
{
  "type": "PIPES:READ",
  "resource": "requests_per_day",
  "fixed_params": {
      "org": "test_org"
  }
}
```
This is particularly useful when you want to pass dynamic values to an API endpoint that are set by your backend and must be safe from user tampering. A good example is multi-tenant applications that require row-level security, where you need to filter data based on a user or tenant ID.

The value for the `org` parameter is always the one specified in the `fixed_params`. Even if you specify a new value in the URL when requesting the endpoint, Tinybird always uses the one specified in the JWT.

## JWT filters (https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-filters)

Filters allow you to pass WHERE clauses to Data Source queries. The filter must be a valid `WHERE` clause. Tinybird automatically appends it to the Data Source at query time.

Consider the following data source with an `org_id` column:

example.datasource
```
SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `org_id` String `json:$.org_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload`

ENGINE MergeTree
ENGINE_SORTING_KEY org_id, timestamp
```
The following JWT payload passes a filter for the data source:

Example filter in a scope
```
{
  "type": "DATASOURCES:READ",
  "resource": "events",
  "filter": "org_id = 'testing'"
}
```
This is particularly useful when you want to pass dynamic filters to queries. A good example is multi-tenant applications that require row-level security, where you need to filter data based on a user or tenant ID.

## JWT example (https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-example)

Consider the following payload with all [required and optional fields](https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-payload):

Example payload
```
{
    "workspace_id": "workspaces_id",
    "name": "frontend_jwt",
    "exp": 123123123123,
    "scopes": [
        {
            "type": "PIPES:READ",
            "resource": "requests_per_day",
            "fixed_params": {
                "org_id": "testing"
            }
        },
        {
            "type": "DATASOURCES:READ",
            "resource": "events",
            "filter": "org_id = 'testing'"
        }
    ],
    "limits": {
      "rps": 10
    }
}
```
Use the workspace admin token as your signing key (`TINYBIRD_SIGNING_KEY`):

Example workspace admin token
```
p.eyJ1IjogIjA1ZDhiYmI0LTdlYjctND...
```
With the payload and admin token, the signed JWT payload would look like this:

Example JWT
```
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ3b3Jrc3BhY2V...
```

## JWT limitations (https://www.tinybird.co/docs/forward/core-concepts/jwt#jwt-limitations)

The following limitations apply to JWTs:

- You can't refresh JWTs individually from inside Tinybird because Tinybird doesn't store them. Refresh them from your app, or globally invalidate all JWTs by refreshing your Admin Token.
- If you refresh your admin token, all the tokens are invalidated.
- If your token expires or is invalidated, you get a 403 HTTP status code from Tinybird when requesting the API endpoint.

## Create a JWT in production (https://www.tinybird.co/docs/forward/core-concepts/jwt#create-a-jwt-in-production)

There is wide support for creating JWTs in many programming languages and frameworks. Any library that supports JWTs should work with Tinybird.

- JavaScript (Next.js)
- Python
Create a JWT in Python using pyjwt
```
import jwt
import datetime
import os

TINYBIRD_SIGNING_KEY = os.getenv('TINYBIRD_SIGNING_KEY')

def generate_jwt():
  expiration_time = datetime.datetime.utcnow() + datetime.timedelta(hours=3)
  payload = {
      "workspace_id": "workspaces_id",
      "name": "frontend_jwt",
      "exp": expiration_time,
      "scopes": [
          {
              "type": "PIPES:READ",
              "resource": "requests_per_day",
              "fixed_params": {
                  "org_id": "testing"
              }
          },
          {
              "type": "DATASOURCES:READ",
              "resource": "events",
              "filter": "org_id = 'testing'"
          },
      ]
  }

  return jwt.encode(payload, TINYBIRD_SIGNING_KEY, algorithm='HS256')
```

## Create a JWT token with Tinybird (https://www.tinybird.co/docs/forward/core-concepts/jwt#create-a-jwt-token-with-tinybird)

If you don't want to self-sign JWTs, Tinybird can create them through the Token API, the Tinybird CLI, or the TypeScript and Python SDKs. These methods require a workspace token with the `ADMIN` or `TOKENS` scope.

- API
- Tinybird CLI
- TypeScript SDK
- Python SDK
Create a JWT with the Tinybird CLI
```
tb token create jwt my_jwt --ttl 1h --scope PIPES:READ --resource my_pipe --fixed-params "column_name=value" --scope DATASOURCES:READ --resource my_datasource --filter "column_name='value'"
```

## Error handling (https://www.tinybird.co/docs/forward/core-concepts/jwt#error-handling)

There are many reasons why a request might return a `403` status code. When a `403` is received, check the following:

1. Confirm the JWT is valid and hasn't expired. The expiration time is in the `exp` field in the JWT's payload.
2. The generated JWTs can only read Tinybird API endpoints or query data sources. Confirm you're not trying to use the JWT to access other APIs.
3. Confirm the JWT has a scope to read the endpoint or data source you are trying to read.
4. If you generated the JWT outside of Tinybird, without using the API or the CLI, make sure you are using the **workspace** `admin token`, not your personal one.

## Rate limits for JWTs (https://www.tinybird.co/docs/forward/core-concepts/jwt#rate-limits-for-jwts)

Check the [limits page](/docs/forward/pricing/limits) for limits on ingestion, queries, API Endpoints, and more.

When you specify a `limits.rps` field in the payload of the JWT, Tinybird uses the name specified in the payload of the JWT to track the number of requests being done. If the number of requests goes beyond the limit, Tinybird starts rejecting new requests and returns an "HTTP 429 Too Many Requests" error.

The following example shows the tracking of all requests done by `frontend_jwt`. Once you reach 10 requests per second, Tinybird would start rejecting requests:

Example payload with global rate limit
```
{
    "workspace_id": "workspaces_id",
    "name": "frontend_jwt",
    "exp": 123123123123,
    "scopes": [
        {
            "type": "PIPES:READ",
            "resource": "requests_per_day",
            "fixed_params": {
                "org_id": "testing"
            }
        },
        {
            "type": "DATASOURCES:READ",
            "resource": "events",
            "filter": "org_id = 'testing'"
        }
    ],
    "limits": {
      "rps": 10
    }
}
```
If `rps <= 0`, Tinybird ignores the limit and assumes there is no limit.

As the `name` field doesn't have to be unique, all the tokens generated using the `name=frontend_jwt` would be under the same umbrella. This can be useful if you want to have a global limit in one of your apps or components.

If you want to limit for each specific user, you can generate a JWT using the following payload. In this case, you would specify a unique name so the limits only apply to each user:

Example of a payload with isolated rate limit
```
{
    "workspace_id": "workspaces_id",
    "name": "frontend_jwt_user_<unique identifier>",
    "exp": 123123123123,
    "scopes": [
        {
            "type": "PIPES:READ",
            "resource": "requests_per_day",
            "fixed_params": {
                "org_id": "testing"
            }
        },
        {
            "type": "DATASOURCES:READ",
            "resource": "events",
            "filter": "org_id = 'testing'"
        }
    ],
    "limits": {
      "rps": 10
    }
}
```

## Next steps (https://www.tinybird.co/docs/forward/core-concepts/jwt#next-steps)

- Learn about [workspaces](/docs/forward/core-concepts/workspaces).
- Learn about [endpoints](/docs/forward/core-concepts/api-endpoints).
- Using [Clerk.com](https://clerk.com/) in your app? [Let Clerk.com create your Tinybird JWTs automatically](https://clerk.com/blog/tinybird-and-clerk).
- Using [Auth0](https://auth0.com/) in your app? [Let Auth0 create your Tinybird JWTs automatically](https://www.tinybird.co/templates/auth0-jwt).

---

URL: https://www.tinybird.co/docs/forward/core-concepts/deployments
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Deployments \| Tinybird"
theme-color: "#171612"
description: "Deploy your data project to Tinybird."
inkeep:version: "forward"
---

# Deployments in Tinybird (https://www.tinybird.co/docs/forward/core-concepts/deployments#deployments-in-tinybird)

Changing state in data infrastructure can be complex. Each state transition must ensure data integrity and consistency.

Tinybird deployments simplify this process by providing robust mechanisms for managing state changes, allowing you to validate and push updates seamlessly while minimizing the risk of data conflicts or loss.

## What is a deployment? (https://www.tinybird.co/docs/forward/core-concepts/deployments#what-is-a-deployment)

Deployments are versions of your project resources and data running on local or cloud infrastructure.

## Types of deployments (https://www.tinybird.co/docs/forward/core-concepts/deployments#types-of-deployments)

There are two types of deployments:

- Staging deployments: Deployments you can use to validate your changes. You access them using the `--staging` flag.
- Live deployments: Deployments that make your changes available to your users.

## Deployment status (https://www.tinybird.co/docs/forward/core-concepts/deployments#deployment-status)

Deployments have the following statuses:

- `In progress`: The deployment is in progress. Use `--wait` to wait for it to finish.
- `Live`: The deployment is active and has been promoted from staging.
- `Staging`: The deployment is active in staging. Use `--staging` to access it.
- `Failed`: The deployment failed. Try `tb deploy --check` to debug the issue.
- `Deleted`: The deployment was deleted as a result of creating new deployments.

## Deployment methods (https://www.tinybird.co/docs/forward/core-concepts/deployments#deployment-methods)

The following deployment methods are available:

- [Manual deployment](/docs/forward/development-workflow/manual-deployment).
- [CI/CD](/docs/forward/development-workflow/cicd).
If your project was initialized with `tb init --dev-mode local` or `tb init --dev-mode branch`, `tb deploy` without explicit environment flags deploys to your cloud main workspace.

## Staging deployments (https://www.tinybird.co/docs/forward/core-concepts/deployments#staging-deployments)

You can write data to, and read data from, a staging deployment before promoting it to live. This is useful when you've made schema changes that might be incompatible with the current live deployment, like adding new columns.

Automatic data source changes that Tinybird applies with `ALTER`, such as adding a column or changing a data source TTL, are applied only when the deployment is promoted to live. Those changes aren't available in staging deployments.

Use staging deployments to validate a deployment before it goes live. For longer-lived development environments that shouldn't block the deployment process, use [Cloud Branches](/docs/forward/core-concepts/branches) instead.

### Writing to staging deployments (https://www.tinybird.co/docs/forward/core-concepts/deployments#writing-to-staging-deployments)

You can use the [Events API](/docs/forward/ingest-data/events-api) to write directly to staging deployments through the `__tb__min_deployment` parameter, which indicates the target deployment ID. For example:

```
curl \
    -H "Authorization: Bearer <import_token>" \
    -d '{"date": "2020-04-05 00:05:38", "city": "Chicago", "new_column": "value"}' \
    'https://<your_host>/v0/events?name=events_test&__tb__min_deployment=5'
```
In the example, if the ID of your current live deployment is 4 and you're creating deployment 5, Tinybird ingests the data into staging deployment 5 only. You can:

1. Make schema changes in a staging deployment.
2. Ingest data compatible with the new schema.
3. Validate the changes work as expected.
4. Promote the deployment to live when ready.
Without the parameter, data would be rejected if it doesn't match the schema of the current live deployment.

If you discard the staging deployment instead of promoting it, the live deployment doesn't contain data written only to that staging deployment.

To get the deployment ID, run `tb deployment ls`.

### Reading from staging deployments (https://www.tinybird.co/docs/forward/core-concepts/deployments#reading-from-staging-deployments)

You can query data from a staging deployment using [pipe endpoints](/docs/forward/core-concepts/api-endpoints). To access a staging endpoint, add the `__tb__deployment` parameter to your API request:

```
curl \
    -H "Authorization: Bearer <query_token>" \
    'https://<your_host>/v0/pipes/my_endpoint?__tb__deployment=5'
```
You can:

1. Test your endpoints with the new schema changes.
2. Validate query results before promoting to live.
3. Ensure your app works correctly with the updated data structure.
To get the deployment ID, run `tb deployment ls`.

### Continuous operation (https://www.tinybird.co/docs/forward/core-concepts/deployments#continuous-operation)

Once the deployment is promoted to live, you can continue using the same API calls. In the previous example, calls using the `__tb__min_deployment=5` or `__tb__deployment=5` parameters continue working without interruption. The parameters ensure compatibility both before and after promotion.

For more details on the Events API parameters, see the [Events API Reference](/docs/api-reference/events-api).

## On-demand compute for deployment populates (https://www.tinybird.co/docs/forward/core-concepts/deployments#on-demand-compute-for-deployment-populates)

When a deployment needs to populate Data Sources, such as when creating Materialized Views or evolving schemas, Tinybird can run the populate on dedicated on-demand compute instead of your main workspace infrastructure. This keeps the populate isolated from production query and ingestion workloads.

Tinybird automatically uses on-demand compute for large deployment populates. For pricing, regional rates, and example calculations, see [On-demand CPUs](/docs/forward/pricing#on-demand-cpus).

## Next steps (https://www.tinybird.co/docs/forward/core-concepts/deployments#next-steps)

- See how to [deploy your project manually](/docs/forward/development-workflow/manual-deployment).
- See how to [deploy your project using CI/CD](/docs/forward/development-workflow/cicd).

---

URL: https://www.tinybird.co/docs/forward/core-concepts/data-sources
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Data sources \| Tinybird"
theme-color: "#171612"
description: "Data sources contain all the data you bring into Tinybird, acting like tables in a database."
inkeep:version: "forward"
---

# Data sources (https://www.tinybird.co/docs/forward/core-concepts/data-sources#data-sources)

When you send data to Tinybird, it's stored in a data source. You then write SQL queries to publish API [endpoints](/docs/forward/core-concepts/api-endpoints), use the [ClickHouse® interface](/docs/forward/query-data/clickhouse-interface), or use [MCP](/docs/forward/query-data/mcp) to query the data.

For example, if your event data lives in a Kafka topic, you can create a data source that connects directly to [Kafka](/docs/forward/ingest-data/connectors/kafka) and writes the events to Tinybird. Similarly, you can [send events](/docs/forward/ingest-data/events-api) or data [from a file](/docs/forward/ingest-data/local-file).

There are also intermediate data sources that are the result of [materialization](/docs/forward/core-concepts/materialized-views) or a [copy pipe](/docs/forward/core-concepts/copy-pipes).

Data sources can be defined with `.datasource` files, in `.ts` files with the TypeScript SDK, or in `.py` files using the Python SDK.

- Tinybird CLI
- TypeScript SDK
- Python SDK
sample.datasource
```
SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "session_id, timestamp"
ENGINE_TTL "timestamp + toIntervalDay(60)"
```
See all syntax options in the [Datafiles reference](/docs/forward/dev-reference/datafiles/datasource-files), [TypeScript SDK reference](/docs/forward/dev-reference/typescript-sdk-resources), and [Python SDK reference](/docs/forward/dev-reference/python-sdk-resources).

## Create Data Sources (https://www.tinybird.co/docs/forward/core-concepts/data-sources#create-data-sources)

To create a new Data Source, define it in your project and then deploy it.

If you need help generating a `.datasource` file, run `tb datasource create`. The CLI asks which type of Data Source you want to create:

- **Blank**: Generates a `.datasource` file with example columns you can edit.
- **Local file**: Creates a `.datasource` file based on the schema of a local file.
- **Remote URL**: Creates a `.datasource` file based on the schema of a file at a remote URL.
- **Kafka**: Creates a Data Source for a [Kafka connection](/docs/forward/ingest-data/connectors/kafka). If you don't have a Kafka connection yet, create it first; the schema is built from the topic you select.
- **Amazon S3**: Creates a Data Source for an [S3 connection](/docs/forward/ingest-data/connectors/s3). You need an existing connection because the schema is built from the file in the bucket you choose.
- **GCS**: Creates a Data Source for a [Google Cloud Storage connection](/docs/forward/ingest-data/connectors/gcs). You need an existing connection because the schema is built from the file in the bucket you choose.
Run `tb datasource create -h` anytime to see this list in the command help.

To convert `.datasource` files to SDK definitions, use <a href="/docs/forward/dev-reference/commands/typescript-sdk-cli#tinybird-migrate">`tinybird migrate`</a> for TypeScript projects or <a href="/docs/forward/dev-reference/commands/python-sdk-cli#tinybird-migrate">`tinybird migrate`</a> for Python projects.

## Delete Data Sources (https://www.tinybird.co/docs/forward/core-concepts/data-sources#delete-data-sources)

To delete a data source in Tinybird, remove its corresponding `.datasource` file or definition from your project and deploy your changes.

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb deploy --allow-destructive-operations
```
The Tinybird CLI and SDK CLIs require the `--allow-destructive-operations` flag to confirm the removal.

This operation permanently removes the Data Source and all its data from your Workspace. Review dependencies such as Pipes or Materialized Views that might rely on the Data Source before deleting it.

## Shared Data Source (https://www.tinybird.co/docs/forward/core-concepts/data-sources#shared-data-source)

Workspace administrators can share a Data Source with another Workspace they have access to in the same Organization.

To share a Data Source, add the destination Workspace names to the Data Source definition and deploy your changes. Use `SHARED_WITH` in `.datasource` files, `sharedWith` in the TypeScript SDK, or `shared_with` in the Python SDK.

- Tinybird CLI
- TypeScript SDK
- Python SDK
origin_datasource.datasource
```
# ... data source definition ...

SHARED_WITH >
    <destination_workspace>,
    <other_destination_workspace>
```

```
tb deploy
```
You can use the shared Data Source to create Pipes in the target Workspace. Users that have access to a shared Data Source can access the `tinybird.datasources_ops_log` and the `tinybird.kafka_ops_log` Service Data Sources.

### Limitations (https://www.tinybird.co/docs/forward/core-concepts/data-sources#limitations)

The following limitations apply to shared Data Sources:

- Shared Data Sources are read-only.
- You can't share a shared Data Source, only the original Workspace can share it.
- You can't check the quarantine of a shared Data Source.
- You can't create a Materialized View from a shared Data Source unless you're migrating from Classic and you already have them.

### Working locally with Shared Data Sources (https://www.tinybird.co/docs/forward/core-concepts/data-sources#working-locally-with-shared-data-sources)

When one Workspace shares a Data Source with another Workspace, deploy order matters. For example, say Workspace A shares a Data Source with Workspace B, and Workspace B uses that Data Source in an endpoint. If you start with a fresh Tinybird Local environment without either Workspace, deploy in this order:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
# Workspace B: first deploy creates the workspace, but fails because the shared Data Source is not available yet.
tb deploy

# Workspace A: deploy the Data Source that is shared with Workspace B.
tb deploy

# Workspace B: deploy again after the shared Data Source is available.
tb deploy
```
`tb build` for datafile projects, or `tinybird build` for SDK projects, hides all this complexity and creates the necessary workspaces and Data Sources to verify that a workspace is valid.

### Keeping .datasource files up-to-date (https://www.tinybird.co/docs/forward/core-concepts/data-sources#keeping-datasource-files-up-to-date)

For datafile projects, run `tb [--cloud] pull --only-vendored` to update the `.datasource` files of Data Sources shared with your Workspace. The command places them in `vendor/<name_of_the_source_workspace>/datasources`.

You can only deploy your project if the files in `vendor/` are up-to-date. If they aren't, your the deployment fails and prompts you to run the preceding command.

If you're using the SDKs, run the corresponding `tinybird migrate` command to convert `.datasource` files to `.ts` or `.py` definitions.

## Quarantine Data Sources (https://www.tinybird.co/docs/forward/core-concepts/data-sources#quarantine-data-sources)

Every Data Source you create in your Workspace has an associated quarantine Data Source that stores rows that don't fit the schema. If a row doesn't match the Data Source schema, Tinybird writes it to the quarantine Data Source instead of failing the whole ingest process.

The quarantine Data Source keeps the columns from the original row and adds metadata columns that explain why the row was quarantined:

- `c__error_column` (`Array(String)`): The columns that contain invalid values.
- `c__error` (`Array(String)`): The ingestion errors that caused the row to be quarantined.
- `c__import_id` (`Nullable(String)`): The import job identifier, when the row was imported through a job.
- `insertion_date` (`DateTime`): The timestamp when Tinybird ingested the row.
Quarantine Data Sources are recreated when you deploy a new version of the original Data Source. If you need to inspect, query, or recover quarantined rows, do it before deploying schema changes to the original Data Source.

Use quarantine Data Sources to inspect schema mismatches, recover failed rows, and validate schema changes during ingest workflows. For step-by-step examples, see the [Quarantine Data Sources guide](/docs/forward/guides/quarantine).

## Data operations (https://www.tinybird.co/docs/forward/core-concepts/data-sources#data-operations)

Data Sources are append-only by default. When you need to update or remove data, use explicit data operations instead of treating Data Sources like mutable OLTP tables.

- **Append data** from events, files, or connectors. See [Ingest Data](/docs/forward/ingest-data).
- **Replace data** when you need to reingest a full Data Source or a partitioned slice. See [Replace and delete data](/docs/forward/guides/replace-and-delete-data) and [Replace data from a file](/docs/forward/ingest-data/local-file#replace-data-from-a-file).
- **Delete data** for selective removals or compliance workflows. See [Delete data selectively](/docs/forward/guides/replace-and-delete-data#delete-data-selectively), [GDPR-compliant data deletion](/docs/forward/guides/gdpr-compliant-data-deletion), and [delete limits](/docs/forward/pricing/limits#delete-limits).
- **Deduplicate data** at query time, with table engines, or by rebuilding derived Data Sources depending on your workload. See [Deduplication strategies](/docs/forward/guides/deduplication-strategies). This is often the best pattern when you need update-like behavior in analytical workloads.
For one-off corrections, use replace or delete operations directly. If updates and deletes are frequent, model the workload as append-only events and deduplicate the latest state at query or processing time. This is usually more efficient and easier to operate at scale.

---

URL: https://www.tinybird.co/docs/forward/core-concepts/copy-pipes
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Copy pipes \| Tinybird"
theme-color: "#171612"
description: "Copy pipes capture the result of a pipe at a moment in time and write the result into a target data source. They can be run on a schedule, or executed on demand."
inkeep:version: "forward"
---

# Copy pipes (https://www.tinybird.co/docs/forward/core-concepts/copy-pipes#copy-pipes)

Copy pipes capture the result of a pipe at a moment in time and write the result into a target data source. They can be run on a schedule, or executed on demand.

Copy pipes should not be confused with [materialized views](/docs/forward/core-concepts/materialized-views). While materialized views continuously update as new events are inserted, copy pipes generate a single snapshot at a specific point in time.

## Create a copy pipe (https://www.tinybird.co/docs/forward/core-concepts/copy-pipes#create-a-copy-pipe)

Copy pipes are defined with SQL nodes and a target Data Source. Use `TYPE COPY` in `.pipe` files, `defineCopyPipe` in the TypeScript SDK, or `define_copy_pipe` in the Python SDK.

Define the queries that filter and transform the data as needed. The final result is written into the target Data Source. Before pushing a copy pipe to your workspace, make sure that the target Data Source already exists and has a schema that matches the query output.

The following example creates a copy pipe scheduled to run every hour and write the query result into the `sales_hour_copy` Data Source.

Copy Pipes can use Tinybird's [templating language](/docs/forward/core-concepts/templating-language) in SQL nodes. Scheduled Copy Pipes must provide defaults for parameters because there is no request-time value.

- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/teams.datasource
```
SCHEMA >
    `starting_date` DateTime,
    `country` String,
    `sales` Float64

ENGINE "MergeTree"
ENGINE_SORTING_KEY "starting_date, country"
```
datasources/sales_hour_copy.datasource
```
SCHEMA >
    `day` DateTime,
    `country` String,
    `total_sales` Float64

ENGINE "MergeTree"
ENGINE_SORTING_KEY "day, country"
```
pipes/sales_hour_copy.pipe
```
NODE daily_sales
SQL >
    %
    SELECT toStartOfDay(starting_date) day, country, sum(sales) as total_sales
    FROM teams
    WHERE
    day BETWEEN toStartOfDay({{DateTime(job_timestamp)}}) - interval 1 day AND toStartOfDay({{DateTime(job_timestamp)}})
    and country = {{ String(country, 'US')}}
    GROUP BY day, country

TYPE COPY
TARGET_DATASOURCE sales_hour_copy
COPY_SCHEDULE 0 * * * *
COPY_MODE append
```
See all syntax options in the [Copy pipes reference](/docs/forward/dev-reference/datafiles/pipe-files#copy-pipes), [TypeScript SDK reference](/docs/forward/dev-reference/typescript-sdk-resources#copy-pipes), and [Python SDK reference](/docs/forward/dev-reference/python-sdk-resources#copy-pipes).

## Schedule a copy pipe (https://www.tinybird.co/docs/forward/core-concepts/copy-pipes#schedule-a-copy-pipe)

You can schedule copy pipes to run at a specific time using a cron expression. To schedule a copy pipe, configure `COPY_SCHEDULE` with a cron expression. On-demand copy pipes are defined by configuring `COPY_SCHEDULE` with the value `@on-demand`.

Copy Pipes scheduled with a cron expression have a maximum timeout defined by the lower of the limit enforced by your plan and half the interval of your cron expression. For example, a Copy Pipe scheduled to run every 5 minutes (300 seconds) has a maximum timeout of 2.5 minutes (150 seconds), even if your plan permits an hour. This prevents your Copy Pipes from overlapping.

All schedules use the UTC time zone. When you configure a schedule that runs at a specific time, convert the desired time from your local time zone to UTC.

## Run, pause, or resume a copy pipe (https://www.tinybird.co/docs/forward/core-concepts/copy-pipes#run-pause-or-resume-a-copy-pipe)

Use the `tb copy` command to run, pause, or resume a copy pipe. See [tb copy](/docs/forward/dev-reference/commands/tb-copy).

You can query `tinybird.jobs_log` to see any running jobs, as well as any jobs that have finished during the last 48 hours.

### Run Copy Pipes in Tinybird Local (https://www.tinybird.co/docs/forward/core-concepts/copy-pipes#run-copy-pipes-in-tinybird-local)

When running Copy Pipes locally, you might encounter this error:

```
Error: Failed creating copy job: Error while running job: There was a problem while copying data: [Error] Cannot schedule a task: no free thread (timeout=0) (threads=2, jobs=2). (CANNOT_SCHEDULE_TASK)
```
This occurs when the local ClickHouse® instance runs out of available threads. To fix this, limit the number of threads the copy job can use by adding the `max_threads` parameter at the top of your SQL query:

```
NODE daily_sales
SQL >
    %
    {{max_threads(1)}}
    SELECT toStartOfDay(starting_date) day, country, sum(sales) as total_sales
    FROM teams
    WHERE
    day BETWEEN toStartOfDay({{DateTime(job_timestamp)}}) - interval 1 day AND toStartOfDay({{DateTime(job_timestamp)}})
    and country = {{ String(country, 'US')}}
    GROUP BY day, country

TYPE COPY
TARGET_DATASOURCE sales_hour_copy
COPY_SCHEDULE 0 * * * *
```
The `max_threads(1)` parameter ensures the copy job uses only one thread, preventing the scheduling error in local environments with limited resources.

## On-demand compute for copy jobs (https://www.tinybird.co/docs/forward/core-concepts/copy-pipes#on-demand-compute-for-copy-jobs)

When running a copy job, you can isolate execution on dedicated on-demand compute, separate from your main workspace infrastructure. This prevents copy jobs from impacting production query performance.

Unlike deployment populates, on-demand compute for copy jobs is not automatically activated based on data size. You must pass the flag explicitly:

```
tb copy run <copy_pipe_name> --on-demand-compute
```
For pricing, regional rates, and example calculations, see [On-demand CPUs](/docs/forward/pricing#on-demand-cpus).

## Next steps (https://www.tinybird.co/docs/forward/core-concepts/copy-pipes#next-steps)

- Learn about [Materialized views](/docs/forward/core-concepts/materialized-views).
- Publish your data with [API Endpoints](/docs/forward/core-concepts/api-endpoints).

---

URL: https://www.tinybird.co/docs/forward/core-concepts/connections
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Connections \| Tinybird"
theme-color: "#171612"
description: "Connections store the credentials and settings Tinybird uses to ingest data from external services."
inkeep:version: "forward"
---

# Connections (https://www.tinybird.co/docs/forward/core-concepts/connections#connections)

Connections store the credentials and settings Tinybird uses to connect to external services such as Kafka, Amazon S3, and Google Cloud Storage. Data Sources reference connections to stream or batch import data into Tinybird.

Connections are managed by Tinybird and are deployed as part of your project. They handle the integration details for:

- Authentication and secure connections
- Schema detection and mapping
- Incremental updates and change data capture
- Error handling and monitoring
- Scheduling and orchestration
Connections are independent from Data Sources. Deleting a Data Source that uses a connection does not delete the connection, so you can reuse the same connection in other Data Sources.

## Define a connection (https://www.tinybird.co/docs/forward/core-concepts/connections#define-a-connection)

You can define connections with `.connection` files, in `.ts` files with the TypeScript SDK, or in `.py` files with the Python SDK. Use `tb_secret()` in datafiles, or `secret()` in the SDKs, to avoid storing credentials in plain text. See [Templating language](/docs/forward/core-concepts/templating-language#secrets) for how secrets are rendered across environments.

- Tinybird CLI
- TypeScript SDK
- Python SDK
connections/events_kafka.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS {{ tb_secret("KAFKA_BOOTSTRAP_SERVERS", "localhost:9092") }}
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("KAFKA_KEY") }}
KAFKA_SECRET {{ tb_secret("KAFKA_SECRET") }}
```
See all syntax options in the [Connection files reference](/docs/forward/dev-reference/datafiles/connection-files), [TypeScript SDK reference](/docs/forward/dev-reference/typescript-sdk-resources#connections), and [Python SDK reference](/docs/forward/dev-reference/python-sdk-resources#connections).

## Create a connection (https://www.tinybird.co/docs/forward/core-concepts/connections#create-a-connection)

To create a connection, add the `.connection` file to the project, or the connection definition to `.ts` or `.py` files and deploy.

For datafile projects, you can scaffold a new connection with <a href="/docs/forward/dev-reference/commands/tb-connection">`tb connection create`</a>:

```
tb connection create <type>
```
Where `<type>` is one of the available connection types:

- `kafka` - Create a [Kafka connection](/docs/forward/ingest-data/connectors/kafka)
- `s3` - Create an [Amazon S3 connection](/docs/forward/ingest-data/connectors/s3)
- `gcs` - Create a [Google Cloud Storage connection](/docs/forward/ingest-data/connectors/gcs)
The Tinybird CLI guides you through the configuration process and prompts you for the required credentials and settings. Once created, the CLI saves the Connection as a `.connection` file in your project's `connections/` folder.

To convert existing `.connection` files to SDK definitions, use <a href="/docs/forward/dev-reference/commands/typescript-sdk-cli#tinybird-migrate">`tinybird migrate`</a> for TypeScript projects or <a href="/docs/forward/dev-reference/commands/python-sdk-cli#tinybird-migrate">`tinybird migrate`</a> for Python projects.

## Use a connection in a Data Source (https://www.tinybird.co/docs/forward/core-concepts/connections#use-a-connection-in-a-data-source)

After you define a connection, reference it from a Data Source. Use `KAFKA_CONNECTION_NAME`, `S3_CONNECTION_NAME`, or `GCS_CONNECTION_NAME` in `.datasource` files, or pass the connection object in the TypeScript and Python SDKs.

- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/events.datasource
```
SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `payload` String `json:$.payload`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "timestamp"

KAFKA_CONNECTION_NAME events_kafka
KAFKA_TOPIC events
KAFKA_GROUP_ID {{ tb_secret("KAFKA_GROUP_ID") }}
```

## Delete a connection (https://www.tinybird.co/docs/forward/core-concepts/connections#delete-a-connection)

To delete a connection, remove its `.connection` file or SDK definition, make sure no Data Sources reference it, and deploy your changes.

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb deploy --allow-destructive-operations
```
The Tinybird CLI and SDK CLIs require the `--allow-destructive-operations` flag to confirm the removal.

Before deleting a connection, make sure no Data Sources use it. If a Data Source references the connection, the deployment fails. Remove the connection reference from all Data Sources first.

## Available connection types (https://www.tinybird.co/docs/forward/core-concepts/connections#available-connection-types)

Tinybird supports the following connection types:

- [Kafka](/docs/forward/ingest-data/connectors/kafka)
- [Amazon S3](/docs/forward/ingest-data/connectors/s3)
- [Google Cloud Storage](/docs/forward/ingest-data/connectors/gcs)

---

URL: https://www.tinybird.co/docs/forward/core-concepts/branches
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Branches \| Tinybird"
theme-color: "#171612"
description: "Create branches from your production workspace to develop and test your project before deploying to production."
inkeep:version: "forward"
---

# Tinybird Cloud Branches (https://www.tinybird.co/docs/forward/core-concepts/branches#tinybird-cloud-branches)

Tinybird Cloud Branches let you develop and test your project in ephemeral cloud environments without affecting your main workspace. They give you a full Tinybird project for development: its own ClickHouse® database, token namespace, API surface, and resource definitions.

A branch is a copy of your Workspace resources, such as Data Sources, Pipes, Endpoints, Tokens, and Connections. When you create a branch manually with `tb branch create`, it copies the workspace structure, not the workspace data. When `tb build` auto-creates a branch with `dev_mode=branch`, it attaches the latest production partition by default, as configured in `branch_data_mode`. This gives you an isolated environment where you can change schemas, queries, and APIs safely. When you need a branch without production data, create it manually without `--last-partition`.

Branches are designed for short-lived development and validation loops. Use them to test schema changes against real production values, run preview environments for pull requests, let coding agents iterate safely, or validate connector-based pipelines before deploying. You can truncate tables, redefine schemas, create endpoints, or start connector ingestion in a branch without changing production.

## When to use Cloud Branches (https://www.tinybird.co/docs/forward/core-concepts/branches#when-to-use-cloud-branches)

Branches are the default way to develop your project, optionally testing against real production data. You should use them in the following scenarios:

- You want to test your changes with real production data.
- You work with preview environments in your CI/CD pipeline before deploying to production.
- You don't use Docker and you want to test your changes without affecting production.

## Data in branches (https://www.tinybird.co/docs/forward/core-concepts/branches#data-in-branches)

When you create a branch with `--last-partition`, Tinybird attaches the latest production partition without copying the underlying data. The branch gets an isolated snapshot backed by immutable ClickHouse parts, so writes, deletes, and schema changes in the branch stay in the branch. Connector data is isolated too: Kafka starts stopped with its own consumer group, and S3, GCS, and DynamoDB connectors can import samples for validation.

## Branches in the development workflow (https://www.tinybird.co/docs/forward/core-concepts/branches#branches-in-the-development-workflow)

Branches are not promoted to production. You use a branch to validate your project, then deploy the project to your main Tinybird Cloud Workspace with `tb deploy`, the SDK deployment flow, or CI/CD.

If your Tinybird project is in a Git repository, you usually don't need to create branches manually. Check out a Git branch and run `tb build`; Tinybird automatically creates or uses a Cloud Branch with the same name as your Git branch and attaches the latest production partition by default.

```
git checkout -b preview_1
tb build
```

## Manually create a branch (https://www.tinybird.co/docs/forward/core-concepts/branches#manually-create-a-branch)

You can also create a branch manually with the Tinybird CLI.

```
tb branch create preview_1
```
If you want to use production data, use the `--last-partition` flag to bring the last partition of production data into the branch.

```
tb branch create preview_1 --last-partition
```

## Start a branch (https://www.tinybird.co/docs/forward/core-concepts/branches#start-a-branch)

After you create a branch manually, start a development session against it:

```
tb --branch=preview_1 dev
```
If the branch was created automatically with `tb build`, start the development session by passing the same branch name. `tb dev` without `--branch` starts Tinybird Local.

```
tb --branch=preview_1 dev
```
Keep this terminal running while you are working in the branch.

`tb dev` watches for changes in your project and rebuilds it automatically.

```
» Building project...
✓ datasources/user_actions.datasource created
✓ endpoints/user_actions_line_chart.pipe created
✓ Rebuild completed in 0.2s
```
Stop the process and the branch with `Ctrl+C`.

```
» Received shutdown signal, stopping...
✓ Branch 'preview_1' session stopped
```

## Make changes to your project (https://www.tinybird.co/docs/forward/core-concepts/branches#make-changes-to-your-project)

While `tb --branch=preview_1 dev` is running, you can edit files in your project and see the changes automatically applied in your branch.

### Using your editor (https://www.tinybird.co/docs/forward/core-concepts/branches#using-your-editor)

Open your editor and start editing your project. The branch detects changes and rebuilds your project automatically:

```
» Building project...
✓ datasources/user_actions.datasource updated
✓ Rebuild completed in 0.3s
```

### Using Tinybird UI (https://www.tinybird.co/docs/forward/core-concepts/branches#using-tinybird-ui)

You can also use Tinybird UI to edit your project. Run the following command to open the Tinybird UI pointing to your branch.

```
tb --branch=preview_1 open
```
`tb dev` exposes your project as API, so you can edit it directly in the browser and see changes applied automatically.

## Test with connector data (https://www.tinybird.co/docs/forward/core-concepts/branches#test-with-connector-data)

When your project uses connector data sources (Kafka, S3, GCS, or DynamoDB), you can test them in branches using the `--with-connections` flag and dedicated CLI commands. Validate schema changes, test pipelines, and verify data transformations against real production data without affecting your production environment.

### Enable connectors in branches (https://www.tinybird.co/docs/forward/core-concepts/branches#enable-connectors-in-branches)

Use `--with-connections` when building or starting a branch:

```
tb --branch=preview_1 dev --with-connections
```
Or when building directly:

```
tb --branch=preview_1 build --with-connections
```
This creates data linkers for your connector data sources (S3, Kafka, GCS, DynamoDB) in the branch.

### Kafka: Pause and resume ingestion (https://www.tinybird.co/docs/forward/core-concepts/branches#kafka-pause-and-resume-ingestion)

By default, Kafka ingestion is stopped in branches. Use `tb datasource start` to begin ingesting and `tb datasource stop` to pause.

Each time you start ingestion, a new consumer group is created with a unique ID for the branch. This means ingestion starts from the latest offset, not from where it previously left off. Consumer group IDs are unique and don't collide with production or other branches. This is by design. Branches are ephemeral testing environments, not production replicas.

Because each start creates a new consumer group, previous consumer groups become orphaned. Depending on your Kafka cluster's consumer group TTL (`offsets.retention.minutes`), these orphan groups may persist until they expire. Keep this in mind if you start and stop ingestion frequently across multiple branches.

```
tb --branch=preview_1 datasource start my_kafka_datasource
# ... observe data flowing in, test your pipelines ...
tb --branch=preview_1 datasource stop my_kafka_datasource
```

### S3/GCS: Import sample data (https://www.tinybird.co/docs/forward/core-concepts/branches#s3gcs-import-sample-data)

Instead of syncing all files from the bucket (as in production), you can import a small sample to validate your schemas and pipelines. The sample import runs as a separate job. It doesn't affect production sync state or offsets.

You can trigger a sample import using the CLI or the API:

Using the CLI
```
tb --branch=preview_1 datasource sample my_datasource --wait
```
Using the API
```
curl -X POST "https://api.tinybird.co/v0/datasources/my_datasource/sample" \
  -H "Authorization: Bearer <your-admin-token>"
```
The response includes a `job_id` to track progress via `GET /v0/jobs/{job_id}`. You can import up to 10 files per request.

This is useful for:

- Validating that your schema matches the data format.
- Testing downstream pipes and endpoints.
- Verifying data transformations.

### DynamoDB: import sample data (https://www.tinybird.co/docs/forward/core-concepts/branches#dynamodb-import-sample-data)

DynamoDB data sources start empty in branches: building or deploying sets up the connection but doesn't trigger a PITR export. Use `tb datasource sample` to load a bounded subset of the table on demand:

```
tb --branch=preview_1 datasource sample my_datasource --wait
```
By default, the sample imports up to 1,500 rows by scanning the table directly, which is faster and cheaper than a full export. Use `--rows` or `--max-bytes` to adjust the cap, or `--full-export` to trigger a full PITR export instead. See [Sample data in branches and Local](/docs/forward/ingest-data/connectors/dynamodb#sample-data-in-branches-and-local) for all options and limits.

Connectors in branches are meant for testing and validation, not for replicating production workloads.

## Alternative: Tinybird Local workflow (https://www.tinybird.co/docs/forward/core-concepts/branches#alternative-tinybird-local-workflow)

If you don't need a shared Cloud environment or production data, you can use [Tinybird Local](/docs/forward/core-concepts/tinybird-local) instead. Tinybird Local runs on your machine and is useful for fast, isolated development and CI checks before you deploy to Tinybird Cloud. For the Local workflow, see [Local development](/docs/forward/development-workflow/local-development).

---

URL: https://www.tinybird.co/docs/forward/core-concepts/api-endpoints
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Endpoints \| Tinybird"
theme-color: "#171612"
description: "Endpoints make it easy to use the results of your queries in applications."
inkeep:version: "forward"
---

# API Endpoints (https://www.tinybird.co/docs/forward/core-concepts/api-endpoints#api-endpoints)

Endpoints are a type of pipe that you can call from other applications. You ingest your data into your Data Sources, build SQL logic inside a pipe, and then publish the result of your query as a REST API endpoint.

## Create an endpoint (https://www.tinybird.co/docs/forward/core-concepts/api-endpoints#create-an-endpoint)

You define Endpoints in `.pipe` files with the `TYPE ENDPOINT` directive, with `defineEndpoint` in the TypeScript SDK, or with `define_endpoint` in the Python SDK.

- Tinybird CLI
- TypeScript SDK
- Python SDK
pipes/top_pages.pipe
```
DESCRIPTION >
    Get the most visited pages

TOKEN dashboard READ

NODE aggregated
SQL >
    %
    SELECT pathname, count() AS views
    FROM page_views
    WHERE timestamp >= {{DateTime(start_date)}}
      AND timestamp <= {{DateTime(end_date)}}
    GROUP BY pathname
    ORDER BY views DESC
    LIMIT {{Int32(limit, 10)}}

TYPE ENDPOINT
```
See all syntax options in the [Endpoint pipes reference](/docs/forward/dev-reference/datafiles/pipe-files#endpoint-pipes), [TypeScript SDK reference](/docs/forward/dev-reference/typescript-sdk-resources#endpoints), and [Python SDK reference](/docs/forward/dev-reference/python-sdk-resources#endpoints).

You can list your endpoints and their URLs and tokens using the `tb endpoint` command. See [tb endpoint](/docs/forward/dev-reference/commands/tb-endpoint).

## Response formats (https://www.tinybird.co/docs/forward/core-concepts/api-endpoints#response-formats)

API Endpoints can return results in different formats. Choose the response format by appending the format extension to the Endpoint URL.

For example:

```
https://api.tinybird.co/v0/pipes/top_pages.json
https://api.tinybird.co/v0/pipes/top_pages.csv
https://api.tinybird.co/v0/pipes/top_pages.ndjson
https://api.tinybird.co/v0/pipes/top_pages.parquet
https://api.tinybird.co/v0/pipes/top_pages.prometheus
```
Supported formats include:

- `.json`: JSON response. This is the default format.
- `.csv`: CSV response.
- `.csvwithnames`: CSV response with column names.
- `.ndjson`: Newline-delimited JSON response.
- `.parquet`: Parquet response.
- `.prometheus`: Prometheus metrics response. Use this when you want observability tools to scrape an Endpoint.
See the [Pipe API reference](/docs/api-reference/pipe-api/api-endpoints#response-formats) for request parameters and response details.

## Query parameters (https://www.tinybird.co/docs/forward/core-concepts/api-endpoints#query-parameters)

Endpoints can receive query parameters to make the published query dynamic at request time. Use parameters for filters, date ranges, pagination, sorting, tenant IDs, and limits.

Endpoints use Tinybird's [templating language](/docs/forward/core-concepts/templating-language) to validate parameter types and render SQL safely before the query runs.

For parameter syntax, defaults, helper functions, pagination, and advanced patterns, see [Query parameters](/docs/forward/query-data/query-parameters).

## Errors and retries (https://www.tinybird.co/docs/forward/core-concepts/api-endpoints#errors-and-retries)

For request behavior, HTTP error codes, and retry guidance, see [Endpoints in Query Data](/docs/forward/query-data/api-endpoints#errors-and-retries).

---

URL: https://www.tinybird.co/docs/forward/ingest-data/table-functions/url
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "URL table function \| Tinybird"
theme-color: "#171612"
description: "Documentation for the Tinybird URL table function"
inkeep:version: "forward"
---

# URL table function (https://www.tinybird.co/docs/forward/ingest-data/table-functions/url#url-table-function)

Use the Tinybird `url()` table function to read data from an existing URL into Tinybird, then schedule a regular Copy Pipe to orchestrate synchronization. You can load full tables, and every run performs a full replace on the Data Source.

To use it, define a node using standard SQL and the `url` function keyword, then publish the node as a copy pipe that does a sync on every run. See [Table functions](../table-functions) for general information and tips.

## Syntax (https://www.tinybird.co/docs/forward/ingest-data/table-functions/url#syntax)

Create a new pipe node. Call the `url` table function and pass the URL. Optionally, pass the format and the structure:

Example query logic
```
SELECT
    JSONExtractString(data, 'article') AS article,
    JSONExtractInt(data, 'views') AS views,
    JSONExtractInt(data, 'rank') AS rank
FROM
    (
        SELECT toJSONString(arrayJoin(items.articles)) AS data
        FROM
            url(
                'https://wikimedia.org/api/rest_v1/metrics/pageviews/top/en.wikipedia.org/all-access/2024/03/all-days',
                'JSONColumns',
                'items Tuple(access Nullable(String), articles Array(Tuple(article Nullable(String), rank Nullable(Int64), views Nullable(Int64))), day Nullable(String), month Nullable(String), project Nullable(String), year Nullable(String))'
            )
    )
```
Publish this node as a copy pipe. You can choose to append only new data or replace all data.

## Example: sync page view rankings from a URL (https://www.tinybird.co/docs/forward/ingest-data/table-functions/url#example-sync-page-view-rankings-from-a-url)

The following example reads the Wikimedia pageviews API, extracts the article rankings, and replaces the target Data Source on demand.

First, define the target Data Source. Then define the copy pipe.

- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/wiki_pageviews.datasource
```
SCHEMA >
    `article` String,
    `views` UInt64,
    `rank` UInt32,
    `day` Date

ENGINE "MergeTree"
ENGINE_SORTING_KEY "day, rank"
```
pipes/wiki_pageviews_sync.pipe
```
NODE pageviews
SQL >
    %
    SELECT
        JSONExtractString(data, 'article') AS article,
        toUInt64(JSONExtractInt(data, 'views')) AS views,
        toUInt32(JSONExtractInt(data, 'rank')) AS rank,
        toDate('2024-03-01') AS day
    FROM (
        SELECT toJSONString(arrayJoin(items.articles)) AS data
        FROM url(
            'https://wikimedia.org/api/rest_v1/metrics/pageviews/top/en.wikipedia.org/all-access/2024/03/01',
            'JSONColumns',
            'items Tuple(access Nullable(String), articles Array(Tuple(article Nullable(String), rank Nullable(Int64), views Nullable(Int64))), day Nullable(String), month Nullable(String), project Nullable(String), year Nullable(String))'
        )
    )

TYPE copy
TARGET_DATASOURCE wiki_pageviews
COPY_MODE replace
COPY_SCHEDULE @on-demand
```
Run the copy pipe on demand when you want to refresh the snapshot.

## See also (https://www.tinybird.co/docs/forward/ingest-data/table-functions/url#see-also)

- [Table functions](../table-functions)

---

URL: https://www.tinybird.co/docs/forward/ingest-data/table-functions/postgresql
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "PostgreSQL table function \| Tinybird"
theme-color: "#171612"
description: "Documentation for the Tinybird PostgreSQL table function"
inkeep:version: "forward"
---

# PostgreSQL table function (https://www.tinybird.co/docs/forward/ingest-data/table-functions/postgresql#postgresql-table-function)

Use the Tinybird `postgresql()` table function to read data from your existing PostgreSQL database into Tinybird, then schedule a regular Copy Pipe to orchestrate synchronization. You can load full tables, and every run performs a full replace on the Data Source.

To use it, define a node using standard SQL and the `postgresql` function keyword, then publish the node as a copy pipe that does a sync on every run. See [Table functions](../table-functions) for general information and tips.

## Setting secrets (https://www.tinybird.co/docs/forward/ingest-data/table-functions/postgresql#setting-secrets)

Table functions require authentication credentials that must be stored securely. In Tinybird Forward, manage those credentials with `tb secret`:

```
tb secret set PG_USERNAME <username>
tb secret set PG_PASSWORD <password>
```
Set the secret in each environment where the copy pipe runs. For example, use `tb --cloud secret set` for Cloud and `tb --branch=<branch_name> secret set` for a branch. Then reference the secrets in SQL with `tb_secret()`.

In Tinybird Classic, use the [Environment Variables API](/docs/api-reference/environment-variables-api) to create the same secret values.

For more details, see [tb secret](/docs/forward/dev-reference/commands/tb-secret).

## Syntax (https://www.tinybird.co/docs/forward/ingest-data/table-functions/postgresql#syntax)

Create a new copy pipe node. Call the `postgresql` table function and pass the hostname and port, database, table, user, and password:

Example query logic
```
SELECT *
FROM postgresql(
  'postgresql.example.com:5432',
  '<YOUR_PG_DB>',
  '<YOUR_PG_TABLE>',
  {{ tb_secret("PG_USERNAME") }},
  {{ tb_secret("PG_PASSWORD") }}
)

TYPE COPY
TARGET_DATASOURCE pg_copy_target_ds
```
Publish this node as a copy pipe. You can choose to append only new data or replace all data.

## Example: sync orders from PostgreSQL (https://www.tinybird.co/docs/forward/ingest-data/table-functions/postgresql#example-sync-orders-from-postgresql)

The following example copies an `orders` table from PostgreSQL into a Tinybird Data Source every 5 minutes.

First, define the target Data Source. Then define a copy pipe that reads from PostgreSQL and appends rows updated in the last 10 minutes.

- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/pg_orders.datasource
```
SCHEMA >
    `order_id` UInt64,
    `customer_id` UInt64,
    `status` String,
    `amount` Float64,
    `updated_at` DateTime

ENGINE "ReplacingMergeTree(updated_at)"
ENGINE_SORTING_KEY "order_id"
```
pipes/pg_orders_sync.pipe
```
NODE updated_orders
SQL >
    %
    SELECT
        order_id,
        customer_id,
        status,
        amount,
        updated_at
    FROM postgresql(
        'postgres.example.com:5432',
        'shop',
        'orders',
        {{ tb_secret("PG_USERNAME") }},
        {{ tb_secret("PG_PASSWORD") }}
    )
    WHERE updated_at >= now() - INTERVAL 10 MINUTE

TYPE copy
TARGET_DATASOURCE pg_orders
COPY_MODE append
COPY_SCHEDULE */5 * * * *
```
Use `ReplacingMergeTree(updated_at)` when the source can send multiple versions of the same row. Query with `FINAL` when you need the latest version at read time.

## Type support and inference (https://www.tinybird.co/docs/forward/ingest-data/table-functions/postgresql#type-support-and-inference)

Here's a detailed conversion table:

| PostgreSQL data type | Tinybird data type |
| --- | --- |
| BOOLEAN | UInt8 or Bool |
| SMALLINT | Int16 |
| INTEGER | Int32 |
| BIGINT | Int64 |
| REAL | Float32 |
| DOUBLE PRECISION | Float64 |
| NUMERIC or DECIMAL | Decimal(p, s) |
| CHAR(n) | FixedString(n) |
| VARCHAR (n) | String |
| TEXT | String |
| BYTEA | String |
| TIMESTAMP | DateTime |
| TIMESTAMP WITH TIME ZONE | DateTime (with appropriate timezone handling) |
| DATE | Date |
| TIME | String (since there is no direct TIME type) |
| TIME WITH TIME ZONE | String |
| INTERVAL | String |
| UUID | UUID |
| ARRAY | Array(T) where T is the array element type |
| JSON | String or JSON |
| JSONB | String |
| INET | String |
| CIDR | String |
| MACADDR | String |
| ENUM | Enum8 or Enum16 |
| GEOMETRY | String |

## Enabling the PostgreSQL table function (https://www.tinybird.co/docs/forward/ingest-data/table-functions/postgresql#enabling-the-postgresql-table-function)

### In production (https://www.tinybird.co/docs/forward/ingest-data/table-functions/postgresql#in-production)

To turn on the PostgreSQL table function in your production Workspace, contact Tinybird support. Support turns on the function for your specific Workspace.

### For local development (https://www.tinybird.co/docs/forward/ingest-data/table-functions/postgresql#for-local-development)

After support turns on the feature for your Workspace, it becomes available for local development automatically. You don't need to take any extra steps to use it locally.

## Use the PostgreSQL table function locally (https://www.tinybird.co/docs/forward/ingest-data/table-functions/postgresql#use-the-postgresql-table-function-locally)

There are two primary scenarios for connecting to a PostgreSQL database from Tinybird Local:

### Connect to PostgreSQL running on your host machine (https://www.tinybird.co/docs/forward/ingest-data/table-functions/postgresql#connect-to-postgresql-running-on-your-host-machine)

When connecting to PostgreSQL running directly on your local machine (not in a container), keep the following considerations in mind:

1. **Network connection**: The connection to your PostgreSQL server originates from within the Tinybird Local container.
2. **Server reachability**: Ensure your PostgreSQL server is reachable from inside the Docker network.
3. **Credentials**: Set the secrets for your PostgreSQL credentials in your local Tinybird project. You can provide default values for these credentials inside the pipe SQL, but not through the CLI:

```
tb secret set PG_USERNAME <YOUR_PG_USERNAME>
tb secret set PG_PASSWORD <YOUR_PG_PASSWORD>
```

1. **Server address**: Use the container-reachable address in your queries, not `localhost`.
Example query:

```
NODE get_ids
SQL >
%
SELECT id
FROM postgresql(
  'host.docker.internal:5432',
  '<YOUR_PG_DB>',
  '<YOUR_PG_TABLE>',
  {{ tb_secret("PG_USERNAME", "<YOUR_DEFAULT_USERNAME>") }},
  {{ tb_secret("PG_PASSWORD", "<YOUR_DEFAULT_PWD>") }}
)

TYPE COPY
TARGET_DATASOURCE pg_copy_target_ds
```

### Connect to PostgreSQL running in a Docker container (https://www.tinybird.co/docs/forward/ingest-data/table-functions/postgresql#connect-to-postgresql-running-in-a-docker-container)

When connecting to a PostgreSQL container running in Docker, follow these steps to set up network communication between Tinybird Local and your PostgreSQL container:

1. Create a shared Docker network for PostgreSQL and Tinybird Local to communicate:

```
docker network create tbnet
```

1. Run PostgreSQL container in the shared network:

```
docker run --name local-postgres \
  --network tbnet \
  -e POSTGRES_USER=tb_user \
  -e POSTGRES_PASSWORD=tb_pass \
  -e POSTGRES_DB=test_db \
  -p 5432:5432 \
  -d postgres:15
```

1. Connect Tinybird Local to the shared network:

```
docker network connect tbnet tinybird-local
```

1. Verify network connectivity by checking that both containers are on the same network. The NetworkID, Gateway, and IPAddress values should match:

```
docker inspect tinybird-local --format '{{json .NetworkSettings.Networks}}' | jq
docker inspect local-postgres --format '{{json .NetworkSettings.Networks}}' | jq
```

1. Set secrets in Tinybird Local to match your PostgreSQL container configuration:

```
tb secret set PG_USERNAME tb_user
tb secret set PG_PASSWORD tb_pass
```

1. Update the PostgreSQL host in your query to use the container name as the hostname:

```
NODE get_ids
SQL >
%
SELECT id
FROM postgresql(
  'local-postgres:5432',
  '<YOUR_PG_DB>',
  '<YOUR_PG_TABLE>',
  {{ tb_secret("PG_USERNAME") }},
  {{ tb_secret("PG_PASSWORD") }}
)

TYPE COPY
TARGET_DATASOURCE pg_copy_target_ds
```

1. Build and deploy your pipe:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb build
tb deploy
```

1. Test that the connection works. Run the copy pipe with `tb copy run <pipe_name>` and check that PostgreSQL data lands in the target Data Source.

## Considerations (https://www.tinybird.co/docs/forward/ingest-data/table-functions/postgresql#considerations)

The following considerations apply to the `postgresql()` table function:

- Tinybird doesn't support all PostgreSQL types directly, so some types are mapped to String, which is the most flexible type for arbitrary data.
- For the `NUMERIC` and `DECIMAL` types, `Decimal(p, s)` in Tinybird requires specifying precision (p) and scale (s).
- Time zone support in Tinybird's `DateTime` can be managed via additional functions or by ensuring consistent storage and retrieval time zones.
- Some types like `INTERVAL` don't have a direct equivalent in Tinybird and are usually stored as String or decomposed into separate fields.

## See also (https://www.tinybird.co/docs/forward/ingest-data/table-functions/postgresql#see-also)

- [Table functions](../table-functions)

---

URL: https://www.tinybird.co/docs/forward/ingest-data/table-functions/mysql
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "MySQL table function \| Tinybird"
theme-color: "#171612"
description: "Documentation for the Tinybird MySQL table function."
inkeep:version: "forward"
---

# MySQL table function (https://www.tinybird.co/docs/forward/ingest-data/table-functions/mysql#mysql-table-function)

Use the Tinybird `mysql()` table function to read data from your existing MySQL database into Tinybird, then schedule a regular Copy Pipe to orchestrate synchronization. You can load full tables, and every run performs a full replace on the Data Source.

To use it, define a node using standard SQL and the `mysql` function keyword, then publish the node as a copy pipe that does a sync on every run. See [Table functions](../table-functions) for general information and tips.

## Setting secrets (https://www.tinybird.co/docs/forward/ingest-data/table-functions/mysql#setting-secrets)

Table functions require authentication credentials that must be stored securely. In Tinybird Forward, manage those credentials with `tb secret`:

```
tb secret set MYSQL_USERNAME <username>
tb secret set MYSQL_PASSWORD <password>
```
Set the secret in each environment where the copy pipe runs. For example, use `tb --cloud secret set` for Cloud and `tb --branch=<branch_name> secret set` for a branch. Then reference the secrets in SQL with `tb_secret()`.

In Tinybird Classic, use the [Environment Variables API](/docs/api-reference/environment-variables-api) to create the same secret values.

For more details, see [tb secret](/docs/forward/dev-reference/commands/tb-secret).

## Syntax (https://www.tinybird.co/docs/forward/ingest-data/table-functions/mysql#syntax)

Create a new pipe node. Call the `mysql` table function and pass the hostname and port, database, table, user, and password:

Example query logic
```
SELECT *
FROM mysql(
  'mysql.example.com:3306',
  'mysql',
  'orders',
  {{ tb_secret("MYSQL_USERNAME") }},
  {{ tb_secret("MYSQL_PASSWORD") }}
)
```
Publish this node as a copy pipe. You can choose to append only new data or replace all data.

## Example: sync orders from MySQL (https://www.tinybird.co/docs/forward/ingest-data/table-functions/mysql#example-sync-orders-from-mysql)

The following example copies an `orders` table from MySQL into a Tinybird Data Source every 5 minutes.

First, define the target Data Source. Then define a copy pipe that reads from MySQL and appends rows updated in the last 10 minutes.

- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/mysql_orders.datasource
```
SCHEMA >
    `order_id` UInt64,
    `customer_id` UInt64,
    `status` String,
    `amount` Float64,
    `updated_at` DateTime

ENGINE "ReplacingMergeTree(updated_at)"
ENGINE_SORTING_KEY "order_id"
```
pipes/mysql_orders_sync.pipe
```
NODE updated_orders
SQL >
    %
    SELECT
        order_id,
        customer_id,
        status,
        amount,
        updated_at
    FROM mysql(
        'mysql.example.com:3306',
        'shop',
        'orders',
        {{ tb_secret("MYSQL_USERNAME") }},
        {{ tb_secret("MYSQL_PASSWORD") }}
    )
    WHERE updated_at >= now() - INTERVAL 10 MINUTE

TYPE copy
TARGET_DATASOURCE mysql_orders
COPY_MODE append
COPY_SCHEDULE */5 * * * *
```
Use `ReplacingMergeTree(updated_at)` when the source can send multiple versions of the same row. Query with `FINAL` when you need the latest version at read time.

## Type support and inference (https://www.tinybird.co/docs/forward/ingest-data/table-functions/mysql#type-support-and-inference)

Here's a detailed conversion table:

| MySQL data type | Tinybird data type |
| --- | --- |
| UNSIGNED TINYINT | UInt8 |
| TINYINT | Int8 |
| UNSIGNED SMALLINT | UInt16 |
| SMALLINT | Int16 |
| UNSIGNED INT, UNSIGNED MEDIUMINT | UInt32 |
| INT, MEDIUMINT | Int32 |
| UNSIGNED BIGINT | UInt64 |
| BIGINT | Int64 |
| FLOAT | Float32 |
| DOUBLE | Float64 |
| DATE | Date |
| DATETIME, TIMESTAMP | DateTime |
| BINARY | FixedString |

## Considerations (https://www.tinybird.co/docs/forward/ingest-data/table-functions/mysql#considerations)

The following considerations apply to the `mysql()` table function:

- Tinybird doesn't support all MySQL types directly, so some types are mapped to String, which is the most flexible type for arbitrary data.
- Time zone support in Tinybird's `DateTime` can be managed via additional functions or by ensuring consistent storage and retrieval time zones.

## See also (https://www.tinybird.co/docs/forward/ingest-data/table-functions/mysql#see-also)

- [Table functions](../table-functions)

---

URL: https://www.tinybird.co/docs/forward/ingest-data/table-functions/iceberg
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Apache Iceberg table function \| Tinybird"
theme-color: "#171612"
description: "Documentation for the Tinybird Iceberg table function."
inkeep:version: "forward"
---

# Iceberg table function (https://www.tinybird.co/docs/forward/ingest-data/table-functions/iceberg#iceberg-table-function)

Use the Tinybird `iceberg()` table function to read data from your existing Apache Iceberg database in S3 into Tinybird, then schedule a regular Copy Pipe to orchestrate synchronization. You can load full tables, and every run performs a full replace on the Data Source.

To use it, define a node using standard SQL and the `iceberg` function keyword, then publish the node as a copy pipe that does a sync on every run. See [Table functions](../table-functions) for general information and tips.

Additionally you can use the `iceberg` table function in API endpoints.

## Setting secrets (https://www.tinybird.co/docs/forward/ingest-data/table-functions/iceberg#setting-secrets)

Table functions require authentication credentials that must be stored securely. In Tinybird Forward, manage those credentials with `tb secret`:

```
tb secret set AWS_ACCESS_KEY_ID <access_key>
tb secret set AWS_SECRET_ACCESS_KEY <secret_key>
```
Set the secret in each environment where the copy pipe runs. For example, use `tb --cloud secret set` for Cloud and `tb --branch=<branch_name> secret set` for a branch. Then reference the secrets in SQL with `tb_secret()`.

In Tinybird Classic, use the [Environment Variables API](/docs/api-reference/environment-variables-api) to create the same secret values.

For more details, see [tb secret](/docs/forward/dev-reference/commands/tb-secret).

## Syntax (https://www.tinybird.co/docs/forward/ingest-data/table-functions/iceberg#syntax)

Create a new pipe node. Call the `iceberg` table function and pass the AWS access key and secret as Tinybird secrets:

Example query logic
```
SELECT *
FROM iceberg(
  's3://your_bucket/iceberg/db/table',
  {{ tb_secret("AWS_ACCESS_KEY_ID") }},
  {{ tb_secret("AWS_SECRET_ACCESS_KEY") }}
)
```
Publish this node as a copy pipe. You can choose to append only new data or replace all data.

Check a full working example in this [GitHub repository](https://github.com/tinybirdco/iceberg-tinybird)

## Example: sync an Iceberg table from S3 (https://www.tinybird.co/docs/forward/ingest-data/table-functions/iceberg#example-sync-an-iceberg-table-from-s3)

The following example copies an Apache Iceberg table stored in S3 into a Tinybird Data Source every hour.

First, define the target Data Source. Then define the copy pipe that reads from Iceberg.

- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/iceberg_orders.datasource
```
SCHEMA >
    `order_id` UInt64,
    `customer_id` UInt64,
    `status` String,
    `amount` Float64,
    `updated_at` DateTime

ENGINE "ReplacingMergeTree(updated_at)"
ENGINE_SORTING_KEY "order_id"
```
pipes/iceberg_orders_sync.pipe
```
NODE iceberg_orders
SQL >
    %
    SELECT
        order_id,
        customer_id,
        status,
        amount,
        updated_at
    FROM iceberg(
        's3://my-lakehouse/warehouse/orders',
        {{ tb_secret("AWS_ACCESS_KEY_ID") }},
        {{ tb_secret("AWS_SECRET_ACCESS_KEY") }}
    )

TYPE copy
TARGET_DATASOURCE iceberg_orders
COPY_MODE replace
COPY_SCHEDULE 0 * * * *
```
Use `COPY_MODE replace` when the Iceberg table is the source of truth and the full table is small enough to refresh on the schedule. For larger tables, filter by an update timestamp and use `COPY_MODE append` with a deduplicating engine.

## See also (https://www.tinybird.co/docs/forward/ingest-data/table-functions/iceberg#see-also)

- [How to effectively use table functions](../table-functions)

---

URL: https://www.tinybird.co/docs/forward/ingest-data/connectors/s3
Last update: 2026-08-10T12:09:01.000Z
Content:
---
title: "S3 connector \| Tinybird"
theme-color: "#171612"
description: "Learn how to configure the S3 connector for Tinybird."
inkeep:version: "forward"
---

# S3 connector (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#s3-connector)

You can set up an S3 connector to load your CSV, NDJSON, or Parquet files into Tinybird from any S3 bucket. Tinybird can detect new files in your buckets and ingest them automatically.

Setting up the S3 connector requires:

1. Configuring AWS [permissions](https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#aws-permissions) using [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html).
2. Defining an S3 Connection in your Tinybird project.
3. Defining a Data Source that uses this Connection.

## Environment considerations (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#environment-considerations)

Before setting up the connector, understand how it works in different environments.

### Cloud environment (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#cloud-environment)

In the Tinybird Cloud environment, Tinybird uses its own AWS account to assume the IAM role you create, allowing it to access your S3 bucket. When you deploy to your main Cloud Workspace, use `tb --cloud deploy` as usual.

When you test S3 connector Data Sources in a Cloud Branch, include `--with-connections` so Tinybird creates the connector data linkers in the branch:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb build --with-connections
```

### Local environment (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#local-environment)

Tinybird Local runs in a container. To use the S3 connector locally, pass your local AWS credentials to the container so Tinybird Local can assume the IAM role you specify in your Connection. See [Local environment details](https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#local-environment-details) for setup options.

When using the S3 connector in the `--local` environment, continuous file ingestion is limited. For continuous ingestion of new files, use the Cloud environment.

## AWS permissions (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#aws-permissions)

The S3 connector requires an IAM Role with specific permissions to access objects in your Amazon S3 bucket:

- `s3:GetObject`
- `s3:ListBucket`
- `s3:GetBucketNotification`
- `s3:PutBucketNotification`
- `s3:GetBucketLocation`
**One IAM role can access multiple buckets**: Update the access policy to include multiple buckets by adding their ARNs to the `Resource` array. Reuse the same IAM role across multiple S3 Connections for different buckets to simplify credential management.

You need to create both an access policy and a trust policy in AWS:

- AWS Access Policy
- AWS Trust Policy

```
{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "s3:GetObject",
          "s3:ListBucket",
          "s3:GetBucketNotification",
          "s3:PutBucketNotification",
          "s3:GetBucketLocation"
        ],
        "Resource": [
          "arn:aws:s3:::{bucket-name}",
          "arn:aws:s3:::{bucket-name}/*"
        ]
      }
    ]
}
```
For the AWS Trust Policy you need to replace placeholder values with values specific to your Tinybird environment:

- `{AWS_ACCOUNT_ID}`:

  - For Cloud environments: Tinybird's AWS account ID, which varies depending on your region and provider
  - For Local environments: The AWS account ID of the credentials you pass to the Docker container with `--use-aws-creds`
- `{EXTERNAL_ID}`: A unique identifier provided by Tinybird and generated from your connection name.
To get the correct values for your Trust Policy:

1. Use the guided CLI process with `tb connection create s3` (recommended)
2. Or access the API endpoint `/v0/integrations/s3/policies/trust-policy?external_id_seed={CONNECTION_NAME}` for your workspace
To allow access from both Local and Cloud environments with a single IAM role, add both account IDs to the `Principal.AWS` in array format: `["arn:aws:iam::{LOCAL_ACCOUNT_ID}:root", "arn:aws:iam::{CLOUD_ACCOUNT_ID}:root"]` in the Trust Policy. This is useful when you want to use the same IAM role for both environments to simplify credential management.

## Set up the connector (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#set-up-the-connector)

1
### Create an S3 connection (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#create-an-s3-connection)

Define the S3 Connection in your project. For Tinybird CLI datafile projects, `tb connection create s3` is a useful helper for setting up the required AWS permissions and generating a `.connection` file you can edit.

Run the guided CLI helper:

```
tb connection create s3
```
When prompted:

1. Enter a name for your Connection.
2. Specify whether you use this Connection for sinking or ingesting data.
3. Enter the S3 bucket name.
4. Enter the AWS region where your bucket is located.
5. Copy the displayed AWS IAM policy to your clipboard. Use it to set up permissions in AWS.
6. Copy the displayed AWS IAM role trust policy for your Local environment, then enter the ARN of the role you create.
7. Copy the displayed AWS IAM role trust policy for your Cloud environment, then enter the ARN of the role you create.
8. Tinybird stores the ARN values securely using [tb secret](/docs/forward/dev-reference/commands/tb-secret). You can use different roles for each environment.
You can also define the Connection manually:

- Tinybird CLI
- TypeScript SDK
- Python SDK
connections/s3_sample.connection
```
TYPE s3
S3_REGION "<S3_REGION>"
S3_ARN {{ tb_secret("S3_ARN") }}
```
When creating your connection manually, you need to set up the required AWS IAM role with appropriate permissions. See the [AWS permissions](https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#aws-permissions) section for details on the required access policy and trust policy configurations.

See [Connection files](/docs/forward/dev-reference/datafiles/connection-files) for more details on how to create a connection file and manage secrets.

You need to create separate connections for each environment you're working with, Local and Cloud.

For example, you can create:

- `my-s3-local` for your Local environment
- `my-s3-cloud` for your Cloud environment
2
### Create an S3 Data Source (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#create-an-s3-data-source)

After creating the Connection, create a Data Source that uses it. For Tinybird CLI datafile projects, `tb datasource create --s3` is a useful helper for generating the `.datasource` file.

```
tb datasource create --s3
```
Define the Data Source schema as with any other Data Source, then attach the S3 Connection. The connection name or object must match the Connection you created in the previous step.

- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/s3_sample.datasource
```
DESCRIPTION >
    Analytics events landing data source

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"
ENGINE_TTL "timestamp + toIntervalDay(60)"

IMPORT_CONNECTION_NAME s3_sample
IMPORT_BUCKET_URI s3://my-bucket/*.csv
IMPORT_SCHEDULE @auto
```
3
### Deploy (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#deploy)

After defining your S3 Data Source and Connection, test it by running a deploy check:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb --cloud deploy --check
```
This runs the Connection locally and checks if the Connection is valid. To see the Connection details, run `tb --cloud connection ls`.

When ready, push the datafile to your Workspace using `tb deploy` to create the S3 Data Source:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb --cloud deploy
```

## .connection settings (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#connection-settings)

The S3 connector uses the following settings in .connection files:

| Instruction | Required | Description |
| --- | --- | --- |
| `S3_REGION` | Yes | Region of the S3 bucket. |
| `S3_ARN` | Yes | ARN of the IAM role with the required permissions. |
Once a Connection is used in a Data Source, you can't change the ARN account ID or region. To modify these values, you must:

1. Remove the Connection from the Data Source.
2. Deploy the changes.
3. Add the Connection again with the new values.

## .datasource settings (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#datasource-settings)

The S3 connector uses the following settings in .datasource files:

| Instruction | Required | Description |
| --- | --- | --- |
| `IMPORT_SCHEDULE` | Yes | Use `@auto` to ingest new files automatically, or `@once` to execute only manually. In the `--local` environment, even if you set `@auto`, the Connector performs only the initial sync and loads all existing files. It doesn't continue to ingest new files automatically. |
| `IMPORT_CONNECTION_NAME` | Yes | Name given to the Connection inside Tinybird. For example, `'my_connection'`. This is the name of the connection file you created in the previous step. |
| `IMPORT_BUCKET_URI` | Yes | Full bucket path, including the `s3://` protocol, bucket name, object path, and an optional pattern to match against object keys. For example, `s3://my-bucket/my-path` discovers all files in the bucket `my-bucket` under the prefix `/my-path`. You can use patterns in the path to filter objects, for example, ending the path with `*.csv` matches all objects that end with the `.csv` suffix. |
| `IMPORT_FORMAT` | No | Format of the files to ingest: `csv`, `ndjson`, or `parquet`. Tinybird infers the format from the extension in `IMPORT_BUCKET_URI`, so you only need this setting when the bucket URI doesn't end with one of the [accepted extensions](https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#supported-file-types). |
| `IMPORT_FROM_TIMESTAMP` | No | Sets the date and time from which to start ingesting files on an S3 bucket. The format is `YYYY-MM-DDTHH:MM:SSZ`. |
The only supported change is updating `IMPORT_SCHEDULE` from `@once` to `@auto`, which makes the connector ingest all files that match the bucket URI pattern since the last on-demand ingestion.

For any other parameter changes, you must:

1. Remove the Connection from the Data Source.
2. Deploy the changes.
3. Add the Connection again with the new values.
4. Deploy again.

## Sync your data (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#sync-your-data)

With the `@on-demand` option for your `IMPORT_SCHEDULE`, you can trigger a **Sync now** action at any time. Run the `tb datasource sync <datasource_name>` command from the CLI. The command prompts for confirmation to sync the Data Source. Enter `y` to confirm. The Data Source then syncs data from its last synchronization point, preventing duplicates.

Be careful when using `IMPORT_SCHEDULE` with `@on-demand`. If you trigger a **Sync now** action while simultaneously uploading a large file to S3, a race condition may cause data loss.

When a file starts uploading, its `creation_time` is unset until the upload completes. If a sync runs during the upload, it doesn't process the file. When the upload completes, `creation_time` is set to the time when the file started uploading, so future syncs don't process it because its `creation_time` is earlier than when the last sync ran.

## Import sample data (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#import-sample-data)

In branches and Tinybird Local, you can import a sample of files from S3 using the API. This is useful for validating schemas and testing pipelines without syncing all files from the bucket.

```
curl -X POST "https://<your_host>/v0/datasources/my_datasource/sample" \
  -H "Authorization: Bearer $TB_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"max_files": 1}'
```
The sample import starts an asynchronous job that imports up to `max_files` files (maximum 10). The response includes a `job_id` that you can use to track progress:

```
curl "https://<your_host>/v0/jobs/{job_id}?token=$TB_TOKEN"
```
The sample import runs as a separate job and doesn't affect production sync state or offsets.

## S3 file URI (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#s3-file-uri)

The S3 connector supports the following wildcard patterns:

- Single asterisk or `*`: matches zero or more characters within a single directory level, excluding `/`. It doesn't cross directory boundaries. For example, `s3://bucket-name/*.ndjson` matches all `.ndjson` files in the root of your bucket but doesn't match files in subdirectories.
- Double asterisk or `**`: matches zero or more characters across multiple directory levels, including `/`. It can cross directory boundaries recursively. For example: `s3://bucket-name/**/*.ndjson` matches all `.ndjson` files in the bucket, regardless of their directory depth.
Use the full S3 file URI and wildcards to select multiple files. The file extension is required to accurately match the desired files in your pattern.

Due to a limitation in Amazon S3 bucket notifications, only one S3 data source with `IMPORT_SCHEDULE=@auto` can be configured per unique bucket URI pattern.

**What counts as a collision?**

Two URI patterns collide if one pattern would match files that the other pattern could also match. This happens when patterns share overlapping prefixes or wildcards.

**Examples of collisions:**

- `s3://my-bucket/stock_*.csv` collides with `s3://my-bucket/stock_prices*.csv` (first pattern would match files from the second)
- `s3://my-bucket/**/*.csv` collides with `s3://my-bucket/transactions/*.csv` (first pattern would match files from the second)
- `s3://my-bucket/*.csv` collides with `s3://my-bucket/export_*.csv` (first pattern would match files from the second)
**Examples that work (non-overlapping prefixes):**

- `s3://my-bucket/export_*.csv` and `s3://my-bucket/import_*.csv` (different prefixes)
- `s3://my-bucket/prod/*.csv` and `s3://my-bucket/staging/*.csv` (different directories)
- `s3://my-bucket/data/*.json` and `s3://my-bucket/data/*.csv` (different file extensions)

### Examples (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#examples)

The following examples show patterns you can use and whether they match the example path:

| File path | S3 File URI | Matches? |
| --- | --- | --- |
| example.ndjson | `s3://bucket-name/*.ndjson` | Yes. Matches files in the root directory with the `.ndjson` extension. |
| example.ndjson.gz | `s3://bucket-name/**/*.ndjson.gz` | Yes. Recursively matches `.ndjson.gz` files anywhere in the bucket. |
| example.ndjson | `s3://bucket-name/example.ndjson` | Yes. Exact match to the path. |
| pending/example.ndjson | `s3://bucket-name/*.ndjson` | No. `*` doesn't cross directory boundaries. |
| pending/example.ndjson | `s3://bucket-name/**/*.ndjson` | Yes. Recursively matches `.ndjson` files in any subdirectory. |
| pending/example.ndjson | `s3://bucket-name/pending/example.ndjson` | Yes. Exact match to the path. |
| pending/example.ndjson | `s3://bucket-name/pending/*.ndjson` | Yes. Matches `.ndjson` files within the `pending` directory. |
| pending/example.ndjson | `s3://bucket-name/pending/**/*.ndjson` | Yes. Recursively matches `.ndjson` files within `pending` and all its subdirectories. |
| pending/example.ndjson | `s3://bucket-name/**/pending/example.ndjson` | Yes. Matches the exact path to `pending/example.ndjson` within any preceding directories. |
| pending/example.ndjson | `s3://bucket-name/other/example.ndjson` | No. Doesn't match because the path includes directories which aren't part of the file's actual path. |
| pending/example.ndjson.gz | `s3://bucket-name/pending/*.csv.gz` | No. The file extension `.ndjson.gz` doesn't match `.csv.gz` |
| pending/o/inner/example.ndjson | `s3://bucket-name/*.ndjson` | No. `*` doesn't cross directory boundaries. |
| pending/o/inner/example.ndjson | `s3://bucket-name/**/*.ndjson` | Yes. Recursively matches `.ndjson` files anywhere in the bucket. |
| pending/o/inner/example.ndjson | `s3://bucket-name/**/inner/example.ndjson` | Yes. Matches the exact path to `inner/example.ndjson` within any preceding directories. |
| pending/o/inner/example.ndjson | `s3://bucket-name/**/ex*.ndjson` | Yes. Recursively matches `.ndjson` files starting with `ex` at any depth. |
| pending/o/inner/example.ndjson | `s3://bucket-name/**/**/*.ndjson` | Yes. Matches `.ndjson` files at any depth, even with multiple `**` wildcards. |
| pending/o/inner/example.ndjson | `s3://bucket-name/pending/**/*.ndjson` | Yes. Matches `.ndjson` files within `pending` and all its subdirectories. |
| pending/o/inner/example.ndjson | `s3://bucket-name/inner/example.ndjson` | No. Doesn't match because the path includes directories which aren't part of the file's actual path. |
| pending/o/inner/example.ndjson | `s3://bucket-name/pending/example.ndjson` | No. Doesn't match because the path includes directories which aren't part of the file's actual path. |
| pending/o/inner/example.ndjson.gz | `s3://bucket-name/pending/*.ndjson.gz` | No. `*` doesn't cross directory boundaries. |
| pending/o/inner/example.ndjson.gz | `s3://bucket-name/other/example.ndjson.gz` | No. Doesn't match because the path includes directories which aren't part of the file's actual path. |

### Considerations (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#considerations)

When using patterns:

- Use specific directory names or even specific file URIs to limit the scope of your search. The more specific your pattern, the narrower the search.
- Combine wildcards: you can combine `**` with other patterns to match files in subdirectories selectively. For example, `s3://bucket-name/**/logs/*.ndjson` matches `.ndjson` files within any logs directory at any depth.
- Avoid unintended matches: be cautious with `**` as it can match many files, which might impact performance and return partial matches.

## Supported file types (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#supported-file-types)

The S3 connector supports the following file types:

| File type | Accepted extensions | Compression formats supported |
| --- | --- | --- |
| CSV | `.csv`, `.csv.gz` | `gzip` |
| NDJSON | `.ndjson`, `.ndjson.gz`, `.jsonl`, `.jsonl.gz`, `.json`, `.json.gz` | `gzip` |
| Parquet | `.parquet`, `.parquet.gz` | `snappy`, `gzip`, `lzo`, `brotli`, `lz4`, `zstd` |
You can upload files with .json extension, provided they follow the Newline Delimited JSON (NDJSON) format. Each line must be a valid JSON object and every line has to end with a `\n` character.

Parquet schemas use the same format as NDJSON schemas, using [JSONPath](/docs/forward/dev-reference/datafiles/datasource-files#jsonpath-expressions) syntax.

## Best practices (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#best-practices)

Choose the right ingestion method based on your data volume and frequency requirements:

### Use S3 connector when: (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#use-s3-connector-when)

- **File size**: Your files are 1 GB or larger (minimum recommended)
- **Frequency**: You have less frequent batch ingestion needs
- **Format**: You're working with CSV, Parquet, or NDJSON files stored in S3
- **Use case**: Historical data loads, periodic batch processing, or large dataset ingestion

### Use Events API or Kafka connector when: (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#use-events-api-or-kafka-connector-when)

- **Frequency**: You need high-frequency, real-time data ingestion
- **Throughput**: You're sending up to 100 requests per second
- **File size**: You're working with smaller, frequent data updates
- **Use case**: Real-time analytics, streaming data, or event-driven architectures
The S3 connector uses Tinybird's <a href="/docs/api-reference/datasource-api">ingestion API (`/v0/datasources`)</a>, which isn't optimized for small, frequent inserts. For streaming use cases, consider:

- **[Events API](/docs/forward/ingest-data/events-api)**: Direct HTTP ingestion for real-time events
- **[Kafka connector](/docs/forward/ingest-data/connectors/kafka)**: For existing Kafka infrastructure

## Limits (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#limits)

The following limits apply to the S3 Connector:

- When using the `auto` mode, Tinybird automatically detects new files uploaded to the bucket, upserts included.
- Tinybird ingests a maximum of 5 files per minute by default. This is a Workspace-level limit, so it's shared across all Data Sources. See [Ingestion limits](/docs/forward/pricing/limits#ingestion-limits).
The following limits apply to file size per type and plan:

| File type | Free plan | Developer, SaaS, and Enterprise plans |
| --- | --- | --- |
| CSV | 10 GB | 32 GB |
| NDJSON | 10 GB | 32 GB |
| Parquet | 1 GB | 5 GB |

## Local environment details (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#local-environment-details)

When using the S3 connector in Tinybird Local, you need to pass your local AWS credentials to the container. These credentials must have the [permissions described in the AWS permissions section](https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#aws-permissions), including access to S3 operations like `GetObject` and `ListBucket`. This allows Tinybird Local to assume the IAM role you specify in your Connection.

To pass your AWS credentials, use the `--use-aws-creds` flag when starting Tinybird Local:

```
tb local start --use-aws-creds

» Starting Tinybird Local...
✓ AWS credentials found and will be passed to Tinybird Local (region: us-east-1)
* Waiting for Tinybird Local to be ready...
✓ Tinybird Local is ready!
```
If you're using a specific AWS profile, specify it using the `AWS_PROFILE` environment variable:

```
AWS_PROFILE=my-profile tb local start --use-aws-creds
```

### Docker Compose setup (https://www.tinybird.co/docs/forward/ingest-data/connectors/s3#docker-compose-setup)

If you're running Tinybird Local via Docker Compose instead of the CLI, pass AWS credentials using environment variables in your `docker-compose.yml`:

```
services:
  tinybird-local:
    image: tinybirdco/tinybird-local:latest
    container_name: tinybird-local
    platform: linux/amd64
    ports:
      - "7181:7181"
    environment:
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN}  # Optional, for temporary credentials
      - AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION} # Optional, add region if available
    volumes:
      - ./:/workspace
      - tinybird-data:/var/lib/tinybird

volumes:
  tinybird-data:
```
You can then start the container with your AWS credentials set as environment variables:

```
# Export your credentials
export AWS_ACCESS_KEY_ID="your-access-key-id"
export AWS_SECRET_ACCESS_KEY="your-secret-access-key"
export AWS_DEFAULT_REGION="us-east-1"

# Start Docker Compose
docker compose up -d
```
Alternatively, if you're using named AWS profiles, use the AWS CLI to export credentials:

```
# Export credentials from a named profile
eval "$(aws configure export-credentials --profile my-profile --format env)"

# Start Docker Compose
docker compose up -d
```

---

URL: https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Kafka connector \| Tinybird"
theme-color: "#171612"
description: "Set up Tinybird's Kafka connector for Confluent Cloud, AWS MSK, Redpanda, and Kafka-compatible platforms with authentication and tuning tips."
inkeep:version: "forward"
---

# Kafka connector (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#kafka-connector)

You can set up a Kafka connector to consume data from a Kafka topic and store it in Tinybird by creating a [.connection](/docs/forward/dev-reference/datafiles/connection-files) and [.datasource](/docs/forward/dev-reference/datafiles/datasource-files) file. Use `tb datasource create --kafka` for a guided Data Source setup.

## Set up the connector (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#set-up-the-connector)

To set up the Kafka connector, follow these steps.

1
### Create a Kafka connection (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#create-a-kafka-connection)

Define the Kafka Connection in your project. The examples use SASL/PLAIN, which is the common setup for managed Kafka providers like Confluent Cloud, Redpanda Cloud, and Aiven.

For Tinybird CLI datafile projects, `tb connection create kafka` is a useful helper. It prompts for the connection name, bootstrap servers, Kafka key, and Kafka secret, then creates a `.connection` file you can edit.

```
tb connection create kafka
```

- Tinybird CLI
- TypeScript SDK
- Python SDK
connections/kafka_connection.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS <BOOTSTRAP_SERVERS:PORT>
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("KAFKA_KEY", "key") }}
KAFKA_SECRET {{ tb_secret("KAFKA_SECRET", "secret") }}
```
Set the Kafka key and secret using [tb secret](/docs/forward/dev-reference/commands/tb-secret):

```
tb [--cloud] secret set KAFKA_KEY <KAFKA_KEY>
tb [--cloud] secret set KAFKA_SECRET <KAFKA_SECRET>
```
If you use the CLI helper and then move to an SDK project, use `tinybird migrate` to convert `.connection` files to TypeScript or Python definitions.

Secrets are only replaced in your resources when you deploy. If you change a secret, you need to deploy for the changes to take effect.

#### Authentication methods (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#authentication-methods)

Tinybird supports multiple authentication methods for Kafka Connections. The initial example uses SASL/PLAIN. Use the settings in this section when your Kafka cluster uses another authentication method.

##### SASL/SCRAM-SHA-256 or SCRAM-SHA-512

For Kafka clusters using SCRAM authentication:

- Tinybird CLI
- TypeScript SDK
- Python SDK
kafka_sasl_scram.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS <BOOTSTRAP_SERVERS:PORT>
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM SCRAM-SHA-256
KAFKA_KEY {{ tb_secret("KAFKA_KEY", "key") }}
KAFKA_SECRET {{ tb_secret("KAFKA_SECRET", "secret") }}
```
Replace `SCRAM-SHA-256` with `SCRAM-SHA-512` if your cluster uses SHA-512.

##### SASL/OAuthBearer (AWS MSK)

For AWS MSK clusters using IAM authentication:

- Tinybird CLI
- TypeScript SDK
- Python SDK
kafka_sasl_oauthbearer_aws.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS <BOOTSTRAP_SERVERS:PORT>
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM OAUTHBEARER
KAFKA_SASL_OAUTHBEARER_METHOD AWS
KAFKA_SASL_OAUTHBEARER_AWS_REGION <AWS_REGION>
KAFKA_SASL_OAUTHBEARER_AWS_ROLE_ARN {{ tb_secret("AWS_ROLE_ARN") }}
KAFKA_SASL_OAUTHBEARER_AWS_EXTERNAL_ID <AWS_EXTERNAL_ID>
```
When using AWS method, you need to set up the required AWS IAM role with appropriate permissions. See the [AWS IAM permissions](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#aws-iam-permissions) section for details on the required access policy and trust policy configurations.

Set the AWS role ARN using [tb secret](/docs/forward/dev-reference/commands/tb-secret):

```
tb [--cloud] secret set AWS_ROLE_ARN <AWS_ROLE_ARN>
```
For detailed setup instructions for specific vendors, see:

- [Confluent Cloud setup guide](/docs/forward/guides/confluent-cloud-setup)
- [AWS MSK setup guide](/docs/forward/guides/aws-msk-setup)
- [Redpanda setup guide](/docs/forward/guides/redpanda-setup)

##### PLAINTEXT (local development only)

For local development with unsecured Kafka:

- Tinybird CLI
- TypeScript SDK
- Python SDK
kafka_plaintext.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS localhost:9092
KAFKA_SECURITY_PROTOCOL PLAINTEXT
```
PLAINTEXT connections are only suitable for local development. Production Kafka clusters should use SASL_SSL.

For more information on local development setup, see the [local development guide](/docs/forward/ingest-data/connectors/kafka/guides/local-development).

2
### Bootstrap servers configuration (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#bootstrap-servers-configuration)

The `KAFKA_BOOTSTRAP_SERVERS` setting specifies one or more Kafka brokers to connect to. Use comma-separated values for multiple brokers:

```
KAFKA_BOOTSTRAP_SERVERS broker1:9092,broker2:9092,broker3:9092
```
**Important considerations:**

- Use the advertised listeners address, not the internal broker address
- For cloud providers, use the public endpoint provided by your Kafka service
- Ensure the port matches your security protocol (9092 for PLAINTEXT, 9093-9096 for SASL_SSL depending on provider)
- For AWS MSK, use the bootstrap broker string from the MSK console
If you encounter connection issues, see the [troubleshooting guide](/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-connection-timeout-or-broker-unreachable) for bootstrap server configuration help.

3
### SSL/TLS certificate configuration (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#ssltls-certificate-configuration)

When using `SASL_SSL` security protocol, you may need to provide a CA certificate, especially for:

- Self-signed certificates
- Private CA certificates
- Aiven Kafka (Private CA port)
The recommended way to provide the certificate is using secrets.

Create the multiline secret with the certificate using [tb secret](/docs/forward/dev-reference/commands/tb-secret#tb-secret-set). This opens an editor to introduce the value.

```
tb [--cloud] secret set --multiline KAFKA_PROD_SSL_CA_PEM
```
Use the secret in the Kafka connection. In `.connection` files, `KAFKA_SSL_CA_PEM` is a [multiline setting](/docs/forward/dev-reference/datafiles#multiple-lines).

- Tinybird CLI
- TypeScript SDK
- Python SDK
/connections/kafka_ssl_ca_pem.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS {{ tb_secret("KAFKA_PROD_SERVER", "kafka:29092") }}
KAFKA_SECURITY_PROTOCOL {{ tb_secret("KAFKA_PROD_SECURITY_PROTOCOL", "PLAINTEXT") }}
KAFKA_SASL_MECHANISM {{ tb_secret("KAFKA_PROD_SASL_MECHANISM", "PLAIN") }}
KAFKA_KEY {{ tb_secret("KAFKA_PROD_KEY", "key") }}
KAFKA_SECRET {{ tb_secret("KAFKA_PROD_SECRET", "secret") }}
KAFKA_SSL_CA_PEM >
  {{ tb_secret("KAFKA_PROD_SSL_CA_PEM") }}
```
**Certificate format requirements:**

- Certificate must be in PEM format (starts with `-----BEGIN CERTIFICATE-----`)
- Include the full certificate chain if required
- For Aiven Kafka, download the CA certificate from the Aiven console
For troubleshooting SSL certificate issues, see the [troubleshooting guide](/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-ssltls-certificate-validation-failed).

4
### Create a Kafka data source (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#create-a-kafka-data-source)

Create the Kafka Data Source in your project. For Tinybird CLI datafile projects, `tb datasource create --kafka` is a useful helper for generating the `.datasource` file.

```
tb datasource create --kafka
```
Define the Data Source schema as with any non-Kafka Data Source, then attach the Kafka connection. The connection name or object must match the connection you created in the previous step.

- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/kafka_sample.datasource
```
SCHEMA >
   `timestamp` DateTime `json:$.timestamp`,
   `session_id` String `json:$.session_id`,
   `action` LowCardinality(String) `json:$.action`,
   `version` LowCardinality(String) `json:$.version`,
   `payload` String `json:$.payload`

KAFKA_CONNECTION_NAME kafka_connection
KAFKA_TOPIC test_topic
KAFKA_GROUP_ID {{ tb_secret("KAFKA_GROUP_ID") }}
```
Set the Kafka consumer group ID using [tb secret](/docs/forward/dev-reference/commands/tb-secret):

```
tb [--cloud] secret set KAFKA_GROUP_ID <KAFKA_GROUP_ID>
```
In `.datasource` files, use [JSONPath expressions](/docs/forward/dev-reference/datafiles/datasource-files#jsonpath-expressions) to extract message fields into separate columns at ingest time. To store the whole message and parse it at query time, add a single `data` column with `json:$` and then read fields with [JSONExtract functions](/docs/sql-reference/functions/json-functions#jsonextract-functions).

In addition to the columns specified in `SCHEMA`, Kafka Data Sources have additional columns that store metadata of the messages ingested. See [Kafka meta columns](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#kafka-meta-columns) for more information.

For a complete list of Kafka data source settings, see [Kafka .datasource settings](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#kafka-datasource-settings).

Use different consumer group values for `KAFKA_GROUP_ID` at different environments to isolate consumers and their committed offset.

If you use [Kafka Tombstones](https://docs.confluent.io/kafka/design/log_compaction.html#compaction-enables-deletes), Tinybird sends these `null` -payload messages to the Quarantine Data Source if your Data Source's schema can't accept them. You can inspect the tombstone records without affecting your main Data Source.

5
### Connectivity check (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#connectivity-check)

After defining your Kafka data source and connection, test the connection and preview data:

```
tb connection data <connection_name>
```
This command prompts you to:

1. Select a Kafka topic
2. Enter a consumer group ID
3. Returns preview data from the topic
This validates that Tinybird can connect to your Kafka broker, authenticate, and consume messages. Remember to set any secrets used by the connection.

## Compatibility (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#compatibility)

The connector is compatible with Apache Kafka and works with any compatible implementation and vendor. The following are tried and tested:

- Apache Kafka
- Confluent Platform and Confluent Cloud Platform
- Redpanda
- AWS MSK
- Azure Event Hubs for Apache Kafka
- Estuary
For vendor-specific setup guides, see:

- [Confluent Cloud setup](/docs/forward/guides/confluent-cloud-setup)
- [AWS MSK setup](/docs/forward/guides/aws-msk-setup)
- [Redpanda setup](/docs/forward/guides/redpanda-setup)

## Schema management and evolution (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#schema-management-and-evolution)

Tinybird supports multiple serialization formats for Kafka messages:

- **JSON** (`json_without_schema` or `json_with_schema`)
- **Avro** (requires Schema Registry)
- **JSON Schema** (requires Schema Registry)
When using Schema Registry, Tinybird automatically handles schema evolution for backward-compatible changes. For detailed information on schema management, including adding fields, handling nullable types, and data type mapping, see the [schema management guide](/docs/forward/ingest-data/connectors/kafka/guides/schema-management).

### Message size limits (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#message-size-limits)

Tinybird has a default message size limit of 10 MB. Messages exceeding this limit are sent to the Quarantine Data Source. For strategies on handling large messages and troubleshooting quarantined messages, see the [message size handling guide](/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling).

## Kafka .datasource settings (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#kafka-datasource-settings)

| Instruction | Required | Description |
| --- | --- | --- |
| `KAFKA_CONNECTION_NAME` | Yes | Name of the configured Kafka connection in Tinybird. It must match the name of the connection file (without the extension). |
| `KAFKA_TOPIC` | Yes | Name of the Kafka topic to consume from. |
| `KAFKA_GROUP_ID` | Yes | Consumer Group ID to use when consuming from Kafka. |
| `KAFKA_AUTO_OFFSET_RESET` | No | Offset to use when no previous offset can be found, like when creating a new consumer. Supported values are `latest` and `earliest`. Default: `latest`. |
| `KAFKA_STORE_HEADERS` | No | Adds a `__headers Map(String, String)` column to the data source, and stores Kafka headers in it for later processing. Default value is `False`. |
| `KAFKA_STORE_RAW_VALUE` | No | Stores the raw message in its entirety in the `__value` column. Default: `False`. |
| `KAFKA_KEY_FORMAT` | No | Format of the message key. Valid values are `avro`, `json_with_schema`, and `json_without_schema`. Using `avro` or `json_with_schema` requires `KAFKA_SCHEMA_REGISTRY_URL` to be set in the connection file used by the data source. |
| `KAFKA_VALUE_FORMAT` | No | Format of the message value. Valid values are `avro`, `json_with_schema`, and `json_without_schema`. Using `avro` or `json_with_schema` requires `KAFKA_SCHEMA_REGISTRY_URL` to be set in the connection file used by the data source. |

## Kafka .connection settings (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#kafka-connection-settings)

| Instruction | Required | Description |
| --- | --- | --- |
| `KAFKA_BOOTSTRAP_SERVERS` | Yes | Comma-separated list of one or more Kafka brokers, including Port numbers. |
| `KAFKA_SECURITY_PROTOCOL` | Yes | Security protocol for the connection. Accepted values are `PLAINTEXT` and `SASL_SSL`. Default value is `SASL_SSL`. |
| `KAFKA_SASL_MECHANISM` | No | SASL mechanism to use for authentication. Supported values are `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, `OAUTHBREARER`. Default value is `PLAIN`. |
| `KAFKA_KEY` | When using SASL `PLAIN` or `SCRAM` | Sometimes called Key, Client Key, or Username depending on the Kafka distribution. |
| `KAFKA_SECRET` | When using SASL `PLAIN` or `SCRAM` | Sometimes called Secret, Secret Key, or Password depending on the Kafka distribution. |
| `KAFKA_SASL_OAUTHBEARER_METHOD` | When using SASL `OAUTHBEARER` | Currently only `AWS` is supported. |
| `KAFKA_SASL_OAUTHBEARER_AWS_REGION` | When using SASL `OAUTHBEARER` | AWS MSK cluster region. |
| `KAFKA_SASL_OAUTHBEARER_AWS_ROLE_ARN` | When using SASL `OAUTHBEARER` | AWS role ARN to assume. |
| `KAFKA_SASL_OAUTHBEARER_AWS_EXTERNAL_ID` | When using SASL `OAUTHBEARER` | External ID used when assuming the role. |
| `KAFKA_SCHEMA_REGISTRY_URL` | No | URL of the Kafka schema registry. Used for `avro` and `json_with_schema` deserialization of keys and values. If Basic Auth is required, include credentials in the URL format: `https://<username>:<password>@<registry_host>` |
| `KAFKA_SSL_CA_PEM` | No | Content of the CA certificate in PEM format for SSL connections. [Multiline setting](/docs/forward/dev-reference/datafiles#multiple-lines) |

## AWS IAM permissions (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#aws-iam-permissions)

To grant Tinybird access to your Kafka cluster, you must create an AWS IAM role with two policies: an access policy and a trust policy.

### Manual configuration (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#manual-configuration)

- AWS Access Policy
- AWS Trust Policy

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeCluster"
            ],
            "Resource": "arn:aws:kafka:<MSK_CLUSTER_REGION>:<MSK_CLUSTER_ACCOUNT_ID>:cluster/<MSK_CLUSTER_NAME>/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:CreateTopic",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData"
            ],
            "Resource": "arn:aws:kafka:<MSK_CLUSTER_REGION>:<MSK_CLUSTER_ACCOUNT_ID>:topic/<MSK_CLUSTER_NAME>/*/<KAFKA_TOPIC>"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": "arn:aws:kafka:<MSK_CLUSTER_REGION>:<MSK_CLUSTER_ACCOUNT_ID>:group/<MSK_CLUSTER_NAME>/*/<KAFKA_GROUP>"
        }
    ]
}
```

- `<MSK_CLUSTER_REGION>`: AWS region where the MSK cluster is located (for example, `us-east-1`).
- `<MSK_CLUSTER_ACCOUNT_ID>`: AWS account identifier of the MSK cluster.
- `<MSK_CLUSTER_NAME>`: Name of the MSK cluster (for example, `demo-cluster-1`).
- `<KAFKA_TOPIC>`: Name of the Kafka topic to give read access. Use `*` to allow all topics.
- `<KAFKA_GROUP>`: Name of the consumer group to give read access. Use `*` to allow all groups.
- `<TINYBIRD_ACCOUNT_ID>`:
  - For Tinybird Cloud environments: Tinybird's AWS account ID, which varies depending on your region and provider
  - For Local environments: The AWS account ID of the credentials you pass to the Docker container with `--use-aws-creds`
- `<EXTERNAL_ID>`: A unique identifier for your connection.

### Get the configuration from the API (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#get-the-configuration-from-the-api)

Check [current Tinybird regions](/docs/api-reference#current-tinybird-regions) to see the available regions and their corresponding API endpoints.

- Access Policy: `/v0/integrations/kafka/policies/read-access-policy`
  - Use the `msk_cluster_arn` paramter to limit the access to only one MSK cluster.
  - Use the `topics` parameter to limit the access to a list topics (comma separated).
  - Use the `groups` parameter to limit the access to a list of consumer groups (comma separated).
- Trust Policy: `/v0/integrations/kafka/policies/trust-policy`
  - Use the connection name in the `external_id_seed` parameter

## Kafka connector in the local environment (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#kafka-connector-in-the-local-environment)

You can use the Kafka connector in [Tinybird Local](/docs/forward/core-concepts/tinybird-local) to consume messages from a local Kafka server or a Kafka server in the cloud.

For detailed local development setup instructions, including Docker Compose examples and environment management, see the [local development guide](/docs/forward/ingest-data/connectors/kafka/guides/local-development).

### Local Kafka server with Docker Compose (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#local-kafka-server-with-docker-compose)

When using a local Kafka server, ensure the Tinybird Local container can access it. If you are running Kafka using Docker, Docker Compose is the best option to set up both Kafka and Tinybird Local in the same network. Here's an example using `apache/kafka`:

```
networks:
  kafka_network:
    driver: bridge

volumes:
  kafka-data:

services:

  tinybird-local:
    image: tinybirdco/tinybird-local:latest
    container_name: tinybird-local
    platform: linux/amd64
    ports:
      - "7181:7181"
    networks:
      - kafka_network

  kafka:
    image: apache/kafka:latest
    hostname: broker
    container_name: broker
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_PROCESS_ROLES: "broker,controller"
      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@broker:29093"
      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,PLAINTEXT_HOST://0.0.0.0:9092,CONTROLLER://0.0.0.0:29093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
    volumes:
      - kafka-data:/var/lib/kafka/data
    networks:
      - kafka_network
```

### Network configuration (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#network-configuration)

The key points about the network configuration:

1. The example uses a bridge network (`kafka_network`) to allow communication between containers
2. The Kafka service exposes ports for both internal container communication and external access
3. Tinybird Local connects to Kafka using the internal network address
4. The bootstrap servers address in your Kafka Connection should match the `KAFKA_ADVERTISED_LISTENERS` in the `docker-compose.yml` file (for example, `kafka:29092`)

### Creating the Kafka connection and data source (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#creating-the-kafka-connection-and-data-source)

The following examples use default secret values suitable for this local setup. When deploying to Tinybird Cloud, set these secrets in the Tinybird Cloud environment instead.

- Tinybird CLI
- TypeScript SDK
- Python SDK
/connections/kafka_conn.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS {{ tb_secret("KAFKA_PROD_SERVER", "kafka:29092") }}
KAFKA_SECURITY_PROTOCOL {{ tb_secret("KAFKA_PROD_SECURITY_PROTOCOL", "PLAINTEXT") }}
KAFKA_SASL_MECHANISM {{ tb_secret("KAFKA_PROD_SASL_MECHANISM", "PLAIN") }}
KAFKA_KEY {{ tb_secret("KAFKA_KEY", "key") }}
KAFKA_SECRET {{ tb_secret("KAFKA_SECRET", "secret") }}
```
datasources/kafka_ds.datasource
```
SCHEMA >
    `data` String `json:$`

KAFKA_CONNECTION_NAME kafka_conn
KAFKA_TOPIC sample-topic
KAFKA_GROUP_ID my_group_id
```

### Usage example (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#usage-example)

1
#### Start the Docker containers (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#start-the-docker-containers)

```
docker compose up
```
2
#### Create the `sample-topic` (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#create-the)

```
docker exec -it broker /opt/kafka/bin/kafka-topics.sh --create --topic sample-topic --bootstrap-server localhost:9092

# Created topic sample-topic.
```
3
#### Deploy the project (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#deploy-the-project)

```
tb deploy
# Running against Tinybird Local

# * Changes to be deployed:
# ------------------------------------------------------------------------
# | status | name       | type       | path                              |
# ------------------------------------------------------------------------
# | new    | kafka_ds   | datasource | datasources/kafka_ds.datasource   |
# | new    | kafka_conn | connection | connections/kafka_conn.connection |
# ------------------------------------------------------------------------
# * No changes in tokens to be deployed

# Deployment URL: http://cloud.tinybird.co/local/7181/None/deployments/1

# * Deployment submitted
# » Waiting for deployment to be ready...
# ✓ Deployment is ready
# » Removing old deployment
# ✓ Old deployment removed
# » Waiting for deployment to be promoted...
# ✓ Deployment #1 is live!
# A deployment with no data is useless. Learn how to ingest at https://www.tinybird.co/docs/forward/ingest-data
```
4
#### Send data to the topic and query it (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#send-data-to-the-topic-and-query-it)

```
echo '{"data": "test"}' | docker exec -i broker /opt/kafka/bin/kafka-console-producer.sh --topic sample-topic --bootstrap-server localhost:9092

tb sql "select * from kafka_ds"
# Running against Tinybird Local
#   data               __value   __topic                  __partition   __offset   __timestamp           __key
#   String             String    LowCardinality(String)         Int16      Int64   DateTime              String
# ───────────────────────────────────────────────────────────────────────────────────────────────────────────────
#   {"data": "test"}             sample-topic                       0          1   2025-06-20 12:17:49
```

### Docker Compose troubleshooting (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#docker-compose-troubleshooting)

If you encounter connection issues:

1. Ensure all containers are running: `docker-compose ps`
2. Check container logs: `docker-compose logs kafka`
3. Ensure the bootstrap servers address in your Connection file matches the `KAFKA_ADVERTISED_LISTENERS` value in your `docker-compose.yml` file.

## Kafka meta columns (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#kafka-meta-columns)

When you connect a data source to Kafka, the following columns are added to store metadata from Kafka messages:

| name | type | description |
| --- | --- | --- |
| `__value` | `String` | A String representing the entire unparsed value of the Kafka message. It is only populated if `KAFKA_STORE_RAW_VALUE` is set to `True`. |
| `__topic` | `LowCardinality(String)` | The topic that the message was read from. |
| `__partition` | `Int16` | The partition that the message was read from. |
| `__offset` | `Int16` | The offset of the message. |
| `__timestamp` | `Datetime` | The timestamp of the message. |
| `__key` | `String` | The key of the message. |
Optionally, when `KAFKA_STORE_HEADERS` is set to `True`, the following column is added and populated:

| name | type | description |
| --- | --- | --- |
| `__headers` | `Map(String, String)` | Kafka headers of the message. |
When you iterate your Kafka data source, you might need to use the meta columns in the [FORWARD_QUERY](/docs/forward/guides/evolve-data-source#forward-query). Tinybird suggests a valid forward query that you can tweak to get the desired values for each column.

## Kafka logs (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#kafka-logs)

You can find global logs in the `datasources_ops_log` [Service Data Source](/docs/forward/monitoring/service-datasources#tinybird-datasources-ops-log). Filter by `datasource_id` to select the correct Data Source, and by `event_type='append-kafka'`.

For example, to select all Kafka related logs in the last day, run the following query:

```
SELECT *
FROM tinybird.datasources_ops_log
WHERE datasource_id = 't_1234'
  AND event_type = 'append-kafka'
  AND timestamp > now() - INTERVAL 1 day
ORDER BY timestamp DESC
```
If you can't find logs in `datasources_ops_log`, the `kafka_ops_log` [Service Data Source](/docs/forward/monitoring/service-datasources#tinybird-kafka-ops-log) contains more detailed logs. Filter by `datasource_id` to select the correct Data Source, and use `msg_type` to select the desired log level (`info`, `warning`, or `error`).

```
SELECT *
FROM tinybird.kafka_ops_log
WHERE datasource_id = 't_1234'
  AND timestamp > now() - interval 1 day
  AND msg_type IN ['info', 'warning', 'error']
```
When your Data Source encounters a high rate of specific Kafka Connection errors, Tinybird temporarily stops ingestion for exponentially increasing periods. This mechanism is called a Circuit Breaker. It prevents repeated errors from filling the logs and activates only for the errors listed in [this section](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#kafka-circuit-breaker-errors).

For example, if your Kafka brokers are unreachable, your logs show a message similar to this: "We have opened a circuit breaker because your error rate is too high. Please, fix your connection and if you think everything is fine, contact support."

If the connection issue persists through all retries, ingestion is paused. Tinybird automatically retries every 12 hours. If a retry succeeds, the Circuit Breaker resets and ingestion resumes from where it left off. After you fix the underlying connection problem, you can also run `tb deploy` to retry before the next automatic retry window.

### Kafka circuit breaker errors (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#kafka-circuit-breaker-errors)

The following Kafka errors trigger the Circuit Breaker.

| Error | Error message |
| --- | --- |
| `_ALL_BROKERS_DOWN` | Local: All broker connections are down |
| `_TRANSPORT` | Local: Broker transport failure |
| `_TIMED_OUT` | Local: Timed out |
| `_DESTROY` | Local: Broker handle destroyed |
| `_AUTHENTICATION` | Local: Authentication failure |
| `_RESOLVE` | Local: Host resolution failure |
| `_UNKNOWN_TOPIC` | Local: Unknown topic |
| `_UNKNOWN_PARTITION` | Local: Unknown partition |
| `SASL_AUTHENTICATION_FAILED` | Broker: SASL Authentication failed |
| `TOPIC_AUTHORIZATION_FAILED` | Broker: Topic authorization failed |
| `GROUP_AUTHORIZATION_FAILED` | Broker: Group authorization failed |
| `CLUSTER_AUTHORIZATION_FAILED` | Broker: Cluster authorization failed |
For comprehensive monitoring of your Kafka connectors, including lag tracking, throughput analysis, and error monitoring, see [Monitor Kafka connectors](/docs/forward/monitoring/kafka-clickhouse-monitoring).

## Pause and resume ingestion (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#pause-and-resume-ingestion)

In branches and Tinybird Local, you can pause and resume Kafka ingestion using the CLI:

```
tb [--branch=BRANCH_NAME] datasource stop my_kafka_datasource
tb [--branch=BRANCH_NAME] datasource start my_kafka_datasource
```
Stopping a Data Source pauses ingestion from the Kafka topic. When you start it again, the behavior depends on the environment. In branches, a new consumer group is created and ingestion starts from the latest offset. In Tinybird Local, ingestion resumes from the last committed offset.

Because each start creates a new consumer group, previous consumer groups become orphaned. Depending on your Kafka cluster's consumer group TTL (`offsets.retention.minutes`), these orphan groups may persist until they expire.

See [tb datasource start](/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-start) and [tb datasource stop](/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-stop) for more details.

## Troubleshooting (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#troubleshooting)

For comprehensive troubleshooting guidance, see the [Kafka connector troubleshooting guide](/docs/forward/ingest-data/connectors/kafka/troubleshooting).

Common issues include:

- Connection timeout or broker unreachable
- Authentication failures
- SSL/TLS certificate validation errors
- Deserialization errors (Avro, JSON)
- Offset and consumer group conflicts
- Consumer lag issues
- Schema mismatches
Each combination of `KAFKA_TOPIC` and `KAFKA_GROUP_ID` can only be used in one data source, otherwise the offsets committed by the consumers of different data sources clash.

If you connect a data source to Kafka using a `KAFKA_TOPIC` and `KAFKA_GROUP_ID` that were previously used by another data source in your workspace, the data source only receives data from the last committed offset, even if `KAFKA_AUTO_OFFSET_RESET` is set to `earliest`.

To prevent these issues, always use unique `KAFKA_GROUP_ID` s when testing Kafka data sources.

See [Kafka logs](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#kafka-logs) to learn how to diagnose any other issues

### Compressed messages (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#compressed-messages)

Tinybird can consume from Kafka topics where Kafka compression is turned on; decompressing the message is a standard function of the Kafka consumer. However, if you compressed the message before passing it through the Kafka producer, Tinybird can't do post-consumer processing to decompress the message.

For example, if you compressed a JSON message through gzip and produced it to a Kafka topic as a `bytes` message, it would be ingested by Tinybird as `bytes`. If you produced a JSON message to a Kafka topic with the Kafka producer setting `compression.type=gzip`, while it would be stored in Kafka as compressed bytes, it would be decoded on ingestion and arrive to Tinybird as JSON.

## Connecting an existing data source to Kafka (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#connecting-an-existing-data-source-to-kafka)

You can connect an existing, default data source to Kafka.

Create the Kafka .connection file if it does not exist, add the desired Kafka settings to the .datasource file, and add a <a href="/docs/forward/guides/evolve-data-source#forward-query">`FORWARD_QUERY`</a> to provide default values for the [Kafka meta columns](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#kafka-meta-columns).

## Disconnecting a data source from Kafka (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#disconnecting-a-data-source-from-kafka)

To disconnect a data source from Kafka, remove the Kafka settings from the .datasource file.

If you want to keep any of the [Kafka meta columns](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#kafka-meta-columns), add them to the schema with a default value and adjust the `FORWARD_QUERY` accordingly.

## Additional resources (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#additional-resources)

### Guides (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#guides)

- [Confluent Cloud setup](/docs/forward/guides/confluent-cloud-setup) - End-to-end setup for Confluent Cloud
- [AWS MSK setup](/docs/forward/guides/aws-msk-setup) - Complete MSK configuration guide
- [Redpanda setup](/docs/forward/guides/redpanda-setup) - Redpanda-specific configuration
- [Local development](/docs/forward/ingest-data/connectors/kafka/guides/local-development) - Local development setup and testing
- [CI/CD and version control](/docs/forward/ingest-data/connectors/kafka/guides/cicd-version-control) - Managing connections across environments
- [Schema management](/docs/forward/ingest-data/connectors/kafka/guides/schema-management) - Schema evolution and data type mapping
- [Partitioning strategies](/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies) - Best practices for Kafka partitioning
- [Message size handling](/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling) - Handling large messages and troubleshooting
- [Performance optimization](/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization) - Throughput optimization and tuning

### Reference (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#reference)

- [Troubleshooting guide](/docs/forward/ingest-data/connectors/kafka/troubleshooting) - Comprehensive error resolution with error lookup table
- [Limits and quotas](/docs/forward/ingest-data/connectors/kafka/limits) - Kafka connector limits and how to request increases

### Related documentation (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka#related-documentation)

- [Monitor Kafka connectors](/docs/forward/monitoring/kafka-clickhouse-monitoring) - Monitoring queries and alerts
- [Kafka Sink](/docs/forward/copy-export-data/kafka-sink) - Exporting data to Kafka
- [Quarantine Data Sources](/docs/forward/guides/quarantine) - Handling failed messages

---

URL: https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs
Last update: 2026-08-10T12:09:01.000Z
Content:
---
title: "GCS Connector \| Tinybird"
theme-color: "#171612"
description: "Learn how to configure the GCS connector for Tinybird."
inkeep:version: "forward"
---

# GCS connector (https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#gcs-connector)

You can set up a GCS connector to load your CSV, NDJSON, or Parquet files into Tinybird from any GCS bucket. Tinybird automatically ingests matching files on the first deployment, but does **not** detect new files afterwards. You must trigger subsequent ingestion manually.

Setting up the GCS connector requires:

1. Configuring a [Service Account](https://cloud.google.com/iam/docs/service-accounts-create) with these [permissions](https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#gcs-permissions) in GCP.
2. Defining a GCS Connection in your Tinybird project.
3. Defining a Data Source that uses this Connection.

## Environment considerations (https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#environment-considerations)

In the Tinybird Cloud environment, Tinybird uses the Service Account credentials you provide to access your GCS bucket. When you deploy to your main Cloud Workspace, use `tb --cloud deploy` as usual.

When you test GCS connector Data Sources in a Cloud Branch, include `--with-connections` so Tinybird creates the connector data linkers in the branch:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb build --with-connections
```
In branches and Tinybird Local, use sample imports to validate schemas and pipelines without syncing every matching file. See [Import sample data](https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#import-sample-data).

## GCS permissions (https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#gcs-permissions)

To authenticate Tinybird with GCS, you need a GCP service account key in JSON format with the **Object Storage Viewer** role.

1. In the Google Cloud Console, create or use an existing service account.
2. Assign the `roles/storage.objectViewer` role.
3. Generate a JSON key file and download it.
4. Store the key as a Tinybird secret in a `.env.local` file to work in local:

```
GCS_KEY='<your-json-key-content>'
```

1. Store the key in Cloud as a Tinybird secret:

```
tb --cloud secret set GCS_KEY '<your-json-key-content>'
```

## Set up the connector (https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#set-up-the-connector)

1
### Create a GCS connection (https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#create-a-gcs-connection)

Define the GCS Connection in your project. For Tinybird CLI datafile projects, `tb connection create gcs` is a useful helper for generating a `.connection` file you can edit.

Run the following command to create a connection:

```
tb connection create gcs
```
The CLI prompts you to enter:

1. A name for your Connection.
2. The GCS bucket name.
3. The service account credentials (JSON key file). You can check [Google Cloud docs](https://cloud.google.com/iam/docs/keys-create-delete) for mode details.
4. Whether to create the connection for your Cloud environment.
You can also define the Connection manually:

- Tinybird CLI
- TypeScript SDK
- Python SDK
connections/gcs_sample.connection
```
TYPE gcs
GCS_SERVICE_ACCOUNT_CREDENTIALS_JSON {{ tb_secret("GCS_KEY") }}
```
Ensure your GCP Service Account has the `roles/storage.objectViewer` role.

Use different Service Account keys for each environment leveraging [Tinybird Secrets](/docs/forward/dev-reference/commands/tb-secret).

2
### Create a GCS Data Source (https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#create-a-gcs-data-source)

After setting up the Connection, create a Data Source that uses it. For Tinybird CLI datafile projects, `tb datasource create --gcs` is a useful helper for generating the `.datasource` file.

```
tb datasource create --gcs
```
Define the Data Source schema as with any other Data Source, then attach the GCS Connection. The connection name or object must match the Connection you created in the previous step.

- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/gcs_sample.datasource
```
DESCRIPTION >
    Analytics events landing data source

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"
ENGINE_TTL "timestamp + toIntervalDay(60)"

IMPORT_CONNECTION_NAME gcs_sample
IMPORT_BUCKET_URI gs://my-bucket/*.csv
IMPORT_SCHEDULE '@on-demand'
```
3
### Sync data (https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#sync-data)

On the first deployment, Tinybird automatically ingests all files that match the `IMPORT_BUCKET_URI` pattern. `@auto` mode is not supported, so you must manually trigger subsequent syncs to ingest new files.

To trigger a manual sync, use the API or the CLI.

#### Using the API (https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#using-the-api)

```
curl -X POST "https://<your_host>/v0/datasources/<datasource_name>/scheduling/runs" \
  -H "Authorization: Bearer <your-tinybird-token>"
```

#### Using the CLI (https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#using-the-cli)

```
tb datasource sync <datasource_name>
```

## .connection settings (https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#connection-settings)

The GCS connector uses the following settings in .connection files:

| Instruction | Required | Description |
| --- | --- | --- |
| `GCS_SERVICE_ACCOUNT_CREDENTIALS_JSON` | Yes | Service Account Key in JSON format, inlined. Use [Tinybird Secrets](/docs/forward/dev-reference/commands/tb-secret). |
Once a Connection is used in a Data Source, you can't change the Service Account Key. To modify it, you must:

1. Remove the Connection from the Data Source.
2. Deploy the changes.
3. Add the Connection again with the new values.

## .datasource settings (https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#datasource-settings)

The GCS connector uses the following settings in .datasource files:

| Instruction | Required | Description |
| --- | --- | --- |
| `IMPORT_CONNECTION_NAME` | Yes | Name given to the Connection inside Tinybird. For example, `'my_connection'`. This is the name of the connection file you created in the previous step. |
| `IMPORT_BUCKET_URI` | Yes | Full bucket path, including the `gs://` protocol, bucket name, object path, and an optional pattern to match against object keys. For example, `gs://my-bucket/my-path` discovers all files in the bucket `my-bucket` under the prefix `/my-path`. You can use patterns in the path to filter objects, for example, ending the path with `*.csv` matches all objects that end with the `.csv` suffix. |
| `IMPORT_SCHEDULE` | Yes | Use `@on-demand` to sync new files as needed. On the first deployment, Tinybird automatically ingests all matching files. After the initial ingestion, when you manually trigger a sync, Tinybird appends only the files added since the last execution. You can also use `@once`, which behaves the same as `@on-demand`. `@auto` mode is not supported; if you use this option, Tinybird only executes the initial sync. |
| `IMPORT_FORMAT` | No | Format of the files to ingest: `csv`, `ndjson`, or `parquet`. Tinybird infers the format from the extension in `IMPORT_BUCKET_URI`, so you only need this setting when the bucket URI doesn't end with one of the [accepted extensions](https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#supported-file-types). |
| `IMPORT_FROM_TIMESTAMP` | No | Sets the date and time from which to start ingesting files on an GCS bucket. The format is `YYYY-MM-DDTHH:MM:SSZ`. |
You can't change these settings after creating the Data Source. To change them:

1. Remove the Connection from the Data Source.
2. Deploy the changes.
3. Add the Connection again with the new values.
4. Deploy again.

## Import sample data (https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#import-sample-data)

In branches and Tinybird Local, you can import a sample of files from GCS using the API. This is useful for validating schemas and testing pipelines without syncing all files from the bucket.

```
curl -X POST "https://<your_host>/v0/datasources/my_datasource/sample" \
  -H "Authorization: Bearer $TB_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"max_files": 1}'
```
The sample import starts an asynchronous job that imports up to `max_files` files (maximum 10). The response includes a `job_id` that you can use to track progress:

```
curl "https://<your_host>/v0/jobs/{job_id}?token=$TB_TOKEN"
```
The sample import runs as a separate job and doesn't affect production sync state or offsets.

## GCS file URI (https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#gcs-file-uri)

Use GCS wildcards to match multiple files:

- `*` (single asterisk): Matches files at one directory level.
  - Example: `gs://bucket-name/*.ndjson` (matches all `.ndjson` files in the root directory, but not in subdirectories).
- `**` (double asterisk): Recursively matches files across multiple directory levels.
  - Example: `gs://bucket-name/**/*.ndjson` (matches all `.ndjson` files anywhere in the bucket).
GCS does not allow overlapping ingestion paths. For example, you cannot have:

- `gs://my_bucket/**/*.csv`
- `gs://my_bucket/transactions/*.csv`

## Supported file types (https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#supported-file-types)

The GCS Connector supports the following formats:

| File Type \| Accepted Extensions \| Supported Compression |  |  |
| --- |  |  |
| CSV \| `.csv`, `.csv.gz` \| `gzip` | NDJSON \| `.ndjson`, `.ndjson.gz`, `.jsonl`, `.jsonl.gz` \| `gzip` | Parquet \| `.parquet`, `.parquet.gz` \| `snappy`, `gzip`, `lzo`, `brotli`, `lz4`, `zstd` |
JSON files must follow the **Newline Delimited JSON (NDJSON)** format. Each line must be a valid JSON object and must end with a `\n` character.

## Limitations (https://www.tinybird.co/docs/forward/ingest-data/connectors/gcs#limitations)

- **No `@auto` mode**: After the initial ingestion on first deployment, you must trigger subsequent ingestion manually.
- **File format support**: Only CSV, NDJSON, and Parquet are supported.
- **Permissions**: Ensure your service account has the correct role assigned.

---

URL: https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "DynamoDB connector \| Tinybird"
theme-color: "#171612"
description: "Tinybird provides you with an easy way to ingest and query large amounts of data with low-latency, and automatically create APIs to consume those queries. This makes it extremely easy to build fast and scalable applications that query your data; no back-end needed!"
inkeep:version: "forward"
---

# DynamoDB connector (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#dynamodb-connector)

Stream data from an Amazon DynamoDB table into a Tinybird data source. Tinybird performs an initial backfill of the table via a PITR (Point-in-Time Recovery) by exporting to S3, then continuously ingests changes through DynamoDB Streams (Change Data Capture).

Use the DynamoDB connector when you want to mirror an operational DynamoDB table into Tinybird for analytics, while keeping it up to date in near real time.

## How it works (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#how-it-works)

When you deploy a DynamoDB data source, Tinybird does two things:

1. **Initial export**: triggers an on-demand PITR export of your table to an S3 bucket you own, then loads that snapshot into the Data Source. AWS exports can take several minutes. The process continues polling until AWS marks the export as `COMPLETED`.
2. **Change Data Capture (CDC)**: starts a worker on Tinybird's infrastructure that reads from DynamoDB Streams and appends inserts, updates, and deletes to the same Data Source. Each row in the Data Source represents a *change* to your table, not the current state. To keep its size under control, DynamoDB Data Sources use the <a href="/docs/sql-reference/engines/replacingmergetree">`ReplacingMergeTree` engine</a>. See [Query the data](https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#query-the-data) for considerations.

## Requirements (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#requirements)

Before you create the connection, make sure your DynamoDB table meets these requirements:

- **Point-in-Time Recovery (PITR)** is enabled on the table.
- **DynamoDB Streams** is enabled, with a stream view type of `NEW_IMAGE` or `NEW_AND_OLD_IMAGES`.
- The table should not be larger than **500 GB** and write no more than **250 WCU (Write Capacity Unit)** (≈ 250 KB/s of writes). If you need higher limits, contact [Tinybird support](/docs/forward/support).

## AWS permissions (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#aws-permissions)

Tinybird ingests from DynamoDB by assuming an IAM role in *your* AWS account via `sts:AssumeRole` with an external ID. The role needs two policies: an **access policy** (what Tinybird may do) and a **trust policy** (who may assume it). You need to create both policies in AWS.

- AWS Access Policy
- AWS Trust Policy

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:Scan",
        "dynamodb:DescribeStream",
        "dynamodb:DescribeExport",
        "dynamodb:GetRecords",
        "dynamodb:GetShardIterator",
        "dynamodb:DescribeTable",
        "dynamodb:DescribeContinuousBackups",
        "dynamodb:ExportTableToPointInTime",
        "dynamodb:UpdateTable",
        "dynamodb:UpdateContinuousBackups"
      ],
      "Resource": [
        "arn:aws:dynamodb:us-east-1:123456789012:table/orders",
        "arn:aws:dynamodb:us-east-1:123456789012:table/orders/stream/*",
        "arn:aws:dynamodb:us-east-1:123456789012:table/orders/export/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::my-orders-exports",
        "arn:aws:s3:::my-orders-exports/*"
      ]
    }
  ]
}
```
The access policy grants read on the table, its stream, and its exports, plus read-write on the export bucket. Scope the resources to your table and bucket. `dynamodb:UpdateTable` and `dynamodb:UpdateContinuousBackups` let the connector enable PITR and Streams (`NEW_AND_OLD_IMAGES`) on the table if they aren't already on.

The trust policy must name Tinybird's connector account for your region and environment, and the Workspace's external ID. The account and external ID differ per region and environment. See [Set up the connector](https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#set-up-the-connector) section to see how to get `<TINYBIRD_CONNECTOR_ACCOUNT>` and `<EXTERNAL_ID>` values.

A `403 ... include the following external ID` error means the trust policy's external ID or `Principal` account doesn't match what Tinybird presents when it assumes the role. If you defined the connection in code without running `tb connection create dynamodb`, the trust policy is likely missing the Workspace-specific external ID entirely. If the external ID is already there, the `Principal` account is wrong for this environment: Tinybird assumes the role from a different account per region and environment. Either way, the `tb connection create dynamodb` output is the source of truth.

One role can serve many Workspaces. `sts:ExternalId` accepts a list, so you can add more Workspaces' external IDs to the same role: `"sts:ExternalId": ["<workspace-a-id>", "<workspace-b-id>"]`.

## Environment considerations (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#environment-considerations)

The DynamoDB Connector behaves differently across the Cloud, Branch, and Local environments. PITR exports and stream reads run in *your* AWS account, but the AWS account that assumes your IAM role changes from one environment to the next. The trust policy you write depends on where the Connector runs.

### Cloud environment (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#cloud-environment)

In Tinybird Cloud, Tinybird uses its own AWS account to assume the IAM role you create. When you deploy to your main Cloud Workspace, use `tb deploy` as usual.

### Branch environment (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#branch-environment)

When you test a data source using the DynamoDB connector in a Cloud Branch, include `--with-connections` so Tinybird sets up the DynamoDB connections in the branch:

```
tb build --with-connections
```
A cloud branch reuses the same connection (and therefore the same IAM role) as the parent Workspace, so no extra AWS setup is needed. To avoid duplicate exports and CDC workers competing over the same DynamoDB stream, point branch Data Sources at a separate test table.

Unlike main Workspaces, Cloud branches and Tinybird Local **don't** trigger a PITR export automatically. Building or deploying a DynamoDB Data Source in a branch or in Local sets up the Connection but leaves the Data Source empty, so you avoid slow exports and unnecessary egress costs while you iterate. Load data on demand with `tb datasource sample` instead. See [Sample data in branches and Local](https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#sample-data-in-branches-and-local).

Main Workspaces still run a full PITR export on deploy, as described in [How it works](https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#how-it-works).

### Local environment (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#local-environment)

Tinybird Local runs in a container. Because PITR exports run in your AWS account, Tinybird Local needs your local AWS credentials to assume the role:

```
tb local restart --use-aws-creds
```
The trust policy differs per environment: Cloud is assumed by Tinybird's AWS account, while Local is assumed by the AWS account of the credentials you pass with `--use-aws-creds`. When you create the Connection, choose **Local**, **Cloud**, or **Both** so the generated trust policy lists the right account IDs. If local credentials aren't available, the CLI warns you and continues Cloud-only. The Connection stays valid for `tb --cloud deploy`, but `tb build` and `tb deploy` against Local skip the DynamoDB resource.

### Sample data in branches and Local (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#sample-data-in-branches-and-local)

Because Cloud branches and Local don't backfill automatically, use `tb datasource sample` to load data into a DynamoDB data source on demand. Instead of a full PITR export, the sample scans the table directly and imports a bounded subset, which is faster and cheaper while you test. Samples are capped at **10 GB** regardless of the `--rows` or `--max-bytes` values you pass. If you need a larger sample, contact [Tinybird support](/docs/forward/support). Build the branch with `--with-connections` first so the data source has a connection to sample from.

```
# Import up to 1500 rows (the default)
tb --branch=my_branch datasource sample orders --wait

# Cap the sample by row count
tb --branch=my_branch datasource sample orders --rows 100000 --wait

# Cap the sample by approximate size
tb --branch=my_branch datasource sample orders --max-bytes 500MB --wait

# Trigger a full PITR export instead of a bounded sample
tb --branch=my_branch datasource sample orders --full-export --wait
```

| Option | Description |
| --- | --- |
| `--rows` | Maximum number of rows to scan and import. Defaults to 1500. Mutually exclusive with `--max-bytes`. |
| `--max-bytes` | Maximum approximate size to import, for example `500MB` or `1GB`. Capped at 10 GB. Mutually exclusive with `--rows`. |
| `--full-export` | Trigger a full PITR export of the whole table instead of a bounded sample. This can import the entire table, so it may be slow and incur egress costs. |
| `--wait` | Block until the import job finishes instead of returning immediately with job info. |
Sampling works the same in Tinybird Local. Drop the `--branch` flag and run the command against your local environment. Like exports, table scans run in your AWS account, so Tinybird Local needs your AWS credentials (`tb local restart --use-aws-creds`).

## Set up the connector (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#set-up-the-connector)

The Tinybird CLI includes a wizard that walks you through the whole flow: creating the IAM role, generating the `.connection` and `.datasource` files, and validating the table.

```
tb connection create dynamodb
```
Working in the TypeScript or Python SDK? Run the Tinybird CLI wizard anyway to handle the IAM role and external ID, then convert the generated `.connection` and `.datasource` files to their SDK equivalents (see the [TypeScript SDK](https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#3-define-the-connection-file) and [Python SDK](https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#3-define-the-connection-file) tabs). The IAM role and secret carry over unchanged.

The CLI asks for:

1. A name for the connection.
2. The DynamoDB **table name** and **export bucket name** (used to scope the IAM policy). Use `*` for unrestricted.
3. The AWS **region** of your table.
4. Which environments use the Connection: **Local**, **Cloud**, or **Both**. Tinybird builds a trust policy containing the AWS account IDs of the selected environments.
The wizard prints a managed IAM **access policy** and **trust policy** with the correct values for you to paste into AWS. After you create the role, paste its ARN back into the CLI. Tinybird then validates the table and writes the connection file.

Finally, the wizard asks for:

- The **DynamoDB table ARN** (for example, `arn:aws:dynamodb:us-east-1:123456789012:table/my-table`).
- The **S3 export bucket** (the bucket name without the `s3://` prefix).
It generates `connections/<name>.connection` and `datasources/<name>.datasource`, ready to deploy. The generated `.datasource` file includes the table's partition key (pk) and sort key (sk) as typed columns, extracted from the change record with `json:` paths and set as the engine sorting key, so you can query and filter on them without writing `JSONExtract*` expressions yourself.

Build the project locally or on a Tinybird Cloud Branch to validate the generated datafiles. Include `--with-connections` flag so the DynamoDB connections are set up:

```
tb build --with-connections
```
When the build succeeds, deploy to Tinybird Cloud:

```
tb --cloud deploy
```

### Manual setup (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#manual-setup)

To write the `.connection` and `.datasource` files manually instead of using the wizard, follow these steps.

#### 1. Create the IAM role (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#1-create-the-iam-role)

Create the IAM role Tinybird assumes to read your table, its stream, and the S3 export bucket. The role needs an **access policy** and a **trust policy** — see [AWS permissions](https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#aws-permissions) for both policy documents, the per-environment placeholders, and the AWS IAM console steps.

The trust policy's `Principal` account and `ExternalId` differ per region and environment.

To create the role in the AWS IAM console:

1. Go to **Policies** → **Create policy**, paste the preceding access policy JSON, and name it (for example, `tinybird-dynamodb-orders`).
2. Go to **Roles** → **Create role** → **Custom trust policy**, and paste the preceding trust policy JSON.
3. Attach the access policy from step 1, then name the role (for example, `TinybirdRole-dynamo`).
4. Copy the role ARN and paste it back into the wizard, or store it as a secret (see [Add the role ARN as a secret](https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#2-add-the-role-arn-as-a-secret)).
Since the `<TINYBIRD_CONNECTOR_ACCOUNT>` and `<EXTERNAL_ID>` values vary per environment, use the Tinybird CLI wizard `tb connection create dynamodb` to get them.

#### 2. Add the role ARN as a secret (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#2-add-the-role-arn-as-a-secret)

Store the role ARN as a Tinybird secret so it isn't checked into your repo. When you create the secret manually, its name **must** follow the format `dynamodb_role_arn_<connection_name>`, where `<connection_name>` matches the name of your `.connection` file. Tinybird looks up the secret by this exact name:

```
tb secret set dynamodb_role_arn_<connection_name> "arn:aws:iam::123456789012:role/tb-my-dynamodb-role"
```
The wizard does this automatically in Local and Cloud when it creates the connection.

#### 3. Define the `.connection` file (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#3-define-the-file)

- Tinybird CLI
- TypeScript SDK
- Python SDK
connections/my_ddb.connection
```
TYPE dynamodb
DYNAMODB_ARN {{ tb_secret("dynamodb_role_arn_my_ddb") }}
DYNAMODB_REGION us-east-1
```

| Instruction | Required | Description |
| --- | --- | --- |
| `TYPE` | Yes | Must be `dynamodb`. |
| `DYNAMODB_ARN` | Yes | The IAM role ARN. Reference via `tb_secret(...)` so it stays out of git. |
| `DYNAMODB_REGION` | Yes | The AWS region the DynamoDB table lives in. Must match the region in `IMPORT_TABLE_ARN`. See [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html) for valid region codes. |

#### 4. Define the `.datasource` file (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#4-define-the-file)

- Tinybird CLI
- TypeScript SDK
- Python SDK
datasources/orders.datasource
```
SCHEMA >
    `<partition_key>` String `json:$.Item.<partition_key>`,
    `<sort_key>` String `json:$.Item.<sort_key>`,
    `_record` String `json:$.NewImage`,
    `_old_record` Nullable(String) `json:$.OldImage`,
    `_timestamp` DateTime64(3) `json:$.ApproximateCreationDateTime`,
    `_event_name` LowCardinality(String) `json:$.eventName`,
    `_is_deleted` UInt8 `json:$._is_deleted`

ENGINE "ReplacingMergeTree"
ENGINE_SORTING_KEY <partition_key>, <sort_key>
ENGINE_VER _timestamp
ENGINE_IS_DELETED _is_deleted

IMPORT_CONNECTION_NAME 'my_ddb'
IMPORT_TABLE_ARN 'arn:aws:dynamodb:us-east-1:123456789012:table/orders'
IMPORT_EXPORT_BUCKET 'my-orders-exports'
```
The first columns are the table's partition key (pk) and sort key (sk), named after your table's key attributes. `tb connection create dynamodb` adds them automatically, pulls them from the item with `json:` paths, and uses them as the `ENGINE_SORTING_KEY`.

DynamoDB data sources **must** use the `ReplacingMergeTree` engine. Other engines are rejected at build time.

| Instruction | Required | Description |
| --- | --- | --- |
| `IMPORT_CONNECTION_NAME` | Yes | Name of the `.connection` file (without the extension). |
| `IMPORT_TABLE_ARN` | Yes | Full ARN of the DynamoDB table to mirror. Must start with `arn:aws:dynamodb:`. |
| `IMPORT_EXPORT_BUCKET` | Yes | Name of the S3 bucket where Tinybird writes PITR exports. Use the bucket name only, without the `s3://` prefix. |

##### Schema columns

Alongside the key columns described in this section, every DynamoDB Data Source has these system columns, each populated from the change record with a `json:` path:

| Column | Type | `json:` path | Description |
| --- | --- | --- | --- |
| `_record` | `String` | `$.NewImage` | JSON-encoded current item image after the change. |
| `_old_record` | `Nullable(String)` | `$.OldImage` | JSON-encoded previous item image. Only present when the stream view type is `NEW_AND_OLD_IMAGES`. |
| `_timestamp` | `DateTime64(3)` | `$.ApproximateCreationDateTime` | Approximate time the change happened in DynamoDB. Used as the `ReplacingMergeTree` version column. |
| `_event_name` | `LowCardinality(String)` | `$.eventName` | `INSERT`, `MODIFY`, `REMOVE`, or `EXPORT` for initial backfill rows. |
| `_is_deleted` | `UInt8` | `$._is_deleted` | `1` for deletes, `0` otherwise. Drives `ReplacingMergeTree` 's deleted-row semantics. |
To extract any other typed columns from your items, query `_record` with `JSONExtract*` functions in a pipe rather than adding more columns to the data source. The connector maps columns from the change-record envelope (`$.NewImage`, `$.eventName`, and so on), not from the attributes inside your item, so item fields aren't available as top-level columns. Keeping the full item in `_record` also means the mirror keeps working when DynamoDB attributes are added, renamed, or retyped, since there's no fixed item schema to migrate. If a field is read often and you want it as a typed, pre-computed column, extract it in a downstream materialized view instead.

## Query the data (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#query-the-data)

Because the Data Source captures *every* change, querying it directly returns multiple rows per item. Use `FINAL` (or rely on the underlying ReplacingMergeTree merges) to get the current state:

```
SELECT
    JSONExtractString(_record, 'id')        AS id,
    JSONExtractString(_record, 'status')    AS status,
    JSONExtractFloat (_record, 'amount')    AS amount
FROM orders FINAL
```

## Deploying (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#deploying)

Deploy to Tinybird Cloud:

- Tinybird CLI
- TypeScript SDK
- Python SDK

```
tb --cloud deploy
```
On deploy, Tinybird:

1. Validates the table (PITR enabled, streams enabled with a supported view type, within size and WCU limits).
2. Triggers the PITR export to your S3 bucket.
3. Streams the export into the data source.
4. Starts the CDC worker.
The CLI displays a message like:

```
△ DynamoDB initial export backfill started for datasource 'orders'.
  AWS exports can stay in progress for several minutes; Tinybird Local will keep
  retrying the import until AWS marks the export as completed.
  Export ARN: arn:aws:dynamodb:us-east-1:123456789012:export/...
```

## Validation errors (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#validation-errors)

`tb deploy` runs the same validation as `tb connection create dynamodb`. Common errors:

| Error | What to do |
| --- | --- |
| `The DynamoDB table was not found.` | Check the table ARN and that the region in the `.connection` file matches the ARN's region. |
| `Point-in-Time Recovery (PITR) must be enabled.` | Enable PITR on the table in the DynamoDB console. |
| `DynamoDB Streams must be enabled.` | Enable streams on the table. |
| `DynamoDB Streams must use NEW_IMAGE or NEW_AND_OLD_IMAGES.` | Change the stream view type — `KEYS_ONLY` and `OLD_IMAGE` are not supported. |
| `The DynamoDB table exceeds the current size limit.` | The table is over 500 GB. Contact support to raise the limit. |
| `The DynamoDB table exceeds the current write-capacity limit.` | The table writes more than 250 WCU. Contact support to raise the limit. |

## Limitations (https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb#limitations)

- One CDC worker per data source. Throughput is bounded by ~250 WCU.
- Stream records have a 24-hour retention in DynamoDB. If CDC is paused for more than 24 hours (for example, a broken IAM role), you can miss some changes and need to backfill again.
- CDC delivery is **at-least-once** — duplicate change events can appear in recovery scenarios. `ReplacingMergeTree` with `_timestamp` as the version column collapses them on read with `FINAL`.

---

URL: https://www.tinybird.co/docs/forward/dev-reference/datafiles/tinyb-file
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: ".tinyb file \| Tinybird"
theme-color: "#171612"
description: "The .tinyb file contains the Tinybird project configuration, including the authentication token."
inkeep:version: "forward"
---

# .tinyb file (https://www.tinybird.co/docs/forward/dev-reference/datafiles/tinyb-file#tinyb-file)

The `.tinyb` file is a configuration file that contains the Tinybird project configuration, including the authentication token obtained by running `tb login`.

Running commands requires a valid `.tinyb` file in the root of your project. If you don't have one, you can create one by running [tb login](/docs/forward/dev-reference/commands/tb-login).

## Location (https://www.tinybird.co/docs/forward/dev-reference/datafiles/tinyb-file#location)

You can place the `.tinyb` file in the root of your app repository or keep it together with your Tinybird datafiles. Tinybird looks for the `.tinyb` file in all parent folders till it reaches the home directory.

If your Tinybird files and folders are in a subdirectory, such as `tinybird/`, add a `cwd` field to specify the relative path.

## File structure (https://www.tinybird.co/docs/forward/dev-reference/datafiles/tinyb-file#file-structure)

The following is a sample `.tinyb` file:

Sample .tinyb file
```
{
    "host": "<tinybird-host>",
    "id": "<workspace-id>",
    "name": "<workspace-name>",
    "scope": "user",
    "token": "<authentication-token>",
    "tokens": {
        "<tinybird-host>": "<authentication-token>",
        "<another-tinybird-host>": "<authentication-token>"
    },
    "cwd": "<optional-path-to-your-tinybird-folder-e.g-./tinybird>",
    "user_email": "<user-email>",
    "user_id": "<user-id>",
    "user_token": "<authentication-token>",
    "version": "<tinybird-version>"
}
```

---

URL: https://www.tinybird.co/docs/forward/dev-reference/datafiles/test-files
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Test files \| Tinybird"
theme-color: "#171612"
description: "Test files are used to test the API endpoints."
inkeep:version: "forward"
---

# Test files (https://www.tinybird.co/docs/forward/dev-reference/datafiles/test-files#test-files)

Test files describe the tests for the API endpoints. See [Test your project](/docs/forward/guides/test-your-project#create-a-test-suite).

Test files are stored in the `tests` folder in your project.

## Test file format (https://www.tinybird.co/docs/forward/dev-reference/datafiles/test-files#test-file-format)

Test files are YAML files that contain the tests for the API endpoints.

Test files are structured as YAML arrays. For example:

```
- name: get_sets_by_theme_year_2000s_range
  description: Test filtering by year range 2000-2005
  expected_http_status: 200
  parameters: year_start=2000&year_end=2005
  expected_result: |
    {"year":2004,"sets_count":1,"parts_count":42,"avg_parts_per_set":42}
    {"year":2005,"sets_count":1,"parts_count":177,"avg_parts_per_set":177}

- name: get_sets_by_theme_year_1990s_range
  description: Test filtering by year range 1990-1999
  expected_http_status: 200
  parameters: year_start=1990&year_end=1999
  expected_result: |
    {"year":1992,"sets_count":1,"parts_count":668,"avg_parts_per_set":668}
    {"year":1993,"sets_count":1,"parts_count":129,"avg_parts_per_set":129}
    {"year":1994,"sets_count":1,"parts_count":1343,"avg_parts_per_set":1343}
    {"year":1996,"sets_count":1,"parts_count":803,"avg_parts_per_set":803}
    {"year":1998,"sets_count":1,"parts_count":1272,"avg_parts_per_set":1272}
    {"year":1999,"sets_count":1,"parts_count":397,"avg_parts_per_set":397}
```
Each test is a YAML object with the following fields:

- `name`: The name of the test.
- `description`: The description of the test.
- `parameters`: The query parameters for the endpoint.
- `expected_result`: The expected result for the test.
- `expected_http_status`: The expected HTTP status, for example `200`.

## Create a test file (https://www.tinybird.co/docs/forward/dev-reference/datafiles/test-files#create-a-test-file)

Create a YAML file manually in the `tests` folder. See [Test file format](https://www.tinybird.co/docs/forward/dev-reference/datafiles/test-files#test-file-format) and the [tb test](/docs/forward/dev-reference/commands/tb-test) command reference.

## Run tests (https://www.tinybird.co/docs/forward/dev-reference/datafiles/test-files#run-tests)

To run tests, use `tb test run`.

---

URL: https://www.tinybird.co/docs/forward/dev-reference/datafiles/pipe-files
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Pipe files \| Tinybird"
theme-color: "#171612"
description: "Pipe files describe your Tinybird pipes. Define the type, data source, and other settings."
inkeep:version: "forward"
---

# Pipe files (.pipe) (https://www.tinybird.co/docs/forward/dev-reference/datafiles/pipe-files#pipe-files-pipe)

Pipe files describe your pipes. You can use .pipe files to define the type, starting node, data source, and other settings of your pipes.

## Nodes (https://www.tinybird.co/docs/forward/dev-reference/datafiles/pipe-files#nodes)

A node is a container for a single SQL `SELECT` statement. Nodes live within pipes, and you can have many sequential nodes inside the same pipe. They allow you to break your query logic down into multiple smaller queries. You can then chain nodes together to build the logic incrementally.

A query in a node can read data from a data source, other nodes inside the same pipe, or from endpoint nodes in other pipes. Each node can be developed and tested individually. This makes it much easier to build complex query logic in Tinybird as you avoid creating large monolithic queries with many subqueries.

## Available instructions (https://www.tinybird.co/docs/forward/dev-reference/datafiles/pipe-files#available-instructions)

The following instructions are available for .pipe files.

| Instruction | Required | Description |
| --- | --- | --- |
| `%` | No | Use as the first character of a node to indicate the node uses the [templating system](/docs/forward/dev-reference/template-functions). |
| `DESCRIPTION <markdown_string>` | No | Sets the description for a node or the complete file. |
| `TAGS <tag_names>` | No | Comma-separated list of tags. |
| `NODE <node_name>` | Yes | Starts the definition of a new node. All the instructions until a new `NODE` instruction or the end of the file are related to this node. |
| `SQL <sql>` | Yes | Defines a block for the SQL of a node. The block must be indented. |
| `TYPE <pipe_type>` | No | Sets the type of the pipe. Valid values are `ENDPOINT`, `MATERIALIZED`, `COPY`, or `SINK`. |
| `DATASOURCE <data_source_name>` | Yes | Required when `TYPE` is `MATERIALIZED`. Sets the destination data source for materialized nodes. |
| `TARGET_DATASOURCE <data_source_name>` | Yes | Required when `TYPE` is `COPY`. Sets the destination data source for copy nodes. |
| `TOKEN <token_name> READ` | No | Grants read access to a pipe or endpoint to the token named `<token_name>`. If you don't specify the token or `<token_name>` doesn't exist, Tinybird creates it automatically. |
| `COPY_SCHEDULE` | No | Cron expression with the frequency to run copy jobs. Must be higher than 5 minutes. For example, `*/5 * * * *`. If undefined, it defaults to `@on-demand`. |
| `COPY_MODE` | No | Strategy to ingest data for copy jobs. One of `append` or `replace`. If empty, the default strategy is `append`. |
| `DEPLOYMENT_METHOD <method>` | No | Deployment method for materialized views. Use `alter` to update existing materialized views using `ALTER TABLE ... MODIFY QUERY` instead of recreating the table. Only valid for `TYPE MATERIALIZED`. |

## Endpoint pipes (https://www.tinybird.co/docs/forward/dev-reference/datafiles/pipe-files#endpoint-pipes)

In a .pipe file you can define how to export the result of a pipe as an HTTP endpoint.

The following example shows how to describe an endpoint pipe. See [Endpoints](/docs/forward/core-concepts/api-endpoints).

tinybird/pipes/sales_by_hour_endpoint.pipe
```
TOKEN dashboard READ
DESCRIPTION endpoint to get sales by hour filtering by date and country

TAGS sales

NODE daily_sales
SQL >
    %
    SELECT day, country, sum(total_sales) as total_sales
    FROM sales_by_hour
    WHERE
    day BETWEEN toStartOfDay(now()) - interval 1 day AND toStartOfDay(now())
    and country = {{ String(country, 'US')}}
    GROUP BY day, country

NODE result
SQL >
    %
    SELECT * FROM daily_sales
    LIMIT {{Int32(page_size, 100)}}
    OFFSET {{Int32(page, 0) * Int32(page_size, 100)}}
TYPE ENDPOINT
```

## Materialized pipes (https://www.tinybird.co/docs/forward/dev-reference/datafiles/pipe-files#materialized-pipes)

In a .pipe file you can define how to materialize each row ingested in the earliest data source in the pipe query to a materialized data source. Materialization happens at ingest. See [Materialized views](/docs/forward/core-concepts/materialized-views).

The following example shows how to describe a materialized pipe.

tinybird/pipes/sales_by_hour_mv.pipe
```
DESCRIPTION Materialized pipe to aggregate sales per hour in the sales_by_hour data source

NODE daily_sales
SQL >
    SELECT toStartOfDay(starting_date) day, country, sum(sales) as total_sales
    FROM teams
    GROUP BY day, country

TYPE MATERIALIZED
DATASOURCE sales_by_hour
```

### Use the ALTER deployment method (https://www.tinybird.co/docs/forward/dev-reference/datafiles/pipe-files#use-the-alter-deployment-method)

When updating an existing materialized view, you can use `DEPLOYMENT_METHOD alter` to modify the query without recreating the table:

tinybird/pipes/sales_by_hour_mv.pipe
```
DESCRIPTION Updated materialized pipe with additional metrics

NODE daily_sales
SQL >
    SELECT
        toStartOfDay(starting_date) day,
        country,
        sum(sales) as total_sales,
        avg(sales) as avg_sales,
        count(*) as transaction_count
    FROM teams
    GROUP BY day, country

TYPE MATERIALIZED
DATASOURCE sales_by_hour
DEPLOYMENT_METHOD alter
```
The `DEPLOYMENT_METHOD alter` directive tells Tinybird to use `ALTER TABLE ... MODIFY QUERY` instead of dropping and recreating the materialized view. This preserves existing data and reduces deployment time. See [Altering materialized views](/docs/forward/core-concepts/materialized-views#altering-materialized-views) for more details.

## Copy pipes (https://www.tinybird.co/docs/forward/dev-reference/datafiles/pipe-files#copy-pipes)

In a .pipe file you can define how to export the result of a pipe to a data source, optionally with a schedule. See [Copy pipes](/docs/forward/core-concepts/copy-pipes).

The following example shows how to describe a copy pipe.

tinybird/pipes/sales_by_hour_cp.pipe
```
DESCRIPTION Copy pipe to export sales hour every hour to the sales_hour_copy data source

NODE daily_sales
SQL >
    %
    SELECT toStartOfDay(starting_date) day, country, sum(sales) as total_sales
    FROM teams
    WHERE
    day BETWEEN toStartOfDay(now()) - interval 1 day AND toStartOfDay(now())
    and country = {{ String(country, 'US')}}
    GROUP BY day, country

TYPE COPY
TARGET_DATASOURCE sales_hour_copy
COPY_SCHEDULE 0 * * * *
```

## Sink Pipes (https://www.tinybird.co/docs/forward/dev-reference/datafiles/pipe-files#sink-pipes)

In a .pipe file you can define how to export the result of a pipe to an external source, optionally with a schedule. See [Sink pipes](/docs/forward/core-concepts/sinks).

The following parameters are available when defining Sink Pipes:

| Instruction | Required | Description |
| --- | --- | --- |
| `EXPORT_CONNECTION_NAME` | Yes | The name of the export connection. |
| `EXPORT_SCHEDULE` | No | Cron expression, in UTC time. Must be higher than 5 minutes. For example, `*/5 * * * *`. |

### Blob storage Sink (https://www.tinybird.co/docs/forward/dev-reference/datafiles/pipe-files#blob-storage-sink)

When using a S3 or GCS connection, you can use the following instructions:

| Instruction | Required | Description |
| --- | --- | --- |
| `EXPORT_BUCKET_URI` | Yes | The desired bucket path for the exported file. Path must not include the filename and extension. |
| `EXPORT_FILE_TEMPLATE` | Yes | Template string that specifies the naming convention for exported files. The template can include dynamic attributes between curly braces based on column data. Tinybird replaces the attributes with real values when exporting. For example: `export_{category}{date,'%Y'}{2}`. |
| `EXPORT_FORMAT` | Yes | Format in which the data is exported. The default value is `csv`. |
| `EXPORT_COMPRESSION` | No | Compression file type. Accepted values are `none`, `gz` for gzip, `br` for brotli, `xz` for LZMA, `zst` for zstd. Default values is `none`. |
| `EXPORT_STRATEGY` | Yes | One of the available strategies. The default is `@new`. |

### Kafka Sink (https://www.tinybird.co/docs/forward/dev-reference/datafiles/pipe-files#kafka-sink)

When using a `kafka` connection, you can use the following instructions:

| Instruction | Required | Description |
| --- | --- | --- |
| `EXPORT_KAFKA_TOPIC` | Yes | The desired topic for the export data. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/datafiles/datasource-files
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Datasource files \| Tinybird"
theme-color: "#171612"
description: "Datasource files describe your data sources. Define the schema, engine, and other settings."
inkeep:version: "forward"
---

# Datasource files (.datasource) (https://www.tinybird.co/docs/forward/dev-reference/datafiles/datasource-files#datasource-files-datasource)

Datasource files describe your data sources. You can use .datasource files to define the schema, engine, and other settings of your data sources.

## Available instructions (https://www.tinybird.co/docs/forward/dev-reference/datafiles/datasource-files#available-instructions)

The following instructions are available for .datasource files.

| Declaration | Required | Description |
| --- | --- | --- |
| `SCHEMA <indented_schema_definition>` | Yes | Defines a block for a data source schema. The block must be indented. |
| `ENGINE <engine_type>` | No | Sets the engine for data source. Default value is `MergeTree`. |
| `ENGINE_SORTING_KEY <sql>` | No | Sets the `ORDER BY` expression for the data source. |
| `ENGINE_PARTITION_KEY <sql>` | No | Sets the `PARTITION` expression for the data source. **Use monthly (`toYYYYMM(date_column)`) or yearly (`toYear(date_column)`) partitions only.** Overly granular partition keys severely degrade write performance. See [MergeTree engine](/docs/sql-reference/engines/mergetree#engine-settings) for details. |
| `ENGINE_TTL <sql>` | No | Sets the `TTL` expression for the data source. |
| `ENGINE_VER <column_name>` | No | Column with the version of the object state. Required when using `ENGINE ReplacingMergeTree`. |
| `ENGINE_SIGN <column_name>` | No | Column to compute the state. Required when using `ENGINE CollapsingMergeTree` or `ENGINE VersionedCollapsingMergeTree`. |
| `ENGINE_VERSION <column_name>` | No | Column with the version of the object state. Required when `ENGINE VersionedCollapsingMergeTree`. |
| `ENGINE_SETTINGS <settings>` | No | Comma-separated list of key-value pairs that describe engine settings for the data source. |
| `FORWARD_QUERY <sql>` | No | Defines a query to execute on the data source. The results of the query are returned instead of the original schema defined in the `SCHEMA` declaration. See [Evolve data sources](/docs/forward/guides/evolve-data-source#forward-query). |
| `BACKFILL skip` | No | Skips the initial data backfill when creating a new Data Source that is the destination for a Materialized View. This instruction is ignored if the Data Source already exists. |
| `TOKEN <token_name> READ\|APPEND` | No | Grants read or append access to a datasource to the token named `<token_name>`. If you don't specify the token or `<token_name>` doesn't exist, Tinybird creates it automatically. |
| `SHARED_WITH <workspace_name>` | No | Shares the Data Source with one or more Workspaces. Workspaces need to be in the same organization. |
The following example shows a typical .datasource file:

tinybird/datasources/example.datasource
```
# A comment
SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"
ENGINE_TTL "timestamp + toIntervalDay(60)"
ENGINE_SETTINGS "index_granularity=8192"
```

### Schema (https://www.tinybird.co/docs/forward/dev-reference/datafiles/datasource-files#schema)

A `SCHEMA` declaration is a newline, comma-separated list of columns definitions. For example:

Example SCHEMA declaration
```
SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload`
```

#### Column definition (https://www.tinybird.co/docs/forward/dev-reference/datafiles/datasource-files#column-definition)

Each column in a `SCHEMA` declaration has the following format:

```
`<column_name>` <data_type> `<json_path>` DEFAULT <default_value> CODEC(<codec>)
```

| Field | Required | Description |
| --- | --- | --- |
| `<column_name>` | Yes | The name of the column in the data source. |
| `<data_type>` | Yes | One of the supported [Data types](/docs/sql-reference/data-types). |
| `<json_path>` | No | JSONPath expression for NDJSON or Parquet data. See [JSONPath expressions](https://www.tinybird.co/docs/forward/dev-reference/datafiles/datasource-files#jsonpath-expressions). |
| `DEFAULT <default_value>` | No | Sets a default value for the column when the value is null or missing. |
| `CODEC(<codec>)` | No | Overrides the default compression codec. See [Column compression codecs](https://www.tinybird.co/docs/forward/dev-reference/datafiles/datasource-files#column-compression-codecs). |
The following example shows a column definition using all available fields:

Complete column definition example
```
SCHEMA >
    `timestamp` DateTime64(3) `json:$.timestamp` DEFAULT now() CODEC(DoubleDelta, ZSTD(1)),
    `session_id` String `json:$.session_id` DEFAULT '' CODEC(ZSTD(1)),
    `status` LowCardinality(String) `json:$.status` DEFAULT 'unknown',
    `count` Int32 `json:$.count` DEFAULT 0,
    `payload` String `json:$`
```

#### Default values (https://www.tinybird.co/docs/forward/dev-reference/datafiles/datasource-files#default-values)

Use `DEFAULT` to set a value for a column when the incoming data is null or missing. This is useful for:

- Providing fallback values for optional fields
- Setting automatic timestamps with `now()`
- Avoiding nullable types when a default makes sense
Example with DEFAULT values
```
SCHEMA >
    `timestamp` DateTime DEFAULT now(),
    `status` String DEFAULT 'pending',
    `count` Int32 DEFAULT 0,
    `is_active` UInt8 DEFAULT 1
```

#### JSONPath expressions (https://www.tinybird.co/docs/forward/dev-reference/datafiles/datasource-files#jsonpath-expressions)

`SCHEMA` definitions need JSONPath expressions when working with Parquet or NDJSON data.

It supports base fields `json:$.field`, arrays `json:$.an_array[:]`, nested fields `json:$.nested.nested_field`, and storing the whole object `json:$`.

For example, given this JSON object:

```
{
  "field": "test",
  "nested": { "nested_field": "bla" },
  "an_array": [1, 2, 3],
  "a_nested_array": { "nested_array": [1, 2, 3] }
}
```
The schema would be something like this:

jsonpaths.datasource
```
SCHEMA >
    field String `json:$.field`,
    nested_nested_field String `json:$.nested.nested_field`,
    an_array Array(Int16) `json:$.an_array[:]`,
    a_nested_array_nested_array Array(Int16) `json:$.a_nested_array.nested_array[:]`,
    whole_message String `json:$`
```
Use brackets for attributes with dots that are not actual nested attributes, like this:

```
{
    "attributes": {
        "otel.attributes": {
            "cli_command": "datasource ls",
        }
    }
}
```
jsonpaths.datasource
```
SCHEMA >
    cli_command String `json:$.attributes.['otel.attributes'].cli_command`
```
Tinybird's JSONPath syntax support has some limitations: It support nested objects at multiple levels, but it **supports nested arrays only at the first level**, as in the example before. To ingest and transform more complex JSON objects, store the whole JSON as a String "<column_name> String `json:$` ", and use [JSONExtract functions](/docs/sql-reference/functions/json-functions#jsonextract-functions) to parse at query time or in materializations.

#### Column compression codecs (https://www.tinybird.co/docs/forward/dev-reference/datafiles/datasource-files#column-compression-codecs)

Tinybird applies compression codecs to data types to optimize storage. You can override the default compression codecs by adding the `CODEC(<codec>)` statement after the type declarations in your .datasource schema. For example:

```
SCHEMA >
    `product_id` Int32 `json:$.product_id`,
    `timestamp` DateTime64(3) `json:$.timestamp` CODEC(DoubleDelta, ZSTD(1)),
```

### Engine settings (https://www.tinybird.co/docs/forward/dev-reference/datafiles/datasource-files#engine-settings)

`ENGINE` declares the engine used for the data source. The default value is `MergeTree`.

See [Engines](/docs/sql-reference/engines) for more information.

### Connector settings (https://www.tinybird.co/docs/forward/dev-reference/datafiles/datasource-files#connector-settings)

A data source file can contain connector settings for certain types of sources, such as Kafka or S3. See [Connections](/docs/forward/core-concepts/connections).

## Forward query (https://www.tinybird.co/docs/forward/dev-reference/datafiles/datasource-files#forward-query)

If you make changes to a .datasource file that are incompatible with the live version, you must use the `FORWARD_QUERY` instruction to transform the data from the live schema to the new one. Otherwise, the deployment fails due to a schema mismatch.

You can't use `ALTER` to change the source of a Materialized View. To do so, you must drop the Materialized View and recreate it with the new source. To prevent data loss during this process, add a `SELECT *` `FORWARD_QUERY` to the destination `.datasource` file.

See [Evolve data sources](/docs/forward/guides/evolve-data-source#forward-query) for more information.

---

URL: https://www.tinybird.co/docs/forward/dev-reference/datafiles/connection-files
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Connection files \| Tinybird"
theme-color: "#171612"
description: "Connection files describe your data connections. They're used by Tinybird connectors."
inkeep:version: "forward"
---

# Connection files (.connection) (https://www.tinybird.co/docs/forward/dev-reference/datafiles/connection-files#connection-files-connection)

Connection files describe your data connections. They're used by Tinybird connectors to establish a connection when deploying to live environments. See [Connections](/docs/forward/core-concepts/connections).

Connection files bear the .connection extension and are stored in the connections folder of your project. See [Datafiles](/docs/forward/dev-reference/datafiles).

Connection files are not active in the local development environment. When using `tb dev`, Tinybird doesn't check connections to external services like Kafka or S3 or ingest any data through them. Connections only become active after deploying to Local or Cloud environments.

## Structure (https://www.tinybird.co/docs/forward/dev-reference/datafiles/connection-files#structure)

All .connection file contains a `TYPE` declaration that specifies the type of connection. For each connection type, a number of settings are mandatory.

The name of the .connection file determines the name used in .datasource files when defining the connection.

The following example shows a .connection file for a Kafka data source:

kafkasample.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS {{ tb_secret("PRODUCTION_KAFKA_SERVERS", "localhost:9092") }}
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("PRODUCTION_KAFKA_USERNAME", "") }}
KAFKA_SECRET {{ tb_secret("PRODUCTION_KAFKA_PASSWORD", "") }}
```
The calls to the `tb_secret` function contain the name of the secret and a default value. See [Templating language](/docs/forward/core-concepts/templating-language#secrets) for how secrets are rendered in datafiles, and [tb secret](/docs/forward/dev-reference/commands/tb-secret) to learn how to set and manage secrets.

Never add credentials as plain text in your datafiles. Use secrets instead.

## List your connections (https://www.tinybird.co/docs/forward/dev-reference/datafiles/connection-files#list-your-connections)

To list your connections, run the [tb connections](/docs/forward/dev-reference/commands/tb-connection) command:

```
tb connection ls
```

## Update a connection (https://www.tinybird.co/docs/forward/dev-reference/datafiles/connection-files#update-a-connection)

To update or refresh a connection after changing settings or secrets, redeploy your project. See [Deployments](/docs/forward/core-concepts/deployments). Connections remain active using the previous settings until you deploy them again.

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/typescript-sdk-cli
Last update: 2026-06-01T10:20:05.000Z
Content:
---
title: "TypeScript SDK CLI commands \| Tinybird"
theme-color: "#171612"
description: "Reference for the Tinybird TypeScript SDK CLI commands."
inkeep:version: "forward"
---

# TypeScript SDK CLI commands (https://www.tinybird.co/docs/forward/dev-reference/commands/typescript-sdk-cli#typescript-sdk-cli-commands)

The TypeScript SDK ships with a `tinybird` CLI. Install it globally to run `tinybird` directly, or use `npx` for one-off usage. It manages TypeScript-based Tinybird projects, syncs resources, and deploys them to Tinybird.

## Install (https://www.tinybird.co/docs/forward/dev-reference/commands/typescript-sdk-cli#install)

```
pnpm add -g @tinybirdco/sdk
```
Or install it in your project and run it with `npx tinybird`:

```
pnpm install @tinybirdco/sdk
```

## Commands (https://www.tinybird.co/docs/forward/dev-reference/commands/typescript-sdk-cli#commands)

### `tinybird init` (https://www.tinybird.co/docs/forward/dev-reference/commands/typescript-sdk-cli#tinybird%2520init)

Scaffolds a new TypeScript project, authenticates you if needed, and can optionally add CI/CD workflow files.

```
tinybird init
tinybird init --force
tinybird init --skip-login
```

### `tinybird dev` (https://www.tinybird.co/docs/forward/dev-reference/commands/typescript-sdk-cli#tinybird%2520dev)

Watch for changes and sync to Tinybird. Defaults to a feature branch; use `--local` to work against Tinybird Local.

```
tinybird dev
tinybird dev --local
tinybird dev --branch my_feature_branch
```

### `tinybird build` (https://www.tinybird.co/docs/forward/dev-reference/commands/typescript-sdk-cli#tinybird%2520build)

Build and push resources to a Tinybird branch. Use this in CI for preview environments.

```
tinybird build
tinybird build --dry-run
tinybird build --local
tinybird build --branch my_feature_branch
```

### `tinybird deploy` (https://www.tinybird.co/docs/forward/dev-reference/commands/typescript-sdk-cli#tinybird%2520deploy)

Deploy resources to your main Tinybird workspace (production).

```
tinybird deploy
tinybird deploy --dry-run
tinybird deploy --check
tinybird deploy --allow-destructive-operations
```

### `tinybird preview` (https://www.tinybird.co/docs/forward/dev-reference/commands/typescript-sdk-cli#tinybird%2520preview)

Creates or replaces an ephemeral preview branch and deploys the project resources to it. Designed for CI pipelines. See [Preview deployments](/docs/forward/development-workflow/cicd#preview-deployments) for details.

```
tinybird preview
tinybird preview --dry-run
tinybird preview --check
tinybird preview --name custom_branch_name
tinybird preview --local
```

### `tinybird migrate` (https://www.tinybird.co/docs/forward/dev-reference/commands/typescript-sdk-cli#tinybird%2520migrate)

Migrate `.datasource`, `.pipe`, and `.connection` files into TypeScript definitions.

```
tinybird migrate "tinybird/**/*.datasource" "tinybird/**/*.pipe" "tinybird/**/*.connection"
tinybird migrate tinybird/legacy --out ./tinybird.migration.ts
tinybird migrate tinybird --dry-run
```

### `tinybird pull` (https://www.tinybird.co/docs/forward/dev-reference/commands/typescript-sdk-cli#tinybird%2520pull)

Download cloud resources as Tinybird datafiles.

```
tinybird pull
tinybird pull --output-dir ./tinybird-datafiles
tinybird pull --force
```

### `tinybird login` (https://www.tinybird.co/docs/forward/dev-reference/commands/typescript-sdk-cli#tinybird%2520login)

Authenticate with Tinybird and write credentials to `.env.local`.

```
tinybird login
```

### `tinybird branch` (https://www.tinybird.co/docs/forward/dev-reference/commands/typescript-sdk-cli#tinybird%2520branch)

Manage Tinybird branches.

```
tinybird branch list
tinybird branch status
tinybird branch delete my_feature_branch
```

### `tinybird info` (https://www.tinybird.co/docs/forward/dev-reference/commands/typescript-sdk-cli#tinybird%2520info)

Display information about the current project and workspace.

```
tinybird info
tinybird info --json
```

## Next steps (https://www.tinybird.co/docs/forward/dev-reference/commands/typescript-sdk-cli#next-steps)

- Follow the TypeScript SDK quickstart. See [Quick start: TypeScript SDK](/docs/forward/quickstarts/typescript-sdk).

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb
Last update: 2026-03-09T11:44:44.000Z
Content:
---
title: "tb \| Tinybird"
theme-color: "#171612"
description: "Root command for the Tinybird CLI"
inkeep:version: "forward"
---

# tb (https://www.tinybird.co/docs/forward/dev-reference/commands/tb#tb)

`tb` is the root command for the Tinybird CLI.

## Usage (https://www.tinybird.co/docs/forward/dev-reference/commands/tb#usage)

Run `tb` without a subcommand to see help:

```
tb
```
Use subcommands for specific actions, for example:

```
tb init
tb build
tb deploy
tb test run
```
`tb --prompt` and interactive AI mode are no longer available.

## Common root options (https://www.tinybird.co/docs/forward/dev-reference/commands/tb#common-root-options)

Global options are passed before the subcommand. For example:

```
tb --cloud deploy
tb --branch=my_feature build
tb --output json deployment ls
```

## Examples (https://www.tinybird.co/docs/forward/dev-reference/commands/tb#examples)

```
tb --help
tb --version
tb init --type cli --dev-mode manual
tb --cloud deploy
```
For the full list of global flags, see [Global options](/docs/forward/dev-reference/commands/global-options).

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes | Selects Tinybird Local as the target for the subcommand. |
| `--cloud` | ✓ Yes | Selects Tinybird Cloud as the target for the subcommand. |
| `--branch=BRANCH_NAME` | ✓ Yes | Selects a Tinybird branch as the target for the subcommand. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-workspace
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "tb workspace \| Tinybird"
theme-color: "#171612"
description: "Use tb workspace to list, inspect, create, switch, and manage Tinybird Cloud Workspaces from the CLI."
inkeep:version: "forward"
---

# tb workspace (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-workspace#tb-workspace)

Manage your workspaces. Global options apply to this command. See [Global options](/docs/forward/dev-reference/commands/global-options).

`tb workspace` is cloud-only. Workspace operations run against Tinybird Cloud account/workspace metadata.

The following subcommands are available:

| Subcommand | Description |
| --- | --- |
| create [OPTIONS] WORKSPACE_NAME | Creates a new workspace for your Tinybird user. |
| current | Shows the workspace you're currently authenticated to. |
| delete [OPTIONS] WORKSPACE_NAME_OR_ID | Deletes a workspace where you are an admin. |
| ls | Lists all the workspaces you have access to in the account you're currently authenticated to. |
| members add [OPTIONS] MEMBERS_EMAILS | Adds members to the current workspace. |
| members ls | Lists members in the current workspace. |
| members rm MEMBERS_EMAILS | Removes members from the current workspace. |
| members set-role [viewer\|admin] MEMBERS_EMAILS | Sets the role for existing workspace members. |
| use WORKSPACE_NAME_OR_ID | Switches to another workspace. Use `tb workspace ls` to list the workspaces you have access to. |

## tb workspace create (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-workspace#tb-workspace-create)

Creates a new workspace for your Tinybird user.

| Option | Description |
| --- | --- |
| --organization-id TEXT | Creates the new workspace in the specified organization. |

## tb workspace delete (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-workspace#tb-workspace-delete)

Deletes a workspace where you are an admin.

| Option | Description |
| --- | --- |
| --yes | Don't ask for confirmation. |

## tb workspace members add (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-workspace#tb-workspace-members-add)

Adds members to the current workspace. Takes a list of members emails as an argument.

| Option | Description |
| --- | --- |
| --role [viewer\|admin] | Sets the role for the members. |
| --user_token TEXT | Override local session with this authentication token. |

## tb workspace members rm (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-workspace#tb-workspace-members-rm)

Removes members from the current workspace. Takes a list of members emails as an argument.

| Option | Description |
| --- | --- |
| --user_token TEXT | Override local session with this authentication token. |

## tb workspace members set-role (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-workspace#tb-workspace-members-set-role)

Sets the role for existing workspace members.

| Option | Description |
| --- | --- |
| --user_token TEXT | Override local session with this authentication token. |

## tb workspace use (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-workspace#tb-workspace-use)

Switches to another workspace. Use `tb workspace ls` to list the workspaces you have access to.

| Option | Description |
| --- | --- |
| --user_token TEXT | Prevents Tinybird from prompting for the authentication Token. |

## Deprecated alias (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-workspace#deprecated-alias)

`tb workspace clear` is deprecated. Use `tb branch clear` instead.

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-workspace#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✗ No | Workspace commands are cloud-only. |
| `--cloud` | ✓ Yes | Manages workspaces in Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✗ No | Workspace commands are not branch-scoped. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-update
Last update: 2025-12-18T09:52:50.000Z
Content:
---
title: "tb update \| Tinybird"
theme-color: "#171612"
description: "Update the Tinybird CLI to the latest version"
inkeep:version: "forward"
---

# tb update (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-update#tb-update)

Updates the CLI to the latest version. Requires `tb local start`.

```
tb update

» Updating Tinybird CLI...
✓ Tinybird CLI updated
```
If you can't update for some reason, manually reinstall the CLI:

- For macOS and Linux
- For Windows

```
curl https://tinybird.co | sh
```

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-update#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✗ No | Updates the CLI binary globally. |
| `--cloud` | ✗ No | Updates the CLI binary globally. |
| `--branch=BRANCH_NAME` | ✗ No | Updates the CLI binary globally. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-token
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "tb token \| Tinybird"
theme-color: "#171612"
description: "Create and manage your workspace tokens."
inkeep:version: "forward"
---

# tb token (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-token#tb-token)

Manage your workspace tokens. See [Authentication](/docs/forward/core-concepts/tokens).

| Subcommand | Description |
| --- | --- |
| copy OPTIONS TOKEN_ID | Copies a token. |
| ls OPTIONS | Lists tokens. Use `--match TEXT` to retrieve any token matching the pattern. For example, `--match _test`. |
| refresh OPTIONS TOKEN_ID | Refreshes a token. Adding `--yes` removes the need for confirmation. |
| rm OPTIONS TOKEN_ID | Removes a token. Adding `--yes` removes the need for confirmation. |
| scopes OPTIONS TOKEN_ID | Lists token scopes. |
| create static OPTIONS TOKEN_NAME | Creates a static token that lasts forever. If a token with the same name already exists, it updates it. |
| create jwt OPTIONS TOKEN_NAME | Creates a JWT token with a fixed expiration time. |

## tb token create static (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-token#tb-token-create-static)

Creates a static token. Tinybird automatically generates [resource-scoped tokens](/docs/forward/core-concepts/static-tokens) when you define and deploy them in a `.datasource` file. This command provides an alternative method to generating resource-scoped tokens.

For example:

```
tb token create static my_static_token --scope ORG_DATASOURCES:READ
```
The following options are available:

| Option | Description |
| --- | --- |
| --scope | Scope for the token, for example `DATASOURCES:READ`. Required. |

## tb token create jwt (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-token#tb-token-create-jwt)

Creates a JWT token.

For example:

```
tb token create jwt my_jwt --ttl 1h --scope PIPES:READ --resource my_pipe --fixed-params "param_name=value"
```
The following options are available:

| Option | Description |
| --- | --- |
| --ttl | Time to live. For example, `1h`, `30min`, `1d`. Required. |
| --scope | Scope for the token. Only `PIPES:READ` is allowed for JWT tokens. Required. |
| --resource | Resource you want to associate the scope with. Required |
| --fixed-params | Fixed parameters in `key=value` format. You can separate multiple values using commas. |

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-token#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Manages tokens for Tinybird Local. |
| `--cloud` | ✓ Yes | Manages tokens in Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✓ Yes | Manages tokens in a branch. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-test
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "tb test \| Tinybird"
theme-color: "#171612"
description: "Run and update tests for your Tinybird project"
inkeep:version: "forward"
---

# tb test (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-test#tb-test)

Runs and updates tests for your Tinybird project.

The following subcommands are available:

| Subcommand | Description |
| --- | --- |
| run [FILES] | Runs all the tests in the project or specific tests passed as arguments. |
| update [FILES] | Updates the test's expectations. |

## Create test files (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-test#create-test-files)

Create test YAML files manually in the `tests/` folder.

For the test file format, see [Test files](/docs/forward/dev-reference/datafiles/test-files).

Example `tests/user_actions.yaml`:

```
- name: user_actions_clicked
  description: Returns clicked actions only
  expected_http_status: 200
  parameters: action=CLICKED
  expected_result: |
    {"action":"CLICKED"}
```
Use `tb test update` to refresh expected results after changing a query.

## tb test run (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-test#tb-test-run)

Runs all the tests in the project `tb test run` or specific tests passed as arguments. Tests are run with fixture data against your local.

For example, `tb test run tests/get_lego_sets_by_year.yaml` returns:

```
Running against Tinybird Local

» Building test environment
✓ Done!

» Running tests
* get_sets_by_theme_year.yaml
✓ get_sets_by_theme_year_2000s_range passed
✓ get_sets_by_theme_year_1990s_range passed
```
Tinybird creates a fresh workspace for each test run. Secrets don't persist between test runs. To avoid test failures, add a default value to your secrets: `{{ tb_secret("secret_name", "default_value") }}`.

## tb test update (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-test#tb-test-update)

Updates the test's expectations. For example: `tb test update get_lego_sets_by_year`.

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-test#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Runs tests locally with fixture data. |
| `--cloud` | ✗ No | Tests run locally, not in cloud. |
| `--branch=BRANCH_NAME` | ✗ No | Tests run locally, not in branches. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-sql
Last update: 2025-12-18T09:52:50.000Z
Content:
---
title: "tb sql \| Tinybird"
theme-color: "#171612"
description: "Run SQL queries over data sources and pipes"
inkeep:version: "forward"
---

# tb sql (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-sql#tb-sql)

Run SQL query over data sources and pipes. Global options apply to this command. See [Global options](/docs/forward/dev-reference/commands/global-options).

## Options (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-sql#options)

| Option | Description |
| --- | --- |
| --rows_limit INTEGER | Max number of rows retrieved. |
| --pipeline TEXT | The name of the pipe to run the SQL Query. |
| --pipe TEXT | The path to the .pipe file to run the SQL Query of a specific NODE. |
| --node TEXT | The NODE name. |
| --stats / --no-stats | Shows query stats. |

## Examples (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-sql#examples)

The following example shows how to run a SELECT statement with count:

```
tb sql "SELECT count(*) from tinybird.endpoint_errors"

Running against Tinybird Local
  count()
   UInt64
───────────
        0
```
The following example shows how to use the `--rows_limit` and `--stats` options:

```
tb sql --rows_limit 5 --stats "SELECT start_datetime, duration, pipe_name from tinybird.pipe_stats_rt"

Running against Tinybird Local
** Query took 0.004044331 seconds
** Rows read: 142
** Bytes read: 10.12 KB
  start_datetime            duration   pipe_name
  DateTime                   Float32   String
──────────────────────────────────────────────────
  2025-03-12 16:07:16     0.05388069   query_api
──────────────────────────────────────────────────
  2025-03-12 16:07:27   0.0040593147   query_api
──────────────────────────────────────────────────
  2025-03-12 16:09:03    0.026257038   query_api
──────────────────────────────────────────────────
  2025-03-12 16:15:27     0.03177452   query_api
──────────────────────────────────────────────────
  2025-03-12 16:15:33    0.010550499   query_api
```

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-sql#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Runs SQL queries locally. |
| `--cloud` | ✓ Yes | Runs SQL queries in Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✓ Yes | Runs SQL queries in a branch. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-sink
Last update: 2025-12-18T09:52:50.000Z
Content:
---
title: "tb sink \| Tinybird"
theme-color: "#171612"
description: "Manage sink pipes in your Tinybird project"
inkeep:version: "forward"
---

# tb sink (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-sink#tb-sink)

Manages sink pipes. Global options apply to this command. See [Global options](/docs/forward/dev-reference/commands/global-options).

The following subcommands are available:

| Subcommand | Description |
| --- | --- |
| ls | Lists all the sink pipes. |
| run [OPTIONS] PIPE_NAME_OR_ID | Runs a sink job. |

## tb sink run (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-sink#tb-sink-run)

Runs a sink job.

| Option | Description |
| --- | --- |
| --wait / --no-wait | Waits for the sink job to finish. |
| --mode [append\|replace] | Defines the sink strategy. |
| --param TEXT | Key and value of the params you want the sink pipe to be called with. For example: tb sink run <my_sink_pipe> --param foo=bar |

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-sink#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Manages sink pipes locally. |
| `--cloud` | ✓ Yes | Manages sink pipes in Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✓ Yes | Manages sink pipes in a branch. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-secret
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "tb secret \| Tinybird"
theme-color: "#171612"
description: "Use tb secret to manage secrets in Tinybird datafiles, set Workspace credentials, and reference sensitive values with tb_secret()."
inkeep:version: "forward"
---

# tb secret (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-secret#tb-secret)

Manage secrets in datafiles, like connection credentials. Secrets consists of a name and a value.

You can add a secret to your workspace like this:

```
tb --cloud secret set KAFKA_USERNAME 12345
```
You can then use the secret in a datafile like this:

```
TYPE kafka
KAFKA_KEY {{ tb_secret("KAFKA_USERNAME", "") }}
```
In the datafile syntax, first argument is the name of the secret, and the second is the default value that's used when the secret is not set.

Secrets are only replaced in your resources when you deploy. If you change a secret, you need to deploy for the changes to take effect.

## Environment variables (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-secret#environment-variables)

When working in Tinybird Local, you can store secrets in `.env.local` or `.env` files. Tinybird loads them automatically when you run `tb dev` or `tb build`.

```
KAFKA_USERNAME=12345
KAFKA_PASSWORD=67890
```

## Subcommands (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-secret#subcommands)

The following subcommands are available:

| Subcommand | Description |
| --- | --- |
| ls | Lists all secrets in the project. |
| rm NAME | Deletes a secret. |
| set NAME [VALUE] | Creates or updates a secret. If the command doesn't include the value, it prompts you for one. |

## tb secret ls (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-secret#tb-secret-ls)

Lists secrets.

| Option | Description |
| --- | --- |
| --match TEXT | Retrieves any resource matching the pattern. |

## tb secret set (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-secret#tb-secret-set)

| Option | Description |
| --- | --- |
| --multiline | Opens an editor to enter a multiline value |

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-secret#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Manages secrets locally. Can use `.env` files. |
| `--cloud` | ✓ Yes | Manages secrets in Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✓ Yes | Manages secrets in a branch. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-pull
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "tb pull \| Tinybird"
theme-color: "#171612"
description: "Download the latest version of your datafiles from a Tinybird Workspace to your local project."
inkeep:version: "forward"
---

# tb pull (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-pull#tb-pull)

Download the latest version of your datafiles from a Tinybird Workspace to your local project. If local files have been modified, the command prompts you to overwrite them.

## Example (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-pull#example)

```
tb pull
```

## Options (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-pull#options)

| Option | Description |
| --- | --- |
| --only-vendored | Only download and update vendored files. |
| -f, --force | Overwrite existing files without prompting for confirmation. |
| --fmt | Format files before saving. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-pipe
Last update: 2026-05-27T12:35:45.000Z
Content:
---
title: "tb pipe \| Tinybird"
theme-color: "#171612"
description: "Use tb pipe to list, inspect, create, unlink, populate, copy, and manage Tinybird Pipes and API Endpoints from the CLI."
inkeep:version: "forward"
---

# tb pipe (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-pipe#tb-pipe)

Manage your pipes. Global options apply to this command. See [Global options](/docs/forward/dev-reference/commands/global-options).

The following subcommands are available:

| Subcommand | Description |
| --- | --- |
| ls | Lists all the pipes you have access to. |

## tb pipe ls (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-pipe#tb-pipe-ls)

Lists all the pipes you have access to.

| Option | Description |
| --- | --- |
| --match TEXT | Retrieves any resource matching the pattern. For example, `--match _test`. |
| --format [json] | Returns the results in the specified format. |

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-pipe#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Manages pipes locally. |
| `--cloud` | ✓ Yes | Manages pipes in Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✓ Yes | Manages pipes in a branch. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-open
Last update: 2025-12-18T09:52:50.000Z
Content:
---
title: "tb open \| Tinybird"
theme-color: "#171612"
description: "Open a Tinybird workspace in your browser."
inkeep:version: "forward"
---

# tb open (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-open#tb-open)

Opens a workspace in Tinybird Cloud. You can get your current workspace by running `tb workspace current`. Global options apply to this command. See [Global options](/docs/forward/dev-reference/commands/global-options).

For example:

```
# Opens the current workspace
tb open

# Opens a specific workspace in the cloud environment
tb --cloud open --workspace someworkspace
```
The following options are available:

| Option | Description |
| --- | --- |
| --workspace | Sets the workspace you want to open. If unset, your current workspace is used. |

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-open#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Opens Tinybird UI for Tinybird Local. |
| `--cloud` | ✓ Yes | Opens Tinybird UI for Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✓ Yes | Opens Tinybird UI for a branch. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-mock
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "tb mock (removed) \| Tinybird"
theme-color: "#171612"
description: "The tb mock command has been removed."
inkeep:version: "forward"
---

# tb mock (removed) (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-mock#tb-mock-removed)

`tb mock` has been removed.

Use one of these alternatives:

- Create fixture files manually in the `fixtures/` folder.
- Append sample data using `tb datasource append`.
For testing workflows, see [Test your project](/docs/forward/guides/test-your-project) and [tb test](/docs/forward/dev-reference/commands/tb-test).

## Examples (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-mock#examples)

Create a fixture file manually:

```
cat > fixtures/user_actions.ndjson <<'EOF'
{"timestamp":"2026-01-01 00:00:00","action":"CLICKED","user_id":"u_1"}
{"timestamp":"2026-01-01 00:01:00","action":"VIEWED","user_id":"u_2"}
EOF
```
Append sample data directly:

```
tb datasource append user_actions --file fixtures/user_actions.ndjson
```

## Options (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-mock#options)

| Option | Description |
| --- | --- |
| N/A | Command removed. |

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-mock#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✗ No | Command removed. |
| `--cloud` | ✗ No | Command removed. |
| `--branch=BRANCH_NAME` | ✗ No | Command removed. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-materialization
Last update: 2025-12-18T09:52:50.000Z
Content:
---
title: "tb materialization \| Tinybird"
theme-color: "#171612"
description: "Manage materialized views in your Tinybird project"
inkeep:version: "forward"
---

# tb materialization (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-materialization#tb-materialization)

Manage materialized views. Global options apply to this command. See [Global options](/docs/forward/dev-reference/commands/global-options).

The following subcommands are available:

| Subcommand | Description |
| --- | --- |
| ls | Lists materialized views. |

## tb materialization ls (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-materialization#tb-materialization-ls)

Lists materialized views.

| Option | Description |
| --- | --- |
| --match TEXT | Retrieves any resource matching the pattern. |
| --format [json] | Returns the results in the specified format. |

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-materialization#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Lists materialized views locally. |
| `--cloud` | ✓ Yes | Lists materialized views in Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✓ Yes | Lists materialized views in a branch. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-logout
Last update: 2026-05-27T12:35:45.000Z
Content:
---
title: "tb logout \| Tinybird"
theme-color: "#171612"
description: "Use tb logout to clear the Tinybird credentials stored in your project directory and end the current CLI session safely."
inkeep:version: "forward"
---

# tb logout (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-logout#tb-logout)

Logs you out of your Tinybird session in the project directory you're in.

```
tb logout

» Logging out from <workspace>...
✓ Logged out!
```
Logging out erases all the credentials stored in the `.tinyb` file. See [.tinyb](/docs/forward/dev-reference/datafiles/tinyb-file).

To log back in, run `tb login`.

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-logout#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✗ No | Logout is global and erases all credentials. |
| `--cloud` | ✗ No | Logout is global and erases all credentials. |
| `--branch=BRANCH_NAME` | ✗ No | Logout is global and erases all credentials. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-login
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "tb login \| Tinybird"
theme-color: "#171612"
description: "Authenticate with your Tinybird account"
inkeep:version: "forward"
---

# tb login (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-login#tb-login)

Authenticates with your Tinybird account.

By default, the command opens a browser window for authentication. You can also use the `--method code` option to authenticate using a temporary code instead of a browser.

The credentials are stored in the `.tinyb` file. See [.tinyb file](/docs/forward/dev-reference/datafiles/tinyb-file).

## Options (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-login#options)

| Option | Description |
| --- | --- |
| --host TEXT | Set a custom host if it's different than https://api.tinybird.co. Check [Regions and endpoints](/docs/api-reference#regions-and-endpoints) for the available list of regions. |
| --auth-host TEXT | Set the host to authenticate to. If unset, the default host is used. |
| --workspace TEXT | Set the workspace to authenticate to. If unset, the default workspace is used. |
| -i, --interactive | Show available regions and prompts you to select one for authentication. |
| --method [browser\|code] | Set the authentication method to use. Default: browser. |
| --use-aws-creds | Use AWS credentials to authenticate. Required to run S3 connections locally. See [Using AWS credentials](/docs/forward/ingest-data/connectors/s3#using-s3-connector-local-environment). |
To log out, run `tb logout`.

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-login#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✗ No | Authentication is global and applies to all environments. |
| `--cloud` | ✗ No | Authentication is global and applies to all environments. |
| `--branch=BRANCH_NAME` | ✗ No | Authentication is global and applies to all environments. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-local
Last update: 2025-12-18T09:52:50.000Z
Content:
---
title: "tb local \| Tinybird"
theme-color: "#171612"
description: "Manage your local development environment"
inkeep:version: "forward"
---

# tb local (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-local#tb-local)

Manages the local development environment.

The following subcommands are available:

| Subcommand | Description | Arguments |
| --- | --- | --- |
| generate-tokens | Generates random default user and workspace tokens for use in dev, CI/CD and testing. | None |
| start | Starts the local development environment. Run with token args to set default values for `TB_LOCAL_USER_TOKEN` and `TB_LOCAL_WORKSPACE_TOKEN`. | `--user-token=<USER_TOKEN>`

`--workspace-token=<WORKSPACE_TOKEN>` |
| restart | Restarts the local development environment. | None |
| stop | Stops the local development environment. | None |
| remove | Removes the local development environment. |  |

## tb local start (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-local#tb-local-start)

Starts the local development environment.

| Option | Description |
| --- | --- |
| --watch | Watches the logs of Tinybird Local while it is running. |

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-local#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✗ No | This command manages Tinybird Local itself. |
| `--cloud` | ✗ No | This command manages Tinybird Local itself. |
| `--branch=BRANCH_NAME` | ✗ No | This command manages Tinybird Local itself. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-job
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "tb job \| Tinybird"
theme-color: "#171612"
description: "Manage jobs in your Tinybird project"
inkeep:version: "forward"
---

# tb job (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-job#tb-job)

Manage jobs. Global options apply to this command. See [Global options](/docs/forward/dev-reference/commands/global-options).

The following subcommands are available:

| Subcommand | Description |
| --- | --- |
| cancel JOB_ID | Tries to cancel a job. |
| details JOB_ID | Gets details for any job created in the last 48 hours. |
| ls [OPTIONS] | Lists jobs. Use `--status [waiting\|working\|done\|error]` or `-s` to show results with the desired status. |

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-job#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Manages jobs locally. |
| `--cloud` | ✓ Yes | Manages jobs in Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✓ Yes | Manages jobs in a branch. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-init
Last update: 2026-03-09T11:44:44.000Z
Content:
---
title: "tb init \| Tinybird"
theme-color: "#171612"
description: "Initialize a new Tinybird project."
inkeep:version: "forward"
---

# tb init (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-init#tb-init)

Initializes a new Tinybird project.

Use this command to scaffold your project structure, set development mode, and optionally generate CI/CD templates.

```
tb init
```

## Options (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-init#options)

| Option | Description |
| --- | --- |
| --type [typescript\|python\|cli] | Project type. |
| --dev-mode [branch\|local\|manual] | Development mode for build/deploy routing. |
| --folder TEXT | Project folder. Default is `tinybird`. |
| --cicd [github\|gitlab\|skip] | Generates CI/CD templates for GitHub or GitLab. |
| --skip-login | Skips login prompt at the end of initialization. |

## Example (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-init#example)

```
tb init --type cli --dev-mode manual --folder . --cicd github
```

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-init#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (implicit) | Initializes local project files. |
| `--cloud` | ✗ No | Not supported. |
| `--branch=BRANCH_NAME` | ✗ No | Not supported. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-infra
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "tb infra \| Tinybird"
theme-color: "#171612"
description: "Select the infrastructure on which you want to run Tinybird."
inkeep:version: "forward"
---

# tb infra (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-infra#tb-infra)

Helps you deploy Tinybird on a cloud service provider of your choice and manage custom regions. See [Self-managed](/docs/forward/core-concepts/self-managed).

To list, add, update, or remove infrastructure regions, you can also go to **Settings**, **Managed regions** in Tinybird Cloud.

## Subcommands (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-infra#subcommands)

The following subcommands are available:

| Subcommand | Description |
| --- | --- |
| init | Initializes the infrastructure and deploys Tinybird Local in a custom region. By default, the command is interactive and prompts you for the necessary information. |
| add | Adds a new self-managed infrastructure region using an existing infrastructure URL. |
| update | Updates the URL of an existing self-managed infrastructure region. |
| rm NAME | Removes an existing infrastructure region. |
| ls | Lists available infrastructure regions. |

## tb infra init (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-infra#tb-infra-init)

Initializes the selected cloud infrastructure provider to host Tinybird Local.

Running the `tb infra init` command requires the following components:

- AWS
The following tools are required to deploy Tinybird Local on AWS:

- [Terraform CLI](https://developer.hashicorp.com/terraform/install) to create the Kubernetes cluster.
- [Kubectl CLI](https://kubernetes.io/docs/tasks/tools/) to manage your Kubernetes installation.
- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) with credentials configured.
Before initiating deployment, you need to set up the following in AWS:

- A zone and domain name in [Route53 zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/CreatingHostedZone.html).

  - Write down the hosted zone name and the zone ID for your domain.
- An [EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html) with the following components:

  - [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/) installed.
  - [external-dns](https://github.com/kubernetes-sigs/external-dns) configured.
  - Both components need sufficient permissions to manage resources.
  - Optionally, set a storage class in the EKS cluster with high IOPS and throughput.
The command requires the following information:

- AWS
When prompted, enter the following information from the first step:

- AWS region. For example, `us-east-1`.
- DNS zone name. For example, `example.com`.
- DNS record name. For example, `tinybird`
- The Kubernetes context to apply the changes on.
- Kubernetes namespace. For example, `default`.
- EKS storage class. For example, `gp3-encrypted`.
Review the changes before applying them. If you want to generate the files without applying them, use the `--skip-apply` flag.

After the deployment is complete, `tb infra` shows the URL to access your Tinybird Local instance. See [tb infra](/docs/forward/dev-reference/commands/tb-infra) for more information and settings.

The following options are available:

| Option | Description |
| --- | --- |
| --name TEXT | Name for identifying the self-managed infrastructure region in Tinybird. |
| --cloud-provider TEXT | Infrastructure provider. Possible values are: `aws`, `gcp`, `azure`. |
| --cloud-region TEXT | AWS region, when using `aws` as the provider. |
| --dns-zone-name TEXT | DNS zone name. |
| --kubernetes-namespace TEXT | Kubernetes namespace for the deployment. |
| --dns-record TEXT | DNS record name to create, without domain. For example, `tinybird`. |
| --kubernetes-storage-class TEXT | Storage class for the Kubernetes StatefulSet. |
| --skip-apply | Create all the configuration files without applying them. |
| --auto-apply | Apply all the configuration automatically, without prompting for confirmation. |
The command always generates the following files inside the `infra/<provider>` directory, where `<provider>` is the infrastructure provider:

- config.json: Contains the configuration for the provider.
- k8s.yaml: Contains the Kubernetes configuration.
- main.tf: Contains the Terraform configuration.

## tb infra add (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-infra#tb-infra-add)

Adds a new self-managed infrastructure region using an existing infrastructure URL. The command prompts you for the necessary information.

For example:

```
tb infra add

Running against Tinybird Cloud: Workspace example_workspace
Enter name: example
Enter host: https://tinybird.example.com
» Adding infrastructure 'example' in Tinybird...
✓ Infrastructure 'example' added
» Required environment variables:
TB_INFRA_TOKEN=example_token
TB_INFRA_WORKSPACE=example_workspace
TB_INFRA_ORGANIZATION=example_organization
TB_INFRA_USER=user@example.com
```
The following options are available:

| Option | Description |
| --- | --- |
| --name TEXT | Name for identifying the self-managed infrastructure in Tinybird. |
| --host TEXT | Host for the infrastructure. |

## tb infra update (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-infra#tb-infra-update)

Updates an existing infrastructure region.

For example:

```
tb infra update example --host https://tinybird-2.example.com

Running against Tinybird Cloud: Workspace example_workspace
» Updating infrastructure 'test' in Tinybird...
```
The following options are available:

| Option | Description |
| --- | --- |
| --name TEXT | Name for identifying the self-managed infrastructure in Tinybird. |
| --host TEXT | Host for the infrastructure. |

## tb infra rm (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-infra#tb-infra-rm)

Removes an existing infrastructure region.

For example:

```
tb infra rm example

Running against Tinybird Cloud: Workspace example_workspace
» Deleting infrastructure 'example' from Tinybird...
✓ Infrastructure 'example' deleted
```

## tb infra ls (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-infra#tb-infra-ls)

Lists all the infrastructure regions.

For example:

```
tb infra ls

Running against Tinybird Cloud: Workspace example_workspace
** Infras:
--------------------------------------------------------------------------
| name         | host                                                    |
--------------------------------------------------------------------------
| example      | https://tinybird.example.com                            |
| example2     | https://tinybird2.example.com                           |
--------------------------------------------------------------------------
```

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-infra#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✗ No | Infrastructure management requires cloud. |
| `--cloud` | ✓ Yes (implicit) | Manages self-managed infrastructure regions. |
| `--branch=BRANCH_NAME` | ✗ No | Infrastructure is managed at account level, not in branches. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-info
Last update: 2025-12-18T09:52:50.000Z
Content:
---
title: "tb info \| Tinybird"
theme-color: "#171612"
description: "Get information about your authentication in use"
inkeep:version: "forward"
---

# tb info (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-info#tb-info)

Displays authentication details stored for Tinybird Cloud and Tinybird Local, along with the current project path configuration. The credentials are stored in the `.tinyb` file. See [.tinyb file](/docs/forward/dev-reference/datafiles/tinyb-file).

For example:

```
tb info

» Tinybird Cloud:
--------------------------------------------------------------------------------------------
user: tinybird@domain.co
workspace_name: forward
workspace_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX
token: YOUR-ADMIN-TOKEN
user_token: YOUR-USER-TOKEN
api: https://api.tinybird.co
ui: https://cloud.tinybird.co/gcp/europe-west2/forward
--------------------------------------------------------------------------------------------

» Tinybird Local:
--------------------------------------------------------------------------------------------
user: tinybird@domain.co
workspace_name: forward
workspace_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX
token: YOUR-LOCAL-ADMIN-TOKEN
user_token: YOUR-LOCAL-USER-TOKEN
api: http://localhost:7181
ui: http://cloud.tinybird.co/local/7181/forward
--------------------------------------------------------------------------------------------

» Project:
---------------------------------------------------
current: /path/to/your/project
.tinyb: /path/to/your/project/.tinyb
project: /path/to/your/project
---------------------------------------------------
```

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-info#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✗ No | Displays info for all environments. |
| `--cloud` | ✗ No | Displays info for all environments. |
| `--branch=BRANCH_NAME` | ✗ No | Displays info for all environments. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-fmt
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "tb fmt \| Tinybird"
theme-color: "#171612"
description: "Formats a .datasource or .pipe file"
inkeep:version: "forward"
---

# tb fmt (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-fmt#tb-fmt)

Formats a .datasource or .pipe file. .connection files are not *yet* supported.

These are the options available for the `fmt` command:

| Option | Description |
| --- | --- |
| --line-length INTEGER | A number indicating the maximum characters per line in the node SQL, lines split based on the SQL syntax and the number of characters passed as a parameter. |
| --dry-run | Don't ask to overwrite the local file. |
| --yes | Don't ask for confirmation to overwrite the local file. |
| --diff | Outputs correctly formatted block (if differs from local file) and prompts to apply correction to local file. |
This command applies opinionated formatting rules and may reorder file properties for consistency. For example, these properties:

```
ENGINE "MergeTree"
ENGINE_SORTING_KEY "name"
ENGINE_PARTITION_KEY "name"
```
The command formats them as follows:

```
ENGINE MergeTree
ENGINE_PARTITION_KEY name
ENGINE_SORTING_KEY name
```
This command removes comments starting with # from the file, so use DESCRIPTION or a comment block instead:

Example comment block
```
%
{% comment this is a comment and fmt keeps it %}

SELECT
  {% comment this is another comment and fmt keeps it %}
  count() c
FROM stock_prices_1m
```
You can add `tb fmt` to your git `pre-commit` hook to format your files. If the SQL formatting results aren't what you expect, you can exclude specific blocks from formatting. Read [how to exclude blocks from formatting](https://docs.sqlfmt.com/getting-started/disabling-sqlfmt).

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-endpoint
Last update: 2025-12-18T09:52:50.000Z
Content:
---
title: "tb endpoint \| Tinybird"
theme-color: "#171612"
description: "Manage endpoints in your Tinybird project"
inkeep:version: "forward"
---

# tb endpoint (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-endpoint#tb-endpoint)

Manage endpoints. Global options apply to this command. See [Global options](/docs/forward/dev-reference/commands/global-options).

The following subcommands are available:

| Subcommand | Description |
| --- | --- |
| data PIPE | Prints data returned by the endpoint. |
| ls [OPTIONS] | Lists all the endpoints. |
| stats [OPTIONS] PIPE | Prints stats of the last 7 days for an endpoint. |
| token PIPE | Retrieves a token to call an endpoint. |
| url PIPE | Prints the URL of an endpoint. |

## tb endpoint data (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-endpoint#tb-endpoint-data)

Prints data returned by the endpoint.

| Option | Description |
| --- | --- |
| --query TEXT | Runs a query over endpoint results. |
| --format [json\|csv] | Returns the results in the specified format. |

## tb endpoint ls (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-endpoint#tb-endpoint-ls)

Lists all the endpoints.

| Option | Description |
| --- | --- |
| --match TEXT | Retrieves any resource matching the pattern. For example, `--match _test`. |
| --format [json] | Returns the results in the specified format. |

## tb endpoint stats (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-endpoint#tb-endpoint-stats)

Prints stats of the last 7 days for an endpoint.

| Option | Description |
| --- | --- |
| --format [json] | Returns the results in the specified format. |

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-endpoint#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Manages endpoints locally. |
| `--cloud` | ✓ Yes | Manages endpoints in Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✓ Yes | Manages endpoints in a branch. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-dev
Last update: 2026-03-09T11:44:44.000Z
Content:
---
title: "tb dev \| Tinybird"
theme-color: "#171612"
description: "Build your Tinybird project and watch for changes"
inkeep:version: "forward"
---

# tb dev (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-dev#tb-dev)

Use `tb dev` to build your project and watch for changes. On each datafile change, the project rebuilds automatically.

For example:

```
tb dev

» Building project...
✓ datasources/user_actions.datasource created
✓ endpoints/user_actions_line_chart.pipe created
✓ endpoints/user_actions_total_widget.pipe created
✓ Rebuild completed in 0.2s

Watching for changes...
```
Run SQL queries and other commands in a separate terminal while `tb dev` is running:

```
tb sql "SELECT * FROM user_actions LIMIT 5"
tb endpoint ls
```

## Options (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-dev#options)

| Option | Description |
| --- | --- |
| --with-connections | Creates data linkers for connection data sources (S3, Kafka, GCS). |
| --no-connections | Disables connector linker creation during `tb dev`. |

## UI access (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-dev#ui-access)

When `tb dev` is running, Tinybird exposes your project in the Tinybird UI:

```
tb dev

» Exposing project to Tinybird UI...
* Access your project at https://cloud.tinybird.co/local/7181/your_workspace/project
```

## Developing with production data (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-dev#developing-with-production-data)

You can develop with production data by pointing to a branch created with the `--last-partition` flag:

```
tb branch create my_feature_branch --last-partition
```
Then run:

```
tb --branch=my_feature_branch dev
```

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-dev#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Starts a dev session in Tinybird Local. |
| `--cloud` | ✗ No | Dev sessions are not available in Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✓ Yes | Starts a dev session in a branch. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-deployment
Last update: 2026-02-17T18:51:22.000Z
Content:
---
title: "tb deployment \| Tinybird"
theme-color: "#171612"
description: "Deploy your project to the Tinybird platform"
inkeep:version: "forward"
---

# tb deployment (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-deployment#tb-deployment)

Deploys your project to the Tinybird platform. Run using `--cloud` to deploy to Tinybird Cloud.

Global options apply to this command. See [Global options](/docs/forward/dev-reference/commands/global-options).

The following subcommands are available:

| Subcommand | Description |
| --- | --- |
| create [OPTIONS] | Validates and deploys the project in Tinybird. |
| ls | Lists all the deployments of your project. |
| promote | Promotes last deploy to ready and remove old one. Accepts the --wait / --no-wait option. |
| discard | Discards the last deployment. Accepts the --wait / --no-wait option. |

## tb deployment create (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-deployment#tb-deployment-create)

Validates and deploys the project in Tinybird. Run using `--cloud` to deploy to Tinybird Cloud. When deploying to Tinybird Cloud, the command shows the URL of the deployment.

| Option | Description |
| --- | --- |
| --wait / --no-wait | Waits for the deployment to finish. Defaults to not waiting. |
| --auto / --no-auto | Automatically promotes the deployment. Only works if `--wait` is used. |
| --check / --no-check | Validates the deployment before creating it. Disabled by default. |
| --allow-destructive-operations / --no-allow-destructive-operations | Allows destructive operations, like removing data sources, dropping historical data from tables, and so on. Disabled by default. |
| --template TEXT | Deploy from a template name or URL. If a URL is provided, it should point to a valid Tinybird project template. |
Removing data sources is an irreversible operation. Be careful when using the `--allow-destructive-operations` flag.

The `--check` flag validates external connections to S3, Kafka, GCS, and databases referenced via table functions. For local success, set connection secrets with `tb secret set` and use `tb local start --use-aws-creds` for S3 connections.

## tb deployment discard (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-deployment#tb-deployment-discard)

Discards the current staging deployment. Use this command to cancel an unfinished deployment. The previous live deployment remains active.

When you run `tb deployment discard`, the following happens:

1. The current staging deployment is cancelled.
2. The deployment resources associated with the staging deployment are removed.
3. The previous live deployment remains active.

| Option | Description |
| --- | --- |
| --wait / --no-wait | Waits for the discard operation to finish. Defaults to not waiting. |
For example, if a deployment is taking a long time due to backfills, you can discard it:

```
tb deployment discard --wait
```
After discarding, the staging deployment and its resources are removed. Your project continues running on the previous live deployment.

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-deployment#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Manages deployments locally. |
| `--cloud` | ✓ Yes | Manages deployments in Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✓ Yes | Manages deployments in a branch. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-deploy
Last update: 2025-12-18T09:52:50.000Z
Content:
---
title: "tb deploy \| Tinybird"
theme-color: "#171612"
description: "Deploy your project to Tinybird Cloud."
inkeep:version: "forward"
---

# tb deploy (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-deploy#tb-deploy)

The `tb deploy` command is an alias of `tb deployment create --wait --auto`. Use `tb deploy` to create and promote a deployment to the Tinybird platform.

For example:

```
tb deploy
Running against Tinybird Local

» Changes to be deployed...

-----------------------------------------------------------------
| status   | name         | path                                |
-----------------------------------------------------------------
| modified | user_actions | datasources/user_actions.datasource |
-----------------------------------------------------------------

✓ Deployment submitted successfully
Deployment is ready
Setting candidate deployment as live
Removing old deployment
Deployment promotion successfully started
Deployment promoted successfully
```
See [tb deployment](/docs/forward/dev-reference/commands/tb-deployment) for more information.

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-deploy#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Deploys to Tinybird Local. |
| `--cloud` | ✓ Yes | Deploys to Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✓ Yes | Deploys to a branch. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-datasource
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "tb datasource \| Tinybird"
theme-color: "#171612"
description: "Manage data sources in your Tinybird project"
inkeep:version: "forward"
---

# tb datasource (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-datasource#tb-datasource)

Manages data sources. Global options apply to this command. See [Global options](/docs/forward/dev-reference/commands/global-options).

The following subcommands are available:

| Subcommand | Description |
| --- | --- |
| create [OPTIONS] | Creates a new .datasource file from a URL, local file or a connector. |
| analyze URL_OR_FILE | Analyzes a URL or a file before creating a new data source. |
| append DATASOURCE_NAME [OPTIONS] | Appends data to an existing data source from URL, local file or via Events API. For example, `tb datasource append my_datasource --url https://my_url.com`. |
| data | Prints data from a data source. |
| delete [OPTIONS] DATASOURCE_NAME | Deletes specific rows from a data source given a SQL condition. |
| export [OPTIONS] DATASOURCE_NAME | Exports data from a data source to a local file in CSV or NDJSON format. |
| ls [OPTIONS] | Lists data sources. |
| replace DATASOURCE_NAME URL | Replaces the data in a data source from a URL, local file or a connector. |
| start DATASOURCE_NAME | Starts Kafka ingestion for a Data Source. |
| stop DATASOURCE_NAME | Stops Kafka ingestion for a Data Source. |
| sync [OPTIONS] DATASOURCE_NAME | Syncs from connector defined in .datasource file. |
| truncate [OPTIONS] DATASOURCE_NAME | Truncates a data source. |

## tb datasource create (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-create)

Creates a new .datasource file. Opens a wizard if no arguments are provided.

| Option | Description |
| --- | --- |
| --name TEXT | Name of the data source |
| --blank | Create a blank data source |
| --file TEXT | Create a data source from a local file |
| --url TEXT | Create a data source from a remote URL |
| --connection-name TEXT | Create a data source from a connection |
| --s3 | Create a data source from a S3 connection |
| --gcs | Create a data source from a GCS connection |
| --kafka | Create a data source from a Kafka connection |

## tb datasource analyze (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-analyze)

Analyzes a URL or a local file before creating a new data source. It prints the column names, data type and nullable status of each column, and the SQL schema of the data file.

For example, `tb datasource analyze telemetry.ndjson` returns the following:

| name | type | nullable |
| --- | --- | --- |
| altitude | Float64 | false |
| latitude | Float32 | false |
| longitude | Float32 | false |
| name | String | false |
| timestamp | DateTime64 | false |

```
altitude Float64 `json:$.altitude`, latitude Float32 `json:$.latitude`, longitude Float32 `json:$.longitude`, name String `json:$.name`, timestamp DateTime64 `json:$.timestamp`
```

## tb datasource append (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-append)

Appends data to an existing data source from URL, local file or a connector.

| Option | Description |
| --- | --- |
| --url TEXT | URL to append data from |
| --file TEXT | Local file to append data from |
| --events TEXT | Events to append data from TEXT in NDJSON format |
| -h, --help | Explains append command and options |

## tb datasource data (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-data)

Prints data from a data source.

| Option | Description |
| --- | --- |
| --limit INTEGER | Limits the number of rows to return |

## tb datasource delete (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-delete)

Deletes rows from a data source with SQL condition. For example: `tb datasource delete [datasource_name] --sql-condition "country='ES'"`

| Option | Description |
| --- | --- |
| --yes | Does not ask for confirmation |
| --wait | Waits for delete job to finish |
| --dry-run | Runs the command without deleting anything |

## tb datasource export (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-export)

Exports data from a data source to a local file in CSV or NDJSON format.

For example:

- Export all rows as CSV: `tb datasource export my_datasource`
- Export 1000 rows as NDJSON: `tb datasource export my_datasource --format ndjson --rows 1000`
- Export to specific file: `tb datasource export my_datasource --target ./data/export.csv`

| Option | Description |
| --- | --- |
| --format [csv\|ndjson] | Output format (CSV or NDJSON) |
| --rows INTEGER | Number of rows to export (default: 100) |
| --where TEXT | Condition to filter data |
| --target TEXT | Target path. Default is `datasource_name.{format}` |
| -h, --help | Explains export commmand and options |

## tb datasource ls (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-ls)

Lists data sources.

| Option | Description |
| --- | --- |
| --match TEXT | Retrieves any resource matching the pattern |
| --format [json] | Returns the results in the specified format |

## tb datasource sync (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-sync)

Sync data source to S3 bucket.

| Option | Description |
| --- | --- |
| --yes | Does not ask for confirmation |

## tb datasource start (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-start)

Starts Kafka ingestion for a Data Source. Only available for Kafka-connected Data Sources in branches or Tinybird Local.

Once started, data is ingested from the Kafka topic. In branches, this creates a new consumer group and starts ingestion from the latest offset. In Tinybird Local, ingestion resumes from the last committed offset.

```
tb [--branch=BRANCH_NAME] datasource start my_kafka_datasource
```
Because each start creates a new consumer group, previous consumer groups become orphaned. Depending on your Kafka cluster's consumer group TTL (`offsets.retention.minutes`), these orphan groups may persist until they expire. This is expected behavior for ephemeral branch environments.

## tb datasource stop (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-stop)

Stops Kafka ingestion for a Data Source. Only available for Kafka-connected Data Sources in branches or Tinybird Local.

Once stopped, no new data is ingested from the Kafka topic until the Data Source is started again.

```
tb [--branch=BRANCH_NAME] datasource stop my_kafka_datasource
```

## tb datasource truncate (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-datasource#tb-datasource-truncate)

Truncates a data source. For example, `tb datasource truncate my_datasource`.

| Option | Description |
| --- | --- |
| --yes | Does not ask for confirmation |
| --cascade | Truncates the dependent data source attached in cascade to the given data source |

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-datasource#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Manages data sources in Tinybird Local. |
| `--cloud` | ✓ Yes | Manages data sources in Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✓ Yes | Manages data sources in a branch. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-create
Last update: 2026-03-09T11:44:44.000Z
Content:
---
title: "tb create (deprecated) \| Tinybird"
theme-color: "#171612"
description: "Deprecated alias for project scaffolding. Use tb init."
inkeep:version: "forward"
---

# tb create (deprecated) (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-create#tb-create-deprecated)

`tb create` is deprecated.

Use [tb init](/docs/forward/dev-reference/commands/tb-init) to scaffold projects.

The command is kept only for backward compatibility and prints a deprecation warning.

## Replacement (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-create#replacement)

Use `tb init` instead:

```
tb init --type cli
```
Common equivalents:

- Choose project folder: `tb init --folder tinybird`
- Set development mode: `tb init --dev-mode manual`
- Generate CI/CD templates: `tb init --cicd github` or `tb init --cicd gitlab`

## Options (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-create#options)

| Option | Description |
| --- | --- |
| -h, --help | Shows the help message. |

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-create#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (implicit) | The deprecation notice is shown locally. |
| `--cloud` | ✗ No | Not supported. |
| `--branch=BRANCH_NAME` | ✗ No | Not supported. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-copy
Last update: 2026-07-27T12:05:10.000Z
Content:
---
title: "tb copy \| Tinybird"
theme-color: "#171612"
description: "Manage copy pipes in your Tinybird project"
inkeep:version: "forward"
---

# tb copy (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-copy#tb-copy)

Manages copy pipes. Global options apply to this command. See [Global options](/docs/forward/dev-reference/commands/global-options).

The following subcommands are available:

| Subcommand | Description |
| --- | --- |
| ls | Lists all the copy pipes. |
| run [OPTIONS] PIPE_NAME_OR_ID | Runs a copy job. |
| pause PIPE_NAME_OR_ID | Pauses a copy job. |
| resume PIPE_NAME_OR_ID | Resumes a copy job. |

## tb copy run (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-copy#tb-copy-run)

Runs a copy job.

| Option | Description |
| --- | --- |
| --wait | Waits for the copy job to finish. |
| --mode [append\|replace] | Defines the copy strategy. |
| --param TEXT | Key-value pair to pass as a parameter to the Copy Pipe. For example: `tb copy run <my_copy_pipe> --param foo=bar` |
| --on-demand-compute | Runs the copy job on a dedicated on-demand compute instance, isolated from your main workspace infrastructure. See [on-demand compute for copy jobs](/docs/forward/core-concepts/copy-pipes#on-demand-compute-for-copy-jobs). |

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-copy#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Manages copy pipes locally. |
| `--cloud` | ✓ Yes | Manages copy pipes in Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✓ Yes | Manages copy pipes in a branch. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-connection
Last update: 2025-12-18T09:52:50.000Z
Content:
---
title: "tb connection \| Tinybird"
theme-color: "#171612"
description: "Manage your data connections from the terminal."
inkeep:version: "forward"
---

# tb connection (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-connection#tb-connection)

Manage your data connections. Global options apply to this command. See [Global options](/docs/forward/dev-reference/commands/global-options).

Connections are configured through .connection files. See [Connection files](/docs/forward/dev-reference/datafiles/connection-files).

## Subcommands (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-connection#subcommands)

The following subcommands are available:

| Subcommand | Description |
| --- | --- |
| create TYPE | Create a new connection. Supported values are: `kafka`, `gcs`, and `s3`. |
| ls | Lists all connections in the project. |

## tb connection ls (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-connection#tb-connection-ls)

Lists connections.

| Option | Description |
| --- | --- |
| --service TEXT | Filter by service. |

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-connection#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Manages connections in Tinybird Local. |
| `--cloud` | ✓ Yes | Manages connections in Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✓ Yes | Manages connections in a branch. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-build
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "tb build \| Tinybird"
theme-color: "#171612"
description: "Build your Tinybird project and validate resources"
inkeep:version: "forward"
---

# tb build (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-build#tb-build)

Builds your project and checks that all the resources are valid.

```
tb build

» Building project...
✓ datasources/events.datasource created
✓ endpoints/endpoint.pipe created

✓ Build completed in 0.2s
```

## Difference with tb deployment create (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-build#difference-with-tb-deployment-create)

While similar, `tb build` and `tb deployment create` have different purposes:

- `tb build` command is a stateless command that validates project [datafiles](/docs/forward/dev-reference/datafiles).
- `tb deployment create --check` checks that you can successfully create the deployment.
- Unlike `tb deployment create --check`, `tb build` does *not* validate external connections.
- **Exception**: [Postgres copy pipes](/docs/forward/ingest-data/table-functions/postgresql) *are* validated with `tb build`. Secrets and DB host must be available locally for a successful build.
For example, when updating a data project in a workspace, `tb build` checks that the new version of your project is valid, while `tb deployment create --check` verifies that you can successfully migrate from the old version to the new one.

## Options (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-build#options)

| Option | Description |
| --- | --- |
| --with-connections | Creates data linkers for connection data sources (S3, Kafka, GCS) during build. Use this flag to enable connector data sources in branches. |
| --watch | Watches for changes and rebuilds automatically. |

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-build#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✓ Yes (default) | Builds your project locally. |
| `--cloud` | ✗ No | Use `tb deployment create --check` to validate your project against Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✓ Yes | Builds your project in a remote branch. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-branch
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "tb branch \| Tinybird"
theme-color: "#171612"
description: "Manage branches in your Tinybird project"
inkeep:version: "forward"
---

# tb branch (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-branch#tb-branch)

Manages branches in your Tinybird project. Branches allow you to develop and test your project in ephemeral environments using production data. Global options apply to this command. See [Global options](/docs/forward/dev-reference/commands/global-options).

The following subcommands are available:

| Subcommand | Description |
| --- | --- |
| create BRANCH_NAME [OPTIONS] | Creates a new branch from your production workspace. |
| rm BRANCH_NAME [OPTIONS] | Removes an existing branch. |
| ls [OPTIONS] | Lists all available branches. |

## tb branch create (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-branch#tb-branch-create)

Creates a new branch from your production workspace. You can optionally include the last partition of production data.

```
tb branch create my_feature_branch
```
To create a branch with production data:

```
tb branch create my_feature_branch --last-partition
```

| Option | Description |
| --- | --- |
| --last-partition | Brings the last partition of production data into the branch |
| --ignore-datasource | Deprecated. Ignored with a warning. Use `tb branch create` without this flag. |
Use the `--last-partition` flag when you want to test your changes with real production data.

## tb branch rm (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-branch#tb-branch-rm)

Removes an existing branch from your workspace.

```
tb branch rm my_feature_branch
```
To skip the confirmation prompt:

```
tb branch rm my_feature_branch --yes
```

| Option | Description |
| --- | --- |
| --yes | Skips confirmation prompt and removes the branch immediately |
Removing a branch is irreversible and deletes all data and changes in the branch.

## tb branch ls (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-branch#tb-branch-ls)

Lists all available branches in your workspace.

```
tb branch ls
```
This command shows all branches you have created, along with their status and creation information.

## Working with branches (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-branch#working-with-branches)

Once you create a branch, you can start working with it using the `--branch` flag with other commands:

```
# Start development in a branch
tb --branch=my_feature_branch dev

# Open Tinybird UI for the branch
tb --branch=my_feature_branch open
```
For more information about working with branches, see [Branches](/docs/forward/core-concepts/branches).

## Environment support (https://www.tinybird.co/docs/forward/dev-reference/commands/tb-branch#environment-support)

| Environment | Supported | Description |
| --- | --- | --- |
| `--local` | ✗ No | Branches are not available locally. |
| `--cloud` | ✗ No | Branches are always created associated with a workspace in Tinybird Cloud. |
| `--branch=BRANCH_NAME` | ✗ No | Use `--branch` with other commands to work in a branch, not for managing branches. |

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli
Last update: 2026-06-01T10:20:05.000Z
Content:
---
title: "Python SDK CLI commands \| Tinybird"
theme-color: "#171612"
description: "Reference for the Tinybird Python SDK CLI commands."
inkeep:version: "forward"
---

# Python SDK CLI commands (https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli#python-sdk-cli-commands)

The Python SDK (`tinybird-sdk`) installs a `tinybird` command.

- `tinybird generate` and `tinybird migrate` are SDK-native commands.
- Other commands are passed through to the installed Tinybird CLI dependency.
Supported Python versions are 3.11, 3.12, and 3.13. Python 3.14 is not supported yet.

## Install (https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli#install)

- pip
- uv

```
pip install tinybird-sdk
```

## Project and deployment commands (https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli#project-and-deployment-commands)

### `tinybird init` (https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli#tinybird%2520init)

Initialize a project.

```
tinybird init --dev-mode branch --cicd skip --folder tb_project
tinybird init --dev-mode local --cicd github --folder tb_project
tinybird init --dev-mode branch --cicd skip --folder tb_project --skip-login
```

### `tinybird dev` (https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli#tinybird%2520dev)

Build server-side and watch for changes.

```
tinybird dev
tinybird dev --with-connections
tinybird dev --no-connections
```

### `tinybird build` (https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli#tinybird%2520build)

Validate and build the current project.

```
tinybird build
tinybird build --watch
tinybird build --with-connections
```

### `tinybird deploy` (https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli#tinybird%2520deploy)

Deploy resources from your project.

```
tinybird deploy
tinybird deploy --check
tinybird deploy --allow-destructive-operations
tinybird deploy --wait --auto
```

### `tinybird preview` (https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli#tinybird%2520preview)

Create a preview environment and deploy project resources.

```
tinybird preview
tinybird preview --dry-run
tinybird preview --check
tinybird preview --name pr_482
```

### `tinybird pull` (https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli#tinybird%2520pull)

Pull latest resources from Tinybird.

```
tinybird pull
tinybird pull --force
```

### `tinybird login` (https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli#tinybird%2520login)

Authenticate with Tinybird.

```
tinybird login
tinybird login --interactive
tinybird login --host https://api.us-east-1.aws.tinybird.co
```

### `tinybird branch` (https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli#tinybird%2520branch)

Manage branches.

```
tinybird branch ls
tinybird branch create my_feature_branch
tinybird branch clear my_feature_branch
tinybird branch rm my_feature_branch
```

### `tinybird info` (https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli#tinybird%2520info)

Show project information.

```
tinybird info
tinybird info --skip-local
```

## SDK-native commands (https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli#sdk-native-commands)

### `tinybird generate` (https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli#tinybird%2520generate)

Generates `.datasource`, `.pipe`, and `.connection` files from the Python resource definitions in your project.

```
tinybird generate
tinybird generate -o ./tinybird_datafiles
tinybird generate -o ./tinybird_datafiles --json
```

### `tinybird migrate` (https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli#tinybird%2520migrate)

Migrate `.datasource`, `.pipe`, and `.connection` files to Python SDK definitions.

```
tinybird migrate "tinybird/**/*.datasource" "tinybird/**/*.pipe" "tinybird/**/*.connection"
tinybird migrate tinybird/legacy --out ./tinybird.migration.py
tinybird migrate tinybird --dry-run --json
tinybird migrate tinybird --no-strict
```

## Next steps (https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli#next-steps)

- Review Python resource definitions: [Python SDK resources](/docs/forward/dev-reference/python-sdk-resources)
- Follow the Python SDK quickstart: [Quick start: Python SDK](/docs/forward/quickstarts/python-sdk)
- Explore the SDK source repository: [tinybird-sdk-python](https://github.com/tinybirdco/tinybird-sdk-python)

---

URL: https://www.tinybird.co/docs/forward/dev-reference/commands/global-options
Last update: 2026-05-19T17:12:19.000Z
Content:
---
title: "Global options \| Tinybird"
theme-color: "#171612"
description: "Global options for the Tinybird CLI"
inkeep:version: "forward"
---

# Global options (https://www.tinybird.co/docs/forward/dev-reference/commands/global-options#global-options)

The following options are available for all commands. Enter them before the command name. For example: `tb --host https://api.tinybird.co workspace ls`.

Environment options (`--cloud`, `--local`, `--build`, `--staging`) don't apply to all commands. For example, `tb dev` and `tb build` don't support `--cloud`. Check the **Environment support** section on each command's page to see which options apply.

| Option | Description |
| --- | --- |
| --host TEXT | Tinybird host. Defaults to the value of the `TB_HOST` environment variable, then to `https://api.tinybird.co`. |
| --token TEXT | Authentication token. Defaults to the value of the `TB_TOKEN` environment variable, then to the `.tinyb` file. See [.tinyb file](/docs/forward/dev-reference/datafiles/tinyb-file). |
| --show-tokens | Shows available tokens. |
| --user-token | Use the user token, defaults to the `TB_USER_TOKEN` environment variable, then to the .tinyb file. See [.tinyb file](/docs/forward/dev-reference/datafiles/tinyb-file). |
| --cloud / --local | Run against Tinybird Cloud or Tinybird Local. Local is the default except when in build mode. |
| --build | Run against the build mode. Default when running `tb dev` or `tb build --watch`. |
| --staging | Run against the staging deployment. |
| --debug / --no-debug | Prints internal representation. Combine it with any command to get more information. |
| --version | Shows the version and exits. |
| -h, --help | Shows help for the command. |

---

URL: https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Kafka connector troubleshooting guide \| Tinybird"
theme-color: "#171612"
description: "Troubleshoot Tinybird Kafka connector errors, including broker connectivity, authentication, deserialization, offset conflicts, and performance issues."
inkeep:version: "forward"
---

# Kafka connector troubleshooting guide (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#kafka-connector-troubleshooting-guide)

This guide helps you diagnose and resolve common issues with Tinybird's Kafka connector. Use the `tinybird.kafka_ops_log` [Service Data Source](/docs/forward/monitoring/service-datasources#tinybird-kafka-ops-log) to monitor errors and warnings in real time.

For setup instructions and configuration details, see the [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka).

## Quick error lookup (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#quick-error-lookup)

Use this table to quickly find errors and their solutions. Errors may appear in `kafka_ops_log` (Kafka connector operations) or `datasources_ops_log` (Data Source ingestion operations).

| Error message / symptom | Category | Log source | Solution link |
| --- | --- | --- | --- |
| Connection timeout or broker unreachable | Connectivity | kafka_ops_log | [Connection timeout](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-connection-timeout-or-broker-unreachable) |
| Authentication failed | Authentication | kafka_ops_log, datasources_ops_log | [Authentication failed](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-authentication-failed) |
| SSL handshake failed | SSL/TLS | kafka_ops_log | [SSL certificate validation](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-ssltls-certificate-validation-failed) |
| Schema Registry connection failed | Deserialization | kafka_ops_log | [Schema Registry](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-schema-registry-connection-failed) |
| Deserialization failed - Avro | Deserialization | kafka_ops_log | [Avro deserialization](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-deserialization-failed---avro) |
| Deserialization failed - JSON | Deserialization | kafka_ops_log | [JSON deserialization](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-deserialization-failed---json) |
| Offset commit failed | Consumer group | kafka_ops_log | [Offset commit](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-offset-commit-failed-or-consumer-group-conflict) |
| Consumer lag continuously increasing | Performance | kafka_ops_log | [Consumer lag](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-consumer-lag-continuously-increasing) |
| Schema mismatch or type conversion failed | Schema | kafka_ops_log | [Schema mismatch](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-schema-mismatch-or-type-conversion-failed) |
| Materialized View errors | Schema | kafka_ops_log, datasources_ops_log | [Materialized View errors](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-materialized-view-errors) |
| Low throughput or processing stall | Performance | kafka_ops_log | [Low throughput](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-low-throughput-or-processing-stall) |
| Uneven partition processing | Performance | kafka_ops_log | [Uneven partitions](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-uneven-partition-processing) |
| Message too large | Message size | kafka_ops_log | [Message size](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-message-too-large-or-quarantined-due-to-size) |
| Compressed message handling | Message format | kafka_ops_log | [Compression](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-compressed-message-handling) |
| Unknown topic or partition | Kafka | datasources_ops_log | [Unknown topic](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-unknown-topic-or-partition) |
| Group authorization failed | Authorization | datasources_ops_log | [Group authorization](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-group-authorization-failed) |
| Topic authorization failed | Authorization | datasources_ops_log | [Topic authorization](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-topic-authorization-failed) |
| Unknown partition | Kafka | datasources_ops_log | [Unknown partition](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-unknown-partition) |
| Table in readonly mode | Data Source | datasources_ops_log | [Readonly mode](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-table-in-readonly-mode) |
| Timeout or memory limit exceeded | Resource | datasources_ops_log | [Timeout](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-timeout-or-memory-limit-exceeded) |

## How to diagnose errors (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#how-to-diagnose-errors)

Use both `kafka_ops_log` and `datasources_ops_log` to diagnose Kafka connector issues:

### Check Kafka connector operations (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#check-kafka-connector-operations)

Query recent errors and warnings from `kafka_ops_log`:

```
SELECT
    timestamp,
    datasource_id,
    topic,
    partition,
    msg_type,
    msg,
    lag
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
  AND msg_type IN ('warning', 'error')
ORDER BY timestamp DESC
```

### Check Data Source ingestion errors (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#check-data-source-ingestion-errors)

Query errors from `datasources_ops_log` to see issues during data ingestion:

```
SELECT
    timestamp,
    datasource_id,
    event_type,
    result,
    error,
    elapsed_time
FROM tinybird.datasources_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
  AND result = 'error'
  AND event_type LIKE '%kafka%'
ORDER BY timestamp DESC
```
This shows errors that occur during the actual data processing phase, even when the Kafka connection itself might be working.

**Set up automated monitoring**: Connect these diagnostic queries to your monitoring and alerting tools. Query the [ClickHouse® HTTP interface](/docs/forward/query-data/clickhouse-interface) directly from tools like Grafana, Datadog, PagerDuty, and Slack. Alternatively, create API endpoints from these queries and export them in [Prometheus format](/docs/forward/core-concepts/api-endpoints#response-formats) for Prometheus-compatible tools. Configure your tools to poll these queries periodically and trigger alerts when errors are detected.

For detailed monitoring queries, see [Monitor Kafka connectors](/docs/forward/monitoring/kafka-clickhouse-monitoring).

## Connectivity errors (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#connectivity-errors)

### Error: Connection timeout or broker unreachable (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-connection-timeout-or-broker-unreachable)

**Symptoms:**

- No messages are being processed
- Errors in `kafka_ops_log` with messages like "Connection timeout" or "Broker unreachable"
- High lag values that continue to increase
**Root causes:**

1. Incorrect `KAFKA_BOOTSTRAP_SERVERS` configuration
2. Network connectivity issues between Tinybird and your Kafka cluster
3. Firewall or security group rules blocking access
4. Kafka broker is down or unreachable
**Solutions:**

1. **Verify bootstrap servers configuration:**

  - Check that `KAFKA_BOOTSTRAP_SERVERS` in your `.connection` file includes the correct host and port
  - Ensure you're using the advertised listeners address, not the internal broker address
  - For multiple brokers, use comma-separated values: `broker1:9092,broker2:9092,broker3:9092`
  - For cloud providers, verify you're using the public endpoint provided by your Kafka service
2. **Test connectivity:**

```
tb connection data <connection_name>
```
Use this command to select a topic and consumer group ID and return preview data. This validates that Tinybird can reach your Kafka broker, authenticate, and consume messages.

1. **Check network configuration:**

  - Verify firewall rules allow outbound connections from Tinybird to your Kafka cluster
  - For AWS MSK, ensure security groups allow inbound traffic on the Kafka port
  - For Confluent Cloud, verify network access settings
  - For PrivateLink setups (Enterprise), verify the PrivateLink connection is active
2. **Verify security protocol:**

  - Ensure `KAFKA_SECURITY_PROTOCOL` matches your Kafka cluster configuration
  - For most cloud providers, use `SASL_SSL`
  - For local development, you may use `PLAINTEXT`
For vendor-specific network configuration help, see:

- [Confluent Cloud setup guide](/docs/forward/guides/confluent-cloud-setup)
- [AWS MSK setup guide](/docs/forward/guides/aws-msk-setup)

### Error: Authentication failed (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-authentication-failed)

**Symptoms:**

- Errors in `kafka_ops_log` with "Authentication failed" or "SASL authentication error"
- Connection check fails with authentication errors
**Root causes:**

1. Incorrect `KAFKA_KEY` or `KAFKA_SECRET` credentials
2. Wrong `KAFKA_SASL_MECHANISM` configuration
3. Expired credentials or tokens
4. For AWS MSK with OAuthBearer, incorrect IAM role configuration
**Solutions:**

1. **Verify credentials:**

```
tb [--cloud] secret get KAFKA_KEY
tb [--cloud] secret get KAFKA_SECRET
```
Ensure the secrets match your Kafka cluster credentials.

1. **Check SASL mechanism:**

  - Verify `KAFKA_SASL_MECHANISM` matches your Kafka cluster (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, or OAUTHBEARER)
  - For Confluent Cloud, typically use `PLAIN`
  - For AWS MSK with IAM, use `OAUTHBEARER` with `KAFKA_SASL_OAUTHBEARER_METHOD AWS`
  - For Redpanda, check your cluster's configured SASL mechanism
2. **For AWS MSK OAuthBearer:**

  - Verify the IAM role ARN is correct: `tb [--cloud] secret get AWS_ROLE_ARN`
  - Check that the IAM role has the correct trust policy allowing Tinybird to assume the role
  - Verify the external ID matches between your connection configuration and IAM trust policy
  - Ensure the IAM role has the required Kafka cluster permissions (see [AWS IAM permissions](/docs/forward/ingest-data/connectors/kafka#aws-iam-permissions))
3. **Rotate credentials if needed:**

  - If credentials have expired, update them using `tb secret set`
  - Redeploy your connection after updating secrets
For detailed authentication setup, see:

- [AWS MSK setup guide](/docs/forward/guides/aws-msk-setup) for IAM authentication
- [Confluent Cloud setup guide](/docs/forward/guides/confluent-cloud-setup) for API key authentication

### Error: SSL/TLS certificate validation failed (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-ssltls-certificate-validation-failed)

**Symptoms:**

- Errors mentioning "SSL handshake failed" or "certificate validation error"
- Connection failures when using `SASL_SSL` security protocol
**Root causes:**

1. Missing or incorrect CA certificate
2. Self-signed certificate not provided
3. Certificate expired or invalid
**Solutions:**

1. **Provide CA certificate:**

```
tb [--cloud] secret set --multiline KAFKA_SSL_CA_PEM
```
Paste your CA certificate in PEM format.

1. **Add certificate to connection file:**

```
KAFKA_SSL_CA_PEM >
   {{ tb_secret("KAFKA_SSL_CA_PEM") }}
```
Note: This is a [multiline setting](/docs/forward/dev-reference/datafiles#multiple-lines).

1. **Verify certificate format:**
  - Ensure the certificate is in PEM format (starts with `-----BEGIN CERTIFICATE-----`)
  - Include the full certificate chain if required
  - For Aiven Kafka, download the CA certificate from the Aiven console

## Deserialization errors (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#deserialization-errors)

### Error: Schema Registry connection failed (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-schema-registry-connection-failed)

**Symptoms:**

- Errors in `kafka_ops_log` mentioning "Schema Registry" or "Failed to fetch schema"
- Messages not being ingested when using Avro or JSON with schema
**Root causes:**

1. Incorrect `KAFKA_SCHEMA_REGISTRY_URL` configuration
2. Missing or incorrect Schema Registry credentials
3. Schema Registry is unreachable
4. Schema not found in Schema Registry
**Solutions:**

1. **Verify Schema Registry URL:**

  - Check that `KAFKA_SCHEMA_REGISTRY_URL` in your `.connection` file is correct
  - For Basic Auth, use format: `https://<username>:<password>@<registry_host>`
  - Ensure the URL is accessible from Tinybird's network
2. **Check schema exists:**

  - Verify the schema exists in your Schema Registry for the topic
  - Ensure the schema subject name matches your topic naming convention
  - For Confluent Schema Registry, check subject names like `{topic-name}-value` or `{topic-name}-key`
3. **Test Schema Registry access:**

  - Use curl or similar tool to verify Schema Registry is reachable
  - Verify credentials work with Schema Registry API
For more information on schema management, see the [schema management guide](guides/schema-management).

### Error: Deserialization failed - Avro (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-deserialization-failed-avro)

**Symptoms:**

- Warnings in `kafka_ops_log` with "Deserialization failed" or "Avro parsing error"
- Messages sent to Quarantine Data Source
- `processed_messages` > `committed_messages` in monitoring queries
**Root causes:**

1. Schema mismatch between message and Schema Registry
2. Schema evolution incompatibility
3. Incorrect `KAFKA_VALUE_FORMAT` or `KAFKA_KEY_FORMAT` configuration
4. Corrupted message data
**Solutions:**

1. **Verify format configuration:**

  - Ensure `KAFKA_VALUE_FORMAT` is set to `avro` for Avro messages
  - Ensure `KAFKA_KEY_FORMAT` is set to `avro` if keys are Avro-encoded
  - Verify `KAFKA_SCHEMA_REGISTRY_URL` is configured
2. **Check schema compatibility:**

  - Verify the message schema matches the schema in Schema Registry
  - Check for schema evolution issues (backward/forward compatibility)
  - Review Quarantine Data Source to see the actual message that failed
3. **Inspect quarantined messages:**

```
SELECT *
FROM your_datasource_quarantine
WHERE timestamp > now() - INTERVAL 1 hour
ORDER BY timestamp DESC
LIMIT 100
```
This helps you see the actual message content and identify the issue.

1. **Schema evolution:**
  - Ensure schema changes are backward compatible
  - Consider using schema versioning strategies
  - Test schema changes in a development environment first
For detailed schema evolution guidance, see the [schema management guide](guides/schema-management).

### Error: Deserialization failed - JSON (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-deserialization-failed-json)

**Symptoms:**

- Warnings in `kafka_ops_log` with "JSON parsing error" or "Invalid JSON"
- Messages in Quarantine Data Source
- Low success rate in throughput monitoring
**Root causes:**

1. Invalid JSON format in message payload
2. Schema mismatch with JSONPath expressions
3. Missing required fields in JSON
4. Incorrect `KAFKA_VALUE_FORMAT` configuration
**Solutions:**

1. **Verify JSON format:**

  - Check that messages are valid JSON
  - Use a JSON validator to test sample messages
  - Review Quarantine Data Source for examples of failed messages
2. **Check JSONPath expressions:**

  - Verify JSONPath expressions in your Data Source schema match the message structure
  - Test JSONPath expressions with sample messages
  - Use `json:$` to store the entire message if you're unsure of the structure
3. **Handle missing fields:**

  - Use nullable types for optional fields: `Nullable(String)`
  - Provide default values in JSONPath: `json:$.field DEFAULT ''`
  - Consider using a schemaless approach with `data String json:$` and extract fields later
4. **Verify format configuration:**

  - Use `json_without_schema` for plain JSON messages
  - Use `json_with_schema` only if you're using Schema Registry for JSON schemas

## Offset and consumer group errors (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#offset-and-consumer-group-errors)

### Error: Offset commit failed or consumer group conflict (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-offset-commit-failed-or-consumer-group-conflict)

**Symptoms:**

- Data Source only receives messages from the last committed offset
- Multiple Data Sources competing for the same consumer group
- Errors about offset commit failures
**Root causes:**

1. Multiple Data Sources using the same `KAFKA_TOPIC` and `KAFKA_GROUP_ID` combination
2. Consumer group already in use by another app
3. Offset reset behavior not working as expected
**Solutions:**

1. **Use unique consumer group IDs:**

  - Each Data Source must use a unique `KAFKA_GROUP_ID` for the same topic
  - Use environment-specific group IDs: `{{ tb_secret("KAFKA_GROUP_ID", "prod-group") }}`
  - For testing, use unique group IDs to avoid conflicts
2. **Check for duplicate configurations:**

```
SELECT
      datasource_id,
      topic,
      count(*) as group_count
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
GROUP BY datasource_id, topic
HAVING group_count > 1
```
This helps identify if multiple Data Sources are consuming from the same topic.

1. **Reset offset behavior:**

  - `KAFKA_AUTO_OFFSET_RESET=earliest` only works for new consumer groups
  - If a consumer group already has committed offsets, it resumes from the last committed offset
  - To start from the beginning, use a new `KAFKA_GROUP_ID` or reset offsets in your Kafka cluster
2. **Best practices:**

  - Use different `KAFKA_GROUP_ID` values for development, staging, and production
  - Document which consumer groups are in use
  - Monitor consumer group activity in your Kafka cluster
For managing consumer groups across environments, see the [CI/CD and version control guide](guides/cicd-version-control).

### Error: Consumer lag continuously increasing (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-consumer-lag-continuously-increasing)

**Symptoms:**

- Lag values in `kafka_ops_log` keep growing
- Messages are not being processed fast enough
- Throughput is lower than message production rate
**Root causes:**

1. Message production rate exceeds processing capacity
2. Consumer autoscaling not keeping up with load
3. Network latency or connectivity issues
4. Data Source schema or Materialized View performance issues
**Solutions:**

1. **Monitor lag trends:**

```
SELECT
      datasource_id,
      topic,
      partition,
      max(lag) as current_lag,
      avg(lag) as avg_lag
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
   AND partition >= 0
GROUP BY datasource_id, topic, partition
ORDER BY current_lag DESC
```

1. **Verify autoscaling:**

  - Tinybird's serverless Kafka connector automatically scales consumers
  - Monitor `kafka_ops_log` to see partition assignment changes
  - If lag continues to increase, there may be a bottleneck in your Data Source or Materialized Views
2. **Check Data Source performance:**

  - Review Materialized View queries that trigger on append
  - Optimize complex Materialized View queries that may slow down ingestion
  - Check for schema issues causing slow parsing
3. **Analyze throughput:**

```
SELECT
      datasource_id,
      topic,
      sum(processed_messages) as processed,
      sum(committed_messages) as committed,
      (sum(committed_messages) * 100.0 / sum(processed_messages)) as success_rate
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
GROUP BY datasource_id, topic
```
Low success rates indicate processing issues.

1. **Review partitioning strategy:**

  - Check if partition distribution is even
  - Review partition key design if lag is uneven across partitions
  - See the [partitioning strategies guide](guides/partitioning-strategies) for optimization tips
2. **Contact support:**

  - If lag continues to increase despite autoscaling, contact Tinybird support
  - Provide `kafka_ops_log` queries showing the issue
  - Include information about message production rates
For performance optimization strategies, see the [performance optimization guide](guides/performance-optimization).

## Data quality and schema errors (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#data-quality-and-schema-errors)

### Error: Schema mismatch or type conversion failed (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-schema-mismatch-or-type-conversion-failed)

**Symptoms:**

- Warnings in `kafka_ops_log` about type mismatches
- Messages in Quarantine Data Source
- Low `committed_messages` compared to `processed_messages`
**Root causes:**

1. Data type mismatch between message and Data Source schema
2. Missing required fields
3. Invalid data formats (for example, date strings that can't be parsed)
4. JSONPath expressions not matching message structure
**Solutions:**

1. **Review schema definition:**

  - Verify column types match the data in messages
  - Use appropriate ClickHouse® types (for example, `DateTime` for timestamps, `Int64` for large integers)
  - Check for nullable vs non-nullable field requirements
2. **Test with sample messages:**

  - Use `tb sql` to test JSONPath expressions with sample data
  - Verify date/time formats can be parsed correctly
  - Check numeric formats and precision
3. **Handle data quality issues:**

  - Use nullable types for fields that may be missing: `Nullable(String)`
  - Provide default values: `json:$.field DEFAULT 0`
  - Use type conversion functions if needed: `toDateTime(JSONExtractString(data, 'timestamp'))`
4. **Inspect quarantined data:**

  - Regularly check Quarantine Data Source for patterns
  - Identify common data quality issues
  - Update schema or data producers to fix root causes
For detailed schema management guidance, see the [schema management guide](guides/schema-management).

### Error: Materialized View errors (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-materialized-view-errors)

**Symptoms:**

- Warnings in `kafka_ops_log` mentioning Materialized View errors
- Data ingested but Materialized Views not updating
- Errors in Materialized View queries
- Errors in Materialized View queries affecting ingestion
**Root causes:**

1. Materialized View query errors
2. Schema changes breaking Materialized View queries
3. Resource constraints (memory, CPU)
4. Circular dependencies between Materialized Views
**Solutions:**

1. **Check Materialized View queries:**

  - Review Materialized View pipe definitions
  - Test Materialized View queries independently
  - Verify queries work with the current Data Source schema
2. **Monitor Materialized View impact:**

  - Monitor overall ingestion throughput in `kafka_ops_log` to see if Materialized Views are slowing down ingestion
  - Check for errors in Materialized View queries that may be blocking ingestion
  - Review Materialized View query complexity and execution time
3. **Optimize Materialized View queries:**

  - Simplify complex aggregations
  - Add appropriate filters to reduce data volume
  - Consider breaking complex Materialized Views into multiple steps
4. **Handle schema evolution:**

  - Update Materialized View queries when Data Source schema changes
  - Test Materialized View changes in development first
  - Use `FORWARD_QUERY` to provide default values for new columns

## Data Source operation errors (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#data-source-operation-errors)

These errors occur during the data ingestion phase and are logged in `datasources_ops_log`. They represent problems that happen during actual data processing, even when the Kafka connection itself might be working.

### Error: Unknown topic or partition (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-unknown-topic-or-partition)

**Error message:**

- "KafkaError[UNKNOWN_TOPIC_OR_PART]: Broker: Unknown topic or partition"
**Symptoms:**

- Errors in `datasources_ops_log` with "Unknown topic or partition"
- No messages being ingested
- Topic name errors
**Root causes:**

1. Topic doesn't exist in Kafka cluster
2. Topic was deleted
3. Topic name typo in configuration
4. Topic retention policies caused data deletion
**Solutions:**

1. **Verify topic exists:**

  - Check your Kafka cluster to confirm the topic exists
  - Use Kafka tools: `kafka-topics.sh --list --bootstrap-server <server>`
  - Verify topic name matches exactly (case-sensitive)
2. **Check topic configuration:**

  - Ensure topic hasn't been deleted
  - Verify topic retention policies haven't removed all data
  - Check if topic was renamed
3. **Verify Data Source configuration:**

```
cat <datasource_name.datasource>
```
Check that `KAFKA_TOPIC` matches the actual topic name.

1. **Create topic if needed:**
  - If topic doesn't exist, create it in your Kafka cluster
  - Ensure proper replication factor and partitions
  - Redeploy the Data Source after creating the topic
**Note:** If the error specifically mentions a partition (not the topic), see [Unknown partition](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-unknown-partition) in the following section for partition-specific troubleshooting.

**Monitor topic errors:**

```
SELECT
    timestamp,
    datasource_id,
    error,
    count(*) as error_count
FROM tinybird.datasources_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
  AND result = 'error'
  AND error LIKE '%UNKNOWN_TOPIC%'
GROUP BY timestamp, datasource_id, error
ORDER BY error_count DESC
```

### Error: Authentication failure (during ingestion) (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-authentication-failure-during-ingestion)

**Error message:**

- "KafkaError[_AUTHENTICATION]: Local: Authentication failure"
**Symptoms:**

- Errors in `datasources_ops_log` with authentication failures
- Connection works initially but fails during ingestion
- Credentials expired or rotated during operation
**Root causes:**

1. SASL credentials expired or invalid during ingestion
2. SSL certificates expired
3. Authentication settings changed on Kafka broker
4. Credentials rotated but not updated in Tinybird
5. Token-based authentication expired mid-operation
**Solutions:**

1. **Verify credentials:**

```
tb [--cloud] secret get KAFKA_KEY
tb [--cloud] secret get KAFKA_SECRET
```
Ensure secrets match your Kafka cluster credentials.

1. **Check for credential expiration:**

  - Some credentials have expiration dates
  - Rotate credentials if they've expired
  - Update secrets and redeploy connection
  - For token-based auth, ensure tokens are refreshed before expiration
2. **Verify SSL certificates:**

  - Check certificate expiration dates
  - Update certificates if expired
  - Verify certificate format is correct
3. **Test connection:**

```
tb connection data <connection_name>
```
This validates authentication is working.

1. **Check for intermittent auth failures:**
  - Monitor `datasources_ops_log` for authentication error patterns
  - If errors occur periodically, credentials may be expiring
  - Set up credential rotation before expiration
**Note:** This error occurs during data ingestion, not during initial connection. If you see authentication errors during connection setup, see [Authentication failed](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-authentication-failed) in the Connectivity errors section.

**Monitor authentication errors:**

```
SELECT
    timestamp,
    datasource_id,
    error,
    count(*) as error_count
FROM tinybird.datasources_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
  AND result = 'error'
  AND error LIKE '%AUTHENTICATION%'
GROUP BY timestamp, datasource_id, error
ORDER BY error_count DESC
```

### Error: Group authorization failed (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-group-authorization-failed)

**Error message:**

- "KafkaError[GROUP_AUTHORIZATION_FAILED]: Broker: Group authorization failed"
**Symptoms:**

- Errors in `datasources_ops_log` with "Group authorization failed"
- Consumer group lacks permissions
- ACLs not configured correctly
**Root causes:**

1. Consumer group lacks proper authorization
2. Kafka ACLs not configured for the consumer group
3. Consumer group name doesn't match ACL configuration
4. Permissions changed on Kafka cluster
**Solutions:**

1. **Check Kafka ACLs:**

  - Verify consumer group has read permissions
  - Check ACLs for the specific consumer group ID
  - Ensure ACLs allow operations on the topic
2. **Verify consumer group ID:**

  - Check the `KAFKA_GROUP_ID` in your Data Source configuration
  - Ensure it matches what's configured in Kafka ACLs
  - Use consistent naming across environments
3. **Update ACLs:**

  - Grant necessary permissions to the consumer group
  - Ensure group has access to read from the topic
  - Verify group can commit offsets
4. **Test with different group ID:**

  - Try a different consumer group ID temporarily
  - If it works, the issue is with ACLs for the original group
  - Update ACLs for the original group
**Monitor authorization errors:**

```
SELECT
    timestamp,
    datasource_id,
    error,
    count(*) as error_count
FROM tinybird.datasources_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
  AND result = 'error'
  AND error LIKE '%GROUP_AUTHORIZATION%'
GROUP BY timestamp, datasource_id, error
ORDER BY error_count DESC
```

### Error: Topic authorization failed (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-topic-authorization-failed)

**Error message:**

- "KafkaError[TOPIC_AUTHORIZATION_FAILED]: Broker: Topic authorization failed"
**Symptoms:**

- Errors in `datasources_ops_log` with "Topic authorization failed"
- Cannot read from topic
- ACLs not configured for topic access
**Root causes:**

1. Kafka client lacks permission to read from topic
2. Topic ACLs not configured
3. Permissions changed on Kafka cluster
4. Credentials don't have topic access
**Solutions:**

1. **Check topic ACLs:**

  - Verify credentials have read permissions on the topic
  - Check ACLs for the specific topic name
  - Ensure ACLs allow consumer operations
2. **Verify credentials:**

  - Ensure credentials have proper topic access
  - Check if topic permissions have changed
  - Update credentials if needed
3. **Update ACLs:**

  - Grant read permissions to the topic
  - Ensure consumer group has topic access
  - Verify ACLs are applied correctly
4. **Test connection:**

```
tb connection data <connection_name>
```
Select the topic to verify access.

**Monitor topic authorization errors:**

```
SELECT
    timestamp,
    datasource_id,
    error,
    count(*) as error_count
FROM tinybird.datasources_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
  AND result = 'error'
  AND error LIKE '%TOPIC_AUTHORIZATION%'
GROUP BY timestamp, datasource_id, error
ORDER BY error_count DESC
```

### Error: Unknown partition (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-unknown-partition)

**Error message:**

- "KafkaError[_UNKNOWN_PARTITION]: Local: Unknown partition"
**Symptoms:**

- Errors in `datasources_ops_log` with "Unknown partition" (note: different from "Unknown topic or partition")
- Specific partition no longer available
- Topic reconfiguration issues
- Partition-specific errors
**Root causes:**

1. Partition no longer exists in topic (topic was reconfigured)
2. Topic reconfiguration changed partition count
3. Broker failures affecting specific partition availability
4. Partition replication issues
5. Partition was deleted or reassigned
**Solutions:**

1. **Check topic configuration:**

  - Verify current partition count for the topic
  - Check if topic was reconfigured (partitions added/removed)
  - Ensure partition assignments are correct
  - Compare current partition count with what the connector expects
2. **Check broker health:**

  - Verify all brokers are healthy
  - Check for broker failures that might affect specific partitions
  - Ensure partition replication is working
  - Review partition leader assignments
3. **Review partition assignments:**

  - Check if partition assignments changed
  - Verify replication factors are correct
  - Consider rebalancing if needed
  - Check if partitions were reassigned to different brokers
4. **Monitor partition availability:**

  - Use `kafka_ops_log` to see which partitions are being accessed
  - Check for partition-specific errors
  - Identify which specific partition is causing issues
  - Contact support if partitions are consistently unavailable
**Note:** This error is different from "Unknown topic or partition" - this specifically indicates a partition issue when the topic exists. If you see "UNKNOWN_TOPIC_OR_PART", see [Unknown topic or partition](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-unknown-topic-or-partition) in the preceding section.

**Monitor partition errors:**

```
SELECT
    timestamp,
    datasource_id,
    error,
    count(*) as error_count
FROM tinybird.datasources_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
  AND result = 'error'
  AND error LIKE '%UNKNOWN_PARTITION%'
  AND error NOT LIKE '%UNKNOWN_TOPIC%'
GROUP BY timestamp, datasource_id, error
ORDER BY error_count DESC
```

### Error: Table in readonly mode (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-table-in-readonly-mode)

**Error message:**

- "Table is in readonly mode: replica_path=..."
**Symptoms:**

- Errors in `datasources_ops_log` with "readonly mode"
- Data Source temporarily unavailable for writes
- Replication or maintenance in progress
**Root causes:**

1. ClickHouse® table in readonly mode during replication
2. Ongoing maintenance operations
3. ClickHouse® cluster issues
4. Replication lag or issues
**Solutions:**

1. **Wait for table to become writable:**

  - This is often a transient state
  - Wait a few minutes and check again
  - Monitor `datasources_ops_log` for resolution
2. **Check ClickHouse® cluster:**

  - Verify cluster health
  - Check for ongoing maintenance
3. **Monitor for resolution:**

```
SELECT
      timestamp,
      datasource_id,
      error,
      count(*) as occurrence_count
FROM tinybird.datasources_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
   AND result = 'error'
   AND error LIKE '%readonly%'
GROUP BY timestamp, datasource_id, error
ORDER BY timestamp DESC
```

1. **Contact support:**
  - If issue persists for extended period
  - Provide `datasources_ops_log` queries showing the issue
  - Include timestamps and Data Source IDs
**Note:** Readonly mode errors are typically transient and resolve automatically. If they persist, contact Tinybird support.

### Error: Timeout or memory limit exceeded (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-timeout-or-memory-limit-exceeded)

**Error message:**

- "memory limit exceeded: would use ... GiB"
- "Waiting timeout for memo"
- Timeout errors during ingestion
**Symptoms:**

- Errors in `datasources_ops_log` with timeout or memory errors
- Large messages or complex transformations
- Resource constraints
**Root causes:**

1. Message size too large
2. Complex Materialized View queries consuming too much memory
3. High message throughput
4. Resource constraints
**Solutions:**

1. **Reduce message size:**

  - Use Kafka compression
  - Split large messages into smaller chunks
  - Move large data to external storage
2. **Optimize Materialized View queries:**

  - Simplify complex aggregations
  - Add filters to reduce data volume
  - Break complex transformations into multiple steps
3. **Monitor memory usage:**

```
SELECT
      timestamp,
      datasource_id,
      error,
      elapsed_time
FROM tinybird.datasources_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
   AND result = 'error'
   AND (error LIKE '%memory%' OR error LIKE '%timeout%')
ORDER BY timestamp DESC
```

1. **Optimize transformations:**

  - Reduce data processed per operation
  - Use more efficient query patterns
  - Consider batching operations
2. **Contact support:**

  - If memory issues persist
  - Discuss resource requirements
  - Consider plan upgrades if needed
For more information on handling large messages, see the [message size handling guide](guides/message-size-handling).

## Performance and throughput issues (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#performance-and-throughput-issues)

### Error: Low throughput or processing stall (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-low-throughput-or-processing-stall)

**Symptoms:**

- `processed_messages` is zero or low
- No recent activity in `kafka_ops_log`
- Data Source not receiving new messages
**Root causes:**

1. Kafka topic has no new messages
2. Consumer has stopped or crashed
3. Network connectivity issues
4. Configuration errors preventing consumption
**Solutions:**

1. **Verify topic has messages:**

  - Check your Kafka cluster to verify messages are being produced
  - Use Kafka tools to verify topic has new messages
  - Check producer metrics
2. **Check connector activity:**

```
SELECT
      datasource_id,
      topic,
      max(timestamp) as last_activity,
      now() - max(timestamp) as time_since_activity
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 7 day
GROUP BY datasource_id, topic
HAVING time_since_activity > INTERVAL 1 hour
```

1. **Verify configuration:**

  - Run `tb connection data <connection_name>` to test the connection and preview data
  - Verify all required settings are present
  - Check for typos in topic names or connection names
2. **Check for errors:**

  - Review recent errors in `kafka_ops_log`
  - Check Quarantine Data Source for issues
  - Review `datasources_ops_log` for Data Source operation errors

### Error: Uneven partition processing (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-uneven-partition-processing)

**Symptoms:**

- Some partitions have high lag while others have low lag
- Uneven message distribution across partitions
- Some partitions processing faster than others
**Root causes:**

1. Uneven message distribution in Kafka topic
2. Partition key design causing hot partitions
3. Consumer assignment imbalance
4. Different message sizes across partitions
**Solutions:**

1. **Analyze partition distribution:**

```
SELECT
      datasource_id,
      topic,
      partition,
      max(lag) as max_lag,
      avg(lag) as avg_lag,
      sum(processed_messages) as total_messages
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 24 hour
   AND partition >= 0
GROUP BY datasource_id, topic, partition
ORDER BY max_lag DESC
```

1. **Review partition key strategy:**

  - Ensure partition keys distribute messages evenly
  - Avoid using keys that create hot partitions
  - Consider using random keys if even distribution is needed
2. **Monitor autoscaling:**

  - Tinybird's connector automatically balances partition assignment
  - Monitor `kafka_ops_log` to see partition assignment changes
  - High lag should trigger additional consumer instances
3. **Optimize at producer level:**

  - Review Kafka producer configuration
  - Adjust partition key strategy if needed
  - Consider increasing topic partitions if needed
For detailed partitioning strategies, see the [partitioning strategies guide](guides/partitioning-strategies).

## Compression and message format errors (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#compression-and-message-format-errors)

### Error: Compressed message handling (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-compressed-message-handling)

**Symptoms:**

- Messages ingested as raw bytes instead of decompressed content
- Warnings about message format
**Root causes:**

1. Messages compressed before being sent to Kafka producer
2. Kafka compression not configured correctly
3. Message format not recognized
**Solutions:**

1. **Understand compression types:**

  - Kafka compression (configured in producer): Automatically decompressed by Kafka consumer
  - App-level compression (compressed before producing): Not automatically decompressed
2. **Use Kafka compression:**

  - Configure Kafka producer with `compression.type=gzip` (or snappy, lz4)
  - Kafka consumer automatically decompresses these messages
  - Messages arrive in Tinybird already decompressed
3. **Handle app-level compression:**

  - If you compress messages before sending to Kafka, you need to handle decompression
  - Consider storing compressed messages and decompressing in Materialized Views
  - Or change producer to use Kafka compression instead
4. **Verify message format:**

  - Check that `KAFKA_VALUE_FORMAT` matches your message format
  - For JSON, use `json_without_schema` or `json_with_schema`
  - For Avro, use `avro` with Schema Registry configured

## Message size errors (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#message-size-errors)

### Error: Message too large or quarantined due to size (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#error-message-too-large-or-quarantined-due-to-size)

**Symptoms:**

- Messages sent to Quarantine Data Source
- Errors about message size limits
- Large messages not being ingested
**Root causes:**

1. Message exceeds Tinybird's 10 MB default limit
2. Large payloads causing memory issues
3. Compression not reducing message size effectively
**Solutions:**

1. **Check message size:**

  - Review Quarantine Data Source for size-related errors
  - Verify message sizes in your Kafka topic
  - Use Kafka tools to inspect message sizes
2. **Implement compression:**

  - Use Kafka compression to reduce message size
  - Consider compressing large payloads before producing to Kafka
3. **Split large messages:**

  - Break large messages into smaller chunks
  - Use message headers to track message parts
  - Reassemble in Materialized Views if needed
4. **Alternative approaches:**

  - Store large payloads in object storage (S3, GCS) and reference them in Kafka messages
  - Use external storage for large binary data
For detailed guidance on handling large messages, see the [message size handling guide](guides/message-size-handling).

## Quarantine Data Source issues (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#quarantine-data-source-issues)

### Understanding Quarantine Data Source (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#understanding-quarantine-data-source)

When messages fail to ingest into your main Data Source, Tinybird automatically sends them to a Quarantine Data Source. This prevents data loss. You can inspect problematic messages in the Quarantine Data Source.

**Common reasons for quarantine:**

- Schema mismatches
- Invalid data formats
- Type conversion errors
- Missing required fields
- Deserialization failures
- Message size limits exceeded
**How to inspect quarantined messages:**

```
SELECT *
FROM your_datasource_quarantine
WHERE timestamp > now() - INTERVAL 24 hour
ORDER BY timestamp DESC
LIMIT 100
```
**How to resolve:**

1. Identify patterns in quarantined messages
2. Fix schema or data quality issues
3. Update Data Source schema if needed
4. Fix data producers to send correct formats
5. Consider reprocessing quarantined messages after fixes
For more information, see [Quarantine Data Sources](/docs/forward/guides/quarantine).

## Getting help (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#getting-help)

If you've tried the preceding solutions and still experience issues:

1. **Collect diagnostic information:**

  - Recent errors from `kafka_ops_log`
  - Recent errors from `datasources_ops_log`
  - Connection configuration (without secrets)
  - Data Source schema
  - Sample of problematic messages (if available)
2. **Check monitoring:**

  - Review [Kafka monitoring guide](/docs/forward/monitoring/kafka-clickhouse-monitoring)
  - Check Service Data Sources for additional context
  - Query both `kafka_ops_log` and `datasources_ops_log` for complete picture
3. **Review related guides:**

  - [Performance optimization guide](guides/performance-optimization) for throughput issues
  - [Schema management guide](guides/schema-management) for schema-related problems
4. **Contact support:**

  - Provide error messages and timestamps from both logs
  - Include relevant queries from `kafka_ops_log` and `datasources_ops_log`
  - Share configuration details (sanitized)
  - Describe steps to reproduce the issue

## Prevention best practices (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/troubleshooting#prevention-best-practices)

1. **Use unique consumer group IDs** for each Data Source and environment
2. **Test schema changes** in development before deploying to production
3. **Monitor both `kafka_ops_log` and `datasources_ops_log`** regularly to catch issues early
4. **Set up automated alerts** for high lag or error rates in both logs using monitoring tools
5. **Review Quarantine Data Source** periodically to identify data quality issues
6. **Test connections** using `tb connection data <connection_name>` to preview data before deploying
7. **Document consumer group usage** to avoid conflicts
8. **Test with sample messages** before connecting production topics
9. **Use environment-specific configurations** for development, staging, and production
10. **Keep credentials secure** using Tinybird secrets, never hardcode them
11. **Regularly review Kafka ACLs** to ensure proper permissions
12. **Monitor for missing tables** and recreate Data Sources if accidentally deleted
13. **Verify topic and partition availability** before deploying connectors
14. **Optimize Materialized View queries** to prevent timeout and memory errors
15. **Set up monitoring for authorization errors** to catch permission issues early
**Integrate with your monitoring stack**: Connect the monitoring queries in this guide to your existing monitoring tools. Query the [ClickHouse® HTTP interface](/docs/forward/query-data/clickhouse-interface) directly from Grafana, Datadog, PagerDuty, Slack, and other alerting systems. You can also create API endpoints from these queries and export them in [Prometheus format](/docs/forward/core-concepts/api-endpoints#response-formats) for Prometheus-compatible tools. This activates proactive monitoring and automated alerting for your Kafka connectors.

For comprehensive monitoring queries and alerts, see [Monitor Kafka connectors](/docs/forward/monitoring/kafka-clickhouse-monitoring).

---

URL: https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/limits
Last update: 2026-08-07T12:15:37.000Z
Content:
---
title: "Kafka connector limits and quotas \| Tinybird"
theme-color: "#171612"
description: "Learn about Kafka connector limits, how they're applied, and how to request limit increases for your Tinybird workspace."
inkeep:version: "forward"
---

# Kafka connector limits and quotas (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/limits#kafka-connector-limits-and-quotas)

Use these limits and quotas when planning Kafka Connector workloads in Tinybird. For general Workspace limits, see [Limits](/docs/forward/pricing/limits).

## How limits are applied (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/limits#how-limits-are-applied)

Kafka connector limits are applied at the **organization level**, meaning all workspaces within your organization share the same limits. This ensures consistent resource allocation across your organization.

## Topic limits (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/limits#topic-limits)

The number of Kafka topics you can connect to depends on your plan:

| Plan | Topic limit | Notes |
| --- | --- | --- |
| Free | 5 topics | Hard limit |
| Developer | 5 topics | Hard limit |
| Enterprise | Unlimited topics | Contact support for high topic counts |

### Requesting a topic limit increase (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/limits#requesting-a-topic-limit-increase)

Enterprise plan users can request an increase to the topic limit by contacting [Tinybird support](/docs/forward/support). Include the following information in your request:

- Your organization name
- Current number of topics in use
- Expected number of topics needed
- Use case description

## Consumer group limits (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/limits#consumer-group-limits)

There are no hard limits on the number of consumer groups you can use. However, each combination of `KAFKA_TOPIC` and `KAFKA_GROUP_ID` can only be used in one Data Source.

**Best practices:**

- Use unique consumer group IDs for each Data Source
- Use environment-specific group IDs (for example, `prod-group`, `staging-group`)
- Document which consumer groups are in use
For managing consumer groups across environments, see the [CI/CD and version control guide](guides/cicd-version-control).

## Connection limits (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/limits#connection-limits)

You can create up to **3 Kafka connections per workspace**. Each connection can be used by multiple Data Sources.

If you need more connections, contact [Tinybird support](/docs/forward/support) to discuss your use case.

## Message size limits (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/limits#message-size-limits)

Tinybird has a default message size limit of **10 MB** per message. Messages exceeding this limit are automatically sent to the Quarantine Data Source.

**Strategies for handling large messages:**

- Use Kafka compression to reduce message size
- Split large messages into smaller chunks
- Store large payloads in object storage (S3, GCS) and reference them in Kafka messages
For detailed guidance, see the [message size handling guide](guides/message-size-handling).

## Throughput limits (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/limits#throughput-limits)

The Kafka connector has the following throughput limits:

- **Minimum flush time**: 4 seconds
- **Throughput (uncompressed)**: 20 MB/s per connection
If you're regularly hitting these limits, contact [Tinybird support](/docs/forward/support). Enterprise customers may have higher limits based on their infrastructure.

## Deployment failures due to limits (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/limits#deployment-failures-due-to-limits)

If a deployment fails due to limits, you see an error message indicating which limit was exceeded. Common scenarios:

1. **Topic limit exceeded**: You've reached the maximum number of topics for your plan
2. **Connection limit exceeded**: You've created the maximum number of connections for your workspace
3. **Message size limit**: A message exceeds the 10 MB limit (sent to quarantine, not a deployment failure)
**To resolve:**

1. Review your current usage against the limits
2. Remove unused topics or connections if needed
3. Contact support to request limit increases (Enterprise plans)

## Best practices for managing limits (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/limits#best-practices-for-managing-limits)

1. **Monitor your usage**: Regularly check how many topics and connections you're using
2. **Clean up unused resources**: Remove Data Sources and connections that are no longer needed
3. **Plan for growth**: Consider your future needs when designing your Kafka integration
4. **Use environment-specific configurations**: Separate development, staging, and production to better manage resources

## Requesting limit increases (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/limits#requesting-limit-increases)

Enterprise plan users can request limit increases by contacting [Tinybird support](/docs/forward/support). Include:

- Your organization name
- Which limit you need increased (topics, connections, throughput)
- Current usage and expected growth
- Use case description
Free and Developer plan users can upgrade to Enterprise to access higher limits. See [Pricing](/docs/forward/pricing) for plan details.

## Related documentation (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/limits#related-documentation)

- [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka) - Main connector setup guide
- [Troubleshooting guide](troubleshooting) - Resolve common issues
- [Performance optimization guide](guides/performance-optimization) - Optimize throughput
- [General limits documentation](/docs/forward/pricing/limits) - All workspace limits

---

URL: https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Kafka connector guides \| Tinybird"
theme-color: "#171612"
description: "Browse Kafka connector guides for Confluent Cloud, AWS MSK, Redpanda, local development, schema management, partitioning, and performance."
inkeep:version: "forward"
---

# Kafka connector guides (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides#kafka-connector-guides)

This section contains detailed guides for setting up, configuring, and optimizing your Kafka connector. Each guide focuses on a specific aspect of working with Kafka in Tinybird.

## Setup guides (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides#setup-guides)

### Vendor-specific setup (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides#vendor-specific-setup)

- **[Confluent Cloud setup](/docs/forward/guides/confluent-cloud-setup)** - Connect Tinybird to your Confluent Cloud cluster with step-by-step instructions for authentication, SSL configuration, and testing.
- **[AWS MSK setup](/docs/forward/guides/aws-msk-setup)** - Integrate Tinybird with Amazon MSK, including IAM authentication, security group configuration, and connection testing.
- **[Redpanda setup](/docs/forward/guides/redpanda-setup)** - Connect to your Redpanda cluster, configure authentication, and set up your first Data Source.
- **[Local development](/docs/forward/ingest-data/connectors/kafka/guides/local-development)** - Set up a local Kafka environment using Docker Compose for development and testing.

## Configuration and management (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides#configuration-and-management)

- **[CI/CD and version control](/docs/forward/ingest-data/connectors/kafka/guides/cicd-version-control)** - Manage Kafka connector configurations in CI/CD pipelines, handle environment-specific settings, and implement best practices for version control.
- **[Schema management](/docs/forward/ingest-data/connectors/kafka/guides/schema-management)** - Understand schema evolution, handle Avro and JSON Schema formats, and implement schema versioning strategies.

## Performance and optimization (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides#performance-and-optimization)

- **[Partitioning strategies](/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies)** - Learn best practices for Kafka topic partitioning, understand partition assignment, and optimize for performance.
- **[Message size handling](/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling)** - Handle large Kafka messages, implement compression strategies, and work within message size limits.
- **[Performance optimization](/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization)** - Optimize throughput, reduce consumer lag, scale consumers effectively, and tune your Kafka connector for maximum performance.

## Related documentation (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides#related-documentation)

- [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka) - Main setup and configuration guide
- [Troubleshooting guide](/docs/forward/ingest-data/connectors/kafka/troubleshooting) - Comprehensive troubleshooting for common issues
- [Monitor Kafka connectors](/docs/forward/monitoring/kafka-clickhouse-monitoring) - Monitoring and alerting

---

URL: https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Schema management and evolution \| Tinybird"
theme-color: "#171612"
description: "Complete guide to managing Kafka message schemas, including adding fields, handling nullable types, data type mapping, and schema evolution strategies."
inkeep:version: "forward"
---

# Schema management and evolution (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#schema-management-and-evolution)

This guide covers managing schemas for Kafka messages in Tinybird, including adding and modifying fields, handling nullable types, data type mapping between Kafka and ClickHouse®, and implementing schema evolution strategies.

## Prerequisites (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#prerequisites)

- Understanding of Kafka message formats (JSON, Avro)
- Basic knowledge of ClickHouse® data types
- Access to Schema Registry (if using Avro or JSON Schema)

## Supported serialization formats (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#supported-serialization-formats)

Tinybird supports the following serialization formats:

- **JSON** (`json_without_schema`) - Plain JSON messages
- **JSON Schema** (`json_with_schema`) - JSON with Schema Registry
- **Avro** (`avro`) - Avro with Schema Registry

## Adding and modifying fields (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#adding-and-modifying-fields)

### Adding new fields (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#adding-new-fields)

When adding new fields to your Kafka messages, the key is making them backward compatible. Old messages (without the field) should still work.

**The pattern:**

1. Update your Data Source schema to include the new field
2. Make it nullable if the field may be missing in older messages
3. Provide default values in JSONPath expressions if needed
**Example:**

datasources/events_updated.datasource
```
SCHEMA >
    `order_id` String `json:$.order_id`,
    `customer_id` String `json:$.customer_id`,
    `order_total` Float64 `json:$.order_total`,
    `payment_method` Nullable(String) `json:$.payment_method`,  -- New field, nullable
    `data` String `json:$`
```
Why nullable? Old messages don't have `payment_method`, so its value is `null`. After all messages include it, you can make it non-nullable if needed.

**For Avro/JSON Schema:**

- Add fields as optional (not in `required` array)
- Provide default values in Schema Registry
- Use nullable types in ClickHouse

### Modifying existing fields (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#modifying-existing-fields)

Modifying existing fields requires more care. Some changes are safe, others require migration.

**Safe modifications:**

- Adding default values to existing fields
- Making fields nullable (if they weren't before)
- Widening types (Int32 → Int64, String → Nullable(String))
**Example of safe modification:**

```
-- Before: String field
`status` String `json:$.status`

-- After: Still String, but with default
`status` String `json:$.status DEFAULT 'pending'`
```
**Unsafe modifications (require FORWARD_QUERY):**

- Changing field types (String → Int32)
- Narrowing types (Int64 → Int32)
- Removing fields
**Example of type change:**

```
-- Before
`count` Int32 `json:$.count`

-- After: Changed to Int64
`count` Int64 `json:$.count`
```
Use FORWARD_QUERY to convert existing data:

```
FORWARD_QUERY >
    SELECT
        *,
        toInt64(count) as count  -- Convert Int32 to Int64
    FROM previous_datasource
```
This migrates existing data while new messages use the new type.

**Deploying changes:**

- Test locally: `tb deploy`
- Deploy to production: `tb --cloud deploy`
- Always test schema changes in development first

## Nullable vs non-nullable (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#nullable-vs-non-nullable)

### When to use nullable (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#when-to-use-nullable)

Use `Nullable()` types when:

- Fields may be missing in some messages
- Fields can have `null` values
- You're adding new optional fields
- You want to handle schema evolution gracefully
**Example:**

```
SCHEMA >
    `order_id` String `json:$.order_id`,                    -- Required
    `email` Nullable(String) `json:$.email`,              -- Optional
    `preferences` Nullable(String) `json:$.preferences`, -- Optional
    `data` String `json:$`
```

### When to use non-nullable (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#when-to-use-non-nullable)

Use non-nullable types when:

- Fields are always present in messages
- You want to enforce data quality
- Fields are required for your use case
**Example:**

```
SCHEMA >
    `order_id` String `json:$.order_id`,        -- Always present
    `timestamp` DateTime `json:$.timestamp`,  -- Always present
    `event_type` String `json:$.event_type`,  -- Always present
    `data` String `json:$`
```

### Handling missing fields (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#handling-missing-fields)

If a field might be missing, provide a default value:

```
`status` String `json:$.status DEFAULT 'unknown'`
`count` Int32 `json:$.count DEFAULT 0`
`is_active` UInt8 `json:$.is_active DEFAULT 0`
```
Defaults handle missing fields gracefully without requiring nullable types.

## Data type mapping (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#data-type-mapping)

### Kafka to ClickHouse® mapping (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#kafka-to-clickhouse-mapping)

| Kafka/JSON Type | ClickHouse Type | Notes |
| --- | --- | --- |
| `string` | `String` | Text data |
| `number` (integer) | `Int32`, `Int64` | Use Int64 for large numbers |
| `number` (float) | `Float32`, `Float64` | Use Float64 for precision |
| `boolean` | `UInt8`, `Boolean` | 0 or 1, or true/false |
| `null` | `Nullable(T)` | Wrap in Nullable |
| `array` | `Array(T)` | Specify element type |
| `object` | `String` (JSON) or `Map(K, V)` | Store as JSON string or Map |

### Common type conversions (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#common-type-conversions)

**String to DateTime:**

```
`timestamp` DateTime `json:$.timestamp`
-- Assumes ISO 8601 format: "2024-01-01T00:00:00Z"
```
**String to number:**

```
`price` Float64 `json:$.price`
-- JSON numbers are automatically converted
```
**Array handling:**

```
`tags` Array(String) `json:$.tags[:]`
-- For JSON arrays of strings
```
**Nested objects:**

```
`metadata` String `json:$.metadata`
-- Store nested object as JSON string, extract later with JSONExtract
```
**Nested fields:**

For nested JSON structures, use JSONPath to extract specific fields:

```
SCHEMA >
    `customer_id` String `json:$.customer.id`,
    `customer_email` String `json:$.customer.email`,
    `order_data` String `json:$.order.data`,
    `data` String `json:$`
```
**Maps:**

Use Map type for key-value structures:

```
`headers` Map(String, String) `json:$.headers`
-- For Kafka headers or metadata maps
```

## Schema Registry integration (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#schema-registry-integration)

### Avro schemas (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#avro-schemas)

When using Avro with Schema Registry:

1. Register schema in Schema Registry
2. Set format to `avro` in Data Source
3. Configure Schema Registry URL in connection

```
KAFKA_VALUE_FORMAT avro
KAFKA_SCHEMA_REGISTRY_URL https://registry.example.com
```
**Schema evolution rules:**

- Add optional fields (with defaults)
- Remove fields (backward compatible)
- Change field types only if compatible
- Don't change required fields to optional without defaults

### JSON Schema (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#json-schema)

When using JSON Schema with Schema Registry:

```
KAFKA_VALUE_FORMAT json_with_schema
KAFKA_SCHEMA_REGISTRY_URL https://registry.example.com
```
**Best practices:**

- Use schema versioning
- Test schema changes in development
- Monitor schema compatibility
- Add new properties without adding them to `required`

## Schema evolution strategies (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#schema-evolution-strategies)

### Backward compatibility (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#backward-compatibility)

Ensure schema changes are backward compatible:

**Safe changes:**

- Adding optional fields
- Removing fields
- Making fields nullable
- Adding default values
- Widening types (Int32 → Int64)
**Breaking changes:**

- Removing required fields
- Changing field types incompatibly
- Renaming fields
- Narrowing types (Int64 → Int32)

### Versioning strategy (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#versioning-strategy)

For breaking changes, use topic versioning:

```
orders-v1
orders-v2
orders-v3
```
Create separate Data Sources for each version, then merge in queries if needed. This is safer than trying to migrate in place.

### Gradual rollout (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#gradual-rollout)

When adding new fields:

1. Add new field as nullable in Data Source
2. Update producers to include new field
3. Monitor for any issues
4. Make field required once all messages include it (optional)
This approach minimizes risk, and you can roll back if needed.

## Monitoring schema changes (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#monitoring-schema-changes)

### Detect schema mismatches (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#detect-schema-mismatches)

Query `kafka_ops_log` for deserialization errors:

```
SELECT
    timestamp,
    datasource_id,
    topic,
    msg,
    count(*) as error_count
FROM tinybird.kafka_ops_log
WHERE msg_type = 'warning'
  AND msg LIKE '%schema%'
  AND timestamp > now() - INTERVAL 1 hour
GROUP BY timestamp, datasource_id, topic, msg
ORDER BY error_count DESC
```
This helps you catch schema evolution issues early.

### Monitor quarantined messages (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#monitor-quarantined-messages)

Check Quarantine Data Source for schema-related issues:

```
SELECT
    timestamp,
    count(*) as quarantined_count
FROM your_datasource_quarantine
WHERE timestamp > now() - INTERVAL 1 hour
GROUP BY timestamp
ORDER BY timestamp DESC
```
Quarantined messages often indicate schema mismatches or data quality issues.

## Common pitfalls (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#common-pitfalls)

### Changing field types incompatibly (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#changing-field-types-incompatibly)

**Problem:** Changing `count` from `String` to `Int32` breaks old messages.

**Solution:** Use `FORWARD_QUERY` to convert types, or add a new field and migrate gradually.

### Removing required fields (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#removing-required-fields)

**Problem:** Removing a field that old messages require causes deserialization failures.

**Solution:** Make the field optional first, wait for all consumers to update, then remove it.

### Not handling nullable fields (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#not-handling-nullable-fields)

**Problem:** Adding a new field as non-nullable breaks old messages that don't have it.

**Solution:** Always make new fields nullable initially. After all messages include it, you can make it non-nullable.

### Ignoring Schema Registry compatibility (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#ignoring-schema-registry-compatibility)

**Problem:** Schema changes that violate compatibility settings cause failures.

**Solution:** Check your compatibility mode (`BACKWARD`, `FORWARD`, `FULL`) and test schema changes before deploying.

## Best practices (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#best-practices)

1. **Use nullable types** for optional fields
2. **Provide default values** for missing fields
3. **Test schema changes** in development first with `tb deploy`
4. **Deploy to production** using `tb --cloud deploy` after testing
5. **Monitor schema evolution** using `kafka_ops_log`
6. **Use Schema Registry** for Avro and JSON Schema
7. **Document schema changes** in your team
8. **Version schemas** for breaking changes
9. **Keep schemas backward compatible** when possible
10. **Start with nullable** when adding new fields

## Troubleshooting schema issues (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#troubleshooting-schema-issues)

### Type conversion failed (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#type-conversion-failed)

**Solutions:**

1. Verify JSONPath expressions match message structure
2. Check data types are compatible
3. Use type conversion functions if needed
4. Make fields nullable if data might be missing

### Schema mismatch (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#schema-mismatch)

**Solutions:**

1. Review Quarantine Data Source for actual message format
2. Update Data Source schema to match messages
3. Check Schema Registry if using Avro/JSON Schema
4. Verify schema evolution is backward compatible

### Missing fields (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#missing-fields)

**Solutions:**

1. Make fields nullable
2. Provide default values
3. Update producers to include required fields
4. Handle missing fields in queries

## Related documentation (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/schema-management#related-documentation)

- [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka) - Main setup and configuration guide
- [Troubleshooting guide](../troubleshooting) - Schema-related error troubleshooting
- [Evolve Data Sources](/docs/forward/guides/evolve-data-source) - Managing schema changes
- [ClickHouse® data types](/docs/sql-reference/data-types) - Available data types

---

URL: https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Kafka connector performance optimization \| Tinybird"
theme-color: "#171612"
description: "Optimize Kafka connector performance with schema optimization, Materialized View tuning, and throughput best practices. Learn how to reduce consumer lag and improve ingestion speed."
inkeep:version: "forward"
---

# Performance optimization (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization#performance-optimization)

This guide covers strategies for optimizing your Kafka connector performance, focusing on schema design, Materialized View optimization, and best practices.

## Schema optimization (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization#schema-optimization)

### Use explicit schemas (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization#use-explicit-schemas)

Explicit schemas are faster and more efficient than schemaless:

**Recommended:**

```
SCHEMA >
    `user_id` String `json:$.user_id`,
    `event_type` LowCardinality(String) `json:$.event_type`,
    `timestamp` DateTime `json:$.timestamp`
```
**Avoid (slower):**

```
SCHEMA >
    `data` String `json:$`  -- Requires parsing at query time
```

### Optimize data types (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization#optimize-data-types)

- Use `LowCardinality(String)` for enum-like fields
- Use smallest integer type needed (`Int32` vs `Int64`)
- Use `DateTime` for timestamps (not `String`)
- Use `Nullable()` only when needed
**Example:**

```
SCHEMA >
    `user_id` String `json:$.user_id`,
    `event_type` LowCardinality(String) `json:$.event_type`,
    `timestamp` DateTime `json:$.timestamp`,
    `count` Int32 `json:$.count`,
    `metadata` Nullable(String) `json:$.metadata`  -- Only if needed
```

## Materialized View optimization (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization#materialized-view-optimization)

Complex Materialized Views can slow down ingestion. Materialized Views that trigger on append operations from Kafka data sources can impact ingestion performance, especially if they perform expensive aggregations or joins.

### Optimization strategies (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization#optimization-strategies)

1. **Simplify aggregations** - Keep aggregations efficient
2. **Add filters** - Reduce data volume processed
3. **Optimize joins** - Use appropriate join strategies
4. **Avoid cascade MVs** - Don't create multiple Materialized Views from the same Kafka data source, as this increases ingestion latency
5. **Limit MVs per data source** - Too many Materialized Views reading from the same Kafka data source can slow down ingestion

## Partition distribution (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization#partition-distribution)

Ensure even partition distribution to maximize throughput. Monitor partition lag:

```
SELECT
    partition,
    max(lag) as max_lag,
    avg(lag) as avg_lag,
    sum(processed_messages) as total_processed
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
  AND partition >= 0
GROUP BY partition
ORDER BY max_lag DESC
```
Uneven distribution may indicate:

- Poor partition key design
- Hot partitions
- Need for more partitions
See the [partitioning strategies guide](partitioning-strategies) for detailed guidance.

## Common performance bottlenecks (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization#common-performance-bottlenecks)

### Schema parsing (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization#schema-parsing)

**Symptoms:**

- High CPU usage
- Slow message processing
- Low throughput
**Solutions:**

1. Use explicit schemas instead of schemaless
2. Optimize JSONPath expressions
3. Reduce schema complexity
4. Use appropriate data types

### Materialized Views (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization#materialized-views)

**Symptoms:**

- Slow ingestion
- High memory usage
- Timeouts in Materialized Views
**Solutions:**

1. Simplify Materialized View queries
2. Add filters to reduce data volume
3. Avoid cascade MVs or multiple MVs from the same Kafka data source
4. Optimize aggregations

### Partition imbalance (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization#partition-imbalance)

**Symptoms:**

- Uneven lag across partitions
- Some partitions slow
- Overall throughput limited
**Solutions:**

1. Review partition key strategy
2. Redistribute messages more evenly
3. Increase partitions if needed
4. Monitor partition distribution

## Best practices (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization#best-practices)

1. **Use explicit schemas** - Faster parsing and better performance
2. **Optimize data types** - Use smallest types needed, `LowCardinality` for enums
3. **Simplify Materialized Views** - Keep MVs efficient to avoid slowing ingestion
4. **Ensure even partition distribution** - Monitor and optimize partition keys
5. **Monitor performance** - Track lag, throughput, and error rates regularly

## Related documentation (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/performance-optimization#related-documentation)

- [Monitor Kafka connectors](/docs/forward/monitoring/kafka-clickhouse-monitoring) - Comprehensive monitoring queries and metrics
- [Partitioning strategies guide](partitioning-strategies) - Optimize partition distribution
- [Troubleshooting guide](../troubleshooting) - Resolve performance issues

---

URL: https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Kafka partitioning strategies \| Tinybird"
theme-color: "#171612"
description: "Best practices for Kafka topic partitioning, including partition key design, impact on ingestion performance, and relationship to ClickHouse® table partitions."
inkeep:version: "forward"
---

# Kafka partitioning strategies (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#kafka-partitioning-strategies)

This guide covers best practices for Kafka topic partitioning and how it affects Tinybird's Kafka connector performance.

## Understanding Kafka partitions (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#understanding-kafka-partitions)

Kafka topics are divided into partitions, which allow:

- **Parallel processing** - Multiple consumers can process different partitions simultaneously
- **Scalability** - Distribute load across partitions
- **Ordering** - Messages with the same key go to the same partition, maintaining order
Tinybird's Kafka connector automatically assigns partitions to consumer instances and scales consumers based on lag and load.

**Partition assignment is automatic**: Tinybird's Kafka connector automatically handles partition assignment. There is no configuration option to explicitly target specific partitions or turn off automatic assignment. The connector manages partition distribution across consumers automatically.

## Kafka partitions vs ClickHouse® table partitions (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#kafka-partitions-vs-clickhouse-table-partitions)

**Kafka partitions** and **ClickHouse® table partitions** are different concepts:

- **Kafka partitions**: Logical divisions of a topic for parallel processing and ordering. Each Kafka partition can be consumed independently.
- **ClickHouse® table partitions**: Physical divisions of data in ClickHouse® tables, typically organized by time (for example, monthly partitions).
**How they relate:**

- Each Kafka partition can write data to multiple ClickHouse® parts within a ClickHouse® partition
- Multiple Kafka partitions can write to the same ClickHouse® partition
- ClickHouse® typically partitions by time (for example, `ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"`)
**TOO_MANY_PARTS error:**

If you have too many Kafka partitions writing to the same ClickHouse® partition, you may encounter the `TOO_MANY_PARTS` error. This happens when ClickHouse® creates too many parts in a single partition, which can impact query performance.

**Best practice:** For ClickHouse®, partitioning by time (for example, monthly or daily) is usually fine for time-series data. This is independent of your Kafka partition key strategy. Kafka partition keys should focus on distributing load evenly across Kafka partitions, while ClickHouse® partitioning organizes data by time for efficient querying.

## Partition design best practices (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#partition-design-best-practices)

### Number of partitions (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#number-of-partitions)

**Guidelines:**

- Start with **3-6 partitions** for most use cases
- Scale up if you see uneven lag distribution
- More partitions = more parallelism, but also more overhead
- Too many partitions can cause frequent rebalancing and ClickHouse® TOO_MANY_PARTS errors
**Factors to consider:**

- Expected message throughput
- Consumer capacity
- ClickHouse® partition strategy

### Kafka partition key strategy (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#kafka-partition-key-strategy)

The partition key determines which Kafka partition a message goes to. This is about **Kafka partitions**, not ClickHouse partitions.

**Goal: Even distribution across Kafka partitions**

- Use keys that distribute messages evenly across Kafka partitions
- Avoid keys that create "hot partitions" (one Kafka partition receiving most messages)
- Consider hash-based keys for even distribution
**Example of good partition key:**

```
# Even distribution using hash - distributes messages across Kafka partitions
partition_key = hash(user_id) % num_partitions
```
**Example of problematic partition key:**

```
# Creates hot partition - all messages go to one Kafka partition
partition_key = "all-users"
```
**Note:** Partition keys are for distributing load across Kafka partitions. For ClickHouse® table partitioning, you typically partition by time (for example, `ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"`), which is usually fine for time-series data. ClickHouse® partitioning is independent of Kafka partition keys.

## Monitoring partition performance (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#monitoring-partition-performance)

### Monitor partition lag (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#monitor-partition-lag)

Partition lag indicates how far behind your consumer is:

```
SELECT
    datasource_id,
    topic,
    partition,
    max(lag) as current_lag,
    avg(lag) as avg_lag
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
  AND partition >= 0
GROUP BY datasource_id, topic, partition
ORDER BY current_lag DESC
```

### Detect hot partitions (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#detect-hot-partitions)

Check if messages are evenly distributed across partitions:

```
SELECT
    __partition,
    count(*) as message_count,
    count(*) * 100.0 / sum(count(*)) OVER () as percentage
FROM your_datasource
WHERE timestamp > now() - INTERVAL 1 hour
GROUP BY __partition
ORDER BY message_count DESC
```
Uneven distribution (for example, one partition has >50% of messages) indicates a hot partition.

### Monitor partition assignment (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#monitor-partition-assignment)

Track how partitions are assigned to consumers:

```
SELECT
    datasource_id,
    topic,
    partition,
    count(*) as assignment_count
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
  AND partition >= 0
GROUP BY datasource_id, topic, partition
ORDER BY assignment_count DESC
```

## How it works in practice (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#how-it-works-in-practice)

1. **Producer sends messages** to Kafka topics with optional partition keys
2. **Kafka assigns messages** to partitions based on the key (or round-robin if no key)
3. **Tinybird connector** consumes from all partitions in parallel
4. **Data is ingested** into ClickHouse®, where it's organized into ClickHouse® partitions (typically by time)
5. **Multiple Kafka partitions** can write to the same ClickHouse® partition
6. **Each Kafka partition** may create multiple ClickHouse® parts within a partition
**Example flow:**

```
Kafka Topic: events (6 partitions)
  ├─ Partition 0 → ClickHouse® partition 202401 (multiple parts)
  ├─ Partition 1 → ClickHouse® partition 202401 (multiple parts)
  ├─ Partition 2 → ClickHouse® partition 202402 (multiple parts)
  └─ ...
```
**Rebalancing:**

When partitions are added, removed, or consumer instances change, Kafka rebalances partition assignments. This causes a brief pause in processing. Monitor for rebalancing by tracking partition assignment changes in `kafka_ops_log`.

## Troubleshooting (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#troubleshooting)

### Issue: Uneven partition lag (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#issue-uneven-partition-lag)

**Symptoms:** Some partitions have high lag while others are idle

**Solutions:**

1. Review partition key strategy
2. Check message distribution across partitions
3. Consider increasing partitions if needed
4. Adjust partition key to distribute more evenly

### Issue: Hot partitions (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#issue-hot-partitions)

**Symptoms:** One partition receives most messages, causing high lag

**Solutions:**

1. Change partition key strategy
2. Use hash-based keys for even distribution
3. Split high-volume keys across multiple partitions

### Issue: TOO_MANY_PARTS error (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#issue-too-many-parts-error)

**Symptoms:** ClickHouse® error indicating too many parts in a partition

**Solutions:**

1. Reduce number of Kafka partitions writing to the same ClickHouse® partition
2. Adjust ClickHouse® merge settings if needed
3. Consider increasing ClickHouse® partition granularity (for example, daily instead of monthly)
4. Note: Kafka partition key strategy doesn't directly affect this - ClickHouse® partitioning by time is usually fine

## Best practices (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#best-practices)

1. **Start with 3-6 Kafka partitions** and scale as needed
2. **Use Kafka partition keys** that distribute messages evenly across Kafka partitions
3. **For ClickHouse®**, partition by time (for example, monthly) - this is usually fine for time-series data
4. **Monitor partition lag** regularly
5. **Avoid hot Kafka partitions** by using hash-based keys
6. **Watch for TOO_MANY_PARTS** errors and adjust ClickHouse® partition granularity if needed
7. **Remember**: Kafka partition keys (for load distribution) and ClickHouse® partitioning (for data organization) serve different purposes

## Related documentation (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/partitioning-strategies#related-documentation)

- [Performance optimization guide](performance-optimization) - Overall performance tuning
- [Monitor Kafka connectors](/docs/forward/monitoring/kafka-clickhouse-monitoring) - Monitoring partition performance
- [Troubleshooting guide](../troubleshooting) - Resolve partition issues

---

URL: https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling
Last update: 2026-05-29T14:27:44.000Z
Content:
---
title: "Kafka message size limits and handling \| Tinybird"
theme-color: "#171612"
description: "Handle large Kafka messages in Tinybird, including 10 MB limits, compression strategies, message splitting, and quarantined message recovery."
inkeep:version: "forward"
---

# Message size handling (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#message-size-handling)

This guide covers handling large Kafka messages in Tinybird, including message size limits and strategies for large messages.

## Message size limits (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#message-size-limits)

Tinybird has a default message size limit of **10 MB** per message. Messages exceeding this limit are automatically sent to the Quarantine Data Source.

## Checking message sizes (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#checking-message-sizes)

Check quarantined messages for size-related issues:

```
SELECT
    timestamp,
    length(__value) as message_size_bytes,
    length(__value) / 1024 / 1024 as message_size_mb,
    msg
FROM your_datasource_quarantine
WHERE timestamp > now() - INTERVAL 1 hour
ORDER BY message_size_bytes DESC
LIMIT 100
```

## Strategies for handling large messages (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#strategies-for-handling-large-messages)

### Option 1: Compression (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#option-1-compression)

Use Kafka compression to reduce message size:

**Producer configuration:**

```
producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    compression_type='gzip',  # or 'snappy', 'lz4'
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)
```
**Compression types:**

- `gzip` - Best compression, higher CPU
- `snappy` - Good balance
- `lz4` - Fast, lower compression

### Option 2: Split large messages (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#option-2-split-large-messages)

Break large messages into smaller chunks on the producer side, then reassemble in a Materialized View if needed.

### Option 3: External storage (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#option-3-external-storage)

Store large payloads in object storage (S3, GCS) and send only references in Kafka:

```
# Upload to S3, send reference in Kafka
message = {
    'message_id': message_id,
    's3_key': s3_key,
    'metadata': {...}
}
producer.send('topic', value=message)
```

### Option 4: Schema optimization (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#option-4-schema-optimization)

Reduce message size by storing only necessary data and using references for large content:

```
{
  "user_id": "123",
  "profile_summary": "key points only",
  "full_profile_s3_key": "s3://bucket/profiles/123.json"
}
```

## Troubleshooting quarantined messages (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#troubleshooting-quarantined-messages)

### Identify size-related quarantines (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#identify-size-related-quarantines)

```
SELECT
    timestamp,
    length(__value) as message_size,
    length(__value) / 1024 / 1024 as size_mb,
    msg
FROM your_datasource_quarantine
WHERE timestamp > now() - INTERVAL 24 hour
  AND length(__value) > 10 * 1024 * 1024  -- Over 10 MB
ORDER BY message_size DESC
```

### Extract useful data from quarantined messages (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#extract-useful-data-from-quarantined-messages)

Even if the full message is too large, you can extract metadata:

```
SELECT
    timestamp,
    JSONExtractString(__value, 'message_id') as message_id,
    JSONExtractString(__value, 'user_id') as user_id,
    length(__value) as original_size
FROM your_datasource_quarantine
WHERE timestamp > now() - INTERVAL 24 hour
```

## Monitoring message sizes (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#monitoring-message-sizes)

### Track message size distribution (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#track-message-size-distribution)

```
SELECT
    quantile(0.5)(message_size) as median_size,
    quantile(0.95)(message_size) as p95_size,
    quantile(0.99)(message_size) as p99_size,
    max(message_size) as max_size
FROM (
    SELECT length(__value) as message_size
    FROM your_datasource
    WHERE timestamp > now() - INTERVAL 1 hour
)
```

### Alert on large messages (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#alert-on-large-messages)

```
SELECT
    timestamp,
    length(__value) as message_size,
    length(__value) / 1024 / 1024 as size_mb
FROM your_datasource
WHERE length(__value) > 8 * 1024 * 1024  -- Over 8MB
  AND timestamp > now() - INTERVAL 1 hour
ORDER BY message_size DESC
```

## Best practices (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#best-practices)

1. **Target size:** Keep messages under 1 MB when possible
2. **Use Kafka compression** for large messages
3. **Store only necessary data** in Kafka messages
4. **Use references** for large binary data (S3, GCS)
5. **Monitor message sizes** regularly to catch issues early

## Common issues and solutions (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#common-issues-and-solutions)

### Issue: Messages consistently over 10 MB (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#issue-messages-consistently-over-10-mb)

**Solutions:**

1. Implement Kafka compression
2. Split messages into chunks
3. Move large data to external storage
4. Optimize schema to reduce size

### Issue: Compression not helping (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#issue-compression-not-helping)

**Solutions:**

1. Check if data is already compressed
2. Try different compression types
3. Verify compression is turned on in producer
4. Consider if data is compressible (text vs binary)

## Related documentation (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/message-size-handling#related-documentation)

- [Troubleshooting guide](../troubleshooting#error-message-too-large-or-quarantined-due-to-size) - Message size error troubleshooting
- [Quarantine Data Sources](/docs/forward/guides/quarantine) - Handling quarantined messages
- [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka) - Main setup and configuration guide

---

URL: https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development
Last update: 2026-07-27T11:34:23.000Z
Content:
---
title: "Kafka connector local development setup \| Tinybird"
theme-color: "#171612"
description: "Set up Kafka connector for local development with Docker Compose. Learn environment management, testing strategies, and how to use default values for local Kafka connections."
inkeep:version: "forward"
---

# Local development setup (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#local-development-setup)

This guide helps you set up the Kafka connector for local development, including running Kafka locally, connecting to cloud Kafka from your local environment, and managing environment-specific configurations.

## Prerequisites (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#prerequisites)

- Docker and Docker Compose installed
- Tinybird Local installed. See [Tinybird Local](/docs/forward/core-concepts/tinybird-local).
- A Tinybird project directory

## Option 1: Local Kafka with Docker Compose (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#option-1-local-kafka-with-docker-compose)

The easiest way to develop locally is to run Kafka in Docker alongside Tinybird Local.

### Docker Compose setup (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#docker-compose-setup)

Create a `docker-compose.yml` file in your project:

```
networks:
  kafka_network:
    driver: bridge

volumes:
  kafka-data:

services:
  tinybird-local:
    image: tinybirdco/tinybird-local:latest
    container_name: tinybird-local
    platform: linux/amd64
    ports:
      - "7181:7181"
    networks:
      - kafka_network
    volumes:
      - ./:/workspace
      - tinybird-data:/var/lib/tinybird

  kafka:
    image: apache/kafka:latest
    hostname: broker
    container_name: broker
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_PROCESS_ROLES: "broker,controller"
      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@broker:29093"
      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,PLAINTEXT_HOST://0.0.0.0:9092,CONTROLLER://0.0.0.0:29093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
    volumes:
      - kafka-data:/var/lib/kafka/data
    networks:
      - kafka_network
```

### Start the services (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#start-the-services)

```
docker compose up -d
```

### Create a topic (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#create-a-topic)

```
docker exec -it broker /opt/kafka/bin/kafka-topics.sh --create \
  --topic test-topic \
  --bootstrap-server localhost:9092 \
  --partitions 3 \
  --replication-factor 1
```

### Connection configuration (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#connection-configuration)

Create a connection file for local development:

connections/kafka_local.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS {{ tb_secret("KAFKA_BOOTSTRAP_SERVERS", "kafka:29092") }}
KAFKA_SECURITY_PROTOCOL {{ tb_secret("KAFKA_SECURITY_PROTOCOL", "PLAINTEXT") }}
```
**Note:** The bootstrap server `kafka:29092` uses the Docker service name, which works from within the Docker network.

### Data source configuration (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#data-source-configuration)

datasources/test_topic.datasource
```
SCHEMA >
    `data` String `json:$`

KAFKA_CONNECTION_NAME kafka_local
KAFKA_TOPIC test-topic
KAFKA_GROUP_ID {{ tb_secret("KAFKA_GROUP_ID", "local-dev-group") }}
```

### Test locally (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#test-locally)

```
# Deploy to Tinybird Local
tb deploy

# Send a test message
echo '{"test": "data"}' | docker exec -i broker /opt/kafka/bin/kafka-console-producer.sh \
  --topic test-topic \
  --bootstrap-server localhost:9092

# Query the data
tb sql "SELECT * FROM test_topic LIMIT 10"
```

## Option 2: Connect to cloud Kafka from local (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#option-2-connect-to-cloud-kafka-from-local)

You can also connect Tinybird Local to a cloud Kafka cluster (Confluent Cloud, AWS MSK, etc.) for testing.

### Connection configuration (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#connection-configuration)

Use the same connection configuration as production, but with local secrets:

connections/kafka_cloud_local.connection
```
TYPE kafka
KAFKA_BOOTSTRAP_SERVERS {{ tb_secret("KAFKA_BOOTSTRAP_SERVERS", "your-cloud-bootstrap:9092") }}
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("KAFKA_KEY", "your-key") }}
KAFKA_SECRET {{ tb_secret("KAFKA_SECRET", "your-secret") }}
```
**Important:** The `tb_secret()` function uses default values when running locally. These defaults are only used in Tinybird Local, not in Cloud.

### Set local secrets (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#set-local-secrets)

```
# Set secrets for local environment
tb secret set KAFKA_BOOTSTRAP_SERVERS "your-cloud-bootstrap:9092"
tb secret set KAFKA_KEY "your-key"
tb secret set KAFKA_SECRET "your-secret"
```

## Environment-specific configurations (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#environment-specific-configurations)

### Using default values for local development (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#using-default-values-for-local-development)

The `tb_secret()` function supports default values for local environments. Use the same Connection and Data Source files across all environments:

```
KAFKA_BOOTSTRAP_SERVERS {{ tb_secret("KAFKA_BOOTSTRAP_SERVERS", "kafka:29092") }}
KAFKA_SECURITY_PROTOCOL {{ tb_secret("KAFKA_SECURITY_PROTOCOL", "PLAINTEXT") }}
KAFKA_KEY {{ tb_secret("KAFKA_KEY", "key") }}
KAFKA_SECRET {{ tb_secret("KAFKA_SECRET", "secret") }}
```

- **Local**: Uses the default values (for example, `kafka:29092` for local Docker Kafka)
- **Cloud**: Uses the secret values set in Tinybird Cloud for each workspace

## Testing strategies (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#testing-strategies)

### Unit testing with sample data (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#unit-testing-with-sample-data)

Create sample messages for testing:

```
# Create a test topic
docker exec -it broker /opt/kafka/bin/kafka-topics.sh --create \
  --topic test-events \
  --bootstrap-server localhost:9092

# Send sample messages
cat <<EOF | docker exec -i broker /opt/kafka/bin/kafka-console-producer.sh \
  --topic test-events \
  --bootstrap-server localhost:9092
{"user_id": "123", "event": "click", "timestamp": "2024-01-01T00:00:00Z"}
{"user_id": "456", "event": "view", "timestamp": "2024-01-01T00:01:00Z"}
{"user_id": "123", "event": "purchase", "timestamp": "2024-01-01T00:02:00Z"}
EOF
```

### Integration testing (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#integration-testing)

Test the full pipeline:

```
# 1. Deploy to local
tb deploy

# 2. Send test messages
# (use your application or kafka-console-producer)

# 3. Verify ingestion
tb sql "SELECT count() FROM your_datasource"

# 4. Test queries
tb sql "SELECT * FROM your_datasource WHERE timestamp > now() - INTERVAL 1 hour"
```

### Schema testing (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#schema-testing)

Test schema changes locally before deploying:

```
# Test schema with sample data
SCHEMA >
    `user_id` String `json:$.user_id`,
    `event` LowCardinality(String) `json:$.event`,
    `timestamp` DateTime `json:$.timestamp`
```

## Debugging local connections (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#debugging-local-connections)

### Check container logs (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#check-container-logs)

```
# Kafka logs
docker compose logs kafka

# Tinybird Local logs
docker compose logs tinybird-local
```

### Verify network connectivity (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#verify-network-connectivity)

```
# From Tinybird Local container
docker exec -it tinybird-local ping kafka

# Test Kafka connectivity
docker exec -it tinybird-local telnet kafka 29092
```

### Common issues (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#common-issues)

**Issue: Connection timeout**

- Verify containers are on the same network
- Check bootstrap server address matches `KAFKA_ADVERTISED_LISTENERS`
- Ensure Kafka container is running: `docker compose ps`
**Issue: Topic not found**

- Create the topic before deploying
- Check topic exists: `docker exec -it broker /opt/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092`
**Issue: No messages received**

- Verify messages are being produced
- Check consumer group: `docker exec -it broker /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list`
- Review `kafka_ops_log` in Tinybird Local

## Best practices (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#best-practices)

1. **Use separate consumer group IDs** for local development to avoid conflicts
2. **Test schema changes locally** before deploying to production
3. **Use default values** in `tb_secret()` for local development
4. **Keep local and cloud configs separate** to avoid accidental deployments
5. **Clean up test topics** regularly to avoid clutter
6. **Use Docker Compose** for consistent local environments
7. **Document local setup** for your team

## Related documentation (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development#related-documentation)

- [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka) - Main setup and configuration guide
- [Troubleshooting guide](../troubleshooting) - Debugging local connection issues
- [CI/CD and version control](cicd-version-control) - Managing environments

---

URL: https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/cicd-version-control
Last update: 2026-06-02T11:48:14.000Z
Content:
---
title: "Kafka connector CI/CD and version control \| Tinybird"
theme-color: "#171612"
description: "Manage Tinybird Kafka connector configuration in CI/CD with environment secrets, GitHub Actions, GitLab CI, and version control practices."
inkeep:version: "forward"
---

# CI/CD and version control (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/cicd-version-control#cicd-and-version-control)

This guide covers managing Kafka connector configurations across different environments (local, staging, production) using secrets.

## Managing secrets across environments (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/cicd-version-control#managing-secrets-across-environments)

### Local development (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/cicd-version-control#local-development)

Use default values in `tb_secret()` for local development:

```
KAFKA_BOOTSTRAP_SERVERS {{ tb_secret("KAFKA_BOOTSTRAP_SERVERS", "kafka:29092") }}
KAFKA_SECURITY_PROTOCOL {{ tb_secret("KAFKA_SECURITY_PROTOCOL", "PLAINTEXT") }}
KAFKA_KEY {{ tb_secret("KAFKA_KEY", "key") }}
KAFKA_SECRET {{ tb_secret("KAFKA_SECRET", "secret") }}
```

- **Local**: Uses the default values (for example, `kafka:29092` for local Docker Kafka)
- **Cloud**: Uses the secret values set in each Tinybird workspace

### Staging and production (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/cicd-version-control#staging-and-production)

Set secrets in each workspace using the `--token` flag:

```
# Staging workspace
tb --cloud --host <STAGING_HOST> --token <STAGING_TOKEN> secret set KAFKA_BOOTSTRAP_SERVERS "staging-kafka:9092"
tb --cloud --host <STAGING_HOST> --token <STAGING_TOKEN> secret set KAFKA_KEY "staging-key"
tb --cloud --host <STAGING_HOST> --token <STAGING_TOKEN> secret set KAFKA_SECRET "staging-secret"

# Production workspace
tb --cloud --host <PROD_HOST> --token <PROD_TOKEN> secret set KAFKA_BOOTSTRAP_SERVERS "prod-kafka:9092"
tb --cloud --host <PROD_HOST> --token <PROD_TOKEN> secret set KAFKA_KEY "prod-key"
tb --cloud --host <PROD_HOST> --token <PROD_TOKEN> secret set KAFKA_SECRET "prod-secret"
```
The same Connection and Data Source files work across all environments - secrets handle the differences.

## CI/CD integration (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/cicd-version-control#cicd-integration)

### GitHub Actions example (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/cicd-version-control#github-actions-example)

```
name: Deploy to Tinybird

on:
  push:
    branches: [main]

env:
  TINYBIRD_HOST: ${{ secrets.TINYBIRD_HOST }}
  TINYBIRD_TOKEN: ${{ secrets.TINYBIRD_TOKEN }}

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Setup Tinybird CLI
        run: |
          curl https://tinybird.co | sh

      - name: Test connection
        run: |
          tb --cloud --host ${{ env.TINYBIRD_HOST }} --token ${{ env.TINYBIRD_TOKEN }} connection data <connection_name>

      - name: Deploy
        run: |
          tb --cloud --host ${{ env.TINYBIRD_HOST }} --token ${{ env.TINYBIRD_TOKEN }} deploy
```
**Secrets setup**: Set secrets in your Tinybird workspace before running the pipeline. Secrets are configured once per workspace, not on every deployment. See [Managing secrets across environments](https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/cicd-version-control#managing-secrets-across-environments) for instructions.

### GitLab CI example (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/cicd-version-control#gitlab-ci-example)

```
deploy:
  image: ubuntu:latest
  before_script:
    - apt update && apt install -y curl
    - curl https://tinybird.co | sh
    - export PATH="$HOME/.local/bin:$PATH"
  script:
    - tb --cloud --host $TINYBIRD_HOST --token $TINYBIRD_TOKEN connection data <connection_name>
    - tb --cloud --host $TINYBIRD_HOST --token $TINYBIRD_TOKEN deploy
  only:
    - main
```

## Consumer group ID management (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/cicd-version-control#consumer-group-id-management)

Always use different consumer group IDs for each environment to avoid conflicts:

```
KAFKA_GROUP_ID {{ tb_secret("KAFKA_GROUP_ID", "dev-events-group") }}
```
Set different group IDs in each workspace:

- Local: Uses default `"dev-events-group"`
- Staging: Set `tb --cloud --host <STAGING_HOST> --token <STAGING_TOKEN> secret set KAFKA_GROUP_ID "staging-events-group"`
- Production: Set `tb --cloud --host <PROD_HOST> --token <PROD_TOKEN> secret set KAFKA_GROUP_ID "prod-events-group"`

## Version control best practices (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/cicd-version-control#version-control-best-practices)

### What to commit (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/cicd-version-control#what-to-commit)

**Commit:**

- Connection file structure (with `tb_secret()` references, not actual secret values)
- Data Source schemas
- Pipe definitions
**Don't commit:**

- Secret values
- API keys
- Passwords
- Production credentials

## Related documentation (https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/cicd-version-control#related-documentation)

- [Kafka connector documentation](/docs/forward/ingest-data/connectors/kafka) - Main setup and configuration guide
- [Development workflow](/docs/forward/development-workflow) - General development and deployment guide
- [Troubleshooting guide](../troubleshooting) - Resolving deployment issues

---