Skill: kaghni/deeplake-skills (version 1.0.1)

--- name: deeplake-managed description: SDK for ingesting data into Deeplake managed tables. Use when users want to store, ingest, or query data in Deeplake. allowed-tools: Bash, Read, Write, Edit, Glob, Grep, WebFetch metadata: openclaw: requires: env: - DEEPLAKE_API_KEY primaryEnv: DEEPLAKE_API_KEY homepage: https://github.com/activeloopai/deeplake-skills --- # Deeplake Managed Service SDK > Agent-friendly SDK for ingesting data into Deeplake managed tables. > Use this skill when users want to store, ingest, or query data in Deeplake. > Available in both **Python** and **Node.js/TypeScript**. --- ## Quick Reference ### Python ```bash pip install deeplake # uv add deeplake ``` **Python import (primary):** ```python from deeplake import Client # Async variant (requires aiohttp: pip install aiohttp): from deeplake.managed import AsyncClient ``` ```python from deeplake import Client # Initialize -- token from DEEPLAKE_API_KEY env var, workspace defaults to "default" client = Client() client = Client([REDACTED_SECRET]", workspace_id="my-workspace") # Ingest files (FILE schema) client.ingest("videos", {"path": ["video1.mp4", "video2.mp4"]}, schema={"path": "FILE"}) # Ingest structured data with indexes for search client.ingest("embeddings", { "text": ["doc1", "doc2", "doc3"], "embedding": [[0.1, 0.2, ...], [0.3, 0.4, ...], [0.5, 0.6, ...]], }, index=["embedding", "text"]) # Ingest from HuggingFace client.ingest("cifar", {"_huggingface": "cifar10"}) # Ingest with format object (see formats.md for CocoPanoptic, Coco, LeRobot, custom) client.ingest("table", format=my_format) # Fluent query results = client.table("videos").select("id", "text").where("file_id = $1", "abc").limit(10)() # Raw SQL results = client.query("SELECT * FROM videos LIMIT 10") # Vector similarity search results = client.query(""" SELECT id, text, embedding <#> $1 AS similarity FROM embeddings ORDER BY similarity DESC LIMIT 10 """, (query_embedding,)) # Table management client.list_tables() client.drop_table("old_table") client.create_index("embeddings", "embedding") ``` ### Node.js / TypeScript ``` npm install deeplake ``` **TypeScript import:** ```typescript import { ManagedClient, initializeWasm } from 'deeplake'; ``` **WASM initialization (required before any operations):** ```typescript await initializeWasm(); ``` Call `initializeWasm()` once at startup before any `ManagedClient` operations (ingest, query, etc.). It initializes the underlying WASM module. ```typescript import { ManagedClient, initializeWasm } from 'deeplake'; await initializeWasm(); const client = new ManagedClient({ [REDACTED_SECRET]', workspaceId: 'my-workspace' }); // Ingest files (FILE schema) await client.ingest("videos", { path: ["video1.mp4"] }, { schema: { path: "FILE" } }); // Ingest structured data await client.ingest("embeddings", { text: ["doc1", "doc2"], embedding: [[0.1, 0.2], [0.3, 0.4]], }); // Ingest with format object (see formats.md) await client.ingest("table", null, { format: myFormat }); // Fluent query (use .execute()) const results = await client.table("videos") .select("id", "text").where("file_id = $1", "abc").limit(10).execute(); // Raw SQL const rows = await client.query("SELECT * FROM videos LIMIT 10"); // Table management await client.listTables(); await client.dropTable("old_table"); await client.createIndex("embeddings", "embedding"); ``` --- ## Dependancies and Prerequisite **Required services:** - Deeplake API server running (default: `https://api.deeplake.ai`) **Optional python dependencies (per file type):** - Video ingestion: `ffmpeg` (`sudo apt-get install ffmpeg`) - PDF ingestion: `pymupdf` (`pip install pymupdf`) - Thumbnail generation: `Pillow` (`pip install Pillow`) - COCO detection format: `pycocotools`, `Pillow`, `numpy` (`pip install pycocotools Pillow numpy`) - LeRobot frames format: `pandas`, `numpy` (`pip install pandas numpy`) **Optional typescript dependencies (per file type):** - Video ingestion: `ffmpeg` (system binary) - PDF ingestion: `pdfjs-dist` (`npm install pdfjs-dist`) - Thumbnail generation: `sharp` (`npm install sharp`) - COCO detection format: no external deps (pure JS mask rendering) ## Architecture ``` Python: Client(token, workspace_id) Node.js: ManagedClient({ token, workspaceId }) |-- .ingest(table, data) -> creates PG table via API, opens al://{ws}/{table} | via deeplake SDK (auto credential rotation) |-- .query(sql) -> POST /workspaces/{id}/tables/query -> list[dict] / QueryRow[] |-- .table(table)... -> fluent SQL builder -> list[dict] / QueryRow[] |-- .create_index(table, col) -> CREATE INDEX USING deeplake_index (for search) |-- .open_table(table) -> deeplake.open("al://{ws}/{table}") with auto creds |-- .list_tables() -> GET /workspaces/{id}/tables -> list[str] / string[] `-- .drop_table(table) -> DELETE /workspaces/{id}/tables/{name} | v REST API -> PostgreSQL + pg_deeplake - All DB operations go through the REST API (no direct PG connection) - Dataset access uses al:// paths with automatic credential resolution - Creds endpoint: GET /api/org/{workspace}/ds/{table}/creds - Vector similarity: embedding <#> query_vec - BM25 text search: text <#> 'search query' - Hybrid search: (embedding, text)::deeplake_hybrid_record ``` --- ## Client Initialization ### Python ```python from deeplake import Client client = Client( token: str = None, # API token (falls back to DEEPLAKE_API_KEY env var) workspace_id: str = "default", # Target workspace (default: "default") api_url: str = None, # API URL (default: https://api.deeplake.ai) ) ``` ### Node.js / TypeScript ```typescript import { ManagedClient, initializeWasm } from 'deeplake'; await initializeWasm(); const client = new ManagedClient({ [REDACTED_SECRET], // API token (required) workspaceId?: string, // Target workspace (default: "default") apiUrl?: string, // API URL (default: https://api.deeplake.ai) }); ``` **Token:** Create API tokens from the Deeplake platform at `https://app.deeplake.ai/<org_name>/workspace/<workspace>/apitoken`. The token is a JWT with `org_id` embedded. Falls back to the `DEEPLAKE_API_KEY` environment variable (Python only). **Backend endpoint:** The client sets the C++ backend endpoint to `api_url` before each dataset open (not on initialization) so that `al://` path resolution (credential fetching) goes through deeplake-api instead of the legacy controlplane. This avoids global state clobbering when multiple clients use different API URLs. Python: `deeplake.client.endpoint = api_url`. Node.js: `deeplakeSetEndpoint(apiUrl)`. **Connection lifecycle:** ```python # Python: just create and use -- no connection to manage client = Client() client.ingest("table", {"path": ["file.txt"]}, schema={"path": "FILE"}) # No close() method -- client is stateless (REST API calls only) ``` --- ## Ingestion ### Python: client.ingest() ```python result = client.ingest( table_name: str, # Table name to create (must not already exist) data: dict[str, list] = None, # Data dict (required unless format= is set). # {"_huggingface": "name"} -> HuggingFace dataset # schema has "FILE" cols -> file paths processed # otherwise -> column data {col: [values]} *, format: Format = None, # Format object (subclass of Format) with # normalize() method. When set, data is ignored. # e.g. CocoPanoptic(images_dir=..., ...) schema: dict[str, str] = None, # Schema override {col: type} # Use "FILE" for columns containing file paths # See reference.md for all type names index: list[str] = None, # Columns to create deeplake_index on after ingestion. # Use for EMBEDDING (vector search) and TEXT (BM25) columns. on_progress: Callable = None, # Progress callback(rows_written, total) chunk_size: int = 1000, # Text chunk size (chars) chunk_overlap: int = 200, # Text chunk overlap (chars) pdf_dpi: int = 150, # PDF render DPI (higher = sharper but slower) ) -> dict ``` ### Node.js: client.ingest() ```typescript const result = await client.ingest( tableName: string, // Table name data?: Record<string, unknown[]> | null, // Data dict (or null when using format) options?: { format?: Format, // Format object with normalize() schema?: Record<string, string>, // Schema override index?: string[], // Columns to create deeplake_index on onProgress?: (processed, total) => void, // Progress callback chunkSize?: number, // Text chunk size (default 1000) chunkOverlap?: number, // Text chunk overlap (default 200) }, ): Promise<IngestResult> ``` **Table existence:** If `table_name` already exists, `ingest()` appends data to the existing table — it does NOT drop and recreate it. To replace an existing table, call `client.drop_table(table_name)` first. The PG table schema must be compatible with the new data being appended. **Returns:** `{"table_name": "videos", "row_count": 150, "dataset_path": "al://workspace_id/videos"}` **Both `data` and `format`:** If both are provided, `format` takes precedence and `data` is ignored. If neither is provided, an `IngestError` is raised. **Thumbnails:** When a format object declares `image_columns()` (columns with pg_schema type `"IMAGE"`), thumbnails are auto-generated at 4 sizes (32x32, 64x64, 128x128, 256x256) and stored in a shared `thumbnails` dataset at `{root_path}/thumbnails`. Requires Pillow (Python) or sharp (Node.js). ### Chunking Strategy by File Type | File Type | Extensions | Strategy | Columns Created | | --------- | ------------------------------------ | ------------------------------- | --------------------------------------------------------------------------- | | **Video** | .mp4, .mov, .avi, .mkv, .webm | 10-second segments + thumbnails | id, file_id, chunk_index, start_time, end_time, video_data, thumbnail, text | | **Image** | .jpg, .jpeg, .png, .gif, .bmp, .webp | Single chunk | id, file_id, image, filename, text | | **PDF** | .pdf | Page-by-page at 150 DPI (configurable via `pdf_dpi`) | id, file_id, page_index, image, text | | **Text** | .txt, .md, .csv, .json, .xml, .html | 1000 char chunks, 200 overlap | id, file_id, chunk_index, text | | **Other** | * | Single binary chunk | id, file_id, data, filename | ### Key Examples ```python # Ingest files (FILE schema) client.ingest("videos", {"path": ["cam1.mp4", "cam2.mp4"]}, schema={"path": "FILE"}) client.ingest("photos", {"path": ["img1.jpg"]}, schema={"path": "FILE"}) client.ingest("manuals", {"path": ["manual.pdf"]}, schema={"path": "FILE"}) # Ingest structured data (dict = column data, schema inferred) client.ingest("vectors", { "text": ["Hello", "Goodbye"], "embedding": [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]], }) # Ingest with explicit schema client.ingest("data", {"name": ["Alice", "Bob"], "age": [30, 25]}, schema={"name": "TEXT", "age": "INT64"}) # Ingest from HuggingFace client.ingest("mnist", {"_huggingface": "mnist"}) # Ingest with a format object (see formats.md for CocoPanoptic, Coco, LeRobot, custom) client.ingest("table", format=my_format) # Ingest with progress callback def progress(rows_written, total): print(f"Written {rows_written} rows...") client.ingest("docs", {"path": pdf_files}, schema={"path": "FILE"}, on_progress=progress) ``` > For custom format classes, see [formats.md](formats.md). > For more ingestion examples, see [examples.md](examples.md). --- ## Training / Streaming ### client.open_table() Open a managed table as a `deeplake.Dataset` for direct access -- bypasses PostgreSQL and returns the native dataset object with built-in ML framework integration. ```python ds = client.open_table(table_name: str) -> deeplake.Dataset ``` **When to use:** Training loops, batch iteration, PyTorch/TensorFlow DataLoaders, async prefetch. ```python # Batch iteration ds = client.open_table("videos") for batch in ds.batches(32): train(batch) # PyTorch DataLoader from torch.utils.data import DataLoader ds = client.open_table("training_data") loader = DataLoader(ds.pytorch(), batch_size=32, shuffle=True, num_workers=4) # TensorFlow tf.data.Dataset ds = client.open_table("training_data") tf_ds = ds.tensorflow().batch(32).prefetch(tf.data.AUTOTUNE) ``` --- ## Querying ### Fluent Query API (Recommended) `client.table(table)` returns a chainable `ManagedQueryBuilder`: ```python # Python: supports both .execute() and () to run the query results = ( client.table("videos") .select("id", "text", "start_time") .where("file_id = $1", "abc123") .where("start_time > $2", 60) .order_by("start_time ASC") .limit(10) .offset(0) )() # or .execute() ``` ```typescript // Node.js: use .execute() only (no () shorthand) // (assumes initializeWasm() already called at startup) const results = await client.table("videos") .select("id", "text", "start_time") .where("file_id = $1", "abc123") .where("start_time > $2", 60) .orderBy("start_time ASC") .limit(10) .offset(0) .execute(); ``` | Method | Python | Node.js | Description | | ----------------------- | --------------------- | --------------------- | -------------------------- | | `.select(*cols)` | `.select("id", "t")` | `.select("id", "t")` | Set columns (default `*`) | | `.where(cond, *params)` | `.where("id=$1","x")` | `.where("id=$1","x")` | Add WHERE (multiple = AND) | | `.order_by(clause)` | `.order_by("col")` | `.orderBy("col")` | Add ORDER BY | | `.limit(n)` | `.limit(10)` | `.limit(10)` | Set LIMIT | | `.offset(n)` | `.offset(20)` | `.offset(20)` | Set OFFSET | | Run query | `.execute()` or `()` | `.execute()` | Execute, return results | **How `.where()` parameters work:** Each `.where("col = $N", value)` call adds an AND condition. The `$1`, `$2`, etc. placeholders are filled by the extra arguments, numbering across all `.where()` calls sequentially. ### Raw SQL: client.query() ```python # Python rows = client.query( sql: str, params: tuple = None, timeout: int = 60, # HTTP timeout in seconds (increase for slow queries) ) -> list[dict] # Examples rows = client.query("SELECT * FROM videos LIMIT 10") rows = client.query("SELECT * FROM documents WHERE file_id = $1", ("abc123",)) rows = client.query("SELECT COUNT(*) FROM big_table", timeout=300) # 5-minute timeout ``` ```typescript // Node.js const rows = await client.query( sql: string, params?: unknown[], options?: { timeoutMs?: number }, // default 60000 (60s) ) -> Promise<QueryRow[]> // Examples const rows = await client.query("SELECT * FROM videos LIMIT 10"); const rows = await client.query("SELECT * FROM documents WHERE file_id = $1", ["abc123"]); const rows = await client.query("SELECT COUNT(*) FROM big", undefined, { timeoutMs: 300_000 }); ``` Queries are sent to the API via `POST /workspaces/{id}/tables/query`. Use `$1`, `$2`, ... for parameterized queries. > **Timeout:** The default query timeout is 60 seconds. For long-running queries (large aggregations, index builds), increase it via the `timeout` / `timeoutMs` parameter. Non-default timeouts are forwarded to the backend as `timeout_ms` so the server can also apply a deadline. > For pg_deeplake SQL features (vector search, BM25, hybrid search, indexes), see [reference.md](reference.md). --- ## Table Management ```python # Python tables = client.list_tables() -> list[str] client.drop_table(table_name: str, if_exists: bool = True) -> None client.create_index(table_name: str, column: str) -> None ``` ```typescript // Node.js const tables = await client.listTables(); await client.dropTable(tableName: string, ifExists?: boolean); // default true await client.createIndex(tableName: string, column: string); ``` ### Index Creation `create_index()` / `createIndex()` creates a `deeplake_index` on a column. Use it for: - **EMBEDDING columns** — enables vector cosine similarity search via `<#>` - **TEXT columns** — enables BM25 text search via `<#>` The method executes `CREATE INDEX IF NOT EXISTS ... USING deeplake_index (column)` and is a no-op if the index already exists. ```python # Python — standalone client.create_index("embeddings", "embedding") # vector index client.create_index("documents", "text") # text index # Python — during ingestion (creates indexes after data is committed) client.ingest("search_index", { "text": documents, "embedding": embeddings, }, index=["embedding", "text"]) ``` ```typescript // Node.js — standalone await client.createIndex("embeddings", "embedding"); await client.createIndex("documents", "text"); // Node.js — during ingestion await client.ingest("search_index", { text: documents, embedding: embeddings, }, { index: ["embedding", "text"] }); ``` --- ## Workspace Management Workspaces are created via the REST API. The SDK clients don't have a built-in `createWorkspace()` method — use `apiRequest` directly. **Important:** The `id` field is **required** when creating a workspace. Omitting it returns an error. ```typescript // Node.js — create workspace via API import { apiRequest, extractOrgId } from 'deeplake'; const orgId = extractOrgId(token); await apiRequest(apiUrl, token, orgId, { method: 'POST', path: '/workspaces', body: { id: 'my-workspace', name: 'My Workspace' }, timeoutMs: 30_000, }); ``` ```python # Python — create workspace via API import requests resp = requests.post( f"{api_url}/workspaces", headers={"Authorization": f"Bearer {token}"}, json={"id": "my-workspace", "name": "My Workspace"}, ) resp.raise_for_status() ``` **List workspaces:** ```typescript // Node.js const resp = await apiRequest(apiUrl, token, orgId, { method: 'GET', path: '/workspaces', }); // resp.data = [{ id, org_id, name, type, created_at }, ...] ``` | Field | Required | Description | | -------- | -------- | ------------------------------------------------- | | `id` | **Yes** | Workspace identifier (used in API paths and `al://` URLs) | | `name` | No | Display name (defaults to `id` if omitted) | > **Note:** Omitting `id` returns HTTP 400 Bad Request with the message "workspace ID is required". --- ## Error Handling Both Python and Node.js share the same error hierarchy: ``` ManagedServiceError # Base class for all errors ├── AuthError # Token invalid/expired │ └── TokenError # Token parsing failed ├── CredentialsError # DB credentials fetch failed ├── IngestError # File ingestion failed ├── TableError # Table operation failed └── WorkspaceError # Workspace not found or inaccessible ``` ```python # Python imports from deeplake.managed import ( ManagedServiceError, AuthError, CredentialsError, IngestError, TableError, TokenError, WorkspaceError, ) ``` ```typescript // Node.js imports import { ManagedServiceError, AuthError, CredentialsError, IngestError, TableError, TokenError, WorkspaceError, } from 'deeplake'; ``` | Error | Cause | Solution | | ------------------------------------------ | ------------------------- | ----------------------------------------------------------- | | `AuthError: Token required` | No token provided | Pass `token=` to Client() or set `DEEPLAKE_API_KEY` env var | | `AuthError: Token does not contain org_id` | Token missing OrgID claim | Ensure token has OrgID claim or API `/me` is accessible | | `IngestError: File not found` | Invalid file path | Check file exists at given path | | `TableError: table creation failed` | API table creation failed | Check API server is running and workspace is accessible | | `WorkspaceError: No storage path` | API returned no path | Check workspace exists and has storage configured | > For troubleshooting details, see [reference.md](reference.md). --- ## Agent Decision Trees ### Decision: How to Initialize Client ``` Need to create a Client | |-- Python? | |-- DEEPLAKE_API_KEY env var is set? | | `-- client = Client() # defaults: token from env, workspace="default" | |-- Have explicit token? | | `-- client = Client([REDACTED_SECRET]") # workspace defaults to "default" | |-- Need specific workspace? | | `-- client = Client(workspace_id="my-ws") # token from env | `-- Need custom API URL? | `-- client = Client(api_url="http://custom:8080") | `-- Node.js? `-- import { ManagedClient, initializeWasm } from 'deeplake'; await initializeWasm(); const client = new ManagedClient({ [REDACTED_SECRET], workspaceId: "my-ws", // optional, default "default" apiUrl: "http://custom:8080", // optional }); ``` ### Decision: How to Ingest Data ``` User wants to ingest data | |-- Is it local files? -> use FILE schema | |-- Python: | | `-- client.ingest("table", {"path": ["f1.mp4", "f2.mp4"]}, | | schema={"path": "FILE"}) | `-- Node.js: | `-- await client.ingest("table", { path: ["f1.mp4"] }, | { schema: { path: "FILE" } }) | |-- Is it structured data (dict/lists)? -> pass a dict directly | |-- Python: | | `-- client.ingest("table", {"col1": [...], "col2": [...]}) | `-- Node.js: | `-- await client.ingest("table", { col1: [...], col2: [...] }) | |-- Is it a HuggingFace dataset? -> use _huggingface key | `-- client.ingest("table", {"_huggingface": "dataset_name"}) | |-- Is it a LeRobot robotics dataset? -> 3-table design (tasks + frames + episodes) | |-- from deeplake.managed.formats import LeRobotTasks, LeRobotFrames, LeRobotEpisodes | |-- client.ingest("tasks", format=LeRobotTasks(dataset_dir)) | |-- client.ingest("frames", format=LeRobotFrames(dataset_dir, chunk_start=0, chunk_end=3)) | `-- client.ingest("episodes", format=LeRobotEpisodes(dataset_dir, chunk_start=0, chunk_end=3)) | Note: chunk_end is inclusive. Episodes requires git lfs. See examples.md Workflow 7. | |-- Is it a domain-specific format (COCO, etc.)? -> use format object | |-- Python: client.ingest("table", format=my_format) | `-- Node.js: await client.ingest("table", null, { format: myFormat }) | See formats.md for built-in formats: CocoPanoptic, Coco, LeRobot, custom | |-- Need custom chunking for text? | `-- client.ingest("table", {"path": ["doc.txt"]}, | schema={"path": "FILE"}, | chunk_size=500, chunk_overlap=100) | `-- Need explicit schema? | `-- client.ingest("table", {...}, schema={ | "name": "TEXT", | "count": "INT64", | "vector": "EMBEDDING", | }) | `-- Need indexes for search performance? `-- client.ingest("table", {...}, index=["embedding", "text"]) Or standalone: client.create_index("table", "embedding") ``` ### Decision: How to Query Data ``` User wants to query data | |-- Simple SELECT (small result)? | |-- Python fluent: client.table("table").select("id", "text").limit(100)() | |-- Python raw: client.query("SELECT * FROM table LIMIT 100") | |-- Node fluent: await client.table("table").select("id", "text").limit(100).execute() | `-- Node raw: await client.query("SELECT * FROM table LIMIT 100") | |-- Large result set (streaming)? | `-- Use client.open_table("table") for direct dataset access | with batch iteration, PyTorch/TF DataLoaders, etc. | |-- Need semantic/vector search? | `-- client.query(""" | SELECT *, embedding <#> $1 AS score | FROM table ORDER BY score DESC LIMIT 10 | """, (query_embedding,)) | |-- Need text search? | `-- client.query(""" | SELECT * FROM table | WHERE text @> $1 | """, ("keyword",)) | `-- Need hybrid search (vector + text)? `-- client.query(""" SELECT *, (embedding, text)::deeplake_hybrid_record <#> deeplake_hybrid_record($1, $2, 0.7, 0.3) AS score FROM table ORDER BY score DESC LIMIT 10 """, (query_emb, "search text")) ``` ### Decision: User Wants to Train on Data ``` User wants to train / iterate over data | |-- Fast native batch iteration (RECOMMENDED for large datasets)? | |-- ds = client.open_table("table") | | for batch in ds.batches(256): # dict of numpy arrays | | states = torch.tensor(np.stack([batch[c] for c in cols], axis=1)) | |-- For column subsets, use query first: | | view = ds.query("SELECT col1, col2 WHERE episode_index < 100") | ` for batch in view.batches(256): ... | |-- Need PyTorch DataLoader (small datasets or need shuffle)? | `-- ds = client.open_table("table") | loader = DataLoader(ds.pytorch(), batch_size=32, shuffle=True) | NOTE: Slower on large remote datasets — prefer ds.batches() above | |-- Need TensorFlow tf.data? | `-- ds = client.open_table("table") | tf_ds = ds.tensorflow().batch(32).prefetch(AUTOTUNE) | `-- Training on LeRobot data? |-- Behavior cloning (state->action): | ds = client.open_table("droid_frames") | view = ds.query("SELECT state_x, ..., action_x, ... WHERE episode_index < 100") | for batch in view.batches(256): | states = torch.tensor(np.stack([batch[c] for c in STATE_COLS], axis=1)) | actions = torch.tensor(np.stack([batch[c] for c in ACTION_COLS], axis=1)) | # train(model, states, actions) | `-- Video-conditioned training: ds = client.open_table("droid_episodes") # Access video bytes via ds[i]["exterior_1_video"], etc. ``` ### Decision: Error Recovery ``` Operation failed with error | |-- AuthError? | |-- "Token required" -> Set DEEPLAKE_API_KEY env var or pass token= to Client() | |-- "Token does not contain org_id" -> Ensure token has OrgID claim | `-- "Token expired" -> Get new token | |-- IngestError? | |-- "data must be a dict" -> Pass a dict, not list/str/int | |-- "data must not be empty" -> Dict must have at least one key | |-- "File not found" -> Check file path exists | |-- "ffmpeg not found" -> Install ffmpeg for video processing | `-- "fitz not found" / "pdfjs-dist not found" -> Install pymupdf (Python) or pdfjs-dist (Node.js) | |-- TableError? | |-- "create_deeplake_table failed" -> Check pg_deeplake extension | |-- "Table already exists" -> Use drop_table() first or different name | |-- "Index creation failed" -> Check column exists and is EMBEDDING or TEXT type | `-- "Query timed out" -> Increase timeout: client.query(sql, timeout=300) | |-- WorkspaceError / "workspace ID is required"? | `-- POST /workspaces requires the "id" field in the request body. | Use: { id: "my-ws", name: "My Workspace" } | |-- "Not found: /workspaces/.../tables"? | `-- Workspace must exist before ingest(). Create it via the API or UI first. | |-- Tables created via raw SQL not visible to ingest()? | `-- Raw SQL tables (CREATE TABLE ... USING deeplake) are not registered with | the managed API. Use client.ingest() to create tables. Use client.query() | for raw SQL operations on manually-created tables. | |-- Thumbnail generation failed? (logged as warning, non-fatal) | |-- Python: Install Pillow (`pip install Pillow`) | `-- Node.js: Install sharp (`npm install sharp`) | `-- ManagedServiceError? `-- Check API server is running at the configured api_url ``` --- ## Supporting Files - **[reference.md](reference.md)** -- pg_deeplake SQL reference (vector search, BM25, hybrid search, indexes), data types, limits, performance tuning, troubleshooting - **[examples.md](examples.md)** -- Complete end-to-end workflow examples and detailed ingestion examples - **[formats.md](formats.md)** -- Format base class, custom format classes, normalize()/schema()/image_columns() rules

path: examples.md
content: # Deeplake Managed Service -- Examples

## Complete Workflow Examples

### Workflow 1: Ingest Videos and Search (Python)

```python
from deeplake import Client

# Initialize -- token from DEEPLAKE_API_KEY env var, workspace defaults to "default"
client = Client()

# Ingest video files (FILE schema)
result = client.ingest("security_videos", {
    "path": ["/path/to/camera1.mp4", "/path/to/camera2.mp4"],
}, schema={"path": "FILE"})
print(f"Ingested {result['row_count']} video segments")

# Fluent query for segments
segments = (
    client.table("security_videos")
        .select("id", "file_id", "start_time", "end_time", "text")
        .where("start_time > $1", 60)
        .limit(10)
)()

for seg in segments:
    print(f"Segment {seg['id']}: {seg['start_time']}s - {seg['end_time']}s")
```

### Workflow 2: Build Semantic Search Index (Python)

```python
from deeplake import Client

client = Client()

# Prepare documents with embeddings
documents = ["Doc about AI", "Doc about ML", "Doc about databases"]
embeddings = [[0.1]*384, [0.2]*384, [0.3]*384]  # Placeholder

# Ingest with indexes for fast search
client.ingest("search_index", {
    "text": documents,
    "embedding": embeddings,
}, index=["embedding", "text"])

# Search (uses deeplake_index automatically)
query_emb = [0.15]*384  # Placeholder

results = client.query("""
    SELECT text, embedding <#> $1 AS similarity
    FROM search_index
    ORDER BY similarity DESC
    LIMIT 5
""", (query_emb,))

for r in results:
    print(f"{r['similarity']:.3f}: {r['text']}")
```

### Workflow 3: Process PDF Documents (Python)

```python
from deeplake import Client

client = Client()

# Ingest PDFs (each page becomes a row)
result = client.ingest("manuals", {
    "path": ["/path/to/manual1.pdf", "/path/to/manual2.pdf"],
}, schema={"path": "FILE"})
print(f"Processed {result['row_count']} pages")

# Search within PDFs
pages = client.query("""
    SELECT file_id, page_index, text
    FROM manuals
    WHERE text @> 'installation'
""")

for page in pages:
    print(f"Found in file {page['file_id']}, page {page['page_index']}")
```

### Workflow 4: Iterate Over Large Datasets (Python)

```python
from deeplake import Client

client = Client()

# Use open_table() for large-scale iteration (bypasses PostgreSQL)
ds = client.open_table("large_table")
for batch in ds.batches(1000):
    process(batch)
```

### Workflow 5: Ingest COCO Panoptic and Query Segments (Python)

```python
from deeplake import Client
from deeplake.managed.formats import CocoPanoptic

client = Client()

# Ingest panoptic dataset using format object
# Thumbnails auto-generated for IMAGE columns
result = client.ingest("panoptic_train", format=CocoPanoptic(
    images_dir="/data/coco/train2017",
    masks_dir="/data/coco/panoptic_train2017",
    annotations="/data/coco/annotations/panoptic_train2017.json",
))
print(f"Ingested {result['row_count']} images")

# Query for images with specific categories
rows = client.query("""
    SELECT coco_image_id, filename, segments_info
    FROM panoptic_train
    LIMIT 10
""")

import json
for row in rows:
    segments = json.loads(row["segments_info"])
    print(f"Image {row['filename']}: {len(segments)} segments")
```

### Workflow 6: Ingest COCO Detection and Query (Python)

```python
from deeplake import Client
from deeplake.managed.formats import Coco

client = Client()

# Ingest detection dataset — auto-detects annotation files from split name
result = client.ingest("coco_val", format=Coco(
    images_dir="/data/coco/val2017",
    max_images=100,  # limit for testing
))
print(f"Ingested {result['row_count']} images")

# Query for images with many objects
rows = client.query("""
    SELECT coco_image_id, filename, num_objects, captions
    FROM coco_val
    WHERE num_objects > 10
    ORDER BY num_objects DESC
    LIMIT 5
""")

import json
for row in rows:
    caps = json.loads(row["captions"])
    print(f"Image {row['filename']}: {row['num_objects']} objects, captions: {caps}")
```

### Workflow 7: Ingest LeRobot Robotics Dataset and Connect to Training (Python)

```python
from deeplake import Client
from deeplake.managed.formats import LeRobotTasks, LeRobotFrames, LeRobotEpisodes

client = Client()

DATASET_DIR = "/path/to/lerobot_dataset"  # HuggingFace LeRobot v2.0 format

# LeRobot uses a 3-table design: tasks, frames, episodes.
# Ingest them in order (tasks first — it's the lookup table).

# 1. Tasks — fast, reads meta/tasks.jsonl (~31K rows for DROID)
client.ingest("droid_tasks", format=LeRobotTasks(DATASET_DIR))

# 2. Frames — per-frame state/action scalars from parquet files
#    chunk_start/chunk_end control which chunks to process (inclusive)
#    Each chunk = 1000 episodes. DROID has 93 chunks total.
client.ingest("droid_frames", format=LeRobotFrames(
    DATASET_DIR, chunk_start=0, chunk_end=3,  # first 4 chunks
))

# 3. Episodes — video data + metadata, pulls LFS videos lazily per chunk
#    Each chunk's videos are pulled via git-lfs, processed, then freed.
#    ~40s per chunk (LFS pull + read 3 camera MP4s per episode).
client.ingest("droid_episodes", format=LeRobotEpisodes(
    DATASET_DIR, chunk_start=0, chunk_end=3,
))

# Query frame data
frames = client.query("""
    SELECT episode_index, frame_index, state_x, state_y, action_x, action_y
    FROM droid_frames
    WHERE episode_index = 0
    ORDER BY frame_index
    LIMIT 10
""")
for f in frames:
    print(f"Frame {f['frame_index']}: state=({f['state_x']:.3f}, {f['state_y']:.3f})")

# Connect to training — open_table() bypasses SQL, returns native dataset
ds = client.open_table("droid_frames")
print(f"Dataset: {len(ds)} frames")

# RECOMMENDED: Use ds.query() + ds.batches() for fast training on large datasets.
# ds.pytorch() with DataLoader works but is slower on remote data due to per-row access.
import torch
import torch.nn as nn
import numpy as np

STATE_COLS = ["state_x","state_y","state_z","state_roll","state_pitch","state_yaw","state_pad","state_gripper"]
ACTION_COLS = ["action_x","action_y","action_z","action_roll","action_pitch","action_yaw","action_gripper"]

# Get a training subset (query runs server-side, fast)
train_view = ds.query("SELECT state_x, state_y, state_z, state_roll, state_pitch, state_yaw, state_pad, state_gripper, action_x, action_y, action_z, action_roll, action_pitch, action_yaw, action_gripper WHERE episode_index < 100")

# Simple behavior cloning model
model = nn.Sequential(nn.Linear(8, 64), nn.ReLU(), nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 7))
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for epoch in range(3):
    total_loss = 0
    n = 0
    for batch in train_view.batches(256):  # dict of numpy arrays, very fast
        states = torch.tensor(np.stack([batch[c] for c in STATE_COLS], axis=1), dtype=torch.float32)
        actions = torch.tensor(np.stack([batch[c] for c in ACTION_COLS], axis=1), dtype=torch.float32)
        loss = nn.MSELoss()(model(states), actions)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
        n += 1
    print(f"Epoch {epoch}: loss={total_loss/n:.6f}")
```

**Important notes for LeRobot ingestion:**
- `chunk_end` is **inclusive** — `chunk_start=0, chunk_end=3` processes chunks 0, 1, 2, 3
- Videos require `git lfs` installed and the dataset cloned (not just downloaded)
- Episodes ingestion pulls ~40s per chunk for LFS, then restores pointers to save disk
- For large datasets, ingest in stages: do chunks 0-9 first, verify, then continue
- Tables support **append**: call `ingest()` again with the same table name and different chunk range to add more data
- Dependencies: `pandas`, `numpy` (for frames); `git lfs` (for episodes)

**Training performance notes:**
- `ds.batches(N)` returns dicts of numpy arrays — ~10x faster than `DataLoader(ds.pytorch())`
- Use `ds.query("SELECT ... WHERE ...")` to filter columns/rows server-side before training
- 3 epochs over 19K frames completes in ~10s with `ds.batches(256)`
- For PyTorch DataLoader compatibility (e.g., shuffling), use on smaller subsets only

### Workflow 8: Node.js -- Ingest and Query (TypeScript)

```typescript
import { ManagedClient, CocoPanoptic, initializeWasm } from 'deeplake';

await initializeWasm();

const client = new ManagedClient({
    [REDACTED_SECRET],
    workspaceId: 'default',
});

// Ingest COCO panoptic
const result = await client.ingest("panoptic", null, {
    format: new CocoPanoptic({
        imagesDir: "coco/train2017",
        masksDir: "coco/panoptic_train2017",
        annotations: "coco/annotations/panoptic_train2017.json",
    }),
});
console.log(`Ingested ${result.rowCount} images`);

// Fluent query
const rows = await client.table("panoptic")
    .select("coco_image_id", "filename", "segments_info")
    .limit(10)
    .execute();

for (const row of rows) {
    const segments = JSON.parse(row.segments_info as string);
    console.log(`Image ${row.filename}: ${segments.length} segments`);
}

// Table management
const tables = await client.listTables();
console.log("Tables:", tables);
```

### Workflow 8: Node.js -- Ingest Structured Data (TypeScript)

```typescript
import { ManagedClient, initializeWasm } from 'deeplake';

await initializeWasm();

const client = new ManagedClient({
    [REDACTED_SECRET],
});

// Ingest structured data
await client.ingest("embeddings", {
    text: ["Hello world", "Goodbye world"],
    embedding: [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]],
    score: [0.9, 0.8],
});

// Vector similarity search
const queryEmb = [0.15, 0.25, 0.35];
const results = await client.query(
    `SELECT text, embedding <#> $1 AS similarity
     FROM embeddings
     ORDER BY similarity DESC
     LIMIT 5`,
    [queryEmb],
);

for (const r of results) {
    console.log(`${r.similarity}: ${r.text}`);
}
```

---

## Detailed Ingestion Examples

### Ingest Video Files (FILE schema)

```python
client.ingest("security_footage", {
    "path": ["camera1_2025-01-15.mp4", "camera2_2025-01-15.mp4"],
}, schema={"path": "FILE"})
# Creates ~10-second segments with thumbnails
# Each segment has: id, file_id, chunk_index, start_time, end_time, video_data, thumbnail, text
```

### Ingest Text Documents

```python
client.ingest("documents", {
    "path": ["report.txt", "notes.md", "data.json"],
}, schema={"path": "FILE"})
# Text is chunked into ~1000 char pieces with 200 char overlap
# Each chunk has: id, file_id, chunk_index, text
```

### Ingest Images

```python
client.ingest("photos", {
    "path": ["image1.jpg", "image2.png"],
}, schema={"path": "FILE"})
# Each image stored as single row
# Columns: id, file_id, image (binary), filename, text
```

### Ingest PDFs

```python
client.ingest("manuals", {"path": ["manual.pdf"]}, schema={"path": "FILE"})
# Each page rendered at 300 DPI as PNG
# Columns: id, file_id, page_index, image (binary), text (extracted)
```

### Ingest with Progress Callback

```python
def progress(rows_written, total):
    print(f"Written {rows_written} rows...")

client.ingest("documents", {"path": pdf_files}, schema={"path": "FILE"}, on_progress=progress)
```

### Ingest Structured Data (dict = column data)

```python
client.ingest("vectors", {
    "id": ["doc1", "doc2", "doc3"],
    "text": ["Hello world", "Goodbye world", "Another doc"],
    "embedding": [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]],
    "score": [0.9, 0.8, 0.7],
})
# Schema inferred from data types
# Embeddings auto-detected as float arrays
```

### Ingest with Explicit Schema

```python
client.ingest("data", {"name": ["Alice", "Bob"], "age": [30, 25]},
              schema={"name": "TEXT", "age": "INT64"})
```

### Ingest from HuggingFace

```python
client.ingest("mnist", {"_huggingface": "mnist"})
client.ingest("cifar", {"_huggingface": "cifar10"})
client.ingest("squad", {"_huggingface": "squad"})
```

### Ingest COCO Panoptic Data (format object)

```python
from deeplake.managed.formats import CocoPanoptic

client.ingest("panoptic", format=CocoPanoptic(
    images_dir="coco/train2017",
    masks_dir="coco/panoptic_train2017",
    annotations="coco/annotations/panoptic_train2017.json",
))
# Each image becomes one row with columns:
# coco_image_id (int), image (IMAGE), mask (SEGMENT_MASK), width (int), height (int),
# filename (str), mask_filename (str), segments_info (JSON str), categories (JSON str)
# Thumbnails auto-generated for IMAGE columns at 32x32, 64x64, 128x128, 256x256
```

### Ingest COCO Detection/Captions Data (format object)

```python
from deeplake.managed.formats import Coco

client.ingest("coco_train", format=Coco(
    images_dir="coco/train2017",
    annotations_dir="coco/annotations",
))
# Each image becomes one row with columns:
# coco_image_id (int), image (IMAGE), mask (SEGMENT_MASK), width (int), height (int),
# filename (str), captions (JSON str), annotations (JSON str),
# categories (JSON str), num_objects (int)
# Requires: pycocotools, Pillow, numpy
```


path: formats.md
content: # Deeplake Managed Service -- Format Classes

Format classes normalize domain-specific dataset layouts into column-oriented batches for `Client.ingest()`. All format classes inherit from the `Format` abstract base class.

Both Python and Node.js implement the same Format interface.

## Format Base Class

### Python

```python
from deeplake.managed.formats import Format

class Format(ABC):
    @abstractmethod
    def normalize(self) -> Iterator[Dict[str, List[Any]]]:
        """Yield column-oriented batches: {col_name: [values...]}."""
        ...

    def schema(self) -> Dict[str, str]:
        """Return {col: deeplake_type} hints. Override to declare types."""
        return {}

    def pg_schema(self) -> Dict[str, str]:
        """Return {col: pg_domain_type} overrides. Override for domain types."""
        return {}

    def image_columns(self) -> List[str]:
        """Return column names containing IMAGE data (for thumbnail generation).
        Default: auto-detect from pg_schema() -- columns with type "IMAGE".
        """
        return [col for col, t in self.pg_schema().items() if t == "IMAGE"]
```

### Node.js / TypeScript

```typescript
import type { Format, Batch, SchemaMap } from 'deeplake';

interface Format {
    normalize(): Iterable<Batch> | AsyncIterable<Batch>;
    schema?(): SchemaMap;               // { col: deeplake_type }
    pg_schema?(): Record<string, string>; // { col: pg_domain_type }
    image_columns?(): string[];          // columns for thumbnail generation
}
```

### Method Summary

| Method            | Required | Purpose                                           |
| ----------------- | -------- | ------------------------------------------------- |
| `normalize()`     | Yes      | Yield `dict[str, list]` batches                   |
| `schema()`        | No       | Declare deeplake storage types (e.g. `"BINARY"`)  |
| `pg_schema()`     | No       | Declare PG domain types (e.g. `"IMAGE"`)          |
| `image_columns()` | No       | List IMAGE columns for thumbnail auto-generation  |

---

## Type Inference

Column types are inferred automatically from Python/JS values when no schema hint is provided:

| Python type   | JS/TS type          | Inferred schema type |
| ------------- | ------------------- | -------------------- |
| `bool`        | `boolean`           | BOOL                 |
| `int`         | `number` (integer)  | INT64                |
| `float`       | `number` (decimal)  | FLOAT64              |
| `str`         | `string`            | TEXT                 |
| `bytes`       | `Buffer/Uint8Array` | BINARY               |
| `list[float]` | `number[]`          | EMBEDDING            |

---

## Minimal Example -- CSV with Linked Images

```python
from deeplake.managed.formats import Format

class CsvWithImages(Format):
    """Format for a CSV file where one column contains image file paths."""

    def __init__(self, csv_path, images_dir):
        self.csv_path = Path(csv_path)
        self.images_dir = Path(images_dir)

    def schema(self):
        return {"image": "BINARY"}

    def pg_schema(self):
        return {"image": "IMAGE"}

    def normalize(self):
        import csv

        with open(self.csv_path) as f:
            rows = list(csv.DictReader(f))

        batch = []
        for row in rows:
            img_path = self.images_dir / row["image_file"]
            if not img_path.exists():
                continue
            batch.append({
                "image": img_path.read_bytes(),        # bytes -> IMAGE (via pg_schema())
                "label": row["label"],                 # str -> TEXT
                "confidence": float(row["confidence"]), # float -> FLOAT64
            })
            if len(batch) >= 100:
                yield {k: [d[k] for d in batch] for k in batch[0]}
                batch = []
        if batch:
            yield {k: [d[k] for d in batch] for k in batch[0]}

# Usage
client.ingest("labeled_images", format=CsvWithImages(
    csv_path="annotations.csv",
    images_dir="images/",
))
# Thumbnails auto-generated for "image" column (declared as IMAGE in pg_schema)
```

---

## Key Rules for `normalize()`

1. **Yield `dict[str, list]`** -- each dict maps column names to equal-length lists (one entry per row)
2. **Use consistent column names** across all yielded batches
3. **Use native Python types** -- `int`, `float`, `str`, `bytes`, `bool`, `list[float]` for embeddings. Types are inferred from the first batch
4. **Batch for memory** -- don't load entire dataset into one dict; yield batches of ~100-1000 rows
5. **Validate in `normalize()`** -- raise `FileNotFoundError` or `ValueError` early for missing files or bad data
6. **Skip bad rows with logging** -- use `logger.warning()` and `continue` for recoverable issues (missing files, etc.)

---

## Optional `schema()` and `pg_schema()` Methods

- `schema()` returns `dict[str, str]` mapping column names to deeplake storage type names (e.g. `{"image": "BINARY", "mask": "BINARY"}`)
- `pg_schema()` returns `dict[str, str]` mapping column names to PostgreSQL domain types (e.g. `{"image": "IMAGE", "mask": "SEGMENT_MASK"}`)
- These hints override type inference for columns where Python types are ambiguous (e.g. `bytes` could be `BINARY`, `IMAGE`, `SEGMENT_MASK`, etc.)
- User-supplied schema to `ingest()` takes precedence over format schema hints
- Available domain types: `IMAGE`, `SEGMENT_MASK`, `BINARY_MASK`, `BOUNDING_BOX`, `CLASS_LABEL`, `POLYGON`, `POINT`, `MESH`, `MEDICAL`, `VIDEO`

---

## Thumbnail Auto-Generation

When a format's `image_columns()` returns column names (default: columns with `pg_schema()` type `"IMAGE"`), the SDK automatically generates thumbnails after ingestion:

- **Sizes:** 32x32, 64x64, 128x128, 256x256 (aspect-preserving)
- **Format:** JPEG, quality 85
- **Storage:** Shared `thumbnails` dataset at `{root_path}/thumbnails` with columns: `file_id`, `column_name`, `dimension`, `content`
- **Requires:** Pillow (Python) or sharp (Node.js)

Override `image_columns()` to customize which columns get thumbnails:

```python
class MyFormat(Format):
    def pg_schema(self):
        return {"photo": "IMAGE", "icon": "IMAGE"}

    def image_columns(self):
        return ["photo"]  # Only generate thumbnails for "photo", not "icon"
```

---

## Multi-File Format with Embeddings

```python
class ImageFolderWithEmbeddings(Format):
    """Images in subfolders (subfolder name = label) + pre-computed embeddings."""

    def __init__(self, root_dir, embeddings_path):
        self.root_dir = Path(root_dir)
        self.embeddings_path = Path(embeddings_path)

    def normalize(self):
        import json

        with open(self.embeddings_path) as f:
            embeddings = json.load(f)

        batch = []
        for label_dir in sorted(self.root_dir.iterdir()):
            if not label_dir.is_dir():
                continue
            label = label_dir.name
            for img_path in sorted(label_dir.glob("*.jpg")):
                emb = embeddings.get(img_path.name)
                if emb is None:
                    continue
                batch.append({
                    "image": img_path.read_bytes(),
                    "label": label,
                    "embedding": emb,
                    "filename": img_path.name,
                })
                if len(batch) >= 100:
                    yield {k: [d[k] for d in batch] for k in batch[0]}
                    batch = []
        if batch:
            yield {k: [d[k] for d in batch] for k in batch[0]}

client.ingest("classified_images", format=ImageFolderWithEmbeddings(
    root_dir="imagenet/train",
    embeddings_path="embeddings.json",
))
```

---

## Built-in Formats

### CocoPanoptic

```python
# Python
from deeplake.managed.formats import CocoPanoptic

client.ingest("panoptic", format=CocoPanoptic(
    images_dir="coco/train2017",
    masks_dir="coco/panoptic_train2017",
    annotations="coco/annotations/panoptic_train2017.json",
    batch_size=100,  # optional, default 100
))
```

```typescript
// Node.js
import { CocoPanoptic } from 'deeplake';

await client.ingest("panoptic", null, {
    format: new CocoPanoptic({
        imagesDir: "coco/train2017",
        masksDir: "coco/panoptic_train2017",
        annotations: "coco/annotations/panoptic_train2017.json",
        batchSize: 100,  // optional, default 100
    }),
});
```

**Parameters:**
- `images_dir` / `imagesDir` -- Directory containing the images (e.g. `train2017/`)
- `masks_dir` / `masksDir` -- Directory containing panoptic segmentation masks (e.g. `panoptic_train2017/`)
- `annotations` -- Path to the panoptic annotations JSON file
- `batch_size` / `batchSize` -- Images per batch (default 100)

**Generated columns:**
| Column          | Type           | Description                    |
| --------------- | -------------- | ------------------------------ |
| `coco_image_id` | int            | COCO image ID                  |
| `image`         | IMAGE          | Image binary data              |
| `mask`          | SEGMENT_MASK   | Panoptic segmentation mask PNG |
| `width`         | int            | Image width in pixels          |
| `height`        | int            | Image height in pixels         |
| `filename`      | str            | Original image filename        |
| `mask_filename` | str            | Panoptic mask PNG filename     |
| `segments_info` | str (JSON)     | Per-segment category/area info |
| `categories`    | str (JSON)     | Category definitions           |

**Note:** No external dependencies for CocoPanoptic -- uses only stdlib/built-in JSON and file I/O.

### Coco (Detection/Captions)

```python
# Python (requires pycocotools, Pillow, numpy)
from deeplake.managed.formats import Coco

client.ingest("coco_train", format=Coco(
    images_dir="coco/train2017",
    annotations_dir="coco/annotations",
))
```

```typescript
// Node.js (no external deps -- pure JS mask rendering)
import { Coco } from 'deeplake';

await client.ingest("coco_train", null, {
    format: new Coco({
        imagesDir: "coco/train2017",
        annotationsDir: "coco/annotations",
    }),
});
```

**Parameters:**
- `images_dir` / `imagesDir` -- Directory containing images
- `annotations_dir` / `annotationsDir` -- Directory containing annotation JSONs (optional, defaults to `<images_dir>/../annotations`)
- `instances_json` / `instancesJson` -- Explicit instances JSON path (optional, auto-detected from split name)
- `captions_json` / `captionsJson` -- Explicit captions JSON path (optional, `False`/`false` to skip)
- `batch_size` / `batchSize` -- Images per batch (default 100)
- `max_images` / `maxImages` -- Limit number of images (optional)

**Generated columns:**
| Column          | Type           | Description                                       |
| --------------- | -------------- | ------------------------------------------------- |
| `coco_image_id` | int            | COCO image ID                                     |
| `image`         | IMAGE          | Source image file bytes                            |
| `mask`          | SEGMENT_MASK   | Combined segmentation mask PNG (category per pixel)|
| `width`         | int            | Image width in pixels                             |
| `height`        | int            | Image height in pixels                            |
| `filename`      | str            | Original image filename                           |
| `captions`      | str (JSON)     | JSON list of caption strings                      |
| `annotations`   | str (JSON)     | JSON list of annotation dicts (bbox, category_id, area, iscrowd) |
| `categories`    | str (JSON)     | Category definitions (same for all rows)          |
| `num_objects`   | int            | Number of object annotations for this image       |

### LeRobot (3-table design)

```python
# Python
from deeplake.managed.formats import LeRobotTasks, LeRobotFrames, LeRobotEpisodes

client.ingest("tasks", format=LeRobotTasks(dataset_dir="lerobot_data"))
client.ingest("frames", format=LeRobotFrames(dataset_dir="lerobot_data", chunk_start=0, chunk_end=10))
client.ingest("episodes", format=LeRobotEpisodes(dataset_dir="lerobot_data", chunk_start=0, chunk_end=10))
```

```typescript
// Node.js
import { LeRobotTasks, LeRobotFrames, LeRobotEpisodes } from 'deeplake';

await client.ingest("tasks", null, { format: new LeRobotTasks({ datasetDir: "lerobot_data" }) });
await client.ingest("frames", null, { format: new LeRobotFrames({ datasetDir: "lerobot_data", chunkStart: 0, chunkEnd: 10 }) });
await client.ingest("episodes", null, { format: new LeRobotEpisodes({ datasetDir: "lerobot_data", chunkStart: 0, chunkEnd: 10 }) });
```

**LeRobotTasks columns:**
| Column       | Type | Description              |
| ------------ | ---- | ------------------------ |
| `task_index` | int  | Task index               |
| `task`       | str  | Task description string  |

**LeRobotFrames columns:**
| Column          | Type  | Description                   |
| --------------- | ----- | ----------------------------- |
| `index`         | int   | Global frame index            |
| `episode_index` | int   | Episode this frame belongs to |
| `frame_index`   | int   | Frame index within episode    |
| `task_index`    | int   | Task index                    |
| `timestamp`     | float | Frame timestamp               |
| `state_x` .. `state_gripper` | float | 8 unpacked state scalars |
| `action_x` .. `action_gripper` | float | 7 unpacked action scalars |

**LeRobotEpisodes columns:**
| Column             | Type   | Description                  |
| ------------------ | ------ | ---------------------------- |
| `episode_index`    | int    | Episode index                |
| `task`             | str    | Task description             |
| `length`           | int    | Number of frames in episode  |
| `exterior_1_video` | VIDEO  | Exterior camera 1 MP4 bytes  |
| `exterior_2_video` | VIDEO  | Exterior camera 2 MP4 bytes  |
| `wrist_video`      | VIDEO  | Wrist camera MP4 bytes       |


path: README.md
content: # Deeplake Agent Skills

Agent skills for working with [Deeplake](https://deeplake.ai), the GPU database.

## Installation

```bash
npx skills add activeloopai/deeplake-skills
```

## Available Skills

| Skill                | Description                                                                                                                                         |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| **deeplake-managed** | SDK for ingesting data into Deeplake managed tables (Python & Node.js/TypeScript). Use when users want to store, ingest, or query data in Deeplake. |

## Usage

Once installed, the skill is automatically available. The agent will use it when it detects tasks related to Deeplake data ingestion, querying, or management.

Example prompts:
- "Ingest these video files into Deeplake"
- "Create a Deeplake table with embeddings"
- "Query my Deeplake dataset for similar vectors"
- "Set up a PyTorch DataLoader from my Deeplake table"
- "Ingest COCO detection data into Deeplake using Node.js"

## Skill Structure

```
skills/
  deeplake-managed/
    SKILL.md        # Main agent instructions (Python + Node.js)
    reference.md    # pg_deeplake SQL reference (vector search, BM25, hybrid search)
    examples.md     # End-to-end workflow examples (Python + Node.js)
    formats.md      # Custom format classes for ingestion (Python + Node.js)
```

## License

MIT


path: reference.md
content: # Deeplake Managed Service -- Reference

## pg_deeplake SQL Reference

Once data is ingested, use these SQL features:

### Vector Similarity Search (Cosine)

The `<#>` operator is overloaded by pg_deeplake -- it performs **vector cosine similarity** when applied to embedding columns and **BM25 text similarity** when applied to text columns. In both cases, higher scores = more similar, so use `ORDER BY ... DESC`.

**Note:** Unlike pgvector's `<=>` (distance, lower = closer), pg_deeplake's `<#>` returns **similarity** (higher = more similar). Always use `ORDER BY ... DESC`.

```sql
-- <#> on embedding column: cosine similarity (higher = more similar)
SELECT id, text, embedding <#> $query_embedding AS similarity
FROM documents
ORDER BY similarity DESC
LIMIT 10;
```

**Python example:**
```python
import numpy as np

query_embedding = model.encode("search query").tolist()  # Must be list, not numpy array

results = client.query("""
    SELECT id, text, embedding <#> $1 AS similarity
    FROM embeddings
    ORDER BY similarity DESC
    LIMIT 10
""", (query_embedding,))

for row in results:
    print(f"{row['similarity']:.3f}: {row['text']}")
```

### BM25 Text Search

> **Important:** Use the `<#>` operator for BM25 search. The `BM25_SIMILARITY()` function form is **not supported** on the managed query endpoint and will return a 400 error.

```sql
-- <#> on text column: BM25 text similarity (higher = more relevant)
SELECT id, text, text <#> 'search query' AS score
FROM documents
ORDER BY score DESC
LIMIT 10;
```

**Python example:**
```python
results = client.query("""
    SELECT id, text, text <#> $1 AS score
    FROM documents
    ORDER BY score DESC
    LIMIT 10
""", ("machine learning",))
```

### Full-Text Contains

```sql
-- Check if text contains keyword
SELECT * FROM documents
WHERE text @> 'important keyword';
```

### Hybrid Search (Vector + Text)

```sql
-- Combine vector and text search with weights
-- deeplake_hybrid_record(embedding, text, vector_weight, text_weight)
-- Weights: 0.7 = 70% vector similarity, 0.3 = 30% BM25 text similarity
SELECT id,
       (embedding, text)::deeplake_hybrid_record <#>
       deeplake_hybrid_record($query_embedding, 'search text', 0.7, 0.3) AS score
FROM documents
ORDER BY score DESC
LIMIT 10;
```

**Python example:**
```python
query_emb = model.encode("neural networks").tolist()
results = client.query("""
    SELECT id, text,
           (embedding, text)::deeplake_hybrid_record <#>
           deeplake_hybrid_record($1, $2, 0.7, 0.3) AS score
    FROM documents
    ORDER BY score DESC
    LIMIT 10
""", (query_emb, "neural networks"))
```

### Create Indexes for Performance

```sql
-- Vector index (speeds up similarity search)
CREATE INDEX ON documents USING deeplake_index (embedding);

-- Text index (speeds up BM25 search)
CREATE INDEX ON documents USING deeplake_index (text);
```

**Via SDK (recommended):**

```python
# Python — standalone
client.create_index("documents", "embedding")
client.create_index("documents", "text")

# Python — during ingestion
client.ingest("documents", data, index=["embedding", "text"])
```

```typescript
// Node.js — standalone
await client.createIndex("documents", "embedding");
await client.createIndex("documents", "text");

// Node.js — during ingestion
await client.ingest("documents", data, { index: ["embedding", "text"] });
```

### Raw SQL: Inserting Image Bytes

When inserting image data via raw SQL (not `client.ingest()`), use `decode()` with the `IMAGE` cast:

```sql
INSERT INTO images (id, img)
VALUES ('img_1', decode('89504e470d0a1a0a...', 'hex')::IMAGE);
```

```python
image_hex = image_bytes.hex()
client.query(
    "INSERT INTO images (id, img) VALUES ($1, decode($2, 'hex')::IMAGE)",
    ("img_1", image_hex)
)
```

### Raw SQL: Vector Literals

When using vector values as SQL literals (e.g. for benchmarking without parameterized queries):

```sql
SELECT id, text, embedding <#> ARRAY[0.1, 0.2, 0.3]::FLOAT4[] AS score
FROM documents
ORDER BY score DESC
LIMIT 10;
```

> **Tip:** For production use, prefer parameterized queries (`$1`) over literals.

---

## Data Types Reference

| Schema Type    | Python Type | JS/TS Type        | Postgres Type             | Example                 |
| -------------- | ----------- | ----------------- | ------------------------- | ----------------------- |
| `TEXT`         | str         | string            | text                      | `"hello"`               |
| `INT32`        | int         | number            | integer                   | `42`                    |
| `INT64`        | int         | number            | bigint                    | `9999999999`            |
| `FLOAT32`      | float       | number            | real                      | `3.14`                  |
| `FLOAT64`      | float       | number            | double precision          | `3.14159265359`         |
| `BOOL`         | bool        | boolean           | boolean                   | `True`                  |
| `BINARY`       | bytes       | Buffer/Uint8Array | bytea                     | `b"\x00\x01"`           |
| `IMAGE`        | bytes       | Buffer/Uint8Array | IMAGE (bytea)             | Image binary data       |
| `VIDEO`        | bytes       | Buffer/Uint8Array | bytea                     | Video binary data       |
| `EMBEDDING`    | list[float] | number[]          | float4[]                  | `[0.1, 0.2, 0.3]`       |
| `SEGMENT_MASK` | bytes       | Buffer/Uint8Array | SEGMENT_MASK (bytea)      | Segmentation mask data  |
| `BINARY_MASK`  | bytes       | Buffer/Uint8Array | BINARY_MASK (bytea)       | Binary mask data        |
| `BOUNDING_BOX` | list[float] | number[]          | BOUNDING_BOX (float4[])   | `[x, y, w, h]`          |
| `CLASS_LABEL`  | int         | number            | CLASS_LABEL (int4)        | Label index             |
| `POLYGON`      | bytes       | Buffer/Uint8Array | DEEPLAKE_POLYGON (bytea)  | Polygon coordinates     |
| `POINT`        | list[float] | number[]          | DEEPLAKE_POINT (float4[]) | `[1.0, 2.0]`            |
| `MESH`         | bytes       | Buffer/Uint8Array | MESH (bytea)              | 3D mesh data (PLY, STL) |
| `MEDICAL`      | bytes       | Buffer/Uint8Array | MEDICAL (bytea)           | Medical imaging (DICOM) |
| `FILE`         | str (path)  | string (path)     | N/A (processed)           | `"/path/to/file.mp4"`   |

> **Note:** `EMBEDDING` is a Deeplake schema type that maps to `float4[]` in PostgreSQL. pg_deeplake treats these columns specially for vector similarity search via the `<#>` operator and `deeplake_index`.
>
> **Note:** `FILE` is a schema directive, not a storage type. Columns marked as `FILE` are treated as file paths during ingestion -- the files are processed (chunked, etc.) and the resulting data is stored in generated columns. The `FILE` column itself is not stored in the dataset.
>
> **Note:** Domain types like `IMAGE`, `SEGMENT_MASK`, `BINARY_MASK`, `BOUNDING_BOX`, `CLASS_LABEL`, `DEEPLAKE_POLYGON`, `DEEPLAKE_POINT`, `MESH`, and `MEDICAL` are PostgreSQL domain types defined by pg_deeplake. They behave like their base types (bytea, float4[], int4) but carry semantic meaning for visualization, search, and type-aware processing.
>
> **Important -- IMAGE columns for UI display:** To display images correctly in the UI, explicitly set `schema={"image_col": "IMAGE"}` during ingestion. Without this, bytes columns are stored as generic `BINARY` and the UI will not render them as images.
>
> **Important -- IMAGE query results:** IMAGE columns returned via `client.query()` may come back as **base64-encoded strings** rather than raw bytes, depending on the backend serialization. Always handle both types:
> ```python
> import base64
> val = row["image"]
> if isinstance(val, str):
>     image_bytes = base64.b64decode(val)
> else:
>     image_bytes = val
> ```

**Schema inference:**
- `bool` / `boolean` -> BOOL
- `int` / `number` (integer) -> INT64
- `float` / `number` (decimal) -> FLOAT64
- `bytes` / `Buffer` -> BINARY
- `str` / `string` -> TEXT
- `list[float]` / `number[]` -> EMBEDDING (size auto-detected)

---

## Thumbnail Table Schema

When a format declares `image_columns()`, the SDK auto-generates a shared `thumbnails` dataset at `{root_path}/thumbnails` with this schema:

| Column        | Type   | Description                                  |
| ------------- | ------ | -------------------------------------------- |
| `file_id`     | TEXT   | UUID of the source row (`_id` column value)  |
| `column_name` | TEXT   | Name of the IMAGE column (e.g. `"image"`)    |
| `dimension`   | TEXT   | Thumbnail size (e.g. `"32x32"`, `"256x256"`) |
| `content`     | BINARY | JPEG thumbnail bytes (quality 85)            |

**Sizes generated:** 32x32, 64x64, 128x128, 256x256 (aspect-preserving via `Image.thumbnail()` in Python, `sharp.resize({ fit: 'inside' })` in Node.js).

---

## Limits

| Resource                     | Limit           |
| ---------------------------- | --------------- |
| Video chunk duration         | 10 seconds      |
| Text chunk size (default)    | 1000 characters |
| Text chunk overlap (default) | 200 characters  |
| PDF rendering resolution     | 150 DPI (configurable via `pdf_dpi`) |
| Batch size (data ingest)     | 1000 rows       |
| Write buffer (flush_every)   | 200 rows        |
| Commit interval              | 2000 rows       |
| File normalization workers   | 4 threads       |
| Storage I/O concurrency      | 32 operations   |

---

## Performance Tuning

The SDK uses several optimizations to handle large-scale ingestion efficiently:

**Buffered writes:** Instead of calling `ds.append()` for each small batch (e.g., one image or text chunk), rows are accumulated in a memory buffer and flushed in larger batches. This reduces Python-to-C++ FFI overhead (or JS-to-WASM overhead in Node.js).

```python
# Default: flush every 200 rows, commit every 2000 rows
client.ingest("table", files)

# These are fixed internal defaults (not configurable via ingest()):
# flush_every=200   -- rows buffered before ds.append()
# commit_every=2000 -- rows between ds.commit() calls
```

**Periodic commits:** `ds.commit()` is called every 2000 rows (default) to:
- Free C++ memory buffers
- Enable crash recovery (partial progress is persisted)
- Bound peak memory usage for very large ingestions

A final `ds.commit()` is always called after all rows are written.

**Parallel file normalization (Python only):** When ingesting multiple files, normalization (ffmpeg for video, PyMuPDF for PDFs, file I/O for images/text) runs in a thread pool (up to 4 workers). Since these operations release the GIL, threads provide real parallelism.

**Storage concurrency (Python only):** The SDK sets `deeplake.storage.set_concurrency(32)` during ingestion to parallelize S3/GCS chunk uploads, significantly improving throughput for large datasets.

| Parameter             | Default | Description                        |
| --------------------- | ------- | ---------------------------------- |
| `flush_every`         | 200     | Rows buffered before `ds.append()` |
| `commit_every`        | 2000    | Rows between `ds.commit()` calls   |
| Normalization workers | 4       | Max threads for file processing    |
| Storage concurrency   | 32      | Parallel storage I/O operations    |

---

## Troubleshooting

**"Token required" error:**
```bash
# Set the DEEPLAKE_API_KEY env var, or pass token= to Client()
export DEEPLAKE_API_KEY="dl_xxx"
```

**"Token does not contain org_id" error:**
```python
# Ensure your token contains an OrgID claim, or that the API /me endpoint
# is accessible to fall back on. All tokens should contain OrgID in their JWT payload.
```

**"ffmpeg not found" for video processing:**
```bash
# Install ffmpeg
sudo apt-get install ffmpeg
```

**"fitz not found" for PDF processing (Python):**
```bash
# Install PyMuPDF
pip install pymupdf
```

**Thumbnail generation skipped (Python):**
```bash
# Install Pillow
pip install Pillow
```

**Thumbnail generation skipped (Node.js):**
```bash
# Install sharp
npm install sharp
```

**Connection refused to API:**
```bash
# Check API server is running
curl https://api.deeplake.ai/health
```

**"workspace ID is required" when creating a workspace (HTTP 400):**
```
# POST /workspaces requires the "id" field in the request body.
# The "id" is used in API paths and al:// URLs.
# Example: { "id": "my-workspace", "name": "My Workspace" }
```

**"Not found: /workspaces/.../tables" during ingest:**
```
# The workspace must exist before calling ingest().
# Create it via the API or UI first, then ingest.
```

**Tables created via raw SQL not visible to ingest():**
```
# Tables created with raw SQL (CREATE TABLE ... USING deeplake) are not
# registered with the managed API. The SDK expects tables created via
# POST /workspaces/{id}/tables, which registers the al:// path.
# Use client.ingest() to create tables, or use client.query() for raw SQL
# operations on manually-created tables.
```

**Query timeout on large datasets:**
```python
# Python: increase timeout (default 60s)
results = client.query("SELECT COUNT(*) FROM big_table", timeout=300)
```
```typescript
// Node.js: increase timeout (default 60s)
const rows = await client.query("SELECT COUNT(*) FROM big", undefined, { timeoutMs: 300_000 });
```

**Note:** All database operations (query, list_tables, drop_table, create_table) go through the REST API. No direct PostgreSQL connection is needed from the Python or Node.js client.


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)