npm package: @kyuda/n8n-nodes-databricks-simplified
Description: Comprehensive Databricks integration for n8n — SQL, Unity Catalog, Jobs, Clusters, Files, Genie, Vector Search, and Model Serving
Author: Kyuda
Latest version: 1.0.0

# @kyuda/n8n-nodes-databricks

Comprehensive Databricks integration for n8n. Provides nodes for SQL, Unity Catalog, Jobs, Clusters, Files, Genie AI, Vector Search, Model Serving, and LangChain AI Agent / Chat / Embedding / Vector Store nodes.

## Features

- **Genie AI Assistant** -- Start conversations, send messages, and execute SQL queries through Databricks' AI assistant
- **Databricks SQL** -- Execute SQL queries with automatic chunked result retrieval
- **Unity Catalog** -- Manage catalogs, schemas, tables, volumes, and functions
- **Model Serving** -- Query deployed ML/LLM endpoints with auto-detected input format
- **Files** -- Upload, download, list, and manage files in Unity Catalog volumes (up to 5 GiB)
- **Vector Search** -- Semantic similarity search with reranking support
- **Jobs** -- Run, create, list, cancel, and monitor Databricks job runs
- **Clusters** -- List, create, start, stop, and delete Databricks compute clusters
- **LmChatDatabricks** -- LangChain Chat Model node for Databricks Foundation Model endpoints
- **EmbeddingsDatabricks** -- LangChain Embeddings node for Databricks embedding endpoints
- **VectorStoreDatabricks** -- LangChain Vector Store node backed by Databricks Vector Search
- **DatabricksAiAgent** -- Full AI Agent node with optional MLflow tracing

## Prerequisites

- Node.js 18+
- pnpm
- A Databricks workspace with credentials (PAT or OAuth Service Principal)

## Installation

### For Development

```bash
git clone https://github.com/kyudahq/n8n.git
cd n8n/@kyuda/n8n-nodes-databricks
pnpm install
pnpm build
```

Link to your n8n installation:

```bash
npm link
cd ~/.n8n/custom
npm link @kyuda/n8n-nodes-databricks
```

### From npm

```bash
npm install @kyuda/n8n-nodes-databricks
```

## Credentials

Two authentication methods are supported:

### Personal Access Token (default)

1. **Host** -- Your Databricks workspace URL (e.g. `https://adb-1234567890123456.7.azuredatabricks.net`)
2. **Token** -- A Databricks personal access token (generate at User Settings > Access Tokens)

### OAuth M2M (Service Principal)

1. **Host** -- Your Databricks workspace URL
2. **Client ID** -- The application (client) ID of a Databricks service principal
3. **Client Secret** -- The secret associated with the service principal

The OAuth flow uses `grant_type=client_credentials` against your workspace's `/oidc/v1/token` endpoint. Tokens are cached and refreshed automatically.

## Resources and Operations

### Databricks (main node)

| Resource | Operations |
|---|---|
| **Genie** | Start Conversation, Create Message, Get Message, Execute Message Query, Get Query Results, Get Space |
| **Databricks SQL** | Execute Query |
| **Unity Catalog** | Create/Get/Update/Delete Catalog, List Catalogs, Create/Get/Update/Delete Schema, List Schemas, Create/Get/Delete Volume, List Volumes, Get/List Tables, Create/Get/Delete Function, List Functions |
| **Model Serving** | Query Endpoint (auto-detects chat/completions/embeddings/dataframe formats) |
| **Files** | Upload File, Download File, Delete File, Get File Info, List Directory, Create Directory, Delete Directory |
| **Vector Search** | Query Index, Get Index, List Indexes |
| **Jobs** | Run Job, Get Run Status, Get Run Output, Create Job, List Jobs, Cancel Run, List Runs |
| **Clusters** | List Clusters, Get Cluster, Create Cluster, Start Cluster, Stop Cluster, Delete Cluster |

### LangChain Nodes

| Node | Description |
|---|---|
| **LmChatDatabricks** | Chat model for Databricks Foundation Model APIs or external model endpoints |
| **EmbeddingsDatabricks** | Text embeddings via Databricks serving endpoints |
| **VectorStoreDatabricks** | Vector store backed by Databricks Vector Search indexes |
| **DatabricksAiAgent** | AI agent with tool calling, memory, structured output, streaming, and optional MLflow tracing |

## Architecture

```
@kyuda/n8n-nodes-databricks/
├── credentials/
│   └── Databricks.credentials.ts      # PAT + OAuth M2M
├── nodes/
│   ├── Databricks/
│   │   ├── Databricks.node.ts          # Main node (slim dispatcher)
│   │   ├── helpers.ts                  # Shared HTTP helper, cache, security utils
│   │   └── resources/
│   │       ├── index.ts                # Barrel exports
│   │       ├── files/                  # operations, parameters, handler
│   │       ├── genie/                  # operations, parameters, handler
│   │       ├── databricksSql/          # operations, parameters, handler
│   │       ├── modelServing/           # operations, parameters, handler
│   │       ├── unityCatalog/           # operations, parameters, handler
│   │       ├── vectorSearch/           # operations, parameters, handler
│   │       ├── jobs/                   # operations, parameters, handler
│   │       └── clusters/              # operations, parameters, handler
│   ├── agents/DatabricksAiAgent/       # AI Agent with MLflow
│   ├── llms/LmChatDatabricks/          # LangChain Chat Model
│   ├── embeddings/EmbeddingsDatabricks/# LangChain Embeddings
│   └── vector_store/VectorStoreDatabricks/ # LangChain Vector Store
└── utils/
    ├── DatabricksVectorStoreLangChain.ts
    ├── N8nTool.ts
    ├── N8nBinaryLoader.ts
    └── descriptions.ts
```

### Execution Flow

```
User selects Resource → Operation → fills Parameters
        ↓
    execute() dispatcher (Databricks.node.ts)
        ↓
    handlers[resource](ctx, operation, itemIndex)
        ↓
    Per-resource handler file (e.g. resources/jobs/handler.ts)
        ↓
    databricksApiRequest() (shared helper with auth, host normalization)
        ↓
    Databricks REST API
```

### Security

- **Token masking** -- Bearer tokens and PAT values are scrubbed from all error messages and logs
- **URL encoding** -- All user-provided path segments are passed through `encodeURIComponent`
- **Path traversal protection** -- File operations reject paths containing `..`
- **Cache isolation** -- Cache keys include a hash of the credential token, preventing cross-user cache leaks

## Databricks AI Agent

The **Databricks AI Agent** node provides a LangChain-based agent with:

- **Tool Calling** -- Supports any LangChain tool or MCP toolkit
- **Memory** -- Conversation history via BaseChatMemory
- **Structured Output** -- Optional output parser for validated JSON
- **Streaming** -- Real-time token streaming support
- **Fallback Models** -- Automatic failover to a secondary model
- **MLflow Tracing** -- Optional observability with automatic experiment management

When MLflow is enabled, traces appear under `/Shared/n8n-workflows-{workflow-id}` with spans for agent execution, LLM calls, and tool invocations.

## Development

```bash
pnpm build      # Build the node
pnpm test       # Run tests
pnpm lint       # Lint
pnpm lintfix    # Auto-fix lint issues
```

## Contributing

Contributions are welcome. Please submit a Pull Request at [github.com/kyudahq/n8n](https://github.com/kyudahq/n8n).

## Resources

- [n8n Documentation](https://docs.n8n.io/)
- [Databricks API Documentation](https://docs.databricks.com/api/)
- [n8n Community](https://community.n8n.io/)

## License

[MIT](LICENSE.md)