npm package: @partnersync/conduit
Description: Conduit - Enterprise ETL Framework for Node.js
Author: Mark Wharton
Latest version: 0.5.3

# @partnersync/conduit

[![npm version](https://img.shields.io/npm/v/@partnersync/conduit.svg)](https://www.npmjs.com/package/@partnersync/conduit)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![Node.js Version](https://img.shields.io/node/v/@partnersync/conduit.svg)](https://nodejs.org)

A modern, type-safe ETL framework for building data integration pipelines with TypeScript.

## Overview

Conduit provides a declarative, configuration-driven approach to data integration with:

- **Type-safe configurations** - Full TypeScript support with compile-time validation
- **Modular architecture** - Pluggable extractors, transformers, and loaders
- **Built-in resilience** - Automatic retries, error handling, and transaction support
- **Multiple data sources** - APIs, databases, files with authentication handling
- **Production-ready** - Event-driven monitoring, atomic commits, and data integrity tracking

## Installation

```bash
npm install @partnersync/conduit
```

## Quick Start

1. **Initialize the database:**
   ```bash
   npx conduit init
   ```

2. **Create an integration:**
   ```typescript
   // integrations/my-api/config.ts
   import { IntegrationConfig } from '@partnersync/conduit/core';

   export default {
     id: 'my-api',
     name: 'My API Integration',
     entities: [{
       id: 'users',
       name: 'Users',
       source: {
         type: 'api',
         url: 'https://api.example.com/users'
       },
       destination: {
         type: 'database',
         tableName: 'Users'
       },
       fields: [
         { source: 'id', target: 'Id', type: 'integer', isPrimaryKey: true },
         { source: 'name', target: 'Name', type: 'string' }
       ]
     }]
   } as IntegrationConfig;
   ```

3. **Deploy and run:**
   ```bash
   npx conduit deploy my-api
   npx conduit run my-api
   ```

## Documentation

- **[Getting Started Guide](./docs/getting-started.md)** - Detailed setup instructions
- **[Configuration Reference](./docs/configuration-reference.md)** - Complete guide to all configuration options
- **[Architecture Documentation](./docs/architecture/)** - Deep dive into framework design
- **[API Reference](./docs/api/)** - Generated TypeDoc documentation

## Environment Variables

The Conduit CLI uses system environment variables exclusively and does NOT load .env files. This follows the pattern of professional CLI tools like kubectl and aws-cli.

### Setting Environment Variables

**Linux/Mac:**
```bash
# Required environment variables
export DATABASE_CONNECTION_STRING="Server=localhost;Database=conduit;User Id=sa;Password=YourPassword;TrustServerCertificate=true"
export STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=youraccount;AccountKey=..."

# Integration-specific variables (examples based on your authentication type)
# For an integration with id: "salesforce-api" using OAuth2:
export SALESFORCE_API_CLIENT_SECRET="your-client-secret"
export SALESFORCE_API_REFRESH_TOKEN="your-refresh-token"

# For an integration with id: "stripe" using API Key:
export STRIPE_API_KEY="your-api-key"
```

**Windows Command Prompt:**
```cmd
rem Required environment variables
set DATABASE_CONNECTION_STRING=Server=localhost;Database=conduit;User Id=sa;Password=YourPassword;TrustServerCertificate=true
set STORAGE_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=youraccount;AccountKey=...

rem Integration-specific variables (examples based on your authentication type)
rem For an integration with id: "salesforce-api" using OAuth2:
set SALESFORCE_API_CLIENT_SECRET=your-client-secret
set SALESFORCE_API_REFRESH_TOKEN=your-refresh-token

rem For an integration with id: "stripe" using API Key:
set STRIPE_API_KEY=your-api-key
```

**Windows PowerShell:**
```powershell
# Required environment variables
$env:DATABASE_CONNECTION_STRING="Server=localhost;Database=conduit;User Id=sa;Password=YourPassword;TrustServerCertificate=true"
$env:STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=youraccount;AccountKey=..."

# Integration-specific variables (examples based on your authentication type)
# For an integration with id: "salesforce-api" using OAuth2:
$env:SALESFORCE_API_CLIENT_SECRET="your-client-secret"
$env:SALESFORCE_API_REFRESH_TOKEN="your-refresh-token"

# For an integration with id: "stripe" using API Key:
$env:STRIPE_API_KEY="your-api-key"
```

### For Library Usage Only

If you're using Conduit as a library (not the CLI), you can create a `.env` file in your project root. See `.env.template` for an example. The .env file is ONLY loaded when using Conduit programmatically, never by the CLI.

### Required Variables by Authentication Type

Based on your integration's authentication configuration:

**OAuth2:**
- `{INTEGRATION_ID}_CLIENT_SECRET`
- `{INTEGRATION_ID}_REFRESH_TOKEN`

**API Key:**
- `{INTEGRATION_ID}_API_KEY`

**Bearer Token:**
- `{INTEGRATION_ID}_BEARER_TOKEN`

**Form Login:**
- `{INTEGRATION_ID}_USERNAME`
- `{INTEGRATION_ID}_PASSWORD`

Note: Replace hyphens with underscores and use uppercase. For example, `api-service` becomes `API_SERVICE`.

## Troubleshooting

### Environment Variable Issues

If you see errors about missing environment variables:

1. **Check the error message** - The CLI provides platform-specific instructions
2. **Verify the variable is set** - Use `echo $VAR_NAME` (Linux/Mac) or `echo %VAR_NAME%` (Windows)
3. **Remember**: The CLI does NOT read .env files - you must set system environment variables
4. **For persistent variables**:
   - Linux/Mac: Add to `~/.bashrc`, `~/.zshrc`, or `~/.profile`
   - Windows: Use System Properties → Environment Variables

## CLI Commands

- `npx conduit init` - Initialize the database with operational schema
- `npx conduit deploy <integration>` - Deploy integration tables and procedures
- `npx conduit run <integration>` - Execute an integration
- `npx conduit health` - Check database health and configuration
- `npx conduit list` - List available integrations

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

## License

MIT - see [LICENSE](LICENSE) for details.