npm package: @seedts/import
Description: Import/export functionality for SeedTS - Load seed data from external files (JSON, CSV, YAML)
Author: SeedTS Contributors
Latest version: 0.1.1

# @seedts/import

Import/export functionality for SeedTS - Load seed data from external files (JSON, CSV, YAML) and export seeded data.

## Installation

```bash
npm install @seedts/import
# or
pnpm add @seedts/import
# or
yarn add @seedts/import
```

## Features

- 📥 **Import** from JSON, CSV, and YAML files
- 📤 **Export** to JSON, CSV, and YAML formats
- ✅ **Schema Validation** using Zod
- 🔄 **Data Transformation** with custom transform functions
- 🔀 **Merge Strategies** (replace, append, update, upsert)
- 🎯 **Type-Safe** with full TypeScript support
- ⚡ **Async/Await** based API

## Quick Start

### Import from JSON

```typescript
import { importJSON, loadJSON } from '@seedts/import';

// Import with validation and transformation
const result = await importJSON({
  path: './data/users.json',
  schema: userSchema,
  transform: (record) => ({
    ...record,
    createdAt: new Date()
  })
});

console.log(`Imported ${result.count} records`);
```

### Use in Seed Definitions

```tsx
import { loadJSON } from '@seedts/import';

export const CategoriesSeed = ({ adapter }) => (
  <Seed name="categories" adapter={adapter}>
    <Action>
      <Entity data={loadJSON('./data/categories.json')} />
    </Action>
  </Seed>
);
```

### Import from CSV

```typescript
import { importCSV, loadCSV } from '@seedts/import';

// Import CSV with headers
const result = await importCSV({
  path: './data/products.csv',
  delimiter: ',',
  header: true,
  dynamicTyping: true
});

// Use in seed definition
<Entity data={loadCSV('./data/products.csv', { dynamicTyping: true })} />
```

### Import from YAML

```typescript
import { importYAML, loadYAML } from '@seedts/import';

const result = await importYAML({
  path: './data/config.yaml',
  schema: configSchema
});

// Use in seed definition
<Entity data={loadYAML('./data/config.yaml')} />
```

## Import API

### importJSON(options)

Import data from a JSON file.

**Options:**
- `path` (string, required) - Path to the JSON file
- `schema` (ZodSchema, optional) - Zod schema for validation
- `transform` (function, optional) - Transform function `(record) => transformedRecord`
- `skipInvalid` (boolean, optional) - Skip invalid records instead of throwing (default: false)
- `encoding` (string, optional) - File encoding (default: 'utf-8')

**Returns:** `Promise<ImportResult<T>>`

```typescript
interface ImportResult<T> {
  data: T[];              // Successfully imported records
  count: number;          // Number of records imported
  invalidCount: number;   // Number of records that failed validation
  errors: Array<{         // Validation errors
    index: number;
    record: any;
    error: string;
  }>;
}
```

**Example:**

```typescript
import { importJSON } from '@seedts/import';
import { z } from 'zod';

const userSchema = z.object({
  name: z.string(),
  email: z.string().email(),
  age: z.number().min(18)
});

const result = await importJSON({
  path: './users.json',
  schema: userSchema,
  transform: (record) => ({
    ...record,
    createdAt: new Date(),
    updatedAt: new Date()
  }),
  skipInvalid: false
});

console.log(`Imported: ${result.count}`);
console.log(`Invalid: ${result.invalidCount}`);
```

### importCSV(options)

Import data from a CSV file.

**Options:** All from `importJSON` plus:
- `delimiter` (string, optional) - Delimiter character (default: ',')
- `header` (boolean, optional) - First row contains headers (default: true)
- `quoteChar` (string, optional) - Quote character (default: '"')
- `skipEmptyLines` (boolean, optional) - Skip empty lines (default: true)
- `dynamicTyping` (boolean, optional) - Auto-detect types (default: false)

**Example:**

```typescript
import { importCSV } from '@seedts/import';

const result = await importCSV({
  path: './products.csv',
  delimiter: ',',
  header: true,
  dynamicTyping: true,
  transform: (record) => ({
    ...record,
    price: parseFloat(record.price),
    inStock: record.inStock === 'true'
  })
});
```

### importYAML(options)

Import data from a YAML file. Same options as `importJSON`.

**Example:**

```typescript
import { importYAML } from '@seedts/import';

const result = await importYAML({
  path: './config.yaml',
  schema: configSchema
});
```

### Helper Functions: loadJSON, loadCSV, loadYAML

These functions return a function that can be used directly in seed definitions.

```typescript
loadJSON<T>(path: string): () => Promise<T[]>
loadCSV<T>(path: string, options?: CSVImportOptions): () => Promise<T[]>
loadYAML<T>(path: string): () => Promise<T[]>
```

**Example:**

```tsx
import { loadJSON, loadCSV, loadYAML } from '@seedts/import';

<Entity data={loadJSON('./data/categories.json')} />
<Entity data={loadCSV('./data/products.csv', { dynamicTyping: true })} />
<Entity data={loadYAML('./data/config.yaml')} />
```

## Export API

### exportJSON(data, options)

Export data to a JSON file.

**Options:**
- `path` (string, required) - Output file path
- `pretty` (boolean, optional) - Pretty print JSON (default: true)
- `transform` (function, optional) - Transform function `(records) => transformedRecords`
- `encoding` (string, optional) - File encoding (default: 'utf-8')

**Returns:** `Promise<ExportResult>`

```typescript
interface ExportResult {
  path: string;   // Path where file was saved
  count: number;  // Number of records exported
  size: number;   // File size in bytes
}
```

**Example:**

```typescript
import { exportJSON } from '@seedts/import';

const result = await exportJSON(userData, {
  path: './output/users.json',
  pretty: true,
  transform: (records) => records.map(r => ({
    ...r,
    password: undefined // Remove sensitive fields
  }))
});

console.log(`Exported ${result.count} records (${result.size} bytes)`);
```

### exportCSV(data, options)

Export data to a CSV file.

**Options:** All from `exportJSON` plus:
- `delimiter` (string, optional) - Delimiter character (default: ',')
- `header` (boolean, optional) - Include headers (default: true)
- `quoteChar` (string, optional) - Quote character (default: '"')
- `columns` (string[], optional) - Columns to include (all if not specified)

**Example:**

```typescript
import { exportCSV } from '@seedts/import';

await exportCSV(userData, {
  path: './output/users.csv',
  delimiter: ',',
  header: true,
  columns: ['id', 'name', 'email', 'createdAt']
});
```

### exportYAML(data, options)

Export data to a YAML file. Same options as `exportJSON` (except `pretty`).

**Example:**

```typescript
import { exportYAML } from '@seedts/import';

await exportYAML(configData, {
  path: './output/config.yaml'
});
```

### exportData(data, options)

Generic export function that routes to the appropriate exporter based on format.

**Example:**

```typescript
import { exportData } from '@seedts/import';

await exportData(userData, {
  path: './output/users.json',
  format: 'json',
  pretty: true
});

await exportData(userData, {
  path: './output/users.csv',
  format: 'csv',
  header: true
});
```

## Merge Strategies

Combine imported data with existing data using different strategies.

### mergeData(existingData, newData, options)

**Strategies:**
- `replace` - Replace all existing data with new data
- `append` - Add new data to existing data
- `update` - Update existing records that match, ignore new records
- `upsert` - Update existing records or insert new ones

**Options:**
- `strategy` (required) - Merge strategy to use
- `keyField` (string | string[]) - Key field(s) for matching (required for update/upsert)
- `matchFn` (function) - Custom match function `(existing, new) => boolean`

**Examples:**

```typescript
import { mergeData } from '@seedts/import';

// Replace all data
const merged = mergeData(existingUsers, newUsers, {
  strategy: 'replace'
});

// Append new data
const merged = mergeData(existingUsers, newUsers, {
  strategy: 'append'
});

// Update by ID
const merged = mergeData(existingUsers, newUsers, {
  strategy: 'update',
  keyField: 'id'
});

// Upsert by email
const merged = mergeData(existingUsers, newUsers, {
  strategy: 'upsert',
  keyField: 'email'
});

// Custom match function
const merged = mergeData(existingUsers, newUsers, {
  strategy: 'upsert',
  matchFn: (existing, newRecord) => existing.email === newRecord.email
});
```

### createMerger(options)

Create a reusable merge function for use in seed hooks.

**Example:**

```tsx
import { createMerger, loadJSON } from '@seedts/import';

const upsertBySlug = createMerger({
  strategy: 'upsert',
  keyField: 'slug'
});

<Seed
  name="categories"
  adapter={adapter}
  beforeInsert={async (newData, ctx) => {
    const existing = await ctx.getSeed('categories') || [];
    return upsertBySlug(newData, existing);
  }}
>
  <Action>
    <Entity data={loadJSON('./data/categories.json')} />
  </Action>
</Seed>
```

## Schema Validation

Use Zod schemas to validate imported data.

```typescript
import { importJSON } from '@seedts/import';
import { z } from 'zod';

const productSchema = z.object({
  name: z.string().min(1),
  price: z.number().positive(),
  sku: z.string().regex(/^[A-Z0-9-]+$/),
  inStock: z.boolean(),
  category: z.enum(['electronics', 'clothing', 'books']),
  tags: z.array(z.string()).optional()
});

const result = await importJSON({
  path: './products.json',
  schema: productSchema,
  skipInvalid: true // Skip invalid records
});

console.log(`Valid: ${result.count}, Invalid: ${result.invalidCount}`);
result.errors.forEach(err => {
  console.log(`Record ${err.index}: ${err.error}`);
});
```

## Complete Example

### Data Files

**users.json:**
```json
[
  {
    "name": "John Doe",
    "email": "john@example.com",
    "age": 30
  },
  {
    "name": "Jane Smith",
    "email": "jane@example.com",
    "age": 25
  }
]
```

**products.csv:**
```csv
name,price,sku,inStock
"Laptop",999.99,LAP-001,true
"Mouse",29.99,MOU-002,true
"Keyboard",79.99,KEY-003,false
```

### Seed Definition

```tsx
import { Seed, Action, Entity, Attribute } from '@seedts/jsx-runtime';
import { loadJSON, loadCSV } from '@seedts/import';
import { z } from 'zod';

const userSchema = z.object({
  name: z.string(),
  email: z.string().email(),
  age: z.number().min(18)
});

export const UsersSeed = ({ adapter }) => (
  <Seed name="users" adapter={adapter}>
    <Action>
      <Entity data={loadJSON('./data/users.json')} />
    </Action>
  </Seed>
);

export const ProductsSeed = ({ adapter }) => (
  <Seed name="products" adapter={adapter}>
    <Action>
      <Entity data={loadCSV('./data/products.csv', {
        dynamicTyping: true
      })} />
    </Action>
  </Seed>
);
```

### Export After Seeding

```typescript
import { Executor } from '@seedts/core';
import { exportJSON, exportCSV } from '@seedts/import';

const executor = new Executor([UsersSeed, ProductsSeed]);
const results = await executor.execute();

// Export users to JSON
const usersData = results.find(r => r.name === 'users')?.data || [];
await exportJSON(usersData, {
  path: './output/users-seeded.json',
  pretty: true
});

// Export products to CSV
const productsData = results.find(r => r.name === 'products')?.data || [];
await exportCSV(productsData, {
  path: './output/products-seeded.csv',
  columns: ['id', 'name', 'price', 'sku']
});
```

## Advanced Usage

### Conditional Import with Merge

```tsx
import { createMerger, loadJSON } from '@seedts/import';

const merger = createMerger({
  strategy: 'upsert',
  keyField: 'slug'
});

<Seed
  name="categories"
  adapter={adapter}
  condition={async (ctx) => {
    const existing = await ctx.getSeed('categories');
    return !existing || existing.length === 0;
  }}
  beforeInsert={async (newData, ctx) => {
    const existing = await adapter.query('categories') || [];
    return merger(newData, existing);
  }}
>
  <Action>
    <Entity data={loadJSON('./data/categories.json')} />
  </Action>
</Seed>
```

### Transform During Import

```typescript
import { importJSON } from '@seedts/import';

const result = await importJSON({
  path: './users.json',
  transform: async (record) => {
    // Add timestamps
    const transformed = {
      ...record,
      createdAt: new Date(),
      updatedAt: new Date()
    };

    // Hash password if present
    if (record.password) {
      transformed.password = await bcrypt.hash(record.password, 10);
    }

    // Normalize email
    transformed.email = record.email.toLowerCase().trim();

    return transformed;
  }
});
```

### Batch Import Multiple Files

```typescript
import { importJSON, mergeData } from '@seedts/import';

const files = [
  './data/users-batch-1.json',
  './data/users-batch-2.json',
  './data/users-batch-3.json'
];

let allUsers = [];

for (const file of files) {
  const result = await importJSON({ path: file });
  allUsers = mergeData(allUsers, result.data, {
    strategy: 'upsert',
    keyField: 'email'
  });
}

console.log(`Total unique users: ${allUsers.length}`);
```

## TypeScript Support

All functions are fully typed:

```typescript
import { importJSON, ImportResult } from '@seedts/import';
import { z } from 'zod';

interface User {
  name: string;
  email: string;
  age: number;
}

const userSchema = z.object({
  name: z.string(),
  email: z.string().email(),
  age: z.number()
});

// Fully typed result
const result: ImportResult<User> = await importJSON<User>({
  path: './users.json',
  schema: userSchema
});

// result.data is User[]
result.data.forEach((user: User) => {
  console.log(user.name, user.email);
});
```

## Error Handling

```typescript
import { importJSON } from '@seedts/import';

try {
  const result = await importJSON({
    path: './data.json',
    schema: mySchema,
    skipInvalid: true
  });

  if (result.invalidCount > 0) {
    console.warn(`${result.invalidCount} invalid records skipped`);
    result.errors.forEach(err => {
      console.error(`Record ${err.index}:`, err.error);
    });
  }

  console.log(`Successfully imported ${result.count} records`);
} catch (error) {
  console.error('Import failed:', error);
}
```

## License

MIT