# 01: Creating Your First Listener
Source: https://flatfile.com/docs/coding-tutorial/101-your-first-listener/101.01-first-listener

Learn to set up a basic Listener with Space configuration to define your data structure and workspace layout.

<Note>
  If you aren't interested in a code-forward approach, we recommend starting with [AutoBuild](/getting-started/quickstart/autobuild.mdx), which uses AI to analyze your template or documentation and then automatically creates and deploys a [Blueprint](/core-concepts/blueprints) (for schema definition) and a [Listener](/core-concepts/listeners) (for validations and transformations) to your [Flatfile App](/core-concepts/apps).

  Once you've started with AutoBuild, you can always download your Listener code and continue building with code from there!
</Note>

## What We're Building

In this tutorial, we'll build a foundational Listener that handles Space configuration - the essential first step for any Flatfile implementation. Our Listener will:

* **Respond to Space creation**: When a user creates a new [Space](/core-concepts/spaces), our Listener will automatically configure it
* **Define the Blueprint**: Set up a single [Workbook](/core-concepts/workbooks) with a single [Sheet](/core-concepts/sheets) with [Field](/core-concepts/fields) definitions for names and emails that establishes the data schema for the Space
* **Handle the complete Job lifecycle**: Acknowledge, update progress, and complete the configuration [Job](/core-concepts/jobs) with proper error handling
* **Provide user feedback**: Give real-time updates during the configuration process

This forms the foundation that you'll build upon in the next parts of this series, where we'll add user Actions and data validation. By the end of this tutorial, you'll have a working Listener that creates a fully configured workspace ready for data import.

## Prerequisites

Before we start coding, you'll need a Flatfile account and a fresh project directory:

1. **Create a new project directory**: Start in a fresh directory for this tutorial (e.g., `mkdir my-flatfile-listener && cd my-flatfile-listener`)
2. **Sign up for Flatfile**: Visit [platform.flatfile.com](https://platform.flatfile.com) and create your free account
3. **Get your credentials**: You'll need your Secret Key and Environment ID from the [Keys & Secrets](https://platform.flatfile.com/dashboard/keys-and-secrets) section later in this tutorial

<Note>
  **New to Flatfile?** If you'd like to understand the broader data structure and concepts before diving into code, we recommend reading through the [Core Concepts](/core-concepts/overview) section first. This covers the foundational elements like [Environments](/core-concepts/environments), [Apps](/core-concepts/apps), and [Spaces](/core-concepts/spaces), as well as our data structure like [Workbooks](/core-concepts/workbooks) and [Sheets](/core-concepts/sheets), and how they all work together.

  Each Listener is deployed to a specific Environment, allowing you to set up separate Environments for development, staging, and production to safely test code changes before deploying to production.
</Note>

## Install Dependencies

Choose your preferred language and follow the setup steps:

<CodeGroup>
  ```bash JavaScript theme={null}
  # Initialize project (skip if you already have package.json)
  npm init -y

  # Install required Flatfile packages
  npm install @flatfile/listener @flatfile/api

  # Note: Feel free to use your preferred JavaScript project setup method instead
  ```

  ```bash TypeScript theme={null}
  # Initialize project (skip if you already have package.json)
  npm init -y

  # Install required Flatfile packages
  npm install @flatfile/listener @flatfile/api

  # Install TypeScript dev dependency
  npm install --save-dev typescript

  # Initialize TypeScript config (skip if you already have tsconfig.json)
  npx tsc --init

  # Note: Feel free to use your preferred TypeScript project setup method instead
  ```
</CodeGroup>

### Authentication Setup

For this step, you'll need to get your Secret Key and environment ID from your [Flatfile Dashboard](https://platform.flatfile.com/dashboard/keys-and-secrets).

Then create a new file called `.env` and add the following (populated with your own values):

```bash theme={null}
# .env
FLATFILE_API_KEY="your_secret_key"
FLATFILE_ENVIRONMENT_ID="us_env_your_environment_id"
```

## Create Your Listener File

Create a new file called `index.js` for Javascript or `index.ts` for TypeScript:

<CodeGroup>
  ```javascript JavaScript theme={null}
  import api from "@flatfile/api";

  export default function (listener) {
    // Configure the Space when it's created
    listener.on("job:ready", { job: "space:configure" }, async (event) => {
      const { jobId, spaceId } = event.context;
      try {
        // Acknowledge the job
        await api.jobs.ack(jobId, {
          info: "Setting up your workspace...",
          progress: 10,
        });
        // Create the Workbook with Sheets, creating the Blueprint for the space
        await api.workbooks.create({
          spaceId,
          name: "My Workbook",
          sheets: [
            {
              name: "contacts",
              slug: "contacts",
              fields: [
                { key: "name", type: "string", label: "Full Name" },
                { key: "email", type: "string", label: "Email" },
              ],
            },
          ],
        });
        // Update progress
        await api.jobs.update(jobId, {
          info: "Workbook created successfully",
          progress: 75,
        });
        // Complete the job
        await api.jobs.complete(jobId, {
          outcome: {
            message: "Workspace configured successfully!",
            acknowledge: true,
          },
        });
      } catch (error) {
        console.error("Error configuring Space:", error);
        // Fail the job if something goes wrong
        await api.jobs.fail(jobId, {
          outcome: {
            message: `Failed to configure workspace: ${error.message}`,
            acknowledge: true,
          },
        });
      }
    });
  }

  ```

  ```typescript TypeScript theme={null}
  import type { FlatfileListener } from "@flatfile/listener";
  import api from "@flatfile/api";

  export default function (listener: FlatfileListener) {
    // Configure the Space when it's created
    listener.on("job:ready", { job: "space:configure" }, async (event) => {
      const { jobId, spaceId } = event.context;
      
      try {
        // Acknowledge the job
        await api.jobs.ack(jobId, {
          info: "Setting up your workspace...",
          progress: 10
        });

        // Create the Workbook with Sheets, creating the Blueprint for the space
        await api.workbooks.create({
          spaceId,
          name: "My Workbook",
          sheets: [
            {
              name: "contacts",
              slug: "contacts",
              fields: [
                { key: "name", type: "string", label: "Full Name" },
                { key: "email", type: "string", label: "Email" },
              ],
            },
          ],
        });

        // Update progress
        await api.jobs.update(jobId, {
          info: "Workbook created successfully",
          progress: 75
        });

        // Complete the job
        await api.jobs.complete(jobId, {
          outcome: {
            message: "Workspace configured successfully!",
            acknowledge: true
          }
        });

      } catch (error) {
        console.error("Error configuring Space:", error);
        
        // Fail the job if something goes wrong
        await api.jobs.fail(jobId, {
          outcome: {
            message: `Failed to configure workspace: ${error instanceof Error ? error.message : 'Unknown error'}`,
            acknowledge: true
          }
        });
      }
    });
  }
  ```
</CodeGroup>

<Note>
  **Complete Example**: The full working code for this tutorial step is available in our Getting Started repository: [JavaScript](https://github.com/FlatFilers/getting-started/tree/main/101.01-first-listener/javascript) | [TypeScript](https://github.com/FlatFilers/getting-started/tree/main/101.01-first-listener/typescript)
</Note>

## Project Structure

After creating your Listener file, your project directory should look like this:

<CodeGroup>
  ```text JavaScript theme={null}
  my-flatfile-listener/
  ├── .env      // Environment variables
  ├── index.js  // Listener code
  |
  |   /* Node-specific files below */
  |
  ├── package.json
  ├── package-lock.json
  └── node_modules/
  ```

  ```text TypeScript theme={null}
  my-flatfile-listener/
  ├── .env      // Environment variables
  ├── index.ts  // Listener code
  |
  |   /* Node and Typescript-specific files below */
  |
  ├── package.json
  ├── package-lock.json
  ├── tsconfig.json
  └── node_modules/
  ```
</CodeGroup>

### Authentication Setup

You'll need to get your Secret Key and Environment ID from your [Flatfile Dashboard](https://platform.flatfile.com/dashboard/keys-and-secrets) to find both values, then add them to a `.env` file:

```bash theme={null}
# .env
FLATFILE_API_KEY="your_secret_key"
FLATFILE_ENVIRONMENT_ID="us_env_your_environment_id"
```

## Testing Your Listener

### Local Development

To test your Listener locally, you can use the `flatfile develop` command. This will start a local server that implements your custom Listener code, and will also watch for changes to your code and automatically reload the server.

```bash theme={null}
# Run locally with file watching
npx flatfile develop
```

### Step-by-Step Testing

After running your listener locally:

<Card title="Testing Steps">
  1. Create a new space in your Flatfile environment
  2. Observe as the new space is configured with a Workbook and Sheet
</Card>

## What Just Happened?

Your Listener is now ready to respond to Space configuration Events! Here's how the space configuration works step by step:

### 1. Exporting your Listener function

This is the base structure of your Listener. At its core, it's just a function that takes a `listener` object as an argument, and then uses that listener to respond to Events.

<CodeGroup>
  ```javascript JavaScript theme={null}
  export default function (listener) {
    // . . . code
  }
  ```

  ```typescript TypeScript theme={null}
  export default function (listener: FlatfileListener) {
    // . . . code
  }
  ```
</CodeGroup>

### 2. Listen for Space Configuration

When a new Space is created, Flatfile automatically triggers a `space:configure` job that your Listener can handle. This code listens for that job using the `job:ready` Event, filtered by the job name `space:configure`.

<CodeGroup>
  ```javascript JavaScript theme={null}
  listener.on("job:ready", { job: "space:configure" }, async (event) => {
    // . . . code
  });
  ```

  ```typescript TypeScript theme={null}
  listener.on("job:ready", { job: "space:configure" }, async (event) => {
    // . . . code
  });
  ```
</CodeGroup>

### 3. Acknowledge the Job

The first step is always to acknowledge that you've received the job and provide initial feedback to users. From this point on, we're responsible for the rest of the job lifecycle, and we'll be doing it all in this Listener. For more information on Jobs, see the [Jobs](/core-concepts/jobs) concept.

<CodeGroup>
  ```javascript JavaScript theme={null}
  await api.jobs.ack(jobId, {
    info: "Setting up your workspace...",
    progress: 10,
  });
  ```

  ```typescript TypeScript theme={null}
  await api.jobs.ack(jobId, {
    info: "Setting up your workspace...",
    progress: 10
  });
  ```
</CodeGroup>

### 4. Define the Blueprint

Next, we create the workbook with sheets and field definitions. This **is** your [Blueprint](/core-concepts/blueprints) definition—establishing the data schema that will govern all data within this Space.

<CodeGroup>
  ```javascript JavaScript theme={null}
  await api.workbooks.create({
    spaceId,
    name: "My Workbook",
    sheets: [
      {
        name: "contacts",
        slug: "contacts", 
        fields: [
          { key: "name", type: "string", label: "Full Name" },
          { key: "email", type: "string", label: "Email" },
        ],
      },
    ],
  });
  ```

  ```typescript TypeScript theme={null}
  await api.workbooks.create({
    spaceId,
    name: "My Workbook",
    sheets: [
      {
        name: "contacts",
        slug: "contacts",
        fields: [
          { key: "name", type: "string", label: "Full Name" },
          { key: "email", type: "string", label: "Email" },
        ],
      },
    ],
  });
  ```
</CodeGroup>

### 5. Update Progress

Keep users informed about what's happening during the configuration process.

<CodeGroup>
  ```javascript JavaScript theme={null}
  await api.jobs.update(jobId, {
    info: "Workbook created successfully",
    progress: 75,
  });
  ```

  ```typescript TypeScript theme={null}
  await api.jobs.update(jobId, {
    info: "Workbook created successfully", 
    progress: 75
  });
  ```
</CodeGroup>

### 6. Complete the Job

Finally, mark the job as complete with a success message, or fail it if something went wrong.

<CodeGroup>
  ```javascript JavaScript theme={null}
  // Success case
  await api.jobs.complete(jobId, {
    outcome: {
      message: "Workspace configured successfully!",
      acknowledge: true,
    },
  });

  // Failure case
  await api.jobs.fail(jobId, {
    outcome: {
      message: `Failed to configure workspace: ${error.message}`,
      acknowledge: true,
    },
  });
  ```

  ```typescript TypeScript theme={null}
  // Success case
  await api.jobs.complete(jobId, {
    outcome: {
      message: "Workspace configured successfully!",
      acknowledge: true
    }
  });

  // Failure case
  await api.jobs.fail(jobId, {
    outcome: {
      message: `Failed to configure workspace: ${error instanceof Error ? error.message : 'Unknown error'}`,
      acknowledge: true
    }
  });
  ```
</CodeGroup>

This follows the standard Job pattern: **acknowledge → update progress → complete** (or fail on error). This provides users with real-time feedback and ensures robust error handling throughout the configuration process.

## Next Steps

Ready to enhance data quality? Continue to [Adding Validation](/coding-tutorial/101-your-first-listener/101.02-adding-validation) to learn how to validate Fields and provide real-time feedback to users.

For more detailed information:

* Understand Job lifecycle patterns in [Jobs](/core-concepts/jobs) and [Spaces](/core-concepts/spaces)
* Learn more about [Events](/reference/events)
* Organize your Listeners with [Namespaces](/guides/namespaces-and-filters)


# 02: Adding Validation to Your Listener
Source: https://flatfile.com/docs/coding-tutorial/101-your-first-listener/101.02-adding-validation

Enhance your listener with data validation capabilities to ensure data quality and provide real-time feedback to users.

In the [previous guide](/coding-tutorial/101-your-first-listener/101.01-first-listener), we created a Listener that configures Spaces and sets up the data structure. Now we'll add data validation to ensure data quality and provide helpful feedback to users as they work with their data.

<Note>
  **Following along?** Download the starting code from our [Getting Started repository](https://github.com/FlatFilers/getting-started/tree/main/101.01-first-listener) and refactor it as we go, or jump directly to the [final version with validation](https://github.com/FlatFilers/getting-started/tree/main/101.02-adding-validation).
</Note>

## What Is Data Validation?

Data validation in Flatfile allows you to:

* Check data formats and business rules
* Provide warnings and errors to guide users
* Ensure data quality before processing
* Give real-time feedback during data entry

Validation can happen at different levels:

* **Field-level**: Validate individual [Field](/core-concepts/fields) values (email format, date ranges, etc.)
* **Record-level**: Validate relationships between [Fields](/core-concepts/fields) in a single [Record](/core-concepts/records)
* **Sheet-level**: Validate across all [Records](/core-concepts/records) (duplicates, unique constraints, etc.)

## Email Validation Example

This example shows how to perform email format validation directly when Records are committed. When users commit their changes, we validate that email addresses have a proper format and provide helpful feedback for any invalid emails.

<Note>
  This approach validates Records as they're committed, providing immediate feedback to users. For more complex validations or when you need an object-oriented approach, we recommend using the [Record Hook](/plugins/record-hook) plugin.
</Note>

<Warning>
  If you use both [Record Hooks](/plugins/record-hook) and regular listener
  validators (like this one) on the same sheet, you may encounter race
  conditions. Record Hooks will clear all existing messages before applying new
  ones, which can interfere with any messages set elsewhere. We have ways to
  work around this, but it's a good idea to avoid using both at the same time.
</Warning>

## What Changes We're Making

To add validation to our basic Listener, we'll add a listener that triggers when users commit their changes and performs validation directly:

```javascript theme={null}
listener.on("commit:created", async (event) => {
  const { sheetId } = event.context;
  
  // Get committed records and validate email format
  const response = await api.records.get(sheetId);
  const records = response.data.records;
  
  // Email validation logic here...
});
```

## Complete Example with Validation

Here's how to add email validation to your existing Listener:

<CodeGroup>
  ```javascript JavaScript theme={null}
  import api from "@flatfile/api";

  export default function (listener) {
    // Configure the space when it's created
    listener.on("job:ready", { job: "space:configure" }, async (event) => {
      const { jobId, spaceId } = event.context;
      try {
        // Acknowledge the job
        await api.jobs.ack(jobId, {
          info: "Setting up your workspace...",
          progress: 10,
        });
        // Create the workbook with sheets
        await api.workbooks.create({
          spaceId,
          name: "My Workbook",
          sheets: [
            {
              name: "contacts",
              slug: "contacts",
              fields: [
                { key: "name", type: "string", label: "Full Name" },
                { key: "email", type: "string", label: "Email" },
              ],
            },
          ],
        });
        // Update progress
        await api.jobs.update(jobId, {
          info: "Workbook created successfully",
          progress: 75,
        });
        // Complete the job
        await api.jobs.complete(jobId, {
          outcome: {
            message: "Workspace configured successfully!",
            acknowledge: true,
          },
        });
      } catch (error) {
        console.error("Error configuring space:", error);
        // Fail the job if something goes wrong
        await api.jobs.fail(jobId, {
          outcome: {
            message: `Failed to configure workspace: ${error.message}`,
            acknowledge: true,
          },
        });
      }
    });

    // Listen for commits and validate email format
    listener.on("commit:created", async (event) => {
      const { sheetId } = event.context;
      try {
        // Get records from the sheet
        const response = await api.records.get(sheetId);
        const records = response.data.records;
        // Simple email validation regex
        const emailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
        // Prepare updates for records with invalid emails
        const updates = [];
        for (const record of records) {
          const emailValue = record.values.email?.value;
          if (emailValue) {
            const email = emailValue.toLowerCase();
            if (!emailRegex.test(email)) {
              updates.push({
                id: record.id,
                values: {
                  email: {
                    value: email,
                    messages: [
                      {
                        type: "error",
                        message:
                          "Please enter a valid email address (e.g., user@example.com)",
                      },
                    ],
                  },
                },
              });
            }
          }
        }
        // Update records with validation messages
        if (updates.length > 0) {
          await api.records.update(sheetId, updates);
        }
      } catch (error) {
        console.error("Error during validation:", error);
      }
    });
  }
  ```

  ```typescript TypeScript theme={null}
  import type { FlatfileListener } from "@flatfile/listener";
  import api, { Flatfile } from "@flatfile/api";

  export default function (listener: FlatfileListener) {
    // Configure the space when it's created
    listener.on("job:ready", { job: "space:configure" }, async (event) => {
      const { jobId, spaceId } = event.context;
      
      try {
        // Acknowledge the job
        await api.jobs.ack(jobId, {
          info: "Setting up your workspace...",
          progress: 10
        });

        // Create the workbook with sheets
        await api.workbooks.create({
          spaceId,
          name: "My Workbook",
          sheets: [
            {
              name: "contacts",
              slug: "contacts",
              fields: [
                { key: "name", type: "string", label: "Full Name" },
                { key: "email", type: "string", label: "Email" },
              ],
            },
          ],
        });

        // Update progress
        await api.jobs.update(jobId, {
          info: "Workbook created successfully",
          progress: 75
        });

        // Complete the job
        await api.jobs.complete(jobId, {
          outcome: {
            message: "Workspace configured successfully!",
            acknowledge: true
          }
        });

      } catch (error) {
        console.error("Error configuring space:", error);
        
        // Fail the job if something goes wrong
        await api.jobs.fail(jobId, {
          outcome: {
            message: `Failed to configure workspace: ${error instanceof Error ? error.message : 'Unknown error'}`,
            acknowledge: true
          }
        });
      }
    });

    // Listen for commits and validate email format
    listener.on("commit:created", async (event) => {
      const { sheetId } = event.context;
      
      try {
        // Get records from the sheet
        const response = await api.records.get(sheetId);
        const records = response.data.records;

        // Simple email validation regex
        const emailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;

        // Prepare updates for records with invalid emails
        const updates: Flatfile.RecordWithLinks[] = [];

        for (const record of records) {
          const emailValue = record.values.email?.value as string; 
          
          if (emailValue) {
            const email = emailValue.toLowerCase();

            if (!emailRegex.test(email)) {
              updates.push({
                id: record.id,
                values: {
                  email: {
                    value: email,
                    messages: [{
                      type: "error",
                      message: "Please enter a valid email address (e.g., user@example.com)",
                    }],
                  },
                },
              });
            }
          }
        }

        // Update records with validation messages
        if (updates.length > 0) {
          await api.records.update(sheetId, updates);
        }

      } catch (error) {
        console.error("Error during validation:", error);
      }
    });
  }
  ```
</CodeGroup>

<Note>
  **Complete Example**: The full working code for this tutorial step is available in our Getting Started repository: [JavaScript](https://github.com/FlatFilers/getting-started/tree/main/101.02-adding-validation/javascript) | [TypeScript](https://github.com/FlatFilers/getting-started/tree/main/101.02-adding-validation/typescript)
</Note>

## Testing Your Validation

### Local Development

To test your Listener locally, you can use the `flatfile develop` command. This will start a local server that will listen for Events and respond to them, and will also watch for changes to your Listener code and automatically reload the server.

```bash theme={null}
# Run locally with file watching
npx flatfile develop
```

### Step-by-Step Testing

After running your listener locally:

<Card title="Testing Steps">
  1. Create a new space in your Flatfile environment
  2. Enter an invalid email address in the Email Field
  3. See error messages appear on invalid email Fields
  4. Fix the emails and see the error messages disappear
</Card>

## What Just Happened?

Your Listener now handles two key Events:

1. **`space:configure`** - Sets up the data structure
2. **`commit:created`** - Validates email format when users commit changes

Here's how the email validation works step by step:

### 1. Listen for Commits

This listener triggers whenever users save their changes to any sheet in the workbook.

<CodeGroup>
  ```javascript JavaScript theme={null}
  listener.on("commit:created", async (event) => {
    const { sheetId } = event.context;
  ```

  ```typescript TypeScript theme={null}
  listener.on("commit:created", async (event) => {
    const { sheetId } = event.context;
  ```
</CodeGroup>

### 2. Get the Records

We retrieve all records from the sheet to validate them.

<CodeGroup>
  ```javascript JavaScript theme={null}
  const response = await api.records.get(sheetId);
  const records = response.data.records;
  ```

  ```typescript TypeScript theme={null}
  const response = await api.records.get(sheetId);
  const records = response.data.records;
  ```
</CodeGroup>

### 3. Validate Email Format

We use a simple regex pattern to check if each email follows the basic `user@domain.com` format.

<CodeGroup>
  ```javascript JavaScript theme={null}
  const emailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;

  for (const record of records) {
    const emailValue = record.values.email?.value;
    if (emailValue && !emailRegex.test(emailValue.toLowerCase())) {
      // Add validation error
    }
  }
  ```

  ```typescript TypeScript theme={null}
  const emailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;

  for (const record of records) {
    const emailValue = record.values.email?.value as string;
    if (emailValue && !emailRegex.test(emailValue.toLowerCase())) {
      // Add validation error
    }
  }
  ```
</CodeGroup>

### 4. Add Error Messages

For invalid emails, we create an update that adds an error message to that specific field.

<CodeGroup>
  ```javascript JavaScript theme={null}
  updates.push({
    id: record.id,
    values: {
      email: {
        value: email,
        messages: [{
          type: "error",
          message: "Please enter a valid email address (e.g., user@example.com)",
        }],
      },
    },
  });
  ```

  ```typescript TypeScript theme={null}
  updates.push({
    id: record.id,
    values: {
      email: {
        value: email,
        messages: [{
          type: "error",
          message: "Please enter a valid email address (e.g., user@example.com)",
        }],
      },
    },
  });
  ```
</CodeGroup>

<Info>
  You can apply different types of validation messages:

  * **`info`**: Informational messages (mouseover tooltip)
  * **`warn`**: Warnings that don't block processing (yellow)
  * **`error`**: Errors that should be fixed, blocks [Actions](/core-concepts/actions) with the `hasAllValid` constraint (red)
</Info>

### 5. Update Records

Finally, we send all validation messages back to the sheet so users can see the errors.

<CodeGroup>
  ```javascript JavaScript theme={null}
  if (updates.length > 0) {
    await api.records.update(sheetId, updates);
  }
  ```

  ```typescript TypeScript theme={null}
  if (updates.length > 0) {
    await api.records.update(sheetId, updates);
  }
  ```
</CodeGroup>

## Next Steps

Ready to make your Listener interactive? Continue to [Adding Actions](/coding-tutorial/101-your-first-listener/101.03-adding-actions) to learn how to handle user submissions and create custom workflows.

For more detailed information:

* Understand Job lifecycle patterns in [Jobs](/core-concepts/jobs) and [Spaces](/core-concepts/spaces)
* Learn more about [Events](/reference/events)
* Organize your Listeners with [Namespaces](/guides/namespaces-and-filters)
* Explore [plugins](/core-concepts/plugins): [Job Handler](/plugins/job-handler) and [Space Configure](/plugins/space-configure)
* Check out [Record Hook](/plugins/record-hook) for simpler Field-level validations


# 03: Adding Actions to Your Listener
Source: https://flatfile.com/docs/coding-tutorial/101-your-first-listener/101.03-adding-actions

Build on your basic Listener by adding user Actions to create interactive data processing workflows.

In the [previous guides](/coding-tutorial/101-your-first-listener/101.02-adding-validation), we created a Listener with Space configuration and data validation. Now we'll extend that Listener to handle user Actions, allowing users to submit and process their data.

<Note>
  **Following along?** Download the starting code from our [Getting Started repository](https://github.com/FlatFilers/getting-started/tree/main/101.02-adding-validation) and refactor it as we go, or jump directly to the [final version with actions](https://github.com/FlatFilers/getting-started/tree/main/101.03-adding-actions).
</Note>

## What Are Actions?

[Actions](/core-concepts/actions) are interactive buttons that appear in the Flatfile interface, allowing users to trigger custom operations on their data. Common Actions include:

* **Submit**: Process your data and POST it to your system via API
* **Validate**: Run custom validation rules
* **Transform**: Apply data transformations
* **Export**: Generate reports or exports

<Frame>
  <img />
</Frame>

For more detail on using Actions, see our [Actions](/guides/using-actions) guide.

## What Changes We're Making

To add Actions to our Listener with validation, we need to make two specific changes:

### 1. Add Actions Array to Blueprint Definition

In the `space:configure` Listener, we'll add an `actions` array to our Workbook creation. This enhances our [Blueprint](/core-concepts/blueprints) to include interactive elements:

```javascript theme={null}
actions: [
  {
    label: "Submit",
    description: "Send data to destination system",
    operation: "submitActionForeground",
    mode: "foreground",
  },
]
```

### 2. Add Action Handler Listener

We'll add a new Listener to handle when users click the Submit button:

```javascript theme={null}
listener.on(
  "job:ready",
  { job: "workbook:submitActionForeground" },
  async (event) => {
    // Handle the action...
  }
);
```

## Complete Example with Actions

<Note>
  This example builds on the Listener we created in the [previous tutorials](/coding-tutorial/101-your-first-listener/101.02-adding-validation). It includes the complete functionality: Space configuration, email validation, and Actions.
</Note>

<CodeGroup>
  ```javascript JavaScript theme={null}
  import api from "@flatfile/api";
  export default function (listener) {
    // Configure the space when it's created
    listener.on("job:ready", { job: "space:configure" }, async (event) => {
      const { jobId, spaceId } = event.context;
      try {
        // Acknowledge the job
        await api.jobs.ack(jobId, {
          info: "Setting up your workspace...",
          progress: 10,
        });
        // Create the Workbook with Sheets and Actions
        await api.workbooks.create({
          spaceId,
          name: "My Workbook",
          sheets: [
            {
              name: "contacts",
              slug: "contacts",
              fields: [
                { key: "name", type: "string", label: "Full Name" },
                { key: "email", type: "string", label: "Email" },
              ],
            },
          ],
          actions: [
            {
              label: "Submit",
              description: "Send data to destination system",
              operation: "submitActionForeground",
              mode: "foreground",
              primary: true,
            },
          ],
        });
        // Update progress
        await api.jobs.update(jobId, {
          info: "Workbook created successfully",
          progress: 75,
        });
        // Complete the job
        await api.jobs.complete(jobId, {
          outcome: {
            message: "Workspace configured successfully!",
            acknowledge: true,
          },
        });
      } catch (error) {
        console.error("Error configuring space:", error);
        // Fail the job if something goes wrong
        await api.jobs.fail(jobId, {
          outcome: {
            message: `Failed to configure workspace: ${error.message}`,
            acknowledge: true,
          },
        });
      }
    });
    // Handle when someone clicks Submit
    listener.on(
      "job:ready",
      { job: "workbook:submitActionForeground" },
      async (event) => {
        const { jobId, workbookId } = event.context;
        try {
          // Acknowledge the job
          await api.jobs.ack(jobId, {
            info: "Starting data processing...",
            progress: 10,
          });
          // Get the data
          const job = await api.jobs.get(jobId);
          // Update progress
          await api.jobs.update(jobId, {
            info: "Retrieving records...",
            progress: 30,
          });
          // Get the sheets
          const { data: sheets } = await api.sheets.list({ workbookId });
          // Get and count the records
          const records = {};
          let recordsCount = 0;
          for (const sheet of sheets) {
            const {
              data: { records: sheetRecords },
            } = await api.records.get(sheet.id);
            records[sheet.name] = sheetRecords;
            recordsCount += sheetRecords.length;
          }
          // Update progress
          await api.jobs.update(jobId, {
            info: `Processing ${sheets.length} sheets with ${recordsCount} records...`,
            progress: 60,
          });
          // Process the data (log to console for now)
          console.log("Processing records:", JSON.stringify(records, null, 2));
          // Complete the job
          await api.jobs.complete(jobId, {
            outcome: {
              message: `Successfully processed ${sheets.length} sheets with ${recordsCount} records!`,
              acknowledge: true,
            },
          });
        } catch (error) {
          console.error("Error processing data:", error);
          // Fail the job if something goes wrong
          await api.jobs.fail(jobId, {
            outcome: {
              message: `Data processing failed: ${error.message}`,
              acknowledge: true,
            },
          });
        }
      },
    );

    // Listen for commits and validate email format
    listener.on("commit:created", async (event) => {
      const { sheetId } = event.context;
      try {
        // Get records from the sheet
        const response = await api.records.get(sheetId);
        const records = response.data.records;
        // Simple email validation regex
        const emailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
        // Prepare updates for records with invalid emails
        const updates = [];
        for (const record of records) {
          const emailValue = record.values.email?.value;
          if (emailValue) {
            const email = emailValue.toLowerCase();
            if (!emailRegex.test(email)) {
              updates.push({
                id: record.id,
                values: {
                  email: {
                    value: email,
                    messages: [
                      {
                        type: "error",
                        message:
                          "Please enter a valid email address (e.g., user@example.com)",
                      },
                    ],
                  },
                },
              });
            }
          }
        }
        // Update records with validation messages
        if (updates.length > 0) {
          await api.records.update(sheetId, updates);
        }
      } catch (error) {
        console.error("Error during validation:", error);
      }
    });
  }
  ```

  ```typescript TypeScript theme={null}
  import type { FlatfileListener } from "@flatfile/listener";
  import api, { Flatfile } from "@flatfile/api";

  export default function (listener: FlatfileListener) {
    // Configure the space when it's created
    listener.on("job:ready", { job: "space:configure" }, async (event) => {
      const { jobId, spaceId } = event.context;
      
      try {
        // Acknowledge the job
        await api.jobs.ack(jobId, {
          info: "Setting up your workspace...",
          progress: 10
        });

        // Create the Workbook with Sheets and Actions
        await api.workbooks.create({
          spaceId,
          name: "My Workbook",
          sheets: [
            {
              name: "contacts",
              slug: "contacts",
              fields: [
                { key: "name", type: "string", label: "Full Name" },
                { key: "email", type: "string", label: "Email" },
              ],
            },
          ],
          actions: [
            {
              label: "Submit",
              description: "Send data to destination system",
              operation: "submitActionForeground",
              mode: "foreground",
              primary: true,
            },
          ],
        });

        // Update progress
        await api.jobs.update(jobId, {
          info: "Workbook created successfully",
          progress: 75
        });

        // Complete the job
        await api.jobs.complete(jobId, {
          outcome: {
            message: "Workspace configured successfully!",
            acknowledge: true
          }
        });

      } catch (error) {
        console.error("Error configuring space:", error);
        
        // Fail the job if something goes wrong
        await api.jobs.fail(jobId, {
          outcome: {
            message: `Failed to configure workspace: ${error instanceof Error ? error.message : 'Unknown error'}`,
            acknowledge: true
          }
        });
      }
    });

    // Handle when someone clicks Submit
    listener.on(
      "job:ready",
      { job: "workbook:submitActionForeground" },
      async (event) => {
        const { jobId, workbookId } = event.context;

        try {
          // Acknowledge the job
          await api.jobs.ack(jobId, {
            info: "Starting data processing...",
            progress: 10
          });

          // Get the data
          const job = await api.jobs.get(jobId);
          
          // Update progress
          await api.jobs.update(jobId, {
            info: "Retrieving records...",
            progress: 30
          });

          // Get the sheets
          const { data: sheets } = await api.sheets.list({ workbookId });

          // Get and count the records
          const records: { [name: string]: any[] } = {};
          let recordsCount = 0;
          for (const sheet of sheets) {
            const { data: { records: sheetRecords}} = await api.records.get(sheet.id);
            records[sheet.name] = sheetRecords;
            recordsCount += sheetRecords.length;
          }

          // Update progress
          await api.jobs.update(jobId, {
            info: `Processing ${sheets.length} sheets with ${recordsCount} records...`,
            progress: 60
          });

          // Process the data (log to console for now)
          console.log("Processing records:", JSON.stringify(records, null, 2));

          // Complete the job
          await api.jobs.complete(jobId, {
            outcome: {
              message: `Successfully processed ${sheets.length} sheets with ${recordsCount} records!`,
              acknowledge: true
            }
          });

        } catch (error) {
          console.error("Error processing data:", error);
          
          // Fail the job if something goes wrong
          await api.jobs.fail(jobId, {
            outcome: {
              message: `Data processing failed: ${error instanceof Error ? error.message : 'Unknown error'}`,
              acknowledge: true
            }
          });
        }
      }
    );

    // Listen for commits and validate email format
    listener.on("commit:created", async (event) => {
      const { sheetId } = event.context;
      
      try {
        // Get records from the sheet
        const response = await api.records.get(sheetId);
        const records = response.data.records;

        // Simple email validation regex
        const emailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;

        // Prepare updates for records with invalid emails
        const updates: Flatfile.RecordWithLinks[] = [];

        for (const record of records) {
          const emailValue = record.values.email?.value as string; 
          
          if (emailValue) {
            const email = emailValue.toLowerCase();

            if (!emailRegex.test(email)) {
              updates.push({
                id: record.id,
                values: {
                  email: {
                    value: email,
                    messages: [{
                      type: "error",
                      message: "Please enter a valid email address (e.g., user@example.com)",
                    }],
                  },
                },
              });
            }
          }
        }

        // Update records with validation messages
        if (updates.length > 0) {
          await api.records.update(sheetId, updates);
        }

      } catch (error) {
        console.error("Error during validation:", error);
      }
    });
  }
  ```
</CodeGroup>

<Note>
  **Complete Example**: The full working code for this tutorial step is available in our Getting Started repository: [JavaScript](https://github.com/FlatFilers/getting-started/tree/main/101.03-adding-actions/javascript) | [TypeScript](https://github.com/FlatFilers/getting-started/tree/main/101.03-adding-actions/typescript)
</Note>

## Understanding Action Modes

Actions can run in different modes:

* **`foreground`**: Runs immediately with real-time progress updates (good for quick operations)
* **`background`**: Runs as a background job (good for longer operations)

The Action operation name (`submitActionForeground`) determines which Listener will handle the Action.

## Testing Your Action

### Local Development

To test your Listener locally, you can use the `flatfile develop` command. This will start a local server that will listen for Events and respond to them, and will also watch for changes to your Listener code and automatically reload the server.

```bash theme={null}
# Run locally with file watching
npx flatfile develop
```

### Step-by-Step Testing

After running your listener locally:

<Card title="Testing Steps">
  1. Create a new Space in your Flatfile Environment
  2. Upload (or manually enter) some data to the contacts Sheet with both valid and invalid email addresses
  3. See validation errors appear on invalid email Fields
  4. Click the "Submit" button
  5. Watch the logging in the terminal as your data is processed and the job is completed
</Card>

## What Just Happened?

Your Listener now handles three key Events and provides a complete data import workflow. Here's how the new action handling works:

### 1. Blueprint Definition with Actions

We enhanced the [Blueprint](/core-concepts/blueprints) definition to include action buttons that users can interact with. Adding actions to your workbook configuration is part of defining your Blueprint.

<CodeGroup>
  ```javascript JavaScript theme={null}
  actions: [
    {
      label: "Submit",
      description: "Send data to destination system",
      operation: "submitActionForeground",
      mode: "foreground",
      primary: true,
    },
  ]
  ```

  ```typescript TypeScript theme={null}
  actions: [
    {
      label: "Submit",
      description: "Send data to destination system",
      operation: "submitActionForeground",
      mode: "foreground", 
      primary: true,
    },
  ]
  ```
</CodeGroup>

### 2. Listen for Action Events

When users click the Submit button, Flatfile triggers a [Job](/core-concepts/jobs) that your listener can handle using the same approach we used for the `space:configure` job in [101.01](/coding-tutorial/101-your-first-listener/101.01-first-listener#2-listen-for-space-configuration).

Jobs are named with the pattern `<domain>:<operation>`. In this case, the domain is `workbook` since we've mounted the Action to the Workbook blueprint, and the operation is `submitActionForeground` as defined in the Action definition.

<CodeGroup>
  ```javascript JavaScript theme={null}
  listener.on(
    "job:ready",
    { job: "workbook:submitActionForeground" },
    async (event) => {
      const { jobId, workbookId } = event.context;
  ```

  ```typescript TypeScript theme={null}
  listener.on(
    "job:ready", 
    { job: "workbook:submitActionForeground" },
    async (event) => {
      const { jobId, workbookId } = event.context;
  ```
</CodeGroup>

### 3. Retrieve and Process Data

Get all the data from the workbook and process it according to your business logic.

<CodeGroup>
  ```javascript JavaScript theme={null}
  // Get the sheets
  const { data: sheets } = await api.sheets.list({ workbookId });

  // Get and count the records
  const records = {};
  let recordsCount = 0;
  for (const sheet of sheets) {
    const { data: { records: sheetRecords } } = await api.records.get(sheet.id);
    records[sheet.name] = sheetRecords;
    recordsCount += sheetRecords.length;
  }
  ```

  ```typescript TypeScript   theme={null}
  // Get the sheets
  const { data: sheets } = await api.sheets.list({ workbookId });

  // Get and count the records
  const records: { [name: string]: any[] } = {};
  let recordsCount = 0;
  for (const sheet of sheets) {
    const { data: { records: sheetRecords }} = await api.records.get(sheet.id);
    records[sheet.name] = sheetRecords;
    recordsCount += sheetRecords.length;
  }
  ```
</CodeGroup>

### 4. Provide User Feedback

Keep users informed about the processing with progress updates and final results.

<CodeGroup>
  ```javascript JavaScript theme={null}
  // Update progress during processing
  await api.jobs.update(jobId, {
    info: `Processing ${sheets.length} sheets with ${recordsCount} records...`,
    progress: 60,
  });

  // Complete with success message
  await api.jobs.complete(jobId, {
    outcome: {
      message: `Successfully processed ${sheets.length} sheets with ${recordsCount} records!`,
      acknowledge: true,
    },
  });
  ```

  ```typescript TypeScript theme={null}
  // Update progress during processing  
  await api.jobs.update(jobId, {
    info: `Processing ${sheets.length} sheets with ${recordsCount} records...`,
    progress: 60
  });

  // Complete with success message
  await api.jobs.complete(jobId, {
    outcome: {
      message: `Successfully processed ${sheets.length} sheets with ${recordsCount} records!`,
      acknowledge: true
    }
  });
  ```
</CodeGroup>

Your complete Listener now handles:

* **`space:configure`** - Defines the Blueprint with interactive actions
* **`commit:created`** - Validates email format when users commit changes
* **`workbook:submitActionForeground`** - Processes data when users click Submit

The Action follows the same Job lifecycle pattern: **acknowledge → update progress → complete** (or fail on error). This provides users with real-time feedback during data processing, while validation ensures data quality throughout the import process.

## Next Steps

Congratulations! You now have a complete Listener that handles Space configuration, data validation, and user Actions.

For more detailed information:

* Learn more about [Actions](/guides/using-actions)
* Understand Job lifecycle patterns in [Jobs](/core-concepts/jobs)
* Learn more about [Events](/reference/events)
* Organize your Listeners with [Namespaces](/guides/namespaces-and-filters)
* Explore [plugins](/core-concepts/plugins): [Job Handler](/plugins/job-handler) and [Space Configure](/plugins/space-configure)


# Coding Tutorial
Source: https://flatfile.com/docs/coding-tutorial/overview

Learn to build Event Listeners that configure and customize Flatfile through hands-on coding tutorials

## Events Drive Everything in Flatfile

**When you configure Flatfile with code, you're building part of an event-driven system.** Every time a user interacts with their data—editing a Record, uploading a file, clicking an [Action](/core-concepts/actions) button, etc.—the Flatfile Platform emits an **Event**. Your custom **Listeners**, defined in your code and generally deployed to the Flatfile Platform as an [Agent](/core-concepts/listeners#agents), respond to these Events by executing whatever logic you define.

**Out of the box, Flatfile is essentially an empty slate.** Everything you see—[Workbooks](/core-concepts/workbooks), [Sheets](/core-concepts/sheets), [Fields](/core-concepts/fields), validation logic—is created by your Listeners responding to Events. This logic includes configuring your [Blueprint](/core-concepts/blueprints) in a `space:configure` job, and may be as simple as logging an Event to the console, or as complex as processing records against existing data from your own API. But given the Event/Listener pattern as basic building blocks, you can build pretty much anything your imagination can conjure.

```mermaid theme={null}
sequenceDiagram
    participant User
    participant Platform as Flatfile Platform  
    participant Listener as Listener Logic

    User->>Platform: Edits data
    Platform->>Platform: Internal processing
    Platform->>Listener: Emits "commit:created"
    Listener->>Listener: Executes your code
```

This event-driven paradigm is fundamentally different from traditional "configuration", where you pre-define static options. Instead, you write reactive code that responds to what users actually do. Events carry structured information about what happened, including the context and any relevant data, giving your Listeners everything they need to respond intelligently.

Common Events include:

* `commit:created` - When a commit is created (after records are added or modified)
* `job:ready` - When [Jobs](/core-concepts/jobs) are ready for execution
* `file:created` - When a file is uploaded

This is just a very small sample of the Events emitted by Flatfile. For a complete list, see the [Events Reference](/reference/events).

Your Listeners define how Flatfile behaves. Want to validate data when records are modified? Listen for `commit:created` Events and perform your validation in the callback. Need to process uploaded files? Listen for `file:created` Events. Want to trigger discrete units of work? Listen for `job:ready` Events (generally filtered by `{job: "domain:operation"}`, but we'll cover that later).

For a more comprehensive understanding of this topic, see [Events and Listeners](/core-concepts/listeners).

## Learning Path: Building Custom Listeners

This tutorial series teaches you to build Event Listeners through hands-on coding. You'll start with basic Space configuration and progressively add more sophisticated functionality.

**Prerequisites**:

* Basic JavaScript or TypeScript knowledge
* Node.js and npm installed (latest LTS version recommended)

You may also find it helpful to review the [Core Concepts](/core-concepts/overview) documentation to get a better understanding of the structures and terminology used in this tutorial, or just jump right into things and keep the Core Concepts documentation in mind in case you come across something you don't understand. You might even keep it open in another tab for handy reference later on.

**Getting Started**: This tutorial is designed to be completed from scratch from an empty folder. You'll build everything step-by-step as you follow along.

**Reference Code**: If you'd like to skip ahead or you'd like a complete reference, you can find complete working examples in our [Getting Started repository](https://github.com/FlatFilers/getting-started/) with both JavaScript and TypeScript versions.

### Listeners 101: Your First Listener

A three-part series that builds a complete Listener from scratch. By the end of this series, you'll have a single-page Listener that configures Spaces, creates Workbooks and Sheets, validates a single field, and adds a custom Action.

This is a great way to learn the basics of Listener development and get a feel for how to build your own Listeners, but it's meant to be a starting point; In the next series, we'll show you how to make your Listeners more concise and structured to prepare for more complex use cases.

<CardGroup>
  <Card title="Section 01: Your First Listener" href="/coding-tutorial/101-your-first-listener/101.01-first-listener">
    Set up your development environment and build a Listener that configures Spaces, creates Workbooks and Sheets, and manages the Job lifecycle. You'll learn the foundational patterns for all Listener development.
  </Card>

  <Card title="Section 02: Adding Validation" href="/coding-tutorial/101-your-first-listener/101.02-adding-validation">
    Add data validation to ensure Record quality. You'll learn to validate individual Fields, provide helpful error messages, and guide users toward clean data.
  </Card>

  <Card title="Section 03: Adding Actions" href="/coding-tutorial/101-your-first-listener/101.03-adding-actions">
    Extend your Listener with custom Actions that let users trigger your code on demand. You'll learn to create interactive workflows and provide user feedback.
  </Card>
</CardGroup>


# Actions
Source: https://flatfile.com/docs/core-concepts/actions

User-triggered operations in Flatfile

An Action is a code-based operation that runs when a user clicks a button or menu item in Flatfile. Actions can be mounted on [Sheets](/core-concepts/sheets), [Workbooks](/core-concepts/workbooks), [Documents](/core-concepts/documents), or Files to trigger custom operations.

Defining a custom Action is a two-step process:

1. Define an Action in your Flatfile blueprint or in your code
2. Create a [Listener](/core-concepts/listeners) to handle the Action

When an Action is triggered, it creates a [Job](/core-concepts/jobs) that your application can listen for and respond to.

Given that Actions are powered by Jobs, the [Jobs Lifecycle](/core-concepts/jobs#jobs-lifecycle) pertains to Actions as well. This means that you can [update progress values/messages](/core-concepts/jobs#updating-job-progress) while an Action is processing, and when it's done you can provide an [Outcome](/core-concepts/jobs#job-outcomes), which allows you to show a success message, automatically [download a generated file](/core-concepts/jobs#file-downloads), or [forward the user](/core-concepts/jobs#internal-navigation) to a generated Document.

<Note>
  For complete implementation details, see our [Using Actions guide](/guides/using-actions).
</Note>

<Frame>
  <img />
</Frame>

## Types of Actions

### Built-in Actions

Resources in Flatfile come with severaldefault built-in actions like:

* Export/download data
* Delete data or files
* Find and replace (Sheets)

### Developer-Created Actions

You can create custom Actions to handle operations specific to your workflow, such as:

* Sending data to your API when data is ready
* Downloading your data in a specific format
* Validating data against external systems
* Moving data between different resources
* Custom data validations andtransformations

## Where Actions Appear

Actions appear in different parts of the UI depending on where they're mounted:

* **Workbook Actions**: Buttons in the top-right corner of Workbooks
* **Sheet Actions**: Dropdown menu in the Sheet toolbar (or top-level button if marked as `primary`)
* **Document Actions**: Buttons in the top-right corner of Documents
* **File Actions**: Dropdown menu for each file in the Files list

## Example Action Configuration

Every Action requires an `operation` (unique identifier) and `label` (display text):

```javascript theme={null}
{
  operation: "submitActionBg",
  mode: "background",
  label: "Submit",
  type: "string",
  description: "Submit this data to a webhook.",
  primary: true,
},
```

Actions support additional options like `primary` status, confirmation dialogs, constraints, and input forms. See the [Using Actions guide](/guides/using-actions) for more details.


# Apps
Source: https://flatfile.com/docs/core-concepts/apps

The anatomy of an App

## Apps

Apps are an organizational unit in Flatfile, designed to manage and coordinate data import workflows across different environments. They serve as containers for organizing related Spaces and provide a consistent configuration that can be deployed across your development pipeline.

Apps can be given [namespaces](/guides/namespaces-and-filters#app-namespaces) to isolate different parts of your application and control which [listeners](/core-concepts/listeners) receive events from which spaces.

<Frame>
  <img />
</Frame>

Apps are available across Development-level environments by default, and optionally available across Production environments with a configuration option:

<Frame>
  <img />
</Frame>


# Blueprints
Source: https://flatfile.com/docs/core-concepts/blueprints

Define your data schema to structure exactly how data should look, behave, and connect

## What is a Blueprint?

Blueprints enable you to create repeatable, reliable data import experiences that scale with your needs while maintaining data quality and user experience standards.

A Blueprint is your complete data definition in Flatfile. It controls how your data should look, behave, and connect—from simple field validations (like `unique` and `required`) to complex [relationships](/core-concepts/fields#reference) between [sheets](/core-concepts/sheets). You can even create [filtered reference fields](/core-concepts/fields#reference-field-filtering) that dynamically control available dropdown options based on other field values. Think of it as an intelligent template that ensures you collect the right data in the right format, every time.

<Note>
  **Terminology Note**: "Blueprint" is Flatfile's term for what might be called a
  "schema" in other systems. Throughout Flatfile's documentation and API, we use
  "Blueprint" as the standard term for data structure definitions to distinguish
  Flatfile's comprehensive data modeling approach from generic schema concepts.
</Note>

## How Blueprints Work

Every [Space](/core-concepts/spaces) has exactly one Blueprint that defines its data structure. Whenever a new space is created, the Flatfile Platform automatically triggers a `space:configure` [Job](/core-concepts/jobs), and you can configure a [Listener](/core-concepts/listeners) to pick up that job and configure the new space by defining its Blueprint. Creating workbooks, sheets, and actions **is** your Blueprint definition, establishing the data schema that will govern all data within that Space.

To make that part easier, we have provided the [Space Configure Plugin](/plugins/space-configure) to abstract away the Job/Listener code, allowing you to focus on what matters: Preparing your space for data.

## Basic Blueprint Structure

* A [Blueprint](/core-concepts/blueprints) defines the data structure for any number of [Spaces](/core-concepts/spaces)
* A [Space](/core-concepts/blueprints) may contain many [Workbooks](/core-concepts/workbooks) and many [Documents](/core-concepts/documents)
* A [Document](/core-concepts/documents) contains static documentation and may contain many [Document-level Actions](/guides/using-actions#document-actions)
* A [Workbook](/core-concepts/workbooks) may contain many [Sheets](/core-concepts/sheets) and many [Workbook-level Actions](/guides/using-actions#workbook-actions)
* A [Sheet](/core-concepts/sheets) may contain many [Fields](/core-concepts/fields) and many [Sheet-level Actions](/guides/using-actions#sheet-actions)
* A [Field](/core-concepts/fields) defines a single column of data, and may contain many [Field-level Actions](/guides/using-actions#field-actions)

<Note>
  **A note about Actions:** Actions also require a listener to respond to the event published by clicking
  on them. For more, see [Using Actions](/guides/using-actions)
</Note>

## Example Blueprint Configuration

<Tip>
  **Recommendation:** Although throughout the documentation we'll be explicitly defining each level of a blueprint, it's important to note that you can split each of your **Workbooks**, **Sheets**, **Documents**, and **Actions** definitions into separate files and import them. Then your Workbook blueprint can be as simple as:

  ```javascript theme={null}
  const companyWorkbook = {
    name: "Company Workbook",
    documents: [dataProcessingSteps]
    sheets: [usersSheet],
    actions: [exportToCRM],
  };
  ```

  This leads to a more maintainable codebase, and the modularity opens the door for code reuse. For instance, you'll be able to use `usersSheet.slug` in your listener code to filter or differentiate between sheets, or re-use `exportToSCRM` in any other workbook that needs to export data to a CRM.
</Tip>

This example shows a Blueprint definition for [Space configuration](/core-concepts/spaces#space-configuration). It creates a single [Workbook](/core-concepts/workbooks) with a single [Document](/core-concepts/documents) and a single [Sheet](/core-concepts/sheets) containing two [Fields](/core-concepts/fields) and one [Action](/core-concepts/actions).

```javascript theme={null}
const workbooks = [{
  name: "Company Workbook",
  documents: [
    {
      title: "Data Processing Walkthrough",
      body: "1. Add Data\n2. Process Data\n3. Export Data",
      actions: [
        {
          operation: "confirm",
          label: "Confirm",
          type: "string",
          primary: true,
        },
      ],
    },
  ],
  sheets: [
    {
      name: "Users",
      slug: "users",
      fields: [
        {
          key: "fname",
          type: "string",
          label: "First Name",
        },
        {
          key: "lname",
          type: "string",
          label: "Last Name",
        },
      ],
      actions: [
        {
          operation: "validate-inventory",
          mode: "background",
          label: "Validate Inventory",
          description: "Check product availability against inventory system",
        },
      ],
    },
  ],
  actions: [
    {
      operation: "export-to-crm",
      mode: "foreground",
      label: "Export to CRM",
      description: "Send validated customers to Salesforce",
    },
  ],
}];
```

## Workbook Folders and Sheet Collections

Although they have no impact on your data itself or its structure, [Workbook Folders](/core-concepts/workbooks#folders) and [Sheet Collections](/core-concepts/sheets#collections) are a powerful way to organize your data in the Flatfile UI.

They are essentially named labels that you assign to your Workbooks and Sheets, which the Flatfile UI interprets to group them together (and apart from others). You can define them directly in your [Blueprint](/core-concepts/blueprints) when [configuring your Space](/core-concepts/spaces#space-configuration) or when otherwise creating or updating a Workbook or Sheet via the [API](https://reference.flatfile.com).

You can think of **Folders** and **Collections** like a filing system:

* [Folders](/core-concepts/workbooks#folders) help you organize your Workbooks within a Space (like organizing binders on a shelf).
* [Collections](/core-concepts/sheets#collections) help you organize Sheets within each Workbook (like organizing tabs within a binder).

This is a great way to declutter your Sidebar and keep your data organized and easy to find in the Flatfile UI.

In the following example, we have several Workbooks grouped into two Folders:

* **Analytics** (folded)
* **Business Operations** (unfolded)

The **Business Operations** Workbooks each contain several Sheets grouped into Collections:

* **Compensation** and **Personel**
* **Stock Management** and **Vendor Management**

<Tabs>
  <Tab title="Screenshot">
    <Frame>
      <img />
    </Frame>
  </Tab>

  <Tab title="Workbook Blueprints">
    ```javascript theme={null}
    const salesReportWorkbook = {
      name: "Sales Analytics",
      folder: "Analytics",
      sheets: [
        // Source Data collection (2 sheets)
        salesDataSheet, 
        revenueSheet,
        // Analytics collection (2 sheets) 
        campaignMetricsSheet, 
        leadSourcesSheet
      ]
    };

    const humanResourcesWorkbook = {
      name: "Human Resources Management",
      folder: "Business Operations",
      sheets: [
        // Personnel collection (2 sheets)
        employeesSheet, 
        departmentsSheet,
        // Compensation collection (2 sheets)
        payrollSheet, 
        benefitsSheet
      ]
    };

    const operationsWorkbook = {
      name: "Operations Management",
      folder: "Business Operations",
      sheets: [
        // Stock Management collection (2 sheets)
        inventorySheet,
        warehousesSheet,
        // Vendor Management collection (2 sheets)
        suppliersSheet,
        purchaseOrdersSheet
      ]
    };
    ```
  </Tab>

  <Tab title="Sheet Blueprints">
    ```javascript theme={null}
    const salesDataSheet = {
      name: "Sales Data",
      collection: "Source Data",
      fields: [
        { key: "name", type: "string", label: "Customer Name" },
        { key: "email", type: "string", label: "Email Address" }
      ]
    };

    const revenueSheet = {
      name: "Revenue",
      collection: "Analytics",
      fields: [
        { key: "revenue", type: "number", label: "Revenue" }
      ]
    };

    const campaignMetricsSheet = {
      name: "Campaign Metrics",
      collection: "Analytics",
      fields: [
        { key: "impressions", type: "number", label: "Impressions" },
        { key: "clicks", type: "number", label: "Clicks" }
      ]
    };

    const leadSourcesSheet = {
      name: "Lead Sources",
      collection: "Analytics",
      fields: [
        { key: "source", type: "string", label: "Source" },
        { key: "conversion_rate", type: "number", label: "Conversion Rate" }
      ]
    };

    const employeesSheet = {
      name: "Employees",
      collection: "Personnel",
      fields: [
        { key: "employee_id", type: "string", label: "Employee ID" },
        { key: "name", type: "string", label: "Full Name" },
        { key: "department", type: "string", label: "Department" },
        { key: "hire_date", type: "date", label: "Hire Date" }
      ]
    };

    const departmentsSheet = {
      name: "Departments",
      collection: "Personnel",
      fields: [
        { key: "dept_code", type: "string", label: "Department Code" },
        { key: "dept_name", type: "string", label: "Department Name" },
        { key: "manager", type: "string", label: "Manager" }
      ]
    };

    const positionsSheet = {
      name: "Job Positions",
      collection: "Personnel",
      fields: [
        { key: "position_id", type: "string", label: "Position ID" },
        { key: "title", type: "string", label: "Job Title" },
        { key: "level", type: "string", label: "Job Level" },
        { key: "department", type: "string", label: "Department" }
      ]
    };

    const payrollSheet = {
      name: "Payroll",
      collection: "Compensation",
      fields: [
        { key: "employee_id", type: "string", label: "Employee ID" },
        { key: "salary", type: "number", label: "Annual Salary" },
        { key: "bonus", type: "number", label: "Bonus" }
      ]
    };

    const benefitsSheet = {
      name: "Benefits",
      collection: "Compensation",
      fields: [
        { key: "benefit_type", type: "string", label: "Benefit Type" },
        { key: "cost", type: "number", label: "Monthly Cost" },
        { key: "coverage", type: "string", label: "Coverage Level" }
      ]
    };

    const bonusesSheet = {
      name: "Performance Bonuses",
      collection: "Compensation",
      fields: [
        { key: "employee_id", type: "string", label: "Employee ID" },
        { key: "performance_rating", type: "string", label: "Performance Rating" },
        { key: "bonus_amount", type: "number", label: "Bonus Amount" },
        { key: "quarter", type: "string", label: "Quarter" }
      ]
    };

    const attendanceSheet = {
      name: "Attendance",
      collection: "Time Tracking",
      fields: [
        { key: "employee_id", type: "string", label: "Employee ID" },
        { key: "date", type: "date", label: "Date" },
        { key: "hours_worked", type: "number", label: "Hours Worked" },
        { key: "overtime", type: "number", label: "Overtime Hours" }
      ]
    };

    const leaveRequestsSheet = {
      name: "Leave Requests",
      collection: "Time Tracking",
      fields: [
        { key: "request_id", type: "string", label: "Request ID" },
        { key: "employee_id", type: "string", label: "Employee ID" },
        { key: "leave_type", type: "string", label: "Leave Type" },
        { key: "start_date", type: "date", label: "Start Date" },
        { key: "end_date", type: "date", label: "End Date" }
      ]
    };

    const inventorySheet = {
      name: "Inventory",
      collection: "Stock Management",
      fields: [
        { key: "sku", type: "string", label: "SKU" },
        { key: "product_name", type: "string", label: "Product Name" },
        { key: "quantity", type: "number", label: "Quantity in Stock" },
        { key: "reorder_level", type: "number", label: "Reorder Level" }
      ]
    };

    const warehousesSheet = {
      name: "Warehouses",
      collection: "Stock Management",
      fields: [
        { key: "warehouse_id", type: "string", label: "Warehouse ID" },
        { key: "location", type: "string", label: "Location" },
        { key: "capacity", type: "number", label: "Storage Capacity" },
        { key: "manager", type: "string", label: "Warehouse Manager" }
      ]
    };

    const stockMovementsSheet = {
      name: "Stock Movements",
      collection: "Stock Management",
      fields: [
        { key: "movement_id", type: "string", label: "Movement ID" },
        { key: "sku", type: "string", label: "SKU" },
        { key: "quantity", type: "number", label: "Quantity" },
        { key: "movement_type", type: "string", label: "Movement Type" },
        { key: "date", type: "date", label: "Date" }
      ]
    };

    const suppliersSheet = {
      name: "Suppliers",
      collection: "Vendor Management",
      fields: [
        { key: "supplier_id", type: "string", label: "Supplier ID" },
        { key: "company_name", type: "string", label: "Company Name" },
        { key: "contact_person", type: "string", label: "Contact Person" },
        { key: "email", type: "string", label: "Email" }
      ]
    };

    const purchaseOrdersSheet = {
      name: "Purchase Orders",
      collection: "Vendor Management",
      fields: [
        { key: "order_id", type: "string", label: "Order ID" },
        { key: "supplier_id", type: "string", label: "Supplier ID" },
        { key: "order_date", type: "date", label: "Order Date" },
        { key: "total_amount", type: "number", label: "Total Amount" }
      ]
    };

    const vendorPerformanceSheet = {
      name: "Vendor Performance",
      collection: "Vendor Management",
      fields: [
        { key: "supplier_id", type: "string", label: "Supplier ID" },
        { key: "on_time_delivery", type: "number", label: "On-Time Delivery %" },
        { key: "quality_rating", type: "number", label: "Quality Rating" },
        { key: "cost_competitiveness", type: "number", label: "Cost Rating" }
      ]
    };
    ```
  </Tab>
</Tabs>


# Documents
Source: https://flatfile.com/docs/core-concepts/documents

Standalone webpages within Flatfile Spaces for guidance and dynamic content

Documents are standalone webpages for your Flatfile [Spaces](/core-concepts/spaces). They can be rendered from [Markdown syntax](https://www.markdownguide.org/basic-syntax/).

Often used for getting started guides, Documents become extremely powerful with dynamically generated content that stays updated as Events occur.

Flatfile also allows you to use HTML tags in your Markdown-formatted text. This is helpful if you prefer certain HTML tags rather than Markdown syntax. Links in documents (both Markdown and HTML) automatically open in a new tab to ensure users don't navigate away from the Flatfile interface.

<Frame>
  <img />
</Frame>

## Key Features

<Note>
  **A note on Documents:** While Documents themselves can be created and updated [dynamically](/core-concepts/documents#dynamic-content), the content inside of a document should be considered to be *static* - that is, you cannot use documents to host interactive elements or single-page webforms. For that sort of functionality, we recommend using [Actions](/core-concepts/actions) to trigger a [Listener](/core-concepts/listeners) to perform the desired functionality.
</Note>

### Markdown-Based Content

Documents support GitHub-flavored Markdown, allowing you to create rich, formatted content with headers, lists, code blocks, and more. You can also use HTML tags within your Markdown for additional formatting flexibility.

### Dynamic Content

Documents can be created and updated programmatically in response to Events, enabling dynamic content that reflects the current state of your Space or data processing workflow.

### Document Actions

Add interactive buttons to your Documents that trigger custom operations. [Actions](/core-concepts/actions) appear in the top right corner and can be configured with different modes, confirmations, and tooltips.

### Embedded Blocks

Documents support embedding interactive data blocks (Workbooks, Sheets, and Diffs) directly within the content. See the [Adding Blocks to Documents](#adding-blocks-to-documents) section for detailed implementation.

## Create a Document

You can create Documents upon Space creation using the [Space Configure Plugin](/plugins/space-configure), or dynamically in a [Listener](/core-concepts/listeners) using the API:

```javascript theme={null}
import api from "@flatfile/api";

export default function flatfileEventListener(listener) {
  listener.on("file:created", async ({ context: { spaceId, fileId } }) => {
    const fileName = (await api.files.get(fileId)).data.name;
    const bodyText =
      "# Welcome\n" +
      "### Say hello to your first customer Space in the new Flatfile!\n" +
      "Let's begin by first getting acquainted with what you're seeing in your Space initially.\n" +
      "---\n" +
      "Your uploaded file, ${fileName}, is located in the Files area.";

    const doc = await api.documents.create(spaceId, {
      title: "Getting Started",
      body: bodyText,
    });
  });
}
```

This Document will now appear in the sidebar of your Space. Learn how to [customize the guest sidebar](/guides/customize-guest-sidebar) for different user types.

In this example, we create a Document when a file is uploaded, but you can also create Documents in response to any other Event. [Read more](/reference/events) about the different Events you can respond to.

## Document Actions

Actions are optional and allow you to run custom operations in response to a user-triggered event from within a Document.

Define Actions on a Document using the `actions` parameter when a document is created:

```javascript theme={null}
import api from "@flatfile/api";

export default function flatfileEventListener(listener) {
  listener.on("file:created", async ({ context: { spaceId, fileId } }) => {
    const fileName = (await api.files.get(fileId)).data.name;
    const bodyText =
      "# Welcome\n" +
      "### Say hello to your first customer Space in the new Flatfile!\n" +
      "Let's begin by first getting acquainted with what you're seeing in your Space initially.\n" +
      "---\n" +
      "Your uploaded file, ${fileName}, is located in the Files area.";

    const doc = await api.documents.create(spaceId, {
      title: "Getting Started",
      body: bodyText,
      actions: [
        {
          label: "Submit",
          operation: "contacts:submit",
          description: "Would you like to submit the contact data?",
          tooltip: "Submit the contact data",
          mode: "foreground",
          primary: true,
          confirm: true,
        },
      ],
    });
  });
}
```

Then configure your listener to handle this Action, and define what should happen in response. Read more about Actions and how to handle them in our [Using Actions guide](/guides/using-actions).

Actions appear as buttons in the top right corner of your Document.

## Document treatments

Documents have an optional `treatments` parameter which takes an array of treatments for your Document. Treatments can be used to categorize your Document. Certain treatments will cause your Document to look or behave differently.

### Ephemeral documents

Giving your Document a treatment of `"ephemeral"` will cause the Document to appear as a full-screen takeover, and it will not appear in the sidebar of your Space like other Documents. You can use ephemeral Documents to create a more focused experience for your end users.

```javascript theme={null}
const ephemeralDoc = await api.documents.create(spaceId, {
  title: "Getting started",
  body: "# Welcome ...",
  treatments: ["ephemeral"],
});
```

Currently, `"ephemeral"` is the only treatment that will change the behavior of your Document.

## Adding Blocks to Documents

Blocks are dynamic, embedded entities that you can use to display data inside a Document. You can add a Block to a Document using the `<embed>` HTML entity in your markdown and specifying which Block type you want to show using the `type` attribute on the entity. Three Block types are currently supported: Embedded Workbook, Embedded Sheet, and Embedded Diff.

### Embedded Workbook

Use this Block to render an entire Workbook with all its Sheets inside a Document, providing users with tabbed navigation between sheets. You can embed a Workbook by passing a workbook ID and optional name. You can also control whether the embedded Workbook is expanded when the document loads and whether to show the header.

```javascript theme={null}
const doc = await api.documents.create(spaceId, {
  title: "Getting started",
  body:
    "# Welcome\n" +
    "\n" +
    "Here is an embedded Workbook:\n" +
    "\n" +
    "<embed type='embedded-workbook' name='Customer Data' defaultExpanded='true' showHeader='true' workbookId='your_workbook_id'>\n" +
    "\n" +
    "Here is another embedded Workbook without header:\n" +
    "\n" +
    "<embed type='embedded-workbook' showHeader='false' workbookId='us_wb_8e0z52gI'>",
});
```

**Properties:**

* `workbookId` (required): The ID of the workbook to embed
* `name` (optional): Display name for the embedded workbook
* `defaultExpanded` (optional): Whether the workbook is expanded when the document loads (defaults to false)
* `showHeader` (optional): Whether to show the workbook header (defaults to true). When false, the workbook is automatically expanded

### Embedded Sheet

Use this Block to render a Sheet along with all its data inside of a Document. You can embed a Sheet into your Document by passing a sheet ID, workbook ID, and name. You can also specify whether the embedded Sheet is expanded or collapsed when the document is loaded, and whether to show the header.

You can include as many embedded Sheets in your Document as you like, but end users will only be able to expand a maximum of 10 embedded Sheets at once.

```javascript theme={null}
const doc = await api.documents.create(spaceId, {
  title: "Getting started",
  body:
    "# Welcome\n" +
    "\n" +
    "Here is an embedded Sheet:\n" +
    "\n" +
    "<embed type='embedded-sheet' name='Contacts' defaultExpanded='true' showHeader='true' sheetId='your_sheet_id' workbookId='your_workbook_id'>\n" +
    "\n" +
    "Here is another embedded Sheet without header:\n" +
    "\n" +
    "<embed type='embedded-sheet' name='Countries' showHeader='false' sheetId='us_sh_TVW95224' workbookId='us_wb_8e0z52gI'>",
});
```

**Properties:**

* `sheetId` (required): The ID of the sheet to embed
* `workbookId` (required): The ID of the workbook containing the sheet
* `name` (optional): Display name for the embedded sheet
* `defaultExpanded` (optional): Whether the sheet is expanded when the document loads (defaults to false)
* `showHeader` (optional): Whether to show the sheet header (defaults to true). When false, the sheet is automatically expanded

### Embedded Diff

Use this Block to show a side-by-side comparison of the data in a Sheet now versus at a previous point in time as captured by a Snapshot. Pass a Sheet ID, Workbook ID, and Snapshot ID. You can optionally pass a `direction` attribute which specified whether the changes are displayed with the Snapshot as the end state (`sheet_to_snapshot`) or the Sheet as the end state (`snapshot_to_sheet`). The default value for direction is `sheet_to_snapshot`.

Use `direction="sheet_to_snapshot"` if you want to show changes that have been made since the time the Snapshot was taken, i.e. to review past changes. Use `direction="snapshot_to_sheet"` if to preview the changes that would occur if you were to revert your Sheet back to the state it was in when the Snapshot was taken.

```javascript theme={null}
const doc = await api.documents.create(spaceId, {
  title: 'Getting started',
  body:
    "# Welcome\n" +
    "\n" +
    "Here is an embedded Diff:\n" +
    "\n" +
    "<embed type='embedded-diff' sheetId='your_sheet_id' workbookId='your_workbook_id' snapshot_id='your_snapshot_id" direction="snapshot_to_sheet">"
  ,
})
```

## Markdown reference

Documents support Github-flavored Markdown. Check out [this guide](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) for a full reference on the entities you can use.


# Environments
Source: https://flatfile.com/docs/core-concepts/environments

Use Environments for testing and authentication

Environments are isolated entities and are intended to be a safe place to create and test different configurations. Environments serve as self-contained, secure domains where diverse configurations can be created and tested. By default, a development and a production environment are set up.

| isProd  | Name        | Description                                                                                  |
| ------- | ----------- | -------------------------------------------------------------------------------------------- |
| *false* | development | Use this default environment, and it's associated test API keys, as you build with Flatfile. |
| *true*  | production  | When you're ready to launch, create a new environment and swap out your keys.                |

The development environment does not count towards your paid credits.

## Listeners

[Listeners](/core-concepts/listeners) are functions you write that respond to Events by executing custom code. They enable all the powerful functionality in your Flatfile implementation: data transformations, validations, integrations, and workflows. Each Listener - whether run locally or deployed as an [Agent](/core-concepts/listeners#agents) – connects to a specific [Environment](/core-concepts/environments) using that Environment's ID and API key as environment variables. To switch between different Environments (or promote a development Environment to production), you simply update your environment variables with the corresponding Environment ID and API key.

By default, Listeners respond to Events from all [Apps](/core-concepts/apps) within an [Environment](/core-concepts/environments). You can use [Namespaces](/guides/namespaces-and-filters#namespaces) to partition your Listeners into isolated functions that only respond to Events from specific Apps.

## Creating an Environment

Your `publishableKey` and `secretKey` are specific to an environment therefore to create a new Environment, you'll have to use a personal access token.

1. Open Settings
2. Click to Personal Tokens
3. You can use the key pair in there to create an access token like:

```bash theme={null}
curl -X POST https://platform.flatfile.com/v1/auth -H 'Content-Type: application/json' -d '{"clientId":"1234-1234", "secret":"1234-1234"}'
```

4. The response will include an `accessToken`. You can present that as your **Bearer `token`** in place of the `secretKey`.

[Or click here to create an environment in the dashboard](https://platform.flatfile.com/dashboard)

## Guest Authentication

Environments support two types of guest authentication:

1. `magic_link`: This method dispatches an email to your guests, which includes a magic link to facilitate login.
2. `shared_link`: This method transforms the Space URL into a public one, typically used in conjunction with embedded Flatfile.

### Additional Info

Should the `guestAuthentication` be left unspecified, both `magic_link` and `shared_link` types are enabled by default.

It's important to note that `guestAuthentication` settings can be applied at both Environment and Space levels. However, in case of any conflicting settings, the authentication type set at the Space level will take precedence over the Environment level setting. This flexibility enables customization based on specific needs, ensuring the right balance of accessibility and security.

## Secret and publishable keys

All Accounts have two key types for each environment. Learn when to use each type of key:

| Type            | Id                    | Description                                                                                                             |
| --------------- | --------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| Secret key      | sk\_23ghsyuyshs7dcrty | **On the server-side:** Store this securely in your server-side code. Don't expose this key in an application.          |
| Publishable key | pk\_23ghsyuyshs7dcert | **On the client-side:** Can be publicly-accessible in your application's client-side code. Use when embedding Flatfile. |

The publishable key only has permissions to create a Space.


# Fields
Source: https://flatfile.com/docs/core-concepts/fields

Blueprint definitions that define the structure and validation rules for your data in Flatfile

## What are Fields?

A Field in Flatfile is like a column in a spreadsheet — it defines the data type, format, and any constraints for a single piece of data in your import.

Fields are defined inside a [Blueprint](/core-concepts/blueprints), which acts as the master schema for your import setup. The hierarchy works like this:

* A Blueprint defines the structure of one or more Workbooks.
* Each Workbook contains Sheets (like tabs in a spreadsheet).
* Each Sheet contains Fields (columns), which describe the individual data points to be collected.

When you configure a Field, you’re telling Flatfile what kind of data to expect in that column—whether it’s text, a number, a date, or another supported type. You can also apply [constraints](#field-constraints) like "required," "unique," or "computed" to control how data is handled during import.

Fields play a key role in ensuring data structure and quality from the moment users start importing.

## Basic Blueprint Structure

* A [Blueprint](/core-concepts/blueprints) defines the data structure for any number of [Spaces](/core-concepts/spaces)
* A [Space](/core-concepts/blueprints) may contain many [Workbooks](/core-concepts/workbooks) and many [Documents](/core-concepts/documents)
* A [Document](/core-concepts/documents) contains static documentation and may contain many [Document-level Actions](/guides/using-actions#document-actions)
* A [Workbook](/core-concepts/workbooks) may contain many [Sheets](/core-concepts/sheets) and many [Workbook-level Actions](/guides/using-actions#workbook-actions)
* A [Sheet](/core-concepts/sheets) may contain many [Fields](/core-concepts/fields) and many [Sheet-level Actions](/guides/using-actions#sheet-actions)
* A [Field](/core-concepts/fields) defines a single column of data, and may contain many [Field-level Actions](/guides/using-actions#field-actions)

## Basic Field Configuration

The following examples demonstrate the configuration of isolated fields, which are intended to be used in the context of a [Blueprint](/core-concepts/blueprints) configuration under a [Sheet](/core-concepts/sheets) definition.

### Simple Field Definition

This example configures a single `string` Field that is `required` and another a `number` Field with a `decimalPlaces` config.

```javascript theme={null}
// Basic string field
const nameField = {
  key: "firstName",
  type: "string",
  label: "First Name",
  description: "The customer's first name",
  required: true,
};

// Number field with config
const ageField = {
  key: "age",
  type: "number",
  label: "Age",
  description: "Customer age in years",
  config: {
    decimalPlaces: 2,
  },
};
```

### Field with Multiple Options

This example configures a single `enum` field with a `status` key, `Customer Status` label, and `Current status of the customer` description. It also defines four options for the field: `active`, `inactive`, `pending`, and `suspended`.

```javascript theme={null}
const statusField = {
  key: "status",
  type: "enum",
  label: "Customer Status",
  description: "Current status of the customer",
  options: [
    { value: "active", label: "Active" },
    { value: "inactive", label: "Inactive" },
    { value: "pending", label: "Pending" },
    { value: "suspended", label: "Suspended" },
  ],
};
```

## Field Types

Flatfile supports 9 field types for defining data structure:

**Basic Types:**

* `string` - Basic text data
* `number` - Integer or floating point numbers
* `boolean` - True/false values
* `date` - GMT date values in YYYY-MM-DD format

**Single Selection:**

* `enum` - Single selection from predefined options

**Multiple Value Types:**

* `string-list` - Array of string values
* `enum-list` - Multiple selections from predefined options

**Reference Types:**

* `reference` - Single reference to another sheet
* `reference-list` - Multiple references to another sheet

### `string`

A property that should be stored and read as a basic string.

```javascript theme={null}
{
  "key": "productCode",
  "label": "Product Code",
  "type": "string",
  "appearance": {
    "size": "m"
  }
}
```

**Note:** String fields don't have type-specific config options, but support appearance settings.

### `string-list`

Stores an array of string values. Useful for fields that contain multiple text entries.

```javascript theme={null}
{
  "key": "tags",
  "label": "Product Tags",
  "type": "string-list"
}
```

### `number`

A property that should be stored and read as either an integer or floating point number.

**config.decimalPlaces**
The number of decimal places to preserve accuracy to.

```javascript theme={null}
{
  "key": "price",
  "label": "Retail Price",
  "type": "number",
  "config": {
    "decimalPlaces": 2
  }
}
```

### `enum`

Defines an enumerated list of options for the user to select from (single selection). The maximum number of options for this list is `100`. For multiple selections, use [`enum-list`](#enum-list).

**config.allowCustom**
Allow users to create new options for this field. When enabled, users will be able to import their custom value to the review table, but it will not be considered a valid enum option.

**config.sortBy**
The field to sort the options by (`label`, `value`, `ordinal`).

**config.options**
An array of valid options the user can select from:

```json theme={null}
{
  "key": "status",
  "label": "Status",
  "type": "enum",
  "config": {
    "options": [
      {
        "value": "active",
        "label": "Active"
      },
      {
        "value": "inactive",
        "label": "Disabled"
      }
    ]
  }
}
```

### `enum-list`

Allows multiple selections from a predefined list of options. Values are stored as an array. Use this instead of `enum` when users need to select multiple options.

**config.allowCustom**
Allow users to create new options for this field. When enabled, users will be able to import their custom value to the review table, but it will not be considered a valid enum option.

**config.sortBy**
Sort options by: `label`, `value`, or `ordinal`.

**config.options**
Array of option objects:

```json theme={null}
{
  "key": "categories",
  "label": "Product Categories",
  "type": "enum-list",
  "config": {
    "allowCustom": false,
    "sortBy": "label",
    "options": [
      {
        "value": "electronics",
        "label": "Electronics"
      },
      {
        "value": "clothing",
        "label": "Clothing"
      },
      {
        "value": "books",
        "label": "Books"
      }
    ]
  }
}
```

### `boolean`

A `true` or `false` value type. Usually displayed as a checkbox.

**config.allowIndeterminate**
Allow a neither true or false state to be stored as `null`.

```json theme={null}
{
  "key": "is_active",
  "label": "Active",
  "type": "boolean",
  "config": {
    "allowIndeterminate": true
  }
}
```

### `date`

Store a field as a GMT date. Data hooks must convert this value into a `YYYY-MM-DD` format in order for it to be considered a valid value.

```json theme={null}
{
  "key": "start_date",
  "label": "Start Date",
  "type": "date"
}
```

### `reference`

Defines a singular one-to-one reference to a field another sheet.

**config.ref**
The sheet slug of the referenced field. Must be in the same workbook.

**config.key**
The key of the property to use as the reference key.

**config.filter**
Optional filter to narrow the set of records in the reference sheet used as valid values. When provided, only records where the `refField` value matches the current record's `recordField` value will be available as options.

```json theme={null}
{
  "key": "author",
  "type": "reference",
  "label": "Authors",
  "config": {
    "ref": "authors",
    "key": "name"
  }
}
```

#### Reference Field Filtering

Reference fields can be filtered to show only relevant options based on the value of another field in the same record. This enables cascading dropdowns and dependent field relationships.

<Note>
  **Dynamic Enums**

  This feature, along with the `ENUM_REFERENCE` [sheet treatment](/core-concepts/sheets#reference-sheets), may be collectively referred to as **Dynamic Enums**. By combining these two features, you can create a drop-down list for any cell in your sheet that's dynamically controlled by the value of another field in the same record – and to the end-user, it will just work like a dynamically-configured `enum` field.
</Note>

**Filter Configuration**

The `filter` property accepts a `ReferenceFilter` object with two required properties:

| Property      | Type     | Description                                            |
| ------------- | -------- | ------------------------------------------------------ |
| `refField`    | `string` | The field key in the referenced sheet to filter with   |
| `recordField` | `string` | The field key in the current record used for filtering |

**How Filtering Works**

When a filter is applied:

1. The system looks at the value in the `recordField` of the current record
2. It then filters the referenced sheet to only show records where `refField` matches that value
3. Only the filtered records become available as options in the reference field

**Example: Country and State Cascading Dropdown**

Consider a scenario where you want state options to be filtered based on the selected country.

<Info>
  This example also demonstrates the use of the `ENUM_REFERENCE` [sheet treatment](/core-concepts/sheets#reference-sheets) to hide the reference sheet from the UI. You may wish to disable this treatment for testing purposes.
</Info>

```json theme={null}
{
  "sheets": [
    {
      "name": "Reference Data",
      "slug": "ref-data",
      "treatments": ["ENUM_REFERENCE"],
      "fields": [
        {
          "key": "country-name",
          "label": "Country",
          "type": "string"
        },
        {
          "key": "state-name",
          "label": "State/Province",
          "type": "string"
        }
      ]
    },
    {
      "name": "Addresses",
      "slug": "addresses",
      "fields": [
        {
          "key": "country",
          "label": "Country",
          "type": "reference",
          "config": {
            "ref": "ref-data",
            "key": "country-name"
          }
        },
        {
          "key": "state",
          "label": "State/Province",
          "type": "reference",
          "config": {
            "ref": "ref-data",
            "key": "state-name",
            "filter": {
              "refField": "country-name",
              "recordField": "country"
            }
          }
        }
      ]
    }
  ]
}
```

**Reference Data Sheet**:

| country-name | state-name |
| ------------ | ---------- |
| USA          | California |
| USA          | New York   |
| USA          | Texas      |
| Canada       | Ontario    |
| Canada       | Quebec     |

**Behavior**

* When **USA** is selected in the country field, the state dropdown will only show **California**, **New York**, and **Texas**
* When **Canada** is selected in the country field, the state dropdown will only show **Ontario** and **Quebec**

The following diagram illustrates how reference field filtering works:

```mermaid theme={null}
graph TD
    A[User selects **USA**] --> B[System filters references]
    B --> C[Show only States where **country-name** = **USA**]
    C --> D[**California**, **New York**, and **Texas** are available]

    E[User selects **Canada**] --> F[System filters references]
    F --> G[Show only States where **country-name** = **Canada**]
    G --> H[**Ontario** and **Quebec** are available]

    style A stroke:#4CD95E, stroke-width:2px
    style E stroke:#4CD95E, stroke-width:2px
    style D stroke:#D9804E, stroke-width:2px
    style H stroke:#D9804E, stroke-width:2px
```

And this is how it looks in the UI:

<Frame>
  <img alt="Reference Field Filtering" />
</Frame>

<Frame>
  <img alt="Reference Field Filtering" />
</Frame>

<Frame>
  <img alt="Reference Field Filtering" />
</Frame>

### `reference-list`

Defines multiple references to records in another sheet within the same workbook.

**config.ref**
The sheet slug of the referenced field. Must be in the same workbook.

**config.key**
The key of the property to use as the reference key.

**config.filter**
Optional filter to narrow the set of records in the reference sheet used as valid values.

```json theme={null}
{
  "key": "authors",
  "type": "reference-list",
  "label": "Book Authors",
  "config": {
    "ref": "authors",
    "key": "name"
  }
}
```

#### Reference List Filtering

The `filter` property on `reference-list` fields works identically to `reference` [field filters](#reference-field-filtering), but allows for multiple selections.

```json theme={null}
{
  "key": "categories",
  "label": "Product Categories",
  "type": "reference-list",
  "config": {
    "ref": "category-data",
    "key": "category-name",
    "filter": {
      "refField": "department",
      "recordField": "product-department"
    }
  }
}
```

**Multi-Level Cascading Example**

You can create complex, multi-level cascading dropdowns by chaining filtered reference fields together. This example shows a product taxonomy where department selection filters available categories, and category selection filters available subcategories:

```json theme={null}
{
  "sheets": [
    {
      "name": "Product Taxonomy",
      "slug": "taxonomy",
      "fields": [
        { "key": "department", "label": "Department", "type": "string" },
        { "key": "category", "label": "Category", "type": "string" },
        { "key": "subcategory", "label": "Subcategory", "type": "string" }
      ]
    },
    {
      "name": "Products",
      "slug": "products",
      "fields": [
        {
          "key": "department",
          "label": "Department",
          "type": "reference",
          "config": { "ref": "taxonomy", "key": "department" }
        },
        {
          "key": "category",
          "label": "Category",
          "type": "reference",
          "config": {
            "ref": "taxonomy",
            "key": "category",
            "filter": { "refField": "department", "recordField": "department" }
          }
        },
        {
          "key": "subcategory",
          "label": "Subcategory",
          "type": "reference-list",
          "config": {
            "ref": "taxonomy",
            "key": "subcategory",
            "filter": { "refField": "category", "recordField": "category" }
          }
        }
      ]
    }
  ]
}
```

**Product Taxonomy Sheet**:

| department  | category    | subcategory   |
| ----------- | ----------- | ------------- |
| Electronics | Computers   | Laptops       |
| Electronics | Computers   | Desktops      |
| Electronics | Computers   | Tablets       |
| Electronics | Audio       | Headphones    |
| Electronics | Audio       | Speakers      |
| Clothing    | Men's       | Shirts        |
| Clothing    | Men's       | Pants         |
| Clothing    | Women's     | Dresses       |
| Clothing    | Women's     | Shoes         |
| Books       | Fiction     | Novels        |
| Books       | Fiction     | Short Stories |
| Books       | Non-Fiction | Biography     |
| Books       | Non-Fiction | History       |

**Behavior**

This creates a three-level cascade: Department → Category → Subcategory, where users can select multiple subcategories from the filtered options.

* When **Electronics** is selected in the department field, the category dropdown will only show **Computers** and **Audio**
* When **Computers** is selected, the subcategory dropdown will show **Laptops**, **Desktops**, and **Tablets**. Users may then select *multiple* subcategories from the filtered options
* When **Clothing** is selected, the category dropdown will only show **Men's** and **Women's**
* When **Men's** is selected, the subcategory dropdown will show **Shirts** and **Pants**. Users may then select *multiple* subcategories from the filtered options

## Field Constraints

Field constraints are system-level validation rules that enforce data integrity and business logic on data in individual fields.

### required

Ensures that a field must have a non-null value. Empty cells are considered null values and will fail this constraint.

```json theme={null}
{
  "key": "email",
  "type": "string",
  "label": "Email Address",
  "constraints": [{ "type": "required" }]
}
```

By default, if a required constraint fails, an error will be added to the field with the message "\<field label> is required".

You can override the message and/or the error level of the message by supplying a `config` object with the constraint.

For example:

```json theme={null}
{
  "key": "email",
  "type": "string",
  "label": "Email Address",
  "constraints": [{
    "type": "required",
    "config": {
      "message": "This record is missing an email address",
      "level": "warn"
    }
  }]
}
```

### unique

Ensures that field values appear only once across all records in the sheet. Note that null values can appear multiple times as they don't count toward uniqueness validation.

```json theme={null}
{
  "key": "employeeId",
  "type": "string",
  "label": "Employee ID",
  "constraints": [{ "type": "unique" }]
}
```

By default, if a required constraint fails, an error will be added to the field with the message "Value is not unique".

You can override the message and/or the error level of the message by supplying a `config` object with the constraint -- see the example with the required constraint above.

### computed

Marks a field as computed, hiding it from the mapping process. Users will not be able to map imported data to fields with this constraint.

```json theme={null}
{
  "key": "calculatedField",
  "type": "string",
  "label": "Calculated Value",
  "constraints": [{ "type": "computed" }]
}
```

<Info>
  Sheet-level constraints that apply to multiple fields are covered in [Sheet Constraints](/core-concepts/sheets#sheet-constraints).
</Info>

### Field-level access

**`readonly`**
On a field level you can restrict a user's interaction with the data to `readonly`. This feature is useful if you're inviting others to view uploaded data, but do not want to allow them to edit that field.

```json theme={null}
{
  "fields": [
    {
      "key": "salary",
      "type": "number",
      "readonly": true
    }
  ]
}
```

## Field Options

Configurable properties for a Field that define its structure and behavior:

| Option          | Type    | Required | Default                                                                                                     | Description                                                                                                                                            |
| --------------- | ------- | -------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **key**         | string  | ✓        |                                                                                                             | The system name of this field. Primarily informs JSON and egress structures                                                                            |
| **type**        | string  | ✓        |                                                                                                             | One of `string`, `number`, `boolean`, `date`, `enum`, `reference`, `string-list`, `enum-list`, `reference-list`. Defines the handling of this property |
| **label**       | string  |          | key                                                                                                         | A user-facing descriptive label designed to be displayed in the UI such as a table header                                                              |
| **description** | string  |          |                                                                                                             | A long form description of the property intended to be displayed to an end user (supports Markdown)                                                    |
| **constraints** | array   |          | \[]                                                                                                         | An array of system level validation rules (max 10). [Learn more about field constraints](#field-constraints)                                           |
| **config**      | object  |          | Configuration relevant to the type of column. See type documentation below                                  |                                                                                                                                                        |
| **readonly**    | boolean |          | false                                                                                                       | Prevents user input on this field                                                                                                                      |
| **appearance**  | object  |          | UI appearance settings. Currently supports `size` property with values: `"xs"`, `"s"`, `"m"`, `"l"`, `"xl"` |                                                                                                                                                        |
| **actions**     | array   |          | \[]                                                                                                         | User actions available for this field. See [Field Actions](/guides/using-actions#field-actions) for detailed configuration                             |

\| **alternativeNames** | array | | \[] | Alternative field names for mapping assistance |
\| **metadata** | object | |  | Arbitrary object of values to pass through to hooks and egress |


# Jobs
Source: https://flatfile.com/docs/core-concepts/jobs

Discrete tasks executed asynchronously in response to events

In Flatfile, a Job represents a large unit of work performed asynchronously on a resource such as a file, [Workbook](/core-concepts/workbooks), or [Sheet](/core-concepts/sheets). The Jobs workflow provides visibility into the status and progress of your Jobs, allowing you to monitor and troubleshoot the data processing pipeline. For handling large datasets, see our [multi-part jobs guide](/guides/multi-part-jobs).

<Note>
  In these examples, we'll show the full Job Listener lifecycle implementation, complete with `ack` to acknowledge the job, `update` to update the job's progress, and `complete` or `fail` to complete or fail the job.

  To make this simpler, we provide a plugin called [Job Handler](/plugins/job-handler) that handles the job lifecycle for you. This plugin works by listening to the `job:ready` event and executing the handler callback. There is also an optional `tick` function which allows you to update the Job's progress.

  For example, this listener implementation contains all the code necessary to handle a job:

  ```typescript theme={null}
  listener.use(jobHandler("workbook:export", async ({ context: { fileId, jobId } }, tick) => {
    const file = await api.files.get(fileId);
    tick(10, "Getting started.");
    // export logic here
    tick(90, "Exported.");
    return {
      outcome: {
        message: "Exporting file contents is complete.",
      }
    }
  }));
  ```
</Note>

## Job Lifecycle

### Lifecycle Events

Jobs fire the following Events during their lifecycle. In chronological order, the Job Events are:

| Event                           | Description                                                                                                                                                                        |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `job:created`                   | Fires when a Job is created, but before it does anything.                                                                                                                          |
| `job:ready`                     | Fires when a Job is ready to move into the execution stage, but before it does anything.                                                                                           |
| `job:updated`                   | Fires when there is an update to a Job while it is executing.                                                                                                                      |
| `job:completed` OR `job:failed` | `job:completed` fires when a Job is completed successfully, `job:failed` fires if a Job is completed but fails. One of these events will fire upon Job completion, but never both. |
| `job:outcome-acknowledged`      | Fires when a user acknowledges the completion of a Job through a UI popup.                                                                                                         |

You can listen on any of these events in your [Listener](/core-concepts/listeners), but the most common event to listen for is `job:ready`.

## Types of Jobs

Jobs can be triggered in a number of ways, most commonly in response to user activity. Jobs are then managed via [Listeners](/core-concepts/listeners), which receive [Events](/core-concepts/listeners#events) published by Flatfile in response to activity.

There are three types of Jobs on the Flatfile Platform:

### Action-Based Jobs

Actions are developer-defined operations that can be mounted on a number of domains (including Sheets, Workbooks, Documents, and Files). Mounting an Action means attaching a custom operation to that domain. That operation can then be triggered by a user event (clicking a button or selecting a menu item).

<Note>
  When an Action is triggered a `job:ready` Event for a Job named `[domain]:[operation]` is published. Your [Listener](/core-concepts/listeners) can then be configured to respond to that Action via its Event.
</Note>

To run an Action based job, two configurations are necessary.

First, create an [Action](/core-concepts/actions) on a domain. Here's an example of a Workbook containing an Action:

```typescript theme={null}
api.workbook.create({
  name: "October Report Workbook",
  actions: [
    {
      label: "Export Data",
      description: "Send data to destination system",
      operation: "export",
      type: "file",
    },
  ],
});
```

Then, create a [Listener](/core-concepts/listeners) to respond to the Action. This listener should listen for the `job:ready` event, filtered by the `domain:operation` Job - where, in this case, `workbook` is the domain, `export` is the operation.

```typescript theme={null}
listener.on(
  "job:ready",
  { job: "workbook:export" },
  async ({ context: { jobId } }) => {
    try {
      await api.jobs.ack(jobId, {
        info: "Starting submit job...",
        progress: 10,
      });

      // Custom code here

      await api.jobs.complete(jobId, {
        outcome: {
          message: "Submit Job was completed successfully.",
        },
      });
    } catch (error) {
      await api.jobs.fail(jobId, {
        outcome: {
          message: "This Job failed.",
        },
      });
    }
  }
);
```

### Custom Jobs

Another trigger option is to create a Custom Job via SDK/API. In the SDK, Jobs are created by calling the `api.jobs.create()` method.

Creating a custom Job in your Listener enables any Event to trigger a Job.

Here's an example of creating a custom Job in a Listener:

```typescript theme={null}
listener.on(
  "commit:created",
  { sheet: "contacts" },
  async ({ context: { workbookId, sheetId } }) => {

    const { data } = await api.jobs.create({
      type: "workbook",
      operation: "myCustomOperation",
      trigger: "immediate",
      source: workbookId
    });
    
  }
);
```

Note that the trigger for this Listener is set to immediate, which means that the Job will be created and executed immediately upon the Event firing.

Therefore, we should have our Listener ready to respond to this Job:

```typescript theme={null}
listener.on(
  "job:ready",
  { job: "workbook:myCustomOperation" },
  async ({ context: { jobId, workbookId } }) => {
    try {
      await api.jobs.ack(jobId, {
        info: "Starting my custom operation.",
        progress: 10,
      });

      // Custom code here.

      await api.jobs.complete(jobId, {
        outcome: {
          message: "Successfully completed my custom operation.",
        },
      });
    } catch {
      await api.jobs.fail(jobId, {
        outcome: {
          message: "Custom operation failed.",
        },
      });
    }
  }
);
```

### System Jobs

Internally, Flatfile uses Jobs to power many of the features of the Flatfile Platform, such as extraction, record mutation, and AI Assist. Here are some examples of Jobs that the Flatfile Platform creates and manages on your behalf:

| Job Name        | Description                                                   |
| --------------- | ------------------------------------------------------------- |
| `Extract`       | Extracts data from the specified source.                      |
| `Map`           | Maps data from its ingress format to Blueprint fields.        |
| `DeleteRecords` | Deletes records from a dataset based on specified criteria.   |
| `Export`        | Exports data to a specified format or destination.            |
| `MutateRecords` | Alters records in a dataset according to defined rules.       |
| `Configure`     | Sets up or modifies the configuration of a Space.             |
| `AiAssist`      | Utilizes AI to assist with tasks such as data categorization. |
| `FindReplace`   | Searches for specific values and replaces them.               |

## Job Parameters

### Required Parameters

When creating a job, the following parameters are required:

* **type** (string) - Workbook, File, Sheet, Space
* **operation** (string) - `export`, `extract`, `map`, `delete`, etc
* **source** (string) - The id of the data source (FileId, WorkbookId, or SheetId)

### Optional Parameters

* **trigger** (string) - `manual` or `immediate`
* **destination** (string) - The id of the data target (if any)
* **status** (string) - `created`, `planning`, `scheduled`, `ready`, `executing`, `complete`, `failed`, `cancelled`
* **progress** (number) - A numerical or percentage value indicating the completion status of the Job
* **estimatedCompletionAt** (date) - An estimated completion time. The UI will display the estimated processing time in the foreground Job overlay
* **info** (string) - Additional information regarding the Job's current status
* **managed** (string) - Indicates whether the Job is managed by the Flatfile platform or not
* **mode** (string) - `foreground`, `background`, `toolbarBlocking`
* **metadata** (object) - Additional metadata for the Job. You can store any additional information here, such as the IDs of Documents or Sheets created during the execution of Job

Please see our [API Reference](https://reference.flatfile.com/api-reference/jobs/) for details on all possible values.

## Working with Jobs

Jobs can be managed via SDK/API. Commonly, Jobs are acknowledged, progressed, and then completed or failed. Here's a look at those steps.

### Acknowledging Jobs

First, acknowledge a Job. This will update the Job's status to `executing`.

```typescript theme={null}
await api.jobs.ack(jobId, {
  info: "Starting submit job...",
  progress: 10,
});
```

### Updating Job Progress

Once a Job is acknowledged, you can begin running your custom operation. Jobs were designed to handle large processing loads, but you can easily update your user by updating the Job with a progress value.

```typescript theme={null}
await api.jobs.update(jobId, {
  progress: 50,
  estimatedCompletionAt: new Date("Tue Aug 23 2023 16:19:42 GMT-0700"),
});
```

`Progress` is a numerical or percentage value indicating the completion status of the work. You may also provide an `estimatedCompletionAt` value which will display your estimate of the remaining processing time in the foreground Job overlay. Additionally, the Jobs Panel will share visibility into the estimated remaining time for acknowledged jobs.

<Frame>
  <img />
</Frame>

### Completing Jobs

Once a job is complete, you can display an alert to the end user using `outcome`.

```typescript theme={null}
await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was completed successfully. ${myData.length} records were processed.`,
    acknowledge: true,
  },
});
```

### Job Outcomes

You can enhance job completion with various outcome options to guide users to their next action:

<Frame>
  <img />
</Frame>

#### Internal Navigation

Add a button to the dialog that will redirect the user somewhere within a Space using next > Id.

In this code below, we will create a button that says "See all downloads" with this path: space/us\_sp\_1234/files?mode=export

```typescript theme={null}
await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was completed successfully. ${myData.length} records were processed.`,
    acknowledge: true,
    next: {
      type: "id",
      id: "dev_sp_1234",
      path: "files",
      query: "mode=export",
      label: "See all downloads",
    },
  },
});
```

#### External Links

Add a button to the dialog that will redirect the user to an external link using next > Url.

In this code below, we will create a button that says "Go to Google". It will open in a new tab.

```typescript theme={null}
await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was completed successfully. ${myData.length} records were processed.`,
    acknowledge: true,
    next: {
      type: "url",
      url: "http://www.google.com",
      label: "Go to Google",
    },
  },
});
```

#### File Downloads

Add a button to the dialog that will redirect the user to an external link using next > Url.

In this code below, we will create a button that says "Download this file".

```typescript theme={null}
await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was completed successfully. ${myData.length} records were processed.`,
    acknowledge: true,
    next: {
      type: "download",
      fileName: "DownloadedFromFlatfile.csv",
      url: "source_of_file.csv",
      label: "Download this file",
    },
  },
});
```

#### Multiple File Downloads

Download files hosted on Flatfile by using next > files.

In this code below, we will create a button that says "Download files".

```typescript theme={null}
await api.jobs.complete(jobId, {
  outcome: {
    message: `The files should download automatically`,
    next: {
      type: "files",
      label: "Download files",
      files: [{fileId: "us_fl_123"}],
    },
  },
});
```

#### Snapshots

Add a button to the dialog that will redirect the user a particular snapshot using next > snapshot.

In this code below, we will create a button that says "Go to Snapshot".

```typescript theme={null}
await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was completed successfully.`,
    acknowledge: true,
    next: {
      type: "snapshot",
      label: "Go to Snapshot",
      snapshotId: snapshot.id,
      sheetId: sheet.id
    },
  },
});
```

#### View Management

Dynamically update visible columns in a sheet using next > view.

In this code below, we will create a button that says "Hide columns":

```typescript theme={null}
await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was completed on sheet ${sheet.name}`,
    acknowledge: true,
    next: {
      type: "view",
      label: "Hide columns",
      sheetId: sheet.id,
      hiddenColumns: ['age', 'phone', 'middleName'] // field keys to hide
    },
  },
});
```

#### Retry Actions

Often in the event of a failure, you may want to add a button to the dialog that will retry the Job using next > Retry.

Provide retry functionality for failed jobs:

```typescript theme={null}
await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was not completed successfully. No records were processed.`,
    acknowledge: true,
    next: {
      type: "retry",
      label: "Try again",
    },
  },
});
```

### Failing Jobs

When a job encounters an error, use the fail method:

```typescript theme={null}
await api.jobs.fail(jobId, {
  outcome: {
    message: "This Job failed due to an error.",
    acknowledge: true,
  },
});
```


# Events and Listeners
Source: https://flatfile.com/docs/core-concepts/listeners

The anatomy of Events and Listeners - core components of the Flatfile Platform

## Events

Flatfile publishes an [Event](/core-concepts/listeners#events) whenever something happens - editing a Record, uploading a file, clicking an Action button. Events carry structured information about what occurred, including the context and any relevant data.

Each Event includes a domain, topic, context, and optional payload:

| Component          | Description                                       | Example                                                 |
| ------------------ | ------------------------------------------------- | ------------------------------------------------------- |
| Domain             | the jurisdiction of the Event                     | Record, Sheet, Workbook, Space, file, Job, Agent        |
| Topic              | a combination of a domain and an action           | Workbook:created Workbook:updated Workbook:deleted      |
| Context            | detailed information about context of the Event   | `{ spaceId: "us_sp_1234", fileId: "us_fl_1234", ... }`  |
| Payload (optional) | detailed information about execution of the Event | `{ status: "complete", workbookId: "us_wb_1234", ... }` |

For a complete list of Events published by Flatfile, see [Events Reference](/reference/events).

## Listeners

Listeners are functions you write that respond to Events by executing custom code. They enable all the powerful functionality in your Flatfile implementation: data transformations, validations, integrations, and workflows. Listeners define how your [Spaces](/core-concepts/spaces) behave and what happens when users interact with your data.

Listeners can run locally during development or be deployed as [Agents](/core-concepts/listeners#agents) to Flatfile's cloud. Each Listener connects to a specific [Environment](/core-concepts/environments) using that Environment's API keys as environment variables. To switch between different Environments (or promote a development Environment to production), you simply update your environment variables with the corresponding API keys.

By default, Listeners respond to Events from all [Apps](/core-concepts/apps) within an [Environment](/core-concepts/environments). You can use [Namespaces](/guides/namespaces-and-filters#namespaces) to partition your Listeners into isolated functions that only respond to Events from specific Apps.

## How Listeners Work

A Listener receives Events as they occur and executes a callback function in response. For example, when a file is uploaded, a Listener can mount an [Action](/core-concepts/actions) buton to the file for performing data transformations, or when a [Record](/core-concepts/records) is created, a Listener can validate the data.

The callback function can be as simple as logging the Event or as advanced as integrating with external systems. Here is an example that logs the topic of any incoming Event:

<Warning>
  **Note:** The `**` wildcard is a catch-all that will match all Events. This is useful for testing, but should be avoided in production as it can lead to performance issues.
</Warning>

<CodeGroup>
  ```javascript JavaScript theme={null}
  export default function (listener) {
    listener.on("**", (event) => {
      console.log(`Received event: ${event.topic}`);
    });
  }
  ```

  For a more concrete and practical example, follow along with our [Coding Tutorial](/coding-tutorial).

  ```typescript TypeScript theme={null}
  import { FlatfileListener, FlatfileEvent } from '@flatfile/listener';

  export default function (listener: FlatfileListener) {
    listener.on("**", (event: FlatfileEvent) => {
      console.log(`Received event: ${event.topic}`);
    });
  }
  ```
</CodeGroup>

## Listener Methods

Flatfile provides two primary methods for configuring listener behavior:

### `listener.on()`

The `listener.on()` method is the fundamental pattern for responding to specific events. This method allows you to provide a callback function that will be executed when the specified event occurs.

For a complete list of Events published by Flatfile, see [Events Reference](/reference/events).

**Type Signature:**

```typescript theme={null}
listener.on(
  eventPattern: string, 
  callback: (event: FlatfileEvent) => void | Promise<void>
): void

listener.on(
  eventPattern: string, 
  filter: object, 
  callback: (event: FlatfileEvent) => void | Promise<void>
): void
```

**Examples:**

<CodeGroup>
  ```javascript JavaScript theme={null}
  // Basic Event handling
  listener.on('file:created', async (event) => {
    console.log(`New file uploaded: ${event.context.fileId}`);
  });

  // Event handling with filters
  listener.on('commit:created', { sheet: 'contacts' }, async (event) => {
    console.log('Processing contact data commit...');
    // Process contact data
  });

  // Job handling
  listener.on('job:ready', { job: 'space:configure' }, async (event) => {
    console.log('Configuring Space...');
    // Configure Space
  });
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileEvent } from '@flatfile/listener';

  // Basic Event handling with proper typing
  listener.on('file:created', async (event: FlatfileEvent) => {
    console.log(`New file uploaded: ${event.context.fileId}`);
  });

  // Event handling with filters
  listener.on('commit:created', { sheet: 'contacts' }, async (event: FlatfileEvent) => {
    console.log('Processing contact data commit...');
    // Process contact data
  });
  ```
</CodeGroup>

### `listener.use()`

The `listener.use()` method is for building and distributing listener functions in a modular manner. It allows you to send your own Listener functions as a callback to the method, each of which receives a `FlatfileListener` instance as their argument.

Inside that callback function, you can use `listener.on()` to register Event handlers or introduce various [Plugins](/core-concepts/plugins) to your listener. The result can be an `index` file with with little more than `listener.use()` calls, each of which distributes your listener function to other places in your codebase:

<CodeGroup>
  ```javascript JavaScript theme={null}
  // import statements

  export default function (listener) {
    // Distribute listener functions to different modules
    listener.use(validateCustomerData);
    listener.use(processFileUploads);
    listener.use(handleDataTransforms);
    listener.use(setupIntegrations);
  }
  ```

  ```typescript TypeScript theme={null}
  // import statements

  export default function (listener: FlatfileListener): void {
    // Distribute listener functions to different modules
    listener.use(validateCustomerData);
    listener.use(processFileUploads);
    listener.use(handleDataTransforms);
    listener.use(setupIntegrations);
  }
  ```
</CodeGroup>

This works particularly well for complex listener functions that need to be broken down into smaller, more manageable pieces.

It also works well when combined with [Namespaces](/guides/namespaces-and-filters) to create isolated listener functions that only respond to Events from a specific namespace.

You'll find this pattern often when using [Plugins](/core-concepts/plugins) to extend the functionality of your listener.

**Type Signature:**

```typescript theme={null}
listener.use(configFunction: (listener: FlatfileListener) => void): void
```

**Examples:**

<CodeGroup>
  ```javascript JavaScript theme={null}
  // Custom configuration function
  function validateCustomerData(listener) {
    listener.on('commit:created', { sheet: 'customers' }, async (event) => {
      // Validation logic here
      console.log('Validating customer data...');
    });
    
    listener.on('record:updated', { sheet: 'customers' }, async (event) => {
      // Update validation logic here
      console.log('Revalidating updated customer data...');
    });
  }

  // Another configuration function
  function setupDataProcessing(listener) {
    listener.on('file:created', async (event) => {
      console.log('Processing uploaded file...');
    });
  }

  export default function (listener) {
    listener.use(validateCustomerData);
    listener.use(setupDataProcessing);
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener, FlatfileEvent } from '@flatfile/listener';

  // Custom configuration function with proper typing
  function validateCustomerData(listener: FlatfileListener): void {
    listener.on('commit:created', { sheet: 'customers' }, async (event: FlatfileEvent) => {
      // Validation logic here
      console.log('Validating customer data...');
    });
    
    listener.on('record:updated', { sheet: 'customers' }, async (event: FlatfileEvent) => {
      // Update validation logic here
      console.log('Revalidating updated customer data...');
    });
  }

  // Another configuration function
  function setupDataProcessing(listener: FlatfileListener): void {
    listener.on('file:created', async (event: FlatfileEvent) => {
      console.log('Processing uploaded file...');
    });
  }

  export default function (listener: FlatfileListener): void {
    listener.use(validateCustomerData);
    listener.use(setupDataProcessing);
  }
  ```
</CodeGroup>

### When to Use Each Method

| Method           | Use For                                                            |
| ---------------- | ------------------------------------------------------------------ |
| `listener.use()` | Creating reusable functions, organizing complex logic into modules |
| `listener.on()`  | Responding to specific Events, implementing direct Event handlers  |

## Event Routing with Namespaces & Filters

There are two more Listener methods to note, specifically intended for routing events to specific Listener functions: `listener.namespace()` and `listener.filter()`.

**Namespaces** provide architectural boundaries for organizing different parts of your application at the App, Space, and Workbook level:

* `space:customer-portal` - Events from Spaces in the "customer-portal" App
* `workbook:staging` - Events from Workbooks with "staging" namespace

**Event filters** enable granular event targeting based on property values, event types, and conditions:

* `{ sheet: 'contacts' }` - Events from the "contacts" Sheet
* `{ domain: 'job', 'payload.job': 'space:configure' }` - Space configuration Jobs

For comprehensive examples, patterns, and detailed configuration options, see our [Namespaces and Filters guide](/guides/namespaces-and-filters).

**Usage Examples:**

<CodeGroup>
  ```javascript JavaScript theme={null}
  export default function (listener) {
    // Use namespaces for architectural organization
    listener.namespace('space:customer-portal', (customerListener) => {
      customerListener.on('commit:created', async (event) => {
        console.log('Processing customer portal data...');
      });
    });

    // Use filters for granular targeting
    listener.filter({ sheet: 'contacts' }, (contactsListener) => {
      contactsListener.on('commit:created', async (event) => {
        console.log('Contact data committed');
      });
    });

    // Combine both for maximum precision
    listener.namespace('space:customer-portal', (customerListener) => {
      customerListener.filter({ sheet: 'enterprise-customers' }, (enterpriseListener) => {
        enterpriseListener.on('commit:created', async (event) => {
          console.log('Processing enterprise customer data');
        });
      });
    });
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener, FlatfileEvent } from '@flatfile/listener';

  export default function (listener: FlatfileListener): void {
    // Use namespaces for architectural organization
    listener.namespace('space:customer-portal', (customerListener: FlatfileListener) => {
      customerListener.on('commit:created', async (event: FlatfileEvent) => {
        console.log('Processing customer portal data...');
      });
    });

    // Use filters for granular targeting
    listener.filter({ sheet: 'contacts' }, (contactsListener: FlatfileListener) => {
      contactsListener.on('commit:created', async (event: FlatfileEvent) => {
        console.log('Contact data committed');
      });
    });

    // Combine both for maximum precision
    listener.namespace('space:customer-portal', (customerListener: FlatfileListener) => {
      customerListener.filter({ sheet: 'enterprise-customers' }, (enterpriseListener: FlatfileListener) => {
        enterpriseListener.on('commit:created', async (event: FlatfileEvent) => {
          console.log('Processing enterprise customer data');
        });
      });
    });
  }
  ```
</CodeGroup>

## Agents

In Flatfile, an **Agent** refers to a server-side listener bundled and deployed to Flatfile to be hosted in our secure cloud. You may deploy multiple Agents to a single Environment, each with its own configurations and codebase. But be careful, as multiple Agents can interfere with each other if not properly managed.

<Note>
  Terminology note: **Agents** in Flatfile should not be confused with **AI Agents**. Flatfile Agents are Event Listeners running in Flatfile's cloud infrastructure, whereas AI Agents refer to a semi-autonomous Artificial Intelligence system.
</Note>

### Agent Deployment

To deploy an Agent, run the following command in your terminal from the root directory containing your listener file:

```bash theme={null}
npx flatfile@latest deploy
```

The CLI will automatically examine your listener code and register itself to listen to only the Events your listener is configured to handle.

#### Deployment Options

**Unique Slug**: Use the `-s` flag to give your Agent a unique slug. This is useful when managing multiple Agents in the same Environment.

```bash theme={null}
npx flatfile@latest deploy -s my-custom-agent
```

**Multiple Agents**: You may deploy multiple Agents to a single Environment, each with its own configurations and codebase. However, be careful as multiple Agents can interfere with each other if not properly managed (see the next section).

If no slug is provided:

* **Single Agent**: Your Agent will be updated and given a slug of `default`
* **Multiple Agents**: The CLI will prompt you to select which Agent to update

### Managing Multiple Agents

When deploying multiple Agents to the same Environment, careful management is essential to prevent race conditions and conflicts. Multiple Agents listening to the same Event topics can interfere with each other and cause unpredictable behavior.

We recommend combining your code into a single codebase and using [Namespaces and Filters](/guides/namespaces-and-filters) to route events to scoped listeners, ideally organized into separate subdirectories. However, if you need multiple Agents, follow these strategies to avoid conflicts:

#### Conflict Prevention Strategies

1. **Use Namespaces and Filters**: Ensure each Agent listens to distinct Event patterns
2. **Segment by Domain**: Separate Agents by functional areas (e.g., data processing vs. integrations) and separate your accout into different [Apps](/core-concepts/apps)
3. **Avoid Topic Overlap**: Never have multiple Agents handling the same Event topics without proper routing

### Agent Configuration

When deploying an Agent, the CLI will automatically reduce the topic scope your Agent listens to by examining your listener code and registering itself to listen to only the topics your listener is configured to act on.

For example if your listener code only listens to `commit:created` Events, the CLI will automatically configure the Agent to only listen to `commit:created` Events. If your listener has a wildcard listener `**`, the CLI will configure the Agent to listen to all Events.

### Agent Logs

When using Flatfile's secure cloud to host your listener, you can view the executions of your Agent in the "Event Logs" tab of your dashboard.

Event logs are useful in monitoring and troubleshooting your listener. Each execution of your agent is recorded here, including any custom console logs you have configured in your listener code.

<Info>If you are running your Agent locally using the `develop` command to test, these logs will not be recorded in your dashboard. You can still view them in your terminal.</Info>

#### Failures

Failures occur when an Agent fails to execute properly. That means either an uncaught error is thrown, or the execution times out.

If you are catching and handling errors within your code, those executions will not be marked as failures. If you would prefer to see them marked this way, re-throw your error after handling to bubble it up to the execution handler.

<CodeGroup>
  ```javascript JavaScript theme={null}
  export default function (listener) {
    //note: listening to all Events with a wildcard can be used while testing but is not
    //recommended for production, as it will capture all Events and may cause performance issues
    listener.on("**", (event) => {
      try {
        // do something
      } catch (error) {
        // handle error
        throw error; // re-throw error to mark execution as failure
      }
    });
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener, FlatfileEvent } from '@flatfile/listener';

  export default function (listener: FlatfileListener) {
    //note: listening to all Events with a wildcard can be used while testing but is not
    //recommended for production, as it will capture all Events and may cause performance issues
    listener.on("**", (event: FlatfileEvent) => {
      try {
        // do something
      } catch (error) {
        // handle error
        throw error; // re-throw error to mark execution as failure
      }
    });
  }
  ```
</CodeGroup>


# Overview
Source: https://flatfile.com/docs/core-concepts/overview

Understanding Flatfile's building blocks and architecture

Our platform follows a hierarchical structure designed to provide secure, organized access to various resources and automation capabilities. Understanding these core components and their relationships will help you build more effective data import solutions.

## Platform Architecture

<Frame>
  <img />

  <img />
</Frame>

Flatfile operates as a hosted platform that integrates with your application through APIs and SDKs. Your users interact with Flatfile's interface while you maintain control over the data flow and business logic through event-driven listeners and customizable workflows.

## Core Concepts

Understanding Flatfile's architecture starts with these fundamental concepts, organized by their role in the platform:

### Infrastructure & Organization

<CardGroup>
  <Card title="Environments" href="/core-concepts/environments">
    Isolated entities that provide secure domains for creating and testing different configurations, with separate API keys for development and production use cases.
  </Card>

  <Card title="Apps" href="/core-concepts/apps">
    Organizational units that manage and coordinate data import workflows across different environments, serving as containers for organizing related Spaces with consistent configuration.
  </Card>

  <Card title="Spaces" href="/core-concepts/spaces">
    Micro-applications with their own database, filestore, and authentication that serve as isolated workspaces for individual data import sessions or customer data management.
  </Card>
</CardGroup>

### Data Structure

<CardGroup>
  <Card title="Blueprints" href="/core-concepts/blueprints">
    Complete data definitions that control how your data should look, behave, and connect, serving as intelligent templates that define structure, validation rules, and relationships between sheets.
  </Card>

  <Card title="Documents" href="/core-concepts/documents">
    Standalone webpages within Spaces that provide guidance, instructions, and dynamically-generated content to help users navigate their data import process.
  </Card>

  <Card title="Workbooks" href="/core-concepts/workbooks">
    Database-like containers configured with type-strict Blueprints that replace spreadsheet templates for data collection, allowing users to validate, correct, and import data with real-time feedback.
  </Card>

  <Card title="Sheets" href="/core-concepts/sheets">
    Individual data tables within Workbooks that organize and structure imported data, similar to database tables or spreadsheet tabs, with each sheet representing a distinct data type or entity.
  </Card>

  <Card title="Fields" href="/core-concepts/fields">
    Individual column definitions within sheets that specify data type, format, and validation constraints for each piece of data, similar to columns in a spreadsheet or database table.
  </Card>

  <Card title="Records" href="/core-concepts/records">
    Individual data rows that contain actual imported values conforming to field definitions, representing the data instances that flow through import workflows from upload to export.
  </Card>
</CardGroup>

### Automation & Processing

<CardGroup>
  <Card title="Events and Listeners" href="/core-concepts/listeners">
    Form the foundation of Flatfile's automation - Events are published whenever something happens in Flatfile, and Listeners are functions you write that respond to these Events by executing custom code.
  </Card>

  <Card title="Jobs" href="/core-concepts/jobs">
    Large units of work performed asynchronously on resources like files, Workbooks, or Sheets, providing visibility into the status and progress of data processing tasks. Jobs are powered by [Events](/core-concepts/listeners#events), and are run within by [Listeners](/core-concepts/listeners).
  </Card>

  <Card title="Actions" href="/core-concepts/actions">
    Code-based operations that run when users click buttons or menu items in Flatfile, which can be mounted on Sheets, Workbooks, Documents, or Files to trigger custom operations. Actions are powered by [Jobs](/core-concepts/jobs), and are run within [Listeners](/core-concepts/listeners).
  </Card>

  <Card title="Plugins" href="/core-concepts/plugins">
    Reusable, modular components that extend Flatfile's data import and processing capabilities, providing pre-built functionality for common tasks like data transformation and integrations.
  </Card>
</CardGroup>


# Plugins
Source: https://flatfile.com/docs/core-concepts/plugins

Reusable functionality modules that extend Flatfile's capabilities

Plugins in Flatfile are reusable, modular components that extend the platform's data import and processing capabilities. They provide pre-built functionality for common data import/extraction, transformation, and processing tasks, integrations, and workflows, enabling rapid development of sophisticated import solutions.

## What are Plugins?

A Plugin is a self-contained module that:

* **Extends Functionality** - Adds new capabilities to your import workflows
* **Encapsulates Logic** - Packages complex operations into reusable components
* **Provides Integration** - Connects to external systems and services
* **Enables Customization** - Allows configuration for specific use cases

Plugins can be used by [Listeners](/core-concepts/listeners) to perform specialized tasks without writing custom code. You can also create your own plugins to extend Flatfile's capabilities.

For detailed guides on our available plugins, see the [Plugins](/plugins) section.


# Records
Source: https://flatfile.com/docs/core-concepts/records

Individual data rows that represent your imported data in Flatfile

Records in Flatfile represent individual rows of data within your import workflows. They are instances of data that conform to the [field](/core-concepts/fields) definitions in your [blueprints](/core-concepts/blueprints), containing the actual values that users import, validate, and transform.

## What are Records?

A Record is a data instance that:

* **Contains Data Values** - Stores the actual imported data for each [field](/core-concepts/fields)
* **Maintains State** - Tracks validation status, errors, and processing history
* **Supports Transformation** - Allows data modification and enrichment

Records are the fundamental data units that flow through your import pipelines, from initial upload through final export.

## Record Structure

### Basic Record Anatomy

This is an example JSON response from the Flatfile API. For working with Records in your language, see our [API Reference](/api-reference) documentation

```json theme={null}
{
  "data": {
    "records": [
      {
        "id": "us_rc_YOUR_ID",
        "values": {
          "firstName": {
            "messages": [],
            "updatedAt": "2023-11-20T16:59:40.286Z",
            "valid": true,
            "value": "John"
          },
          "lastName": {
            "messages": [],
            "updatedAt": "2023-11-20T16:59:40.286Z",
            "valid": true,
            "value": "Smith"
          },
          "email": {
            "messages": [],
            "updatedAt": "2023-11-20T16:59:40.286Z",
            "valid": true,
            "value": "john.smith@example.com"
          }
        },
        "valid": true,
        "metadata": {},
        "config": {}
      }
    ],
    "success": true,
    "commitId": "us_vr_YOUR_ID",
    "counts": {
      "total": 1000,
      "valid": 1000,
      "error": 0
    },
    "versionId": "us_vr_YOUR_ID"
  }
}
```


# Sheets
Source: https://flatfile.com/docs/core-concepts/sheets

Individual data tables within Workbooks that organize and structure imported data

## What are Sheets?

Sheets are individual data tables within [Workbooks](/core-concepts/workbooks) that organize and structure imported data. Each Sheet represents a distinct data type or entity, similar to tables in a database or tabs in a spreadsheet.

Sheets serve as containers for [Records](/core-concepts/records) and are defined by [Blueprints](/core-concepts/blueprints) that specify their structure, validation rules, and data types. They provide the fundamental building blocks for organizing data within the Flatfile platform.

## Basic Blueprint Structure

* A [Blueprint](/core-concepts/blueprints) defines the data structure for any number of [Spaces](/core-concepts/spaces)
* A [Space](/core-concepts/blueprints) may contain many [Workbooks](/core-concepts/workbooks) and many [Documents](/core-concepts/documents)
* A [Document](/core-concepts/documents) contains static documentation and may contain many [Document-level Actions](/guides/using-actions#document-actions)
* A [Workbook](/core-concepts/workbooks) may contain many [Sheets](/core-concepts/sheets) and many [Workbook-level Actions](/guides/using-actions#workbook-actions)
* A [Sheet](/core-concepts/sheets) may contain many [Fields](/core-concepts/fields) and many [Sheet-level Actions](/guides/using-actions#sheet-actions)
* A [Field](/core-concepts/fields) defines a single column of data, and may contain many [Field-level Actions](/guides/using-actions#field-actions)

## Basic Sheet Definition

The following examples demonstrate the configuration of isolated Sheets, which are intended to be used in the context of a [Workbook](/core-concepts/workbooks) configuration.

### Single-Sheet Sheet Configuration

This example configures a single [Sheet](/core-concepts/sheets) containing three [Fields](/core-concepts/fields) and one [Action](/core-concepts/actions) and defining [access controls](#sheet-level-access).

```javascript theme={null}
const customerSheet = {
  name: "Customers",
  slug: "customers",
  fields: [
    {
      key: "firstName",
      type: "string",
      label: "First Name",
      constraints: [{ type: "required" }],
    },
    {
      key: "email",
      type: "string",
      label: "Email Address",
      constraints: [{ type: "required" }, { type: "unique" }],
    },
    {
      key: "company",
      type: "string",
      label: "Company Name",
    },
  ],
  access: ["add", "edit", "delete"],
  actions: [
    {
      operation: "validate-inventory",
      mode: "background",
      label: "Validate Inventory",
      description: "Check product availability against inventory system",
    },
  ],
};
```

## Sheet level access

With `access` you can control Sheet-level access for users.

| Access Level      | Description                                                                                     |
| ----------------- | ----------------------------------------------------------------------------------------------- |
| `"*"` *(default)* | A user can use all access actions to this Sheet                                                 |
| `"add"`           | A user can add a record(s) to the Sheet                                                         |
| `"delete"`        | A user can delete record(s) from the Sheet                                                      |
| `"edit"`          | A user can edit records (field values) in the Sheet                                             |
| `"import"`        | A user can import CSVs to this Sheet                                                            |
| `<empty>`         | If no parameters are specified in the access array, sheet-level readOnly access will be applied |

<Warning>
  If you use `"*"` access control, users will gain new functionalities as we expand
  access controls. Use an exhaustive list today to block future functionality from
  being added automatically.
</Warning>

```javascript theme={null}
{
  "sheets": [
    {
      "name": "Contacts",
      "slug": "contacts",
      "access": ["add", "edit"]
      // Define fields
    }
  ]
}
```

## Sheet Constraints

Sheet constraints apply validation rules across multiple fields or entire sheets. These constraints ensure data integrity at the sheet level and work in conjunction with field-level constraints.

### Composite Uniqueness

Ensures that combinations of multiple field values are unique across all records in the sheet. This is useful when individual fields can have duplicate values, but their combination should be unique.

```javascript theme={null}
{
  name: "Customers",
  slug: "customers",
  constraints: [
    {
      name: "unique-customer-location",
      type: "unique",
      fields: ["customerId", "locationId"],
      requiredFields: ["customerId"], // Only enforce when customerId has a value
      strategy: "concat",
      config: {
        message: "Customers must be unique per location",
        level: "error"
      }
    }
  ],
  fields: [
    // field definitions
  ]
}
```

#### Configuration Properties

| Property         | Type      | Required | Description                                                                          |
| ---------------- | --------- | -------- | ------------------------------------------------------------------------------------ |
| `name`           | string    | ✓        | Unique identifier for the constraint                                                 |
| `type`           | string    | ✓        | Must be `"unique"` for composite uniqueness constraints                              |
| `fields`         | string\[] | ✓        | Array of field names that must be unique together                                    |
| `requiredFields` | string\[] |          | Subset of `fields` that when empty, disables constraint validation                   |
| `strategy`       | string    | ✓        | Either `"concat"` or `"hash"` - determines how uniqueness is calculated              |
| `config`         | object    |          | Configuration options for the validation message and level when the constraint fails |

**Strategy Options:**

| Strategy | Description                                            | Best Used When                                             |
| -------- | ------------------------------------------------------ | ---------------------------------------------------------- |
| `concat` | Concatenates field values to determine uniqueness      | Simple field types, debugging needed, performance critical |
| `hash`   | Uses SHA1 hash of combined field values for uniqueness | Complex values, collision avoidance, data privacy          |

**Config Options:**

| Key       | Type   | Description                                                                                                           |
| --------- | ------ | --------------------------------------------------------------------------------------------------------------------- |
| `message` | string | The message shown to the user if this constraint is violated. Defaults to "Composite \<constraint name> is not unique |
| `level`   | string | The error level when this constraint is violated. Must be one of `"error"` (default), `"warn"` or `"info`".           |

#### Choosing the Right Strategy

The `strategy` property determines how uniqueness is calculated. You can choose to simply concatenate the field values as a single string or use a SHA1 hash function to create a unique identifier. Consider the following when choosing a strategy:

**Use `concat` when:**

* You aren't concerned about concatenation collisions (see example below)
* Performance is critical (string concatenation is faster than SHA1)
* You have short/consistent value sizes

**Use `hash` when:**

* You want to avoid concatenation collisions
* You aren't concerned about the performance cost (SHA1 calculation is slower than string concatenation)
* You have long/inconsistent value sizes (SHA1 hashes are always 20 bytes)

**Concatenation Collision Example:**

In this example, the `concat` strategy would consider the following records to be duplicates:

<CardGroup>
  <Card>
    ```json theme={null}
      {
        "firstName": "John",
        "lastName": "Smith"
      }
    ```
  </Card>

  <Card>
    ```json theme={null}
      {
        "firstName": "JohnS",
        "lastName": "mith"
      }
    ```
  </Card>
</CardGroup>

Constraint configuration with `concat` strategy:

```javascript theme={null}
{
  // sheet configuration
  constraints: [
    {
      name: "unique-name-combination",
      type: "unique",
      fields: ["firstName", "lastName"],
      strategy: "concat"
    }
  ]
}
```

This is because both records have the same concatenated value:

```javascript theme={null}
"John" + "Smith" = "JohnSmith"
"JohnS" + "mith" = "JohnSmith"
```

But the `hash` strategy would prevent this, because the hash function creates a unique identifier based on each field's invividual value rather than a simple concatenation.

```javascript theme={null}
hash(["John", "Smith"]) = "9e03c21f9beff9d943843c1b0623848fe63e2beb"
hash(["JohnS", "mith"]) = "2fd09d2f5e62d980988094b43640966a3bffbde9"
```

Constraint configuration with `hash` strategy:

```javascript theme={null}
{
  // sheet configuration
  constraints: [
    {
      name: "unique-name-combination",
      type: "unique",
      fields: ["firstName", "lastName"],
      strategy: "hash" // Prevents collision
    }
  ]
}
```

#### Conditional Validation with Required Fields

The `requiredFields` property enables conditional uniqueness validation. When any field specified in `requiredFields` is empty (null, undefined, or empty string), the entire constraint is ignored for that record.

**Use Cases:**

* **Partial data imports** - Allow incomplete records during staged import processes
* **Optional relationships** - Handle cases where some composite key fields are optional
* **Data migration** - Gradually enforce constraints as required fields get populated
* **Conditional business rules** - Only enforce uniqueness when critical fields have values

**Important Notes:**

* `requiredFields` should contain only fields that exist in the `fields` array
* If ANY required field is empty, the constraint is completely ignored
* Empty fields are: `null`, `undefined`, or empty strings (`""`)
* If `requiredFields` is omitted, the constraint always applies

#### Example: Customer Registration System

Consider a customer registration system where you want unique combinations of email and company, but only when `email` is provided:

```javascript theme={null}
{
  name: "unique-customer-profile",
  type: "unique",
  fields: ["email", "company"],
  requiredFields: ["email"], // Only enforce when email exists
  strategy: "hash"
}
```

**Data Behavior:**

| Email             | Company       | Validation Result | Reason                               |
| ----------------- | ------------- | ----------------- | ------------------------------------ |
| `"john@acme.com"` | `"Acme Corp"` | ✅ Enforced        | Email provided                       |
| `"jane@acme.com"` | `"Acme Corp"` | ❌ Duplicate       | Same email+company combination       |
| `""`              | `"Acme Corp"` | ⏭️ Ignored        | Email empty (required field)         |
| `null`            | `"Beta Inc"`  | ⏭️ Ignored        | Email null (required field)          |
| `"bob@beta.com"`  | `""`          | ✅ Enforced        | Email provided, company can be empty |

**Without `requiredFields`:**
All records would be validated, potentially causing errors during partial data imports.

#### Example: Multiple Required Fields

For more complex scenarios, you can specify multiple required fields:

```javascript theme={null}
{
  name: "unique-order-item",
  type: "unique",
  fields: ["orderId", "productId", "customerId"],
  requiredFields: ["orderId", "productId"], // Both must have values
  strategy: "hash"
}
```

In this case, the constraint only applies when **both** `orderId` and `productId` have non-empty values. If either is empty, the entire constraint is ignored.

<Info>
  Individual field constraints like required and unique are covered in [Field Constraints](/core-concepts/fields#field-constraints).
</Info>

## Collections

Collections provide a way to organize Sheets within a Workbook into named groupings in the Flatfile UI. This helps to visually organize complex Workbooks with many Sheets, making it easier to navigate and understand the data structure. Collections have no functional impact on the data itself; their only purpose is to help you organize your Sheets visually.

### Configuring Sheets with Collections

Assigning a Collection to a Sheet is as simple as adding a `collection` property to your Sheet [Blueprint](/core-concepts/blueprints) when [configuring your Space](/core-concepts/spaces#space-configuration) (or otherwise creating or updating a Sheet via the API). If you add the same Collection to multiple Sheets, they will be grouped together in the UI.

The following example depicts a Blueprint defining a single Workbook with three Sheets organized into two Collections: **Source Data** with one Sheet, and **Processed** with two Sheets.

<Info>
  Workbooks also have a similar feature called [Folders](/core-concepts/workbooks#folders), which you can use to group associated Workbooks.

  You can think of **Folders** and **Collections** like a filing system:

  * [Folders](/core-concepts/workbooks#folders) help you organize your Workbooks within a Space (like organizing binders on a shelf)
  * **Collections** help you organize Sheets within each Workbook (like organizing tabs within a binder).
</Info>

<Tabs>
  <Tab title="Screenshot">
    <Frame>
      <img />
    </Frame>
  </Tab>

  <Tab title="Blueprints">
    ```javascript theme={null}
    const workbook = {
      name: "Data Processing Pipeline",
      sheets: [
        {
          name: "Raw Customer Data",
          collection: "Source Data", // Assign to "Source Data" collection (only Sheet)
          slug: "raw_customers",
          fields: [
            { key: "name", type: "string", label: "Customer Name" },
            { key: "email", type: "string", label: "Email Address" }
          ]
        },
        {
          name: "Processed Customers",
          collection: "Processed", // Assign to "Processed" collection (first Sheet)
          slug: "clean_customers", 
          fields: [
            { key: "name", type: "string", label: "Full Name" },
            { key: "email", type: "string", label: "Email" },
            { key: "validation_status", type: "string", label: "Status" }
          ]
        },
        {
          name: "Customer Reports", 
          collection: "Processed", // Assign to "Processed" collection (second Sheet)
          slug: "customer_reports",
          fields: [
            { key: "metric", type: "string", label: "Metric Name" },
            { key: "value", type: "number", label: "Value" }
          ]
        }
      ]
    };
    ```
  </Tab>
</Tabs>

## Sheet Treatments

Sheets have an optional `treatments` parameter which takes an array of treatments for your Sheet. Treatments can be used to categorize your Sheet and control its behavior. Certain treatments will cause your Sheet to look or behave differently.

### Reference sheets

Giving your Sheet a treatment of `"ENUM_REFERENCE"` will mark it as reference data for other sheets. Reference sheets are currently hidden from view, allowing you to generate a number of reference values without adding visual distraction for the user.

<Note>
  **Dynamic Enums**

  This feature, along with the [Reference Field Filtering](/core-concepts/fields#reference-field-filtering) feature, may be collectively referred to as **Dynamic Enums**. By combining these two features, you can create a drop-down list for any cell in your sheet that's dynamically controlled by the value of another field in the same record – and to the end-user, it will just work like a dynamically-configured `enum` field.
</Note>

<Warning>
  Please note that this feature is not intended for situations where PII or other sensitive data must be hidden from view of the user - for situations like that, reach out to support or your CSM for best practices.
</Warning>

```javascript theme={null}
const referenceSheet = {
  name: "Countries",
  slug: "countries",
  treatments: ["ENUM_REFERENCE"],
  fields: [
    {
      key: "code",
      type: "string",
      label: "Country Code"
    },
    {
      key: "name",
      type: "string",
      label: "Country Name"
    }
  ]
};
```

Currently, `"ENUM_REFERENCE"` is the only treatment that changes the behavior of your Sheet.

## Sheet Options

Configurable properties for a Sheet that control its behavior and appearance:

| Option                         | Type    | Required | Description                                                                                                                                                                                             |
| ------------------------------ | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **name**                       | string  | ✓        | The name of your Sheet as it will appear to your end users                                                                                                                                              |
| **description**                | string  |          | A sentence or two describing the purpose of your Sheet                                                                                                                                                  |
| **slug**                       | string  |          | A unique identifier for your Sheet. Used to reference your Sheet in code, for example in a Record Hook                                                                                                  |
| **readonly**                   | boolean |          | A boolean specifying whether or not this sheet is read only. Read only sheets are not editable by end users                                                                                             |
| **allowAdditionalFields**      | boolean |          | When set to `true`, your Sheet will accept additional fields beyond what you specify in its configuration. These additional fields can be added via API, or by end users during the file import process |
| **access**                     | array   |          | An array specifying the access controls for this Sheet. Valid values: `"*"`, `"add"`, `"edit"`, `"delete"`, `"import"`. [Read more about access controls](#sheet-level-access)                          |
| **fields**                     | array   | ✓        | This is where you define your Sheet's data schema. The collection of fields in your Sheet determines the shape of data you wish to accept (minimum: 1, maximum: 1000)                                   |
| **actions**                    | array   |          | An array of actions that end users can perform on this Sheet. [Read more about actions](/core-concepts/actions)                                                                                         |
| **constraints**                | array   |          | An array of sheet-level validation constraints that apply across multiple fields or entire sheets. [Read more about sheet constraints](#sheet-constraints)                                              |
| **metadata**                   | object  |          | Use `metadata` to store any extra contextual information about your Sheet. Must be valid JSON                                                                                                           |
| **mappingConfidenceThreshold** | number  |          | Configure the minimum required confidence for mapping jobs targeting that Sheet (default: 0.5, range: 0-1)                                                                                              |


# Spaces
Source: https://flatfile.com/docs/core-concepts/spaces

Micro-applications for content and data storage

Flatfile Spaces are micro-applications, each having their own database, filestore, and auth. Use Spaces to integrate Flatfile into your data exchange workflow, whether that happens directly in your application or as part of an offline process.

You can think of a Space as a home for any one of your Customers' data, or as place for a discrete data migration session; you can create a new Space any time you need an isolated place to migrate new data.

<Note>
  Terminology note: **Spaces** are sometimes referred to as **Projects** in Flatfile conversations - this is because we often consider a Space to be a single import or series of imports from a single end-customer, or, a "Project".
</Note>

## Anatomy

The following example depicts a **Space** with:

* 1 [Document](/core-concepts/documents) named **Customers**
* 2 [Workbooks](/core-concepts/workbooks):
  * **Customers Workbook**
    * 3 [Sheets](/core-concepts/sheets) named **Customers**, **Orders**, and **Products**
  * **Company Workbook**
    * 3 [Sheets](/core-concepts/sheets) named **Users**, **Departments**, and **Projects**
    * 2 [Workbook Actions](/core-concepts/actions) labeled **Download** and **Submit**

The current view has the **Users** sheet under the **Company Workbook** selected.

<Frame>
  <img />
</Frame>

## Basic Blueprint Structure

* A [Blueprint](/core-concepts/blueprints) defines the data structure for any number of [Spaces](/core-concepts/spaces)
* A [Space](/core-concepts/blueprints) may contain many [Workbooks](/core-concepts/workbooks) and many [Documents](/core-concepts/documents)
* A [Document](/core-concepts/documents) contains static documentation and may contain many [Document-level Actions](/guides/using-actions#document-actions)
* A [Workbook](/core-concepts/workbooks) may contain many [Sheets](/core-concepts/sheets) and many [Workbook-level Actions](/guides/using-actions#workbook-actions)
* A [Sheet](/core-concepts/sheets) may contain many [Fields](/core-concepts/fields) and many [Sheet-level Actions](/guides/using-actions#sheet-actions)
* A [Field](/core-concepts/fields) defines a single column of data, and may contain many [Field-level Actions](/guides/using-actions#field-actions)

### Blueprints and Space Configuration

Every Space has exactly one [Blueprint](/core-concepts/blueprints) that defines its data structure. When you configure a Space by creating workbooks, sheets, and fields, you're defining the Space's Blueprint - the schema that controls how data should look, behave, and connect within that Space.

The Blueprint is established during the `space:configure` [Job](/core-concepts/jobs) and remains consistent throughout the Space's lifecycle (though you can change it via the [API](https://reference.flatfile.com/api-reference/workbooks/update) at any time). This ensures all data imported into the Space follows the same structure and validation rules you've defined.

<Note>
  By default, the same Blueprint configuration is applied to all Spaces unless you [namespace your listeners](/guides/namespaces-and-filters) to specific scopes. This makes it easy to maintain consistent data structures across multiple Spaces for similar use cases or different customers with the same data requirements.
</Note>

## Creating Spaces

Spaces can be created manually via your [Flatfile Dashboard](https://platform.flatfile.com/dashboard), or — quite commonly — programmatically via the [Flatfile API](https://reference.flatfile.com/api-reference/spaces/create).

Here are some example workflows for creating Spaces:

### Backend Automation

When integrating Flatfile into your application workflows:

* **CRM Integration**: When a customer in your CRM moves to the "onboarding" stage, you may create a new Space via the API for them to complete their data onboarding process.
* **Account Creation**: When a new account is created in your application, you may automatically create a new Space for them via the API.

```mermaid theme={null}
graph TD
    A[CRM Customer] -->|Moves to onboarding stage| B[Create Space via API]
    C[New Account Created] -->|In your application| D[Create Space via API]
    
    B --> E[Customer completes data onboarding]
    D --> F[Account setup and data import]

```

### User-Initiated Creation

For direct customer collaboration and embedded applications:

* **Direct Collaboration**: When working with a new customer directly, you may create a new Space via the Dashboard and invite them to the Space.
* **Embedding Flatfile**: In an [Embedded App](/embedding/overview), you may create a new Space every time a user clicks a button to start a new import.

```mermaid theme={null}
graph TD
    A[Working with Customer] -->|Direct collaboration| B[Create Space via Dashboard]
    C[Embedded App User] -->|Clicks import button| D[Create Space via API]
    
    B --> E[Invite customer to Space]
    D --> F[User starts new import session]
```

You may also rename "Space" to any other term you prefer, such as "Project" or "Customer". Each App may have a different name for a Space, but they all have the same underlying structure.

<Frame>
  <img />
</Frame>

### Space Namespaces

Spaces can be assigned [namespaces](/guides/namespaces-and-filters) to control which [listeners](/core-concepts/listeners) receive events from that space. This enables isolation between different parts of your application or different customers. See the [Namespaces and Filters guide](/guides/namespaces-and-filters) for detailed examples and patterns.

## Space Configuration

<Info>
  For complete theming capabilities, see our comprehensive [Theme Your Space guide](/guides/theme-your-space).
</Info>

If you're using [Autobuild](/getting-started/quickstart/autobuild), space configuration is handled automatically via its own deployed agent.

If you're configuring Flatfile in code, you can configure the Space via a [Listener](/core-concepts/listeners).
When a new space is created, Flatfile automatically creates a `space:configure` [Job](/core-concepts/jobs) for you to configure the Space. In your job listener, you can create workbooks, sheets, and actions, as well as configure the Space's theme and metadata. This workbook and sheet configuration defines your Space's [Blueprint](/core-concepts/blueprints) - the data schema that controls how your data should look, behave, and connect.

<Note>
  In this example, we'll show the full Job Listener lifecycle implementation, complete with `ack` to acknowledge the job, `update` to update the job's progress, and `complete` or `fail` to complete or fail the job.

  As in other places throughout the documentation, you *could* use the [Job Handler](/plugins/job-handler) plugin to handle the job lifecycle for you. This may be useful if you have a particularly complex space configuration with custom logic, but don't need complete control over the job's entire lifecycle.

  However, for most implementations, we recommend using the [Space Configure](/plugins/space-configure) plugin. This plugin takes care of even more of the heavy lifting for you; not only does it handle the Job lifecycle, but it also takes care of all of the API calls necessary to configure the Space and create its Workbooks and documents.

  With this plugin, you can configure your entire space with a single configuration object rather than perforing any API calls.

  For example, this listener implementation configures a space with a [Blueprint](/core-concepts/blueprints) containing two [workbooks](/core-concepts/workbooks) and a Welcome Guide [document](/core-concepts/documents) – both defined in another file – as well as adding some light [theming](/guides/theme-your-space):

  ```javascript theme={null}
  listener.use(configureSpace({
    workbooks: [companyWorkbook, customerWorkbook],
    documents: [welcomeGuideDocument],
    space: {
      metadata: {
        theme: {
          root: {
            primaryColor: "#4F46E5",
            actionColor: "#000000"
          },
          sidebar: {
            logo: "www.example.com/logo.png"
          },
        }
      }
    }
  }));
  ```
</Note>

Here's how to manually configure a Space using the job lifecycle approach. This example demonstrates creating a [Blueprint](/core-concepts/blueprints) by defining workbooks with sheets and fields, custom actions, a welcome document, and theme customization, with complete control over the configuration process and proper job lifecycle management.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import api from "@flatfile/api";

  export default function (listener) {

    // Listen for the space:configure job
    listener.on("job:ready", { job: "space:configure" }, async (event) => {

      // Get the job ID and space ID from the event context
      const { jobId, spaceId } = event.context;
      
      try {
        // Acknowledge the job
        await api.jobs.ack(jobId, {
          info: "Setting up customer data workspace",
          progress: 10
        });

        // Create the workbook with a sheet and an action
        const { data: workbook } = await api.workbooks.create({
          spaceId,
          name: "Customer Data",
          sheets: [
            {
              name: "Contacts",
              slug: "contacts",
              fields: [
                { key: "firstName", type: "string", label: "First Name" },
                { key: "lastName", type: "string", label: "Last Name" },
                { key: "email", type: "string", label: "Email Address" },
                { key: "phone", type: "string", label: "Phone Number" }
              ]
            }
          ],
          actions: [
            {
              operation: "submit",
              label: "Submit Data",
              mode: "foreground",
              primary: true,
              description: "Submit customer data for processing",
              constraints: [{ type: "hasData" }]
            }
          ]
        });

        // Update progress
        await api.jobs.update(jobId, {
          info: "Customer workbook created",
          progress: 50
        });

        // Create a welcome document
        await api.documents.create(spaceId, {
          title: "Welcome Guide",
          body: `
            <h1>Welcome to Customer Data Import</h1>
            <p>This Space helps you import and validate customer contact information.</p>
            <h2>Getting Started</h2>
            <ol>
              <li>Upload your CSV file or enter data manually</li>
              <li>Review and correct any validation errors</li>
              <li>Click "Submit Data" when ready</li>
            </ol>
          `
        });

        // Update progress
        await api.jobs.update(jobId, {
          info: "Welcome document added",
          progress: 75
        });

        // Configure the space theme
        await api.spaces.update(spaceId, {
          metadata: {
            theme: {
              root: {
                primaryColor: "#4F46E5",
                borderColor: "#E5E7EB",
                fontFamily: "Inter, system-ui, sans-serif"
              },
              sidebar: {
                backgroundColor: "#1F2937",
                textColor: "#F9FAFB",
                titleColor: "#4F46E5"
              }
            }
          }
        });

        // Complete the job
        await api.jobs.complete(jobId, {
          outcome: {
            message: "Customer data workspace configured successfully",
            acknowledge: true
          }
        });

      } catch (error) {
        console.error("Error configuring space:", error);
        
        // Fail the job if something goes wrong
        await api.jobs.fail(jobId, {
          outcome: {
            message: `Failed to configure space: ${error.message}`,
            acknowledge: true
          }
        });
      }
    });
  }
  ```

  ```typescript TypeScript theme={null}
  import type { FlatfileListener } from "@flatfile/listener";
  import api from "@flatfile/api";

  export default function (listener: FlatfileListener) {

    // Listen for the space:configure job
    listener.on("job:ready", { job: "space:configure" }, async (event) => {

      // Get the job ID and space ID from the event context
      const { jobId, spaceId } = event.context;
      
      try {
        // Acknowledge the job
        await api.jobs.ack(jobId, {
          info: "Setting up customer data workspace",
          progress: 10
        });

        // Create the workbook with a sheet and an action
        const { data: workbook } = await api.workbooks.create({
          spaceId,
          name: "Customer Data",
          sheets: [
            {
              name: "Contacts",
              slug: "contacts",
              fields: [
                { key: "firstName", type: "string", label: "First Name" },
                { key: "lastName", type: "string", label: "Last Name" },
                { key: "email", type: "string", label: "Email Address" },
                { key: "phone", type: "string", label: "Phone Number" }
              ]
            }
          ],
          actions: [
            {
              operation: "submit",
              label: "Submit Data",
              mode: "foreground",
              primary: true,
              description: "Submit customer data for processing",
              constraints: [{ type: "hasData" }]
            }
          ]
        });

        // Update progress
        await api.jobs.update(jobId, {
          info: "Customer workbook created",
          progress: 50
        });

        // Create a welcome document
        await api.documents.create(spaceId, {
          title: "Welcome Guide",
          body: `
            <h1>Welcome to Customer Data Import</h1>
            <p>This Space helps you import and validate customer contact information.</p>
            <h2>Getting Started</h2>
            <ol>
              <li>Upload your CSV file or enter data manually</li>
              <li>Review and correct any validation errors</li>
              <li>Click "Submit Data" when ready</li>
            </ol>
          `
        });

        // Update progress
        await api.jobs.update(jobId, {
          info: "Welcome document added",
          progress: 75
        });

        // Configure the space theme
        await api.spaces.update(spaceId, {
          metadata: {
            theme: {
              root: {
                primaryColor: "#4F46E5",
                borderColor: "#E5E7EB",
                fontFamily: "Inter, system-ui, sans-serif"
              },
              sidebar: {
                backgroundColor: "#1F2937",
                textColor: "#F9FAFB",
                titleColor: "#4F46E5"
              }
            }
          }
        });

        // Complete the job
        await api.jobs.complete(jobId, {
          outcome: {
            message: "Customer data workspace configured successfully",
            acknowledge: true
          }
        });

      } catch (error) {
        console.error("Error configuring space:", error);
        
        // Fail the job if something goes wrong
        await api.jobs.fail(jobId, {
          outcome: {
            message: `Failed to configure space: ${error.message}`,
            acknowledge: true
          }
        });
      }
    });
  }
  ```
</CodeGroup>


# Workbooks
Source: https://flatfile.com/docs/core-concepts/workbooks

Database-like containers with type-strict Blueprints for data import

## What are Workbooks?

Workbooks are analogous to a database, and like a database, they are configured with a type-strict [Blueprint](/core-concepts/blueprints). A Workbook replaces the spreadsheet template you may share with your users today when requesting data during the data collection phase of customer onboarding or other file-based data exchange processes.

Unlike a spreadsheet, Workbooks are designed to allow your team and users to validate, correct, and import data with real-time feedback.

Workbooks are effectively *containers* for related [Sheets](/core-concepts/sheets). Inside a any Workbook, you can define multiple Sheets that can [reference](/core-concepts/fields#reference) each other's data.

## Basic Blueprint Structure

* A [Blueprint](/core-concepts/blueprints) defines the data structure for any number of [Spaces](/core-concepts/spaces)
* A [Space](/core-concepts/blueprints) may contain many [Workbooks](/core-concepts/workbooks) and many [Documents](/core-concepts/documents)
* A [Document](/core-concepts/documents) contains static documentation and may contain many [Document-level Actions](/guides/using-actions#document-actions)
* A [Workbook](/core-concepts/workbooks) may contain many [Sheets](/core-concepts/sheets) and many [Workbook-level Actions](/guides/using-actions#workbook-actions)
* A [Sheet](/core-concepts/sheets) may contain many [Fields](/core-concepts/fields) and many [Sheet-level Actions](/guides/using-actions#sheet-actions)
* A [Field](/core-concepts/fields) defines a single column of data, and may contain many [Field-level Actions](/guides/using-actions#field-actions)

## Example Workbook Configuration

The following examples demonstrate the configuration of isolated Workbooks, which are intended to be used in the context of a [Blueprint](/core-concepts/blueprints) configuration.

### Single-Sheet Workbook Configuration

This example configures a single [Workbook](/core-concepts/workbooks) with a single [Sheet](/core-concepts/sheets) containing two [Fields](/core-concepts/fields).

```javascript theme={null}
const customerWorkbook = {
  name: "Customer Import",
  description: "Import and validate customer data",
  sheets: [
    {
      name: "customers",
      slug: "customers",
      fields: [
        {
          key: "name",
          type: "string",
          label: "Full Name",
          constraints: [{ type: "required" }],
        },
        {
          key: "email",
          type: "string",
          label: "Email Address",
          constraints: [{ type: "required" }, { type: "unique" }],
        },
      ],
    },
  ],
};
```

### Multi-Sheet Workbook Configuration

This example configures a [Workbook](/core-concepts/workbooks) with three [Sheets](/core-concepts/sheets) containing several [Fields](/core-concepts/fields) each.

```javascript theme={null}
const ecommerceWorkbook = {
  name: "E-commerce Data Import",
  description: "Import customers, products, and orders",
  sheets: [
    {
      name: "customers",
      slug: "customers",
      fields: [
        { key: "customerId", type: "string", label: "Customer ID" },
        { key: "email", type: "string", label: "Email" },
        { key: "firstName", type: "string", label: "First Name" },
        { key: "lastName", type: "string", label: "Last Name" },
      ],
    },
    {
      name: "products",
      slug: "products",
      fields: [
        { key: "productId", type: "string", label: "Product ID" },
        { key: "name", type: "string", label: "Product Name" },
        { key: "price", type: "number", label: "Price" },
        { key: "category", type: "string", label: "Category" },
      ],
    },
    {
      name: "orders",
      slug: "orders",
      fields: [
        { key: "orderId", type: "string", label: "Order ID" },
        {
          key: "customerId",
          type: "reference",
          label: "Customer",
          config: { ref: "customers", key: "customerId" },
        },
        {
          key: "productId",
          type: "reference",
          label: "Product",
          config: { ref: "products", key: "productId" },
        },
        { key: "quantity", type: "number", label: "Quantity" },
        { key: "orderDate", type: "date", label: "Order Date" },
      ],
    },
  ],
};
```

### Actions and Workflows

This example configures a [Workbook](/core-concepts/workbooks) with three [Actions](/core-concepts/actions) that can be used to validate and enrich the data in the workbook.

<Note>
  **A note about Actions:** Actions also require a listener to respond to the event published by clicking
  on them. For more, see [Using Actions](/guides/using-actions)
</Note>

```javascript theme={null}
const workbookWithActions = {
  name: "Advanced Customer Import",
  sheets: [customerSheet, productsSheet],

  actions: [
    {
      operation: "validate-all",
      mode: "foreground",
      label: "Validate All Data",
      description: "Run comprehensive validation on all sheets",
      primary: true,
      constraints: [
        {
          type: "hasData",
        },
      ],
    },
    {
      operation: "enrich-data",
      mode: "background",
      label: "Enrich Customer Data",
      description: "Add company information and social profiles",
      constraints: [
        {
          type: "hasData",
        },
      ],
    },
    {
      operation: "export-to-crm",
      mode: "foreground",
      label: "Export to CRM",
      description: "Send validated customers to Salesforce",
      constraints: [
        {
          type: "hasAllValid",
        },
      ],
    },
  ],
};
```

### Workbook Metadata

For comprehensive metadata usage patterns, see our [metadata guide](/guides/utilizing-metadata).

Add contextual information:

```javascript theme={null}
const metadataWorkbook = {
  name: "Q1 2024 Customer Import",
  description: "Quarterly customer data migration for Q1 2024",

  metadata: {
    version: "2.1",
    department: "Sales",
    projectId: "PROJ-2024-001",
  },

  sheets: [customerSheet],
};
```

### Workbook Namespaces

Workbooks can be assigned [namespaces](/guides/namespaces-and-filters) to enable granular event filtering and isolation within the same space:

```javascript theme={null}
const workbook = {
  name: 'Employee Data Processing',
  namespace: 'staging', // Simple string namespace
  sheets: [/* your sheets */]
};
```

Namespaces allow you to apply different [listeners](/core-concepts/listeners) and processing logic to different types of workbooks within the same space. For detailed examples and patterns, see the [Namespaces and Filters guide](/guides/namespaces-and-filters).

## Folders

Folders provide a simple way to organize Workbooks within a Space into named groupings in the Flatfile UI. Similar to organizing files in folders on your computer, you can group related Workbooks together for better organization and navigation. Folders have no functional impact on the data itself; their only purpose is to help you organize your Workbooks visually.

### Configuring Workbooks with Folders

Assigning a Folder to a Workbook is as simple as adding a `folder` property to your Workbook [Blueprint](/core-concepts/blueprints) when [configuring your Space](/core-concepts/spaces#space-configuration) (or otherwise creating or updating a Workbook via the API). If you add the same folder to multiple Workbooks, they will be grouped together in the UI.

The following example defines four Workbooks (**Marketing Data**, **Sales Report**, **Customer Onboarding**, and **Support Tickets**) split evenly between two folders (**Analytics** and **Customer Data**).

<Info>
  Sheets also have an similar feature called [Collections](/core-concepts/sheets#collections), which you can use to group associated Sheets within a Workbook.

  You can think of **Folders** and **Collections** like a filing system:

  * **Folders** help you organize your Workbooks within a Space (like organizing binders on a shelf)
  * [Collections](/core-concepts/sheets#organizing-sheets-with-collections) help you organize Sheets within each Workbook (like organizing tabs within a binder).
</Info>

<Tabs>
  <Tab title="Screenshot">
    <Frame>
      <img />
    </Frame>
  </Tab>

  <Tab title="Blueprints">
    ```javascript theme={null}
    // Analytics folder
    const marketingDataWorkbook = {
      name: "Marketing Data", 
      folder: "Analytics",
      sheets: [campaignMetricsSheet, leadSourcesSheet]
    };
    const salesReportWorkbook = {
      name: "Sales Report",
      folder: "Analytics",
      sheets: [salesDataSheet, revenueSheet]
    };

    // Customer Data folder
    const onboardingWorkbook = {
      name: "Customer Onboarding",
      folder: "Customer Data",
      sheets: [newCustomersSheet, onboardingStepsSheet]
    };

    const supportTicketsWorkbook = {
      name: "Support Tickets",
      folder: "Customer Data", 
      sheets: [ticketsSheet, resolutionsSheet]
    };
    ```
  </Tab>
</Tabs>


# Advanced Configuration
Source: https://flatfile.com/docs/embedding/advanced-configuration

Complete configuration reference for embedded Flatfile

This reference covers all configuration options for embedded Flatfile, from basic setup to advanced customization.

## Authentication & Security

### publishableKey

Your publishable key authenticates your application with Flatfile. This key is safe to include in client-side code.

**Where to find it:**

1. Log into [Platform Dashboard](https://platform.flatfile.com)
2. Navigate to **Developer Settings** → **API Keys**
3. Copy your **Publishable Key** (starts with `pk_`)

```javascript theme={null}
// Example usage
const config = {
  publishableKey: "pk_1234567890abcdef", // Your actual key
};
```

### Security Best Practices

#### Environment Variables

Store your publishable key in environment variables rather than hardcoding:

```javascript theme={null}
// ✅ Good - using environment variable
const config = {
  publishableKey: process.env.REACT_APP_FLATFILE_KEY,
};

// ❌ Avoid - hardcoded keys
const config = {
  publishableKey: "pk_1234567890abcdef",
};
```

## Common Configuration Options

These options are shared across all SDK implementations:

### Authentication

| Option           | Type   | Required | Description                                  |
| ---------------- | ------ | -------- | -------------------------------------------- |
| `publishableKey` | string | ✅        | Your publishable key from Platform Dashboard |

### User Identity

| Option                 | Type   | Required | Description                                                                     |
| ---------------------- | ------ | -------- | ------------------------------------------------------------------------------- |
| `userInfo`             | object | ❌        | User metadata for space creation                                                |
| `userInfo.userId`      | string | ❌        | Unique user identifier                                                          |
| `userInfo.name`        | string | ❌        | User's display name - this is displayed in the dashboard as the associated user |
| `userInfo.companyId`   | string | ❌        | Company identifier                                                              |
| `userInfo.companyName` | string | ❌        | Company display name                                                            |
| `externalActorId`      | string | ❌        | Unique identifier for embedded users                                            |

### Space Setup

| Option          | Type     | Required | Description                               |
| --------------- | -------- | -------- | ----------------------------------------- |
| `name`          | string   | ✅        | Name of the space                         |
| `environmentId` | string   | ✅        | Environment identifier                    |
| `spaceId`       | string   | ❌        | ID of existing space to reuse             |
| `workbook`      | object   | ❌        | Workbook configuration for dynamic spaces |
| `listener`      | Listener | ❌        | Event listener for responding to events   |

### Look & Feel

| Option                            | Type    | Required | Description                                                          |
| --------------------------------- | ------- | -------- | -------------------------------------------------------------------- |
| `themeConfig`                     | object  | ❌        | Theme values for Space, sidebar and data table                       |
| `spaceBody`                       | object  | ❌        | Space options for creating new Space; used with Angular and Vue SDKs |
| `sidebarConfig`                   | object  | ❌        | Sidebar UI configuration                                             |
| `sidebarConfig.defaultPage`       | object  | ❌        | Landing page configuration                                           |
| `sidebarConfig.showDataChecklist` | boolean | ❌        | Toggle data config, defaults to false                                |
| `sidebarConfig.showSidebar`       | boolean | ❌        | Show/hide sidebar                                                    |
| `document`                        | object  | ❌        | Document content for space                                           |
| `document.title`                  | string  | ❌        | Document title                                                       |
| `document.body`                   | string  | ❌        | Document body content (markdown)                                     |

### CSS Customization

You can customize the embedded Flatfile iframe and its container elements using CSS variables and class selectors. This allows you to control colors, sizing, borders, and other visual aspects of the iframe wrapper to match your application's design.

#### CSS Variables

Define these CSS variables in your application's stylesheet to control the appearance of Flatfile's embedded components:

```css theme={null}
:root {
  --ff-primary-color: #4c48ef;
  --ff-secondary-color: #616a7d;
  --ff-text-color: #090b2b;
  --ff-dialog-border-radius: 4px;
  --ff-border-radius: 5px;
  --ff-bg-fade: rgba(0, 0, 0, 0.2);
}
```

#### Container Elements

Target these elements to customize the iframe container:

```css theme={null}
/* The default mount element */
#flatfile_iFrameContainer {
  /* Your custom styles */
}

/* A div around the iframe that contains Flatfile */
.flatfile_iframe-wrapper {
  /* Your custom styles */
}

/* The actual iframe that contains Flatfile */
#flatfile_iframe {
  /* Your custom styles */
}
```

#### Modal Display Customization

When `displayAsModal` is set to `true`, customize the modal appearance:

```css theme={null}
/* Container styles when displayed as modal */
.flatfile_displayAsModal {
  padding: 50px !important;
  width: calc(100% - 100px) !important;
  height: calc(100vh - 100px) !important;
}

.flatfile_iframe-wrapper.flatfile_displayAsModal {
  background: var(--ff-bg-fade);
}

/* Close button styles */
.flatfile_displayAsModal .flatfile-close-button {
  /* Your custom styles */
}

.flatfile_displayAsModal .flatfile-close-button svg {
  fill: var(--ff-secondary-color);
}

/* Iframe border radius when displayed as modal */
.flatfile_displayAsModal #flatfile_iframe {
  border-radius: var(--ff-border-radius);
}
```

#### Exit Confirmation Dialog

Customize the confirmation dialog that appears when closing Flatfile:

```css theme={null}
/* Modal backdrop */
.flatfile_outer-shell {
  background-color: var(--ff-bg-fade);
  border-radius: var(--ff-border-radius);
}

/* Inner container */
.flatfile_inner-shell {
  /* Your custom styles */
}

/* Dialog box */
.flatfile_modal {
  border-radius: var(--ff-dialog-border-radius);
}

/* Button container */
.flatfile_button-group {
  /* Your custom styles */
}

/* All buttons */
.flatfile_button {
  /* Your custom styles */
}

/* Primary "Yes, cancel" button */
.flatfile_primary {
  border: 1px solid var(--ff-primary-color);
  background-color: var(--ff-primary-color);
  color: #fff;
}

/* Secondary "No, stay" button */
.flatfile_secondary {
  color: var(--ff-secondary-color);
}

/* Dialog heading */
.flatfile_modal-heading {
  color: var(--ff-text-color);
}

/* Dialog description text */
.flatfile_modal-text {
  color: var(--ff-secondary-color);
}
```

#### Error Component

Customize the error display component:

```css theme={null}
/* Error container */
.ff_error_container {
  /* Your custom styles */
}

/* Error heading */
.ff_error_heading {
  /* Your custom styles */
}

/* Error description */
.ff_error_text {
  /* Your custom styles */
}
```

### Basic Behavior

| Option                 | Type     | Required | Description                                |
| ---------------------- | -------- | -------- | ------------------------------------------ |
| `closeSpace`           | object   | ❌        | Options for closing iframe                 |
| `closeSpace.operation` | string   | ❌        | Operation type                             |
| `closeSpace.onClose`   | function | ❌        | Callback when space closes                 |
| `displayAsModal`       | boolean  | ❌        | Display as modal or inline (default: true) |

## Advanced Configuration Options

These options provide specialized functionality for custom implementations:

### Space Reuse

| Option        | Type   | Required | Description                                   |
| ------------- | ------ | -------- | --------------------------------------------- |
| `id`          | string | ✅        | Space ID                                      |
| `accessToken` | string | ✅        | Access token for space (obtained server-side) |

**Important:** To reuse an existing space, you must retrieve the spaceId and access token server-side using your secret key, then pass the `accessToken` to the client. See [Server Setup Guide](./server-setup) for details.

### UI Overrides

| Option                    | Type         | Required | Description                                                      |
| ------------------------- | ------------ | -------- | ---------------------------------------------------------------- |
| `mountElement`            | string       | ❌        | Element to mount Flatfile (default: "flatfile\_iFrameContainer") |
| `loading`                 | ReactElement | ❌        | Custom loading component                                         |
| `exitTitle`               | string       | ❌        | Exit dialog title (default: "Close Window")                      |
| `exitText`                | string       | ❌        | Exit dialog text (default: "See below")                          |
| `exitPrimaryButtonText`   | string       | ❌        | Primary button text (default: "Yes, exit")                       |
| `exitSecondaryButtonText` | string       | ❌        | Secondary button text (default: "No, stay")                      |
| `errorTitle`              | string       | ❌        | Error dialog title (default: "Something went wrong")             |

### On-Premises Configuration

| Option     | Type   | Required | Description                                                                                      |
| ---------- | ------ | -------- | ------------------------------------------------------------------------------------------------ |
| `apiUrl`   | string | ❌        | API endpoint (default: "[https://platform.flatfile.com/api](https://platform.flatfile.com/api)") |
| `spaceUrl` | string | ❌        | Spaces API URL (default: "[https://platform.flatfile.com/s](https://platform.flatfile.com/s)")   |

URLs for other regions can be found [here](../reference/cli#regional-servers).

## Configuration Examples

### Basic Space Creation

```javascript theme={null}
const config = {
  publishableKey: "pk_1234567890abcdef",
  name: "Customer Data Import",
  environmentId: "us_env_abc123",
  workbook: {
    // your workbook configuration
  },
  userInfo: {
    userId: "user_123",
    name: "John Doe",
  },
};
```

### Space Reuse with Access Token

```javascript theme={null}
// Client-side: Use space with access token from server
const config = {
  space: {
    id: "us_sp_abc123def456",
    accessToken: "at_1234567890abcdef", // Retrieved server-side
  },
};
```

### Advanced UI Customization

```javascript theme={null}
const config = {
  publishableKey: "pk_1234567890abcdef",
  mountElement: "custom-flatfile-container",
  exitTitle: "Are you sure you want to leave?",
  exitText: "Your progress will be saved.",
  themeConfig: {
    // custom theme configuration
  },
};
```

## Troubleshooting

### Invalid publishableKey

**Error:** `"Invalid publishable key"`

**Solution:**

* Verify key starts with `pk_`
* Check for typos or extra spaces
* Ensure key is from correct environment

### Space Not Found

**Error:** `"Space not found"` or `403 Forbidden`

**Solution:**

* Verify Space ID format (`us_sp_` prefix)
* Ensure Space exists and is active
* Check Space permissions in dashboard

### CORS Issues

**Error:** `"CORS policy blocked"`

**Solution:**

* Add your domain to allowed origins in Platform Dashboard
* Ensure you're using publishable key (not secret key)
* Check browser network tab for specific CORS errors

### Access Token Issues

**Error:** `"Invalid access token"` when using space reuse

**Solution:**

* Ensure access token is retrieved server-side using secret key
* Check that token hasn't expired
* Verify space ID matches the token

## Testing Setup

For development and testing:

```javascript theme={null}
// Development configuration
const config = {
  publishableKey: "pk_test_1234567890abcdef", // publishable key from development environment
};
```

Create separate test Spaces for development to avoid affecting production data.

## Next Steps

Once configured:

* Deploy your event listener to Flatfile
* Configure data validation and transformation rules
* Test the embedding in your application
* Deploy to production with production keys

For server-side space reuse patterns, see our [Server Setup Guide](./server-setup).


# Angular Embedding
Source: https://flatfile.com/docs/embedding/angular

Embed Flatfile in Angular applications

Embed Flatfile in your Angular application using our Angular SDK. This provides Angular components and services for seamless integration.

## Installation

```bash theme={null}
npm install @flatfile/angular-sdk
```

## Basic Implementation

### 1. Import the Module

Add the `SpaceModule` to your Angular module:

```typescript theme={null}
import { NgModule } from "@angular/core";
import { SpaceModule } from "@flatfile/angular-sdk";

@NgModule({
  imports: [
    SpaceModule,
    // your other imports
  ],
  // ...
})
export class AppModule {}
```

### 2. Create Component

Create a component to handle the Flatfile embed:

```typescript theme={null}
import { Component } from "@angular/core";
import { SpaceService, ISpace } from "@flatfile/angular-sdk";

@Component({
  selector: "app-import",
  template: `
    <div>
      <h1>Welcome to our app</h1>
      <button (click)="openFlatfile()">Import Data</button>
    </div>
  `,
})
export class ImportComponent {
  constructor(private spaceService: SpaceService) {}

  spaceProps: ISpace = {
    publishableKey: "pk_your_publishable_key",
    displayAsModal: true,
  };

  openFlatfile() {
    this.spaceService.OpenEmbed(this.spaceProps);
  }
}
```

### 3. Get Your Credentials

**publishableKey**: Get from [Platform Dashboard](https://platform.flatfile.com) → Developer Settings

**Authentication & Security**: For production applications, implement proper authentication and space management on your server. See [Advanced Configuration](./advanced-configuration) for authentication guidance.

## Complete Example

<Note>
  The example below will open an empty space. To create the sheet your users
  should land on, you'll want to create a workbook as shown further down this
  page.
</Note>

```typescript theme={null}
// app.module.ts
import { NgModule } from "@angular/core";
import { BrowserModule } from "@angular/platform-browser";
import { SpaceModule } from "@flatfile/angular-sdk";

import { AppComponent } from "./app.component";

@NgModule({
  declarations: [AppComponent],
  imports: [BrowserModule, SpaceModule],
  providers: [],
  bootstrap: [AppComponent],
})
export class AppModule {}
```

```typescript theme={null}
// app.component.ts
import { Component } from "@angular/core";
import { SpaceService, ISpace } from "@flatfile/angular-sdk";

@Component({
  selector: "app-root",
  template: `
    <div>
      <h1>My Application</h1>
      <button (click)="openFlatfile()">Import Data</button>
    </div>
  `,
})
export class AppComponent {
  constructor(private spaceService: SpaceService) {}

  spaceProps: ISpace = {
    publishableKey: "pk_your_publishable_key",
    displayAsModal: true,
  };

  openFlatfile() {
    this.spaceService.OpenEmbed(this.spaceProps);
  }
}
```

## Creating New Spaces

To create a new Space each time:

1. Add a `workbook` configuration object. Read more about workbooks [here](../core-concepts/workbooks).
2. Optionally [deploy](../core-concepts/listeners) a `listener` for custom data processing. Your listener will contain your validations and transformations

```typescript theme={null}
spaceProps: ISpace = {
  publishableKey: "pk_your_publishable_key",
  workbook: {
    name: "My Import",
    sheets: [
      {
        name: "Contacts",
        slug: "contacts",
        fields: [
          { key: "name", type: "string", label: "Name" },
          { key: "email", type: "string", label: "Email" },
        ],
      },
    ],
  },
  displayAsModal: true,
};
```

For detailed workbook configuration, see the [Workbook API Reference](https://reference.flatfile.com/api-reference/workbooks).

## Reusing Existing Spaces

For production applications, implement proper space management on your server to ensure security and proper access control:

```typescript theme={null}
// Frontend Component
@Component({
  selector: "app-import",
  template: `
    <div>
      <button (click)="openFlatfile()" [disabled]="loading">
        {{ loading ? "Loading..." : "Import Data" }}
      </button>
    </div>
  `,
})
export class ImportComponent {
  loading = false;

  constructor(private spaceService: SpaceService, private http: HttpClient) {}

  async openFlatfile() {
    this.loading = true;

    try {
      // Get space credentials from your server
      const response = await this.http
        .get<{
          publishableKey: string;
          spaceId: string;
          accessToken?: string;
        }>("/api/flatfile/space")
        .toPromise();

      const spaceProps: ISpace = {
        space: {
          spaceId: response.spaceId,
          accessToken: response.accessToken,
        },
        displayAsModal: true,
      };

      this.spaceService.OpenEmbed(spaceProps);
    } catch (error) {
      console.error("Failed to load Flatfile space:", error);
    } finally {
      this.loading = false;
    }
  }
}
```

For server implementation details, see the [Server Setup](/embedding/server-setup) guide.

## Configuration Options

For detailed configuration options, authentication settings, and advanced features, see the [Advanced Configuration](./advanced-configuration) guide.

## Using Space Component Directly

You can also use the `flatfile-space` component directly in your template:

```typescript theme={null}
@Component({
  selector: "app-import",
  template: `
    <flatfile-space
      [spaceProps]="spaceProps"
      [showSpace]="showSpace"
      (closeSpace)="onCloseSpace()"
    >
    </flatfile-space>
    <button (click)="toggleSpace()">
      {{ showSpace ? "Close" : "Open" }} Import
    </button>
  `,
})
export class ImportComponent {
  showSpace = false;

  spaceProps: ISpace = {
    publishableKey: "pk_your_publishable_key",
    displayAsModal: true,
  };

  toggleSpace() {
    this.showSpace = !this.showSpace;
  }

  onCloseSpace() {
    this.showSpace = false;
  }
}
```

## TypeScript Support

The Angular SDK is built with TypeScript and includes full type definitions:

```typescript theme={null}
import { ISpace, SpaceService } from "@flatfile/angular-sdk";

interface ImportData {
  name: string;
  email: string;
}

@Component({
  // component definition
})
export class ImportComponent {
  spaceProps: ISpace;

  constructor(private spaceService: SpaceService) {
    this.spaceProps = {
      publishableKey: "pk_your_publishable_key",
      spaceId: "us_sp_your_space_id",
    };
  }
}
```

## Next Steps

* **Advanced Configuration**: Set up [authentication, listeners, and advanced options](./advanced-configuration)
* **Server Setup**: Implement [backend integration and space management](./server-setup)
* **Data Processing**: Set up Listeners in your Space for custom data transformations
* **API Integration**: Use [Flatfile API](https://reference.flatfile.com) to retrieve processed data
* **Angular SDK Documentation**: See [@flatfile/angular-sdk documentation](https://www.npmjs.com/package/@flatfile/angular-sdk)

## Quick Links

<CardGroup>
  <Card title="Advanced Configuration" icon="gear" href="/embedding/advanced-configuration">
    Authentication, listeners, and advanced options
  </Card>

  <Card title="Server Setup" icon="server" href="/embedding/server-setup">
    Backend integration and space management
  </Card>
</CardGroup>

## Example Projects

<CardGroup>
  <Card title="Angular Example" icon="angular" href="https://github.com/FlatFilers/create-flatfile-angular">
    Complete Angular application with Flatfile embedding
  </Card>
</CardGroup>


# JavaScript Embedding
Source: https://flatfile.com/docs/embedding/javascript

Embed Flatfile using vanilla JavaScript

Embed Flatfile directly into any web application using our JavaScript SDK. This approach works with any framework or vanilla JavaScript setup.

## Quick Links

* [Advanced Configuration](./advanced-configuration) - Complete configuration options reference
* [Server Setup](./server-setup) - Server-side setup for space reuse

## Installation

Add the Flatfile JavaScript SDK to your page:

```html theme={null}
<script src="https://unpkg.com/@flatfile/javascript/dist/index.js"></script>
```

Or install via npm:

```bash theme={null}
npm install @flatfile/javascript
```

## Basic Implementation

### 1. HTML Structure

Create a button to trigger the Flatfile embed:

```html theme={null}
<!DOCTYPE html>
<html>
  <head>
    <title>Data Import</title>
    <script src="https://unpkg.com/@flatfile/javascript/dist/index.js"></script>
  </head>
  <body>
    <button id="import-btn">Import Data</button>

    <script>
      // Implementation goes here
    </script>
  </body>
</html>
```

### 2. Initialize Flatfile

```javascript theme={null}
document.getElementById("import-btn").addEventListener("click", function () {
  window.FlatFileJavaScript.startFlatfile({});
});
```

### 3. Get Your Credentials

**publishableKey**: Get from [Platform Dashboard](https://platform.flatfile.com) → Developer Settings

```javascript theme={null}
document.getElementById("import-btn").addEventListener("click", function () {
  window.FlatFileJavaScript.startFlatfile({
    publishableKey: "pk_your_publishable_key",
  });
});
```

For detailed authentication and security guidance, see [Advanced Configuration](./advanced-configuration).

## Complete Example

<Note>
  The example below will open an empty space. To create the sheet your users
  should land on, you'll want to create a workbook as shown further down this
  page.
</Note>

```html theme={null}
<!DOCTYPE html>
<html>
  <head>
    <title>Data Import</title>
    <script src="https://unpkg.com/@flatfile/javascript/dist/index.js"></script>
  </head>
  <body>
    <h1>Welcome to our app</h1>
    <button id="import-btn">Import Data</button>

    <script>
      document
        .getElementById("import-btn")
        .addEventListener("click", function () {
          window.FlatFileJavaScript.startFlatfile({
            publishableKey: "pk_your_publishable_key",
            displayAsModal: true,
          });
        });
    </script>
  </body>
</html>
```

## Creating New Spaces

To create a new Space each time:

1. Add a `workbook` configuration object. Read more about workbooks [here](../core-concepts/workbooks).
2. Optionally [deploy](../core-concepts/listeners) a `listener` for custom data processing. Your listener will contain your validations and transformations

<Note>
  You can also deploy and create your workbook via a server-side listener as
  shown in the above link, rather than passing it in as a client-side property.
</Note>

```javascript theme={null}
window.FlatFileJavaScript.startFlatfile({
  publishableKey: "pk_your_publishable_key",
  workbook: {
    name: "My Import",
    sheets: [
      {
        name: "Contacts",
        slug: "contacts",
        fields: [
          { key: "name", type: "string", label: "Name" },
          { key: "email", type: "string", label: "Email" },
        ],
      },
    ],
  },
});
```

For detailed workbook configuration, see the [Workbook API Reference](https://reference.flatfile.com/api-reference/workbooks).

## Configuration Options

For complete configuration options including user identity, theming, and advanced features, see [Advanced Configuration](./advanced-configuration).

## Reusing Existing Spaces

To reuse an existing space, you need to set up a server-side component that retrieves the space access token. The basic client-side approach shown above only works for initial space creation.

For space reuse, you'll need:

1. **Server-side setup**: Use your secret key to retrieve the space and its access token
2. **Client-side update**: Use the access token instead of just the publishable key

```javascript theme={null}
// Client-side: Get space data from your server
const spaceData = await fetch("/api/spaces/us_sp_your_space_id").then((r) =>
  r.json()
);

// Use the space with access token
window.FlatFileJavaScript.startFlatfile({
  space: {
    id: spaceData.id,
    accessToken: spaceData.accessToken, // Retrieved from server
  },
});
```

For complete server-side implementation, see [Server Setup](./server-setup).

## Next Steps

* **Advanced Configuration**: See [Advanced Configuration](./advanced-configuration) for complete options reference
* **Server Setup**: Set up [Server Setup](./server-setup) for space reuse functionality
* **Styling**: Customize the embedded experience in your Platform Dashboard Space settings
* **Data Processing**: Set up Listeners in your Space for custom data transformations
* **API Integration**: Use [Flatfile API](https://reference.flatfile.com) to retrieve processed data

## Example Projects

<CardGroup>
  <Card title="Basic JavaScript Example" icon="js" href="https://github.com/FlatFilers/create-flatfile-javascript">
    Complete working example with HTML and JavaScript
  </Card>
</CardGroup>


# Overview
Source: https://flatfile.com/docs/embedding/overview

Embed Flatfile data import directly into your application

Flatfile can be embedded directly into your web application, allowing users to import data without leaving your interface. This creates a seamless experience where data import feels like a native part of your app.

## When to Embed Flatfile

Embed Flatfile when you want to:

* **Seamless user experience** - Keep users within your application flow
* **Custom branding** - Maintain your application's look and feel
* **Integrated workflows** - Connect data import directly to your business logic
* **Control over timing** - Trigger imports at the right moment in your user journey

## How Embedding Works

Flatfile embedding uses an iframe-based approach that loads the Flatfile interface within your application. You provide credentials and configuration to load a pre-configured Space or create a new Space with your data schema and workflows.

The embedded experience handles file upload, data mapping, validation, and transformation automatically based on your Space configuration.

## Available SDKs

Choose the SDK that matches your frontend framework:

<CardGroup>
  <Card title="JavaScript" icon="js" href="/embedding/javascript">
    Vanilla JavaScript for any web application
  </Card>

  <Card title="React" icon="react" href="/embedding/react">
    React components and hooks
  </Card>

  <Card title="Angular" icon="angular" href="/embedding/angular">
    Angular components and services
  </Card>

  <Card title="Vue" icon="vuejs" href="/embedding/vue">
    Vue.js components and composables
  </Card>
</CardGroup>

## Getting Started

1. **Create a Space** in the [Platform Dashboard](https://platform.flatfile.com) with your data schema
2. **Get your credentials** from Developer Settings
3. **Choose your SDK** and follow the integration guide

## Configuration & Setup

<CardGroup>
  <Card title="Advanced Configuration" icon="gear" href="/embedding/advanced-configuration">
    Complete configuration options reference
  </Card>

  <Card title="Server Setup" icon="server" href="/embedding/server-setup">
    Server-side setup for space reuse
  </Card>
</CardGroup>

## Two Approaches: Existing vs Dynamic Spaces

These guides show both approaches:

* **Existing Spaces**: Connect to a pre-existing Space
* **Dynamic Spaces**: Create a new Space programmatically

Choose the approach that best fits your workflow. See the implementation instructions for both patterns in each SDK guide.

For advanced configuration options like themes, custom workflows, and data transformations, see [Advanced Configuration](./advanced-configuration) or configure these in your Space through the Platform Dashboard.


# React Embedding
Source: https://flatfile.com/docs/embedding/react

Embed Flatfile in React applications

Embed Flatfile in your React application using our React SDK. This provides React components and hooks for seamless integration.

## Installation

```bash theme={null}
npm install @flatfile/react
```

## Basic Implementation

### 1. Wrap Your App

Wrap your application with the `FlatfileProvider`:

```jsx theme={null}
import { FlatfileProvider } from "@flatfile/react";

function App() {
  return (
    <FlatfileProvider publishableKey="pk_your_publishable_key">
      <YourApp />
    </FlatfileProvider>
  );
}
```

### 2. Add Import Trigger

Use the `useFlatfile` hook to open Flatfile:

```jsx theme={null}
import { useFlatfile } from "@flatfile/react";

function YourComponent() {
  const { openPortal } = useFlatfile();

  const handleImport = () => {
    openPortal({});
  };

  return (
    <div>
      <h1>Welcome to our app</h1>
      <button onClick={handleImport}>Import Data</button>
    </div>
  );
}
```

### 3. Get Your Credentials

**publishableKey**: Get from [Platform Dashboard](https://platform.flatfile.com) → Developer Settings

For authentication guidance including JWT tokens and user management, see [Advanced Configuration](./advanced-configuration).

## Complete Example

<Note>
  The example below will open an empty space. To create the sheet your users
  should land on, you'll want to create a workbook as shown further down this
  page.
</Note>

```jsx theme={null}
import React from "react";
import { FlatfileProvider, useFlatfile } from "@flatfile/react";

function ImportButton() {
  const { openPortal } = useFlatfile();

  const handleImport = () => {
    openPortal({});
  };

  return <button onClick={handleImport}>Import Data</button>;
}

function App() {
  return (
    <FlatfileProvider publishableKey="pk_your_publishable_key">
      <div>
        <h1>My Application</h1>
        <ImportButton />
      </div>
    </FlatfileProvider>
  );
}

export default App;
```

## Making API Requests

<Info>
  The `publishableKey` is designed exclusively for initializing `FlatfileProvider`. To make API requests, use the `accessToken` from the `useFlatfileInternal` hook.
</Info>

Use `useFlatfileInternal` to access the `accessToken` for making authenticated API calls:

```jsx theme={null}
import { useEffect, useState } from "react";
import { useFlatfileInternal } from "@flatfile/react";
import { FlatfileClient } from "@flatfile/api";

function MyComponent() {
  const { accessToken, sessionSpace } = useFlatfileInternal();
  const [data, setData] = useState([]);

  useEffect(() => {
    // Wait for the space to be created and token to be available
    if (!accessToken || !sessionSpace?.id) return;

    // Initialize the API client with the accessToken
    const api = new FlatfileClient({ token: accessToken });

    const fetchData = async () => {
      try {
        const workbooks = await api.workbooks.list({ 
          spaceId: sessionSpace.id 
        });
        setData(workbooks.data);
      } catch (error) {
        console.error("API Error:", error);
      }
    };

    fetchData();
  }, [accessToken, sessionSpace?.id]);

  return <div>{data.length} workbooks found</div>;
}
```

For a complete guide on authentication and making API requests, see the [React Authentication Guide](/sdks/react/authentication).

## Configuration Options

For all configuration options including authentication, theming, and advanced settings, see [Advanced Configuration](./advanced-configuration).

## Using Space Component

For more control, you can use the `Space` component directly:

```jsx theme={null}
import { FlatfileProvider, Space } from "@flatfile/react";

function App() {
  return (
    <FlatfileProvider
      publishableKey="pk_your_publishable_key"
      config={{ displayAsModal: true }}
    >
      <Space config={{ name: "Embedded Space" }} />
    </FlatfileProvider>
  );
}
```

## Creating New Spaces

To create a new Space each time:

1. Add a `workbook` configuration object. Read more about workbooks [here](../core-concepts/workbooks).
2. Optionally [deploy](../core-concepts/listeners) a `listener` for custom data processing. Your listener will contain your validations and transformations

```jsx theme={null}
import { FlatfileProvider, Workbook } from "@flatfile/react";

const workbookConfig = {
  name: "My Import",
  sheets: [
    {
      name: "Contacts",
      slug: "contacts",
      fields: [
        { key: "name", type: "string", label: "Name" },
        { key: "email", type: "string", label: "Email" },
      ],
    },
  ],
};

function App() {
  return (
    <FlatfileProvider publishableKey="pk_your_publishable_key">
      <Workbook config={workbookConfig} />
    </FlatfileProvider>
  );
}
```

For detailed workbook configuration, see the [Workbook API Reference](https://reference.flatfile.com/api-reference/workbooks).

## Reusing Existing Spaces

For production applications, you should create Spaces on your server and pass the Space ID to your React application:

### Server-side (Node.js/Express)

```javascript theme={null}
// Create Space on your server
const space = await api.spaces.create({
  name: "User Import",
  workbooks: [workbookConfig],
});

// Pass spaceId to your React app
res.json({ spaceId: space.data.id, token: space.data.accessToken });
```

### Client-side (React)

```jsx theme={null}
import { FlatfileProvider, useFlatfile } from "@flatfile/react";

function ImportComponent() {
  const { openPortal } = useFlatfile();
  const [spaceId, setSpaceId] = useState(null);

  useEffect(() => {
    // Get Space ID from your server
    fetch("/api/create-space")
      .then((res) => res.json())
      .then((data) => setSpaceId(data.spaceId))
      .then((data) => setToken(data.token));
  }, []);

  <FlatfileProvider accessToken="token">
    <Space id="spaceId">
  </FlatfileProvider>

  const handleImport = () => {
    if (spaceId) {
      openPortal({});
    }
  };

  return <button onClick={handleImport}>Import Data</button>;
}
```

This approach allows you to:

* Create Spaces with server-side authentication
* Add custom event listeners and data processing
* Control Space lifecycle and cleanup

For complete server setup examples, see [Server Setup](/embedding/server-setup).

## TypeScript Support

The React SDK includes full TypeScript support:

```tsx theme={null}
import { FlatfileProvider, useFlatfile } from "@flatfile/react";

interface Props {
  onDataImported?: (data: any) => void;
}

function ImportComponent({ onDataImported }: Props) {
  const { openPortal } = useFlatfile();

  const handleImport = (): void => {
    openPortal({});
  };

  return <button onClick={handleImport}>Import Data</button>;
}
```

## Next Steps

* **React Authentication**: [Complete guide to using accessToken for API requests](/sdks/react/authentication)
* **Advanced Configuration**: [Authentication, theming, and advanced options](./advanced-configuration)
* **Server Setup**: [Backend integration and event handling](./server-setup)
* **API Integration**: Use [Flatfile API](https://reference.flatfile.com) to retrieve processed data
* **Package Documentation**: See [@flatfile/react documentation](https://www.npmjs.com/package/@flatfile/react)

## Quick Links

<CardGroup>
  <Card title="React Authentication" icon="key" href="/sdks/react/authentication">
    Using accessToken for secure API requests
  </Card>

  <Card title="Advanced Configuration" icon="gear" href="/embedding/advanced-configuration">
    Authentication, theming, and advanced options
  </Card>

  <Card title="Server Setup" icon="server" href="/embedding/server-setup">
    Backend integration and event handling
  </Card>
</CardGroup>

## Example Projects

<CardGroup>
  <Card title="React Example" icon="react" href="https://github.com/FlatFilers/create-flatfile-react">
    Complete React application with Flatfile embedding
  </Card>
</CardGroup>


# Server Setup for Space Reuse
Source: https://flatfile.com/docs/embedding/server-setup

Server-side implementation guide for reusing existing spaces

To reuse existing spaces with embedded Flatfile, you need a server-side component that retrieves space access tokens using your secret key. This guide shows you how to set up the server-side pattern correctly.

## Why Server-Side Setup is Required

When reusing an existing space, you cannot use your publishable key directly. Instead, you must:

1. **Server-side**: Use your secret key to retrieve the space and its access token
2. **Client-side**: Use the access token (not publishable key) to launch the space

This pattern ensures security by keeping your secret key on the server while providing the necessary access token to the client.

## Server-Side Implementation

### Environment Setup

Create a `.env` file with your credentials:

```bash theme={null}
# .env
FLATFILE_API_KEY=sk_1234567890abcdef  # Your secret key
SPACE_ID=us_sp_abc123def456           # Space ID to reuse
BASE_URL=http://localhost:3000        # Your application URL
```

### Basic Server Example

Here's a Node.js/Express server that retrieves space access tokens:

```javascript theme={null}
// server.js
import express from "express";
import cors from "cors";
import { FlatfileClient } from "@flatfile/api";
import dotenv from "dotenv";

dotenv.config();

const app = express();
const PORT = process.env.PORT || 3001;

// Initialize Flatfile API with secret key
const flatfile = new FlatfileClient({
  token: process.env.FLATFILE_API_KEY, // Secret key
});

// Enable CORS for your frontend
app.use(
  cors({
    origin: process.env.BASE_URL || "http://localhost:3000",
  })
);

app.use(express.json());

// Endpoint to retrieve space with access token
app.get("/api/spaces/:spaceId", async (req, res) => {
  try {
    const { spaceId } = req.params;

    // Retrieve space using secret key
    const space = await flatfile.spaces.get(spaceId);

    // Return space data including access token
    res.json({
      success: true,
      data: space.data, // Contains both id and accessToken
    });
  } catch (error) {
    console.error("Error retrieving space:", error);
    res.status(500).json({
      success: false,
      error: "Failed to retrieve space",
    });
  }
});

// Health check endpoint
app.get("/health", (req, res) => {
  res.json({ status: "OK" });
});

app.listen(PORT, () => {
  console.log(`Server running on port ${PORT}`);
});
```

### Enhanced Server with Multiple Spaces

For applications that need to handle multiple spaces:

```javascript theme={null}
// enhanced-server.js
import express from "express";
import cors from "cors";
import { FlatfileClient } from "@flatfile/api";
import dotenv from "dotenv";

dotenv.config();

const app = express();
const flatfile = new FlatfileClient({
  token: process.env.FLATFILE_API_KEY,
});

app.use(
  cors({
    origin: process.env.BASE_URL || "http://localhost:3000",
  })
);

app.use(express.json());

// Get space by ID
app.get("/api/spaces/:spaceId", async (req, res) => {
  try {
    const { spaceId } = req.params;
    const space = await flatfile.spaces.get(spaceId);

    res.json({
      success: true,
      data: {
        id: space.data.id,
        name: space.data.name,
        accessToken: space.data.accessToken,
      },
    });
  } catch (error) {
    res.status(500).json({
      success: false,
      error: "Failed to retrieve space",
    });
  }
});

// List available spaces
app.get("/api/spaces", async (req, res) => {
  try {
    const spaces = await flatfile.spaces.list();

    res.json({
      success: true,
      data: spaces.data.map((space) => ({
        id: space.id,
        name: space.name,
        createdAt: space.createdAt,
      })),
    });
  } catch (error) {
    res.status(500).json({
      success: false,
      error: "Failed to list spaces",
    });
  }
});

// Create new space (optional)
app.post("/api/spaces", async (req, res) => {
  try {
    const { name, environmentId } = req.body;

    const space = await flatfile.spaces.create({
      name,
      environmentId,
      // Additional space configuration
    });

    res.json({
      success: true,
      data: {
        id: space.data.id,
        name: space.data.name,
        accessToken: space.data.accessToken,
      },
    });
  } catch (error) {
    res.status(500).json({
      success: false,
      error: "Failed to create space",
    });
  }
});

const PORT = process.env.PORT || 3001;
app.listen(PORT, () => {
  console.log(`Server running on port ${PORT}`);
});
```

## Client-Side Implementation

### Fetching Space Data

Your client application should fetch the space data from your server:

```javascript theme={null}
// client.js
async function getSpaceData(spaceId) {
  try {
    const response = await fetch(`/api/spaces/${spaceId}`);
    const result = await response.json();

    if (!result.success) {
      throw new Error(result.error);
    }

    return result.data; // Contains id and accessToken
  } catch (error) {
    console.error("Failed to fetch space data:", error);
    throw error;
  }
}
```

### Using the Space Data

Once you have the space data, use it to initialize Flatfile:

```javascript theme={null}
window.openFlatfile = () => {
  fetch(server_url + "/space") // Make a request to the server endpoint
    .then((response) => response.json())
    .then((space) => {
      const flatfileOptions = {
        space: {
          id: space && space.data && space.data.id,
          accessToken: space && space.data && space.data.accessToken,
        },
        sidebarConfig: {
          showSidebar: false,
        },
        // Additional props...
      };
      initializeFlatfile(flatfileOptions);
    })
    .catch((error) => {
      console.error("Error retrieving space in client:", error);
    });
};
```

## Complete Workflow Example

Here's how the complete workflow works:

### 1. Server Setup

```javascript theme={null}
// server.js
app.get("/api/spaces/:spaceId", async (req, res) => {
  const space = await flatfile.spaces.get(req.params.spaceId);
  res.json({ data: space.data });
});
```

### 2. Client Request

```javascript theme={null}
// client.js
const response = await fetch("/api/spaces/us_sp_abc123def456");
const { data } = await response.json();
```

### 3. Client Usage

```javascript theme={null}
const flatfileOptions = {
  space: {
    id: space && space.data && space.data.id,
    accessToken: space && space.data && space.data.accessToken,
  },
  sidebarConfig: {
    showSidebar: false,
  },
  // Additional props...
};
initializeFlatfile(flatfileOptions);
```

## Security Considerations

### Environment Variables

Never expose your secret key in client-side code:

```javascript theme={null}
// ✅ Good - server-side only
const flatfile = new FlatfileClient({
  token: process.env.FLATFILE_API_KEY, // Secret key on server
});

// ❌ Bad - never do this in client code
const flatfile = new FlatfileClient({
  token: "sk_1234567890abcdef", // Secret key exposed
});
```

### CORS Configuration

Configure CORS to only allow your frontend domain:

```javascript theme={null}
app.use(
  cors({
    origin: process.env.BASE_URL, // Only your frontend
    credentials: true,
  })
);
```

### Access Token Handling

* Access tokens are temporary and space-specific
* They should be fetched fresh for each session
* Store them securely in your client application

## Error Handling

### Server-Side Errors

```javascript theme={null}
app.get("/api/spaces/:spaceId", async (req, res) => {
  try {
    const space = await flatfile.spaces.get(req.params.spaceId);
    res.json({ success: true, data: space.data });
  } catch (error) {
    if (error.status === 404) {
      res.status(404).json({
        success: false,
        error: "Space not found",
      });
    } else if (error.status === 403) {
      res.status(403).json({
        success: false,
        error: "Access denied",
      });
    } else {
      res.status(500).json({
        success: false,
        error: "Server error",
      });
    }
  }
});
```

### Client-Side Errors

```javascript theme={null}
async function getSpaceData(spaceId) {
  try {
    const response = await fetch(`/api/spaces/${spaceId}`);

    if (!response.ok) {
      throw new Error(`HTTP ${response.status}: ${response.statusText}`);
    }

    const result = await response.json();

    if (!result.success) {
      throw new Error(result.error);
    }

    return result.data;
  } catch (error) {
    console.error("Failed to fetch space data:", error);
    throw error;
  }
}
```

## Testing

### Local Development

1. Start your server: `node server.js`
2. Test the endpoint: `curl http://localhost:3001/api/spaces/us_sp_abc123def456`
3. Verify the response includes `id` and `accessToken`

### Production Deployment

1. Set environment variables on your server
2. Deploy your server application
3. Update your client to use the production server URL
4. Test the complete workflow

## Next Steps

Once your server is set up:

* Update your client applications to use the server endpoint
* Test the space reuse functionality
* Monitor server logs for any issues
* Consider adding authentication for your server endpoints

For client-side SDK implementation, see the framework-specific guides:

* [JavaScript SDK](./javascript)
* [React SDK](./react)
* [Vue SDK](./vue)
* [Angular SDK](./angular)


# Vue Embedding
Source: https://flatfile.com/docs/embedding/vue

Embed Flatfile in Vue.js applications

Embed Flatfile in your Vue.js application using our Vue SDK. This provides Vue components and composables for seamless integration.

## Installation

```bash theme={null}
npm install @flatfile/vue
```

## Basic Implementation

### 1. Initialize Flatfile

Use the `initializeFlatfile` composable in your Vue component:

```vue theme={null}
<script setup>
import { ref } from "vue";
import { initializeFlatfile } from "@flatfile/vue";

const showSpace = ref(false);

const spaceProps = {
  publishableKey: "pk_your_publishable_key",
};

const { Space, OpenEmbed } = initializeFlatfile(spaceProps);

const openImport = () => {
  showSpace.value = true;
  OpenEmbed();
};
</script>

<template>
  <div>
    <h1>Welcome to our app</h1>
    <button @click="openImport">Import Data</button>

    <div v-if="showSpace">
      <Space />
    </div>
  </div>
</template>
```

### 2. Get Your Credentials

**publishableKey**: Get from [Platform Dashboard](https://platform.flatfile.com) → Developer Settings

For authentication and security guidance, see [Advanced Configuration](./advanced-configuration).

## Complete Example

<Note>
  The example below will open an empty space. To create the sheet your users
  should land on, you'll want to create a workbook as shown further down this
  page.
</Note>

```vue theme={null}
<script setup>
import { ref } from "vue";
import { initializeFlatfile } from "@flatfile/vue";

const showSpace = ref(false);

const spaceProps = {
  publishableKey: "pk_your_publishable_key",
  closeSpace: {
    onClose: () => {
      showSpace.value = false;
    },
  },
};

const { Space, OpenEmbed } = initializeFlatfile(spaceProps);

const toggleImport = () => {
  showSpace.value = !showSpace.value;
  if (showSpace.value) {
    OpenEmbed();
  }
};
</script>

<template>
  <div>
    <h1>My Application</h1>
    <button @click="toggleImport">
      {{ showSpace ? "Close" : "Open" }} Import
    </button>

    <div v-if="showSpace" class="space-wrapper">
      <Space />
    </div>
  </div>
</template>

<style>
.space-wrapper {
  margin-top: 20px;
}
</style>
```

## Options API Syntax

If you prefer the Options API:

```vue theme={null}
<script>
import { initializeFlatfile } from "@flatfile/vue";

export default {
  data() {
    return {
      showSpace: false,
    };
  },
  setup() {
    const spaceProps = {
      publishableKey: "pk_your_publishable_key",
      spaceId: "us_sp_your_space_id",
    };

    const { Space, OpenEmbed } = initializeFlatfile(spaceProps);

    return {
      Space,
      OpenEmbed,
    };
  },
  methods: {
    openImport() {
      this.showSpace = true;
      this.OpenEmbed();
    },
  },
};
</script>

<template>
  <div>
    <button @click="openImport">Import Data</button>
    <div v-if="showSpace">
      <Space />
    </div>
  </div>
</template>
```

## Creating New Spaces

To create a new Space each time:

1. Add a `workbook` configuration object. Read more about workbooks [here](../core-concepts/workbooks).
2. Optionally [deploy](../core-concepts/listeners) a `listener` for custom data processing. Your listener will contain your validations and transformations

```vue theme={null}
<script setup>
import { initializeFlatfile } from "@flatfile/vue";

const spaceProps = {
  publishableKey: "pk_your_publishable_key",
  workbook: {
    name: "Contacts Import",
    sheets: [
      {
        name: "Contacts",
        slug: "contacts",
        fields: [
          { key: "name", type: "string", label: "Name" },
          { key: "email", type: "string", label: "Email" },
        ],
      },
    ],
  },
};

const { Space, OpenEmbed } = initializeFlatfile(spaceProps);
</script>
```

For detailed workbook configuration, see the [Workbook API Reference](https://reference.flatfile.com/api-reference/workbooks).

## Reusing Existing Spaces

When reusing existing Spaces, the proper pattern is to let your server handle Space creation and management:

```vue theme={null}
<script setup>
import { ref } from "vue";
import { initializeFlatfile } from "@flatfile/vue";

const showSpace = ref(false);

// Client-side initialization with existing Space
const spaceProps = {
  space: {
    spaceId: "us_sp_yourSpaceId",
    token: "accessToken",
  },
};

const { Space, OpenEmbed } = initializeFlatfile(spaceProps);

const openImport = () => {
  showSpace.value = true;
  OpenEmbed();
};
</script>
```

For server-side Space creation and management patterns, see [Server Setup](/embedding/server-setup).

## TypeScript Support

The Vue SDK supports TypeScript:

```vue theme={null}
<script setup lang="ts">
import { ref, Ref } from "vue";
import { initializeFlatfile } from "@flatfile/vue";

interface SpaceConfig {
  publishableKey: string;
}

const showSpace: Ref<boolean> = ref(false);

const spaceProps: SpaceConfig = {
  publishableKey: "pk_your_publishable_key",
};

const { Space, OpenEmbed } = initializeFlatfile(spaceProps);
</script>
```

## Styling

Add custom styles to integrate with your Vue application:

```vue theme={null}
<style scoped>
.import-button {
  background-color: #4c48ef;
  color: white;
  border: none;
  padding: 12px 24px;
  border-radius: 6px;
  cursor: pointer;
}

.import-button:hover {
  background-color: #3f3ccc;
}

.space-container {
  margin-top: 20px;
  border: 1px solid #e0e0e0;
  border-radius: 8px;
  overflow: hidden;
}
</style>
```

## Configuration Options

For complete configuration options including authentication, theming, and advanced settings, see [Advanced Configuration](./advanced-configuration).

## Next Steps

* **Authentication**: Set up secure authentication with [Advanced Configuration](./advanced-configuration)
* **Server Setup**: Configure backend data processing with [Server Setup](./server-setup)
* **Styling**: Customize the embedded experience in your Platform Dashboard Space settings
* **API Integration**: Use [Flatfile API](https://reference.flatfile.com) to retrieve processed data

## Quick Links

<CardGroup>
  <Card title="Advanced Configuration" icon="gear" href="/embedding/advanced-configuration">
    Authentication, theming, and advanced options
  </Card>

  <Card title="Server Setup" icon="server" href="/embedding/server-setup">
    Backend integration and data processing
  </Card>
</CardGroup>

## Example Projects

<CardGroup>
  <Card title="Vue Example" icon="vuejs" href="https://github.com/FlatFilers/create-flatfile-vuejs">
    Complete Vue.js application with Flatfile embedding
  </Card>
</CardGroup>


# Getting Started with AutoBuild
Source: https://flatfile.com/docs/getting-started/quickstart/autobuild

Get up and running with Flatfile in minutes using AutoBuild to create a complete data import solution

## What is AutoBuild?

The easiest way to get started with Flatfile is using AutoBuild.

With AutoBuild, you can transform existing import templates or documentation into a fully
functional Flatfile app in minutes. Simply drop your example files into AutoBuild, and it will automatically create and deploy a [Blueprint](/core-concepts/blueprints) (for schema definition) and a [Listener](/core-concepts/listeners) (for validations and transformations) to your Flatfile [App](/core-concepts/apps).

Once you've started with AutoBuild, you can always download your Listener code and continue building with code from there!

## Setting Up Your Account

To get started, you'll need to [sign up for a Flatfile account](https://platform.flatfile.com/oauth/login).

During account setup, enter your company name and select "Start with an existing template or project file."

<Frame>
  <img />
</Frame>

<Accordion title="Using AutoBuild in an existing Flatfile account">
  If you already have an active Flatfile account, you can still use AutoBuild to create a new app.

  From the Flatfile dashboard, click the "New App" button.

  <Frame>
    <img />
  </Frame>

  Then select "Build with AutoBuild."

  <Note>
    If the AutoBuild option isn't available on your account, please reach out to
    support via [Slack](https://flatfile.com/join-slack/) or
    [Email](mailto:support@flatfile.com) to gain access!
  </Note>
</Accordion>

## Uploading Files and Context

Next, you'll upload files and provide additional context to the AutoBuild agent.

You can upload any of the following to help the AI understand your requirements:

* Import templates
* System documentation
* Complete data files
* Any other files that provide useful context

You may also provide an additional prompt to guide the AutoBuild agent. Use this to give
context about your uploaded files, explain specific data challenges, or outline additional requirements.

When you're ready, click "Get Started." The Flatfile AutoBuild agent will now build your space template.

## Working in Build Mode

After a few moments, you'll be taken to your new Flatfile app in Build Mode, which you can
access anytime to make changes.

On the right side, you'll see the blueprint of your space. Here you can inspect and edit the sheets
and fields that the AutoBuild agent has generated. You can easily add or remove fields,
update constraints and validations, or make other basic edits to your blueprint.

<Frame>
  <img />
</Frame>

For more advanced changes, you can chat with the Flatfile Assistant. The Assistant can help you
with anything from small tweaks to complex validations, data egress actions, or large reorganization
of your sheets.

<Frame>
  <img />
</Frame>

At any point, you can check the Data Preview tab to see what your Flatfile project will look like for
your users. You can add or edit data to test your validations and transformations.

<Frame>
  <img />
</Frame>

## Deploying Your App

When you're finished building your space, click "Configure & Deploy."

You'll be prompted to give your app a name, and then it's ready to be deployed!

From here, you'll be taken to your new app in the dashboard.

Your autobuild agent is deployed and you're ready to create your first project and start importing data!


# Creating Your First Project
Source: https://flatfile.com/docs/getting-started/quickstart/first-project

Running through the import flow with your Flatfile App

<Note>
  If you haven't created an app yet, start with the [Building Your First App
  with AutoBuild guide](/getting-started/quickstart/autobuild).
</Note>

## Understanding Projects

With your app created, it's time to create your first data import project!

A project is an instance of your app - think of your app as a template for onboarding customers,
and you'll create a project for each customer you onboard.
Each project gives you an isolated workspace with its own database, permissions, and workflow.

## Creating a New Project

To create a project, click the create button in the upper right corner of the
app page in your dashboard. The text in this button will depend on how you've named
your app's entity.

<Frame>
  <img />
</Frame>

The AutoBuild agent will create a new space based on your app's template.

You'll be taken to your new space, ready to receive data.

<Frame>
  <img />
</Frame>

If you'd like, you can manually import data using Flatfile's intuitive spreadsheet interface.

## Uploading and Mapping Data

Most users will start by uploading a data file.

You can drag and drop a file from your computer directly onto the sheet interface. The file will
automatically be extracted and you'll be taken to the mapping interface.

<Frame>
  <img />
</Frame>

Here you can map columns from your uploaded file to fields in your source sheet. With your
mappings in place, click "Continue."

The data from your file will be mapped into your sheet.

<Frame>
  <img />
</Frame>

## Learn More

Now that you've created your first project, explore these helpful resources:

* [Core Concepts](/core-concepts/overview) - Understand Flatfile's fundamental building blocks
* [Handling Data](/guides/using-record-hooks) - Advanced data transformation and validation techniques
* [Using Actions](/guides/using-actions) - Create custom workflows and automations
* [API Reference](/api-reference) - Complete technical documentation

## Working with Your Data

From here, any transformations and validations you've defined will run automatically. You can resolve
any data issues in your sheet using AI transforms, find and replace, custom
actions, or by simply editing the data manually. You can collaborate with others on
data issues using comments and data clips.

## Exporting Your Data

When you're ready to move your perfected data out of Flatfile, you've got options!

You can download your data directly from the sheets interface. You can retrieve the sheet data
via API. Or, you can create a custom action to ship the data directly into your system.


# Accepting Additional Fields
Source: https://flatfile.com/docs/guides/accepting-additional-fields

Create additional fields on the fly

The `allowAdditionalFields` feature offers a fluid integration experience, allowing users to effortlessly map to new or unconfigured fields in your Blueprints.

## How it works

* By enabling `allowAdditionalFields`, your Sheet isn't restricted to the initial configuration. It can adapt to include new fields, whether they're anticipated or not.
* These supplementary fields can either be added through API calls or input directly by users during the file import process.
* To ensure clarity, any field that wasn't part of the original Blueprint configuration is flagged with a `treatment` property labeled `user-defined`.
* When adding a custom field, there's no need to fuss over naming the field. The system intuitively adopts the header name from the imported file, streamlining the process.

In essence, the `allowAdditionalFields` feature is designed for scalability and ease, ensuring your Blueprints are always ready for unexpected data fields.

## Example Blueprint w/ `allowAdditionalFields`

```json theme={null}
{
  "sheets": [
    {
      "name": "Contacts",
      "slug": "contacts",
      "allowAdditionalFields": true,
      "fields": [
        {
          "key": "firstName",
          "label": "First Name",
          "type": "string"
        },
        {
          "key": "lastName",
          "label": "Last Name",
          "type": "string"
        },
        {
          "key": "email",
          "label": "Email",
          "type": "string"
        }
      ]
    }
  ]
}
```


# Advanced Filters
Source: https://flatfile.com/docs/guides/advanced-filters

Learn how to use Flatfile's Advanced Filters to efficiently filter and search through your data

Advanced Filters in Flatfile provide a powerful way to filter and search through your data with complex conditions. This feature allows you to create sophisticated filter combinations to quickly find the exact records you need.

## Overview

The Advanced Filters feature enables you to:

* Create multiple filter conditions with different fields
* Combine conditions using logical operators (AND/OR)
* Filter by various data types with appropriate operators
* Save and reuse filter combinations
* Apply filters to large datasets efficiently

## Using Advanced Filters

### Accessing Advanced Filters

You can access Advanced Filters in the Flatfile interface through the Filter button in the sheet toolbar:

1. Navigate to any sheet in your workbook
2. Click the "Filter" button in the toolbar
3. Select a field to filter by, or click "Advanced filter" to create a complex filter

### Creating Filter Conditions

Each filter condition consists of three parts:

1. **Field** - The column you want to filter on
2. **Operator** - The comparison type (equals, contains, greater than, etc.)
3. **Value** - The specific value to filter by

For example, you might create a filter like: `firstName is "John"` or `age > 30`.

### Combining Multiple Filters

Advanced Filters allow you to combine multiple conditions:

1. Create your first filter condition
2. Click the "Add condition" button
3. Select whether to join with "AND" or "OR" logic
4. Add your next condition

This allows for complex queries like: `firstName is "John" AND age > 30` or `status is "pending" OR status is "review"`.

### Available Operators

Different field types support different operators:

| Field Type | Available Operators                             |
| ---------- | ----------------------------------------------- |
| String     | is, is not, like, is empty, not empty           |
| Number     | is, is not, >, \<, >=, \<=, is empty, not empty |
| Boolean    | is true, is false, is empty, not empty          |
| Date       | is, is not, >, \<, >=, \<=, is empty, not empty |
| Enum       | is, is not, is empty, not empty                 |

### Horizontal Scrolling

When you add multiple filter conditions that extend beyond the available width of the screen, the filter area will automatically enable horizontal scrolling. This allows you to create complex filter combinations without being limited by screen space.

Simply scroll horizontally to see all your filter conditions when they extend beyond the visible area.

## Advanced Filter Examples

Here are some examples of how you might use Advanced Filters:

### Example 1: Finding Specific Customer Records

```
firstName is "Sarah" AND status is "active" AND lastPurchase > "2023-01-01"
```

This filter would show all active customers named Sarah who made a purchase after January 1, 2023.

### Example 2: Identifying Records Needing Attention

```
(status is "pending" OR status is "review") AND createdDate < "2023-06-01"
```

This filter would show all records that are either pending or in review, and were created before June 1, 2023.

### Example 3: Finding Missing Data

```
email is not empty AND phone is empty
```

This filter would show all records that have an email address but are missing a phone number.

## Best Practices

* **Start simple**: Begin with a single filter condition and add more as needed
* **Use AND/OR strategically**: "AND" narrows results (both conditions must be true), while "OR" broadens results (either condition can be true)
* **Consider performance**: Very complex filters on large datasets may take longer to process
* **Save common filters**: If you frequently use the same filter combinations, consider saving them as views

## Troubleshooting

If you encounter issues with Advanced Filters:

* Ensure your filter values match the expected format for the field type
* Check that you're using appropriate operators for each field type
* For complex filters, try breaking them down into simpler components to identify issues
* Verify that the data you're filtering actually exists in your dataset

Advanced Filters provide a powerful way to work with your data in Flatfile, allowing you to quickly find and focus on the records that matter most to your workflow.


# Authentication and Authorization
Source: https://flatfile.com/docs/guides/authentication

Complete guide to authenticating with Flatfile using API keys, Personal Access Tokens, and managing roles and permissions

This guide covers all aspects of authentication with Flatfile, including API keys, Personal Access Tokens, and role-based access control for your team and customers.

## API Keys

Flatfile provides two different kinds of environment-specific API keys you can use to interact with the API. In addition, you can work with a development key or a production environment.

<Info>
  API keys are created automatically. Use the [API Keys and Secrets](https://platform.flatfile.com/dashboard/keys-and-secrets) page to see your API keys for
  any given Environment.
</Info>

### Testing and Development

[Environments](/core-concepts/environments) are isolated entities and are intended to be a safe place to create and test different configurations. A `development` and `production` environment are created by default.

| isProd  | Name          | Description                                                                                 |
| ------- | ------------- | ------------------------------------------------------------------------------------------- |
| *false* | `development` | Use this default environment, and its associated test API keys, as you build with Flatfile. |
| *true*  | `production`  | When you're ready to launch, create a new environment and swap out your keys.               |

<Note>
  The development environment does not count towards your paid credits.
</Note>

### Secret and Publishable Keys

All Accounts have two key types for each environment. Learn when to use each type of key:

| Type            | Id                     | Description                                                                                                             |
| --------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| Secret key      | `sk_23ghsyuyshs7dcrty` | **On the server-side:** Store this securely in your server-side code. Don't expose this key in an application.          |
| Publishable key | `pk_23ghsyuyshs7dcert` | **On the client-side:** Can be publicly-accessible in your application's client-side code. Use when embedding Flatfile. |

<Note>
  The `accessToken` provided from `publishableKey` will remain valid for a
  duration of 24 hours.
</Note>

## Personal Access Tokens

Personal Access Tokens (PATs) provide a secure way to authenticate with the Flatfile API. Unlike environment-specific API keys, PATs are user-scoped tokens that inherit the permissions of the user who created them.

Personal Access Tokens:

* Are user-scoped authentication tokens
* Have the same auth scope as the user who created them
* Can be used in place of a JWT for API authentication
* Are ideal for scripts, automation, and integrations that need to act on behalf of a user

This opens up possibilities for various use cases, including building audit logs, managing Spaces, and monitoring agents across environments.

### Creating a Token

1. Log in to your Flatfile account
2. Click on your user profile dropdown in the top-right corner
3. Select "Personal Access Tokens"
4. Click "Create Token"
5. Enter a descriptive name for your token
6. Copy the generated token immediately - it will only be shown once

<Note>
  Make sure to copy your token when it's first created. For security reasons,
  you won't be able to view the token again after leaving the page.
</Note>

### Exchanging Credentials for an Access Token

You can exchange your email and password credentials for an access token using the auth endpoint. See the [Authentication Examples](/guides/deeper/auth-examples#creating-a-pat-via-api) for the complete API call.

The response will include an access token that you can use for API authentication.

### Retrieving a Personal Access Token (Legacy Method)

Your `publishableKey` and `secretKey` are specific to an environment. Therefore, to interact at a higher level, you can use a personal access token.

1. From the dashboard, open **Settings**

2. Click to **Personal Tokens**

3. Retrieve your `clientId` and `secret`.

4. Using the key pair, call the auth endpoint. See the [Authentication Examples](/guides/deeper/auth-examples#legacy-client-credentials-flow) for the complete API call.

5. The response will include an `accessToken`. Present that as your **Bearer `token`** in place of the `secretKey`.

### Using a Token

Use your Personal Access Token directly as a bearer token in API requests. No exchange step is needed.

```bash theme={null}
curl -X GET https://platform.flatfile.com/api/v1/spaces \
  -H "Authorization: Bearer pat_your_token_here"
```

### Rotating API Keys

You can use a Personal Access Token to programmatically rotate your environment's API keys. This is useful for automated key rotation workflows, especially for accounts that use SSO and cannot authenticate with email and password.

1. Create a new secret key:

```bash theme={null}
curl -X POST "https://platform.flatfile.com/api/v1/auth/api-key?type=SECRET&environmentId=us_env_yourEnvironmentId" \
  -H "Authorization: Bearer pat_your_token_here"
```

2. Update your application to use the new key.

3. Delete the old key:

```bash theme={null}
curl -X DELETE "https://platform.flatfile.com/api/v1/auth/api-key?key=sk_old_key&environmentId=us_env_yourEnvironmentId" \
  -H "Authorization: Bearer pat_your_token_here"
```

<Note>
  Both keys will be valid simultaneously until you delete the old one, so you
  can rotate without downtime.
</Note>

### Managing Tokens

You can view all your active tokens in the Personal Access Tokens page. For each token, you can see:

* Name
* Creation date
* Last used date (if applicable)

To delete a token:

1. Navigate to the Personal Access Tokens page
2. Find the token you want to delete
3. Click the menu icon (three dots) next to the token
4. Select "Delete"
5. Confirm the deletion

<Warning>
  Deleting a token immediately revokes access for any applications or scripts
  using it. Make sure you update any dependent systems before deleting a token.
</Warning>

### Best Practices

* Create separate tokens for different applications or use cases
* Use descriptive names that identify where the token will be used
* Regularly review and delete unused tokens
* Rotate tokens periodically for enhanced security
* Never share your tokens with others - each user should create their own tokens

### Example Use Cases

#### Building an Audit Log

Query for all events across all environments and combine them with user and guest data to create a comprehensive audit log, providing a detailed history of actions within the application.

#### Managing Spaces Across Environments

Determine the number of Spaces available and identify which Spaces exist in different environments, allowing you to efficiently manage and organize your data.

#### Monitoring Agents Across Environments

Keep track of agents deployed to various environments by retrieving information about their presence, ensuring smooth and efficient data import processes.

## Roles & Permissions

Grant your team and customers access with role-based permissions.

### Administrator Roles

Administrator roles have full access to your accounts, including inviting additional admins and seeing developer keys.

<Note>
  The `accessToken` provided will remain valid for a duration of 24 hours.
</Note>

| Role          | Details                                                                                                                                                                               |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Administrator | This role is meant for any member of your team who requires full access to the Account.<br /> <br />✓ Can add other administrators <br />✓ Can view secret keys <br />✓ Can view logs |

### Guest Roles

Guest roles receive access via a magic link or a shared link depending on the [Environment](https://platform.flatfile.com/dashboard) `guestAuthentication` type. Guests roles can invite other Guests unless you turn off this setting in the [Guest Sidebar](/guides/customize-guest-sidebar).

<Note>
  The `accessToken` provided will remain valid for a duration of 1 hour.
</Note>

Data Clips can provide granular guest access to a specific [sheet](/core-concepts/sheets) by sharing only selected records from that sheet within a [workbook](/core-concepts/workbooks). However, if the clipped sheet includes [reference fields](/core-concepts/fields#reference) that point to other sheets in the same workbook, guests will also receive read access to those referenced records to support lookups and validation. Learn more in the [Data Clips guide](/legacy-docs/advanced-guides/dataclips).

#### Space Grant

| Role               | Details                                                                                                                                                        |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Single-Space Guest | This role is meant for a guest who has access to only one Space. Such guests can be invited to additional Spaces at any time.                                  |
| Multi-Space Guest  | This role is meant for a guest who has access to multiple Spaces. They will see a drop-down next to the Space name that enables them to switch between Spaces. |

#### Workbook Grant

| Role                  | Details                                                                                    |
| --------------------- | ------------------------------------------------------------------------------------------ |
| Single-Workbook Guest | This role is meant for a guest who should have access to only one Workbook within a Space. |
| Multi-Workbook Guest  | This role is intended for a guest who has access to multiple Workbooks within a Space.     |

<Info>This role can only be configured using code. See code example.</Info>

```js theme={null}
const createGuest = await api.guests.create({
  environmentId: "us_env_hVXkXs0b",
  email: "guest@example.com",
  name: "Mr. Guest",
  spaces: [
    {
      id: "us_sp_DrdXetPN",
      workbooks: [
        {
          id: "us_wb_qGZbKwDW",
        },
      ],
    },
  ],
});
```

#### Guest Lifecycle

When a guest user is deleted, all their space connections are automatically removed to ensure security. This means:

* The guest loses access to all previously connected spaces
* They cannot regain access to these spaces without being explicitly re-invited

This automatic cleanup ensures that deleted guests cannot retain any access to spaces, even if they are later recreated with the same email address.

## API Reference

For detailed API documentation on authentication endpoints, see the [Authentication API Reference](https://reference.flatfile.com/api-reference/auth).


# Custom Extractors
Source: https://flatfile.com/docs/guides/custom-extractors

Build custom file processing plugins to handle unique data formats and transform files into structured data

## What Are Custom Extractors?

Custom extractors are specialized plugins that enable you to handle file formats that aren't natively supported by Flatfile's existing [plugins](/plugins). They process uploaded files, extract structured data, and provide that data for mapping into [Sheets](/core-concepts/sheets) as [Records](/core-concepts/records). This guide covers everything you need to know to build custom extractors.

Common use cases include:

* Legacy system data exports (custom delimited files, fixed-width formats)
* Industry-specific formats (healthcare, finance, manufacturing)
* Multi-format processors (handling various formats in one extractor)
* Binary file handlers (images with metadata, proprietary formats)

## Architecture Overview

### Core Components

Custom extractors are built using the `@flatfile/util-extractor` utility, which provides a standardized framework for file processing:

```javascript theme={null}
import { Extractor } from "@flatfile/util-extractor";

export const MyCustomExtractor = (options = {}) => {
  return Extractor(".myformat", "custom", myCustomParser, options);
};
```

Once you've created your extractor, you must register it in a [listener](/core-concepts/listeners) to be used. This will ensure that the extractor responds to the `file:created` [event](/reference/events#file%3Acreated) and processes your files.

```javascript theme={null}
// . . . other imports
import { MyCustomExtractor } from "./my-custom-extractor";

export default function (listener) {
  // . . . other listener setup
  listener.use(MyCustomExtractor());
}
```

### Handling Multiple File Extensions

To support multiple file extensions, use a RegExp pattern:

```javascript theme={null}
// Support both .pipe and .custom extensions
export const MultiExtensionExtractor = (options = {}) => {
  return Extractor(/\.(pipe|custom)$/i, "pipe", parseCustomFormat, options);
};

// Support JSON variants
export const JSONExtractor = (options = {}) => {
  return Extractor(/\.(json|jsonl|jsonlines)$/i, "json", parseJSONFormat, options);
};
```

### Key Architecture Elements

| Component           | Purpose                                                        | Required |
| ------------------- | -------------------------------------------------------------- | -------- |
| **File Extension**  | String or RegExp of supported file extension(s)                | ✓        |
| **Extractor Type**  | String identifier for the extractor type                       | ✓        |
| **Parser Function** | Core logic that converts file buffer to structured data        | ✓        |
| **Options**         | Configuration for chunking, parallelization, and customization | -        |

### Data Flow

1. **File Upload** → Flatfile receives file with matching extension
2. **Event Trigger** → `file:created` [event](/reference/events#file%3Acreated) fires
3. **Parser Execution** → Your parser function processes the file buffer
4. **Data Structuring** → Raw data is converted to WorkbookCapture format and provided to Flatfile for mapping into [Sheets](/core-concepts/sheets) as [Records](/core-concepts/records)
5. **Job Completion** → Processing status is reported to user

## Getting Started

Remember that custom extractors are powerful tools for handling unique data formats. Start with simple implementations and gradually add complexity as needed.

### Prerequisites

Install the required packages. You may also want to review our [Coding Tutorial](/coding-tutorial/overview) if you haven't created a [Listener](/core-concepts/listeners) yet.

```bash theme={null}
npm install @flatfile/util-extractor @flatfile/listener @flatfile/api
```

### Basic Implementation

Let's create a simple custom extractor for a pipe-delimited format. This will be used to process files with the `.pipe` or `.psv` extension that look like this:

```psv theme={null}
name|email|phone
John Doe|john@example.com|123-456-7890
Jane Smith|jane@example.com|098-765-4321
```

```javascript theme={null}
import { Extractor } from "@flatfile/util-extractor";

// Parser function - converts Buffer to WorkbookCapture
function parseCustomFormat(buffer) {
  const content = buffer.toString('utf-8');
  const lines = content.split('\n').filter(line => line.trim());
  
  if (lines.length === 0) {
    throw new Error('Empty file');
  }
  
  // First line contains headers
  const headers = lines[0].split('|').map(h => h.trim());
  
  // Remaining lines contain data
  const data = lines.slice(1).map(line => {
    const values = line.split('|').map(v => v.trim());
    const record = {};
    
    headers.forEach((header, index) => {
      record[header] = {
        value: values[index] || ''
      };
    });
    
    return record;
  });
  
  return {
    Sheet1: {
      headers,
      data
    }
  };
}

// Create the extractor
export const CustomPipeExtractor = (options = {}) => {
  return Extractor(/\.(pipe|psv)$/i, "pipe", parseCustomFormat, options);
};
```

And now let's import and register it in your [Listener](/core-concepts/listeners).

```javascript theme={null}
// . . . other imports
import { CustomPipeExtractor } from "./custom-pipe-extractor";

export default function (listener) {
  // . . . other listener setup
  listener.use(CustomPipeExtractor());
}
```

That's it! Your extractor is now registered and will be used to process pipe-delimited files with the `.pipe` or `.psv` extension.

## Advanced Examples

### Multi-Sheet Parser

Let's construct an Extractor to handle files that contain multiple data sections. This will be used to process files with the `.multi` or `.sections` extension that look like this:

```text theme={null}
---SECTION---
SHEET:Sheet1
name,email,phone
John Doe,john@example.com,123-456-7890
Jane Smith,jane@example.com,098-765-4321
---SECTION---
SHEET:Sheet2
name,email,phone
Jane Doe,jane@example.com,123-456-7891
John Smith,john@example.com,098-765-4322
---SECTION---
```

```javascript theme={null}
function parseMultiSheetFormat(buffer) {
  const content = buffer.toString('utf-8');
  const sections = content.split('---SECTION---');
  
  const workbook = {};
  
  sections.forEach((section, index) => {
    if (!section.trim()) return;
    
    const lines = section.trim().split('\n');
    const sheetName = lines[0].replace('SHEET:', '').trim() || `Sheet${index + 1}`;
    
    const headers = lines[1].split(',').map(h => h.trim());
    const data = lines.slice(2).map(line => {
      const values = line.split(',').map(v => v.trim());
      const record = {};
      
      headers.forEach((header, idx) => {
        record[header] = {
          value: values[idx] || ''
        };
      });
      
      return record;
    });
    
    workbook[sheetName] = { headers, data };
  });
  
  return workbook;
}

export const MultiSheetExtractor = (options = {}) => {
  return Extractor(/\.(multi|sections)$/i, "multi-sheet", parseMultiSheetFormat, options);
};
```

Now let's register it in your [Listener](/core-concepts/listeners).

```javascript theme={null}
// . . . other imports
import { MultiSheetExtractor } from "./multi-sheet-extractor";

export default function (listener) {
  // . . . other listener setup
  listener.use(MultiSheetExtractor());
}
```

### Binary Format Handler

This example will be used to process binary files with structured data. This will be used to process binary files with the `.bin` or `.dat` extension. Due to the nature of binary format, we can't easily present a sample import here.

```javascript theme={null}
function parseBinaryFormat(buffer) {
  // Example: Custom binary format with header + records
  let offset = 0;
  
  // Read header (first 16 bytes)
  const magic = buffer.readUInt32LE(offset); offset += 4;
  const version = buffer.readUInt16LE(offset); offset += 2;
  const recordCount = buffer.readUInt32LE(offset); offset += 4;
  const fieldCount = buffer.readUInt16LE(offset); offset += 2;
  
  if (magic !== 0xDEADBEEF) {
    throw new Error('Invalid file format');
  }
  
  // Read field definitions
  const headers = [];
  for (let i = 0; i < fieldCount; i++) {
    const nameLength = buffer.readUInt16LE(offset); offset += 2;
    const name = buffer.toString('utf-8', offset, offset + nameLength);
    offset += nameLength;
    const type = buffer.readUInt8(offset); offset += 1;
    
    headers.push(name);
  }
  
  // Read records
  const data = [];
  for (let i = 0; i < recordCount; i++) {
    const record = {};
    
    headers.forEach(header => {
      const valueLength = buffer.readUInt16LE(offset); offset += 2;
      const value = buffer.toString('utf-8', offset, offset + valueLength);
      offset += valueLength;
      
      record[header] = { value };
    });
    
    data.push(record);
  }
  
  return {
    Sheet1: { headers, data }
  };
}

export const BinaryExtractor = (options = {}) => {
  return Extractor(/\.(bin|dat)$/i, "binary", parseBinaryFormat, options);
};
```

And, once again, let's register it in your [Listener](/core-concepts/listeners).

```javascript theme={null}
// . . . other imports
import { BinaryExtractor } from "./binary-extractor";

export default function (listener) {
  // . . . other listener setup
  listener.use(BinaryExtractor());
}
```

### Configuration-Driven Extractor

Create a flexible extractor that can be configured for different formats. This will be used to process files in a manner that handles different delimiters, line endings, and other formatting options.

```javascript theme={null}
function createConfigurableParser(config) {
  return function parseConfigurableFormat(buffer) {
    const content = buffer.toString(config.encoding || 'utf-8');
    let lines = content.split(config.lineDelimiter || '\n');
    
    // Skip header lines if specified
    if (config.skipLines) {
      lines = lines.slice(config.skipLines);
    }
    
    // Filter empty lines
    if (config.skipEmptyLines) {
      lines = lines.filter(line => line.trim());
    }
    
    if (lines.length === 0) {
      throw new Error('No data found');
    }
    
    // Extract headers
    let headers;
    let dataStartIndex = 0;
    
    if (config.explicitHeaders) {
      headers = config.explicitHeaders;
    } else {
      headers = lines[0].split(config.fieldDelimiter || ',').map(h => h.trim());
      dataStartIndex = 1;
    }
    
    // Process data
    const data = lines.slice(dataStartIndex).map(line => {
      const values = line.split(config.fieldDelimiter || ',');
      const record = {};
      
      headers.forEach((header, index) => {
        let value = values[index] || '';
        
        // Apply transformations
        if (config.transforms && config.transforms[header]) {
          value = config.transforms[header](value);
        }
        
        // Type conversion
        if (config.typeConversion) {
          if (!isNaN(value) && value !== '') {
            value = Number(value);
          } else if (value.toLowerCase() === 'true' || value.toLowerCase() === 'false') {
            value = value.toLowerCase() === 'true';
          }
        }
        
        record[header] = { value };
      });
      
      return record;
    });
    
    return {
      [config.sheetName || 'Sheet1']: { headers, data }
    };
  };
}

export const ConfigurableExtractor = (userConfig = {}) => {
  const defaultConfig = {
    encoding: 'utf-8',
    lineDelimiter: '\n',
    fieldDelimiter: ',',
    skipLines: 0,
    skipEmptyLines: true,
    typeConversion: false,
    sheetName: 'Sheet1'
  };
  
  const config = { ...defaultConfig, ...userConfig };
  
  return Extractor(
    config.fileExtension || ".txt", 
    "configurable", 
    createConfigurableParser(config),
    {
      chunkSize: config.chunkSize || 10000,
      parallel: config.parallel || 1
    }
  );
};
```

Now let's register two different configurable extractors in our [Listener](/core-concepts/listeners).

The first will be used to process files with the `.custom` extension that look like this, while transforming dates and amount values:

```text theme={null}
Extraneous text
More extraneous text
name & date & amount
John Doe & 1/1/2021 & 100.00
Jane Smith & 1/2/2021 & 200.00
```

The second will be used to process files with the `.pipe` or `.special` extension that look like this:

```text theme={null}
Extraneous text
More extraneous text
name|date|amount
John Doe|2021-01-01|100.00
Jane Smith|2021-01-02|200.00
```

```javascript theme={null}
// . . . other imports
import { ConfigurableExtractor } from "./configurable-extractor";

export default function (listener) {
  // . . . other listener setup

  // Custom extractor with configuration for .custom files
  listener.use(ConfigurableExtractor({
    fileExtension: ".custom",
    fieldDelimiter: " & ",
    skipLines: 2,
    typeConversion: true,
    transforms: {
      'date': (value) => new Date(value).toISOString(),
      'amount': (value) => parseFloat(value).toFixed(2)
    }
  }));

  // Custom extractor with configuration for .pipe and .special files
  listener.use(ConfigurableExtractor({
    fileExtension: /\.(pipe|special)$/i,
    fieldDelimiter: "|",
    skipLines: 2,
    typeConversion: true
  }));
}
```

## Reference

### API

```typescript theme={null}
function Extractor(
  fileExt: string | RegExp,
  extractorType: string,
  parseBuffer: (
    buffer: Buffer,
    options: any
  ) => WorkbookCapture | Promise<WorkbookCapture>,
  options?: Record<string, any>
): (listener: FlatfileListener) => void
```

| Parameter       | Type                  | Description                                                                |
| --------------- | --------------------- | -------------------------------------------------------------------------- |
| `fileExt`       | `string` or `RegExp`  | File extension to process (e.g., `".custom"` or `/\.(custom\|special)$/i`) |
| `extractorType` | `string`              | Identifier for the extractor type (e.g., "custom", "binary")               |
| `parseBuffer`   | `ParserFunction`      | Function that converts Buffer to WorkbookCapture                           |
| `options`       | `Record<string, any>` | Optional configuration object                                              |

#### Options

| Option      | Type      | Default | Description                            |
| ----------- | --------- | ------- | -------------------------------------- |
| `chunkSize` | `number`  | `5000`  | Records to process per batch           |
| `parallel`  | `number`  | `1`     | Number of concurrent processing chunks |
| `debug`     | `boolean` | `false` | Enable debug logging                   |

#### Parser Function Options

Your `parseBuffer` function receives additional options beyond what you pass to `Extractor`:

| Option                   | Type      | Description                                       |
| ------------------------ | --------- | ------------------------------------------------- |
| `fileId`                 | `string`  | The ID of the file being processed                |
| `fileExt`                | `string`  | The file extension (e.g., ".csv")                 |
| `headerSelectionEnabled` | `boolean` | Whether header selection is enabled for the space |

### Data Structures

#### WorkbookCapture Structure

The parser function must return a `WorkbookCapture` object:

```javascript theme={null}
const workbookCapture = {
  "SheetName1": {
    headers: ["field1", "field2", "field3"],
    data: [
      {
        field1: { value: "value1" },
        field2: { value: "value2" },
        field3: { value: "value3" }
      },
      // ... more records
    ]
  },
  "SheetName2": {
    headers: ["col1", "col2"],
    data: [
      {
        col1: { value: "data1" },
        col2: { value: "data2" }
      }
    ]
  }
};
```

#### Cell Value Objects

Each cell value should use the `Flatfile.RecordData` format:

```javascript theme={null}
const recordData = {
  field1: { value: "john@example.com" },
  field2: { value: "John Doe" },
  field3: { 
    value: "invalid-email",
    messages: [
      {
        type: "error",
        message: "Invalid email format"
      }
    ]
  }
};
```

#### Message Types

| Type      | Description           | UI Effect                                                                                    |
| --------- | --------------------- | -------------------------------------------------------------------------------------------- |
| `error`   | Validation error      | Red highlighting, blocks [Actions](/core-concepts/actions) with the `hasAllValid` constraint |
| `warning` | Warning message       | Yellow highlighting, allows submission                                                       |
| `info`    | Informational message | Mouseover tooltip, allows submission                                                         |

### TypeScript Interfaces

```typescript theme={null}
type ParserFunction = (
  buffer: Buffer,
  options: any
) => WorkbookCapture | Promise<WorkbookCapture>;

type WorkbookCapture = Record<string, SheetCapture>;

type SheetCapture = {
  headers: string[];
  descriptions?: Record<string, string | null> | null;
  data: Flatfile.RecordData[];
  metadata?: { rowHeaders: number[] };
};
```

## Troubleshooting Common Issues

### Files Not Processing

**Symptoms**: Files upload but no extraction occurs

**Solutions**:

* Verify file extension matches `fileExt` configuration
* Check [Listener](/core-concepts/listeners) is properly deployed and running
* Enable debug logging to see processing details

```javascript theme={null}
const extractor = CustomExtractor({
  debug: true
}); // Make sure file extensions match in the Extractor call
```

### Parser Errors

**Symptoms**: Jobs fail with parsing errors

**Solutions**:

* Add try-catch blocks in parser function
* Validate input data before processing
* Return helpful error messages

```javascript theme={null}
function parseCustomFormat(buffer) {
  try {
    const content = buffer.toString('utf-8');
    
    if (!content || content.trim() === '') {
      throw new Error('File is empty');
    }
    
    // ... parsing logic
    
  } catch (error) {
    throw new Error(`Parse error: ${error.message}`);
  }
}
```

### Memory Issues

**Symptoms**: Large files cause timeouts or memory errors

**Solutions**:

* Reduce chunk size for large files
* Implement streaming for very large files
* Use parallel processing carefully

```javascript theme={null}
const extractor = CustomExtractor({
  chunkSize: 1000,  // Smaller chunks
  parallel: 1       // Reduce parallelization
});
```

### Performance Problems

**Symptoms**: Slow processing, timeouts

**Solutions**:

* Optimize parser algorithm
* Use appropriate chunk sizes
* Consider parallel processing for I/O-bound operations

```javascript theme={null}
// Optimize for large files
const extractor = CustomExtractor({
  chunkSize: 5000,
  parallel: 3
});
```


# Data Egress
Source: https://flatfile.com/docs/guides/egress

Get data out of Flatfile using various transmission methods and file formats

After your customers have imported and validated their data in Flatfile, you need to get that data to its final destination. Data egress refers to the process of extracting and transmitting data from Flatfile to external systems, files, or destinations.

## Egress Methods

Since Flatfile listeners run in TypeScript/JavaScript environments, you can egress data using any mechanism available in the language ecosystem. This includes direct API calls, file generation, database connections, message queues, or any custom integration pattern your application requires.

### API Transmission

Send data directly to external systems via HTTP requests to REST APIs, GraphQL endpoints, streaming services, or database APIs. Use any HTTP client library or built-in fetch to POST data to your existing endpoints.

### File Generation & Download

Build files in any format - CSV, XLSX, XML, JSON, PDF, or custom formats - using the appropriate TypeScript libraries. Generate formatted reports, structured documents, or data exports that users can download directly.

### File Transfer

Transmit generated files to external destinations like cloud storage, FTP servers, email attachments, or content delivery networks using the corresponding SDK or protocol library.

## Example: API Egress with Workbook Actions

The following example demonstrates egressing data via HTTP POST when users complete a workbook. This pattern listens for a workbook action and sends the data to an external webhook endpoint.

### Implementation

The example listener responds to `workbook:submitActionFg` events, retrieves all sheets and records from the workbook, and posts the data to a configured webhook URL. It uses the `event.secrets()` method to securely access the destination URL.

### Testing with webhook.site

To test this pattern:

1. Navigate to [webhook.site](https://webhook.site) and copy your unique URL
2. Add the URL as a Flatfile secret named `WEBHOOK_SITE_URL`
3. Configure a workbook action in your blueprint to trigger the egress
4. The listener will POST the workbook data to your webhook.site URL where you can inspect the payload

### Example Projects

* [TypeScript implementation](https://github.com/FlatFilers/flatfile-docs-kitchen-sink/blob/main/typescript/egress/index.ts)
* [JavaScript implementation](https://github.com/FlatFilers/flatfile-docs-kitchen-sink/blob/main/javascript/egress/index.js)

## Example: File Export with Export Workbook Plugin

For file generation and download, Flatfile provides the Export Workbook Plugin that generates Excel files from workbook data. This plugin handles the entire process of extracting data, formatting it into Excel sheets, and making it available for download.

### Installation

```bash theme={null}
npm install @flatfile/plugin-export-workbook
```

### Implementation

```javascript theme={null}
import { FlatfileListener } from "@flatfile/listener";
import { exportWorkbookPlugin } from "@flatfile/plugin-export-workbook";

export default function (listener) {
  listener.use(
    exportWorkbookPlugin({
      // Only export validated records
      recordFilter: 'valid',
      // Automatically trigger download
      autoDownload: true,
      // Custom filename
      filename: 'exported-data',
      // Transform column names for readability
      columnNameTransformer: (columnName) => {
        return columnName.replace(/([A-Z])/g, ' $1').replace(/^./, str => str.toUpperCase());
      }
    })
  );
}
```

### Workbook Configuration

Add a download action to your workbook to trigger the export:

```javascript theme={null}
{
  name: "Customer Data",
  actions: [
    {
      operation: "downloadWorkbook",
      mode: "foreground", 
      label: "Download Excel File",
      description: "Export all data as Excel spreadsheet",
      primary: true
    }
  ],
  sheets: [
    // your sheet definitions
  ]
}
```

When users click the download action, the plugin generates an Excel file containing all workbook data and either triggers an automatic download or directs users to the Files page to download manually.

For complete configuration options and advanced usage, see the [Export Workbook Plugin documentation](/plugins/export-workbook).

## Example: Custom XML File Generation

For custom file formats, you can build files programmatically using any TypeScript library. This example demonstrates creating XML files using the `xml2js` library to generate structured data exports.

### Installation

```bash theme={null}
npm install xml2js
npm install @types/xml2js --save-dev
```

### Implementation

```javascript theme={null}
import api from "@flatfile/api";
import { FlatfileListener } from "@flatfile/listener";
import { Builder } from 'xml2js';
import fs from 'fs';

export default function (listener) {
  listener.on(
    "job:ready",
    { job: "workbook:downloadXML" },
    async ({ context: { jobId, workbookId } }) => {
      try {
        await api.jobs.ack(jobId, {
          info: "Starting XML generation...",
          progress: 10,
        });

        // Get workbook and find the target sheet
        const workbook = await api.workbooks.get(workbookId);
        const sheet = workbook?.data.sheets?.find(sheet => sheet.slug === "customers");
        const records = await api.records.get(sheet?.id || "");

        await api.jobs.ack(jobId, {
          info: "Processing records...",
          progress: 30,
        });

        // Initialize XML builder
        const builder = new Builder({
          xmldec: { version: '1.0', encoding: 'UTF-8' }
        });

        // Transform records to structured data
        const customers = records.data.records.map(record => {
          const values = record.values;
          return {
            id: values.id?.value || '',
            name: values.name?.value || '',
            email: values.email?.value || '',
            phone: values.phone?.value || '',
            address: {
              street: values.street?.value || '',
              city: values.city?.value || '',
              state: values.state?.value || '',
              zip: values.zip?.value || ''
            }
          };
        });

        // Create XML structure
        const xmlObj = {
          export: {
            $: { 
              generated: new Date().toISOString(),
              recordCount: customers.length 
            },
            customers: {
              customer: customers
            }
          }
        };

        // Generate XML string
        const xml = builder.buildObject(xmlObj);

        await api.jobs.ack(jobId, {
          info: "Creating XML file...",
          progress: 70,
        });

        // Write XML to temporary file
        const fileName = `customer_export_${new Date().toISOString().split('T')[0]}.xml`;
        fs.writeFileSync(fileName, xml);

        // Upload file to Flatfile
        const file = fs.createReadStream(fileName);
        const fileUpload = await api.files.upload(file, {
          spaceId: workbook.data.spaceId,
          environmentId: workbook.data.environmentId,
        });

        // Complete job with download link
        await api.jobs.complete(jobId, {
          outcome: {
            message: "XML file generated successfully",
            next: {
              type: "files",
              label: "Download XML",
              files: [{ fileId: fileUpload?.data?.id }],
            },
          },
        });

        // Clean up temporary file
        fs.unlinkSync(fileName);

      } catch (error) {
        await api.jobs.fail(jobId, {
          outcome: {
            message: `Failed to generate XML: ${error.message}`,
          },
        });
      }
    }
  );
}
```

### Workbook Configuration

Add an XML download action to trigger the export:

```javascript theme={null}
{
  name: "Customer Data",
  actions: [
    {
      operation: "downloadXML",
      mode: "foreground",
      label: "Download XML",
      description: "Export data as XML file",
      primary: false
    }
  ],
  sheets: [
    {
      name: "Customers",
      slug: "customers",
      fields: [
        { key: "id", type: "string", label: "Customer ID" },
        { key: "name", type: "string", label: "Name" },
        { key: "email", type: "string", label: "Email" },
        { key: "phone", type: "string", label: "Phone" },
        { key: "street", type: "string", label: "Street" },
        { key: "city", type: "string", label: "City" },
        { key: "state", type: "string", label: "State" },
        { key: "zip", type: "string", label: "ZIP Code" }
      ]
    }
  ]
}
```

This pattern can be adapted for any file format by substituting the appropriate library and transformation logic. The key steps remain the same: fetch data, transform it, generate the file, upload it to Flatfile, and provide a download link to users.


# Multi-Part Jobs
Source: https://flatfile.com/docs/guides/multi-part-jobs

Split up Jobs into Parts

A Job can be split up into multiple parts depending on the intended functionality. This can help spread out large data sets or long tasks across many separate listeners. When a job is split up into parts, the original part is considered the "parent" part and it's children are considered "parts". Each Part is considered it's own separate Job and has the full ecosystem of events attached to it. When all Job Parts have been completed, a new event `job:parts-completed` is emitted, so that you can listen for that event to complete the Parent Job. This event has a payload that has summary statistics about the parts.

```json theme={null}
{
  "payload": {
    "parts": {
      "total": 10,
      "completed": 10,
      "failed": 0,
      "canceled": 0
    }
  }
}
```

By inspecting this payload, you can determine if the all have completed successfully or not. Based on that information, you are expected to complete or fail the parent Job.

Here's an example of a listener that creates parts and waits for the `job:parts-completed` event:

```typescript theme={null}
import api from "@flatfile/api";
import { FlatfileListener } from "@flatfile/listener";

export default function (listener: FlatfileListener) {
  // Parent and Part Jobs have the same operation name.
  // Filtering by isPart: false, ensures that the Job is the Parent
  // The job operation name is an example, you can define your own.
  listener.filter(
    { job: "sheet:submitLargeSheet", isPart: false },
    (submitLargeSheet) => {
      submitLargeSheet.on("job:ready", async (event) => {
        const {
          context: { jobId, sheetId },
        } = event;
        console.log("job:ready [PARENT]", { jobId: event.context.jobId });

        const { data: counts } = await api.sheets.getRecordCounts(sheetId);
        const { total } = counts.counts;
        await api.jobs.ack(jobId, {
          info: `Splitting Job`,
          progress: 10,
        });
        console.log("splitting job: ", { jobId, total });

        const batchSize = 1000;
        const totalParts = Math.ceil(total / batchSize);
        const splitjob = await api.jobs.split(jobId, { parts: totalParts });
        console.log("splitjob: ", { splitjob });

        await api.jobs.ack(jobId, {
          info: `Job Split into ${total} parts.`,
          progress: 20,
        });
      });

      // Listen for all parts to finish and then complete the parent Job
      submitLargeSheet.on("job:parts-completed", async (event) => {
        const {
          context: { jobId },
        } = event;

        console.log("job:parts-completed: ", jobId);

        await api.jobs.complete(jobId, {
          outcome: {
            message: "This job is now complete.",
          },
        });
      });
    }
  );
}
```

Here's an example of a listener that does logic on each Part:

```typescript theme={null}
import api from "@flatfile/api";
import { FlatfileListener } from "@flatfile/listener";

export default function (listener: FlatfileListener) {
  // Parent and Part Jobs have the same operation name.
  // Filtering by isPart: true, ensures that the Job is a Part
  // The job operation name is an example, you can define your own.
  listener.filter(
    { job: "sheet:submitLargeSheet", isPart: true },
    (submitLargeSheet) => {
      submitLargeSheet.on("job:ready", async (event) => {
        const {
          context: { jobId },
        } = event;
        await api.jobs.ack(jobId);
        const job = await api.jobs.get(jobId);
        console.dir({ job }, { depth: 10 });
        const { partData, parentId } = job.data;
        const { records } = await event.data({
          pageSize: 1,
          pageNumber: partData.part + 1,
        });

        console.log({ record: records[0].values });
        console.log("submitting part: ", jobId, partData);

        await new Promise((r) => setTimeout(r, 1000));

        await api.jobs.complete(jobId, {
          outcome: {
            message: "This job is now complete.",
          },
        });
      });
    }
  );
}
```


# Namespaces and Filters
Source: https://flatfile.com/docs/guides/namespaces-and-filters

Isolate and organize your Flatfile Listeners using namespace filtering and advanced Event filtering patterns

Flatfile provides two complementary approaches for organizing and scoping your Event handling: **namespaces** and **Event filters**. Together, they enable sophisticated Event routing, better code organization, and precise control over which [Listeners](/core-concepts/listeners) respond to specific Events.

<CardGroup>
  <Card title="Namespaces" href="#namespaces">
    *Organize and isolate different parts of your application at the App, Space, and Workbook level.*
  </Card>

  <Card title="Event Filters" href="#event-filters">
    *Target specific Sheets or event properties to create precise Event handling rules for your Listeners.*
  </Card>
</CardGroup>

Both approaches help organize and scope Event handling, reduce noise by ensuring Listeners only respond to relevant Events, and can be combined for the most flexible and maintainable Event organization.

## Namespaces

### Understanding Namespaces

Namespaces are simple string identifiers that you can assign to [Apps](/core-concepts/apps), [Spaces](/core-concepts/spaces), and [Workbooks](/core-concepts/workbooks) to organize and isolate different parts of your Flatfile application. When you assign a namespace to a resource, Events from that resource can be filtered using Listener namespace patterns.

#### How Namespace Filtering Works

Listeners use prefix patterns to filter Events based on the namespace of the resource that generated the Event:

* `space:namespace` - Listen to Events from [Spaces](/core-concepts/spaces) with the given namespace or belonging to [Apps](/core-concepts/apps) with that namespace
* `workbook:namespace` - Listen to Events from [Workbooks](/core-concepts/workbooks) with the given namespace

### App Namespaces

The most common namespace pattern is creating separate [Apps](/core-concepts/apps) with distinct namespaces, then using namespaced Listeners to handle different configurations for each app.

#### Setting App Namespaces

Namespaces are set when creating an App via the [Flatfile Dashboard](https://platform.flatfile.com/dashboard):

<Frame>
  <img />
</Frame>

Let's walk through an example of routing events to different Listeners based on the App namespace.

Imagine creating three separate Apps with distinct namespaces:

<CardGroup>
  <Card title="Customer Portal">
    **App Name:** Customer Portal\
    **Namespace:** `customer-portal`\
    **Purpose:** External customer data import
  </Card>

  <Card title="Internal Tools">
    **App Name:** Internal Tools\
    **Namespace:** `internal-tools`\
    **Purpose:** Admin and operations
  </Card>

  <Card title="Partner Integration">
    **App Name:** Partner Integration\
    **Namespace:** `partner-integration`\
    **Purpose:** B2B data exchange
  </Card>
</CardGroup>

#### Listening to App Namespace Events

Now you can use `listener.namespace()` to create separate Listeners for Spaces within each App. This will provide a new, filtered Listener object scoped to your callback function.

This pattern allows each App to have completely different:

* Space configurations and blueprints
* Data validation rules
* Processing workflows
* User experiences
* Integration behaviors

<CodeGroup>
  ```javascript JavaScript theme={null}
  export default function (listener) {
    // Customer Portal - External customer data import
    listener.namespace('space:customer-portal', (customerListener) => {
      customerListener.use(configureCustomerPortalSpace);
      customerListener.use(configureGuidedSetup);
      customerListener.use(validateCustomerData);
    });

    // Internal Tools - Admin and operations
    listener.namespace('space:internal-tools', (internalListener) => {
      internalListener.use(configureInternalToolsSpace);
      internalListener.use(validateAdvancedRules);
      internalListener.use(applyBusinessLogic);
    });

    // Partner Integration - B2B data exchange
    listener.namespace('space:partner-integration', (partnerListener) => {
      partnerListener.use(configurePartnerIntegrationSpace);
      partnerListener.use(processAutomatically);
      partnerListener.use(syncPartnerData);
    });
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';

  export default function (listener: FlatfileListener) {
    // Customer Portal - External customer data import
    listener.namespace('space:customer-portal', (customerListener: FlatfileListener) => {
      customerListener.use(configureCustomerPortalSpace);
      customerListener.use(configureGuidedSetup);
      customerListener.use(validateCustomerData);
    });

    // Internal Tools - Admin and operations
    listener.namespace('space:internal-tools', (internalListener: FlatfileListener) => {
      internalListener.use(configureInternalToolsSpace);
      internalListener.use(validateAdvancedRules);
      internalListener.use(applyBusinessLogic);
    });

    // Partner Integration - B2B data exchange
    listener.namespace('space:partner-integration', (partnerListener: FlatfileListener) => {
      partnerListener.use(configurePartnerIntegrationSpace);
      partnerListener.use(processAutomatically);
      partnerListener.use(syncPartnerData);
    });
  }
  ```
</CodeGroup>

For more information on Events and Listeners (including `listener.use()`, as in this example), see [Events and Listeners](/core-concepts/listeners).

### Space Namespaces

Beyond App-level namespacing, you can also apply namespaces directly to individual Spaces, enabling unique configurations and behaviors on a space-by-space basis.

<Warning>
  **Important**: [Spaces](/core-concepts/spaces) inherit their [App's](/core-concepts/apps) namespace by default. When you create [listeners](/core-concepts/listeners) that filter on `space:app-namespace`, they receive Events from spaces within that App.

  If you override a Space's namespace to be different from its App, Listeners filtering on the App's namespace will no longer receive Events from that Space. For example:

  1. App has namespace `"my-app"`
  2. You create a Space with explicit namespace `"my-space"`
  3. Listeners filtering on `space:my-space` *will* receive Events from that Space
  4. Listeners filtering on `space:my-app` *will not* receive Events from that Space

  This applies regardless of whether the Space belongs to that App. Any other Spaces created without explicit namespaces will continue to inherit the App namespace normally.
</Warning>

#### Setting Space Namespaces

Assign a namespace when creating a Space via the API:

<CodeGroup>
  ```javascript JavaScript theme={null}
  const templateSpace = await api.spaces.create({
    name: "Template Manager",
    namespace: "templates", // Simple string identifier
    // ...space configuration
  });
  ```

  ```typescript TypeScript theme={null}
  import { Flatfile } from '@flatfile/api';

  const templateSpace = await api.spaces.create({
    name: "Template Manager",
    namespace: "templates", // Simple string identifier
    // ...space configuration
  });
  ```
</CodeGroup>

You can also set the namespace during the `space:configure` [Job](/core-concepts/jobs) by updating the Space. This may be useful if you're creating Spaces from the [Flatfile Dashboard](https://platform.flatfile.com/dashboard), which doesn't support setting a namespace during creation.

This example shows how to set the namespace based on the Space name, allowing you to have multiple configurations in the same App:

<Note>
  In this example, we'll show the full [Job](/core-concepts/jobs) Listener lifecycle implementation, complete with `ack` to acknowledge the job, `update` to report progress, and `complete` or `fail` to finish the job.

  However, for most implementations, we recommend using the [Space Configure](/plugins/space-configure) plugin. This plugin takes care of even more of the heavy lifting for you; not only does it handle the Job lifecycle, but it also takes care of all of the API calls necessary to configure the Space and create its Workbooks and documents.
</Note>

<CodeGroup>
  ```javascript JavaScript theme={null}
  // Set namespace based on space name patterns
  listener.on("job:ready", { job: "space:configure" }, async (event) => {
    const { jobId, spaceId } = event.context;

    try {
      await api.jobs.ack(jobId, { info: "Configuring space with namespace" });

      // Get Space details to determine namespace
      const space = await api.spaces.get(spaceId);
      
      // Route spaces to different namespaces based on naming conventions
      let namespace;
      if (space.data.name.toLowerCase().endsWith('portal')) {
        namespace = 'customer-portal';
      } else if (space.data.name.toLowerCase().endsWith('admin')) {
        namespace = 'internal-tools';
      } else if (space.data.name.toLowerCase().endsWith('partner')) {
        namespace = 'partner-integration';
      } else {
        namespace = 'general'; // Default namespace
      }
      
      await api.spaces.update(spaceId, {
        namespace: namespace,
      });

      // ...create workbooks, sheets, documents

      await api.jobs.complete(jobId, {
        outcome: { message: `Space configured with namespace: ${namespace}` }
      });
    } catch (error) {
      console.error(error);
      await api.jobs.fail(jobId, { 
        outcome: { message: `Configuration failed: ${error.message}` }
      });
    }
  });
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener, FlatfileEvent } from '@flatfile/listener';
  import { Flatfile } from '@flatfile/api';

  // Set namespace based on space name patterns
  listener.on("job:ready", { job: "space:configure" }, async (event: FlatfileEvent) => {
    const { jobId, spaceId } = event.context;

    try {
      await api.jobs.ack(jobId, { info: "Configuring space with namespace" });

      // Get Space details to determine namespace
      const space = await api.spaces.get(spaceId);
      
      // Route spaces to different namespaces based on naming conventions
      let namespace: string;
      if (space.data.name.toLowerCase().endsWith('portal')) {
        namespace = 'customer-portal';
      } else if (space.data.name.toLowerCase().endsWith('admin')) {
        namespace = 'internal-tools';
      } else if (space.data.name.toLowerCase().endsWith('partner')) {
        namespace = 'partner-integration';
      } else {
        namespace = 'general'; // Default namespace
      }
      
      await api.spaces.update(spaceId, {
        namespace: namespace,
      });

      // ...create workbooks, sheets, documents

      await api.jobs.complete(jobId, {
        outcome: { message: `Space configured with namespace: ${namespace}` }
      });
    } catch (error) {
      console.error(error);
      await api.jobs.fail(jobId, { 
        outcome: { message: `Configuration failed: ${error.message}` }
      });
    }
  });
  ```
</CodeGroup>

For complete Space configuration examples, see [Creating Spaces](/core-concepts/spaces#creating-spaces).

#### Listening to Space Namespace Events

Use `listener.namespace()` to filter Events from Spaces with specific namespaces. This will provide a new, filtered Listener object scoped to your callback function:

<CodeGroup>
  ```javascript JavaScript theme={null}
    // Listen to Events from customer portal Spaces
    listener.namespace('space:customer-portal', (customerPortalListener) => {
      customerPortalListener.use(applyCustomerBranding);
      customerPortalListener.use(configureGuidedOnboarding);
    });

    // Listen to Events from internal tools Spaces
    listener.namespace('space:internal-tools', (internalToolsListener) => {
      internalToolsListener.use(validateAdminData);
      internalToolsListener.use(enableAuditLogging);
    });

    // Listen to Events from partner integration Spaces
    listener.namespace('space:partner-integration', (partnerIntegrationListener) => {
      partnerIntegrationListener.use(configureApiWebhooks);
      partnerIntegrationListener.use(enableBulkProcessing);
    });

    // Listen to Events from general Spaces (default)
    listener.namespace('space:general', (generalSpaceListener) => {
      generalSpaceListener.use(validateBasicData);
    });
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';

  // Listen to Events from customer portal Spaces
  listener.namespace('space:customer-portal', (customerPortalListener: FlatfileListener) => {
    customerPortalListener.use(applyCustomerBranding);
    customerPortalListener.use(configureGuidedOnboarding);
  });

  // Listen to Events from internal tools Spaces
  listener.namespace('space:internal-tools', (internalToolsListener: FlatfileListener) => {
    internalToolsListener.use(validateAdminData);
    internalToolsListener.use(enableAuditLogging);
  });

  // Listen to Events from partner integration Spaces
  listener.namespace('space:partner-integration', (partnerIntegrationListener: FlatfileListener) => {
    partnerIntegrationListener.use(configureApiWebhooks);
    partnerIntegrationListener.use(enableBulkProcessing);
  });

  // Listen to Events from general Spaces (default)
  listener.namespace('space:general', (generalSpaceListener: FlatfileListener) => {
    generalSpaceListener.use(validateBasicData);
  });
  ```
</CodeGroup>

### Workbook Namespaces

[Workbooks](/core-concepts/workbooks) can also have namespaces for more granular Event filtering within the same [Space](/core-concepts/spaces). This can be set directly in the Workbook's [Blueprint](/core-concepts/blueprints).

This example configures a structure with two Workbooks in the same Space, with a flow for moving data from the staging Workbook to the production Workbook.

#### Setting Workbook Namespaces

<CodeGroup>
  ```javascript JavaScript theme={null}
  const workbook = {
    name: 'Employee Data Processing',
    namespace: 'staging', // Simple string namespace
    sheets: [employeesSheet, departmentsSheet, payrollSheet]
  };
  ```

  ```typescript TypeScript theme={null}
  import { Workbook } from '@flatfile/api';

  const workbook: Workbook = {
    name: 'Employee Data Processing',
    namespace: 'staging', // Simple string namespace
    sheets: [employeesSheet, departmentsSheet, payrollSheet]
  };
  ```
</CodeGroup>

<CodeGroup>
  ```javascript JavaScript theme={null}
  const workbook = {
    name: 'Employee Data Processing: ',
    namespace: 'production', // Simple string namespace
    sheets: [employeesSheet, departmentsSheet, payrollSheet]
  };
  ```

  ```typescript TypeScript theme={null}
  import { Workbook } from '@flatfile/api';

  const workbook: Workbook = {
    name: 'Employee Data Processing: ',
    namespace: 'production', // Simple string namespace
    sheets: [employeesSheet, departmentsSheet, payrollSheet]
  };
  ```
</CodeGroup>

#### Listening to Workbook Namespace Events

Events from Workbooks with namespaces are filtered using the `workbook:namespace` pattern.

<CodeGroup>
  ```javascript JavaScript theme={null}
  listener.use(configureAllWorkbooks);
  listener.use(validateData);

  // Listen to Events from Workbooks with "staging" namespace
  listener.namespace('workbook:staging', (stagingWorkbookListener) => {
    stagingWorkbookListener.use(migrateStagingDataToProduction);
  });

  // Listen to Events from Workbooks with "production" namespace
  listener.namespace('workbook:production', (productionWorkbookListener) => {
    productionWorkbookListener.use(applyBusinessLogic);
    productionWorkbookListener.use(enableAuditing);
  });
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';

  listener.use(configureAllWorkbooks);
  listener.use(validateData);

  // Listen to Events from Workbooks with "staging" namespace
  listener.namespace('workbook:staging', (stagingWorkbookListener: FlatfileListener) => {
    stagingWorkbookListener.use(migrateStagingDataToProduction);
  });

  // Listen to Events from Workbooks with "production" namespace
  listener.namespace('workbook:production', (productionWorkbookListener: FlatfileListener) => {
    productionWorkbookListener.use(applyBusinessLogic);
    productionWorkbookListener.use(enableAuditing);
  });
  ```
</CodeGroup>

### Listening to Multiple Namespaces

The `listener.namespace()` function can accept an array of namespace patterns as its first argument, allowing you to listen to Events from multiple namespaces with a single Listener configuration. This is useful when you want to apply the same processing logic across different namespaces.

You can also mix different namespace types in the same array. The Listener in this example will receive Events from both the `space:admin-tools` and `workbook:critical-data` namespaces:

<CodeGroup>
  ```javascript JavaScript theme={null}
  // Listen to events from specific Spaces and Workbooks
  listener.namespace(['space:admin-tools', 'workbook:critical-data'], (adminCriticalListener) => {
    adminCriticalListener.use(enableHighSecurityMode);
    adminCriticalListener.use(requireApproval);
    adminCriticalListener.use(flagForReview);
  });
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';

  // Listen to events from specific Spaces and Workbooks
  listener.namespace(['space:admin-tools', 'workbook:critical-data'], (adminCriticalListener: FlatfileListener) => {
    adminCriticalListener.use(enableHighSecurityMode);
    adminCriticalListener.use(requireApproval);
    adminCriticalListener.use(flagForReview);
  });
  ```
</CodeGroup>

### Nested Namespace Example

You can also nest namespacing: let's say you have two Apps (`customer-portal` and `vendor-portal`), and each App processes data through different Workbook types (`invoices` and `orders`). You want to ensure that each App's Workbooks are completely isolated from the other App, but within each App you want specific handling for each Workbook type.

This example demonstrates how App, Space, and Workbook namespaces work together with nested Listeners:

#### Setting Up Multiple Apps

First, create two Apps via the [Flatfile Dashboard](https://platform.flatfile.com/dashboard), each with distinct namespaces:

<CardGroup>
  <Card title="Customer Portal">
    **App Name:** Customer Portal\
    **Namespace:** `customer-portal`\
    **Purpose:** External customer data import and processing
  </Card>

  <Card title="Vendor Portal">
    **App Name:** Vendor Portal\
    **Namespace:** `vendor-portal`\
    **Purpose:** B2B vendor data exchange and management
  </Card>
</CardGroup>

#### Nested Listener Configuration

<Info>
  To help reduce scrolling for this example, we've split it into two tabs: **Blueprints** and **Listener Configuration**.
</Info>

<Tabs>
  <Tab title="Blueprints" icon="table">
    Each App requires different field structures for their invoice and order processing, so we define App-specific Blueprint configurations. See the next tab for the Listener Configuration.

    <CodeGroup>
      ```javascript JavaScript theme={null}
      // Customer Portal Blueprint configurations
      const customerInvoiceWorkbook = {
        name: "Customer Invoice Processing",
        namespace: "invoices",
        sheets: [{
          name: "Customer Invoices",
          slug: "invoices",
          fields: [
            { key: "invoice_number", type: "string", label: "Invoice Number" },
            { key: "customer_name", type: "string", label: "Customer Name" },
            { key: "billing_amount", type: "number", label: "Billing Amount" }
          ]
        }]
      };

      const customerOrderWorkbook = {
        name: "Customer Order Processing",
        namespace: "orders",
        sheets: [{
          name: "Customer Orders",
          slug: "orders",
          fields: [
            { key: "order_id", type: "string", label: "Order ID" },
            { key: "customer_name", type: "string", label: "Customer Name" },
            { key: "order_total", type: "number", label: "Order Total" }
          ]
        }]
      };

      // Vendor Portal Blueprint configurations
      const vendorInvoiceWorkbook = {
        name: "Vendor Invoice Processing",
        namespace: "invoices",
        sheets: [{
          name: "Vendor Invoices",
          slug: "invoices",
          fields: [
            { key: "vendor_invoice_id", type: "string", label: "Vendor Invoice ID" },
            { key: "vendor_company", type: "string", label: "Vendor Company" },
            { key: "payment_due", type: "number", label: "Payment Due" }
          ]
        }]
      };

      const vendorOrderWorkbook = {
        name: "Vendor Purchase Orders",
        namespace: "orders",
        sheets: [{
          name: "Purchase Orders",
          slug: "orders",
          fields: [
            { key: "po_number", type: "string", label: "PO Number" },
            { key: "supplier_name", type: "string", label: "Supplier Name" },
            { key: "purchase_amount", type: "number", label: "Purchase Amount" }
          ]
        }]
      };
      ```

      ```typescript TypeScript theme={null}
      import { Flatfile } from '@flatfile/api';

      // Customer Portal Blueprint configurations
      const customerInvoiceWorkbook: Flatfile.CreateWorkbookConfig = {
        name: "Customer Invoice Processing",
        namespace: "invoices",
        sheets: [{
          name: "Customer Invoices",
          slug: "invoices",
          fields: [
            { key: "invoice_number", type: "string", label: "Invoice Number" },
            { key: "customer_name", type: "string", label: "Customer Name" },
            { key: "billing_amount", type: "number", label: "Billing Amount" }
          ]
        }]
      };

      const customerOrderWorkbook: Flatfile.CreateWorkbookConfig = {
        name: "Customer Order Processing",
        namespace: "orders",
        sheets: [{
          name: "Customer Orders",
          slug: "orders",
          fields: [
            { key: "order_id", type: "string", label: "Order ID" },
            { key: "customer_name", type: "string", label: "Customer Name" },
            { key: "order_total", type: "number", label: "Order Total" }
          ]
        }]
      };

      // Vendor Portal Blueprint configurations
      const vendorInvoiceWorkbook: Flatfile.CreateWorkbookConfig = {
        name: "Vendor Invoice Processing",
        namespace: "invoices",
        sheets: [{
          name: "Vendor Invoices",
          slug: "invoices",
          fields: [
            { key: "vendor_invoice_id", type: "string", label: "Vendor Invoice ID" },
            { key: "vendor_company", type: "string", label: "Vendor Company" },
            { key: "payment_due", type: "number", label: "Payment Due" }
          ]
        }]
      };

      const vendorOrderWorkbook: Flatfile.CreateWorkbookConfig = {
        name: "Vendor Purchase Orders",
        namespace: "orders",
        sheets: [{
          name: "Purchase Orders",
          slug: "orders",
          fields: [
            { key: "po_number", type: "string", label: "PO Number" },
            { key: "supplier_name", type: "string", label: "Supplier Name" },
            { key: "purchase_amount", type: "number", label: "Purchase Amount" }
          ]
        }]
      };
      ```
    </CodeGroup>
  </Tab>

  <Tab title="Listener Configuration" icon="code">
    <Note>
      Unlike other examples, this one uses the [Space Configure](/plugins/space-configure) plugin to configure the Space. The alternative – Space configuration using the full Job Lifecycle management approach – is more instructive, but significantly more verbose. See the [Space Configuration](/core-concepts/spaces#space-configuration) section for examples of that.

      The Space Configure plugin takes a lot of heavy lifting off your hands; not only does it handle the Job lifecycle, but it also takes care of all of the API calls necessary to configure the Space and create its Workbooks and documents.

      With this plugin, you can configure your entire space with a single configuration object rather than perforing any API calls.
    </Note>

    <CodeGroup>
      ```javascript JavaScript theme={null}
      import { configureSpace } from '@flatfile/plugin-space-configure';

      export default function (listener) {
        // Namespaced for the Customer Portal App
        listener.namespace("space:customer-portal", (customerListener) => {

          // Configure Space using the plugin
          customerListener.use(configureSpace({
            workbooks: [customerInvoiceWorkbook, customerOrderWorkbook]
          }));

          // Handle Events from Invoice Workbooks in the Customer Portal App
          customerListener.namespace("workbook:invoices", (customerInvoiceListener) => {
            customerInvoiceListener.use(validateCustomerInvoiceData);
          });

          // Handle Events from Order Workbooks in the Customer Portal App
          customerListener.namespace("workbook:orders", (customerOrderListener) => {
            customerOrderListener.use(validateCustomerOrderData);
          });
        });

        // Namespaced for the Vendor Portal App
        listener.namespace("space:vendor-portal", (vendorListener) => {

          // Configure Space using the plugin
          vendorListener.use(configureSpace({
            workbooks: [vendorInvoiceWorkbook, vendorOrderWorkbook]
          }));

          // Handle Events from Invoice Workbooks in the Vendor Portal App
          vendorListener.namespace("workbook:invoices", (vendorInvoiceListener) => {
            vendorInvoiceListener.use(validateVendorInvoices);
          });

          // Handle Events from Order Workbooks in the Vendor Portal App
          vendorListener.namespace("workbook:orders", (vendorOrderListener) => {
            vendorOrderListener.use(validateVendorOrders);
          });
        });
      }
      ```

      ```typescript TypeScript theme={null}
      import { FlatfileListener } from '@flatfile/listener';
      import { Flatfile } from '@flatfile/api';
      import { configureSpace } from '@flatfile/plugin-space-configure';

      export default function (listener: FlatfileListener) {

        // Namespaced for the Customer Portal App
        listener.namespace("space:customer-portal", (customerListener: FlatfileListener) => {

          // Configure Space using the plugin
          customerListener.use(configureSpace({
            workbooks: [customerInvoiceWorkbook, customerOrderWorkbook]
          }));

          // Handle Events from Invoice Workbooks in the Customer Portal App
          customerListener.namespace("workbook:invoices", (customerInvoiceListener: FlatfileListener) => {
            customerInvoiceListener.use(validateCustomerInvoiceData);
          });

          // Handle Events from Order Workbooks in the Customer Portal App
          customerListener.namespace("workbook:orders", (customerOrderListener: FlatfileListener) => {
            customerOrderListener.use(validateCustomerOrderData);
          });
        });

        // Namespaced for the Vendor Portal App
        listener.namespace("space:vendor-portal", (vendorListener: FlatfileListener) => {

          // Configure Space using the plugin
          vendorListener.use(configureSpace({
            workbooks: [vendorInvoiceWorkbook, vendorOrderWorkbook]
          }));

          // Handle Events from Invoice Workbooks in the Vendor Portal App
          vendorListener.namespace("workbook:invoices", (vendorInvoiceListener: FlatfileListener) => {
            vendorInvoiceListener.use(validateVendorInvoices);
          });

          // Handle Events from Order Workbooks in the Vendor Portal App
          vendorListener.namespace("workbook:orders", (vendorOrderListener: FlatfileListener) => {
            vendorOrderListener.use(validateVendorOrders);
          });
        });
      }
      ```
    </CodeGroup>
  </Tab>
</Tabs>

## Event Filters

### Understanding Event Filtering

The `listener.filter()` method creates filtered Listener instances that only respond to Events matching specific criteria, returning a new `FlatfileListener` instance with the applied filter conditions.

### Basic Filtering

The most fundamental use of `listener.filter()` is to respond to specific Events based on simple criteria. This approach is useful when you want different processing logic for different types of Jobs or Events within the same namespace, without creating separate Event handlers for each case.

<CodeGroup>
  ```javascript JavaScript theme={null}
  // With callback function for multiple handlers
  listener.filter({ sheet: 'contacts' }, (contactsListener) => {
    contactsListener.on('commit:created', async (event) => {
      console.log('Contact data committed');
      // Process contact data validation
    });

    contactsListener.on('records:created', async (event) => {
      console.log('New contacts added');
      // Handle contact creation workflow
    });
  });

  // Chaining `filter()` with `on()`
  listener
    .filter({ sheet: 'contacts' })
    .on('records:updated', async (event) => {
      console.log('Contact updated');
    // Handle contact updates
    });
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener, FlatfileEvent } from '@flatfile/listener';

  // With callback function for multiple handlers
  listener.filter({ sheet: 'contacts' }, (contactsListener: FlatfileListener) => {
    contactsListener.on('commit:created', async (event: FlatfileEvent) => {
      console.log('Contact data committed');
      // Process contact data validation
    });

    contactsListener.on('records:created', async (event: FlatfileEvent) => {
      console.log('New contacts added');
      // Handle contact creation workflow
    });
  });

  // Chaining `filter()` with `on()`
  listener
    .filter({ sheet: 'contacts' })
    .on('records:updated', async (event: FlatfileEvent) => {
    console.log('Contact updated');
    // Handle contact updates
  });
  ```
</CodeGroup>

### Wildcard Filtering

Filters support wildcard patterns using `*` to match partial values. This may be useful when you want to filter by ID patterns or prefixes.

This example filters for `commit:created` events that were initiated by [Jobs](/core-concepts/jobs) rather than users. When a Job causes changes to your data (commits), the `actorId` in the event context will be the job ID (starting with `"us_jb"`):

<CodeGroup>
  ```javascript JavaScript theme={null}
  // Use wildcard to filter for events initiated by jobs (actorId starts with "us_jb")
  listener
    .filter({actorId: "us_jb*"}) // note the * wildcard
    .on("commit:created", async (event) => {
      // Get the job details that caused this commit
      const { data: job } = await api.jobs.get(event.context.actorId);
      console.log(`Job ${job.operation} caused a commit in sheet ${event.context.sheetId}`);
      // ...React to job-initiated commits
    });
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener, FlatfileEvent } from '@flatfile/listener';

  // Use wildcard to filter for events initiated by jobs (actorId starts with "us_jb")
  listener
    .filter({actorId: "us_jb*"}) // note the * wildcard
    .on("commit:created", async (event: FlatfileEvent) => {
      // Get the job details that caused this commit
      const { data: job } = await api.jobs.get(event.context.actorId);
      console.log(`Job ${job.operation} caused a commit in sheet ${event.context.sheetId}`);
      // ...React to job-initiated commits
    });
  ```
</CodeGroup>

### Chaining Filters and Namespaces

You can also chain multiple filters along with namespaces to isolate highly specific events. This example shows how to combine a namespace with two filters to handle failed submit jobs for third-party integrations:

<CodeGroup>
  ```javascript JavaScript theme={null}
  // Progressive filtering for highly specific event targeting
  listener
    .namespace('space:third-party-integrations')
    .filter({job: `workbook:submit`})
    .filter({"payload.status": "failed"})
    .on('job:updated', async (event) => {
      const { data: job } = await api.jobs.get(event.context.jobId);
      handleFailedThirdPartySubmissions(job);
    });
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener, FlatfileEvent } from '@flatfile/listener';

  // Progressive filtering for highly specific event targeting
  listener
    .namespace('space:third-party-integrations')
    .filter({job: `workbook:submit`})
    .filter({"payload.status": "failed"})
    .on('job:updated', async (event: FlatfileEvent) => {
      const { data: job } = await api.jobs.get(event.context.jobId);
      handleFailedThirdPartySubmissions(job);
    });
  ```
</CodeGroup>

### Filter Properties

The `listener.filter()` method accepts an object defining filter criteria based on Event properties.

<Warning>
  **Important**: This is not intended to be a comprehensive list of all possible filter properties, but a reference for commonly-used ones.

  Event properties vary significantly by event type, and using a filter property that doesn't exist for an event type will result in no matches.

  Refer to the [Event Reference](/reference/events) for the specific properties available for each event you want to filter on. You can always `console.log()` your events prior to filtering to see what properties are available.
</Warning>

#### Universal Properties

These properties are available for filtering across most or all event types:

| Property        | Description                       | Example                           |
| --------------- | --------------------------------- | --------------------------------- |
| `topic`         | Event topic pattern               | `{ topic: 'records:created' }`    |
| `domain`        | Event domain (supports wildcards) | `{ domain: 'space' }`             |
| `environmentId` | Environment identifier            | `{ environmentId: 'us_env_123' }` |

#### Common Context Properties

These properties are available in many events but not all:

| Property    | Description        | Example                       | Available In                         |
| ----------- | ------------------ | ----------------------------- | ------------------------------------ |
| `spaceId`   | Specific space     | `{ spaceId: 'us_sp_789' }`    | Most events (not environment events) |
| `actorId`   | Specific actor     | `{ actorId: 'us_usr_123' }`   | Most user-initiated events           |
| `accountId` | Account identifier | `{ accountId: 'us_acc_123' }` | Most events                          |

#### Specific Context Properties

These properties are only available for certain event types:

| Property     | Description       | Example                       | Available In                             |
| ------------ | ----------------- | ----------------------------- | ---------------------------------------- |
| `workbookId` | Specific workbook | `{ workbookId: 'us_wb_456' }` | Workbook, sheet, record, some job events |
| `sheetId`    | Specific sheet ID | `{ sheetId: 'us_sh_123' }`    | Sheet, record, program events            |
| `sheet`      | Sheet slug        | `{ sheet: 'contacts' }`       | Sheet, record, program events            |
| `jobId`      | Specific job ID   | `{ jobId: 'us_jb_123' }`      | Job events only                          |
| `fileId`     | File identifier   | `{ fileId: 'us_fl_123' }`     | File events only                         |

#### Job Event Filters

Special filter properties for job events:

| Property            | Description         | Example                                |
| ------------------- | ------------------- | -------------------------------------- |
| `job`               | Job type identifier | `{ job: 'workbook:submit' }`           |
| `payload.status`    | Job status          | `{ 'payload.status': 'failed' }`       |
| `payload.domain`    | Event domain        | `{ 'payload.domain': 'space' }`        |
| `payload.operation` | Operation name      | `{ 'payload.operation': 'configure' }` |

#### Record Event Filters

Specific to record creation, update, and deletion events:

| Property              | Description                  | Example                                |
| --------------------- | ---------------------------- | -------------------------------------- |
| `payload.recordIds`   | Array of affected record IDs | `{ 'payload.recordIds': ['rec_123'] }` |
| `payload.recordCount` | Number of records affected   | `{ 'payload.recordCount': 5 }`         |
| `payload.sheetId`     | Sheet containing the records | `{ 'payload.sheetId': 'us_sh_123' }`   |

#### File Event Filters

Specific to file upload and processing events:

| Property             | Description            | Example                                 |
| -------------------- | ---------------------- | --------------------------------------- |
| `payload.status`     | File processing status | `{ 'payload.status': 'completed' }`     |
| `payload.workbookId` | Associated workbook    | `{ 'payload.workbookId': 'us_wb_123' }` |

You can also use various pattern matching approaches including exact matches, arrays of values, and wildcard patterns with a `*`.


# Guides Overview
Source: https://flatfile.com/docs/guides/overview

Comprehensive guides to help you master Flatfile's features and capabilities

Welcome to Flatfile's comprehensive guide collection. These guides provide step-by-step instructions and best practices for implementing advanced features and customizations in your Flatfile integration.

## Core Functionality

Essential guides for working with Flatfile's core features and data processing capabilities.

<CardGroup>
  <Card title="Authentication" href="/guides/authentication">
    Complete guide to authenticating with Flatfile using API keys, Personal Access Tokens, and managing roles and permissions for your team and customers.
  </Card>

  <Card title="Using Actions" href="/guides/using-actions">
    Learn how to trigger operations based on user input. Actions allow you to create custom workflows that execute when users interact with buttons in your Flatfile interface.
  </Card>

  <Card title="Handling Data" href="/guides/using-record-hooks">
    Master data processing with Data Hooks. This guide covers how to validate, transform, and process records as they're imported into Flatfile.
  </Card>

  <Card title="Accepting Additional Fields" href="/guides/accepting-additional-fields">
    Enable dynamic field creation during data import. Learn how to allow users to add new fields on the fly when their data doesn't match your predefined Blueprint.
  </Card>
</CardGroup>

## Data Processing & Filtering

Advanced techniques for filtering, querying, and transforming your data.

<CardGroup>
  <Card title="Advanced Filters" href="/guides/advanced-filters">
    Create sophisticated filter combinations with complex conditions. Learn how to use logical operators, multiple criteria, and various data types to find exactly the records you need.
  </Card>

  <Card title="Custom Extractors" href="/guides/custom-extractors">
    Build custom file parsers to handle proprietary or uncommon file formats. Learn how to create extractors that can parse any file type and integrate seamlessly with Flatfile's processing pipeline.
  </Card>

  <Card title="Using Flatfile Query Language" href="/reference/ffql">
    Master FFQL (Flatfile Query Language) to filter data in sheets with powerful query syntax. Build complex queries to search and manipulate your data programmatically.
  </Card>
</CardGroup>

## Configuration & Customization

Guides for customizing the look, feel, and behavior of your Flatfile implementation.

<CardGroup>
  <Card title="Theme Your Space" href="/guides/theme-your-space">
    Customize the look and feel of Flatfile to match your brand. This comprehensive guide covers all theming options including colors, fonts, and UI elements.
  </Card>

  <Card title="Translating Your Space" href="/guides/translating-your-space">
    Implement internationalization with text overrides and translations. Make your Flatfile interface accessible to users in different languages and regions.
  </Card>

  <Card title="Customize the Guest Sidebar" href="/guides/customize-guest-sidebar">
    Control what guests see in their sidebar interface. Learn how to limit and customize the sidebar experience for different user types.
  </Card>
</CardGroup>

## Data Management

Guides for managing data lifecycle, retention, and export processes.

<CardGroup>
  <Card title="Egress" href="/guides/egress">
    Get data out of Flatfile to external destinations. Master various patterns for exporting and syncing data to your systems and third-party services.
  </Card>

  <Card title="Utilizing Metadata" href="/guides/utilizing-metadata">
    Store and manage descriptive information that provides additional context to your data. Learn how to effectively use metadata throughout your Flatfile implementation.
  </Card>

  <Card title="Multi-Part Jobs" href="/guides/multi-part-jobs">
    Split complex operations into manageable parts for better performance and user experience. Learn how to break down large jobs into smaller, trackable components.
  </Card>
</CardGroup>

## Advanced Features

Specialized guides for advanced Flatfile features and enterprise capabilities.

<CardGroup>
  <Card title="Namespaces and Filters" href="/guides/namespaces-and-filters">
    Organize and isolate resources using namespaces. Learn how to narrow down the scope of Spaces, Workbooks, and Sheets for better organization and security.
  </Card>

  <Card title="Share Secrets" href="/guides/share-secrets">
    Securely manage and use credentials in your listeners. Learn best practices for handling sensitive information like API keys and authentication tokens.
  </Card>

  <Card title="Authentication Examples" href="/guides/deeper/auth-examples">
    Standardized authentication setup patterns for consistent integration across all Flatfile projects.
  </Card>

  <Card title="Webhook Examples" href="/guides/deeper/webhook-examples">
    Standardized webhook patterns for reliable data submission to external systems.
  </Card>
</CardGroup>

## Getting Help

Need additional assistance? Here are some resources:

* **[Community](https://flatfile.com/join-slack/)** - Join our Slack community for questions and discussions
* **[Support](https://flatfile.com/join-slack/)** - Get help from our support team
* **[API Reference](https://reference.flatfile.com)** - Detailed API documentation
* **[Platform](https://platform.flatfile.com)** - Access your Flatfile dashboard

Each guide includes practical examples, code snippets, and best practices to help you implement these features effectively in your own applications.


# Secrets
Source: https://flatfile.com/docs/guides/share-secrets

Securely use credentials in listeners

With Secrets you can securely share credentials with listener implementations without developers explicitly knowing the secret values upfront. Secret values are set in the user interface, but retrieved via the SDK or API.

## Overview

### Creating Secrets

Secrets in Flatfile, defined as Name/Value pairs, are securely stored and associated with an Environment or a Space. Spaces will inherit Secrets from their respective Environment but you may choose to override any Environment Secret for a given Space. To define Secrets shared with every Space in an Environment, navigate to the "Developer Settings" screen for that environment. To override an Environment value, navigate to the specific Space and select "Secrets" in the left navigation.

While Flatfile encrypts all data, both during transit and at rest in our datastore, Secrets have an additional layer of protection. Secrets are encrypted/decrypted on demand using a unique set of keys. As such, a potential intruder would need not only access to the plaintext datastore, but also these extra keys to decrypt and compromise these sensitive values.

### Consuming Secrets

While Secrets are defined in administrative interfaces for Environments and Spaces, respectively, they are designed to be consumed by Listeners. While it might be trivial to pass in secret values through environment variables in a self-hosted Listener, with a Flatfile hosted Agent based Listener one must use the Secrets features. See Usage below for some example consumer patterns.

## Usage Examples

### Sensitive Credentials

The principal utility of Secrets lies in securely storing sensitive credentials/tokens within an Environment/Space for connecting Listeners to third-party APIs. For instance, you might store a secret named `SLACK_TOKEN` with a value, allowing you to communicate with a Slack bot each time a custom action is triggered.

For complete examples of secure credential management, see the [Authentication Examples](/guides/deeper/auth-examples#secure-credential-management) guide.

#### Example Listener

In this example, we use an `event.secrets` call to pull a sensitive Slack token for use within a listener context. We then can use the credential to post a message to Slack.

```javascript theme={null}
export default function flatfileEventListener(listener) {
  //note: listening to all events with a wildcard can be used while testing but is not
  //recommended for production, as it will capture all events and may cause performance issues
  listener.on("**", async (event) => {
    const tok = await event.secrets("SLACK_TOKEN");
    console.log(tok);
    /* pseudo code for an example
    slack = new Slack(tok);
    slack.api(
      "chat.postMessage",
      {
        text: "Flatfile event received!",
        channel: "#integration-flatfile",
      },
      function (err, response) {
        console.log(response || err);
      }
    ); */
  });
}
```

#### Example Listener using optional props

The `options` parameter for the secrets fetch function allows optionally choosing a different Environment or Space than the event occurred within.

```javascript theme={null}
export default function flatfileEventListener(listener) {
  //note: listening to all events with a wildcard can be used while testing but is not
  //recommended for production, as it will capture all events and may cause performance issues
  listener.on("**", async (event) => {
    // Hardcode specific environment and space for this listener's case
    const credential = await event.secrets("MY_CREDENTIAL", {
      environmentId: "us_env_123",
      spaceId: "us_spa_123",
    });
    console.log(credential);
  });
}
```

## Metadata

While it might seem creative to use the Secrets feature to hold non-sensitive metadata, we encourage you to learn more about utilizing metadata within your Spaces, Records, or Fields.


# Theming
Source: https://flatfile.com/docs/guides/theme-your-space

Learn how to customize the look and feel of Flatfile to match your brand

Flatfile supports modifying most UI elements including colors, fonts, borders, padding, and more via the [Space](/core-concepts/spaces) endpoint.

1. Start by simply updating `theme.root.primaryColor` and `theme.sidebar.logo` when calling `spaces.update()`.
2. If needed, you can customize the theme further with additional css variables.

## Building a theme

Learn how to create a Space with a theme, and update a theme from an Event listener.

```javascript theme={null}
import api from "@flatfile/api";

export default function flatfileEventListener(listener) {
  //listen for space:configure job and build out space
  listener.filter({ job: "space:configure" }, (configure) => {});

  //theme during creation or update your space after it is created
  listener.on("space:created", async ({ context: { spaceId } }) => {
    const updateSpace = await api.spaces.update(spaceId, {
      metadata: {
        theme: {
          root: {
            primaryColor: "red",
          },
          sidebar: {
            logo: "https://image.png",
          },
          // See reference for all possible variables
        },
      },
    });
  });
}
```

See full code example in our [flatfile-docs-kitchen-sink Github repo](https://github.com/FlatFilers/flatfile-docs-kitchen-sink/blob/main)

## Theme reference

### theme.root

<img alt="Theme root configuration example" />

<Tip>
  The sidebar, table and document will automatically inherit theming from your
  root variables!
</Tip>

#### Font

**fontFamily** `string`\
The font family used throughout the application. Only system fonts are supported at the moment

#### Colors

**primaryColor** `string`\
The primary color used throughout the application.

**dangerColor** `string`\
The color used for error messages.

**warningColor** `string`\
The color used for warning messages.

**borderColor** `string`\
The color used for borders throughout the application.

### theme.sidebar

<img alt="Theme sidebar configuration example" />

<Info>
  You can override the default colors of the sidebar below. If these are not set
  they will inherit from your root colors.
</Info>

**logo** `string`\
The img path for the logo displayed in the sidebar.

**textColor** `string`\
The color of the text in the sidebar.

**titleColor** `string`\
The color of the title in the sidebar.

**focusBgColor** `string`\
The background color of the active navigation link. The hover state will use the same color with 30% opacity applied.

**focusTextColor** `string`\
The text color of the active/focused navigation link.

**backgroundColor** `string`\
The background color of the sidebar.

### theme.table

<img alt="Theme table configuration example" />

<Info>
  You can override the default colors and values for different elements in the
  table below. If these are not set they will inherit from your root values.
</Info>

#### Font

**fontFamily** `string`\
The font family used throughout the table. Only system fonts are supported at the moment

#### Active cell

**cell.active.borderWidth** `string`\
The width of the border around the active cell.

**cell.active.borderShadow** `string`\
The box shadow around the active cell.

**cell.number.fontFamily** `string`\
The font family for number cells.

#### Column header

<img alt="Theme table column configuration" />

**column.header.backgroundColor** `string`\
The background color of the column headers in the table.

**column.header.color** `string`\
The text color of the column headers in the table.

#### Index column

<img alt="Theme table index column configuration" />

**indexColumn.backgroundColor** `string`\
The background color of the first column in the table.

**indexColumn.color** `string`\
The text color of the first column in the table.

**indexColumn.selected.backgroundColor** `string`\
The background color of the first column in the table when selected.

#### Checkboxes

<img alt="Theme table inputs configuration" />

**inputs.checkbox.color** `string`\
The color of the checkboxes in the table.

**inputs.checkbox.borderColor** `string`\
The color of the border for the checkboxes in the table.

#### Filters

**filters.outerBorderRadius** `string`\
The border radius of the table filters

**filters.innerBorderRadius** `string`\
The border radius for the individual filters

**filters.outerBorder** `string`\
The border of the table filters. By default there is no border.

> When nested objects share the same border radius, small gaps may appear. To resolve this, adjust the inner and outer border radius on the filters to seamlessly fill any gaps.

#### Buttons

**buttons.iconColor** `string`\
The color of the icon buttons in the toolbar and table

**buttons.pill.color** `string`\
The color of the pill buttons in the toolbar

### theme.email

You can theme the guest invite emails as well as guest email verification emails via the properties below. These are sent to a guest when they are invited to a Space via the guest management page in your Space.

> Email theming is only available on the pro and enterprise plans.

**logo** `string`\
The URL of the image displayed at the top of the email

**textColor** `string`\
The color of the text in the email

**titleColor** `string`\
The color of the title in the email

**buttonBgColor** `string`\
The background color of the button

**buttonTextColor** `string`\
The text color of the button

**footerTextColor** `string`\
The text color of the footer text

**backgroundColor** `string`\
The background color of the email

#### theme.email.dark

If your default email theme (as set above) is light, we suggest adding a dark mode email theme. Different email providers handle dark and light mode differently. While Apple Mail and some other clients will respect dark mode variables, some email providers do not support dark mode and will display your default theme. We suggest you test your themes across various email clients before going to production to ensure consistent appearance.

**logo** `string`\
The URL of the image displayed at the top of the email in dark mode

**textColor** `string`\
The color of the text in the email in dark mode

**titleColor** `string`\
The color of the title in the email in dark mode

**buttonBgColor** `string`\
The background color of the button in dark mode

**buttonTextColor** `string`\
The text color of the button in dark mode

**footerTextColor** `string`\
The text color of the footer text in dark mode

**backgroundColor** `string`\
The background color of the email in dark mode

## Deprecation Notice

Flatfile's new design system released in Q1 2025 supports a more streamlined experience when configuring theme. The new system accepts a more limited set of properties, but ensures those supplied properties are cohesively applied across the application.

As a result, many of the original `theme.root` properties which applied to specific UI elements have been deprecated. We have taken steps to ensure some level of backwards compatibility for these properties in the new system - however we recommend any usage of these properties be updated to the new system as soon as possible.

## Example Project

Find the theming example in the Flatfile GitHub repository.

* [Clone the theming example in Typescript](https://github.com/FlatFilers/flatfile-docs-kitchen-sink/blob/main/typescript/theming/index.ts)
* [Clone the theming example in Javascript](https://github.com/FlatFilers/flatfile-docs-kitchen-sink/blob/main/javascript/theming/index.js)


# Internationalization
Source: https://flatfile.com/docs/guides/translating-your-space

Translate and customize your Space with text overrides

Tailor the language and text in your Flatfile Space with ease! This guide will lead you step-by-step to allow you to both translate and personalize the text in your Space in just a couple of simple steps.

## Available Languages

We support translations into these languages:

| Language                | Code       |
| ----------------------- | ---------- |
| German                  | de         |
| English                 | en         |
| English - Great Britain | en-GB      |
| English - Canadian      | en-CA      |
| English - South African | en-ZA      |
| Spanish                 | es         |
| Spanish - Latin America | es-419     |
| French                  | fr         |
| French - Canadian       | fr-CA      |
| French - France         | fr-FR      |
| Indonesian              | id         |
| Italian                 | it         |
| Japanese                | jp         |
| Korean                  | kr         |
| Malay                   | ms         |
| Dutch                   | nl         |
| Polish                  | pl         |
| Portuguese              | pt         |
| Portuguese - Brazilian  | pt-BR      |
| Swedish                 | sv         |
| Thai                    | th         |
| Turkish                 | tr         |
| Vietnamese              | vi         |
| Chinese - Simplified    | zh         |
| Chinese - Traditional   | zh-Hant    |
| Chinese - Hong Kong     | zh-Hant-HK |

Need more languages? Reach out to us at [support@flatfile.io](mailto:support@flatfile.io).

## Setting the language on a Space

Flatfile will auto-detect the user's language based on their browser settings. If the language is not detectable or is currently unsupported, the system will automatically be set to English.

### Overriding the language

You can manually set the language on a Space using the `languageOverride` property. This property can be set either directly on the Space or at the environment level if the Space has not yet been created.

All users who use the Space will only see the language set in the language override property. Browser settings will no longer be respected.

To specify the language, simply assign the appropriate language code (e.g., 'en' for English, 'fr' for French, etc.) to the languageOverride property.

```typescript listener.ts theme={null}
listener.on("space:created", async ({ context: { spaceId } }) => {
  // This space will be in English regardless of the users browser settings
  const updateSpace = await api.spaces.update(spaceId, {
    languageOverride: "en",
  });
});
```

## Adding translations

### Creating your translation files

All our translation files can be found in our [Platform-Translations repository](https://github.com/FlatFilers/Platform-Translations). You can fork the repo or simply copy paste the files in the repository to create your own translation files.

Simply update the values in each file with the text you would like to see.

Once you have created your own translation files, you need to make them available to Flatfile. This is done via the translationsPath property.

While creating your JSON translation files, you can opt for partial or full JSON objects:

**Partial JSON Objects:** Just include those values you want to change. Be sure to maintain the same JSON structure to avoid errors.

**Full JSON Objects:** Copy the translation file from our public repository and modify the values as required.

### Setting your Translations Path

To provide translations and text overrides you can use the translationsPath parameter. You have the flexibility to set up translationsPaths at two levels:

* **Space:** Overrides the environment level setting.
* **Environment:** Used when no Space-level path is defined.

Set your translationsPath to the URL of your **english JSON file**. This should be the URL ending in `/locales/en/translation.json`.

Don't have an english file? You can use any other language instead.

Your `translationsPath` must meet a few requirements to work as expected:

1. It must be a string.
2. The string should be a publicly available URL that returns your JSON translation file.
3. It must follow this folder structure:
   ```
   yourCustomTranslationsDirectory/
       └── locales/
           ├── en/
           │   └── translation.json
           ├── es/
           │   └── translation.json
           ...
   ```
   where `yourCustomTranslationsDirectory` can be swapped out for any path desired.

For our translations repository we would set our translationsPath to: [https://raw.githubusercontent.com/FlatFilers/Platform-Translations/main/locales/en/translation.json](https://raw.githubusercontent.com/FlatFilers/Platform-Translations/main/locales/en/translation.json)

### Implementing Translations Path

Set up your translations path using our API or the listener. This can be done at both the Space and environment levels.

```typescript listener.ts theme={null}
listener.on("space:created", async ({ context: { spaceId } }) => {
  //set the translation path for the Space
  const updateSpace = await api.spaces.update(spaceId, {
    translationsPath:
      "https://raw.githubusercontent.com/FlatFilers/Platform-Translations/kitchen-sink/locales/en/translation.json",
  });
});
```

<Info>
  Currently we only offer translations and text overrides for the guest
  experience within your Spaces. These means there will be admin only areas that
  are not currently customizable.
</Info>

## Translating Your Custom Actions and Jobs

You can translate your own custom text used in your Actions.

Simply add the new keys to your translation.json file:

```json .../locales/en/translation.json theme={null}
{
  "mySubmitAction": {
    "label": "Submit",
    "description": "Done editing the data? Hit submit to send your data",
    "tooltip": "Submit your data",
    "outcome": {
      "heading": "Successful import!",
      "message": "All data has been submitted! You can now click the link below.",
      "urlLabel": "Return to the homepage"
    }
  }
}
```

Then instead of hardcoding strings directly into your action scripts, use JSON translation keys. Refer to these keys in your script as shown below:

<CodeGroup>
  ```javascript sheet.js theme={null}
  sheets : [
    {
      name: "Sheet Name",
      actions: [
        {
          operation: 'Submit',
          mode: 'foreground',
          label: 'mySubmitAction.label',
          type: 'string',
          description: 'mySubmitAction.description',
          primary: true,
          tooltip: 'mySubmitAction.tooltip'
        },
        {...}
      ]
    }
  ]
  ```

  ```javascript listener.js theme={null}
  sheets : [
    {
      name: "Sheet Name",
      actions: [
        {
          operation: 'Submit',
          mode: 'foreground',
          label: 'mySubmitAction.label',
          type: 'string',
          description: 'mySubmitAction.description',
          primary: true,
          tooltip: 'mySubmitAction.tooltip'
        },
        {...}
      ]
    }
  ]
  ```
</CodeGroup>

If a translation key is missing from your `translation.json` file, the system will display the key itself as a fallback. Therefore, it's encouraged to always have a default language translation available to avoid showing raw keys to the users.

## Translating Your Documents

The same logic that applies to Actions also applies to your Documents. Just pass in the custom translation keys that you added to your translation files!

<CodeGroup>
  ```json .../locales/en/translation.json theme={null}
  {
    "myDocument": {
      "title": "Your first Space!",
      "body": "# Welcome\n\n### Say hello to your first Space in the new Flatfile!"
    }
  }
  ```

  ```javascript listener.js theme={null}
  // Reference the translation keys in your document configuration
  ```
</CodeGroup>

## Guidelines for Creating JSON Keys

When creating JSON keys, adhere to the following guidelines:

* **Alphanumeric Keys**: Ensure keys consist of alphanumeric characters to prevent issues with parsing.
* **Nested Structure**: Utilize at least one level of nesting (e.g., `key1.key2`) to organize translations effectively and maintain readability.
* **Avoid Whitespace and Control Characters**: Refrain from using spaces, newlines, or tabs in your JSON keys to avoid potential issues.
* **Don't End Keys with File Extensions**: Avoid using file extensions (e.g., .csv, .txt) as the terminal segment in a key. For instance, myFileAction.csv is discouraged, but myFileAction.csv.uploaded is acceptable.

Always validate your `translation.json` file to ensure it follows the correct JSON format to prevent errors during runtime.

## Example Project

Find the Localization example in the Flatfile GitHub repository.

* [Clone the Localization example in Typescript](https://github.com/FlatFilers/flatfile-docs-kitchen-sink/blob/main/typescript/localization/index.ts)
* [Clone the Localization example in Javascript](https://github.com/FlatFilers/flatfile-docs-kitchen-sink/blob/main/javascript/localization/index.js)


# Actions
Source: https://flatfile.com/docs/guides/using-actions

Trigger custom operations based on user input

An Action is a code-based operation that runs where that Action is mounted. Actions run when a user clicks the corresponding user prompt in Flatfile.

## Overview

When an Action is triggered, it creates a [Job](/core-concepts/jobs) that your application can listen for and respond to.

Given that Actions are powered by Jobs, the [Jobs Lifecycle](/core-concepts/jobs#jobs-lifecycle) pertains to Actions as well. This means that you can [update progress values/messages](/core-concepts/jobs#updating-job-progress) while an Action is processing, and when it's done you can provide an [Outcome](/core-concepts/jobs#job-outcomes), which allows you to show a success message, automatically [download a generated file](/core-concepts/jobs#file-downloads), or [forward the user](/core-concepts/jobs#internal-navigation) to a generated Document.

[Actions](/core-concepts/actions) are mounted on resources like **Workbooks**, **Sheets**, **Fields**, **Documents**, and **Files**.

Generally, [Workbook](#workbook-actions), [Sheet](#sheet-actions), [Field](#field-actions), and [Document](#document-actions) Actions are configured within a Blueprint object, while [File](#file-actions) Actions are appended to the file during the upload process. Alternatively, Actions can be mounted to any of these resources via API in a [Listener](/core-concepts/listeners).

[Sheet](#sheet-actions) Actions can be executed on the entire Sheet, for a filtered view of the Sheet, or selectively for the chosen records. See [Sheet Action Execution Modes](#sheet-action-execution-modes) for details on how actions handle different data selections.

<Note>
  In these examples, we'll show the full Job Listener lifecycle implementation, complete with `ack` to acknowledge the job, `update` to update the job's progress, and `complete` or `fail` to complete or fail the job.

  To make this simpler in practice, we provide a plugin called [Job Handler](/plugins/job-handler) that handles the job lifecycle for you. This plugin works by listening to the `job:ready` event and executing the handler callback, even catching errors to fail the job. There is also an optional `tick` function which allows you to update the Job's progress.

  For example: With the [Job Handler](/plugins/job-handler) plugin, the 35-line [File Action Listener](#listener-implementation-4) defined below would be implemented simply as:

  ```typescript theme={null}
  listener.use(jobHandler("file:logFileContents", async ({ context: { fileId, jobId } }, tick) => {
    const file = await api.files.get(fileId);
    tick(10, "Getting started.");
    console.log({ file });
    tick(90, "Logged.");
    return {
      outcome: {
        message: "Logging file contents is complete.",
      }
    }
  }));
  ```
</Note>

## Workbook Actions

<Frame>
  <img />
</Frame>

Once a user has extracted and mapped data into a Workbook, it may be more efficient to run an operation on the entire dataset rather than making atomic transformations at the record- or field-level.

For example:

* Sending a webhook that notifies your API of the data's readiness
* Populating a Sheet with data from another source
* Adding two different fields together after a user review's initial validation checks
* Moving valid data from an editable Sheet to a read-only Sheet

Workbook-mounted actions are represented as buttons in the top right of the Workbook.

### Usage

If you configure `primary: true` on an Action, its button will be highlighted in the Workbook.

If you configure `trackChanges: true`, it will disable your actions until all commits are complete (usually data hooks).

#### Blueprint Configuration

First, configure your action in your Blueprint:

```javascript theme={null}
// Add this to your workbook configuration
actions: [
  {
    operation: "downloadXML",
    mode: "background",
    label: "Download XML",
    type: "string",
    description: "Generates and downloads an XML file with all the users in the workbook",
    primary: false,
  },
  {
    operation: "submitToAPI",
    mode: "background",
    label: "Submit to API",
    type: "string",
    description: "Submits the users to our API",
    primary: true,
  },
],
settings: {
  trackChanges: true,
}
```

#### Listener Implementation

Next, create a listener to handle the `job:ready` event for your action.

```typescript theme={null}
listener.on(
  "job:ready",
  { job: "workbook:submitToAPI" },
  async (event) => {
    const { jobId, workbookId } = event.context;
    const { data: workbook } = await api.workbooks.get(workbookId);
    const { data: workbookSheets } = await api.sheets.list({ workbookId });
    
    // Collect all sheet data
    const sheets = [];
    for (const [_, element] of workbookSheets.entries()) {
      const { data: records } = await api.records.get(element.id);
      sheets.push({
        ...element,
        ...records,
      });
    }
    
    try {
      // Acknowledge the job start
      await api.jobs.ack(jobId, {
        info: "Starting job to submit action to webhook",
        progress: 10,
      });
      
      // Replace with your actual webhook URL
      const webhookReceiver = "https://your-app.com/webhook/flatfile";
      
      // Submit data to external system
      const response = await fetch(webhookReceiver, {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
        },
        body: JSON.stringify({
          workbook: {
            ...workbook,
            sheets,
          },
        }),
      });
      
      if (response.status === 200) {
        await api.jobs.complete(jobId, {
          outcome: {
            message: `Data was successfully submitted to ${webhookReceiver}.`,
          },
        });
      } else {
        throw new Error("Failed to submit data to webhook");
      }
    } catch (error) {
      console.error(error);
      await api.jobs.fail(jobId, {
        outcome: {
          message: "This job failed. Check your webhook URL and try again.",
        },
      });
    }
  }
);
```

## Sheet Actions

<Frame>
  <img />
</Frame>

Each Sheet has built-in Actions like download.

Sheet-mounted actions are represented as a dropdown in the toolbar of the Sheet.

### Usage

If you configure `primary: true` on an Action, it creates a top-level button as well as placing it in the dropdown menu.

#### Blueprint Configuration

First, configure your action on your Blueprint. Add the action configuration to your sheet definition:

```javascript theme={null}
sheets : [
  {
    name: "Users",
    actions: [
      {
        operation: "populateHealthScores",
        mode: "background",
        label: "Populate Health Scores",
        type: "string",
        description: "Populate health scores for all users",
        primary: true,
      },
      {
        operation: "validateHealthScores",
        mode: "background",
        label: "Validate Health Scores",
        type: "string",
        description: "Validate health scores for all users",
        primary: false,
      },
    ]
  }
]
```

#### Listener Implementation

Next, listen for a `job:ready` and filter on the `domain` (sheet) and the `operation` of where the action was placed. Be sure to complete the job when it's done.

```typescript theme={null}
listener.on(
  "job:ready",
  { job: "sheet:populateHealthScores" },
  async ({ context: { jobId } }) => {
    try {
      await api.jobs.ack(jobId, {
        info: "Populating health scores...",
        progress: 10,
        estimatedCompletionAt: new Date("Tue Aug 23 2023 16:19:42 GMT-0700"),
      });

      // Populate health scores for all users
      // Implementation logic here

      await api.jobs.complete(jobId, {
        info: "Health scores populated successfully.",
      });
    } catch (error) {
      console.error("Error:", error.stack);

      await api.jobs.fail(jobId, {
        info: "Failed to populate health scores.",
      });
    }
  }
);
```

### Sheet Action Execution Modes

Sheet Actions are powerful because they can operate on different subsets of data within a sheet. When a Sheet Action is triggered, it automatically receives context about what data should be processed based on the user's current view and selections.

#### How Sheet Actions Handle Data Context

When a Sheet Action job is created, it receives a `query` object in the job's subject that specifies which records to process:

```javascript theme={null}
// Job subject structure for Sheet Actions
{
  type: "collection", 
  resource: string,              // Usually sheetId or "records" 
  query: {
    filter: "valid" | "error" | "all" | "none",  // Filter type
    filterField: string,         // Field to filter on (optional)
    searchField: string,         // Field to search in (optional)  
    searchValue: string,         // Search term (optional)
    q: string,                   // FFQL (Flatfile Query Language) query (optional)
    ids: string[]               // Record IDs - INCLUDE when no filter, EXCLUDE when filter applied
  },
  params: {
    sheetId: string,            // Always present for sheet actions
    // Additional params depending on action type
  }
}
```

#### Three Execution Contexts

**1. Entire Sheet (No Filter, No Selection)**

* Processes all records in the sheet
* `query.filter` is `"all"` or undefined
* No `ids` or `exceptions` arrays

**2. Filtered View**

* Processes only records matching the current filter
* `query.filter` indicates the filter type:
  * `"valid"` - Only valid records (no errors)
  * `"error"` - Only records with validation errors
  * Custom filters if applied
* May include search parameters if user has searched
* May include [FFQL queries](/reference/ffql) via the `q` parameter for advanced filtering

**3. Selected Records**

* Processes only user-selected records
* **When no other filters applied**: `query.ids` contains specific record IDs to INCLUDE
* **When filters are applied**: `query.ids` contains record IDs to EXCLUDE from the filtered results

#### Implementation Example

```typescript theme={null}
listener.on(
  "job:ready", 
  { job: "sheet:processData" },
  async (event) => {
    const { jobId, sheetId } = event.context;
    
    // Get the job to access the query context
    const { data: job } = await api.jobs.get(jobId);
    const query = job.subject?.query || {};
    
    // Pass the query directly to the records API
    // The API handles the inclusion/exclusion logic for ids based on filters
    const { data: records } = await api.records.get(sheetId, {
      ...query  // Includes filter, ids, searchField, searchValue, etc.
    });
    
    console.log(`Processing ${records.length} records`);
    console.log(`Query context:`, query);
    
    // Process the records
    for (const record of records) {
      // Your processing logic here...
    }
    
    await api.jobs.complete(jobId, {
      outcome: {
        message: `Successfully processed ${records.length} records`,
      },
    });
  }
);
```

#### Built-in Action Examples

Built-in actions like Delete and Download demonstrate these execution modes:

* **Delete All** - Removes all records when no selection/filter
* **Delete Selected** - Removes only selected records
* **Delete Valid/Invalid** - Removes records matching the current filter
* **Download Filtered** - Exports only records matching current view

Your custom Sheet Actions automatically inherit this same context-aware behavior.

## Field Actions

<Frame>
  <img />
</Frame>

Field-mounted actions are operations that can be performed on individual columns/fields within a Sheet. They appear as options in the dropdown menu when you click on a column header.

Field actions are particularly useful for:

* Column-specific data transformations (e.g., capitalizing all values in a name field)
* Field-level validation operations
* Data formatting specific to a column type
* Column-specific cleanup operations

### Usage

The `primary` property does not affect the UI for Field actions.

<Note>
  Note: Field Actions are essentially an extension of Sheet actions - therefore their operation names are prefixed with `sheet:`, and the column key is available in the job data.
</Note>

#### Blueprint Configuration

First, configure your field action in your Blueprint by adding it to the field definition:

```javascript theme={null}
// Add this to your field configuration within a sheet
{
  key: "firstName",
  type: "string",
  label: "First Name",
  description: "Customer's first name",
  actions: [
    {
      operation: "capitalize",
      mode: "foreground",
      label: "Capitalize All Values",
      description: "Capitalizes the values of the selected column",
    }
  ]
}
```

#### Listener Implementation

Next, create a listener to handle the `job:ready` event for your field action. Field actions use the `sheet:operationName` job pattern and provide field context through the job's subject parameters.

```typescript theme={null}
listener.on(
  "job:ready",
  { job: "sheet:capitalize" },
  async (event) => {
    const { jobId, sheetId } = event.context;
    
    try {
      // Acknowledge the job start
      await api.jobs.ack(jobId, {
        info: "Starting job to capitalize all values",
        progress: 10,
      });

      // Get the field key from the job subject
      const { data: { subject } } = await api.jobs.get(jobId);
      const fieldKey = subject.params.columnKey;

      // Get all records from the sheet
      const { data: { records } } = await api.records.get(sheetId);

      await api.jobs.update(jobId, {
        info: "Capitalizing all values",
        progress: 20,
      });

      // Process records for the specific field
      const updatedRecords = [];
      records.forEach(record => {
        const value = record.values[fieldKey].value as string;
        if (value && value.length > 0) {
          const capitalizedValue = value.toUpperCase();
          if (value !== capitalizedValue) {
            record.values[fieldKey].value = capitalizedValue;
            updatedRecords.push(record);
          }
        }
      });

      await api.jobs.update(jobId, {
        info: "Updating records",
        progress: 60,
      });

      // Update only the modified records
      if (updatedRecords.length > 0) {
        await api.records.update(sheetId, updatedRecords);
      }
      
      await api.jobs.complete(jobId, {
        outcome: {
          message: `Successfully capitalized ${updatedRecords.length} values in the ${fieldKey} column.`,
        },
      });
    } catch (error) {
      console.error(error);
      await api.jobs.fail(jobId, {
        outcome: {
          message: "Failed to capitalize field values: " + error.message,
        },
      });
    }
  }
);
```

### Key Differences from Other Action Types

Field actions have several unique characteristics:

* **Job Pattern**: Use `sheet:operationName`
* **Context Access**: Field key is available via `job.subject.params.columnKey`
* **UI Location**: Appear in column header dropdown menus
* **Configuration**: Defined within individual field objects in the Blueprint

## Document Actions

<Frame>
  <img />
</Frame>

Document-mounted actions are similar to Workbook-mounted actions. They appear in the top right corner of your Document.

### Usage

If you configure `primary: true` on an Action, it will be highlighted in the Document.

#### Document Configuration

Define Document-mounted Actions using the `actions` parameter when you create a Document.

```typescript theme={null}
import api from "@flatfile/api";

export default function flatfileEventListener(listener) {
  listener.on("file:created", async ({ context: { spaceId, fileId } }) => {
    const fileName = (await api.files.get(fileId)).data.name;
    const bodyText =
      "# Welcome\n" +
      "### Say hello to your first customer Space in the new Flatfile!\n" +
      "Let's begin by first getting acquainted with what you're seeing in your Space initially.\n" +
      "---\n" +
      `Your uploaded file, ${fileName}, is located in the Files area.`;

    const doc = await api.documents.create(spaceId, {
      title: "Getting Started",
      body: bodyText,
      actions: [
        {
          label: "Submit",
          operation: "contacts:submit",
          description: "Would you like to submit the contact data?",
          tooltip: "Submit the contact data",
          mode: "foreground",
          primary: true,
          confirm: true,
        },
      ],
    });
  });
}
```

#### Listener Implementation

In your listener, listen for the job's event and perform your desired operations.

```typescript theme={null}
export default function flatfileEventListener(listener) {
  listener.on(
    "job:ready",
    { job: "document:contacts:submit" },
    async (event) => {
      const { context, payload } = event;
      const { jobId, workbookId } = context;

      try {
        await api.jobs.ack(jobId, {
          info: "Starting submit job...",
          progress: 10,
          estimatedCompletionAt: new Date("Tue Aug 23 2023 16:19:42 GMT-0700"),
        });

        // Do your work here

        await api.jobs.complete(jobId, {
          outcome: {
            message: `Submit job was completed succesfully.`,
          },
        });
      } catch (error) {
        console.log(`There was an error: ${JSON.stringify(error)}`);
        await api.jobs.fail(jobId, {
          outcome: {
            message: `This job failed.`,
          },
        });
      }
    }
  );
}
```

## File Actions

<Frame>
  <img />
</Frame>

Each file has built-in actions like Delete and Download.

File-mounted actions are represented as a dropdown menu for each file in the Files list. You can attach additional actions to a File.

### Usage

You can attach additional actions to a File by listening for file events and updating the file's actions array.

#### File Configuration

First, listen for a `file:ready` event and add one or more actions to the file.

```typescript theme={null}
listener.on("file:created", async ({ context: { fileId } }) => {
  const file = await api.files.get(fileId);
  const actions = file.data?.actions || [];
  const newActions = [
    ...actions,
    {
      operation: "logFileContents",
      label: "Log File Metadata",
      description: "This will log the file metadata.",
      primaty: true,
    }
  ];
  await api.files.update(fileId, {
    actions: newActions,
  });
});
```

#### Listener Implementation

Next, listen for `job:ready` and filter on the `domain` (file) and the `operation` of where the Action was placed. Be sure to complete the job when it's done.

```typescript theme={null}
listener.on(
  "job:ready",
  { job: "file:logFileContents" },
  async ({ context: { fileId, jobId } }) => {
    try {
      await api.jobs.ack(jobId, {
        info: "Getting started.",
        progress: 10,
        estimatedCompletionAt: new Date(Date.now() + 10000),
      });

      const file = await api.files.get(fileId);
      console.log({ file });

      await api.jobs.update(jobId, {
        info: "Logged.",
        progress: 90,
        estimatedCompletionAt: new Date(Date.now() + 10000),
      });

      
      await api.jobs.complete(jobId, {
        outcome: {
          message: "Logging file contents is complete.",
        },
      });
    } catch (error) {
      await api.jobs.fail(jobId, {
        outcome: {
          message: "Logging file contents failed: " + error.message,
        },
      });
    }

  }
);
```

## Action Parameters

### Required Parameters

| Parameter   | Type     | Description                                                                                                                |
| ----------- | -------- | -------------------------------------------------------------------------------------------------------------------------- |
| `operation` | `string` | A unique identifier for the Action that is used by the listener to determine what work to do as part of the resulting Job. |
| `label`     | `string` | The text that will be displayed to the user in the UI as a button or menu item.                                            |

### Optional Parameters

| Parameter     | Type        | Default        | Description                                                                                                            |
| ------------- | ----------- | -------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `primary`     | `boolean`   | `false`        | Whether the Action is a primary Action for the resource. Primary Actions are displayed more prominently in the UI.     |
| `confirm`     | `boolean`   | `true`         | When set to true, a modal is shown to confirm the Action.                                                              |
| `description` | `string`    | -              | The text displayed to the user when a confirmation modal is used. Phrase this as a question.                           |
| `icon`        | `string`    | lightning bolt | The icon to be displayed. Use `'none'` to omit the icon. See [Flatfile icons](/reference/icons) for available options. |
| `tooltip`     | `string`    | -              | Text displayed as a tooltip when hovering over the action button or menu item.                                         |
| `messages`    | `array[{}]` | -              | Custom messages displayed as tooltips based on action state. Supports `[{ type: 'error' }]` and `[{ type: 'info' }]`.  |
| `constraints` | `array[{}]` | -              | Conditions that disable the action. Options: `hasAllValid`, `hasSelection`, `hasData`.                                 |
| `mode`        | `string`    | `background`   | Execution mode: `foreground`, `background`, or `toolbarBlocking`.                                                      |

#### Constraint Types

* `hasAllValid`: Disables action when there are invalid records
* `hasSelection`: Disables action when no records are selected (Sheet actions only)
* `hasData`: Disables action when there are no records

#### Mode Types

* `foreground`: Prevents interacting with the entire resource until complete
* `background`: Runs in the background without blocking the UI
* `toolbarBlocking`: Disables sheet-level toolbar and column header menus while allowing manual record entry

### Usage

An Action with all of the above properties would look like this:

```javascript theme={null}
{
  operation: 'my-action',
  label: 'My Action',
  primary: true,
  confirm: true,
  description: 'Are you sure you want to run this action?',
  constraints: [{ type: 'hasAllValid' }, { type: 'hasSelection' }],
  mode: 'foreground'
  tooltip: 'Click to run action'
}
```

## Input Forms

<Frame>
  <img />
</Frame>

When initiating an action, there may be instances where additional information is required from the end user to successfully complete the intended task. For example, you might want to enable users to specify the name of the file they intend to export.

In such cases, if you configure input fields for your action, a secondary dialog will be presented to the end user, prompting them to provide the necessary information. Once the user has entered the required details, they can proceed with the action seamlessly.

### Configuration

| Parameter | Type            | Required | Description                                                  |
| --------- | --------------- | -------- | ------------------------------------------------------------ |
| `type`    | `string`        | ✓        | The type of the input form. Currently accepts: `simple`      |
| `fields`  | `array[object]` | ✓        | An array of field objects representing the input form fields |

### Fields

| Parameter      | Type            | Required | Description                                                      |
| -------------- | --------------- | -------- | ---------------------------------------------------------------- |
| `key`          | `string`        | ✓        | The key for the field                                            |
| `label`        | `string`        | ✓        | The label for the field                                          |
| `type`         | `string`        | ✓        | Field type: `string`, `textarea`, `number`, `boolean`, or `enum` |
| `defaultValue` | `string`        | -        | The default value for the field                                  |
| `description`  | `string`        | -        | A description of the field                                       |
| `config`       | `object`        | -        | Configuration options for the field (required for `enum` type)   |
| `constraints`  | `array[object]` | -        | An array of constraints for the field                            |

### Config (for enum fields)

| Parameter | Type            | Required | Description                       |
| --------- | --------------- | -------- | --------------------------------- |
| `options` | `array[object]` | ✓        | An array of options for the field |

### Options

| Parameter     | Type     | Required | Description                                         |
| ------------- | -------- | -------- | --------------------------------------------------- |
| `value`       | `string` | ✓        | The value or ID of the option                       |
| `label`       | `string` | -        | A visual label for the option                       |
| `description` | `string` | -        | A short description of the option                   |
| `meta`        | `object` | -        | An arbitrary JSON object associated with the option |

### Field Constraints

| Parameter | Type     | Required | Description                                           |
| --------- | -------- | -------- | ----------------------------------------------------- |
| `type`    | `string` | ✓        | The type of constraint. Currently accepts: `required` |

### Usage

First, configure your action to have an inputForm on your Blueprint. These will appear once the action button is clicked.

```javascript theme={null}
actions: [
  {
    operation: "submitActionFg",
    mode: "foreground",
    label: "Submit data elsewhere",
    type: "string",
    description: "Submit this data to a webhook.",
    primary: true,
    inputForm: {
      type: "simple",
      fields: [
        {
          key: "priority",
          label: "Priority level",
          description: "Set the priority level.",
          type: "enum",
          defaultValue: "80ce8718a21c",
          config: {
            options: [
              {
                value: "80ce8718a21c",
                label: "High Priority",
                description:
                  "Setting a value to High Priority means it will be prioritized over other values",
              },
            ],
          },
          constraints: [
            {
              type: "required",
            },
          ],
        },
      ],
    },
  },
];
```

Next, listen for a `job:ready` and filter on the `job` you'd like to process. Grab the data entered in the form from the job itself and leverage it as required for your use case.

```typescript theme={null}
import api from "@flatfile/api";

export default async function (listener) {
  listener.on(
    "job:ready",
    { job: "workbook:actionWithInput" },
    async (event) => {
      const { jobId } = event.context;

      try {
        await api.jobs.ack(jobId, {
          info: "Acknowledging job",
          progress: 1,
        });

        // retrieve input
        const { data: job } = await api.jobs.get(jobId);
        const input = job.input;
        console.log({ input });

        // do something with input...

        await api.jobs.complete(jobId, {
          outcome: {
            message: "Action was successful",
          },
        });
        return;
      } catch (error) {
        console.error(error);
        await api.jobs.fail(jobId, {
          outcome: {
            message: "Action failed",
          },
        });
        return;
      }
    }
  );
}
```

## Constraints

### Usage

#### Workbook & Sheet Actions

1. Adding a `hasAllValid` constraint on an Action will disable a Workbook Action when there are invalid records.
2. Adding a `hasData` on an Action will disable a Workbook Action when there are no records.

```javascript theme={null}
actions: [
  {
    operation: 'submitActionFg',
    mode: 'foreground',
    label: 'Submit data elsewhere',
    description: 'Submit this data to a webhook.',
    primary: true,
    constraints: [{ type: 'hasAllValid'},{ type: 'hasData' }]
  },
  {...}
],
```

#### Sheet Actions Only

Adding a constraint of `hasSelection` on an Action will disable a Sheet Action when no records in the Sheet are selected.

```javascript theme={null}
sheets : [
  {
    name: "Sheet Name",
    actions: [
      {
        operation: 'duplicate',
        mode: 'background',
        label: 'Duplicate selected names',
        description: 'Duplicate names for selected rows',
        constraints: [{ type: 'hasSelection' }],
        primary: true,
      },
      {...}
    ]
  }
]
```

## Messages

Add custom messages to actions, tailored according to their state:

* Error
* Info

These messages will be displayed as tooltips when users hover over an action, providing context-specific text that corresponds to the action's current state. When an error message is present on an action, the action will be disabled.

### Usage

Simply add a messages property to your action configuration. This property should be an array of objects, each specifying a message type and its content.

```javascript theme={null}
  actions: [
    {
      operation: 'duplicate',
      mode: 'background',
      label: 'Duplicate selected names',
      description: 'Duplicate names for selected rows',
      messages: [
        { type: 'error', content: 'This is an error message' },
      ],
      primary: true,
    },
    {...}
  ]
```


# Using Record Hooks
Source: https://flatfile.com/docs/guides/using-record-hooks

Process data with Record Hooks

**Hooks** are concise functions that automatically **re-format**, **correct**, **validate**, and **enrich** data during the data import process. These hooks can be executed on a complete record, or row, of data using methods on the `FlatfileRecord` class.

Record-level hooks have access to all fields in a row and should be utilized for operations that require access to multiple fields or when creating a new field.

## Getting started

The `FlatfileRecord` class provides methods to manipulate and interact with a record in the Flatfile format, including setting values, retrieving values, adding information messages, performing computations, validating fields, and converting the record to JSON format.

Use the [Record Hook plugin](/plugins/record-hook) to listen for updates to data records and respond with three types of record hooks: `compute`, `computeIfPresent`, and `validate`. These hooks are available as methods on the `FlatfileRecord` class.

For complete installation instructions, configuration options, and detailed examples, see the [Record Hook plugin documentation](/plugins/record-hook).

## Record Hook Types

Record hooks provide three main operations for data processing:

### `compute`

Transforms field values based on the original value or other fields in the record. Always runs, even when no initial value is set. Use this for generating new values or creating computed fields.

### `computeIfPresent`

Similar to `compute`, but only runs when the field has an initial value. Ideal for transformations that might fail on empty values, such as string operations or mathematical calculations.

### `validate`

Validates field values against custom conditions and marks fields as invalid with error messages when they fail validation. Use this for complex validation rules beyond built-in field constraints.

For detailed syntax, parameters, and implementation examples of these record hooks, see the [Record Hook plugin documentation](/plugins/record-hook).

## Record Messages

The `FlatfileRecord` class provides methods to attach contextual messages to fields:

* **`addInfo()`**: Adds informational messages in a tooltip to help users understand transformations or provide context
* **`addWarning()`**: Adds warning messages to alert users of potential issues
* **`addError()`**: Marks fields as invalid with error messages, blocks [Actions](/core-concepts/actions) with the `hasAllValid` constraint

These methods accept either a single field name or an array of field names, along with a message string. For detailed syntax and examples, see the [Record Hook plugin documentation](/plugins/record-hook).

## Accessing External APIs

Record hooks can integrate with external APIs to enrich, validate, or transform data in real-time. This is useful for:

* **Data Enrichment**: Fetching additional information from external services
* **Real-time Validation**: Verifying data against external systems
* **Data Transformation**: Using external services to process or format data
* **Webhook Integration**: Sending processed data to external systems

### Common Use Cases

* **GET Requests**: Retrieve data from external APIs to enrich records
* **POST Requests**: Send record data to webhooks or external systems for processing
* **Authentication**: Validate credentials or tokens against external services
* **Geocoding**: Convert addresses to coordinates using mapping services

For complete examples of API integration patterns, including error handling and async processing, see the [Record Hook plugin documentation](/plugins/record-hook).

## getLinks

The `getLinks` method is a feature of the FlatfileRecord class. When a field in your record is of the Reference Field type and links to another record, getLinks can fetch those linked fields for you.

### Usage

When processing a record, you may find references to a related record. To retrieve the fields from the related record, use the `getLinks` method. Provide the field key of the Reference Field type, the part of the record that holds the reference to the other records, like this:

```javascript theme={null}
const relatedRecords = record.getLinks("referenceFieldKey");
```

The getLinks method will then return an array containing all fields from the linked record associated with the provided 'referenceFieldKey'. If there is not a linked record associated with this field, an empty array will be returned.

### Benefits

Using `getLinks` provides access to all related information. It's particularly useful when you want to perform operations similar to VLOOKUPs in spreadsheets, or when you need to compare data across referenced fields.

For instance, you could use `getLinks` to fetch all the fields related to a particular record and enrich your data, or compare the related records for validation.

With `getLinks`, processing related datasets becomes much more manageable in Flatfile. This method provides you with an effective way to navigate, enrich, and validate your data.

## Deleting records

There are primarily two use cases for deleting records:

1. Deleting a subset of records
2. Deleting all records in a sheet

### Deleting a subset of records

To delete a subset of records first import the `@flatfile/api` package, then use the `api.records.delete()` helper method. This method takes in an array of record IDs and deletes them from the sheet.

```javascript theme={null}
await api.records.delete(sheetId, { ids: [...] });
```

### Deleting all records in a sheet

For clearing an entire sheet of its records, set up a bulk delete job. This task will comprehensively wipe out every record on the specified sheet. Check out our [jobs documentation](/core-concepts/jobs).

```javascript theme={null}
await api.jobs.create({
  type: "workbook",
  operation: "delete-records",
  trigger: "immediate",
  source: workbookId,
  config: {
    sheet: sheetId,
    filter: "all",
  },
});
```

* [Clone the Handling Data example in Typescript](https://github.com/FlatFilers/flatfile-docs-kitchen-sink/blob/main/typescript/handling-data/index.ts)
* [Clone the Handling Data example in Javascript](https://github.com/FlatFilers/flatfile-docs-kitchen-sink/blob/main/javascript/handling-data/index.js)


# Metadata
Source: https://flatfile.com/docs/guides/utilizing-metadata

Store descriptive information or data that provides additional context

Metadata allows you to store and retrieve additional data about any Flatfile resource without exposing it to end users. This key-value object provides a flexible way to attach custom information, track state, store references, or add contextual data to your resources.

## Universal Usage

Metadata can be added during resource creation or updated later using the Flatfile API. The metadata object accepts any valid JSON data and is accessible in all listeners and webhooks.

```javascript theme={null}
await api.spaces.update(spaceId, {
  metadata: {
    userId: "user123",
    companyId: "company456",
    customField: "any value"
  }
});
```

## Resource Types

**Environment** - Store deployment details, version info, or environment state\
**Space** - Track user IDs, company information, or session data\
**Workbook** - Add expiration dates, processing flags, or workflow state\
**Record** - Store computed values, external IDs, or transformation flags\
**Field** - Define formatting rules, validation context, or display preferences


# Welcome to Flatfile
Source: https://flatfile.com/docs/index



If you're ready to dive right in, start here:

<CardGroup>
  <Card title="Getting started with AutoBuild" icon="rocket" href="/getting-started/quickstart/autobuild">
    Build your first data import experience in minutes using our powerful AI tooling
  </Card>

  <Card title="Getting started with Code" icon="code" href="/coding-tutorial/overview">
    Build your first data import experience with code using our robust, event-driven SDK
  </Card>
</CardGroup>

## What is Flatfile?

Flatfile is an AI-powered platform that eliminates the weeks and months typically spent migrating customer data, cutting project timelines and labor hours by over 70%. Build seamless data onboarding experiences with universal file support, effortless mapping, intelligent formatting, and collaborative resolution tools.

<iframe title="Flatfile Demo" />

## How it works

Every import follows the same proven workflow:

<Steps>
  <Step title="Upload File">
    Users drag and drop their CSV, Excel, or other data files
  </Step>

  <Step title="Prepare Data">
    AI automatically detects and cleans common data issues
  </Step>

  <Step title="Map Fields">
    Smart column mapping matches user data to your Blueprint
  </Step>

  <Step title="Transform & Validate">
    Apply business rules and validate data in real-time
  </Step>

  <Step title="Export Clean Data">
    Get production-ready data delivered to your system
  </Step>
</Steps>


# Autocast Plugin
Source: https://flatfile.com/docs/plugins/autocast

Automatically convert data in a Flatfile Sheet to match the data types defined in the corresponding Blueprint

The Autocast plugin is an opinionated transformer for Flatfile that automatically converts data in a Sheet to match the data types defined in the corresponding Blueprint (Schema). It operates on the `commit:created` event, meaning it processes records after they have been committed to the sheet.

Its primary purpose is to clean and standardize data by ensuring that values intended to be numbers, booleans, or dates are correctly typed, even if they are imported as strings. For example, it can convert the string "1,000" to the number `1000`, the string "yes" to the boolean `true`, and the string "08/16/2023" to a standardized UTC date string.

This plugin is useful in any scenario where source data may have inconsistent or incorrect data types, saving developers from writing manual data-casting logic.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-autocast
```

## Configuration & Parameters

### Main Plugin Function

The `autocast` function accepts the following parameters:

<ParamField type="string">
  The slug of the sheet that the plugin should monitor and apply autocasting to.
</ParamField>

<ParamField type="string[]">
  An optional array of field keys. If provided, the plugin will only attempt to cast values in the specified fields.

  **Default Behavior:** If not provided, the plugin will automatically attempt to cast all fields in the sheet that are not of type 'string' in the Blueprint (i.e., it will target number, boolean, and date fields by default).
</ParamField>

<ParamField type="object">
  Configuration options for performance and debugging:

  <Expandable title="options properties">
    <ParamField type="number">
      Specifies the number of records to process in each batch. This is passed down to the underlying bulk record hook.
    </ParamField>

    <ParamField type="number">
      Specifies how many chunks to process in parallel. This is passed down to the underlying bulk record hook.
    </ParamField>

    <ParamField type="boolean">
      An optional flag to enable debug logging.
    </ParamField>
  </Expandable>
</ParamField>

## Usage Examples

### Basic Usage

Apply autocasting to all supported fields on a sheet:

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { autocast } from '@flatfile/plugin-autocast';

  const listener = new FlatfileListener();

  listener.use(autocast('contacts'));

  export default listener;
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { autocast } from '@flatfile/plugin-autocast';

  const listener = new FlatfileListener();

  listener.use(autocast('contacts'));

  export default listener;
  ```
</CodeGroup>

### Targeted Field Casting

Apply autocasting to only specific fields:

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { autocast } from '@flatfile/plugin-autocast';

  const listener = new FlatfileListener();

  listener.use(autocast('contacts', ['annualRevenue', 'subscribed']));

  export default listener;
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { autocast } from '@flatfile/plugin-autocast';

  const listener = new FlatfileListener();

  listener.use(autocast('contacts', ['annualRevenue', 'subscribed']));

  export default listener;
  ```
</CodeGroup>

### Advanced Configuration

Configure field filters and adjust performance settings:

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { autocast } from '@flatfile/plugin-autocast';

  const listener = new FlatfileListener();

  listener.use(
    autocast('contacts', ['annualRevenue', 'subscribed'], {
      chunkSize: 5000,
      parallel: 2,
      debug: true,
    })
  );

  export default listener;
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { autocast } from '@flatfile/plugin-autocast';

  const listener = new FlatfileListener();

  listener.use(
    autocast('contacts', ['annualRevenue', 'subscribed'], {
      chunkSize: 5000,
      parallel: 2,
      debug: true,
    })
  );

  export default listener;
  ```
</CodeGroup>

### Using Utility Functions

The plugin also exports individual casting utility functions:

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { castNumber, castBoolean, castDate } from '@flatfile/plugin-autocast';

  // Cast numbers (handles commas)
  const num1 = castNumber('1,234.56'); // Returns 1234.56
  const num2 = castNumber(99); // Returns 99

  // Cast booleans
  const bool1 = castBoolean('yes'); // Returns true
  const bool2 = castBoolean(0); // Returns false
  const bool3 = castBoolean('f'); // Returns false

  // Cast dates
  const date1 = castDate('08/16/2023'); // Returns 'Wed, 16 Aug 2023 00:00:00 GMT'
  const date2 = castDate(1692144000000); // Returns 'Wed, 16 Aug 2023 00:00:00 GMT'
  ```

  ```typescript TypeScript theme={null}
  import { castNumber, castBoolean, castDate, TRecordValue } from '@flatfile/plugin-autocast';

  // Cast numbers (handles commas)
  const num1: number = castNumber('1,234.56'); // Returns 1234.56
  const num2: number = castNumber(99); // Returns 99

  // Cast booleans
  const bool1: boolean = castBoolean('yes'); // Returns true
  const bool2: boolean = castBoolean(0); // Returns false
  const bool3: boolean = castBoolean('f'); // Returns false

  // Cast dates
  const date1: string = castDate('08/16/2023'); // Returns 'Wed, 16 Aug 2023 00:00:00 GMT'
  const date2: string = castDate(1692144000000); // Returns 'Wed, 16 Aug 2023 00:00:00 GMT'
  ```
</CodeGroup>

## Troubleshooting

### Error Handling with Utility Functions

The individual casting utility functions throw errors when values cannot be converted:

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { castNumber, castBoolean, castDate } from '@flatfile/plugin-autocast';

  try {
    const invalidNum = castNumber('not a number');
  } catch (e) {
    console.error(e.message); // Prints: Invalid number
  }

  try {
    const invalidBool = castBoolean('maybe');
  } catch (e) {
    console.error(e.message); // Prints: Invalid boolean
  }

  try {
    const invalidDate = castDate('not a date');
  } catch (e) {
    console.error(e.message); // Prints: Invalid date
  }
  ```

  ```typescript TypeScript theme={null}
  import { castNumber, castBoolean, castDate } from '@flatfile/plugin-autocast';

  try {
    const invalidNum = castNumber('not a number');
  } catch (e: any) {
    console.error(e.message); // Prints: Invalid number
  }

  try {
    const invalidBool = castBoolean('maybe');
  } catch (e: any) {
    console.error(e.message); // Prints: Invalid boolean
  }

  try {
    const invalidDate = castDate('not a date');
  } catch (e: any) {
    console.error(e.message); // Prints: Invalid date
  }
  ```
</CodeGroup>

## Notes

### Event Trigger

The plugin is designed to run on the `listener.on('commit:created')` event.

### Plugin Order

This plugin runs on the same event as `recordHook` and `bulkRecordHook`. The order in which you `.use()` the plugins in your listener matters, as they will execute sequentially.

### Error Handling Pattern

The main `autocast` plugin does not throw errors. Instead, if a value cannot be cast, it attaches an error message directly to the record's cell using `record.addError()`. This makes the errors visible to the user in the Flatfile UI. The individual `cast*` utility functions, however, do throw an `Error` on failure.

### Supported Types

The plugin automatically targets fields of type `number`, `boolean`, and `date` as defined in the Sheet's Blueprint. It does not attempt to cast `string` fields by default.

### Boolean Casting

* **Truthy values:** `'1'`, `'yes'`, `'true'`, `'on'`, `'t'`, `'y'`, and `1`
* **Falsy values:** `'-1'`, `'0'`, `'no'`, `'false'`, `'off'`, `'f'`, `'n'`, `0`, and `-1`

### Date Casting

All parsed dates are converted to a standardized UTC string format. ISO 8601 formats like `YYYY-MM-DD` are treated as UTC, while other formats like `MM/DD/YYYY` are assumed to be local time and are converted to UTC.


# Automap Plugin
Source: https://flatfile.com/docs/plugins/automap

Automatically map columns for headless data import workflows in Flatfile with configurable confidence levels

The Automap plugin is designed for headless data import workflows in Flatfile. Its primary purpose is to automate the column mapping process. The plugin listens for successfully extracted files, and when a matching file is found, it automatically creates and executes a mapping job to a specified destination Sheet.

This is ideal for scenarios where files with consistent schemas are uploaded programmatically, bypassing the need for a user to manually map columns in the UI. The plugin determines whether to proceed with the mapping based on a configurable confidence level, ensuring that only high-quality matches are automated. If the mapping confidence is too low, it can trigger a failure callback for custom notifications or alternative handling.

## Installation

<CodeGroup>
  ```bash npm theme={null}
  npm install @flatfile/plugin-automap
  ```

  ```bash yarn theme={null}
  yarn add @flatfile/plugin-automap
  ```
</CodeGroup>

## Configuration & Parameters

The `automap` function accepts an `AutomapOptions` configuration object with the following parameters:

### Required Parameters

<ParamField type="'confident' | 'exact'">
  Controls the minimum confidence level required for the plugin to automatically execute the mapping job.

  * `'confident'`: All mapped fields must have a confidence level of 'strong' (> 90%) or 'absolute' (100%)
  * `'exact'`: All mapped fields must have a confidence level of 'absolute' (100%)
</ParamField>

### Optional Parameters

<ParamField type="boolean">
  Toggles verbose logging for development and troubleshooting. When true, the plugin will output detailed information about its progress, decisions, and any errors it encounters to the console.
</ParamField>

<ParamField type="string | function">
  Specifies the destination sheet for the imported data.

  * If a string is provided, it must be the exact slug of the target sheet
  * If a function is provided, it receives the uploaded file's name and the event payload, and must return the target sheet slug (or a Promise that resolves to it)
  * **Default behavior**: If not provided, the plugin will not be able to map a single-sheet file automatically unless more advanced logic is implemented by the user
</ParamField>

<ParamField type="RegExp">
  A regular expression used to filter which files the plugin should process.

  * **Default behavior**: If not provided, the plugin will attempt to automap every file that is uploaded
  * The plugin will only act on files whose names pass a `test()` against this regex
</ParamField>

<ParamField type="function">
  A callback function that is executed if the automapping process is aborted due to low mapping confidence.

  * **Default behavior**: Nothing happens on failure, though a warning may be logged if `debug` is true
  * This can be used to trigger notifications (e.g., email, SMS, webhook) to alert a user that manual intervention is required
</ParamField>

<ParamField type="string">
  Specifies the destination Workbook by its ID or name.

  * **Default behavior**: If not provided, the plugin searches for a suitable workbook in the space. It filters out workbooks associated with raw files (those with a 'file' label). If only one workbook remains, it is chosen. If multiple remain, it will select the one with the 'primary' label.
</ParamField>

<ParamField type="boolean">
  Prevents the plugin from updating the name of the processed file in the Flatfile UI.

  * By default, the plugin prepends "⚡️" to the file name on processing and appends the destination sheet name on success to provide visual feedback
  * Setting this to `true` disables this behavior
</ParamField>

## Usage Examples

### Basic Usage

This example shows the simplest way to use the automap plugin, targeting a specific sheet for all uploaded CSV files.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { automap } from '@flatfile/plugin-automap';

  const listener = FlatfileListener.create((listener) => {
    listener.use(
      automap({
        accuracy: 'confident',
        defaultTargetSheet: 'Contacts',
        matchFilename: /\.csv$/,
      })
    );
  });
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { automap } from '@flatfile/plugin-automap';

  const listener = FlatfileListener.create((listener) => {
    listener.use(
      automap({
        accuracy: 'confident',
        defaultTargetSheet: 'Contacts',
        matchFilename: /\.csv$/,
      })
    );
  });
  ```
</CodeGroup>

### Configuration with Failure Handling

This example demonstrates a more complete configuration, including a failure callback and targeting a specific workbook.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { automap } from '@flatfile/plugin-automap';

  const listener = FlatfileListener.create((listener) => {
    listener.use(
      automap({
        accuracy: 'confident',
        defaultTargetSheet: 'Contacts',
        targetWorkbook: 'MyPrimaryWorkbook',
        matchFilename: /^(contacts|people|users)\.csv$/i,
        debug: true,
        onFailure: (event) => {
          console.error(
            `Automap failed for file in space ${event.context.spaceId}. Please map manually.`
          );
          // Add custom logic here, like sending an email or Slack message.
        },
      })
    );
  });
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { automap } from '@flatfile/plugin-automap';
  import type { FlatfileEvent } from '@flatfile/listener';

  const listener = FlatfileListener.create((listener) => {
    listener.use(
      automap({
        accuracy: 'confident',
        defaultTargetSheet: 'Contacts',
        targetWorkbook: 'MyPrimaryWorkbook',
        matchFilename: /^(contacts|people|users)\.csv$/i,
        debug: true,
        onFailure: (event: FlatfileEvent) => {
          console.error(
            `Automap failed for file in space ${event.context.spaceId}. Please map manually.`
          );
          // Add custom logic here, like sending an email or Slack message.
        },
      })
    );
  });
  ```
</CodeGroup>

### Dynamic Sheet Targeting

This example uses a function for `defaultTargetSheet` to dynamically route data to different sheets based on the filename.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { automap } from '@flatfile/plugin-automap';

  const listener = FlatfileListener.create((listener) => {
    listener.use(
      automap({
        accuracy: 'exact',
        defaultTargetSheet: (fileName) => {
          if (fileName.includes('invoice')) {
            return 'Invoices';
          } else if (fileName.includes('contact')) {
            return 'Contacts';
          }
          // Return a default or handle cases where no match is found
          return 'DefaultSheet';
        },
        onFailure: (event) => {
          console.log('Automap failed, manual mapping required.');
        },
      })
    );
  });
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { automap } from '@flatfile/plugin-automap';
  import type { FlatfileEvent } from '@flatfile/listener';

  const listener = FlatfileListener.create((listener) => {
    listener.use(
      automap({
        accuracy: 'exact',
        defaultTargetSheet: (fileName?: string): string => {
          if (fileName?.includes('invoice')) {
            return 'Invoices';
          } else if (fileName?.includes('contact')) {
            return 'Contacts';
          }
          // Return a default or handle cases where no match is found
          return 'DefaultSheet';
        },
        onFailure: (event: FlatfileEvent) => {
          console.log('Automap failed, manual mapping required.');
        },
      })
    );
  });
  ```
</CodeGroup>

## Troubleshooting

The most effective way to troubleshoot the plugin is to set the `debug: true` option in the configuration. This will provide a step-by-step log of the plugin's execution, including:

* Which files are matched
* What workbooks and sheets are targeted
* The contents of the mapping plan
* The reason for any failures

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { automap } from '@flatfile/plugin-automap';

  const listener = FlatfileListener.create((listener) => {
    listener.use(
      automap({
        accuracy: 'exact',
        defaultTargetSheet: 'Contacts',
        debug: true, // Enable verbose logging
        onFailure: (event) => {
          const { spaceId, fileId } = event.context;
          console.error(
            `Could not automap file ${fileId} with 'exact' accuracy. ` +
            `Please visit space ${spaceId} to map it manually.`
          );
        },
      })
    );
  });
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { automap } from '@flatfile/plugin-automap';
  import type { FlatfileEvent } from '@flatfile/listener';

  const listener = FlatfileListener.create((listener) => {
    listener.use(
      automap({
        accuracy: 'exact',
        defaultTargetSheet: 'Contacts',
        debug: true, // Enable verbose logging
        onFailure: (event: FlatfileEvent) => {
          const { spaceId, fileId } = event.context;
          console.error(
            `Could not automap file ${fileId} with 'exact' accuracy. ` +
            `Please visit space ${spaceId} to map it manually.`
          );
        },
      })
    );
  });
  ```
</CodeGroup>

## Notes

### Default Behavior

* **File processing**: If no `matchFilename` is provided, the plugin will attempt to automap every uploaded file
* **Target sheet**: It is highly recommended to set `defaultTargetSheet` for basic workflows, as the plugin cannot map single-sheet files automatically without it
* **Workbook selection**: When `targetWorkbook` is not specified, the plugin filters out file-associated workbooks and selects the remaining one, or the one with the 'primary' label if multiple exist
* **File naming**: By default, the plugin updates file names with status indicators ("⚡️" during processing, destination sheet name on success)

### Special Considerations

* This plugin is intended for use in a server-side listener, not in the browser
* The plugin relies on two key events: `job:completed:file:extract` to start the process, and `job:updated:workbook:map` to check the mapping plan
* The logic for selecting a `targetWorkbook` works best when there's a clear primary workbook in the space

### Limitations

* The `accuracy` check is all-or-nothing. If even one column mapping does not meet the required confidence level, the entire automatic mapping job is aborted
* The plugin's default behavior works best with single-sheet source files. For multi-sheet source files, you must provide more complex logic
* For internal errors (e.g., API call failures, inability to find a file or workbook), the plugin uses `try/catch` blocks and logs errors to the console, which are more verbose when `debug` is set to `true`

### Error Handling Patterns

The primary pattern for user-defined error handling is the `onFailure` callback, which is triggered when mapping confidence is too low. This allows you to implement custom notification systems or alternative workflows when automatic mapping cannot proceed.


# Boolean Validator
Source: https://flatfile.com/docs/plugins/boolean

Comprehensive boolean validation plugin for Flatfile that handles various representations of boolean values with multi-language support and flexible configuration options.

The Boolean Validator plugin for Flatfile provides comprehensive boolean validation for specified fields. It is designed to handle various representations of boolean values, not just `true` and `false`. Key features include two main validation modes: 'strict' (only accepts true/false boolean types) and 'truthy' (accepts values like 'yes', 'no', 'y', 'n', etc.). The plugin offers multi-language support for these truthy values (English, Spanish, French, German) and allows for custom mappings. It is highly configurable, with options to control case sensitivity, how null/undefined values are handled, and whether to automatically convert non-boolean values.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-validate-boolean
```

## Configuration & Parameters

The plugin is configured with a single object, `BooleanValidatorConfig`, containing the following options:

### Required Parameters

**`fields`** `string[]`

* An array of field keys (column names) to which the boolean validation should be applied.

**`validationType`** `'strict' | 'truthy'`

* The type of validation to perform:
  * `'strict'`: Only allows `true` and `false` boolean values
  * `'truthy'`: Allows string representations like 'yes', 'no', etc.

### Optional Parameters

**`sheetSlug`** `string`

* The slug of a specific sheet to apply the validation to
* Default: `'**'` (all sheets)

**`language`** `'en' | 'es' | 'fr' | 'de'`

* Specifies the language for predefined 'truthy' mappings
* Default: `'en'`

**`customMapping`** `Record<string, boolean>`

* Custom string-to-boolean mappings that override language-specific mappings
* Example: `{ 'ja': true, 'nein': false }`

**`caseSensitive`** `boolean`

* Controls case sensitivity for string comparisons during 'truthy' validation
* Default: `false`

**`handleNull`** `'error' | 'false' | 'true' | 'skip'`

* Defines how to handle `null` or `undefined` values:
  * `'error'`: Adds an error to the record
  * `'false'`: Converts the value to `false`
  * `'true'`: Converts the value to `true`
  * `'skip'`: Ignores the value without adding an error
* Default: `'skip'`

**`convertNonBoolean`** `boolean`

* Attempts to convert non-boolean values using JavaScript's `Boolean()` casting
* Default: `false`

**`defaultValue`** `boolean | 'skip'`

* Default value for invalid inputs instead of adding an error
* Default: `undefined` (raises an error)

**`customErrorMessages`** `object`

* Custom error messages for validation failures
* Properties: `invalidBoolean`, `invalidTruthy`, `nullValue`

## Usage Examples

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateBoolean } from '@flatfile/plugin-validate-boolean';

  export default function(listener) {
    // Basic strict validation
    listener.use(
      validateBoolean({
        fields: ['isActive'],
        validationType: 'strict',
      })
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateBoolean } from '@flatfile/plugin-validate-boolean';

  export default function(listener: FlatfileListener) {
    // Basic strict validation
    listener.use(
      validateBoolean({
        fields: ['isActive'],
        validationType: 'strict',
      })
    );
  }
  ```
</CodeGroup>

### Advanced Configuration

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateBoolean } from '@flatfile/plugin-validate-boolean';

  export default function(listener) {
    listener.use(
      validateBoolean({
        sheetSlug: 'contacts',
        fields: ['hasSubscription', 'isPremium'],
        validationType: 'truthy',
        language: 'es', // Use Spanish mappings: 'sí', 'no'
        handleNull: 'false', // Treat null/undefined as false
        defaultValue: false, // Set invalid values to false instead of erroring
        customErrorMessages: {
          nullValue: 'El campo no puede estar vacío.',
        },
      })
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateBoolean } from '@flatfile/plugin-validate-boolean';

  export default function(listener: FlatfileListener) {
    listener.use(
      validateBoolean({
        sheetSlug: 'contacts',
        fields: ['hasSubscription', 'isPremium'],
        validationType: 'truthy',
        language: 'es', // Use Spanish mappings: 'sí', 'no'
        handleNull: 'false', // Treat null/undefined as false
        defaultValue: false, // Set invalid values to false instead of erroring
        customErrorMessages: {
          nullValue: 'El campo no puede estar vacío.',
        },
      })
    );
  }
  ```
</CodeGroup>

### Using Helper Functions

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { validateBooleanField } from '@flatfile/plugin-validate-boolean';

  const myValue = 'Y';
  const result = validateBooleanField(myValue, {
    fields: ['customField'],
    validationType: 'truthy',
    language: 'en', // 'y' is a valid mapping in English
  });

  if (result.error) {
    console.error(`Validation failed: ${result.error}`);
  } else {
    console.log(`Validated value: ${result.value}`); // Outputs: Validated value: true
  }
  ```

  ```typescript TypeScript theme={null}
  import { validateBooleanField } from '@flatfile/plugin-validate-boolean';

  const myValue = 'Y';
  const result = validateBooleanField(myValue, {
    fields: ['customField'],
    validationType: 'truthy',
    language: 'en', // 'y' is a valid mapping in English
  });

  if (result.error) {
    console.error(`Validation failed: ${result.error}`);
  } else {
    console.log(`Validated value: ${result.value}`); // Outputs: Validated value: true
  }
  ```
</CodeGroup>

## API Reference

### validateBoolean

The main entry point for the plugin that configures and returns a Flatfile listener.

**Signature:**

```typescript theme={null}
validateBoolean(config: BooleanValidatorConfig): (listener: FlatfileListener) => void
```

**Parameters:**

* `config` - Configuration object for the validator

**Returns:**
A function that can be passed to `listener.use()` to register the plugin.

### validateBooleanField

A utility function that runs the complete validation logic for a single value.

**Signature:**

```typescript theme={null}
validateBooleanField(value: any, config: BooleanValidatorConfig): { value: boolean | null; error: string | null }
```

**Parameters:**

* `value` - The value to validate
* `config` - The configuration object

**Returns:**
Object with `value` (validated boolean or null) and `error` (error message or null)

### validateStrictBoolean

Validates that a value is strictly a boolean `true` or `false`.

**Signature:**

```typescript theme={null}
validateStrictBoolean(value: any, config: BooleanValidatorConfig): { value: boolean | null; error: string | null }
```

### validateTruthyBoolean

Validates that a value corresponds to a "truthy" or "falsy" representation.

**Signature:**

```typescript theme={null}
validateTruthyBoolean(value: any, config: BooleanValidatorConfig): { value: boolean | null; error: string | null }
```

### handleNullValue

Processes a `null` or `undefined` value according to the `handleNull` configuration.

**Signature:**

```typescript theme={null}
handleNullValue(value: any, config: BooleanValidatorConfig): { value: boolean | null; error: string | null }
```

### handleInvalidValue

Processes a value that has been identified as invalid.

**Signature:**

```typescript theme={null}
handleInvalidValue(value: any, config: BooleanValidatorConfig): { value: boolean | null; error: string | null }
```

## Troubleshooting

### Validation Not Applied

* Ensure the `fields` array contains the correct field keys
* Verify the `sheetSlug` (if used) matches the target sheet

### Case Sensitivity Issues

For 'truthy' validation, if values like 'YES' aren't being validated correctly, check the `caseSensitive` option. It defaults to `false`, but if set to `true`, the case must match exactly.

### Unexpected Results

Remember the order of operations:

1. Null handling is checked first
2. Specific validation type ('strict' or 'truthy') is applied
3. `defaultValue` is used as a final fallback for invalid values

## Notes

### Default Behavior

If only the required `fields` and `validationType` options are provided, the plugin will apply validation to the specified fields on all sheets. For 'truthy' validation, it uses case-insensitive English mappings ('yes'/'no'). Null or undefined values are skipped by default.

### Special Considerations

* The plugin supports built-in truthy/falsy mappings for English ('en'), Spanish ('es'), French ('fr'), and German ('de')
* Custom mappings (`customMapping`) take precedence over language-based default mappings
* The `sheetSlug` option allows applying different validation rules to different sheets within the same workbook

### Error Handling Patterns

* The main plugin does not throw exceptions; it adds errors directly to Flatfile records
* When a `defaultValue` is provided, the plugin corrects invalid values and adds an informational message for auditing
* Helper functions return a consistent `{ value, error }` object pattern for easy error checking


# Constraints Plugin
Source: https://flatfile.com/docs/plugins/constraints

Extend Flatfile validation capabilities with custom validation logic for complex field and sheet-level constraints

The Constraints plugin extends Flatfile's validation capabilities by allowing developers to define custom validation logic, called "external constraints," within a listener. These custom rules can then be applied to specific fields or to the entire sheet through the blueprint configuration.

The main purpose is to handle complex validation scenarios that are not covered by Flatfile's standard built-in constraints. Use cases include:

* Field-level validation based on complex logic (e.g., checking a value's format against a specific regular expression not available by default)
* Cross-field validation where the validity of one field depends on the value of another (e.g., ensuring 'endDate' is after 'startDate')
* Validating data against an external system or API (e.g., checking if a product SKU exists in an external database)
* Applying a single validation rule to multiple fields simultaneously

The plugin works by matching a `validator` key in the blueprint with a corresponding handler registered in the listener.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-constraints
```

## Configuration & Parameters

Configuration for this plugin is not set on the plugin itself, but within the Sheet's blueprint configuration. The plugin reads this blueprint to apply the correct logic.

### Field-Level Constraints

For field-level constraints (used with `externalConstraint`), add a constraint object to a field's `constraints` array:

| Parameter   | Type   | Required | Description                                                                              |
| ----------- | ------ | -------- | ---------------------------------------------------------------------------------------- |
| `type`      | string | Yes      | Must be set to 'external' to indicate it's a custom validation rule                      |
| `validator` | string | Yes      | A unique name for your validator used to link the blueprint rule to the validation logic |
| `config`    | object | No       | An arbitrary object containing any parameters or settings your validation logic needs    |

### Sheet-Level Constraints

For sheet-level constraints (used with `externalSheetConstraint`), add a constraint object to the sheet's top-level `constraints` array:

| Parameter   | Type      | Required | Description                                                 |
| ----------- | --------- | -------- | ----------------------------------------------------------- |
| `type`      | string    | Yes      | Must be set to 'external'                                   |
| `validator` | string    | Yes      | A unique name for your sheet-level validator                |
| `fields`    | string\[] | Yes      | An array of field keys that this constraint applies to      |
| `config`    | object    | No       | An arbitrary object with settings for your validation logic |

### Default Behavior

If no `external` type constraints are defined in the blueprint, the plugin will have no effect. The validation logic only runs when a matching `validator` is found in the blueprint for the current sheet.

## Usage Examples

### Basic Field-Level Constraint

<CodeGroup>
  ```javascript JavaScript theme={null}
  // In your listener file (e.g., index.js)
  import { listener } from '@flatfile/listener'
  import { externalConstraint } from '@flatfile/plugin-constraints'

  listener.use(
    externalConstraint('minLength', (value, key, { config, record }) => {
      if (typeof value === 'string' && value.length < config.len) {
        record.addError(key, `Must be at least ${config.len} characters.`)
      }
    })
  )

  // In your blueprint file (e.g., workbook.js)
  const blueprint = {
    sheets: [
      {
        name: 'Promotions',
        slug: 'promos',
        fields: [
          {
            key: 'promo_code',
            type: 'string',
            label: 'Promo Code',
            constraints: [
              { type: 'external', validator: 'minLength', config: { len: 8 } },
            ],
          },
        ],
      },
    ],
  }
  ```

  ```typescript TypeScript theme={null}
  // In your listener file (e.g., index.ts)
  import { listener } from '@flatfile/listener'
  import { externalConstraint } from '@flatfile/plugin-constraints'

  listener.use(
    externalConstraint('minLength', (value, key, { config, record }) => {
      if (typeof value === 'string' && value.length < config.len) {
        record.addError(key, `Must be at least ${config.len} characters.`)
      }
    })
  )

  // In your blueprint file (e.g., workbook.ts)
  const blueprint = {
    sheets: [
      {
        name: 'Promotions',
        slug: 'promos',
        fields: [
          {
            key: 'promo_code',
            type: 'string',
            label: 'Promo Code',
            constraints: [
              { type: 'external', validator: 'minLength', config: { len: 8 } },
            ],
          },
        ],
      },
    ],
  }
  ```
</CodeGroup>

### Configurable Constraint

<CodeGroup>
  ```javascript JavaScript theme={null}
  // In your listener file (e.g., index.js)
  import { listener } from '@flatfile/listener'
  import { externalConstraint } from '@flatfile/plugin-constraints'

  // This 'length' validator can be used for min or max length checks
  listener.use(
    externalConstraint('length', (value, key, { config, record }) => {
      if (typeof value !== 'string') return

      if (config.max && value.length > config.max) {
        record.addError(key, `Text must be under ${config.max} characters.`)
      }
      if (config.min && value.length < config.min) {
        record.addError(key, `Text must be over ${config.min} characters.`)
      }
    })
  )

  // In your blueprint file (e.g., workbook.js)
  const blueprint = {
    sheets: [
      {
        name: 'Content',
        slug: 'content',
        fields: [
          {
            key: 'title',
            type: 'string',
            label: 'Title',
            constraints: [
              { type: 'external', validator: 'length', config: { max: 50 } },
            ],
          },
          {
            key: 'description',
            type: 'string',
            label: 'Description',
            constraints: [
              { type: 'external', validator: 'length', config: { min: 10 } },
            ],
          },
        ],
      },
    ],
  }
  ```

  ```typescript TypeScript theme={null}
  // In your listener file (e.g., index.ts)
  import { listener } from '@flatfile/listener'
  import { externalConstraint } from '@flatfile/plugin-constraints'

  // This 'length' validator can be used for min or max length checks
  listener.use(
    externalConstraint('length', (value, key, { config, record }) => {
      if (typeof value !== 'string') return

      if (config.max && value.length > config.max) {
        record.addError(key, `Text must be under ${config.max} characters.`)
      }
      if (config.min && value.length < config.min) {
        record.addError(key, `Text must be over ${config.min} characters.`)
      }
    })
  )

  // In your blueprint file (e.g., workbook.ts)
  const blueprint = {
    sheets: [
      {
        name: 'Content',
        slug: 'content',
        fields: [
          {
            key: 'title',
            type: 'string',
            label: 'Title',
            constraints: [
              { type: 'external', validator: 'length', config: { max: 50 } },
            ],
          },
          {
            key: 'description',
            type: 'string',
            label: 'Description',
            constraints: [
              { type: 'external', validator: 'length', config: { min: 10 } },
            ],
          },
        ],
      },
    ],
  }
  ```
</CodeGroup>

### Sheet-Level Constraint

<CodeGroup>
  ```javascript JavaScript theme={null}
  // In your listener file (e.g., index.js)
  import { listener } from '@flatfile/listener'
  import { externalSheetConstraint } from '@flatfile/plugin-constraints'

  listener.use(
    externalSheetConstraint('contact-required', (values, keys, { record }) => {
      if (!values.email && !values.phone) {
        const message = 'Either Email or Phone must be provided.'
        // Add the error to both fields
        keys.forEach((key) => record.addError(key, message))
      }
    })
  )

  // In your blueprint file (e.g., workbook.js)
  const blueprint = {
    sheets: [
      {
        name: 'Contacts',
        slug: 'contacts',
        fields: [
          { key: 'email', type: 'string', label: 'Email' },
          { key: 'phone', type: 'string', label: 'Phone' },
        ],
        constraints: [
          {
            type: 'external',
            validator: 'contact-required',
            fields: ['email', 'phone'],
          },
        ],
      },
    ],
  }
  ```

  ```typescript TypeScript theme={null}
  // In your listener file (e.g., index.ts)
  import { listener } from '@flatfile/listener'
  import { externalSheetConstraint } from '@flatfile/plugin-constraints'

  listener.use(
    externalSheetConstraint('contact-required', (values, keys, { record }) => {
      if (!values.email && !values.phone) {
        const message = 'Either Email or Phone must be provided.'
        // Add the error to both fields
        keys.forEach((key) => record.addError(key, message))
      }
    })
  )

  // In your blueprint file (e.g., workbook.ts)
  const blueprint = {
    sheets: [
      {
        name: 'Contacts',
        slug: 'contacts',
        fields: [
          { key: 'email', type: 'string', label: 'Email' },
          { key: 'phone', type: 'string', label: 'Phone' },
        ],
        constraints: [
          {
            type: 'external',
            validator: 'contact-required',
            fields: ['email', 'phone'],
          },
        ],
      },
    ],
  }
  ```
</CodeGroup>

## API Reference

### externalConstraint

Registers a listener for a field-level custom validation rule. The provided callback function will be executed for every record on each field that has a matching `external` constraint in the blueprint.

**Signature:**

```typescript theme={null}
externalConstraint(
  validator: string, 
  cb: (
    value: any, 
    key: string, 
    support: { 
      config: any, 
      record: FlatfileRecord, 
      property: Flatfile.Property, 
      event: FlatfileEvent 
    }
  ) => any | Promise<any>
)
```

**Parameters:**

* `validator` (string): The name of the validator. This must match the `validator` property in the field's constraint configuration in the blueprint.
* `cb` (function): A callback function that contains the validation logic. It receives:
  * `value` (any): The value of the cell being validated
  * `key` (string): The key of the field being validated
  * `support` (object): An object containing helpful context:
    * `config` (any): The `config` object from the blueprint constraint
    * `record` (FlatfileRecord): The full record object, which can be used to get other values or add errors
    * `property` (Flatfile.Property): The full property (field) definition from the sheet schema
    * `event` (FlatfileEvent): The raw event that triggered the validation

**Error Handling Examples:**

<CodeGroup>
  ```javascript JavaScript theme={null}
  // Using record.addError() (Recommended)
  listener.use(
    externalConstraint('must-be-positive', (value, key, { record }) => {
      if (typeof value === 'number' && value <= 0) {
        record.addError(key, 'Value must be a positive number.')
      }
    })
  )

  // Throwing an Error
  listener.use(
    externalConstraint('must-be-positive', (value) => {
      if (typeof value === 'number' && value <= 0) {
        throw 'Value must be a positive number.'
      }
    })
  )
  ```

  ```typescript TypeScript theme={null}
  // Using record.addError() (Recommended)
  listener.use(
    externalConstraint('must-be-positive', (value, key, { record }) => {
      if (typeof value === 'number' && value <= 0) {
        record.addError(key, 'Value must be a positive number.')
      }
    })
  )

  // Throwing an Error
  listener.use(
    externalConstraint('must-be-positive', (value) => {
      if (typeof value === 'number' && value <= 0) {
        throw 'Value must be a positive number.'
      }
    })
  )
  ```
</CodeGroup>

### externalSheetConstraint

Registers a listener for a sheet-level custom validation rule that involves multiple fields. The callback is executed once per record for each matching `external` constraint in the sheet's top-level `constraints` array.

**Signature:**

```typescript theme={null}
externalSheetConstraint(
  validator: string, 
  cb: (
    values: Record<string, TRecordValue>, 
    keys: string[], 
    support: { 
      config: any, 
      record: FlatfileRecord, 
      properties: Flatfile.Property[], 
      event: FlatfileEvent 
    }
  ) => any | Promise<any>
)
```

**Parameters:**

* `validator` (string): The name of the validator. This must match the `validator` property in the sheet's constraint configuration.
* `cb` (function): A callback function that contains the validation logic. It receives:
  * `values` (Record\<string, TRecordValue>): An object where keys are the field keys from the constraint's `fields` array and values are the corresponding cell values for the current record
  * `keys` (string\[]): An array of the field keys this constraint applies to (from the `fields` property in the blueprint)
  * `support` (object): An object containing helpful context:
    * `config` (any): The `config` object from the blueprint constraint
    * `record` (FlatfileRecord): The full record object
    * `properties` (Flatfile.Property\[]): An array of the full property (field) definitions for the fields involved in this constraint
    * `event` (FlatfileEvent): The raw event that triggered the validation

**Error Handling Examples:**

<CodeGroup>
  ```javascript JavaScript theme={null}
  // Using record.addError() - allows different error messages for different fields
  listener.use(
    externalSheetConstraint('date-range', (values, keys, { record }) => {
      if (values.startDate && values.endDate && values.startDate > values.endDate) {
        record.addError('startDate', 'Start date must be before end date.')
        record.addError('endDate', 'End date must be after start date.')
      }
    })
  )

  // Throwing an Error - applies same error message to ALL fields
  listener.use(
    externalSheetConstraint('date-range', (values) => {
      if (values.startDate && values.endDate && values.startDate > values.endDate) {
        throw 'Start date must be before end date.'
      }
    })
  )
  ```

  ```typescript TypeScript theme={null}
  // Using record.addError() - allows different error messages for different fields
  listener.use(
    externalSheetConstraint('date-range', (values, keys, { record }) => {
      if (values.startDate && values.endDate && values.startDate > values.endDate) {
        record.addError('startDate', 'Start date must be before end date.')
        record.addError('endDate', 'End date must be after start date.')
      }
    })
  )

  // Throwing an Error - applies same error message to ALL fields
  listener.use(
    externalSheetConstraint('date-range', (values) => {
      if (values.startDate && values.endDate && values.startDate > values.endDate) {
        throw 'Start date must be before end date.'
      }
    })
  )
  ```
</CodeGroup>

## Troubleshooting

* **Validator Not Firing:** Ensure the `validator` string in your blueprint constraint exactly matches the string you passed to `externalConstraint` or `externalSheetConstraint` in your listener.
* **Constraint Not Recognized:** Double-check that the constraint object in your blueprint has `type: 'external'`.
* **Sheet Constraint Issues:** For `externalSheetConstraint`, make sure the sheet-level constraint in the blueprint includes the `fields` array, listing the keys of all fields involved in the validation.

## Notes

### Special Considerations

* The plugin fetches and caches the sheet schema (blueprint) once per data submission (`commit:created` event). For very high-frequency operations, this could be a performance consideration, but for most use cases, it is not an issue.
* The plugin relies on `@flatfile/plugin-record-hook` to process records in bulk.

### Error Handling Patterns

The plugin supports two primary error handling patterns within the validation callback:

1. **Imperative:** Call `record.addError(key, message)` to add an error to a specific field. This is useful for sheet-level constraints where you might want to flag only one of the involved fields.
2. **Declarative:** `throw new Error(message)` or `throw "message"`. The plugin will catch the thrown error. For `externalConstraint`, the error is added to the field being validated. For `externalSheetConstraint`, the same error message is added to *all* fields listed in the constraint's `fields` array.


# Currency Conversion Plugin
Source: https://flatfile.com/docs/plugins/currency

Automatically converts currency values from a source currency to a target currency using Open Exchange Rates API with support for historical exchange rates.

This plugin automatically converts currency values from a source currency to a target currency for records within a Flatfile Sheet. It utilizes the Open Exchange Rates API to fetch both the latest and historical exchange rates.

The primary use case is for processing financial data, such as transaction logs or expense reports, where amounts need to be standardized into a single currency. The plugin can use a date from another field in the record to fetch the correct historical rate for the conversion. It can optionally store the calculated exchange rate and the date of conversion back into the record. The plugin operates by hooking into the record processing lifecycle, making it a seamless part of the data import process.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-convert-currency
```

## Configuration & Parameters

The plugin requires a configuration object with the following parameters:

### Required Parameters

| Parameter              | Type   | Description                                                            |
| ---------------------- | ------ | ---------------------------------------------------------------------- |
| `sheetSlug`            | string | The slug of the sheet the plugin should operate on                     |
| `sourceCurrency`       | string | The three-letter currency code (e.g., "USD") of the source amounts     |
| `targetCurrency`       | string | The three-letter currency code (e.g., "EUR") to convert the amounts to |
| `amountField`          | string | The field key/slug that contains the numerical amount to be converted  |
| `convertedAmountField` | string | The field key/slug where the converted amount will be stored           |

### Optional Parameters

| Parameter             | Type   | Description                                                                                          | Default Behavior                                 |
| --------------------- | ------ | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| `dateField`           | string | The field key/slug containing the date (in YYYY-MM-DD format) for fetching historical exchange rates | Uses current date to fetch latest exchange rates |
| `exchangeRateField`   | string | The field key/slug where the calculated exchange rate for the conversion will be stored              | Exchange rate is not stored on the record        |
| `conversionDateField` | string | The field key/slug where the timestamp of the conversion will be stored in ISO format                | Conversion date is not stored on the record      |

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { currencyConverterPlugin } from "@flatfile/plugin-convert-currency";

  export default function (listener) {
    listener.use(
      currencyConverterPlugin({
        sheetSlug: "transactions",
        sourceCurrency: "USD",
        targetCurrency: "EUR",
        amountField: "amount",
        convertedAmountField: "amountInEUR",
      })
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { currencyConverterPlugin } from "@flatfile/plugin-convert-currency";

  export default function (listener: FlatfileListener) {
    listener.use(
      currencyConverterPlugin({
        sheetSlug: "transactions",
        sourceCurrency: "USD",
        targetCurrency: "EUR",
        amountField: "amount",
        convertedAmountField: "amountInEUR",
      })
    );
  }
  ```
</CodeGroup>

### Full Configuration with Historical Rates

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { currencyConverterPlugin } from "@flatfile/plugin-convert-currency";

  export default function (listener) {
    listener.use(
      currencyConverterPlugin({
        sheetSlug: "transactions",
        sourceCurrency: "USD",
        targetCurrency: "EUR",
        amountField: "amount",
        dateField: "transactionDate",
        convertedAmountField: "amountInEUR",
        exchangeRateField: "exchangeRate",
        conversionDateField: "conversionDate",
      })
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { currencyConverterPlugin } from "@flatfile/plugin-convert-currency";

  export default function (listener: FlatfileListener) {
    listener.use(
      currencyConverterPlugin({
        sheetSlug: "transactions",
        sourceCurrency: "USD",
        targetCurrency: "EUR",
        amountField: "amount",
        dateField: "transactionDate",
        convertedAmountField: "amountInEUR",
        exchangeRateField: "exchangeRate",
        conversionDateField: "conversionDate",
      })
    );
  }
  ```
</CodeGroup>

### Using Utility Functions

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { 
    validateAmount, 
    validateDate, 
    convertCurrency, 
    calculateExchangeRate 
  } from "@flatfile/plugin-convert-currency";

  // Validate an amount
  const amountResult = validateAmount(150.75);
  // Returns: { value: 150.75 }

  // Validate a date
  const dateResult = validateDate('2023-10-27');
  // Returns: { value: '2023-10-27' }

  // Convert currency
  const converted = convertCurrency(100, 0.92, 1.0);
  // Returns: 108.6957

  // Calculate exchange rate
  const rate = calculateExchangeRate(0.92, 0.80);
  // Returns: 0.869565
  ```

  ```typescript TypeScript theme={null}
  import { 
    validateAmount, 
    validateDate, 
    convertCurrency, 
    calculateExchangeRate 
  } from "@flatfile/plugin-convert-currency";

  // Validate an amount
  const amountResult = validateAmount(150.75);
  // Returns: { value: 150.75 }

  // Validate a date
  const dateResult = validateDate('2023-10-27');
  // Returns: { value: '2023-10-27' }

  // Convert currency
  const converted: number = convertCurrency(100, 0.92, 1.0);
  // Returns: 108.6957

  // Calculate exchange rate
  const rate: number = calculateExchangeRate(0.92, 0.80);
  // Returns: 0.869565
  ```
</CodeGroup>

## Troubleshooting

### Common Error Messages

| Error                            | Cause                                           | Solution                                                      |
| -------------------------------- | ----------------------------------------------- | ------------------------------------------------------------- |
| "Invalid source/target currency" | Currency codes are not valid three-letter codes | Check that currency codes are valid and supported by the API  |
| "Network error" or "Status: 401" | API key issues                                  | Verify `OPENEXCHANGERATES_API_KEY` is correct and not expired |
| "Amount must be a valid number"  | Invalid amount data                             | Ensure amount field contains numeric values                   |
| "Invalid date format"            | Date not in YYYY-MM-DD format                   | Ensure date field uses YYYY-MM-DD format                      |

### Error Handling

The plugin handles errors gracefully by attaching them directly to records in Flatfile:

* **Validation errors**: Attached to specific fields using `record.addError(fieldName, message)`
* **API/Network errors**: Attached as general record errors using `record.addError('general', message)`

## Notes

### Requirements

* An active subscription to the Open Exchange Rates API is required
* The `OPENEXCHANGERATES_API_KEY` environment variable must be set in your Flatfile Space with your API key

### Limitations

* All currency conversions are routed through USD as a base currency due to Open Exchange Rates API limitations on free/lower-tier plans
* The `dateField` must contain dates in `YYYY-MM-DD` format only
* Converted amounts are fixed to 4 decimal places
* Exchange rates are fixed to 6 decimal places

### Default Behavior

* When `dateField` is not provided, the plugin uses the current date to fetch the latest exchange rates
* When `exchangeRateField` is not provided, the calculated exchange rate is not stored on the record
* When `conversionDateField` is not provided, the conversion timestamp is not stored on the record
* Empty date fields default to the current date in YYYY-MM-DD format


# Date Format Normalizer
Source: https://flatfile.com/docs/plugins/date

Automatically parse and standardize date values during data import, converting various date formats into a consistent output format.

The Date Format Normalizer plugin for Flatfile is designed to automatically parse and standardize date values during the data import process. Its primary purpose is to detect various common date and time formats within specified fields and convert them into a single, consistent output format. This is useful when importing data from different sources that may use different date conventions (e.g., 'MM/DD/YYYY', 'YYYY-MM-DD', 'Jan 15, 2023'). The plugin can be configured to operate on specific fields across one or all sheets, and it can handle both date-only and date-time values. If a date string cannot be parsed, the plugin adds an error to the corresponding cell, alerting the user to the issue.

## Installation

Install the plugin via npm:

```bash theme={null}
npm install @flatfile/plugin-validate-date
```

## Configuration & Parameters

The plugin accepts a configuration object with the following parameters:

### sheetSlug

* **Type:** `string` (optional)
* **Default:** `'**'` (all sheets)
* **Description:** The slug of the sheet to which the date normalization should be applied. If this option is omitted, the plugin will apply to all sheets in the workbook.

### dateFields

* **Type:** `string[]` (required)
* **Description:** An array of field keys (the column names) that contain date values needing normalization. The plugin will process each field listed in this array for every record.

### outputFormat

* **Type:** `string` (required)
* **Description:** A string defining the desired output format for the dates, following the `date-fns` format patterns (e.g., 'MM/dd/yyyy', 'yyyy-MM-dd HH:mm:ss').

### includeTime

* **Type:** `boolean` (required)
* **Description:** A boolean that determines whether to include the time component in the final output. If set to `false`, any time information from the parsed date will be stripped, leaving only the date part. If `true`, the time will be included as formatted by `outputFormat`.

### locale

* **Type:** `string` (optional)
* **Default:** `'en-US'` (hardcoded)
* **Description:** Specifies the locale for date parsing. Note: Although this option exists in the configuration interface, the current implementation hardcodes the locale to 'en-US' and does not use the value provided in this parameter.

## Usage Examples

### Basic Usage

This example applies date normalization to the 'start\_date' field on all sheets, converting dates to 'YYYY-MM-DD' format.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { validateDate } from '@flatfile/plugin-validate-date'

  export default function (listener) {
    listener.use(
      validateDate({
        dateFields: ['start_date'],
        outputFormat: 'yyyy-MM-dd',
        includeTime: false
      })
    )
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { validateDate } from '@flatfile/plugin-validate-date'

  export default function (listener: FlatfileListener) {
    listener.use(
      validateDate({
        dateFields: ['start_date'],
        outputFormat: 'yyyy-MM-dd',
        includeTime: false
      })
    )
  }
  ```
</CodeGroup>

### Configuration Example

This example configures the plugin to run only on the 'contacts' sheet. It normalizes two different date fields, 'birth\_date' and 'registration\_date', to the 'MM/dd/yyyy' format and excludes time.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { validateDate } from '@flatfile/plugin-validate-date'

  export default function (listener) {
    listener.use(
      validateDate({
        sheetSlug: 'contacts',
        dateFields: ['birth_date', 'registration_date'],
        outputFormat: 'MM/dd/yyyy',
        includeTime: false
      })
    )
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { validateDate } from '@flatfile/plugin-validate-date'

  export default function (listener: FlatfileListener) {
    listener.use(
      validateDate({
        sheetSlug: 'contacts',
        dateFields: ['birth_date', 'registration_date'],
        outputFormat: 'MM/dd/yyyy',
        includeTime: false
      })
    )
  }
  ```
</CodeGroup>

### Advanced Usage (Including Time)

This example normalizes the 'event\_timestamp' field to a format that includes both date and time.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { validateDate } from '@flatfile/plugin-validate-date'

  export default function (listener) {
    listener.use(
      validateDate({
        sheetSlug: 'event_logs',
        dateFields: ['event_timestamp'],
        outputFormat: 'yyyy-MM-dd HH:mm:ss',
        includeTime: true
      })
    )
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { validateDate } from '@flatfile/plugin-validate-date'

  export default function (listener: FlatfileListener) {
    listener.use(
      validateDate({
        sheetSlug: 'event_logs',
        dateFields: ['event_timestamp'],
        outputFormat: 'yyyy-MM-dd HH:mm:ss',
        includeTime: true
      })
    )
  }
  ```
</CodeGroup>

### Error Handling Example

If a date string cannot be parsed, the plugin adds an error to the specific cell. For example, if you try to import a record with `due_date: 'not a real date'`, the plugin will not change the value but will attach an error message.

<CodeGroup>
  ```javascript JavaScript theme={null}
  // Source Record:
  // { due_date: 'not a real date' }

  // After plugin runs, the record in Flatfile will have an error:
  // Field: 'due_date'
  // Value: 'not a real date'
  // Error Message: 'Unable to parse date string'

  import { FlatfileListener } from '@flatfile/listener'
  import { validateDate } from '@flatfile/plugin-validate-date'

  export default function (listener) {
    listener.use(
      validateDate({
        sheetSlug: 'tasks',
        dateFields: ['due_date'],
        outputFormat: 'MM/dd/yyyy',
        includeTime: false
      })
    )
  }
  ```

  ```typescript TypeScript theme={null}
  // Source Record:
  // { due_date: 'not a real date' }

  // After plugin runs, the record in Flatfile will have an error:
  // Field: 'due_date'
  // Value: 'not a real date'
  // Error Message: 'Unable to parse date string'

  import { FlatfileListener } from '@flatfile/listener'
  import { validateDate } from '@flatfile/plugin-validate-date'

  export default function (listener: FlatfileListener) {
    listener.use(
      validateDate({
        sheetSlug: 'tasks',
        dateFields: ['due_date'],
        outputFormat: 'MM/dd/yyyy',
        includeTime: false
      })
    )
  }
  ```
</CodeGroup>

## Troubleshooting

If dates are not being normalized as expected, consider the following:

* **Check Configuration:** Verify that the `sheetSlug` and `dateFields` in the configuration correctly match your workbook setup.
* **Validate Format String:** Ensure that the `outputFormat` string is a valid format recognized by `date-fns`.
* **Locale Issues:** If a valid date is being marked with an error, it may be in a format not recognized by `chrono-node` or it may conflict with the hardcoded 'en-US' locale (e.g., a DD/MM/YYYY format might be misinterpreted as MM/DD/YYYY).

## Notes

### Default Behavior

The plugin hooks into the `commit:created` event. For each committed record, it checks the fields specified in `dateFields`. If a value exists, it attempts to parse it as a date. If successful, it reformats the date according to `outputFormat` and updates the record. If parsing fails, it adds an error message to the cell and leaves the original value unchanged. By default, it operates on all sheets unless a specific `sheetSlug` is provided.

### Special Considerations

* The plugin relies on the `chrono-node` library for date parsing, which supports a wide variety of natural language and standard date formats.
* The plugin hooks into the `commit:created` event, meaning it runs after a user submits their data and before it is finalized.
* The `outputFormat` string must be compatible with the `date-fns` formatting library.

### Limitations

* The `locale` configuration option is not currently implemented. The plugin defaults to using the 'en-US' locale for parsing, regardless of the value passed in the configuration. This may affect parsing of formats where the day and month order are ambiguous (e.g., '01/02/2023').

### Error Handling

The plugin's error handling is simple: if `chrono-node` cannot parse the date string from a given field, the function returns `null`. The plugin then calls `record.addError(field, 'Unable to parse date string')` to flag the cell with an error message in the Flatfile UI. The original, un-parsable value is kept in the cell.


# Deduplicate Sheet Records
Source: https://flatfile.com/docs/plugins/dedupe

A Flatfile plugin that provides functionality to find and remove duplicate records from a sheet based on specified field values or custom logic.

The Dedupe plugin provides functionality to find and remove duplicate records from a sheet within Flatfile. It is designed to be used as a server-side listener that is triggered by a custom action configured on a specific sheet.

The primary use case is data cleaning. For example, when importing a list of contacts, you can use this plugin to automatically remove entries that have the same email address. The plugin is flexible, allowing you to specify which field to check for duplicates (e.g., 'email', 'orderId'). You can configure it to keep either the first or the last occurrence of a duplicate record. For more complex deduplication logic, you can provide your own custom function.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-dedupe
```

## Configuration & Parameters

The `dedupePlugin` function takes two parameters:

### Parameters

* **jobOperation** (string, required): The operation name that you define in a Sheet-level action. The plugin will only run when an action with this exact operation name is triggered.
* **opts** (PluginOptions, required): An object containing the configuration options for the plugin.

### Configuration Options

| Option   | Type              | Default   | Description                                                                                                                                                     |
| -------- | ----------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `on`     | string            | undefined | The field key (e.g., 'email') to check for duplicate values. Required when using the `keep` option.                                                             |
| `keep`   | 'first' \| 'last' | undefined | Determines which record to keep when a duplicate is found. 'first' keeps the first record encountered, 'last' keeps the last record encountered.                |
| `custom` | function          | undefined | A custom function that receives a batch of records and a set of unique values. Should return an array of record IDs to be deleted. Overrides the `keep` option. |
| `debug`  | boolean           | false     | When set to `true`, the plugin may output additional logging for development and debugging purposes.                                                            |

### Default Behavior

By default, without any configuration in the `opts` object, the plugin will not perform any deduplication. You must provide either the `keep` and `on` options or a `custom` function for the plugin to work. If the `keep` option is used, the `on` option becomes mandatory.

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from "./listener-instance"; // Your listener instance
  import { dedupePlugin } from "@flatfile/plugin-dedupe";

  listener.use(
    dedupePlugin("dedupe-email", {
      on: "email",
      keep: "last",
    })
  );

  // You also need to configure the action on your Sheet
  /*
  const contactsSheet = {
    name: 'Contacts',
    // ... other fields
    actions: [
      {
        operation: "dedupe-email",
        mode: "background",
        label: "Dedupe emails",
        description: "Remove duplicate emails"
      }
    ]
  }
  */
  ```

  ```typescript TypeScript theme={null}
  import { listener } from "./listener-instance"; // Your listener instance
  import { dedupePlugin } from "@flatfile/plugin-dedupe";
  import { Flatfile } from "@flatfile/api";

  listener.use(
    dedupePlugin("dedupe-email", {
      on: "email",
      keep: "last",
    })
  );

  // You also need to configure the action on your Sheet
  /*
  const contactsSheet: Flatfile.SheetConfig = {
    name: 'Contacts',
    // ... other fields
    actions: [
      {
        operation: "dedupe-email",
        mode: "background",
        label: "Dedupe emails",
        description: "Remove duplicate emails"
      }
    ]
  }
  */
  ```
</CodeGroup>

This example configures the plugin to trigger when a sheet action with the operation "dedupe-email" is clicked. It will find duplicates in the 'email' field and keep the last record found, deleting any previous ones.

### Custom Deduplication Logic

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from "./listener-instance"; // Your listener instance
  import { dedupePlugin } from "@flatfile/plugin-dedupe";

  listener.use(
    dedupePlugin("dedupe-email", {
      custom: (records) => {
        const uniques = new Set();
        const toDelete = [];

        records.forEach(record => {
          const emailValue = record.values["email"]?.value;
          if (emailValue) {
            if (uniques.has(emailValue)) {
              toDelete.push(record.id);
            } else {
              uniques.add(emailValue);
            }
          }
        });

        return toDelete;
      },
    })
  );
  ```

  ```typescript TypeScript theme={null}
  import { listener } from "./listener-instance"; // Your listener instance
  import { dedupePlugin } from "@flatfile/plugin-dedupe";
  import { Flatfile } from "@flatfile/api";

  listener.use(
    dedupePlugin("dedupe-email", {
      custom: (records: Flatfile.RecordsWithLinks) => {
        const uniques = new Set();
        const toDelete: string[] = [];

        records.forEach(record => {
          const emailValue = record.values["email"]?.value;
          if (emailValue) {
            if (uniques.has(emailValue)) {
              toDelete.push(record.id);
            } else {
              uniques.add(emailValue);
            }
          }
        });

        return toDelete;
      },
    })
  );
  ```
</CodeGroup>

This example shows how to use a custom function for more complex deduplication logic. The custom function identifies records to delete based on the 'email' field and returns their IDs.

### Keep First Record

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { dedupePlugin } from "@flatfile/plugin-dedupe";

  listener.use(
    dedupePlugin("dedupe:contacts-email", {
      on: "email",
      keep: "first",
    })
  );
  ```

  ```typescript TypeScript theme={null}
  import { dedupePlugin } from "@flatfile/plugin-dedupe";

  listener.use(
    dedupePlugin("dedupe:contacts-email", {
      on: "email",
      keep: "first",
    })
  );
  ```
</CodeGroup>

This example keeps the first record encountered for each unique email value and deletes subsequent duplicates.

## Troubleshooting

### Common Issues

**Plugin Not Triggering**

* Check for a mismatch between the `jobOperation` string in your listener code and the `operation` value in your Sheet configuration's action. They must be identical.

**Incorrect Field Error**

* Ensure the field key passed to the `on` option exists in your Sheet configuration and is spelled correctly.

**No Duplicates Removed**

* Verify your data to ensure duplicates actually exist for the specified `on` field.
* If using a `custom` function, add logging to debug its logic.

### Error Scenarios

The plugin will throw descriptive errors for common misconfigurations:

* **Missing `on` option**: `Error: \`on\` is required when \`keep\` is first\`
* **Field not found**: `Error: Field "non_existent_field" not found`
* **Invalid context**: `Error: Dedupe must be called from a sheet-level action`

## Notes

### Requirements and Limitations

* **Server-Side Requirement**: This plugin must be deployed in a server-side listener. It is not intended for client-side use.
* **Sheet Action Requirement**: The plugin is triggered by a job. To trigger this job, you must configure a Sheet-level action in your Sheet configuration. The `operation` property of this action must exactly match the `jobOperation` string passed to the `dedupePlugin` function.
* **Large Dataset Limitation**: The `keep: 'last'` option may not function as expected on very large datasets where duplicate records are spread across different pages of data. The `keep: 'first'` option is generally more reliable for large datasets as it correctly tracks unique values across all pages. For a reliable "keep last" implementation on large datasets, a `custom` function should be used.

### Error Handling

The plugin is wrapped in the `jobHandler` utility, which provides standardized job management. Any error thrown during the dedupe function's execution will be caught, and the job will be marked as 'failed' with the corresponding error message. The plugin also performs its own configuration checks and will throw descriptive errors for common misconfigurations.


# Delimited File Zip Exporter
Source: https://flatfile.com/docs/plugins/delimited-zip

Export data from all sheets within a Flatfile Workbook into delimited text files and compress them into a single ZIP archive for download.

This plugin is designed to be used in a server-side Flatfile listener. Its primary purpose is to export data from all sheets within a Flatfile Workbook into delimited text files (such as CSV or TSV). After generating a file for each sheet, it compresses all of them into a single ZIP archive. This ZIP file is then uploaded back into the Flatfile space, making it available for download. This is useful for users who need to download all their processed data from a workbook in a portable, compressed format for use in other systems or for archival purposes. The plugin is triggered by a `job:ready` event.

## Installation

Install the plugin via npm:

```bash theme={null}
npm install @flatfile/plugin-export-delimited-zip
```

## Configuration & Parameters

The plugin is configured by passing an options object to the `exportDelimitedZip` function.

| Parameter       | Type      | Default               | Description                                                                                                                     |
| --------------- | --------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `job`           | `string`  | `'downloadDelimited'` | The job name that will trigger the export process. The listener will be configured to listen for `workbook:${job}`.             |
| `delimiter`     | `string`  | `','`                 | The character to use as a delimiter to separate values in the output files. For example, use ',' for CSV or '\t' for TSV.       |
| `fileExtension` | `string`  | `'csv'`               | The file extension to use for the generated delimited files (e.g., 'csv', 'txt', 'tsv').                                        |
| `debug`         | `boolean` | `false`               | When set to true, the plugin will print detailed logs to the console during its execution, which is useful for troubleshooting. |

### Default Behavior

By default, the plugin listens for a job named `workbook:downloadDelimited`. When triggered, it will process all sheets in the workbook, convert them to CSV files (using a comma delimiter), zip them up, and upload the final archive. Debug logging is disabled.

## Usage Examples

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    import { exportDelimitedZip } from '@flatfile/plugin-export-delimited-zip'

    export default function (listener) {
      // Using default options for a job named 'downloadDelimited'
      // that exports to .csv files
      listener.use(exportDelimitedZip({
          job: 'downloadDelimited',
          delimiter: ',',
          fileExtension: 'csv'
      }))
    }
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import type { FlatfileListener } from '@flatfile/listener'
    import { exportDelimitedZip } from '@flatfile/plugin-export-delimited-zip'

    export default function (listener: FlatfileListener) {
      // Using default options for a job named 'downloadDelimited'
      // that exports to .csv files
      listener.use(exportDelimitedZip({
          job: 'downloadDelimited',
          delimiter: ',',
          fileExtension: 'csv'
      }))
    }
    ```
  </Tab>
</Tabs>

### Custom Configuration

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    import { exportDelimitedZip } from '@flatfile/plugin-export-delimited-zip'

    export default function (listener) {
      // Custom configuration to create tab-separated files (.tsv)
      // triggered by a job named 'export-workbook-tsv'
      // with debug logging enabled.
      listener.use(exportDelimitedZip({
          job: 'export-workbook-tsv',
          delimiter: '\t',
          fileExtension: 'tsv',
          debug: true
        }))
    }
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import type { FlatfileListener } from '@flatfile/listener'
    import { exportDelimitedZip } from '@flatfile/plugin-export-delimited-zip'

    export default function (listener: FlatfileListener) {
      // Custom configuration to create tab-separated files (.tsv)
      // triggered by a job named 'export-workbook-tsv'
      // with debug logging enabled.
      listener.use(exportDelimitedZip({
          job: 'export-workbook-tsv',
          delimiter: '\t',
          fileExtension: 'tsv',
          debug: true
        }))
    }
    ```
  </Tab>
</Tabs>

## API Reference

### exportDelimitedZip(options)

This function registers a listener plugin that handles the process of exporting workbook data to a compressed ZIP file. It sets up a job handler for a `job:ready` event. When the specified job is executed, the plugin fetches all sheets in the workbook, streams their records, and writes them to local temporary files using the specified delimiter. These files are then added to a ZIP archive, which is uploaded to the Flatfile space. Finally, the temporary files and directory are cleaned up.

**Parameters:**

* `options` (PluginOptions) - An object containing configuration for the plugin
  * `job` (string) - The job name to listen for
  * `delimiter` (string) - The delimiter character for the output files
  * `fileExtension` (string) - The file extension for the output files
  * `debug` (boolean, optional) - An optional flag to enable verbose console logging

**Return Value:**
Returns a `FlatfileListener` plugin instance that can be passed to `listener.use()`. The job itself, when completed successfully, returns an `outcome` object to the Flatfile UI containing a message and a link to the generated ZIP file.

## Troubleshooting

The primary method for troubleshooting is to enable the `debug: true` configuration option. This will output detailed step-by-step logs to the console, including retrieved sheets, file paths, record counts, and any caught errors. This provides visibility into where the process might be failing.

### Error Handling

If any part of the process fails (e.g., reading sheets, writing temporary files, zipping, or uploading), the function will catch the error and fail the job with a generic message: "This job failed probably because it couldn't write to the \[EXTENSION] files, compress them into a ZIP file, or upload it.". To diagnose the specific cause of failure, set the `debug` option to `true` to see detailed error logs in the console where the listener is running.

The core logic is wrapped in a single `try...catch` block. If an error occurs at any stage, it is caught, and the job is marked as failed with a general error message. Specific warnings are logged to the console if the cleanup of temporary files fails, but these do not cause the job to fail.

## Notes

### Requirements and Limitations

* **Server-Side Execution**: This plugin must be deployed in a server-side listener environment (e.g., Node.js) as it requires access to the file system (`fs`) to create temporary files and directories.
* **Temporary Files**: The plugin writes temporary delimited files and a temporary ZIP file to the operating system's temporary directory (`os.tmpdir()`). It attempts to clean these files up after the upload is complete, but in case of an unhandled crash, temporary files might be left behind.
* **File Name Sanitization**: The plugin sanitizes both the workbook name and sheet names to create valid file names. It removes special characters (`[<>:"/\\|?*]`) and replaces spaces with underscores.
* **Sheet Name Length**: Sheet names are trimmed to a maximum of 31 characters after sanitization to avoid issues with file system or ZIP format limitations.
* **Dependencies**: The plugin relies on external libraries `adm-zip` for creating ZIP archives and `csv-stringify` for generating the delimited file content.


# Delimiter Extractor Plugin
Source: https://flatfile.com/docs/plugins/delimiter-extractor

Parse text files with custom delimiters and automatically extract structured data for import into Flatfile

The Delimiter Extractor plugin is designed to parse text files that use non-standard delimiters to separate values. Its primary purpose is to automatically detect and extract structured data from these files when they are uploaded to Flatfile. It supports a variety of single-character delimiters such as `;`, `:`, `~`, `^`, and `#`.

This plugin is useful in scenarios where data is provided in custom formats that are not natively handled by the Flatfile platform's standard CSV, TSV, or PSV parsers. It operates within a server-side listener, triggering on the `file:created` event to process the file, identify headers, and structure the data into records for import.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-delimiter-extractor
```

## Configuration & Parameters

### Required Parameters

<ParamField type="string">
  The file extension (e.g., ".txt", ".dat") that the plugin should listen for. This is the first argument to the `DelimiterExtractor` function.
</ParamField>

<ParamField type="string">
  The character used to separate values in the file. Supported delimiters are `;`, `:`, `~`, `^`, `#`.
</ParamField>

### Optional Parameters

<ParamField type="boolean">
  If set to `true`, the plugin will attempt to convert numeric and boolean strings into their corresponding types. For example, "123" becomes `123` and "true" becomes `true`.
</ParamField>

<ParamField type="boolean | 'greedy'">
  Controls how empty lines in the file are handled:

  * `true`: Skips lines that are completely empty
  * `'greedy'`: Skips lines that contain only whitespace characters
  * `false`: Includes all lines, even empty ones
</ParamField>

<ParamField type="function">
  A function that is applied to each individual cell value during parsing. The return value of the function will replace the original value. This is applied before `dynamicTyping`.
</ParamField>

<ParamField type="number">
  The number of records to process in each batch or chunk when inserting data into Flatfile.
</ParamField>

<ParamField type="number">
  The number of chunks to process concurrently.
</ParamField>

<ParamField type="object">
  An advanced configuration object to control the header detection strategy. Allows for specifying explicit headers, looking for headers in specific rows, or using different detection algorithms. Default uses the 'default' algorithm, which selects the row with the most non-empty cells within the first 10 rows as the header.
</ParamField>

<ParamField type="string[]">
  An array of delimiter characters to try if a specific `delimiter` is not provided. The parser will use the first one that successfully parses the data.
</ParamField>

<ParamField type="boolean">
  Enables debug logging.
</ParamField>

## Usage Examples

### Basic Usage

Configure the listener to use the plugin for any `.txt` file, specifying that the data is separated by a colon:

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from "@flatfile/platform";
  import { DelimiterExtractor } from "@flatfile/plugin-delimiter-extractor";

  listener.use(DelimiterExtractor(".txt", { delimiter: ":" }));
  ```

  ```typescript TypeScript theme={null}
  import { listener } from "@flatfile/platform";
  import { DelimiterExtractor } from "@flatfile/plugin-delimiter-extractor";

  listener.use(DelimiterExtractor(".txt", { delimiter: ":" }));
  ```
</CodeGroup>

### Advanced Configuration

This example shows a more detailed configuration for `.data` files with type conversion, empty line handling, and value transformation:

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from "@flatfile/platform";
  import { DelimiterExtractor } from "@flatfile/plugin-delimiter-extractor";

  const options = {
    delimiter: "#",
    dynamicTyping: true,
    skipEmptyLines: 'greedy',
    transform: (value) => {
      if (typeof value === 'string') {
        return value.toUpperCase();
      }
      return value;
    },
  };

  listener.use(DelimiterExtractor(".data", options));
  ```

  ```typescript TypeScript theme={null}
  import { listener } from "@flatfile/platform";
  import { DelimiterExtractor } from "@flatfile/plugin-delimiter-extractor";

  const options = {
    delimiter: "#",
    dynamicTyping: true,
    skipEmptyLines: 'greedy' as const,
    transform: (value: any) => {
      if (typeof value === 'string') {
        return value.toUpperCase();
      }
      return value;
    },
  };

  listener.use(DelimiterExtractor(".data", options));
  ```
</CodeGroup>

### Custom Header Detection

This example demonstrates how to use advanced header detection options to explicitly define headers:

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from "@flatfile/platform";
  import { DelimiterExtractor } from "@flatfile/plugin-delimiter-extractor";

  const advancedOptions = {
    delimiter: "~",
    headerDetectionOptions: {
      algorithm: 'explicitHeaders',
      headers: ['product_id', 'product_name', 'quantity', 'price'],
      skip: 1 // Skip the first row in the file
    }
  };

  listener.use(DelimiterExtractor(".inv", advancedOptions));
  ```

  ```typescript TypeScript theme={null}
  import { listener } from "@flatfile/platform";
  import { DelimiterExtractor } from "@flatfile/plugin-delimiter-extractor";

  const advancedOptions = {
    delimiter: "~",
    headerDetectionOptions: {
      algorithm: 'explicitHeaders' as const,
      headers: ['product_id', 'product_name', 'quantity', 'price'],
      skip: 1 // Skip the first row in the file
    }
  };

  listener.use(DelimiterExtractor(".inv", advancedOptions));
  ```
</CodeGroup>

### Direct Buffer Parsing

For advanced use cases where you need to parse a buffer directly:

<CodeGroup>
  ```javascript JavaScript theme={null}
  import * as fs from 'fs';
  import { delimiterParser } from "@flatfile/plugin-delimiter-extractor";

  async function parseLocalFile() {
    const fileBuffer = fs.readFileSync('my-data.txt');
    const options = { delimiter: '|', dynamicTyping: true };

    const workbookData = await delimiterParser(fileBuffer, options);
    console.log(workbookData.Sheet1.headers);
    console.log(workbookData.Sheet1.data[0]);
  }

  parseLocalFile();
  ```

  ```typescript TypeScript theme={null}
  import * as fs from 'fs';
  import { delimiterParser } from "@flatfile/plugin-delimiter-extractor";

  async function parseLocalFile(): Promise<void> {
    const fileBuffer = fs.readFileSync('my-data.txt');
    const options = { delimiter: '|', dynamicTyping: true };

    const workbookData = await delimiterParser(fileBuffer, options);
    console.log(workbookData.Sheet1.headers);
    console.log(workbookData.Sheet1.data[0]);
  }

  parseLocalFile();
  ```
</CodeGroup>

## Troubleshooting

### No Data Appears After Upload

If a file is uploaded but no data appears, check the following:

1. **File Extension**: Ensure the file extension matches the one configured in `DelimiterExtractor(fileExt, ...)`
2. **Delimiter**: Verify that the `delimiter` option matches the actual delimiter used in the file
3. **Empty Files**: If the file is empty or contains no parsable data, the plugin will log "No data found in the file" to the console and produce no records

### Unsupported File Types Error

<CodeGroup>
  ```javascript JavaScript theme={null}
  try {
    // This will throw an error
    const csvExtractor = DelimiterExtractor(".csv", { delimiter: "," });
  } catch (e) {
    console.error(e.message); 
    // -> ".csv is a native file type and not supported by the delimiter extractor."
  }
  ```

  ```typescript TypeScript theme={null}
  try {
    // This will throw an error
    const csvExtractor = DelimiterExtractor(".csv", { delimiter: "," });
  } catch (e: any) {
    console.error(e.message); 
    // -> ".csv is a native file type and not supported by the delimiter extractor."
  }
  ```
</CodeGroup>

## Notes

### Limitations

* The plugin explicitly does not support file types that are natively handled by Flatfile: `.csv` (comma-separated), `.tsv` (tab-separated), and `.psv` (pipe-separated)
* The list of supported delimiters is fixed to: `;`, `:`, `~`, `^`, `#`
* This plugin is intended to be run in a server-side listener environment within the Flatfile Platform

### Error Handling

* The main `DelimiterExtractor` function includes a guard clause that throws an `Error` if an unsupported native file type is provided
* The internal parsing function uses try-catch blocks to handle parsing errors, which are logged to the console and re-thrown, causing the associated Flatfile job to fail

### Default Behavior

* By default, the plugin does not perform type conversion (`dynamicTyping: false`)
* Empty lines are included in the output unless explicitly configured otherwise (`skipEmptyLines: false`)
* The plugin processes 10,000 records per chunk with no parallel processing (`chunkSize: 10000`, `parallel: 1`)
* Header detection uses the 'default' algorithm, selecting the row with the most non-empty cells within the first 10 rows


# Email Validation Plugin
Source: https://flatfile.com/docs/plugins/email

Validate email addresses in Flatfile data imports with format checking, required field validation, and disposable domain blocking

The Email Validation plugin for Flatfile provides a convenient way to validate email addresses in your data. It integrates into the Flatfile Listener as a record hook, automatically checking specified fields on each record. The plugin's core functionalities include validating the email format, checking for required values, and blocking emails from disposable or temporary domains. It is highly configurable, allowing you to specify which fields and sheets to target, provide a custom list of disposable domains, and customize the error messages shown to the user for different validation failures. This is useful for ensuring data quality for contact lists, user sign-ups, and any dataset containing email addresses.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-validate-email
```

## Configuration & Parameters

The plugin is configured with a single object passed to the `validateEmail` function.

### sheetSlug

* **Type:** `string`
* **Required:** No
* **Default:** `'**'`
* **Description:** The slug of the sheet to apply the validation to. The default value `'**'` means the plugin will run on all sheets in the workspace.

### emailFields

* **Type:** `string[]`
* **Required:** Yes
* **Description:** An array of strings, where each string is the field key (or column name) that should be validated as an email.

### disposableDomains

* **Type:** `string[]`
* **Required:** No
* **Default:** `[]` (empty array)
* **Description:** A custom list of email domains that should be considered disposable and rejected. The comparison is case-insensitive.

### errorMessages

* **Type:** `object`
* **Required:** No
* **Description:** An object to override the default error messages. The available keys are:
  * `required`: Message for a missing email value. Default: "Email is required"
  * `invalid`: Message for an improperly formatted email. Default: "Invalid email format"
  * `disposable`: Message for an email from a blocked domain. Default: "Disposable email addresses are not allowed"
  * `domain`: Not currently used by the plugin

## Usage Examples

### Basic Usage

This example validates the 'email' field in all sheets with default error messages.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateEmail } from '@flatfile/plugin-validate-email';

  export default function (listener) {
    listener.use(validateEmail({
      emailFields: ['email']
    }));
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateEmail } from '@flatfile/plugin-validate-email';

  export default function (listener: FlatfileListener) {
    listener.use(validateEmail({
      emailFields: ['email']
    }));
  }
  ```
</CodeGroup>

### Configuration with Custom Messages

This example validates two email fields, provides custom error messages, and applies the validation only to the 'contacts' sheet.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateEmail } from '@flatfile/plugin-validate-email';

  export default function (listener) {
    listener.use(validateEmail({
      sheetSlug: 'contacts',
      emailFields: ['primary_email', 'secondary_email'],
      errorMessages: {
        required: 'Please enter an email.',
        invalid: 'The email you entered is not valid.',
        disposable: 'Temporary email addresses are not permitted.'
      }
    }));
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateEmail } from '@flatfile/plugin-validate-email';

  export default function (listener: FlatfileListener) {
    listener.use(validateEmail({
      sheetSlug: 'contacts',
      emailFields: ['primary_email', 'secondary_email'],
      errorMessages: {
        required: 'Please enter an email.',
        invalid: 'The email you entered is not valid.',
        disposable: 'Temporary email addresses are not permitted.'
      }
    }));
  }
  ```
</CodeGroup>

### Advanced Usage with Disposable Domain Blocking

This example demonstrates using a custom list of disposable domains to block.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateEmail } from '@flatfile/plugin-validate-email';

  export default function (listener) {
    const blockedDomains = ['mailinator.com', 'temp-mail.org', '10minutemail.com'];

    listener.use(validateEmail({
      emailFields: ['email'],
      disposableDomains: blockedDomains,
      errorMessages: {
        disposable: 'This email provider is not allowed. Please use a permanent email address.'
      }
    }));
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateEmail } from '@flatfile/plugin-validate-email';

  export default function (listener: FlatfileListener) {
    const blockedDomains = ['mailinator.com', 'temp-mail.org', '10minutemail.com'];

    listener.use(validateEmail({
      emailFields: ['email'],
      disposableDomains: blockedDomains,
      errorMessages: {
        disposable: 'This email provider is not allowed. Please use a permanent email address.'
      }
    }));
  }
  ```
</CodeGroup>

## API Reference

### validateEmail(config)

A factory function that creates a Flatfile record hook. The hook validates specified fields on a record to ensure they are properly formatted and non-disposable email addresses.

**Parameters:**

* `config` (object): A configuration object with the following properties:
  * `sheetSlug` (string, optional): The slug of the sheet to target. Defaults to `'**'` to target all sheets.
  * `emailFields` (string\[], required): An array of field keys to validate.
  * `disposableDomains` (string\[], optional): A list of domains to reject.
  * `errorMessages` (object, optional): An object for custom error messages.

**Return Value:**
Returns a `FlatfileListener` instance configured with the record hook, which can be passed to `listener.use()`.

## Troubleshooting

* **Validation not triggering:** Verify that the `sheetSlug` in the configuration correctly matches the slug of your target Sheet.
* **Fields not being validated:** Ensure the strings in the `emailFields` array exactly match the field keys in your Sheet configuration.
* **Plugin not working:** Confirm that the plugin is correctly registered with a Flatfile listener using `listener.use()`.

## Notes

### Default Behavior

By default, the plugin applies to all sheets. If `errorMessages` are not provided, it uses its own internal messages for required, invalid, and disposable email errors. If `disposableDomains` is not provided, it will not perform any disposable domain checks, only format and presence validation.

### Special Considerations

* This plugin is a "Record Hook" type, meaning it runs on every record as it is processed.
* The validation logic uses a regular expression for format checking. While robust, it may not cover 100% of all edge-case email address formats defined in RFCs.
* The `domain` key within the `errorMessages` configuration object is defined in the type interface but is not currently used in the validation logic.
* The domain check for disposable emails is case-insensitive.

### Error Handling

* The plugin does not throw errors or stop the import process.
* Validation failures are handled by adding an error message to the specific field on the `FlatfileRecord` using the `record.addError()` method.
* This approach allows users to see all data issues inline within the Flatfile interface and correct them before finalizing the import.


# Export Workbook Plugin
Source: https://flatfile.com/docs/plugins/export-workbook

A server-side utility for Flatfile that allows users to export data from an entire Flatfile Workbook into a single, downloadable Microsoft Excel (.xlsx) file.

The Export Workbook Plugin is a server-side utility for Flatfile that allows users to export data from an entire Flatfile Workbook into a single, downloadable Microsoft Excel (`.xlsx`) file.

Its primary purpose is to provide a simple way to get data out of Flatfile in a widely-used format. The plugin is triggered by a user action, typically a "Download" button configured on the workbook. It then iterates through all the sheets in the workbook (unless some are excluded), fetches the records, and compiles them into a corresponding sheet in the generated Excel file.

Use cases include:

* Allowing end-users to download a copy of their cleaned and validated data after an import
* Creating backups or snapshots of data within a Flatfile Space
* Exporting data for use in other systems that accept Excel files
* Providing a final report of all imported data, including any validation messages as comments in the Excel cells

## Installation

<CodeGroup>
  ```bash npm theme={null}
  npm install @flatfile/plugin-export-workbook
  ```
</CodeGroup>

## Configuration & Parameters

The plugin is configured by passing an options object to the `exportWorkbookPlugin` function.

| Parameter               | Type                                                | Default                       | Description                                                                                                                                                                                                |
| ----------------------- | --------------------------------------------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `jobName`               | `string`                                            | `'workbook:downloadWorkbook'` | The name of the job operation that the plugin will listen for. This must match the `operation` name of an action configured on the workbook.                                                               |
| `excludedSheets`        | `string[]`                                          | `undefined`                   | An array of sheet slugs to be excluded from the export.                                                                                                                                                    |
| `excludeFields`         | `string[]`                                          | `undefined`                   | An array of field keys (column names) to be excluded from the export across all sheets.                                                                                                                    |
| `excludeMessages`       | `boolean`                                           | `false`                       | If set to `true`, validation messages on records will not be included as comments in the cells of the exported Excel file.                                                                                 |
| `recordFilter`          | `'valid' \| 'error' \| 'all'`                       | `undefined`                   | Filters the records to be exported. Can be set to 'valid' to export only records without errors, or 'error' to export only records with errors.                                                            |
| `includeRecordIds`      | `boolean`                                           | `false`                       | If set to `true`, a 'recordId' column containing the Flatfile internal record ID will be added as the first column in each sheet.                                                                          |
| `autoDownload`          | `boolean`                                           | `false`                       | If set to `true`, the exported file will be downloaded automatically in the user's browser upon job completion. If `false`, the user is directed to the "Files" page in the Flatfile Space to download it. |
| `filename`              | `string`                                            | `undefined`                   | A custom filename for the exported file (without the `.xlsx` extension). If not provided, a filename is generated using the workbook name and a timestamp.                                                 |
| `debug`                 | `boolean`                                           | `false`                       | If set to `true`, the plugin will output verbose logging to the console, which is useful for development and troubleshooting.                                                                              |
| `sheetOptions`          | `Record<string, ExportSheetOptions>`                | `undefined`                   | An object that maps a sheet slug to sheet-specific export options.                                                                                                                                         |
| `columnNameTransformer` | `(columnName: string, sheetSlug: string) => string` | `undefined`                   | A callback function to dynamically transform column names before they are written to the Excel file.                                                                                                       |

### Sheet Options

The `sheetOptions` parameter allows you to configure specific sheets with the following options:

| Option              | Type                                      | Description                                                                                                                                 |
| ------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `skipColumnHeaders` | `boolean`                                 | If `true`, the header row with column names is omitted for that sheet.                                                                      |
| `origin`            | `number \| {row: number, column: number}` | Sets the starting cell for the data in the sheet. A number sets the starting row, while an object can set both the starting row and column. |

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { exportWorkbookPlugin } from "@flatfile/plugin-export-workbook";

  export default function (listener) {
    // Use the plugin with default settings
    listener.use(exportWorkbookPlugin());
  }

  /*
  // In your workbook.config.json, you need an action to trigger the plugin:
  "actions": [
    {
      "operation": "downloadWorkbook",
      "mode": "foreground",
      "label": "Download Excel Workbook",
      "description": "Downloads all data in an Excel file.",
      "primary": true
    }
  ]
  */
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { exportWorkbookPlugin } from "@flatfile/plugin-export-workbook";

  export default function (listener: FlatfileListener) {
    // Use the plugin with default settings
    listener.use(exportWorkbookPlugin());
  }

  /*
  // In your workbook.config.json, you need an action to trigger the plugin:
  "actions": [
    {
      "operation": "downloadWorkbook",
      "mode": "foreground",
      "label": "Download Excel Workbook",
      "description": "Downloads all data in an Excel file.",
      "primary": true
    }
  ]
  */
  ```
</CodeGroup>

### Configuration Example

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { exportWorkbookPlugin } from "@flatfile/plugin-export-workbook";

  export default function (listener) {
    listener.use(
      exportWorkbookPlugin({
        // Only export records that have passed validation
        recordFilter: 'valid',
        // Exclude the 'internal_notes' field from all sheets
        excludeFields: ['internal_notes'],
        // Exclude the 'raw_data' sheet entirely
        excludedSheets: ['raw_data'],
        // Automatically start the download for the user
        autoDownload: true,
        // Enable verbose logging for troubleshooting
        debug: true,
        // Add custom options for the 'contacts' sheet
        sheetOptions: {
          contacts: {
            // Omit the header row for the 'contacts' sheet
            skipColumnHeaders: true,
          },
        },
      })
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { exportWorkbookPlugin } from "@flatfile/plugin-export-workbook";

  export default function (listener: FlatfileListener) {
    listener.use(
      exportWorkbookPlugin({
        // Only export records that have passed validation
        recordFilter: 'valid',
        // Exclude the 'internal_notes' field from all sheets
        excludeFields: ['internal_notes'],
        // Exclude the 'raw_data' sheet entirely
        excludedSheets: ['raw_data'],
        // Automatically start the download for the user
        autoDownload: true,
        // Enable verbose logging for troubleshooting
        debug: true,
        // Add custom options for the 'contacts' sheet
        sheetOptions: {
          contacts: {
            // Omit the header row for the 'contacts' sheet
            skipColumnHeaders: true,
          },
        },
      })
    );
  }
  ```
</CodeGroup>

### Advanced Usage with Column Transformer

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { exportWorkbookPlugin } from "@flatfile/plugin-export-workbook";

  export default function (listener) {
    listener.use(
      exportWorkbookPlugin({
        // Use a custom job name
        jobName: 'export:customExcel',
        // Transform column names to be more user-friendly
        columnNameTransformer: (columnName, sheetSlug) => {
          // Example: transform 'firstName' to 'First Name'
          const friendlyName = columnName.replace(/([A-Z])/g, ' $1').replace(/^./, (str) => str.toUpperCase());
          
          // Add a prefix for a specific sheet
          if (sheetSlug === 'users') {
            return `User - ${friendlyName}`;
          }
          
          return friendlyName;
        },
      })
    );
  }

  /*
  // In your workbook.config.json, the action must match the custom jobName:
  "actions": [
    {
      "operation": "export:customExcel",
      "mode": "foreground",
      "label": "Download Custom Excel Report",
      "primary": true
    }
  ]
  */
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { exportWorkbookPlugin } from "@flatfile/plugin-export-workbook";

  export default function (listener: FlatfileListener) {
    listener.use(
      exportWorkbookPlugin({
        // Use a custom job name
        jobName: 'export:customExcel',
        // Transform column names to be more user-friendly
        columnNameTransformer: (columnName: string, sheetSlug: string) => {
          // Example: transform 'firstName' to 'First Name'
          const friendlyName = columnName.replace(/([A-Z])/g, ' $1').replace(/^./, (str) => str.toUpperCase());
          
          // Add a prefix for a specific sheet
          if (sheetSlug === 'users') {
            return `User - ${friendlyName}`;
          }
          
          return friendlyName;
        },
      })
    );
  }

  /*
  // In your workbook.config.json, the action must match the custom jobName:
  "actions": [
    {
      "operation": "export:customExcel",
      "mode": "foreground",
      "label": "Download Custom Excel Report",
      "primary": true
    }
  ]
  */
  ```
</CodeGroup>

## Troubleshooting

### Enable Debug Mode

The most important troubleshooting tool is the `debug: true` option. When enabled, the plugin prints detailed logs to the console, including which sheets are being processed, which are skipped, and the status of file writing and uploading.

### Check Action Operation

If the plugin does not trigger when the action button is clicked, ensure the `operation` value in your `workbook.config.json` action exactly matches the `jobName` used to configure the plugin.

### No Data Exported

If the exported file is empty or missing sheets, check if a `recordFilter` is unintentionally filtering out all records or if `excludedSheets` is misconfigured. The `debug` logs will show if sheets are being skipped. If all sheets are empty, the plugin will throw an error: `No data to write to Excel file.`

## Notes

### Server-Side Execution

This plugin must be deployed in a server-side listener environment, not in the browser.

### Action Configuration

For the plugin to be triggered, a corresponding `action` must be configured on the `Workbook` in your `workbook.config.json`. The `operation` of this action must match the `jobName` option of the plugin (which defaults to `workbook:downloadWorkbook`).

### File System Access

The plugin temporarily writes the `.xlsx` file to the `/tmp` directory of the execution environment, which is standard for serverless functions.

### Sheet Name Sanitization

Excel sheet names have limitations (e.g., max 31 characters, no invalid characters like `\ / ? * [ ]`). The plugin automatically sanitizes sheet names from your workbook to comply with these rules. If a name becomes empty after sanitization, it will be replaced with a default like `Sheet1`, `Sheet2`, etc.

### Default Behavior

* By default, all records from all sheets are exported
* Validation messages are included as comments in the Excel cells
* The exported file is made available in the "Files" page rather than auto-downloading
* Column headers are included in the export
* A filename is auto-generated using the workbook name and timestamp if not specified

### Error Handling

The plugin wraps its entire logic in a `try...catch` block. If any critical step fails (fetching records, writing the file to disk, uploading the file to Flatfile), it logs the error and throws a new `Error`. This causes the associated job in Flatfile to fail and display the error message to the user, providing feedback on what went wrong.


# Foreign Database Extractor for Microsoft SQL Server
Source: https://flatfile.com/docs/plugins/foreign-db-extractor

Automatically extract data from Microsoft SQL Server backup files (.bak) into Flatfile Workbooks for easy data review and processing.

This plugin automates the process of extracting data from a Microsoft SQL Server (MSSQL) database backup file (`.bak`). When a user uploads a `.bak` file to a Flatfile Space, this plugin triggers a multi-step process. First, it uploads the backup file to a secure storage location. Then, it restores the backup to a Flatfile-hosted MSSQL database instance. After the database is restored and available, the plugin inspects its schema to identify all tables. Finally, it creates a read-only Flatfile Workbook where each database table is represented as a separate Sheet. This allows users to view and interact with data from large MSSQL databases directly within the Flatfile UI without needing to manually export data to CSV or Excel first. It is ideal for scenarios where the source of truth is an MSSQL database and users need to bring that data into the Flatfile ecosystem for review or processing.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-foreign-db-extractor
```

## Configuration & Parameters

This plugin does not have any user-configurable options that are passed during initialization. The plugin operates with a default, built-in configuration that automatically listens for `file:created` events. If the uploaded file has a `.bak` extension and is not for export, it initiates a job named 'extract-foreign-mssql-db'. This job handles the entire workflow: creating a workbook, uploading the file to S3, restoring the database, polling for status, generating sheets from the database tables, and linking everything together. The process is entirely self-contained.

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { foreignDBExtractor } from '@flatfile/plugin-foreign-db-extractor';

  export default function (listener) {
    // Simply use the plugin
    listener.use(foreignDBExtractor());

    // Add other listeners as needed
  }
  ```

  ```typescript TypeScript theme={null}
  import type { FlatfileListener } from '@flatfile/listener';
  import { foreignDBExtractor } from '@flatfile/plugin-foreign-db-extractor';

  export default function (listener: FlatfileListener) {
    // Simply use the plugin
    listener.use(foreignDBExtractor());

    // Add other listeners as needed
  }
  ```
</CodeGroup>

### Complete Setup Example

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { foreignDBExtractor } from '@flatfile/plugin-foreign-db-extractor';

  const listener = new FlatfileListener();

  // Register the plugin with the listener instance
  listener.use(foreignDBExtractor());

  // The listener is now configured to handle .bak file extractions
  export default listener;
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { foreignDBExtractor } from '@flatfile/plugin-foreign-db-extractor';

  const listener = new FlatfileListener();

  // Register the plugin with the listener instance
  listener.use(foreignDBExtractor());

  // The listener is now configured to handle .bak file extractions
  export default listener;
  ```
</CodeGroup>

## API Reference

### foreignDBExtractor()

A factory function that returns a pre-configured plugin for a Flatfile Listener. The returned plugin contains all the necessary logic to listen for `.bak` file uploads and manage the extraction process. This includes creating a corresponding job, restoring the database on a Flatfile-managed service, polling for its availability, generating Sheets from the database tables, and updating the Flatfile Workbook and File entities with the results.

**Parameters:**

* None

**Return Value:**

* `(listener: FlatfileListener) => void` - A function that accepts a FlatfileListener instance and registers the necessary event handlers

**Error Handling:**

The plugin has built-in error handling. If any step in the extraction job fails (e.g., database restore fails, polling times out), the `try...catch` block within the `job:ready` listener will catch the exception. It then updates the associated File status to 'failed' and fails the Job with a descriptive error message, making the failure visible in the Flatfile UI.

<CodeGroup>
  ```javascript JavaScript theme={null}
  // Error handling is automatic - failures will be visible in the Flatfile UI
  try {
    listener.use(foreignDBExtractor());
  } catch (error) {
    console.error('Plugin registration failed:', error);
  }
  ```

  ```typescript TypeScript theme={null}
  // Error handling is automatic - failures will be visible in the Flatfile UI
  try {
    listener.use(foreignDBExtractor());
  } catch (error) {
    console.error('Plugin registration failed:', error);
  }
  ```
</CodeGroup>

## Troubleshooting

To troubleshoot issues, check the status and outcome message of the 'extract-foreign-mssql-db' job in the Flatfile dashboard. The error message from the failed step (e.g., "Database restore failed", "Failed to retrieve user credentials", or an error from the API) will be available in the job's `info` field. Common causes of failure could be a corrupted `.bak` file or the database restore process exceeding the built-in timeout.

## Notes

### Requirements

* The `@flatfile/plugin-foreign-db-extractor` and `@flatfile/listener` packages must be installed
* This feature must be enabled for your Flatfile account. Please contact Flatfile support to get access
* The environment where the listener runs must have `AGENT_INTERNAL_URL` and `FLATFILE_BEARER_TOKEN` environment variables set, which are typically provided by the Flatfile platform

### Limitations

* The plugin only activates for files with a `.bak` extension. Other file types are ignored
* The created Workbook and Sheets are read-only representations of the restored database
* The database restore process has a polling timeout of 3 minutes. If the restore takes longer, the job will fail
* The process to retrieve database user credentials has a polling timeout of 50 seconds (10 attempts with a 5-second delay)

### Error Handling Patterns

The primary error handling is centralized in the `job:ready` listener. It uses a `try...catch` block to wrap the entire extraction workflow. Upon catching an error, it uses the Flatfile API to update the status of the associated file and job to reflect the failure:

* `api.files.update(fileId, { status: 'failed' })`
* `api.jobs.fail(jobId, { info: e.message })`

This ensures that failures are clearly communicated to the user through the Flatfile UI. Internal helper functions throw standard `Error` objects with specific messages that are propagated up to this central handler.


# Geocode Address Data
Source: https://flatfile.com/docs/plugins/geocode

Automatically enrich location data using the Google Maps Geocoding API to convert addresses into geographic coordinates and vice-versa.

The Geocode plugin for Flatfile automatically enriches location data using the Google Maps Geocoding API. Its primary purpose is to convert addresses into geographic coordinates (latitude and longitude) and vice-versa.

Use cases include:

* **Forward Geocoding**: Automatically populating latitude and longitude fields from a given address field
* **Reverse Geocoding**: Automatically populating a formatted address from given latitude and longitude fields
* **Data Enrichment**: Extract and add supplementary location data to records, such as country and postal code

The plugin operates during the data commit phase (`commit:created`), processing records in bulk before they are finalized.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-enrich-geocode
```

## Configuration & Parameters

The plugin is configured by passing a configuration object to the `enrichGeocode` function.

| Parameter        | Type    | Default       | Description                                                               |
| ---------------- | ------- | ------------- | ------------------------------------------------------------------------- |
| `sheetSlug`      | string  | `"addresses"` | The slug of the sheet you want the plugin to process                      |
| `addressField`   | string  | `"address"`   | The field key/name in your sheet that contains the address to be geocoded |
| `latitudeField`  | string  | `"latitude"`  | The field key/name in your sheet for the latitude value                   |
| `longitudeField` | string  | `"longitude"` | The field key/name in your sheet for the longitude value                  |
| `autoGeocode`    | boolean | `true`        | A flag to enable or disable the automatic geocoding process               |

### Default Behavior

If no configuration is provided, the plugin will attempt to run on a sheet with the slug "addresses", looking for an "address" field to geocode and populating "latitude" and "longitude" fields.

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import enrichGeocode from "@flatfile/plugin-enrich-geocode";

  export default function (listener) {
    // Assumes a sheet with slug 'addresses' and fields 'address', 'latitude', 'longitude'
    listener.use(enrichGeocode({}));
  }
  ```

  ```typescript TypeScript theme={null}
  import type { FlatfileListener } from "@flatfile/listener";
  import enrichGeocode from "@flatfile/plugin-enrich-geocode";

  export default function (listener: FlatfileListener) {
    // Assumes a sheet with slug 'addresses' and fields 'address', 'latitude', 'longitude'
    listener.use(enrichGeocode({}));
  }
  ```
</CodeGroup>

### Custom Configuration

<CodeGroup>
  ```javascript JavaScript theme={null}
  import enrichGeocode from "@flatfile/plugin-enrich-geocode";

  export default function (listener) {
    listener.use(
      enrichGeocode({
        sheetSlug: 'contacts',
        addressField: 'full_address',
        latitudeField: 'lat',
        longitudeField: 'lon'
      })
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import type { FlatfileListener } from "@flatfile/listener";
  import enrichGeocode from "@flatfile/plugin-enrich-geocode";

  export default function (listener: FlatfileListener) {
    listener.use(
      enrichGeocode({
        sheetSlug: 'contacts',
        addressField: 'full_address',
        latitudeField: 'lat',
        longitudeField: 'lon'
      })
    );
  }
  ```
</CodeGroup>

### Advanced Usage - Direct API Function

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { performGeocoding } from "@flatfile/plugin-enrich-geocode";

  async function geocodeSingleAddress(address, apiKey) {
    console.log(`Geocoding address: ${address}`);
    const result = await performGeocoding({ address }, apiKey);

    if ('message' in result) {
      console.error(`Error: ${result.message}`);
    } else {
      console.log(`Coordinates: ${result.latitude}, ${result.longitude}`);
      console.log(`Formatted Address: ${result.formatted_address}`);
    }
  }
  ```

  ```typescript TypeScript theme={null}
  import { performGeocoding } from "@flatfile/plugin-enrich-geocode";

  async function geocodeSingleAddress(address: string, apiKey: string) {
    console.log(`Geocoding address: ${address}`);
    const result = await performGeocoding({ address }, apiKey);

    if ('message' in result) {
      console.error(`Error: ${result.message}`);
    } else {
      console.log(`Coordinates: ${result.latitude}, ${result.longitude}`);
      console.log(`Formatted Address: ${result.formatted_address}`);
    }
  }
  ```
</CodeGroup>

## API Reference

### enrichGeocode(config)

The main entry point for the plugin. Returns a `bulkRecordHook` compatible with the Flatfile listener's `use()` method.

**Parameters:**

* `config` (object): Configuration object with the parameters described above

**Returns:**
A function that can be passed to `listener.use()` and will be executed on the `commit:created` event.

### performGeocoding(input, apiKey)

An async function that makes a direct call to the Google Maps Geocoding API for both forward and reverse geocoding.

**Parameters:**

* `input` (object): Must contain either:
  * `address` (string): The address to geocode
  * `latitude` (number) and `longitude` (number): The coordinates to reverse geocode
* `apiKey` (string): Your Google Maps Geocoding API key

**Returns:**
A Promise that resolves to either:

**Success (GeocodingResult):**

```javascript theme={null}
{
  latitude: number,
  longitude: number,
  formatted_address: string,
  country?: string,
  postal_code?: string
}
```

**Failure (GeocodingError):**

```javascript theme={null}
{
  message: string,
  field: string // 'address', 'coordinates', or 'input'
}
```

**Example with Error Handling:**

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { performGeocoding } from "@flatfile/plugin-enrich-geocode";

  async function findCoordinates(apiKey) {
    const result = await performGeocoding({ address: "Eiffel Tower" }, apiKey);
    
    if ('message' in result) {
      console.error(`Geocoding failed on field '${result.field}': ${result.message}`);
    } else {
      console.log(`Coordinates: ${result.latitude}, ${result.longitude}`);
    }
  }
  ```

  ```typescript TypeScript theme={null}
  import { performGeocoding } from "@flatfile/plugin-enrich-geocode";

  async function findCoordinates(apiKey: string) {
    const result = await performGeocoding({ address: "Eiffel Tower" }, apiKey);
    
    if ('message' in result) {
      console.error(`Geocoding failed on field '${result.field}': ${result.message}`);
    } else {
      console.log(`Coordinates: ${result.latitude}, ${result.longitude}`);
    }
  }
  ```
</CodeGroup>

## Notes

### API Key Requirement

A valid Google Maps Geocoding API key is required for this plugin to function. The plugin will look for the key in the environment variables as `GOOGLE_MAPS_API_KEY` or in Flatfile secrets with the name `GOOGLE_MAPS_API_KEY`.

### Data Enrichment and Sheet Configuration

Upon successful geocoding, the plugin will attempt to set values for the following fields on the record: `latitude`, `longitude`, `formatted_address`, `country`, and `postal_code`. Your Flatfile Sheet must be configured with these fields to store the enriched data. The latitude and longitude fields are configurable; the others are hardcoded.

### Error Handling Pattern

If the Google Maps API returns an error (e.g., `ZERO_RESULTS`, `REQUEST_DENIED`) or if the network request fails, the plugin will not halt the import process. Instead, it will add an error message to the specific record and field that caused the issue using `record.addError()`. This allows users to see which records failed to geocode and why directly in the Flatfile UI.

### Event Trigger

The plugin is designed to run on the `listener.on('commit:created')` event. This means it processes data after a user has reviewed their data and clicked the final submit button, but before the data is sent to its final destination.


# GPX File Parser and Analyzer
Source: https://flatfile.com/docs/plugins/gpx

Automatically processes GPX (GPS Exchange Format) files to extract waypoints, tracks, routes, and calculate geographic statistics like distance and elevation gain.

This plugin automatically processes GPX (GPS Exchange Format) files attached to records in a Flatfile Sheet. When a commit is created, the plugin triggers on a specified Sheet. It reads GPX data from a designated field in a record, parses the XML content, and extracts waypoints, tracks, and routes.

The primary purpose is to enrich records with structured data derived from GPX files. It can convert the geographic data into a tabular format, calculate aggregate statistics like total distance and elevation gain, and extract metadata like the name and description of the activity.

Use cases include analyzing fitness activities, processing geographic survey data, or managing collections of GPS routes, where users provide GPX files and the system needs to automatically extract and display key information and statistics.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-enrich-gpx
```

## Configuration & Parameters

The plugin is configured with an object containing the following properties, which map to field slugs in your Sheet:

### Required Parameters

* **sheetSlug** (`string`): The slug of the sheet the plugin should operate on.
* **gpxFileField** (`string`): The field slug in the source record that contains the GPX file content as a string.

### Optional Parameters

* **removeDuplicatesField** (`string`): The field slug that acts as a boolean flag. If the value in this field for a given record is the string "true", duplicate points will be removed from the parsed data.
* **filterDatesField** (`string`): The field slug that acts as a boolean flag. If the value in this field for a given record is the string "true", the parsed data will be filtered by a date range.
* **startDateField** (`string`): The field slug containing the start date for filtering. This is only used if `filterDatesField` is set to "true". The value should be a valid date string.
* **endDateField** (`string`): The field slug containing the end date for filtering. This is only used if `filterDatesField` is set to "true". The value should be a valid date string.

### Default Behavior

By default, the plugin will parse the GPX file without removing duplicates or filtering by date. These processing steps are only activated if the corresponding fields (`removeDuplicatesField`, `filterDatesField`) in the record are explicitly set to the string "true". If filtering is enabled but the start or end date fields are empty or invalid, the date filter will not be applied.

## Usage Examples

### Basic Setup

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { enrichGpx } from "@flatfile/plugin-enrich-gpx";

  export default function (listener) {
    listener.use(enrichGpx(listener, {
      sheetSlug: 'gpx-data',
      gpxFileField: 'gpx_file',
      removeDuplicatesField: 'remove_duplicates',
      filterDatesField: 'filter_dates',
      startDateField: 'start_date',
      endDateField: 'end_date'
    }));
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { enrichGpx } from "@flatfile/plugin-enrich-gpx";

  export default function (listener: FlatfileListener) {
    listener.use(enrichGpx(listener, {
      sheetSlug: 'gpx-data',
      gpxFileField: 'gpx_file',
      removeDuplicatesField: 'remove_duplicates',
      filterDatesField: 'filter_dates',
      startDateField: 'start_date',
      endDateField: 'end_date'
    }));
  }
  ```
</CodeGroup>

### Complete Configuration Example

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { enrichGpx } from "@flatfile/plugin-enrich-gpx";

  export default function (listener) {
    const config = {
      sheetSlug: 'gpx-data', // The sheet to target
      gpxFileField: 'gpx_file', // Field with GPX string content
      removeDuplicatesField: 'remove_duplicates', // Field to enable/disable deduplication
      filterDatesField: 'filter_dates', // Field to enable/disable date filtering
      startDateField: 'start_date', // Field with the filter start date
      endDateField: 'end_date' // Field with the filter end date
    };

    listener.use(enrichGpx(listener, config));
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { enrichGpx, GpxParserConfig } from "@flatfile/plugin-enrich-gpx";

  export default function (listener: FlatfileListener) {
    const config: GpxParserConfig = {
      sheetSlug: 'gpx-data', // The sheet to target
      gpxFileField: 'gpx_file', // Field with GPX string content
      removeDuplicatesField: 'remove_duplicates', // Field to enable/disable deduplication
      filterDatesField: 'filter_dates', // Field to enable/disable date filtering
      startDateField: 'start_date', // Field with the filter start date
      endDateField: 'end_date' // Field with the filter end date
    };

    listener.use(enrichGpx(listener, config));
  }
  ```
</CodeGroup>

### Using Utility Functions

The plugin exports several utility functions that can be used independently for custom data processing tasks:

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { calculateDistance } from "@flatfile/plugin-enrich-gpx";

  // Define two waypoints
  const point1 = { latitude: 40.7128, longitude: -74.0060 }; // New York City
  const point2 = { latitude: 34.0522, longitude: -118.2437 }; // Los Angeles

  // Calculate the distance between them
  const distanceInKm = calculateDistance(point1, point2);

  console.log(`The distance is approximately ${distanceInKm.toFixed(2)} km.`);
  // Expected output: The distance is approximately 3944.42 km.
  ```

  ```typescript TypeScript theme={null}
  import { calculateDistance, Waypoint } from "@flatfile/plugin-enrich-gpx";

  // Define two waypoints
  const point1: Waypoint = { latitude: 40.7128, longitude: -74.0060 }; // New York City
  const point2: Waypoint = { latitude: 34.0522, longitude: -118.2437 }; // Los Angeles

  // Calculate the distance between them
  const distanceInKm: number = calculateDistance(point1, point2);

  console.log(`The distance is approximately ${distanceInKm.toFixed(2)} km.`);
  // Expected output: The distance is approximately 3944.42 km.
  ```
</CodeGroup>

## API Reference

### enrichGpx(listener, config)

The main plugin entry point that configures and attaches a record hook to a Flatfile listener.

**Parameters:**

* `listener`: FlatfileListener - The Flatfile listener instance to attach the hook to
* `config`: GpxParserConfig - Configuration object containing field mappings

**Returns:** void

### calculateDistance(point1, point2)

Calculates the great-circle distance between two geographical points using the Haversine formula.

**Parameters:**

* `point1`: Waypoint - Object with `latitude` and `longitude` properties
* `point2`: Waypoint - Object with `latitude` and `longitude` properties

**Returns:** number - The distance between the two points in kilometers

### removeDuplicatePoints(points)

Removes duplicate waypoints from an array based on latitude, longitude, elevation, and time.

**Parameters:**

* `points`: Waypoint\[] - Array of waypoint objects

**Returns:** Waypoint\[] - New array with duplicate waypoints removed

### filterByDateRange(points, startDate, endDate)

Filters waypoints to include only those within a specified date range.

**Parameters:**

* `points`: Waypoint\[] - Array of waypoint objects with optional `time` property
* `startDate`: Date - Start of the date range
* `endDate`: Date - End of the date range

**Returns:** Waypoint\[] - Filtered array of waypoints

### calculateStatistics(tabularData)

Calculates total distance and cumulative positive elevation gain for a path of waypoints.

**Parameters:**

* `tabularData`: Waypoint\[] - Sorted array of waypoint objects

**Returns:** object - Object with `totalDistance` (km) and `elevationGain` (meters) properties

### convertToTabularFormat(waypoints, tracks, routes)

Merges separate arrays of waypoints, tracks, and routes into a single, flat array sorted by time.

**Parameters:**

* `waypoints`: Waypoint\[] - Array of waypoint objects
* `tracks`: Track\[] - Array of track objects
* `routes`: Route\[] - Array of route objects

**Returns:** Waypoint\[] - Single, sorted array containing all points

## Troubleshooting

### Common Issues

* **Data not processing**: Ensure the `sheetSlug` in the configuration exactly matches the slug of your target Sheet
* **Field mapping errors**: Verify that all field slugs in the configuration match the field keys in your Sheet template
* **Invalid GPX content**: Check that the `gpxFileField` contains valid GPX XML content
* **Date filtering not working**: Ensure the `filterDatesField` contains the string "true" and that date fields contain valid date strings

### Error Handling

The plugin includes comprehensive error handling:

* **Missing GPX content**: Adds error message 'GPX file content is required' to the `gpxFileField`
* **Invalid XML**: Adds generic error message 'Failed to parse GPX content' to the `gpxFileField`
* **Processing continues**: Records are always returned to ensure other records can be processed

## Notes

### Important Considerations

* The plugin operates on the `commit:created` event using a `recordHook`
* GPX data must be provided as a complete XML string within the specified field
* Boolean-like options are controlled by the string value "true", not native boolean types
* The plugin will overwrite values in target fields with extracted GPX data
* Processing steps (deduplication, date filtering) are only activated when explicitly enabled

### Limitations

* Does not handle file uploads directly, only processes string content from uploads
* Requires valid GPX XML format for successful parsing
* Date filtering requires valid date strings in the specified format


# GraphQL Schema to Flatfile Blueprint Converter
Source: https://flatfile.com/docs/plugins/graphql-schema

Automatically generate Flatfile Space configurations by converting GraphQL schemas into Workbooks and Sheets, streamlining data import setup for GraphQL APIs.

This plugin automates the creation of a Flatfile Space configuration by converting a GraphQL schema into a Flatfile Blueprint. It introspects a given GraphQL source—which can be a live endpoint URL, a static Schema Definition Language (SDL) string, or a GraphQL.js schema object—and generates corresponding Workbooks and Sheets.

The primary purpose is to significantly speed up the setup process for developers who need to import data that conforms to an existing GraphQL API. It maps GraphQL object types to Flatfile Sheets and their fields to Flatfile Fields, automatically handling scalar types, object relationships (as reference fields), and non-null constraints.

**Use cases include:**

* Rapidly scaffolding a data importer for a headless CMS or backend service that exposes a GraphQL API
* Creating a consistent data onboarding experience based on a single source of truth (the GraphQL schema)
* Migrating data from other systems into an application with a GraphQL-based data model

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-graphql-schema
```

## Configuration & Parameters

The plugin is configured via a `setupFactory` object passed to the `configureSpaceGraphQL` function.

### Main Configuration

| Parameter   | Type                      | Required | Description                                                                  |
| ----------- | ------------------------- | -------- | ---------------------------------------------------------------------------- |
| `workbooks` | `PartialWorkbookConfig[]` | Yes      | Array of workbook configuration objects. Each object generates one workbook. |
| `space`     | `object`                  | No       | Configuration for the Space itself (e.g., metadata, themes).                 |
| `documents` | `object[]`                | No       | Array of document configurations to add to the Space.                        |

### PartialWorkbookConfig

| Parameter | Type                                  | Default              | Description                                                                                                   |
| --------- | ------------------------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------- |
| `source`  | `string \| GraphQLSchema \| function` | Required             | GraphQL schema source - can be a URL, SDL string, GraphQLSchema object, or function returning a GraphQLSchema |
| `sheets`  | `PartialSheetConfig[]`                | `undefined`          | Optional array to filter and customize generated sheets                                                       |
| `name`    | `string`                              | `'GraphQL Workbook'` | User-friendly name for the workbook                                                                           |

### PartialSheetConfig

| Parameter | Type     | Default                              | Description                                           |
| --------- | -------- | ------------------------------------ | ----------------------------------------------------- |
| `slug`    | `string` | Required                             | Sheet slug that must match a GraphQL object type name |
| `name`    | `string` | `capitalCase` of GraphQL object name | User-friendly name for the sheet                      |

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { configureSpaceGraphQL } from '@flatfile/plugin-graphql-schema'

  const listener = FlatfileListener.create((listener) => {
    listener.on('**', (event) => {
      console.log(`Event: ${event.topic}`)
    })

    listener.use(
      configureSpaceGraphQL({
        workbooks: [
          {
            name: 'SpaceX Launches',
            source: 'https://spacex-production.up.railway.app/',
          },
        ],
      })
    )
  })
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { configureSpaceGraphQL } from '@flatfile/plugin-graphql-schema'

  const listener = FlatfileListener.create((listener) => {
    listener.on('**', (event) => {
      console.log(`Event: ${event.topic}`)
    })

    listener.use(
      configureSpaceGraphQL({
        workbooks: [
          {
            name: 'SpaceX Launches',
            source: 'https://spacex-production.up.railway.app/',
          },
        ],
      })
    )
  })
  ```
</CodeGroup>

### Filtering Sheets

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { configureSpaceGraphQL } from '@flatfile/plugin-graphql-schema'

  const listener = FlatfileListener.create((listener) => {
    listener.use(
      configureSpaceGraphQL({
        workbooks: [
          {
            name: 'SpaceX Capsules',
            source: 'https://spacex-production.up.railway.app/',
            // Only generate a sheet for the 'Capsule' GraphQL object type
            sheets: [{ slug: 'Capsule' }],
          },
        ],
      })
    )
  })
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { configureSpaceGraphQL } from '@flatfile/plugin-graphql-schema'

  const listener = FlatfileListener.create((listener) => {
    listener.use(
      configureSpaceGraphQL({
        workbooks: [
          {
            name: 'SpaceX Capsules',
            source: 'https://spacex-production.up.railway.app/',
            // Only generate a sheet for the 'Capsule' GraphQL object type
            sheets: [{ slug: 'Capsule' }],
          },
        ],
      })
    )
  })
  ```
</CodeGroup>

### Advanced Configuration

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { configureSpaceGraphQL } from '@flatfile/plugin-graphql-schema'

  const listener = FlatfileListener.create((listener) => {
    listener.use(
      configureSpaceGraphQL(
        {
          workbooks: [
            {
              name: 'Custom SpaceX Workbook',
              source: 'https://spacex-production.up.railway.app/',
              sheets: [
                {
                  name: 'Capsules', // Custom sheet name
                  slug: 'Capsule',
                  actions: [
                    {
                      operation: 'dedupe',
                      mode: 'background',
                      label: 'Deduplicate Records',
                    },
                  ],
                },
              ],
            },
          ],
          space: {
            metadata: {
              theme: {
                root: { primaryColor: 'blue' },
              },
            },
          },
        },
        async (event, workbookIds, tick) => {
          const { spaceId } = event.context
          console.log('Space configured successfully!', { spaceId, workbookIds })
          await tick(100, 'Configuration complete')
        }
      )
    )
  })
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { configureSpaceGraphQL } from '@flatfile/plugin-graphql-schema'
  import type { FlatfileEvent } from '@flatfile/listener'

  const listener = FlatfileListener.create((listener) => {
    listener.use(
      configureSpaceGraphQL(
        {
          workbooks: [
            {
              name: 'Custom SpaceX Workbook',
              source: 'https://spacex-production.up.railway.app/',
              sheets: [
                {
                  name: 'Capsules', // Custom sheet name
                  slug: 'Capsule',
                  actions: [
                    {
                      operation: 'dedupe',
                      mode: 'background',
                      label: 'Deduplicate Records',
                    },
                  ],
                },
              ],
            },
          ],
          space: {
            metadata: {
              theme: {
                root: { primaryColor: 'blue' },
              },
            },
          },
        },
        async (event: FlatfileEvent, workbookIds: string[], tick) => {
          const { spaceId } = event.context
          console.log('Space configured successfully!', { spaceId, workbookIds })
          await tick(100, 'Configuration complete')
        }
      )
    )
  })
  ```
</CodeGroup>

### Using Local Schema File

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { configureSpaceGraphQL } from '@flatfile/plugin-graphql-schema'
  import fs from 'fs'
  import path from 'path'

  const listener = FlatfileListener.create((listener) => {
    // Read a GraphQL schema from a local file
    const schemaSDL = fs.readFileSync(
      path.join(__dirname, 'schema.graphql'),
      'utf8'
    )

    listener.use(
      configureSpaceGraphQL({
        workbooks: [
          {
            name: 'My Local Schema Workbook',
            source: schemaSDL,
          },
        ],
      })
    )
  })
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { configureSpaceGraphQL } from '@flatfile/plugin-graphql-schema'
  import fs from 'fs'
  import path from 'path'

  const listener = FlatfileListener.create((listener) => {
    // Read a GraphQL schema from a local file
    const schemaSDL = fs.readFileSync(
      path.join(__dirname, 'schema.graphql'),
      'utf8'
    )

    listener.use(
      configureSpaceGraphQL({
        workbooks: [
          {
            name: 'My Local Schema Workbook',
            source: schemaSDL,
          },
        ],
      })
    )
  })
  ```
</CodeGroup>

## Troubleshooting

### Missing Sheets or Fields

Check the console logs for messages about "unsupported" field types or missing reference tables. The plugin logs warnings for non-critical issues.

### Invalid GraphQL Source

Ensure that any URL provided in the `source` option points to a valid, publicly accessible GraphQL endpoint that responds to introspection queries.

### Sheet Filtering Issues

When using the `sheets` filter, verify that the `slug` for each sheet configuration exactly matches the name of the corresponding GraphQL object type.

### Reference Field Problems

If reference fields are not working, confirm that the sheet being referenced is also being generated (i.e., it's included in your `sheets` filter or the filter is omitted entirely).

## Notes

### Default Behavior

By default, the plugin introspects the entire GraphQL schema from the provided `source`. It creates one sheet for each GraphQL `OBJECT` type, excluding standard types like `Query`, `Mutation`, `Subscription`, and internal types (those starting with `__`).

**Field Type Mapping:**

* GraphQL scalar types are mapped to Flatfile field types (`Int` → `number`, `String` → `string`)
* GraphQL object types are mapped to Flatfile reference fields
* If a GraphQL field is `NON_NULL`, a `required` constraint is added

### Supported Field Types

The plugin currently supports GraphQL `SCALAR`, `OBJECT`, and `NON_NULL` types. Other GraphQL types like `LIST`, `UNION`, `INTERFACE`, and `ENUM` are not explicitly supported and will be skipped during sheet generation.

### Reference Field Generation

For a reference field to be created correctly, the corresponding GraphQL object type must also be generated as a sheet in the same workbook. The plugin automatically selects the first non-reference field from the referenced sheet to use as the relationship key.

### Error Handling

The plugin uses console logging to provide feedback. Critical errors, such as failure to fetch or parse the GraphQL schema, will throw an exception, causing the `configureSpace` job to fail.


# HTML Table Extractor
Source: https://flatfile.com/docs/plugins/html-table

Parse HTML files and extract data from tables within them, converting structured data into Flatfile-compatible format

This plugin for Flatfile is designed to parse HTML files and extract data from tables within them. Its main purpose is to automatically convert structured data found in HTML `<table>` elements into a format that Flatfile can process.

The plugin can handle multiple tables within a single HTML file, creating a separate sheet for each one. It is capable of interpreting complex table layouts that use `colspan` and `rowspan` attributes to merge cells, ensuring the data is correctly aligned. Use cases include importing data from legacy systems that export reports as HTML pages, scraping data from web pages, or processing any structured data provided in an HTML table format.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-extract-html-table
```

## Configuration & Parameters

The plugin accepts the following configuration options:

| Parameter       | Type    | Default | Description                                                                                                                                         |
| --------------- | ------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `handleColspan` | boolean | `true`  | When true, the plugin will correctly handle cells with a `colspan` attribute by duplicating the cell's value across the specified number of columns |
| `handleRowspan` | boolean | `true`  | When true, the plugin will attempt to handle cells with a `rowspan` attribute by carrying the cell's value down into the subsequent rows            |
| `maxDepth`      | number  | `3`     | Defines the maximum depth for parsing nested tables (Note: not currently implemented)                                                               |
| `debug`         | boolean | `false` | When set to true, the plugin will output detailed logs to the console during the parsing process                                                    |

### Default Behavior

By default, the plugin processes HTML files with `handleColspan` and `handleRowspan` enabled, meaning it will attempt to correctly structure data from cells that span multiple columns or rows. Debug logging is disabled, and the nesting depth for tables is notionally set to 3.

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { HTMLTableExtractor } from '@flatfile/plugin-extract-html-table';

  const listener = new FlatfileListener();

  // Use the extractor with default options
  listener.use(HTMLTableExtractor());
  ```

  ```typescript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { HTMLTableExtractor } from '@flatfile/plugin-extract-html-table';

  const listener = new FlatfileListener();

  // Use the extractor with default options
  listener.use(HTMLTableExtractor());
  ```
</CodeGroup>

### Configuration Example

<CodeGroup>
  ```javascript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { HTMLTableExtractor } from '@flatfile/plugin-extract-html-table';

  const listener = new FlatfileListener();

  // Use the extractor with custom options
  listener.use(
    HTMLTableExtractor({
      handleColspan: true,
      handleRowspan: false, // Disable rowspan handling
      debug: true // Enable verbose logging for troubleshooting
    })
  );
  ```

  ```typescript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { HTMLTableExtractor } from '@flatfile/plugin-extract-html-table';

  const listener = new FlatfileListener();

  // Use the extractor with custom options
  listener.use(
    HTMLTableExtractor({
      handleColspan: true,
      handleRowspan: false, // Disable rowspan handling
      debug: true // Enable verbose logging for troubleshooting
    })
  );
  ```
</CodeGroup>

### Direct Parser Usage

This example shows how to use the parser function directly, outside of a Flatfile listener, to process an HTML file:

<CodeGroup>
  ```javascript theme={null}
  import * as fs from 'fs';
  import { htmlTableParser } from '@flatfile/plugin-extract-html-table';

  // Read an HTML file into a buffer
  const fileBuffer = fs.readFileSync('path/to/your/table.html');

  // Define parser options
  const options = {
    handleColspan: true,
    handleRowspan: true,
    debug: false
  };

  // Parse the buffer to get structured data
  try {
    const workbookData = htmlTableParser(fileBuffer, options);
    console.log('Extracted Workbook:', JSON.stringify(workbookData, null, 2));
  } catch (error) {
    console.error('An error occurred during parsing:', error);
  }
  ```

  ```typescript theme={null}
  import * as fs from 'fs';
  import { htmlTableParser } from '@flatfile/plugin-extract-html-table';

  // Read an HTML file into a buffer
  const fileBuffer = fs.readFileSync('path/to/your/table.html');

  // Define parser options
  const options = {
    handleColspan: true,
    handleRowspan: true,
    debug: false
  };

  // Parse the buffer to get structured data
  try {
    const workbookData = htmlTableParser(fileBuffer, options);
    console.log('Extracted Workbook:', JSON.stringify(workbookData, null, 2));
  } catch (error) {
    console.error('An error occurred during parsing:', error);
  }
  ```
</CodeGroup>

### Example with HTML Content

<CodeGroup>
  ```javascript theme={null}
  import { htmlTableParser } from '@flatfile/plugin-extract-html-table';

  const htmlContent = '<table><thead><tr><th>Name</th><th>Age</th></tr></thead><tbody><tr><td>John</td><td>30</td></tr></tbody></table>';
  const buffer = Buffer.from(htmlContent, 'utf-8');
  const workbook = htmlTableParser(buffer, {});

  // workbook will be:
  // {
  //   "Table_1": {
  //     "headers": ["Name", "Age"],
  //     "data": [{ "Name": { "value": "John" }, "Age": { "value": "30" } }]
  //   }
  // }
  ```

  ```typescript theme={null}
  import { htmlTableParser } from '@flatfile/plugin-extract-html-table';

  const htmlContent = '<table><thead><tr><th>Name</th><th>Age</th></tr></thead><tbody><tr><td>John</td><td>30</td></tr></tbody></table>';
  const buffer = Buffer.from(htmlContent, 'utf-8');
  const workbook = htmlTableParser(buffer, {});

  // workbook will be:
  // {
  //   "Table_1": {
  //     "headers": ["Name", "Age"],
  //     "data": [{ "Name": { "value": "John" }, "Age": { "value": "30" } }]
  //   }
  // }
  ```
</CodeGroup>

## Troubleshooting

If data is missing or incorrect, enable `debug: true` in the configuration to see a step-by-step log of the parsing process:

<CodeGroup>
  ```javascript theme={null}
  listener.use(
    HTMLTableExtractor({
      debug: true
    })
  );
  ```

  ```typescript theme={null}
  listener.use(
    HTMLTableExtractor({
      debug: true
    })
  );
  ```
</CodeGroup>

Ensure the source HTML file contains well-structured `<table>` elements with `<th>` tags for headers and `<td>` tags for data cells. The plugin's effectiveness is highly dependent on the quality of the input HTML.

## Notes

### Important Considerations

* **Supported File Type**: The plugin is hardcoded to only process files with the `.html` extension
* **Event Listener**: It operates on the `listener.on('file:created')` event
* **Multiple Tables**: Each `<table>` element found in the HTML document will be extracted into its own separate sheet within the Flatfile workbook. Sheets are named sequentially: `Table_1`, `Table_2`, and so on
* **Header Extraction**: Headers are extracted from `<th>` elements. If a table has no `<th>` elements, the `headers` array for that sheet will be empty, and data rows will likely not be mapped correctly

### Limitations

* **`maxDepth` Limitation**: The `maxDepth` configuration option is defined in the options type but is not currently implemented in the parsing logic. Nested tables are processed, but their depth is not limited by this setting
* **`rowspan` Implementation**: The current implementation for `handleRowspan` may not function as expected because it attempts to re-parse trimmed text content of a cell to find an attribute, which is not possible. This feature should be considered unreliable

### Error Handling

* The primary method for diagnosing issues is to set the `debug` option to `true`. This will print detailed logs of the extraction process, including tables found, headers extracted, and cell data
* If a data row contains more cells than there are headers, a warning is logged (in debug mode) and the extra cell data is ignored to prevent data misalignment
* The underlying HTML parser is generally resilient to malformed HTML, but if the table structure (`<table>`, `<tr>`, `<th>`, `<td>`) is invalid, the function may return an empty object or partially extracted data


# Overview
Source: https://flatfile.com/docs/plugins/index

Extend your Flatfile data engine with powerful plugins for extraction, transformation, validation, and integration

Flatfile plugins extend your data processing capabilities with pre-built, reusable modules for common data operations. They handle everything from file extraction and data transformation to validation and external integrations, allowing you to build sophisticated data workflows quickly.

## Open Source

All Flatfile plugins are **open source** and available on GitHub at [github.com/FlatFilers/flatfile-plugins](https://github.com/FlatFilers/flatfile-plugins).

## Plugin Categories

### 🔄 Transform

Plugins for data transformation, validation, and cleaning.

<CardGroup>
  <Card title="Record Hook" href="/plugins/record-hook">
    Run custom logic on individual data records in real-time
  </Card>

  <Card title="Autocast" href="/plugins/autocast">
    Automatically cast values to appropriate data types
  </Card>

  <Card title="Constraints" href="/plugins/constraints">
    Extend schemas with external validation constraints
  </Card>

  <Card title="Dedupe" href="/plugins/dedupe">
    Remove duplicate records via sheet-level actions
  </Card>
</CardGroup>

### 📤 Extractors

Plugins for parsing and extracting data from various file formats.

<CardGroup>
  <Card title="Delimiter Extractor" href="/plugins/delimiter-extractor">
    Parse delimited files (CSV, TSV, etc.)
  </Card>

  <Card title="XLSX Extractor" href="/plugins/xlsx-extractor">
    Extract data from Excel spreadsheets
  </Card>

  <Card title="JSON Extractor" href="/plugins/json-extractor">
    Parse JSON files and convert to tabular data
  </Card>

  <Card title="PDF Extractor" href="/plugins/pdf-extractor">
    Extract structured data from PDF documents
  </Card>
</CardGroup>

### 📋 Schemas

Plugins for converting external schemas to Flatfile format.

<CardGroup>
  <Card title="JSON Schema Converter" href="/plugins/json-schema">
    Convert JSON Schema to Flatfile Blueprint
  </Card>

  <Card title="OpenAPI Schema Converter" href="/plugins/openapi-schema">
    Convert OpenAPI schema to Flatfile Blueprint
  </Card>

  <Card title="SQL DDL Converter" href="/plugins/sql-ddl-converter">
    Convert SQL DDL to Flatfile Blueprint
  </Card>

  <Card title="YAML Schema Converter" href="/plugins/yaml-schema">
    Convert YAML Schema to Flatfile Blueprint
  </Card>
</CardGroup>

### 📤 Export

Plugins for exporting processed data to external systems.

<CardGroup>
  <Card title="Export Workbook" href="/plugins/export-workbook">
    Export data from Flatfile to various formats
  </Card>

  <Card title="Webhook Egress" href="/plugins/webhook-egress">
    Send data to external webhooks
  </Card>

  <Card title="Export Delimited ZIP" href="/plugins/delimited-zip">
    Export workbooks as delimited files in ZIP archives
  </Card>

  <Card title="Export Pivot Table" href="/plugins/pivot-table">
    Generate pivot tables from sheet data
  </Card>
</CardGroup>

### 🔧 Core

Essential plugins for Flatfile operations and configuration.

<CardGroup>
  <Card title="Space Configure" href="/plugins/space-configure">
    Configure Flatfile spaces with workbooks and settings
  </Card>

  <Card title="Job Handler" href="/plugins/job-handler">
    Handle Flatfile jobs with custom logic
  </Card>

  <Card title="View Mapped" href="/plugins/view-mapped">
    Show only mapped columns in post-mapping view
  </Card>

  <Card title="Rollout" href="/plugins/rollout">
    Automatically roll out workbook changes
  </Card>
</CardGroup>

### ✅ Validation

Specialized validation plugins for data quality.

<CardGroup>
  <Card title="Validate Email" href="/plugins/email">
    Comprehensive email address validation
  </Card>

  <Card title="Validate Phone" href="/plugins/phone">
    Phone number formatting and validation
  </Card>

  <Card title="Validate Date" href="/plugins/date">
    Date format normalization and validation
  </Card>

  <Card title="Validate Number" href="/plugins/number">
    Number validation and formatting
  </Card>
</CardGroup>

### 🔗 Enrich

Plugins for data enrichment using external APIs.

<CardGroup>
  <Card title="Geocode" href="/plugins/geocode">
    Geocode addresses using Google Maps API
  </Card>

  <Card title="Sentiment Analysis" href="/plugins/sentiment">
    Analyze sentiment of text fields
  </Card>

  <Card title="Text Summarization" href="/plugins/summarize">
    Summarize and extract key phrases from text
  </Card>

  <Card title="What3Words" href="/plugins/what3words">
    Convert What3Words addresses to standard addresses
  </Card>
</CardGroup>

## Getting Started with Plugins

### Installation

Install plugins using npm in your Flatfile project:

```bash theme={null}
npm install @flatfile/plugin-record-hook @flatfile/plugin-autocast
```

### Basic Usage

Add plugins to your listener:

```typescript theme={null}
import { FlatfileListener } from '@flatfile/listener'
import { recordHook } from '@flatfile/plugin-record-hook'
import { autocast } from '@flatfile/plugin-autocast'

const listener = FlatfileListener.create(async (client) => {
  // Add autocast for automatic type conversion
  client.use(autocast())
  
  // Add record hook for custom validation
  client.use(
    recordHook('customers', (record) => {
      // Custom record processing logic
      const email = record.get('email')
      if (email && !isValidEmail(email)) {
        record.addError('email', 'Invalid email format')
      }
      return record
    })
  )
})
```

### Plugin Configuration

Most plugins accept configuration options:

```typescript theme={null}
import { configureSpace } from '@flatfile/plugin-space-configure'
import { dedupePlugin } from '@flatfile/plugin-dedupe'

client.use(
  configureSpace({
    workbooks: [workbookConfig],
    space: {
      name: "Customer Import",
      namespace: "customers"
    }
  })
)

client.use(
  dedupePlugin({
    sheetSlug: 'customers',
    dedupe: {
      key: 'email',
      keep: 'first',
      custom: {
        operation: 'dedupe-customers',
        label: 'Remove Duplicate Customers'
      }
    }
  })
)
```

## Plugin Development

### Creating Custom Plugins

You can create custom plugins for reusable functionality:

```typescript theme={null}
import { FlatfilePlugin } from '@flatfile/listener'

export function myCustomPlugin(options: any): FlatfilePlugin {
  return (client) => {
    // Plugin initialization logic
    client.filter({ job: 'sheet:myCustomOperation' }, (configure) => {
      configure.on('job:ready', async (event) => {
        // Plugin functionality
        const { jobId } = event.context
        
        try {
          await client.jobs.ack(jobId, {
            info: 'Running custom operation...'
          })
          
          // Your custom logic here
          await performCustomOperation(options)
          
          await client.jobs.complete(jobId, {
            outcome: { message: 'Custom operation completed successfully' }
          })
        } catch (error) {
          await client.jobs.fail(jobId, {
            outcome: { message: `Operation failed: ${error.message}` }
          })
        }
      })
    })
  }
}

// Usage
client.use(myCustomPlugin({ customOption: 'value' }))
```

### Plugin Best Practices

1. **Single Responsibility**: Each plugin should handle one specific functionality
2. **Configuration**: Accept configuration options for flexibility
3. **Error Handling**: Implement robust error handling and logging
4. **Documentation**: Provide clear documentation and examples
5. **Testing**: Include comprehensive tests for your plugin

## Popular Plugin Combinations

### Basic Data Import

```typescript theme={null}
// Essential plugins for most data import scenarios
client.use(autocast())
client.use(recordHook('sheet-name', validateRecord))
client.use(dedupePlugin({ sheetSlug: 'sheet-name' }))
```

### Advanced Validation

```typescript theme={null}
// Comprehensive data validation
client.use(autocast())
client.use(recordHook('customers', validateCustomer))
client.use(validateEmail({ sheetSlug: 'customers', fieldKey: 'email' }))
client.use(validatePhone({ sheetSlug: 'customers', fieldKey: 'phone' }))
client.use(validateDate({ sheetSlug: 'customers', fieldKey: 'birthdate' }))
```

### Schema Conversion

```typescript theme={null}
// Convert external schemas to Flatfile
client.use(convertJsonSchema({ source: 'api-schema.json' }))
client.use(convertOpenApiSchema({ source: 'swagger.yaml' }))
```

### Data Enrichment

```typescript theme={null}
// Enrich data with external services
client.use(enrichGeocode({ 
  sheetSlug: 'addresses',
  addressField: 'full_address'
}))
client.use(enrichSentiment({
  sheetSlug: 'feedback', 
  textField: 'comments'
}))
```

## Plugin Repository

All Flatfile plugins are open source and available on GitHub:

* **Main Repository**: [github.com/FlatFilers/flatfile-plugins](https://github.com/FlatFilers/flatfile-plugins)
* **NPM Organization**: [@flatfile](https://www.npmjs.com/org/flatfile)
* **Plugin Marketplace**: [flatfile.com/plugins](https://flatfile.com/plugins/)

## Contributing

We welcome contributions to the Flatfile plugin ecosystem:

1. **Bug Reports**: Report issues on the GitHub repository
2. **Feature Requests**: Suggest new plugins or enhancements
3. **Pull Requests**: Contribute code improvements or new plugins
4. **Documentation**: Help improve plugin documentation

## Support

* **Documentation**: Each plugin includes detailed documentation
* **Community**: Join our [Slack community](https://flatfile.com/join-slack/) for support
* **GitHub Issues**: Report bugs or request features on GitHub
* **Professional Support**: Contact our team for enterprise support


# ISBN Validator Plugin
Source: https://flatfile.com/docs/plugins/isbn

A Flatfile plugin that validates and formats International Standard Book Number (ISBN) data during the import process, supporting both ISBN-10 and ISBN-13 standards with automatic formatting and conversion capabilities.

The ISBN Validator plugin for Flatfile is designed to ensure the quality and consistency of International Standard Book Number (ISBN) data during the import process. It automatically validates fields containing ISBNs against both ISBN-10 and ISBN-13 standards, checking for correct structure and valid check digits.

Key features include the ability to automatically format valid ISBNs with hyphens for better readability and to convert ISBNs between formats (e.g., from ISBN-10 to ISBN-13). The plugin provides clear, field-level error messages for invalid ISBNs and informational messages for successful transformations. It is highly configurable, allowing users to specify which sheets and fields to target, making it a flexible tool for any data onboarding workflow involving book information.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-validate-isbn
```

## Configuration & Parameters

The plugin accepts a configuration object with the following parameters:

### `sheetSlug`

* **Type:** `string`
* **Default:** `'**'`
* **Description:** The slug of the sheet to apply the validation to. The default `'**'` applies it to all sheets in the workspace.

### `isbnFields`

* **Type:** `string[]`
* **Default:** `['isbn']`
* **Description:** An array of field keys that contain ISBN values to be validated.

### `autoFormat`

* **Type:** `boolean`
* **Default:** `true`
* **Description:** If set to true, the plugin will automatically format valid ISBNs with hyphens (e.g., '9780306406157' becomes '978-0-306-40615-7').

### `format`

* **Type:** `string` (optional)
* **Default:** `undefined`
* **Description:** If specified, the plugin will attempt to convert the ISBN to the given format. Possible values are `'isbn13'`, `'isbn13h'` (with hyphens), `'isbn10'`, and `'isbn10h'` (with hyphens).

## Usage Examples

### Basic Usage

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    import { FlatfileListener } from "@flatfile/listener";
    import validateISBN from "@flatfile/plugin-validate-isbn";

    const listener = new FlatfileListener();

    // Use the plugin with default settings
    // Validates the 'isbn' field on all sheets and auto-formats it.
    listener.use(validateISBN());
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import { FlatfileListener } from "@flatfile/listener";
    import validateISBN from "@flatfile/plugin-validate-isbn";

    const listener = new FlatfileListener();

    // Use the plugin with default settings
    // Validates the 'isbn' field on all sheets and auto-formats it.
    listener.use(validateISBN());
    ```
  </Tab>
</Tabs>

### Custom Configuration

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    import { FlatfileListener } from "@flatfile/listener";
    import validateISBN from "@flatfile/plugin-validate-isbn";

    const listener = new FlatfileListener();

    // Use the plugin with custom configuration
    listener.use(
      validateISBN({
        sheetSlug: "books",
        isbnFields: ["primary_isbn", "secondary_isbn"],
        autoFormat: true,
        format: "isbn13h", // Convert all valid ISBNs to hyphenated ISBN-13
      })
    );
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import { FlatfileListener } from "@flatfile/listener";
    import validateISBN from "@flatfile/plugin-validate-isbn";

    const listener = new FlatfileListener();

    // Use the plugin with custom configuration
    listener.use(
      validateISBN({
        sheetSlug: "books",
        isbnFields: ["primary_isbn", "secondary_isbn"],
        autoFormat: true,
        format: "isbn13h", // Convert all valid ISBNs to hyphenated ISBN-13
      })
    );
    ```
  </Tab>
</Tabs>

### Standalone Validation Function

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    // Using the exported utility function for standalone validation
    import { validateAndFormatISBN } from "@flatfile/plugin-validate-isbn";

    const isbnToTest = "0306406152";

    // Validate and convert an ISBN-10 to a hyphenated ISBN-13
    const result = validateAndFormatISBN(isbnToTest, false, "isbn13h");

    if (result.isValid) {
      console.log("Validation successful!");
      console.log("Converted ISBN:", result.convertedISBN); // Output: 978-0-306-40615-7
    } else {
      console.error("Validation failed:", result.message);
    }
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    // Using the exported utility function for standalone validation
    import { validateAndFormatISBN } from "@flatfile/plugin-validate-isbn";

    const isbnToTest = "0306406152";

    // Validate and convert an ISBN-10 to a hyphenated ISBN-13
    const result = validateAndFormatISBN(isbnToTest, false, "isbn13h");

    if (result.isValid) {
      console.log("Validation successful!");
      console.log("Converted ISBN:", result.convertedISBN); // Output: 978-0-306-40615-7
    } else {
      console.error("Validation failed:", result.message);
    }
    ```
  </Tab>
</Tabs>

## Troubleshooting

Common troubleshooting steps include:

1. **Sheet Configuration:** Verify that the `sheetSlug` in the configuration exactly matches the slug of the target sheet in your Flatfile Space.
2. **Field Mapping:** Ensure the strings in the `isbnFields` array match the field keys in your sheet's data model.
3. **Installation:** Confirm that the plugin is correctly installed (`npm install @flatfile/plugin-validate-isbn`) and imported.

## Notes

### Default Behavior

By default, the plugin runs on all sheets, targets the field with the key 'isbn', and automatically formats any valid ISBNs by adding hyphens. It does not perform any format conversion unless the `format` option is explicitly set.

### Dependencies

This plugin has an external dependency on the `isbn3` library, which handles the core validation and conversion logic. The plugin operates using `recordHook`, which is triggered by the `commit:created` event. This means validation runs after a user submits their data but before the commit is finalized.

### Error Handling

The plugin follows standard Flatfile error handling patterns:

* For validation failures, it uses `record.addError(field, message)` to attach a blocking error to the specific field.
* For successful formatting or conversion, it uses `record.addInfo(field, message)` to provide non-blocking feedback to the user.
* If a configured ISBN field is empty or null for a given record, the plugin skips it without adding an error.


# Job Handler Plugin
Source: https://flatfile.com/docs/plugins/job-handler

A Flatfile plugin that streamlines the handling of asynchronous Jobs by managing their lifecycle, progress reporting, and completion status.

The Job Handler plugin is designed to streamline the handling of Flatfile Jobs. A Job is a large unit of work performed asynchronously, such as processing a file or configuring a Space. This plugin listens for the `job:ready` event and executes a custom handler function when a matching job is triggered.

Its main purpose is to provide a structured way to manage the lifecycle of a job. It automatically acknowledges the job upon receipt, provides a `tick` function to report progress back to the user, and handles the final completion or failure status. This is useful for any long-running process initiated within Flatfile, allowing developers to focus on the business logic of the task while the plugin manages the communication with the Flatfile API.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-job-handler
```

## Configuration & Parameters

### job

* **Type:** `string | EventFilter`
* **Required:** Yes
* **Description:** A filter to specify which job the listener should handle. This is typically a string in the format "domain:operation", for example, "space:configure" or "workbook:submit".

### handler

* **Type:** `Function`
* **Required:** Yes
* **Description:** An async callback function that contains the logic to be executed for the job. It receives the `event` object and a `tick` function as arguments. Throwing an error within this function will cause the job to fail. If it completes successfully, it can optionally return a `JobCompleteDetails` object to customize the outcome message.

**Function signature:**

```typescript theme={null}
(event: FlatfileEvent, tick: TickFunction) => Promise<void | Flatfile.JobCompleteDetails>
```

### opts.debug

* **Type:** `boolean`
* **Default:** `false`
* **Description:** An optional parameter to enable detailed logging for the plugin in the console, which is useful for troubleshooting.

### Default Behavior

By default, the plugin listens for the specified `job:ready` event. When triggered, it immediately acknowledges the job with the Flatfile API, setting its status to "Accepted" and progress to 0%. It then executes the user-provided `handler`. If the handler completes without returning a value, the plugin marks the job as complete with a default message "Job complete". If the handler throws an error, the plugin catches it and marks the job as failed, using the error message as the reason.

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { jobHandler } from '@flatfile/plugin-job-handler';

  const listener = new FlatfileListener();

  listener.use(
    jobHandler('space:configure', async (event, tick) => {
      // Acknowledge the job has started
      await tick(10, 'Starting configuration...');

      // ... your custom logic to configure the space ...
      console.log('Configuring space:', event.context.spaceId);

      // Update progress
      await tick(50, 'Halfway there!');

      // ... more logic ...

      // Job is finished
      await tick(100, 'Configuration complete.');
    })
  );
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { jobHandler } from '@flatfile/plugin-job-handler';
  import type { FlatfileEvent } from '@flatfile/listener';

  const listener = new FlatfileListener();

  listener.use(
    jobHandler('space:configure', async (event: FlatfileEvent, tick) => {
      // Acknowledge the job has started
      await tick(10, 'Starting configuration...');

      // ... your custom logic to configure the space ...
      console.log('Configuring space:', event.context.spaceId);

      // Update progress
      await tick(50, 'Halfway there!');

      // ... more logic ...

      // Job is finished
      await tick(100, 'Configuration complete.');
    })
  );
  ```
</CodeGroup>

### Configuration with Debug

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { jobHandler } from '@flatfile/plugin-job-handler';

  const listener = new FlatfileListener();

  // Using the 'opts' parameter to enable debug logging
  listener.use(
    jobHandler(
      'workbook:submit',
      async (event) => {
        console.log('Processing submitted workbook...');
        // ... processing logic ...
      },
      { debug: true }
    )
  );
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { jobHandler } from '@flatfile/plugin-job-handler';
  import type { FlatfileEvent } from '@flatfile/listener';

  const listener = new FlatfileListener();

  // Using the 'opts' parameter to enable debug logging
  listener.use(
    jobHandler(
      'workbook:submit',
      async (event: FlatfileEvent) => {
        console.log('Processing submitted workbook...');
        // ... processing logic ...
      },
      { debug: true }
    )
  );
  ```
</CodeGroup>

### Advanced Usage with Custom Outcomes

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { jobHandler } from '@flatfile/plugin-job-handler';

  const listener = new FlatfileListener();

  listener.use(
    jobHandler('file:process', async (event, tick) => {
      try {
        await tick(25, 'Processing file...');
        // ... your processing logic ...

        // Return a custom outcome on success
        return {
          outcome: {
            message: 'File processed successfully. 10 new records created.',
            acknowledge: true,
          },
        };
      } catch (error) {
        console.error('An error occurred:', error);
        // Throwing the error will automatically fail the job
        throw new Error('Failed to process the file.');
      }
    })
  );
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { jobHandler } from '@flatfile/plugin-job-handler';
  import type { FlatfileEvent } from '@flatfile/listener';
  import type { Flatfile } from '@flatfile/api';

  const listener = new FlatfileListener();

  listener.use(
    jobHandler('file:process', async (event: FlatfileEvent, tick): Promise<Flatfile.JobCompleteDetails> => {
      try {
        await tick(25, 'Processing file...');
        // ... your processing logic ...

        // Return a custom outcome on success
        return {
          outcome: {
            message: 'File processed successfully. 10 new records created.',
            acknowledge: true,
          },
        };
      } catch (error) {
        console.error('An error occurred:', error);
        // Throwing the error will automatically fail the job
        throw new Error('Failed to process the file.');
      }
    })
  );
  ```
</CodeGroup>

### Error Handling Example

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { jobHandler } from '@flatfile/plugin-job-handler';

  const listener = new FlatfileListener();

  listener.use(
    jobHandler('data:validate', async (event, tick) => {
      await tick(1, 'Starting validation...');
      const records = await getRecords(event); // Fictional function

      if (records.length === 0) {
        // This will be caught by the plugin and fail the job
        throw new Error('Validation failed: No records found to validate.');
      }

      // ... continue validation ...
    })
  );
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { jobHandler } from '@flatfile/plugin-job-handler';
  import type { FlatfileEvent } from '@flatfile/listener';

  const listener = new FlatfileListener();

  listener.use(
    jobHandler('data:validate', async (event: FlatfileEvent, tick) => {
      await tick(1, 'Starting validation...');
      const records = await getRecords(event); // Fictional function

      if (records.length === 0) {
        // This will be caught by the plugin and fail the job
        throw new Error('Validation failed: No records found to validate.');
      }

      // ... continue validation ...
    })
  );
  ```
</CodeGroup>

## Troubleshooting

The main tool for troubleshooting is the `debug` option. Setting `opts.debug = true` when calling `jobHandler` will enable verbose logging from the plugin, showing when a job is received and when it is completed. Fatal errors are logged using `console.error`, which can be monitored in your server logs.

## Notes

### Requirements and Considerations

This plugin is designed to be used within a Flatfile listener environment, such as a Node.js server running the `@flatfile/listener` package. It relies on the Flatfile API to update job statuses, so proper API keys and permissions are required for the environment where the listener is running.

### Error Handling Patterns

The primary error handling pattern is to wrap the logic inside the `handler` function in a `try...catch` block if you need to perform cleanup actions, or to simply `throw` an error to immediately fail the job. The plugin automatically handles the API call to `api.jobs.fail()` when an unhandled exception is thrown from the `handler`.

### Function Signatures

```typescript theme={null}
function jobHandler(
  job: string | { job: string },
  handler: (event: FlatfileEvent, tick: TickFunction) => Promise<void | Flatfile.JobCompleteDetails>,
  opts?: { debug?: boolean }
): (listener: FlatfileListener) => void

type TickFunction = (progress: number, info?: string) => Promise<Flatfile.JobResponse>
```

### Return Values

* The `jobHandler` function returns a plugin function to be passed to `listener.use()`
* The `handler` function can return either `void` or a `Flatfile.JobCompleteDetails` object
* If `void` is returned, the job is completed with a default success message
* If a `JobCompleteDetails` object is returned, it is used to set a custom outcome for the completed job
* The `tick` function returns a `Promise<Flatfile.JobResponse>` which can be awaited but is not required


# JSON Extractor
Source: https://flatfile.com/docs/plugins/json-extractor

Parse JSON and JSON Lines files uploaded to Flatfile and extract data into Sheets within a Workbook

The JSON Extractor plugin is designed to parse JSON (`.json`) and JSON Lines (`.jsonl`, `.jsonlines`) files uploaded to Flatfile. It automatically detects the file type and extracts the data into one or more Sheets within a Flatfile Workbook.

The primary use case is to allow users to upload structured JSON data and have it seamlessly transformed into a tabular format for review and import. The plugin can handle two main JSON structures:

1. A JSON object where each top-level key is a sheet name and its value is an array of objects (records). This creates a multi-sheet Workbook.
2. A single JSON array of objects. This creates a single-sheet Workbook with the default sheet name "Sheet1".

The plugin also intelligently handles nested JSON objects by flattening them into a single-level structure, using dot notation for headers (e.g., an object `{ "address": { "city": "..." } }` becomes a column named `Address.City`). This plugin is intended to be used in a server-side listener.

## Installation

Install the JSON Extractor plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-json-extractor
```

## Configuration & Parameters

The `JSONExtractor` function accepts an optional options object with the following properties:

| Parameter   | Type      | Default | Description                                                                                                                                        |
| ----------- | --------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `chunkSize` | `number`  | `10000` | Controls the number of records to process in each batch or "chunk" when inserting data into Flatfile                                               |
| `parallel`  | `number`  | `1`     | Controls how many chunks are processed concurrently. Increasing this can speed up the import of very large files but may use more server resources |
| `debug`     | `boolean` | `false` | If set to true, enables more verbose logging for debugging purposes                                                                                |

### Default Behavior

If no options are provided, the plugin processes files by inserting data in chunks of 10,000 records, with only one chunk being processed at a time. Debug logging is disabled.

## Usage Examples

### Basic Usage

This example shows how to add the JSON extractor to a Flatfile listener with its default settings.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from '@flatfile/listener';
  import { JSONExtractor } from '@flatfile/plugin-json-extractor';

  listener.use(JSONExtractor());
  ```

  ```typescript TypeScript theme={null}
  import { listener } from '@flatfile/listener';
  import { JSONExtractor } from '@flatfile/plugin-json-extractor';

  listener.use(JSONExtractor());
  ```
</CodeGroup>

### Configuration Example

This example configures the extractor to use a smaller chunk size and process two chunks in parallel.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from '@flatfile/listener';
  import { JSONExtractor } from '@flatfile/plugin-json-extractor';

  listener.use(
    JSONExtractor({
      chunkSize: 5000,
      parallel: 2,
    })
  );
  ```

  ```typescript TypeScript theme={null}
  import { listener } from '@flatfile/listener';
  import { JSONExtractor } from '@flatfile/plugin-json-extractor';

  listener.use(
    JSONExtractor({
      chunkSize: 5000,
      parallel: 2,
    })
  );
  ```
</CodeGroup>

### Advanced Usage (Standalone Parser)

This example shows how to use the exported `jsonParser` function directly to parse a JSON file buffer outside of the Flatfile listener context.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import * as fs from 'fs';
  import { jsonParser } from '@flatfile/plugin-json-extractor';

  const fileBuffer = fs.readFileSync('path/to/my-data.json');
  const workbookData = jsonParser(fileBuffer);

  console.log(workbookData);
  // Output: { Sheet1: { headers: [...], data: [...] } }
  ```

  ```typescript TypeScript theme={null}
  import * as fs from 'fs';
  import { jsonParser } from '@flatfile/plugin-json-extractor';

  const fileBuffer = fs.readFileSync('path/to/my-data.json');
  const workbookData = jsonParser(fileBuffer);

  console.log(workbookData);
  // Output: { Sheet1: { headers: [...], data: [...] } }
  ```
</CodeGroup>

## API Reference

### JSONExtractor

This is the main factory function for the plugin. It returns a configured extractor instance that can be registered with a Flatfile listener. The extractor listens for `.json`, `.jsonl`, and `.jsonlines` file uploads, parses them, and creates the corresponding Sheets and Records in Flatfile.

**Parameters:**

* `options` (optional): `PluginOptions`
  * `chunkSize` (number): Records per chunk. Default: 10000
  * `parallel` (number): Number of chunks to process in parallel. Default: 1
  * `debug` (boolean): Enable verbose logging. Default: false

**Return Value:**
Returns a Flatfile `Extractor` instance, which is a type of listener middleware.

**Usage Example:**

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from '@flatfile/listener';
  import { JSONExtractor } from '@flatfile/plugin-json-extractor';

  // Register the plugin with default options
  listener.use(JSONExtractor());

  // Or with custom options
  listener.use(JSONExtractor({ chunkSize: 1000, parallel: 5 }));
  ```

  ```typescript TypeScript theme={null}
  import { listener } from '@flatfile/listener';
  import { JSONExtractor } from '@flatfile/plugin-json-extractor';

  // Register the plugin with default options
  listener.use(JSONExtractor());

  // Or with custom options
  listener.use(JSONExtractor({ chunkSize: 1000, parallel: 5 }));
  ```
</CodeGroup>

**Error Handling:**
The extractor is built on the `@flatfile/util-extractor` utility, which automatically handles job lifecycle events. If the parser throws an error (e.g., due to malformed JSON), the utility will catch it and fail the corresponding job in Flatfile, providing feedback to the user in the UI.

### jsonParser

A standalone function that performs the core logic of parsing a file buffer into a `WorkbookCapture` object. This is useful for testing or for use cases where parsing is needed without the full listener integration. It handles both standard JSON and JSONL formats.

**Parameters:**

* `buffer`: `Buffer` - A Node.js Buffer containing the raw file content
* `options` (optional): `{ readonly fileExt?: string }`
  * `fileExt` (string): Can be set to 'jsonl' to explicitly trigger JSON Lines parsing logic, even if the file name is not available

**Return Value:**
Returns a `WorkbookCapture` object, which has sheet names as keys and `SheetCapture` objects as values.
Example: `{ "MySheet": { headers: ["id", "name"], data: [{ id: {value: 1}, name: {value: "Test"} }] } }`

**Usage Example:**

<CodeGroup>
  ```javascript JavaScript theme={null}
  import * as fs from 'fs';
  import { jsonParser } from '@flatfile/plugin-json-extractor';

  // For a standard JSON array file
  const jsonBuffer = Buffer.from('[{"id": 1, "name": "Alice"}]');
  const workbook = jsonParser(jsonBuffer);
  // workbook -> { Sheet1: { headers: ['id', 'name'], data: [...] } }

  // For a JSONL file
  const jsonlBuffer = Buffer.from('{"id": 1}\n{"id": 2}');
  const workbookL = jsonParser(jsonlBuffer, { fileExt: 'jsonl' });
  // workbookL -> { Sheet1: { headers: ['id'], data: [...] } }
  ```

  ```typescript TypeScript theme={null}
  import * as fs from 'fs';
  import { jsonParser } from '@flatfile/plugin-json-extractor';

  // For a standard JSON array file
  const jsonBuffer = Buffer.from('[{"id": 1, "name": "Alice"}]');
  const workbook = jsonParser(jsonBuffer);
  // workbook -> { Sheet1: { headers: ['id', 'name'], data: [...] } }

  // For a JSONL file
  const jsonlBuffer = Buffer.from('{"id": 1}\n{"id": 2}');
  const workbookL = jsonParser(jsonlBuffer, { fileExt: 'jsonl' });
  // workbookL -> { Sheet1: { headers: ['id'], data: [...] } }
  ```
</CodeGroup>

**Error Handling:**

* If the buffer contains fundamentally invalid JSON, the function will throw a `SyntaxError`
* For JSONL files, it will skip any individual lines that are not valid JSON, log an error to the console for each invalid line, and continue processing the rest of the file
* If the input data is not an object or array (e.g., a simple string or number), it will log an error and return an empty object

## Troubleshooting

* **File not being processed**: Ensure it has a `.json` or `.jsonl` extension and that the listener is correctly configured with `listener.use(JSONExtractor())`
* **Data appears in wrong sheet or not at all**: Check the root structure of your JSON file. It must be either an array of objects or an object of arrays
* **Some rows from JSONL file are missing**: Check the server logs for "Invalid JSON line" errors to identify and correct malformed lines in the source file

## Notes

### Special Considerations

* **Deployment**: This plugin is designed to run in a server-side environment as part of a Flatfile listener
* **Supported File Types**: The plugin automatically triggers for files with extensions `.json`, `.jsonl`, and `.jsonlines`
* **Data Structure for Multi-Sheet Extraction**: To create multiple sheets from a single file, the root of the JSON must be an object where each key is a string (the sheet name) and each value is an array of uniform objects
* **Data Structure for Single-Sheet Extraction**: If the root of the JSON is an array of objects, the plugin will create a single sheet named "Sheet1"
* **Nested Objects**: Nested objects are flattened into columns using dot notation. For example, `{ "user": { "name": "John" } }` will result in a column named `user.name`

### Error Handling Patterns

* The core `parseBuffer` function is wrapped in a try/catch block. On a fatal parsing error (like malformed JSON), it re-throws the error, which is then handled by the extractor utility to fail the job
* For JSONL files, the parser processes the file line-by-line. If a line contains invalid JSON, it is skipped, an error is logged to the console, and processing continues with the next valid line. This makes it resilient to partially corrupted JSONL files


# JSON Schema Converter
Source: https://flatfile.com/docs/plugins/json-schema

Automatically transform JSON Schema into Flatfile Blueprint for streamlined data model setup

The JSON Schema Converter plugin for Flatfile automatically transforms a JSON Schema into a Flatfile Blueprint. A Blueprint is Flatfile's Data Definition Language (DDL) used for defining data models, validations, and transformations.

The primary purpose of this plugin is to streamline the setup of a Flatfile Space. Instead of manually defining each field for your Sheets, you can provide a JSON Schema, and the plugin will generate the corresponding Workbooks and Sheets configuration. This is particularly useful for projects that already use JSON Schema for data validation or API definitions, allowing for a single source of truth for data structures.

The plugin is designed to be used in a server-side Flatfile listener, typically on the `space:configure` event. It can fetch schemas from a URL, use a direct JSON object, or execute a function to retrieve the schema dynamically.

## Installation

```bash theme={null}
npm install @flatfile/plugin-convert-json-schema
```

## Configuration & Parameters

The plugin is configured via a single object passed to the `configureSpaceWithJsonSchema` function. This object is of type `JsonSetupFactory`.

### JsonSetupFactory

| Parameter   | Type                                   | Required | Description                                                         |
| ----------- | -------------------------------------- | -------- | ------------------------------------------------------------------- |
| `workbooks` | `PartialWorkbookConfig[]`              | Yes      | An array of workbook configurations to be created in the Space      |
| `space`     | `Partial<Flatfile.spaces.SpaceConfig>` | No       | Configuration details for the Space itself, like metadata or themes |

### PartialWorkbookConfig

| Parameter | Type                   | Required | Description                                          |
| --------- | ---------------------- | -------- | ---------------------------------------------------- |
| `name`    | `string`               | Yes      | The name of the Workbook                             |
| `sheets`  | `PartialSheetConfig[]` | Yes      | An array of sheet configurations within the Workbook |
| `actions` | `Flatfile.Action[]`    | No       | Actions available at the Workbook level              |

### PartialSheetConfig

| Parameter | Type                                                    | Required | Description                                                                                                                                                  |
| --------- | ------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `source`  | `object \| string \| (() => object \| Promise<object>)` | Yes      | The JSON Schema for the sheet. Can be a direct JSON Schema object, a string URL pointing to a schema file, or an async function that returns a schema object |
| `name`    | `string`                                                | No       | The name of the Sheet. Defaults to the `title` property from the root of the JSON Schema if not provided                                                     |
| `slug`    | `string`                                                | No       | A unique identifier for the sheet                                                                                                                            |
| `actions` | `Flatfile.Action[]`                                     | No       | Actions available at the Sheet level                                                                                                                         |

### Default Behavior

By default, the plugin will create Workbooks and Sheets as defined in the `workbooks` array. For each sheet, it parses the `source` JSON schema, converting its `properties` into Flatfile fields. Nested objects in the schema are flattened into fields with names like `parentKey_childKey`. The sheet's name defaults to the schema's `title` if not explicitly provided.

## Usage Examples

### Basic Usage with URL Schema

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { configureSpaceWithJsonSchema } from "@flatfile/plugin-convert-json-schema";
  import { FlatfileListener } from "@flatfile/listener";

  export default function (listener) {
    listener.use(
      configureSpaceWithJsonSchema({
        workbooks: [
          {
            name: "JSON Schema Workbook",
            sheets: [
              {
                name: "Person Sheet",
                source: "https://example.com/schemas/person.json",
              },
            ],
          },
        ],
      })
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import { configureSpaceWithJsonSchema } from "@flatfile/plugin-convert-json-schema";
  import { FlatfileListener } from "@flatfile/listener";

  export default function (listener: FlatfileListener) {
    listener.use(
      configureSpaceWithJsonSchema({
        workbooks: [
          {
            name: "JSON Schema Workbook",
            sheets: [
              {
                name: "Person Sheet",
                source: "https://example.com/schemas/person.json",
              },
            ],
          },
        ],
      })
    );
  }
  ```
</CodeGroup>

### Local Schema with Callback

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { configureSpaceWithJsonSchema } from "@flatfile/plugin-convert-json-schema";
  import { FlatfileListener } from "@flatfile/listener";
  import api from "@flatfile/api";

  export default function (listener) {
    const personSchema = {
      title: "Person",
      type: "object",
      properties: {
        firstName: { type: "string" },
        lastName: { type: "string" },
        age: { type: "integer", minimum: 0 },
      },
    };

    const callback = async (event, workbookIds, tick) => {
      const { spaceId } = event.context;
      await api.documents.create(spaceId, {
        title: "Welcome",
        body: "<h1>Welcome to your new Space!</h1>",
      });
      await tick(100, "Space setup complete");
    };

    listener.use(
      configureSpaceWithJsonSchema(
        {
          workbooks: [
            {
              name: "My Workbook",
              sheets: [{ source: personSchema }],
            },
          ],
        },
        callback
      )
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import { configureSpaceWithJsonSchema } from "@flatfile/plugin-convert-json-schema";
  import { FlatfileListener, FlatfileEvent } from "@flatfile/listener";
  import { TickFunction } from "@flatfile/plugin-job-handler";
  import api from "@flatfile/api";

  export default function (listener: FlatfileListener) {
    const personSchema = {
      title: "Person",
      type: "object",
      properties: {
        firstName: { type: "string" },
        lastName: { type: "string" },
        age: { type: "integer", minimum: 0 },
      },
    };

    const callback = async (event: FlatfileEvent, workbookIds: string[], tick: TickFunction) => {
      const { spaceId } = event.context;
      await api.documents.create(spaceId, {
        title: "Welcome",
        body: "<h1>Welcome to your new Space!</h1>",
      });
      await tick(100, "Space setup complete");
    };

    listener.use(
      configureSpaceWithJsonSchema(
        {
          workbooks: [
            {
              name: "My Workbook",
              sheets: [{ source: personSchema }],
            },
          ],
        },
        callback
      )
    );
  }
  ```
</CodeGroup>

### Dynamic Schema from Function

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { configureSpaceWithJsonSchema, fetchExternalReference } from "@flatfile/plugin-convert-json-schema";
  import { FlatfileListener } from "@flatfile/listener";

  export default function (listener) {
    listener.use(
      configureSpaceWithJsonSchema({
        workbooks: [
          {
            name: "Dynamic Workbook",
            sheets: [
              {
                name: "Product Sheet",
                source: async () => {
                  // Imagine custom logic here, e.g., adding auth headers
                  const schemaUrl = "https://api.my-service.com/schemas/product.json";
                  return await fetchExternalReference(schemaUrl);
                },
              },
            ],
          },
        ],
      })
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import { configureSpaceWithJsonSchema, fetchExternalReference } from "@flatfile/plugin-convert-json-schema";
  import { FlatfileListener } from "@flatfile/listener";

  export default function (listener: FlatfileListener) {
    listener.use(
      configureSpaceWithJsonSchema({
        workbooks: [
          {
            name: "Dynamic Workbook",
            sheets: [
              {
                name: "Product Sheet",
                source: async () => {
                  // Imagine custom logic here, e.g., adding auth headers
                  const schemaUrl = "https://api.my-service.com/schemas/product.json";
                  return await fetchExternalReference(schemaUrl);
                },
              },
            ],
          },
        ],
      })
    );
  }
  ```
</CodeGroup>

### Using fetchExternalReference with Error Handling

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { fetchExternalReference } from "@flatfile/plugin-convert-json-schema";

  try {
    const schema = await fetchExternalReference("https://example.com/schemas/product.json");
    // Use schema
  } catch (error) {
    console.error("Failed to fetch schema:", error.message);
    // Handle the error, perhaps by falling back to a default schema
  }
  ```

  ```typescript TypeScript theme={null}
  import { fetchExternalReference } from "@flatfile/plugin-convert-json-schema";

  try {
    const schema = await fetchExternalReference("https://example.com/schemas/product.json");
    // Use schema
  } catch (error) {
    console.error("Failed to fetch schema:", error.message);
    // Handle the error, perhaps by falling back to a default schema
  }
  ```
</CodeGroup>

## Troubleshooting

### Job Fails on `space:configure`

Check the Job status in the Flatfile Dashboard for error messages. Common causes include:

* An invalid URL was provided for a schema `source`
* The schema at the URL is not valid JSON
* A `$ref` in the schema points to a location that cannot be resolved

### Fields Not Appearing as Expected

* Ensure the JSON schema has a `properties` object at the level you expect fields to be generated from
* Nested objects are flattened with an underscore (`_`) separator. For example, an `address` object with a `street` property becomes a field with the key `address_street`
* Check the data types in your schema. Supported types include `string`, `number`, `integer`, `boolean`, and `array` (which becomes `string-list`). `enum` is also supported. Unrecognized types will be ignored

## Notes

### Special Considerations

* This plugin is intended for use in server-side listeners, specifically for the `space:configure` event
* The plugin handles JSON schema references (`$ref`). It can resolve local references within the same schema (e.g., `#/definitions/address`) and external references to other files (e.g., `common.json#/definitions/name`). External references are resolved relative to the `$id` property of the schema they are in
* The plugin makes API calls on behalf of the user, including `api.spaces.update` and `api.workbooks.create`

### Error Handling

Errors during schema fetching (`fetchExternalReference`) or reference resolution will bubble up. When used within a Flatfile listener, these unhandled exceptions will cause the associated Job to fail, which is the standard error handling pattern in the Flatfile ecosystem. The Job's execution history will contain details about the error.

It is the user's responsibility to ensure that the provided JSON schemas are valid and that any URLs are accessible from the server environment where the listener is running.


# Markdown File Extractor
Source: https://flatfile.com/docs/plugins/markdown

Automatically parse Markdown files and extract tables into Flatfile Sheets

The Markdown Extractor plugin is designed to automatically parse Markdown files (`.md`) uploaded to Flatfile. Its primary purpose is to find and extract any tables present in the markdown content. For each table it discovers, the plugin creates a new Sheet within the Flatfile Space. The table's header row is used to define the fields (columns) of the Sheet, and the subsequent rows are converted into records. This is useful for importing data that is documented or stored in Markdown files, such as in project readmes, wikis, or technical documentation.

## Installation

Install the Markdown Extractor plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-markdown-extractor
```

## Configuration & Parameters

The plugin accepts the following configuration options:

### maxTables

* **Type:** `number`
* **Default:** `Infinity`
* **Description:** Sets the maximum number of tables to extract from a single Markdown file. By default, it will extract all tables it finds.

### errorHandling

* **Type:** `'strict' | 'lenient'`
* **Default:** `'lenient'`
* **Description:** Controls how parsing errors are handled:
  * `'strict'`: The plugin will throw an error and stop processing if it encounters a malformed table (e.g., a data row with a different number of columns than the header)
  * `'lenient'`: The plugin will log a warning to the console, skip the problematic table, and continue processing the rest of the file

### debug

* **Type:** `boolean`
* **Default:** `false`
* **Description:** When set to true, enables verbose logging to the console. This is useful for troubleshooting issues with file parsing, as it shows the content being parsed, tables found, and the final normalized data.

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from '@flatfile/listener';
  import { MarkdownExtractor } from '@flatfile/plugin-markdown-extractor';

  export default function(listener) {
    listener.use(MarkdownExtractor());
  }
  ```

  ```typescript TypeScript theme={null}
  import { listener } from '@flatfile/listener';
  import { MarkdownExtractor } from '@flatfile/plugin-markdown-extractor';

  export default function(listener: any) {
    listener.use(MarkdownExtractor());
  }
  ```
</CodeGroup>

### Configuration Example

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from '@flatfile/listener';
  import { MarkdownExtractor } from '@flatfile/plugin-markdown-extractor';

  export default function(listener) {
    const options = {
      maxTables: 5,
      errorHandling: 'strict',
      debug: true
    };

    listener.use(MarkdownExtractor(options));
  }
  ```

  ```typescript TypeScript theme={null}
  import { listener } from '@flatfile/listener';
  import { MarkdownExtractor } from '@flatfile/plugin-markdown-extractor';

  export default function(listener: any) {
    const options = {
      maxTables: 5,
      errorHandling: 'strict' as const,
      debug: true
    };

    listener.use(MarkdownExtractor(options));
  }
  ```
</CodeGroup>

### Direct Parser Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import * as fs from 'fs';
  import { markdownParser } from '@flatfile/plugin-markdown-extractor';

  const markdownContent = '| ID | Name |\n|----|------|\n| 1  | Test |';
  const buffer = Buffer.from(markdownContent, 'utf-8');

  const workbookData = markdownParser(buffer, { errorHandling: 'lenient' });

  console.log(JSON.stringify(workbookData, null, 2));
  ```

  ```typescript TypeScript theme={null}
  import * as fs from 'fs';
  import { markdownParser } from '@flatfile/plugin-markdown-extractor';

  const markdownContent = '| ID | Name |\n|----|------|\n| 1  | Test |';
  const buffer = Buffer.from(markdownContent, 'utf-8');

  const workbookData = markdownParser(buffer, { errorHandling: 'lenient' });

  console.log(JSON.stringify(workbookData, null, 2));
  ```
</CodeGroup>

## API Reference

### MarkdownExtractor(options?)

The main factory function used to create and configure the markdown extractor plugin for use with a Flatfile listener.

**Parameters:**

* `options` (optional): Configuration settings object

**Returns:** An `Extractor` instance that can be passed to `listener.use()`

**Example:**

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from '@flatfile/listener';
  import { MarkdownExtractor } from '@flatfile/plugin-markdown-extractor';

  export default function(myListener) {
    // Use with default options
    myListener.use(MarkdownExtractor());

    // Use with custom options
    myListener.use(MarkdownExtractor({ maxTables: 1, errorHandling: 'strict' }));
  }
  ```

  ```typescript TypeScript theme={null}
  import { listener } from '@flatfile/listener';
  import { MarkdownExtractor } from '@flatfile/plugin-markdown-extractor';

  export default function(myListener: any) {
    // Use with default options
    myListener.use(MarkdownExtractor());

    // Use with custom options
    myListener.use(MarkdownExtractor({ maxTables: 1, errorHandling: 'strict' }));
  }
  ```
</CodeGroup>

### markdownParser(buffer, options)

A low-level function that directly parses a Buffer containing markdown content into a Flatfile `WorkbookCapture` object.

**Parameters:**

* `buffer`: A Node.js Buffer containing the UTF-8 encoded content of a markdown file
* `options`: Configuration settings object

**Returns:** A `WorkbookCapture` object, which is a map of sheet names to sheet data

**Example:**

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { markdownParser } from '@flatfile/plugin-markdown-extractor';

  const markdownContent = '| ID | Name |\n|----|------|\n| 1  | Test |';
  const buffer = Buffer.from(markdownContent, 'utf-8');

  const workbookData = markdownParser(buffer, { errorHandling: 'lenient' });
  console.log(JSON.stringify(workbookData, null, 2));
  ```

  ```typescript TypeScript theme={null}
  import { markdownParser } from '@flatfile/plugin-markdown-extractor';

  const markdownContent = '| ID | Name |\n|----|------|\n| 1  | Test |';
  const buffer = Buffer.from(markdownContent, 'utf-8');

  const workbookData = markdownParser(buffer, { errorHandling: 'lenient' });
  console.log(JSON.stringify(workbookData, null, 2));
  ```
</CodeGroup>

## Troubleshooting

### Error Handling Example

This example demonstrates how the `errorHandling` option affects behavior when parsing a malformed table:

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { markdownParser } from '@flatfile/plugin-markdown-extractor';

  // Table has a header with 2 columns but a data row with 3
  const malformedContent = '| A | B |\n|---|---|\n| 1 | 2 | 3 |';
  const buffer = Buffer.from(malformedContent, 'utf-8');

  // 'strict' mode will throw an error
  try {
    markdownParser(buffer, { errorHandling: 'strict' });
  } catch (e) {
    console.error('Caught error in strict mode:', e.message);
    //-> Caught error in strict mode: Data row length does not match header row length
  }

  // 'lenient' mode will not throw but will log a warning and skip the table
  const result = markdownParser(buffer, { errorHandling: 'lenient' });
  console.log('Result in lenient mode:', result);
  // (A warning is logged to the console)
  //-> Result in lenient mode: {}
  ```

  ```typescript TypeScript theme={null}
  import { markdownParser } from '@flatfile/plugin-markdown-extractor';

  // Table has a header with 2 columns but a data row with 3
  const malformedContent = '| A | B |\n|---|---|\n| 1 | 2 | 3 |';
  const buffer = Buffer.from(malformedContent, 'utf-8');

  // 'strict' mode will throw an error
  try {
    markdownParser(buffer, { errorHandling: 'strict' });
  } catch (e: any) {
    console.error('Caught error in strict mode:', e.message);
    //-> Caught error in strict mode: Data row length does not match header row length
  }

  // 'lenient' mode will not throw but will log a warning and skip the table
  const result = markdownParser(buffer, { errorHandling: 'lenient' });
  console.log('Result in lenient mode:', result);
  // (A warning is logged to the console)
  //-> Result in lenient mode: {}
  ```
</CodeGroup>

### Debug Mode

To diagnose parsing issues, enable debug mode:

<CodeGroup>
  ```javascript JavaScript theme={null}
  listener.use(MarkdownExtractor({ debug: true }));
  ```

  ```typescript TypeScript theme={null}
  listener.use(MarkdownExtractor({ debug: true }));
  ```
</CodeGroup>

## Notes

### Default Behavior

* The plugin extracts all tables found in a markdown file by default (`maxTables: Infinity`)
* Uses lenient error handling by default, skipping malformed tables and continuing processing
* Debug logging is disabled by default

### Special Considerations

* This plugin is intended for use in a server-side Flatfile listener
* It operates on files with the `.md` extension
* For each table found in a markdown file, a new Sheet is created with auto-generated names (`Table_1`, `Table_2`, etc.)
* The parser expects standard GitHub-flavored markdown table syntax
* Highly complex or non-standard tables may not be parsed correctly

### Error Handling Patterns

* **Lenient mode (default)**: Logs warnings for malformed tables and continues processing
* **Strict mode**: Throws errors immediately when encountering malformed tables, failing the entire file processing

### Troubleshooting Tips

* Set `debug: true` to enable detailed logging for diagnosing parsing issues
* Check console output for warnings about skipped tables in lenient mode
* Ensure markdown tables follow standard GitHub-flavored markdown syntax
* Verify that header and data rows have matching column counts


# Merge.dev Connection Plugin
Source: https://flatfile.com/docs/plugins/merge-connection

Connect Flatfile to the Merge.dev unified API platform to sync data from hundreds of third-party integrations including HRIS, ATS, CRM, and Accounting systems.

The Merge.dev Connection Plugin enables users to sync data from hundreds of third-party integrations directly into Flatfile through the Merge.dev unified API platform. When a user establishes a connection to a service via Merge within the Flatfile UI, the plugin automatically creates a new Flatfile Workbook with a schema that matches the data models for that integration. The plugin performs an initial data sync and allows users to manually trigger full data refreshes at any time.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-connect-via-merge
```

## Configuration & Parameters

This plugin is configured through Flatfile Secrets rather than code-based options.

### Required Secret

<ParamField type="Secret (String)">
  Your API key from your Merge.dev account. This secret must be created in your Flatfile Space and is used to authenticate with the Merge API for token exchange, schema fetching, and data synchronization.

  **Default Behavior:** If the `MERGE_ACCESS_KEY` secret is not present or invalid, the plugin will fail during workbook creation or data sync, resulting in a failed Flatfile Job with an error message indicating the missing key.
</ParamField>

### Function Signature

The plugin exports a single function:

```typescript theme={null}
mergePlugin(): (listener: FlatfileListener) => void
```

**Parameters:** None

**Returns:** A function that registers job handlers with the Flatfile listener for:

* `space:createConnectedWorkbook`: Creates workbooks based on Merge schemas
* `workbook:syncConnectedWorkbook`: Syncs data from connected services

## Usage Examples

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { mergePlugin } from "@flatfile/plugin-connect-via-merge";

  export default function (listener) {
    // Add the Merge.dev plugin to the listener
    listener.use(mergePlugin());
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { mergePlugin } from "@flatfile/plugin-connect-via-merge";

  export default function (listener: FlatfileListener) {
    // Add the Merge.dev plugin to the listener
    listener.use(mergePlugin());
  }
  ```
</CodeGroup>

### Complete Setup Example

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { mergePlugin } from "@flatfile/plugin-connect-via-merge";

  export default function (listener) {
    // Register the Merge.dev plugin handlers
    listener.use(mergePlugin());

    // Add other listeners as needed
    listener.on('**', (event) => {
      console.log(`Event received: ${event.topic}`);
    });
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { mergePlugin } from "@flatfile/plugin-connect-via-merge";

  export default function (listener: FlatfileListener) {
    // Register the Merge.dev plugin handlers
    listener.use(mergePlugin());

    // Add other listeners as needed
    listener.on('**', (event) => {
      console.log(`Event received: ${event.topic}`);
    });
  }
  ```
</CodeGroup>

### Configuration Setup

1. **Create the Secret:** In your Flatfile Space settings, create a new Secret with the name `MERGE_ACCESS_KEY` and your Merge.dev API key as the value.

2. **Use the Plugin:** The plugin automatically looks for the `MERGE_ACCESS_KEY` secret in the Space where it's running.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { mergePlugin } from "@flatfile/plugin-connect-via-merge";

  export default function (listener) {
    // The plugin automatically uses the 'MERGE_ACCESS_KEY' secret
    listener.use(mergePlugin());
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { mergePlugin } from "@flatfile/plugin-connect-via-merge";

  export default function (listener: FlatfileListener) {
    // The plugin automatically uses the 'MERGE_ACCESS_KEY' secret
    listener.use(mergePlugin());
  }
  ```
</CodeGroup>

## Troubleshooting

### Missing API Key Error

If you encounter errors related to missing Merge API keys, ensure that:

1. The `MERGE_ACCESS_KEY` secret is properly set in your Flatfile Space
2. The API key value is correct and active in your Merge.dev account

### Sync Timeout Issues

If sync operations timeout, the plugin polls Merge.dev for up to 5 minutes (30 attempts at 10-second intervals). If Merge doesn't complete its sync within this window, the job will fail with a timeout error.

### Job Failures

Check the Flatfile Job logs in your dashboard for specific error messages. The plugin provides descriptive error messages for common issues like missing credentials or API failures.

## Notes

### Prerequisites

* The `connections` feature flag must be enabled for your Flatfile account (contact [support@flatfile.com](mailto:support@flatfile.com))
* Active Merge.dev account required
* `MERGE_ACCESS_KEY` secret must be configured in your Flatfile Space

### Alpha Release Warning

This plugin is an alpha release. Functionality and APIs may change in future versions.

### Data Sync Behavior

* Each sync performs a **full refresh** of data
* All existing records are deleted before inserting current data
* Manual changes made in Flatfile will be overwritten on next sync
* This ensures data consistency with the source system

### Automatic Secret Management

The plugin automatically creates workbook-specific secrets named `<workbookId>:MERGE_X_ACCOUNT_TOKEN` to store account tokens for each connection. These are managed internally by the plugin.

### Error Handling Pattern

The plugin uses centralized error handling that:

* Logs original errors to the console for debugging
* Throws user-friendly error messages displayed in the Flatfile UI
* Causes failed jobs to show descriptive error messages in the dashboard

### Manual Sync Actions

After initial setup, the plugin automatically adds a "Sync" action to connected workbooks. Users can trigger this action from the Flatfile UI to refresh data without additional code.


# Number Validation Plugin
Source: https://flatfile.com/docs/plugins/number

Comprehensive validation for numeric data during the Flatfile import process with support for ranges, formats, and special number types.

The Number Validation Plugin for Flatfile provides comprehensive validation for numeric data during the import process. It attaches to the `commit:created` listener event to analyze and validate data from specified fields.

Its main purpose is to ensure that numeric data conforms to a wide range of rules before being accepted into the system. Use cases include:

* Enforcing value ranges (e.g., age must be between 18 and 65)
* Validating data formats like integers, currency, or numbers with specific precision and scale
* Ensuring numbers follow specific rules, such as being a multiple of a certain value (step validation)
* Checking for special numeric properties like being even, odd, or prime
* Automatically cleaning up number formats by handling thousands separators and decimal points
* Optionally rounding or truncating numbers before validation

The plugin modifies records by setting the cleaned, parsed numeric value and attaching any validation errors or warnings directly to the relevant field.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-validate-number
```

## Configuration & Parameters

The `validateNumber` function accepts a single configuration object with the following properties:

### Required Parameters

| Parameter | Type       | Description                                                                           |
| --------- | ---------- | ------------------------------------------------------------------------------------- |
| `fields`  | `string[]` | An array of field keys (column names) to which the validation rules should be applied |

### Optional Parameters

| Parameter            | Type       | Default   | Description                                                                                          |
| -------------------- | ---------- | --------- | ---------------------------------------------------------------------------------------------------- |
| `sheetSlug`          | `string`   | `'**'`    | The slug of the sheet to apply the validation to. Defaults to all sheets                             |
| `min`                | `number`   | undefined | The minimum allowed value for the number                                                             |
| `max`                | `number`   | undefined | The maximum allowed value for the number                                                             |
| `inclusive`          | `boolean`  | `false`   | Determines if the `min` and `max` values are inclusive                                               |
| `integerOnly`        | `boolean`  | `false`   | If true, the value must be an integer (no decimal part)                                              |
| `precision`          | `number`   | undefined | The total maximum number of digits allowed (requires `scale` to be set)                              |
| `scale`              | `number`   | undefined | The maximum number of digits allowed after the decimal point (requires `precision` to be set)        |
| `currency`           | `boolean`  | `false`   | If true, validates that the number is a valid currency format, allowing up to two decimal places     |
| `step`               | `number`   | undefined | The value must be a multiple of this number                                                          |
| `thousandsSeparator` | `string`   | `','`     | The character used as a thousands separator in the input string                                      |
| `decimalPoint`       | `string`   | `'.'`     | The character used as the decimal point in the input string                                          |
| `specialTypes`       | `string[]` | undefined | An array of special number types to validate against. Supported values: `'prime'`, `'even'`, `'odd'` |
| `round`              | `boolean`  | `false`   | If true, the number is rounded to the nearest integer before validation                              |
| `truncate`           | `boolean`  | `false`   | If true, the decimal part of the number is removed before validation                                 |

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateNumber } from '@flatfile/plugin-validate-number';

  const listener = new FlatfileListener();

  listener.use(
    validateNumber({
      fields: ['quantity'],
      integerOnly: true,
    })
  );
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateNumber } from '@flatfile/plugin-validate-number';

  const listener = new FlatfileListener();

  listener.use(
    validateNumber({
      fields: ['quantity'],
      integerOnly: true,
    })
  );
  ```
</CodeGroup>

### Currency Validation

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateNumber } from '@flatfile/plugin-validate-number';

  const listener = new FlatfileListener();

  listener.use(
    validateNumber({
      sheetSlug: 'products',
      fields: ['price'],
      min: 0,
      max: 1000000,
      inclusive: true,
      currency: true,
      precision: 9, // 7 digits before decimal, 2 after
      scale: 2,
      thousandsSeparator: ',',
      decimalPoint: '.',
    })
  );
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateNumber } from '@flatfile/plugin-validate-number';

  const listener = new FlatfileListener();

  listener.use(
    validateNumber({
      sheetSlug: 'products',
      fields: ['price'],
      min: 0,
      max: 1000000,
      inclusive: true,
      currency: true,
      precision: 9, // 7 digits before decimal, 2 after
      scale: 2,
      thousandsSeparator: ',',
      decimalPoint: '.',
    })
  );
  ```
</CodeGroup>

### Direct Validation Function

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { validateNumberField } from '@flatfile/plugin-validate-number';

  const customValidation = () => {
    const inputValue = '1,234.56';
    const config = { thousandsSeparator: ',', decimalPoint: '.' };

    const result = validateNumberField(inputValue, config);

    if (result.errors.length > 0) {
      console.error('Validation Errors:', result.errors);
    } else if (result.warnings.length > 0) {
      console.warn('Validation Warnings:', result.warnings);
    } else {
      console.log('Validated Value:', result.value); // Outputs: 1234.56
    }
  };
  ```

  ```typescript TypeScript theme={null}
  import { validateNumberField } from '@flatfile/plugin-validate-number';

  const customValidation = (): void => {
    const inputValue = '1,234.56';
    const config = { thousandsSeparator: ',', decimalPoint: '.' };

    const result = validateNumberField(inputValue, config);

    if (result.errors.length > 0) {
      console.error('Validation Errors:', result.errors);
    } else if (result.warnings.length > 0) {
      console.warn('Validation Warnings:', result.warnings);
    } else {
      console.log('Validated Value:', result.value); // Outputs: 1234.56
    }
  };
  ```
</CodeGroup>

## API Reference

### validateNumber

The main plugin entry point that returns a function for use with `listener.use()`.

**Parameters:**

* `config` (object): Configuration object containing validation rules and target fields

**Returns:**
A function of type `(listener: FlatfileListener) => void` that registers the validation hook.

**Example:**

<CodeGroup>
  ```javascript JavaScript theme={null}
  listener.use(
    validateNumber({
      fields: ['score'],
      min: 0,
      max: 100,
      inclusive: true,
    })
  );
  ```

  ```typescript TypeScript theme={null}
  listener.use(
    validateNumber({
      fields: ['score'],
      min: 0,
      max: 100,
      inclusive: true,
    })
  );
  ```
</CodeGroup>

### validateNumberField

A standalone utility function that validates a single value against validation rules.

**Parameters:**

* `value` (string | number): The input value to validate
* `config` (NumberValidationConfig): Configuration object with validation rules

**Returns:**
`NumberValidationResult` object with:

* `value` (number | null): The parsed numeric value or null if parsing failed
* `errors` (string\[]): Array of error messages for fundamental parsing failures
* `warnings` (string\[]): Array of warning messages for validation rule violations

**Error Handling Example:**

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { validateNumberField } from '@flatfile/plugin-validate-number';

  const result = validateNumberField('not-a-number', {});
  console.log(result.errors); // Outputs: ['Must be a number']
  console.log(result.value); // Outputs: null

  const result2 = validateNumberField('99.99', { integerOnly: true });
  console.log(result2.warnings); // Outputs: ['Must be an integer']
  console.log(result2.value); // Outputs: 99.99
  ```

  ```typescript TypeScript theme={null}
  import { validateNumberField } from '@flatfile/plugin-validate-number';

  const result = validateNumberField('not-a-number', {});
  console.log(result.errors); // Outputs: ['Must be a number']
  console.log(result.value); // Outputs: null

  const result2 = validateNumberField('99.99', { integerOnly: true });
  console.log(result2.warnings); // Outputs: ['Must be an integer']
  console.log(result2.value); // Outputs: 99.99
  ```
</CodeGroup>

### isPrime

A helper function to check if a number is prime. Used internally when `specialTypes: ['prime']` is configured.

**Parameters:**

* `num` (number): The number to check

**Returns:**
`boolean` - True if the number is prime, false otherwise

## Troubleshooting

* **Validation not working**: Check the `fields` and `sheetSlug` configuration to ensure the plugin is targeting the correct data
* **Unexpected number formats**: Verify the `thousandsSeparator` and `decimalPoint` settings match your data format
* **Reference examples**: The test file `src/validate.number.plugin.spec.ts` provides clear examples of expected outcomes for every configuration option

## Notes

### Default Behavior

* The `min`/`max` validation is **exclusive by default**. To include boundary values, set `inclusive: true`
* Default `thousandsSeparator` is `','` and `decimalPoint` is `'.'`
* If both `round` and `truncate` are true, rounding occurs first, then truncation

### Special Considerations

* The plugin operates on the `commit:created` event, running after user submission but before data finalization
* The plugin modifies `FlatfileRecord` in place, overwriting original values with parsed numeric values
* Fundamental parsing failures (e.g., "abc" as a number) result in "errors"
* Rule violations (e.g., number outside min/max range) result in "warnings"
* The plugin follows standard Flatfile patterns by adding errors/warnings to records rather than throwing exceptions


# OpenAPI Schema to Flatfile Blueprint Converter
Source: https://flatfile.com/docs/plugins/openapi-schema

Automatically converts OpenAPI v3.0.3 schemas into Flatfile Blueprints to streamline Space setup using existing API data structures

This plugin automatically converts an OpenAPI v3.0.3 schema into a Flatfile Blueprint. Its primary purpose is to streamline the setup of a Flatfile Space by using an existing OpenAPI specification as the single source of truth for data models. When a new Space is configured, the plugin fetches the specified OpenAPI schema, parses its components, and generates corresponding Workbooks and Sheets with correctly typed fields, constraints, and relationships. This is ideal for developers who want to quickly create a data import experience that matches their existing API data structures without manually defining each field in Flatfile.

## Installation

```bash theme={null}
npm install @flatfile/plugin-convert-openapi-schema
```

## Configuration & Parameters

The plugin is configured via a single `setupFactory` object passed to the `configureSpaceWithOpenAPI` function.

### setupFactory

The main configuration object containing the following properties:

* **workbooks** (required): `Array<PartialWorkbookConfig>` - An array of workbook configurations to create from OpenAPI schemas
  * **source** (required): `string` - The URL pointing to the raw OpenAPI schema JSON file
  * **sheets** (required): `Array<PartialSheetConfig>` - An array of sheet configurations, mapping models from the schema to Flatfile Sheets
    * **model** (required): `string` - The name of the model in the OpenAPI `components.schemas` section to use for this sheet
    * **name** (optional): `string` - A custom display name for the Sheet. Defaults to the `model` name
    * **slug** (optional): `string` - A custom slug for the Sheet. Defaults to the `model` name
    * **actions** (optional): `Flatfile.Action[]` - An array of actions to add to the sheet
  * **name** (optional): `string` - A custom name for the Workbook. Defaults to the `info.title` property from the OpenAPI schema
  * **actions** (optional): `Flatfile.Action[]` - An array of actions to add to the workbook
* **space** (optional): `object` - A partial `Flatfile.spaces.SpaceConfig` object to apply custom configurations to the Space
* **debug** (optional): `boolean` - A flag for enabling debug mode

### callback (optional)

An optional function that executes after the Space and Workbooks have been successfully configured:

* **event**: `FlatfileEvent` - The event that triggered the configuration
* **workbookIds**: `string[]` - An array of IDs for the newly created workbooks
* **tick**: `TickFunction` - A function to report progress on the configuration job

## Usage Examples

### Basic Usage

This example sets up a listener that configures a Space with a single Workbook and Sheet based on a remote OpenAPI schema.

<CodeGroup>
  ```javascript theme={null}
  import { configureSpaceWithOpenAPI } from "@flatfile/plugin-convert-openapi-schema";

  export default function (listener) {
    listener.on(
      "space:configure",
      configureSpaceWithOpenAPI({
        workbooks: [
          {
            source:
              "https://raw.githubusercontent.com/OAI/OpenAPI-Specification/main/examples/v3.1/webhook-example.json",
            sheets: [{ model: "Pet" }],
          },
        ],
      })
    );
  }
  ```

  ```typescript theme={null}
  import { configureSpaceWithOpenAPI } from "@flatfile/plugin-convert-openapi-schema";
  import { FlatfileListener } from "@flatfile/listener";

  export default function (listener: FlatfileListener) {
    listener.on(
      "space:configure",
      configureSpaceWithOpenAPI({
        workbooks: [
          {
            source:
              "https://raw.githubusercontent.com/OAI/OpenAPI-Specification/main/examples/v3.1/webhook-example.json",
            sheets: [{ model: "Pet" }],
          },
        ],
      })
    );
  }
  ```
</CodeGroup>

### Custom Configuration

This example shows how to customize the names of the Workbook and Sheet, and how to define multiple Sheets from the same source schema.

<CodeGroup>
  ```javascript theme={null}
  import { configureSpaceWithOpenAPI } from "@flatfile/plugin-convert-openapi-schema";

  export default function (listener) {
    listener.on(
      "space:configure",
      configureSpaceWithOpenAPI({
        workbooks: [
          {
            name: "My Custom Workbook",
            source: "https://api.merge.dev/api/accounting/v1/schema",
            sheets: [
              { model: "Account" },
              { model: "Address", name: "Mailing Addresses", slug: "addresses" },
              { model: "Invoice" },
            ],
          },
        ],
      })
    );
  }
  ```

  ```typescript theme={null}
  import { configureSpaceWithOpenAPI } from "@flatfile/plugin-convert-openapi-schema";
  import { FlatfileListener } from "@flatfile/listener";

  export default function (listener: FlatfileListener) {
    listener.on(
      "space:configure",
      configureSpaceWithOpenAPI({
        workbooks: [
          {
            name: "My Custom Workbook",
            source: "https://api.merge.dev/api/accounting/v1/schema",
            sheets: [
              { model: "Account" },
              { model: "Address", name: "Mailing Addresses", slug: "addresses" },
              { model: "Invoice" },
            ],
          },
        ],
      })
    );
  }
  ```
</CodeGroup>

### Advanced Usage with Callback

This example demonstrates adding custom actions to Workbooks and Sheets, and using the optional callback function to perform additional setup tasks after the Space is configured.

<CodeGroup>
  ```javascript theme={null}
  import api from "@flatfile/api";
  import { configureSpaceWithOpenAPI } from "@flatfile/plugin-convert-openapi-schema";

  export default function (listener) {
    const callback = async (event, workbookIds, tick) => {
      const { spaceId } = event.context;
      // Create a getting started document
      await api.documents.create(spaceId, {
        title: "Getting Started",
        body: "<h1>Welcome!</h1><p>Follow the steps to import your data.</p>",
      });
      await tick(100, "Space setup complete");
    };

    listener.use(
      configureSpaceWithOpenAPI(
        {
          workbooks: [
            {
              source: "https://api.merge.dev/api/accounting/v1/schema",
              actions: [
                {
                  operation: "submitActionFg",
                  mode: "foreground",
                  label: "Submit",
                  primary: true,
                },
              ],
              sheets: [{ model: "Account" }, { model: "Invoice" }],
            },
          ],
        },
        callback
      )
    );
  }
  ```

  ```typescript theme={null}
  import api from "@flatfile/api";
  import type { FlatfileListener } from "@flatfile/listener";
  import { configureSpaceWithOpenAPI } from "@flatfile/plugin-convert-openapi-schema";

  export default function (listener: FlatfileListener) {
    const callback = async (event, workbookIds, tick) => {
      const { spaceId } = event.context;
      // Create a getting started document
      await api.documents.create(spaceId, {
        title: "Getting Started",
        body: "<h1>Welcome!</h1><p>Follow the steps to import your data.</p>",
      });
      await tick(100, "Space setup complete");
    };

    listener.use(
      configureSpaceWithOpenAPI(
        {
          workbooks: [
            {
              source: "https://api.merge.dev/api/accounting/v1/schema",
              actions: [
                {
                  operation: "submitActionFg",
                  mode: "foreground",
                  label: "Submit",
                  primary: true,
                },
              ],
              sheets: [{ model: "Account" }, { model: "Invoice" }],
            },
          ],
        },
        callback
      )
    );
  }
  ```
</CodeGroup>

## Troubleshooting

If a sheet fails to generate, check the server logs for the message: `Schema not found for table name [sheet slug]`. This indicates that the `model` name provided in your configuration does not match any model name found in the `components.schemas` section of your OpenAPI file. Ensure the `model` name is spelled correctly and exists in the schema.

Common error scenarios:

* **Network errors**: If the `source` URL cannot be reached or returns a non-200 status code, an error like `Error fetching or processing schema: API returned status 404: Not Found` will be thrown
* **Missing models**: If a model specified in the `sheets` configuration is not found in the schema, an error is logged to the console and that sheet is skipped
* **Invalid schema**: Malformed OpenAPI schemas will cause the configuration job to fail

## Notes

### Default Behavior

* The plugin creates one Workbook for each entry in the `workbooks` array
* Workbook names default to the `info.title` field from the OpenAPI schema
* Sheet names and slugs default to the `model` name
* Fields are generated based on the model's properties, with OpenAPI types (`string`, `number`, `integer`, `boolean`) mapped to corresponding Flatfile field types
* Properties marked as `required` in the schema will have a `required` constraint in Flatfile

### Limitations and Considerations

* This plugin is designed for server-side execution within a Flatfile listener
* Officially supports OpenAPI version 3.0.3 (compatibility with other versions is not guaranteed)
* Depends on `@flatfile/plugin-space-configure` to apply the generated configuration
* OpenAPI `array` types are mapped to Flatfile's `string-list` type by default
* Complex arrays (arrays of objects) are not explicitly handled and may not convert as expected
* Schema references (`$ref`) are resolved only within the same document (e.g., `#/components/schemas/ModelName`)
* External file references are not supported
* Errors are logged to the console and re-thrown, which can aid in debugging within the listener's execution environment


# PDF Extractor Plugin
Source: https://flatfile.com/docs/plugins/pdf-extractor

Extract tabular data from PDF files and convert them to CSV format using the pdftables.com service

The PDF Extractor plugin is designed to process PDF files uploaded to Flatfile. Its main purpose is to extract tabular data from a PDF document and make it available for import. When a user uploads a PDF file, this plugin automatically sends the file to the external `pdftables.com` service, which converts the data into a CSV format. The resulting CSV file is then uploaded back into the same Flatfile Space as a new file, ready to be mapped and imported. This is useful for workflows where source data is provided in PDF reports or documents instead of standard spreadsheet formats.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-pdf-extractor
```

## Configuration & Parameters

The `pdfExtractorPlugin` function accepts a configuration object with the following options:

| Parameter | Type      | Required | Default | Description                                |
| --------- | --------- | -------- | ------- | ------------------------------------------ |
| `apiKey`  | `string`  | Yes      | -       | Your API key from `pdftables.com` service  |
| `debug`   | `boolean` | No       | `false` | Enable verbose logging for troubleshooting |

### Default Behavior

By default, the plugin operates with debugging messages turned off. It will listen for any `.pdf` file uploaded in `import` mode. If a matching file is detected, it will attempt the conversion process. If the required `apiKey` is not provided, the plugin will log an error and fail silently.

## Usage Examples

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    // listener.js
    import { pdfExtractorPlugin } from "@flatfile/plugin-pdf-extractor";

    export default function (listener) {
      // A pdftables.com API key is required
      listener.use(pdfExtractorPlugin({ 
        apiKey: "YOUR_PDFTABLES_API_KEY" 
      }));
    }
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    // listener.ts
    import { FlatfileListener } from "@flatfile/listener";
    import { pdfExtractorPlugin } from "@flatfile/plugin-pdf-extractor";

    export default function (listener: FlatfileListener) {
      // A pdftables.com API key is required
      listener.use(pdfExtractorPlugin({ 
        apiKey: "YOUR_PDFTABLES_API_KEY" 
      }));
    }
    ```
  </Tab>
</Tabs>

### Configuration with Debug Mode

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    // listener.js
    import { pdfExtractorPlugin } from "@flatfile/plugin-pdf-extractor";

    export default function (listener) {
      // Using the plugin with debug option enabled for detailed logging
      listener.use(
        pdfExtractorPlugin({
          apiKey: "YOUR_PDFTABLES_API_KEY",
          debug: true,
        })
      );
    }
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    // listener.ts
    import { FlatfileListener } from "@flatfile/listener";
    import { pdfExtractorPlugin } from "@flatfile/plugin-pdf-extractor";

    export default function (listener: FlatfileListener) {
      // Using the plugin with debug option enabled for detailed logging
      listener.use(
        pdfExtractorPlugin({
          apiKey: "YOUR_PDFTABLES_API_KEY",
          debug: true,
        })
      );
    }
    ```
  </Tab>
</Tabs>

## Troubleshooting

If the plugin is not working properly, follow these troubleshooting steps:

1. **Check API Key**: Ensure that a valid `apiKey` has been provided in the configuration
2. **Enable Debug Mode**: Set `debug: true` in the plugin configuration to get detailed logs in your listener's console output
3. **Network Access**: Ensure the listener environment has network access to `pdftables.com`

### Error Handling

The plugin has built-in error handling and will log errors to the console. Common errors include:

* "Found invalid API key"
* "Failed to convert PDF on pdftables.com"
* "Error writing file to disk"
* "Failed to upload PDF->CSV file"

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    // Enable debug mode to see detailed error messages
    listener.use(
      pdfExtractorPlugin({
        apiKey: "YOUR_PDFTABLES_API_KEY",
        debug: true, // This will log errors from the API call or file upload
      })
    );
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    // Enable debug mode to see detailed error messages
    listener.use(
      pdfExtractorPlugin({
        apiKey: "YOUR_PDFTABLES_API_KEY",
        debug: true, // This will log errors from the API call or file upload
      })
    );
    ```
  </Tab>
</Tabs>

## Notes

### Requirements and Limitations

* **External Dependency**: This plugin requires a subscription and a valid API key from the third-party service `pdftables.com`
* **File Scope**: The plugin only processes files with a `.pdf` extension that are uploaded in `import` mode. It will ignore all other files
* **Environment**: This plugin is intended to be run in a server-side listener environment (e.g., Node.js) as it interacts with the file system

### Processing Details

* **Asynchronous Processing**: The PDF conversion and re-upload process is asynchronous. After a user uploads a PDF, the new converted CSV file will appear in the "Files" list in the Flatfile Space a few moments later
* **File Naming**: The converted file will be named based on the original, with ` (Converted PDF)-{timestamp}.csv` appended to it


# Phone Number Validation and Formatting Plugin
Source: https://flatfile.com/docs/plugins/phone

Validates and formats phone numbers in Flatfile using country-specific validation with libphonenumber-js library

This plugin validates phone numbers in a specified field against a corresponding country code from another field. It leverages the `libphonenumber-js` library for robust, country-specific validation. The primary use case is to ensure that phone number data ingested into Flatfile is correctly formatted and valid. It can automatically correct and reformat phone numbers into various standard formats (e.g., NATIONAL, INTERNATIONAL, E164) or simply add an error to the record if the number is invalid.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-validate-phone
```

## Configuration & Parameters

The plugin accepts a configuration object with the following parameters:

| Parameter       | Type    | Required | Default      | Description                                                                                 |
| --------------- | ------- | -------- | ------------ | ------------------------------------------------------------------------------------------- |
| `phoneField`    | string  | Yes      | -            | The API key (name) of the field containing the phone number to validate                     |
| `countryField`  | string  | Yes      | -            | The API key (name) of the field containing the country code (e.g., 'US', 'GB')              |
| `sheetSlug`     | string  | No       | `'**'`       | The slug of the sheet to apply the validation to. Defaults to all sheets                    |
| `autoConvert`   | boolean | No       | `true`       | If true, automatically updates the phone number field with the correctly formatted version  |
| `format`        | string  | No       | `'NATIONAL'` | The desired output format: 'NATIONAL', 'INTERNATIONAL', 'E164', 'RFC3966', or 'SIGNIFICANT' |
| `concurrency`   | number  | No       | `10`         | Number of records to process concurrently                                                   |
| `debug`         | boolean | No       | `false`      | Enables verbose debug logging in the console                                                |
| `formatOptions` | object  | No       | -            | Additional formatting options for libphonenumber-js library                                 |

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import validatePhone from '@flatfile/plugin-validate-phone';

  export default function (listener) {
    listener.use(validatePhone({
      phoneField: 'phone',
      countryField: 'country',
      autoConvert: false
    }));
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import validatePhone from '@flatfile/plugin-validate-phone';

  export default function (listener: FlatfileListener) {
    listener.use(validatePhone({
      phoneField: 'phone',
      countryField: 'country',
      autoConvert: false
    }));
  }
  ```
</CodeGroup>

This example validates the 'phone' field using the 'country' field in all sheets. It will add errors for invalid numbers but will not automatically reformat them.

### Configuration Example

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import validatePhone from '@flatfile/plugin-validate-phone';

  export default function (listener) {
    listener.use(validatePhone({
      sheetSlug: 'contacts',
      phoneField: 'phone_number',
      countryField: 'country_code',
      autoConvert: true,
      format: 'INTERNATIONAL',
      debug: true
    }));
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import validatePhone from '@flatfile/plugin-validate-phone';

  export default function (listener: FlatfileListener) {
    listener.use(validatePhone({
      sheetSlug: 'contacts',
      phoneField: 'phone_number',
      countryField: 'country_code',
      autoConvert: true,
      format: 'INTERNATIONAL',
      debug: true
    }));
  }
  ```
</CodeGroup>

This example validates and formats phone numbers in the "contacts" sheet using the 'phone\_number' and 'country\_code' fields. Valid numbers are automatically converted to INTERNATIONAL format.

### Advanced Usage with Format Options

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import validatePhone from '@flatfile/plugin-validate-phone';

  export default function (listener) {
    listener.use(validatePhone({
      sheetSlug: 'my-sheet',
      phoneField: 'phone_number',
      countryField: 'country_code',
      format: 'INTERNATIONAL',
      formatOptions: { formatExtension: 'national' }
    }));
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import validatePhone from '@flatfile/plugin-validate-phone';

  export default function (listener: FlatfileListener) {
    listener.use(validatePhone({
      sheetSlug: 'my-sheet',
      phoneField: 'phone_number',
      countryField: 'country_code',
      format: 'INTERNATIONAL',
      formatOptions: { formatExtension: 'national' }
    }));
  }
  ```
</CodeGroup>

This example demonstrates using the formatOptions parameter to provide more specific formatting rules from the underlying libphonenumber-js library.

## Troubleshooting

If validation is not working as expected:

1. **Check field names**: Ensure that the `phoneField` and `countryField` values in the configuration exactly match the field keys in your Sheet template.

2. **Verify country codes**: Ensure the `countryField` in your data contains valid two-letter ISO 3166-1 alpha-2 country codes (e.g., "US", "GB", "DE").

3. **Enable debug logging**: Set the `debug` option to `true` in the configuration to see more detailed logging output in your environment's console, which can help diagnose issues.

## Notes

### Default Behavior

* By default, the plugin applies to all sheets (`sheetSlug: '**'`)
* Phone numbers are automatically converted to the specified format when `autoConvert` is `true` (default)
* The default format is 'NATIONAL'
* Records are processed with a concurrency of 10

### Error Handling

The plugin adds errors to specific fields rather than throwing exceptions:

* **Empty phone number**: "Phone number is required"
* **Empty country field**: "Country is required for phone number formatting"
* **Invalid phone number**: "Invalid phone number format for \[country]"

### Requirements and Limitations

* This plugin has an external dependency on `libphonenumber-js`
* The `countryField` must contain a valid two-letter ISO 3166-1 alpha-2 country code
* The plugin operates as a listener on the `commit:created` event, processing records in batches
* While the documentation mentions specific support for US, UK, India, Germany, France, and Brazil, the underlying library supports a wider range of countries


# Pivot Table Exporter
Source: https://flatfile.com/docs/plugins/pivot-table

A Flatfile plugin that performs data analysis and summarization by generating pivot tables from workbook data and uploading them as Markdown documents.

The Pivot Table Exporter plugin for Flatfile is designed to perform data analysis and summarization directly within the Flatfile environment. It listens for a specific job trigger ('workbook:generatePivotTable'), fetches all records from the first sheet in a workbook, and then generates a pivot table based on user-defined configuration. The resulting pivot table is formatted into a Markdown document and uploaded back to the Flatfile space. This is useful for users who need to quickly aggregate, group, and analyze data during an import process without leaving the Flatfile UI. For example, it can be used to summarize sales data by region and category, or count records based on different attributes.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-export-pivot-table
```

## Configuration & Parameters

The plugin accepts a configuration object with the following parameters:

### Required Parameters

<ParamField type="string">
  The name of the column in your sheet to use as the main pivot (rows in the output table).
</ParamField>

<ParamField type="string">
  The name of the column containing the numerical data to be aggregated.
</ParamField>

<ParamField type="'sum' | 'average' | 'count' | 'min' | 'max'">
  The mathematical operation to perform on the `aggregateColumn`.
</ParamField>

### Optional Parameters

<ParamField type="string">
  An optional column to group the data by. If provided, this will create distinct columns in the output table for each unique value in the `groupByColumn`.
</ParamField>

<ParamField type="boolean">
  If set to true, the plugin will log any caught errors to the console, which is useful for server-side debugging.
</ParamField>

### Default Behavior

By default, the plugin requires `pivotColumn`, `aggregateColumn`, and `aggregationMethod` to be defined. If `groupByColumn` is not provided, the plugin will aggregate data across all records for each unique `pivotColumn` value and present the result in a single "Total" column. The `debug` mode is disabled by default, meaning errors are only reported back to the Flatfile job status without being logged to the console.

## Usage Examples

### Basic Usage

This example shows the simplest way to use the plugin with the required configuration.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { pivotTablePlugin } from '@flatfile/plugin-export-pivot-table';

  export default function(listener) {
    listener.use(
      pivotTablePlugin({
        pivotColumn: 'country',
        aggregateColumn: 'order_value',
        aggregationMethod: 'sum',
      })
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { pivotTablePlugin } from '@flatfile/plugin-export-pivot-table';

  export default function(listener: FlatfileListener) {
    listener.use(
      pivotTablePlugin({
        pivotColumn: 'country',
        aggregateColumn: 'order_value',
        aggregationMethod: 'sum',
      })
    );
  }
  ```
</CodeGroup>

### Configuration with Optional Parameters

This example demonstrates a more complete configuration, including the optional `groupByColumn` and `debug` options.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { pivotTablePlugin } from '@flatfile/plugin-export-pivot-table';

  export default function(listener) {
    listener.use(
      pivotTablePlugin({
        pivotColumn: 'Region',
        aggregateColumn: 'Sales',
        aggregationMethod: 'average',
        groupByColumn: 'Category',
        debug: true,
      })
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { pivotTablePlugin } from '@flatfile/plugin-export-pivot-table';

  export default function(listener: FlatfileListener) {
    listener.use(
      pivotTablePlugin({
        pivotColumn: 'Region',
        aggregateColumn: 'Sales',
        aggregationMethod: 'average',
        groupByColumn: 'Category',
        debug: true,
      })
    );
  }
  ```
</CodeGroup>

### Error Handling with Debug Mode

The plugin has built-in error handling. To see more detailed logs on your server, enable the `debug` flag.

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { pivotTablePlugin } from '@flatfile/plugin-export-pivot-table';

  const listener = new FlatfileListener();

  // Enable debug mode to log errors to the server console
  listener.use(
    pivotTablePlugin({
      pivotColumn: 'non_existent_column', // This will cause an error
      aggregateColumn: 'Sales',
      aggregationMethod: 'sum',
      debug: true,
    })
  );

  export default listener;
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { pivotTablePlugin } from '@flatfile/plugin-export-pivot-table';

  const listener = new FlatfileListener();

  // Enable debug mode to log errors to the server console
  listener.use(
    pivotTablePlugin({
      pivotColumn: 'non_existent_column', // This will cause an error
      aggregateColumn: 'Sales',
      aggregationMethod: 'sum',
      debug: true,
    })
  );

  export default listener;
  ```
</CodeGroup>

## Troubleshooting

If you encounter issues with the pivot table plugin, follow these troubleshooting steps:

1. **Check the job outcome message**: If a job fails, check the "outcome" message on the job in the Flatfile UI for initial error information.

2. **Enable debug mode**: If the error message is not clear, re-run the job with the `debug: true` configuration option enabled and check your server-side logs for a more detailed error stack trace.

3. **Verify column names**: Common errors include providing incorrect column names in the configuration that do not match the columns in the sheet.

4. **Check data types**: Ensure that the `aggregateColumn` contains numeric data. Non-numeric values will be treated as 0.

## Notes

### Requirements and Limitations

* The plugin must be deployed in a server-side listener environment
* It triggers on a specific job: `job: 'workbook:generatePivotTable'`. This job must be initiated from the Flatfile UI or via an API call
* The plugin processes data from the first sheet it finds in the workbook (`sheets.data[0]`). It does not currently support selecting a specific sheet by name or ID
* The values in the `aggregateColumn` are expected to be numbers. Non-numeric values will be treated as 0

### Error Handling

The plugin uses a `try...catch` block to manage errors during its execution:

* On start, it acknowledges the job with `api.jobs.ack`
* On success, it completes the job with `api.jobs.complete` and a success message
* On failure, it fails the job with `api.jobs.fail` and an error message
* If the `debug: true` option is set, the caught error object is also logged to the console


# Record Hook Plugin
Source: https://flatfile.com/docs/plugins/record-hook

Execute custom logic on individual data records within a Flatfile Sheet with support for validation, transformation, enrichment, and cleaning.

The Record Hook plugin provides a way to execute custom logic on individual data records within a Flatfile Sheet. It works by attaching a listener to the `commit:created` event, which fires when new data is added or existing data is updated.

This plugin is ideal for a variety of use cases, including:

* **Data Validation:** Implementing complex validation rules that go beyond Flatfile's built-in validators, such as checking for valid email formats or ensuring conditional field requirements.
* **Data Transformation:** Modifying data on the fly, like capitalizing names, standardizing formats, or setting default values.
* **Data Enrichment:** Augmenting records with additional information from external sources.
* **Data Cleaning:** Removing whitespace, correcting common typos, or normalizing values.

The plugin offers two main functions: `recordHook` for processing records one-by-one, and `bulkRecordHook` for processing records in batches, which is more efficient for large datasets.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-record-hook
```

## Configuration & Parameters

### recordHook

Processes individual records one at a time. The provided callback function is executed for each record in the commit.

**Parameters:**

* `sheetSlug` (string, required): The slug of the Sheet you want the hook to apply to
* `callback` (function, required): A function that contains your custom logic. It receives the record and the event object as arguments
* `options` (object, optional): Configuration options
  * `concurrency` (number, default: 10): Controls how many individual record handlers can run in parallel
  * `debug` (boolean, default: false): When set to `true`, enables verbose logging to the console

### bulkRecordHook

Processes records in batches (chunks). The provided callback function is executed for each chunk of records.

**Parameters:**

* `sheetSlug` (string, required): The slug of the Sheet you want the hook to apply to
* `callback` (function, required): A function that contains your custom logic. It receives an array of records and the event object as arguments
* `options` (object, optional): Configuration options
  * `chunkSize` (number, default: 10000): Specifies the number of records to process in each batch. Maximum recommended value is 5000
  * `parallel` (number, default: 1): Specifies how many chunks of records to process in parallel
  * `debug` (boolean, default: false): When set to `true`, enables verbose logging to the console

**Default Behavior:**
By default, the plugin processes records for the specified `sheetSlug` after a commit is created. `bulkRecordHook` processes all records in a single chunk sequentially (`parallel: 1`). `recordHook` processes up to 10 records concurrently. Debug logging is disabled.

## Usage Examples

### Basic Single Record Processing

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { recordHook } from "@flatfile/plugin-record-hook";
  import { FlatfileListener } from "@flatfile/listener";

  export default function (listener) {
    listener.use(
      recordHook("contacts", (record) => {
        const firstName = record.get("firstName");
        if (firstName) {
          record.set("firstName", firstName.charAt(0).toUpperCase() + firstName.slice(1));
        }
        return record;
      })
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import { recordHook } from "@flatfile/plugin-record-hook";
  import { FlatfileListener } from "@flatfile/listener";

  export default function (listener: FlatfileListener) {
    listener.use(
      recordHook("contacts", (record) => {
        const firstName = record.get("firstName") as string;
        if (firstName) {
          record.set("firstName", firstName.charAt(0).toUpperCase() + firstName.slice(1));
        }
        return record;
      })
    );
  }
  ```
</CodeGroup>

### Basic Bulk Record Processing

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { bulkRecordHook } from "@flatfile/plugin-record-hook";
  import { FlatfileListener } from "@flatfile/listener";

  export default function (listener) {
    listener.use(
      bulkRecordHook("contacts", (records) => {
        records.forEach((record) => {
          const firstName = record.get("firstName");
          if (firstName) {
            record.set("firstName", firstName.charAt(0).toUpperCase() + firstName.slice(1));
          }
        });
        return records;
      })
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import { bulkRecordHook } from "@flatfile/plugin-record-hook";
  import { FlatfileListener } from "@flatfile/listener";
  import { FlatfileRecord } from "@flatfile/hooks";

  export default function (listener: FlatfileListener) {
    listener.use(
      bulkRecordHook("contacts", (records: FlatfileRecord[]) => {
        records.forEach((record) => {
          const firstName = record.get("firstName") as string;
          if (firstName) {
            record.set("firstName", firstName.charAt(0).toUpperCase() + firstName.slice(1));
          }
        });
        return records;
      })
    );
  }
  ```
</CodeGroup>

### Configuration Example

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { bulkRecordHook } from "@flatfile/plugin-record-hook";
  import { FlatfileListener } from "@flatfile/listener";

  export default function (listener) {
    listener.use(
      bulkRecordHook(
        "contacts",
        (records) => {
          // Your processing logic here
          return records;
        },
        { chunkSize: 1000, parallel: 5, debug: true }
      )
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import { bulkRecordHook } from "@flatfile/plugin-record-hook";
  import { FlatfileListener } from "@flatfile/listener";
  import { FlatfileRecord } from "@flatfile/hooks";

  export default function (listener: FlatfileListener) {
    listener.use(
      bulkRecordHook(
        "contacts",
        (records: FlatfileRecord[]) => {
          // Your processing logic here
          return records;
        },
        { chunkSize: 1000, parallel: 5, debug: true }
      )
    );
  }
  ```
</CodeGroup>

### Advanced Usage - Email Validation

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { recordHook } from "@flatfile/plugin-record-hook";
  import { FlatfileListener } from "@flatfile/listener";

  export default function (listener) {
    listener.use(
      recordHook("contacts", (record) => {
        const email = record.get("email");
        if (!email) {
          record.addError("email", "Email address is required.");
          return record;
        }

        const validEmailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
        if (!validEmailRegex.test(email)) {
          record.addError("email", "Please enter a valid email address.");
        }
        return record;
      })
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import { recordHook } from "@flatfile/plugin-record-hook";
  import { FlatfileListener } from "@flatfile/listener";

  export default function (listener: FlatfileListener) {
    listener.use(
      recordHook("contacts", (record) => {
        const email = record.get("email") as string;
        if (!email) {
          record.addError("email", "Email address is required.");
          return record;
        }

        const validEmailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
        if (!validEmailRegex.test(email)) {
          record.addError("email", "Please enter a valid email address.");
        }
        return record;
      })
    );
  }
  ```
</CodeGroup>

## Troubleshooting

The most effective way to troubleshoot issues is to set the `debug: true` option in the plugin configuration. This will print detailed logs to the console, showing the timing and status of each major step: fetching data, running the handler, filtering modified records, and updating records. This can help identify performance bottlenecks or see why records are not being updated as expected.

## Notes

### Special Considerations

* The plugin operates on the `commit:created` event. This means it runs after Flatfile's initial parsing and validation but before the data is finalized.
* To save changes, your callback function **must** return the modified record (for `recordHook`) or the array of modified records (for `bulkRecordHook`). If you return `null`, `undefined`, or nothing, your changes will not be persisted.
* The plugin intelligently detects which records have actually been modified and only sends those records back to the Flatfile API for an update, optimizing performance.

### Limitations

* The `chunkSize` for `bulkRecordHook` has a recommended maximum of 5000 records to ensure stability and performance.

### Error Handling

* The plugin is designed to be robust. It wraps the execution of the user-provided callback in a `try...catch` block.
* If an error is thrown inside your callback, the plugin will log the error message to the console with a `[FATAL]` prefix.
* Crucially, it will then ensure the commit is marked as complete with the Flatfile API, preventing the import process from stalling due to an error in custom code.


# Rollout: Automatic Workbook Updates
Source: https://flatfile.com/docs/plugins/rollout

Automatically apply schema changes to existing, live workbooks when a new version of a Flatfile Agent is deployed

The Rollout plugin automates the process of applying schema changes to existing, live workbooks. When a new version of a Flatfile Agent is deployed (triggering `agent:created` or `agent:updated` events), this plugin identifies relevant workbooks and initiates an update process.

Its primary use case is to ensure that all workbooks associated with a particular configuration stay up-to-date with the latest schema definitions without manual intervention. It works by creating a job to apply the new schema via a user-defined `updater` function. After the schema is updated, it cleverly re-triggers all data hooks on all records in the updated sheets, ensuring data validity and transformations are re-evaluated against the new schema.

This plugin should be deployed in a server-side listener.

## Installation

```bash theme={null}
npm install @flatfile/plugin-rollout
```

## Configuration & Parameters

The rollout plugin accepts a configuration object with the following parameters:

### `namespace` (required)

* **Type:** `string`
* **Description:** A string used to filter which spaces the plugin should operate on. The plugin will only consider spaces matching this namespace. The expected format is typically `workbook:your-namespace`.

### `dev` (required)

* **Type:** `boolean`
* **Description:** When set to `true`, the plugin will also trigger an update check when the agent starts up in a local development environment (i.e., not running on AWS Lambda). This is useful for testing changes locally without needing to deploy a new agent version.
* **Default:** Effectively `false` - updates in local development are suppressed unless this is explicitly set to `true`.

### `updater` (required)

* **Type:** `(space: Flatfile.Space, workbooks: Flatfile.Workbook[]) => Promise<undefined | Flatfile.Workbook[]>`
* **Description:** An asynchronous callback function you provide to perform the actual schema updates. It receives the `space` being processed and a list of its `workbooks`. You are responsible for using the Flatfile API within this function to apply your new schema to the workbooks. The function should return an array of the workbooks that were successfully updated.

## Usage Examples

<CodeGroup>
  ```javascript JavaScript - Basic Usage theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { rollout } from '@flatfile/plugin-rollout'
  import { FlatfileClient } from '@flatfile/api'

  const listener = new FlatfileListener()
  const api = new FlatfileClient()

  const rolloutPlugin = rollout({
    namespace: 'workbook:my-app',
    dev: process.env.NODE_ENV === 'development',
    updater: async (space, workbooks) => {
      // In a real scenario, you would update the workbook's sheets
      // with a new schema configuration here.
      console.log(`Updating workbooks in space ${space.id}`)
      // For example:
      // const newSchema = require('./schemas/my-new-schema')
      // await api.sheets.update(workbooks[0].sheets[0].id, { config: newSchema.config })
      
      // Return the workbooks you updated to re-trigger hooks.
      return workbooks
    },
  })

  // Register the job handler on a filtered listener
  listener.namespace(['workbook:my-app'], (filteredListener) => {
    filteredListener.use(rolloutPlugin)
  })

  // Register the root event handler on the main listener
  // This is required to catch the agent deployment events.
  listener.use(rolloutPlugin.root)
  ```

  ```typescript TypeScript - Basic Usage theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { rollout } from '@flatfile/plugin-rollout'
  import { FlatfileClient } from '@flatfile/api'
  import { Flatfile } from '@flatfile/api'

  const listener = new FlatfileListener()
  const api = new FlatfileClient()

  const rolloutPlugin = rollout({
    namespace: 'workbook:my-app',
    dev: process.env.NODE_ENV === 'development',
    updater: async (space: Flatfile.Space, workbooks: Flatfile.Workbook[]) => {
      // In a real scenario, you would update the workbook's sheets
      // with a new schema configuration here.
      console.log(`Updating workbooks in space ${space.id}`)
      // For example:
      // const newSchema = require('./schemas/my-new-schema')
      // await api.sheets.update(workbooks[0].sheets[0].id, { config: newSchema.config })
      
      // Return the workbooks you updated to re-trigger hooks.
      return workbooks
    },
  })

  // Register the job handler on a filtered listener
  listener.namespace(['workbook:my-app'], (filteredListener) => {
    filteredListener.use(rolloutPlugin)
  })

  // Register the root event handler on the main listener
  // This is required to catch the agent deployment events.
  listener.use(rolloutPlugin.root)
  ```
</CodeGroup>

<CodeGroup>
  ```javascript JavaScript - Advanced Configuration theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { rollout } from '@flatfile/plugin-rollout'
  import { FlatfileClient } from '@flatfile/api'
  import { myNewSheetConfig } from './new-sheet.config'

  const listener = new FlatfileListener()
  const api = new FlatfileClient()

  const rolloutPluginWithConfig = rollout({
    // Only act on spaces with this namespace
    namespace: 'workbook:customer-onboarding',
    
    // Enable local dev updates
    dev: true,
    
    // Provide the logic to update the workbook schemas
    updater: async (space, workbooks) => {
      const updatedWorkbooks = []
      for (const workbook of workbooks) {
        // Assuming we want to update the first sheet of each workbook
        if (workbook.sheets && workbook.sheets.length > 0) {
          const sheetId = workbook.sheets[0].id
          try {
            await api.sheets.update(sheetId, {
              config: myNewSheetConfig,
            })
            console.log(`Successfully updated schema for sheet ${sheetId}`)
            updatedWorkbooks.push(workbook)
          } catch (e) {
            console.error(`Failed to update sheet ${sheetId}:`, e)
          }
        }
      }
      // Return only the workbooks that were successfully updated
      return updatedWorkbooks
    },
  })

  // The plugin returns two parts that must both be registered.
  listener.namespace(['workbook:customer-onboarding'], (filteredListener) => {
    filteredListener.use(rolloutPluginWithConfig)
  })
  listener.use(rolloutPluginWithConfig.root)
  ```

  ```typescript TypeScript - Advanced Configuration theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { rollout } from '@flatfile/plugin-rollout'
  import { FlatfileClient } from '@flatfile/api'
  import { Flatfile } from '@flatfile/api'
  import { myNewSheetConfig } from './new-sheet.config'

  const listener = new FlatfileListener()
  const api = new FlatfileClient()

  const rolloutPluginWithConfig = rollout({
    // Only act on spaces with this namespace
    namespace: 'workbook:customer-onboarding',
    
    // Enable local dev updates
    dev: true,
    
    // Provide the logic to update the workbook schemas
    updater: async (space: Flatfile.Space, workbooks: Flatfile.Workbook[]) => {
      const updatedWorkbooks: Flatfile.Workbook[] = []
      for (const workbook of workbooks) {
        // Assuming we want to update the first sheet of each workbook
        if (workbook.sheets && workbook.sheets.length > 0) {
          const sheetId = workbook.sheets[0].id
          try {
            await api.sheets.update(sheetId, {
              config: myNewSheetConfig,
            })
            console.log(`Successfully updated schema for sheet ${sheetId}`)
            updatedWorkbooks.push(workbook)
          } catch (e) {
            console.error(`Failed to update sheet ${sheetId}:`, e)
          }
        }
      }
      // Return only the workbooks that were successfully updated
      return updatedWorkbooks
    },
  })

  // The plugin returns two parts that must both be registered.
  listener.namespace(['workbook:customer-onboarding'], (filteredListener) => {
    filteredListener.use(rolloutPluginWithConfig)
  })
  listener.use(rolloutPluginWithConfig.root)
  ```
</CodeGroup>

## API Reference

### `rollout(config)`

Initializes the rollout plugin. It sets up listeners for agent deployment events and a job handler to perform the updates. It relies on a user-provided `updater` function to apply the specific schema changes.

**Parameters:**

* `config` (object): Configuration object with the following properties:
  * `namespace` (string, required): The namespace to filter spaces for updates
  * `dev` (boolean, required): Set to `true` to enable updates on local agent reloads
  * `updater` (function, required): Async function that performs the schema update

**Return Value:**
A listener callback function with a `.root` property (also a listener callback). The main callback handles the `space:auto-update` job, while the `.root` callback handles `agent:created` and `agent:updated` events.

<CodeGroup>
  ```javascript JavaScript - Error Handling theme={null}
  const rolloutWithErrorHandling = rollout({
    namespace: 'workbook:my-app',
    dev: true,
    updater: async (space, workbooks) => {
      const updated = []
      for (const wb of workbooks) {
        try {
          // Attempt to update a sheet
          const sheetId = wb.sheets[0].id
          // ... your api call to update the sheet ...
          console.log(`Updated sheet ${sheetId}`)
          updated.push(wb)
        } catch (error) {
          console.error(`Could not update workbook ${wb.id}. Reason:`, error)
          // Continue to the next workbook without stopping the whole process
        }
      }
      return updated
    },
  })
  ```

  ```typescript TypeScript - Error Handling theme={null}
  const rolloutWithErrorHandling = rollout({
    namespace: 'workbook:my-app',
    dev: true,
    updater: async (space: Flatfile.Space, workbooks: Flatfile.Workbook[]) => {
      const updated: Flatfile.Workbook[] = []
      for (const wb of workbooks) {
        try {
          // Attempt to update a sheet
          const sheetId = wb.sheets[0].id
          // ... your api call to update the sheet ...
          console.log(`Updated sheet ${sheetId}`)
          updated.push(wb)
        } catch (error) {
          console.error(`Could not update workbook ${wb.id}. Reason:`, error)
          // Continue to the next workbook without stopping the whole process
        }
      }
      return updated
    },
  })
  ```
</CodeGroup>

## Troubleshooting

### Updates not triggering

* Ensure the `agent:created` or `agent:updated` events are firing upon deployment
* Check that the target space has the correct namespace and the required secret (`FF_AUTO_UPDATE` or `FF_AUTO_UPDATE_DEV`) is set to `'true'`

### Local updates not working

* Make sure you have set `dev: true` in the plugin configuration

### Hooks not re-running

* Verify that your `updater` function is returning an array containing the workbooks that you successfully updated
* If an empty or `undefined` value is returned, the hook re-triggering step will be skipped

## Notes

### Default Behavior

By default, the plugin will only trigger updates for production spaces that have a secret named `FF_AUTO_UPDATE` with a value of `'true'`. If the `dev: true` option is used, it will trigger updates for development spaces that have a secret named `FF_AUTO_UPDATE_DEV` with a value of `'true'`. This secret-based mechanism acts as an explicit opt-in for each space, preventing accidental updates.

### Special Considerations

* **Dual Listener Registration:** The plugin returns a function with a `.root` property. You MUST register both. The main function handles the job execution and should be on a namespaced listener. The `.root` function handles the global events that trigger the job and must be on the root listener.
* **Server-Side Only:** This plugin is designed to run in a server-side listener environment, not in the browser.
* **Opt-In via Secrets:** The plugin will not update a space unless it contains a specific secret. For production environments, a secret named `FF_AUTO_UPDATE` must exist with the value `'true'`. For local development (with `dev: true`), the secret must be `FF_AUTO_UPDATE_DEV` with the value `'true'`.
* **Hook Re-triggering:** To re-run data hooks after a schema change, the plugin updates the metadata of every record in the updated sheets by adding a `_autoUpdateKey` with a random UUID. This modification forces Flatfile to re-evaluate each record.

### API Permissions

The agent using this plugin requires API permissions for the following actions:

* `space:read` (for `spaces.list`, `spaces.get`)
* `secret:read` (for `secrets.list`)
* `workbook:read` (for `workbooks.list`)
* `job:write` (for `jobs.create`)
* Your `updater` function will likely require additional permissions, such as `sheet:write` to update sheet configurations.


# Sentiment Analysis Plugin
Source: https://flatfile.com/docs/plugins/sentiment

Automatically analyze sentiment in text fields and add sentiment scores and categories to your Flatfile records

This plugin analyzes the sentiment of text within specified fields of records in a Flatfile Sheet. It operates during the data processing phase by hooking into the record commit lifecycle. For each designated text field, the plugin calculates a sentiment score, categorizes it as 'positive', 'negative', or 'neutral', and adds this information back to the record in two new fields: `<field_name>_sentiment_score` and `<field_name>_sentiment_category`. It also adds informational messages to the record detailing the outcome of the analysis. This is useful for automatically classifying customer feedback, product reviews, support tickets, or any other free-text data to quickly gauge user sentiment.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-enrich-sentiment
```

## Configuration & Parameters

The plugin accepts a configuration object with the following parameters:

### `sheetSlug` (required)

* **Type:** `string`
* **Description:** The slug of the sheet that this plugin should run on.

### `textFields`

* **Type:** `string[]`
* **Description:** An array of field keys (column names) that contain the text to be analyzed.
* **Default:** `['description']`

### `automaticValidation`

* **Type:** `boolean`
* **Description:** A flag to enable or disable the sentiment analysis. If set to `false`, the plugin will add an info message to each record indicating that analysis is disabled but will not perform any analysis.
* **Default:** `false` - The analysis only runs if this is explicitly set to `true`.

### `errorMessages`

* **Type:** `object`
* **Description:** A configuration object for custom error messages. Note: This option is defined in the type interface but is not used in the current plugin implementation.

## Usage Examples

### Basic Usage

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    import { FlatfileListener } from '@flatfile/listener';
    import { enrichSentiment } from '@flatfile/plugin-enrich-sentiment';

    export default function(listener) {
      // Analyzes the 'description' field in the 'contacts' sheet.
      listener.use(enrichSentiment({
        sheetSlug: 'contacts',
        textFields: ['description'],
        automaticValidation: true
      }));
    }
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import { FlatfileListener } from '@flatfile/listener';
    import { enrichSentiment } from '@flatfile/plugin-enrich-sentiment';

    export default function(listener: FlatfileListener) {
      // Analyzes the 'description' field in the 'contacts' sheet.
      listener.use(enrichSentiment({
        sheetSlug: 'contacts',
        textFields: ['description'],
        automaticValidation: true
      }));
    }
    ```
  </Tab>
</Tabs>

### Multiple Text Fields

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    import { FlatfileListener } from '@flatfile/listener';
    import { enrichSentiment } from '@flatfile/plugin-enrich-sentiment';

    export default function(listener) {
      // A more detailed configuration for a customer feedback sheet.
      // Analyzes both 'comment' and 'feedback' fields.
      listener.use(enrichSentiment({
        sheetSlug: 'customer-feedback',
        textFields: ['comment', 'feedback'],
        automaticValidation: true
      }));
    }
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import { FlatfileListener } from '@flatfile/listener';
    import { enrichSentiment } from '@flatfile/plugin-enrich-sentiment';

    export default function(listener: FlatfileListener) {
      // A more detailed configuration for a customer feedback sheet.
      // Analyzes both 'comment' and 'feedback' fields.
      listener.use(enrichSentiment({
        sheetSlug: 'customer-feedback',
        textFields: ['comment', 'feedback'],
        automaticValidation: true
      }));
    }
    ```
  </Tab>
</Tabs>

### Advanced Custom Usage

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    // This example shows how to use the exported helper functions
    // inside a custom recordHook for more fine-grained control.
    import { FlatfileListener } from '@flatfile/listener';
    import { recordHook } from '@flatfile/plugin-record-hook';
    import { performEnrichSentiment } from '@flatfile/plugin-enrich-sentiment';

    export default function(listener) {
      listener.use(recordHook('reviews', async (record) => {
        const reviewText = String(record.get('review_text'));

        // Use the exported helper function directly
        const { error, result } = performEnrichSentiment(reviewText, 'review_text');

        if (error) {
          record.addWarning('review_text', error);
        } else if (result) {
          // Custom logic: only add a 'positive_review' flag
          if (result.category === 'positive' && result.score > 3) {
            record.set('positive_review', true);
            record.addInfo('review_text', 'This is a highly positive review!');
          }
        }
        return record;
      }));
    }
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    // This example shows how to use the exported helper functions
    // inside a custom recordHook for more fine-grained control.
    import { FlatfileListener } from '@flatfile/listener';
    import { recordHook } from '@flatfile/plugin-record-hook';
    import { performEnrichSentiment } from '@flatfile/plugin-enrich-sentiment';

    export default function(listener: FlatfileListener) {
      listener.use(recordHook('reviews', async (record) => {
        const reviewText = String(record.get('review_text'));

        // Use the exported helper function directly
        const { error, result } = performEnrichSentiment(reviewText, 'review_text');

        if (error) {
          record.addWarning('review_text', error);
        } else if (result) {
          // Custom logic: only add a 'positive_review' flag
          if (result.category === 'positive' && result.score > 3) {
            record.set('positive_review', true);
            record.addInfo('review_text', 'This is a highly positive review!');
          }
        }
        return record;
      }));
    }
    ```
  </Tab>
</Tabs>

### Using Utility Functions

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    import { analyzeSentiment } from '@flatfile/plugin-enrich-sentiment';

    const analysis = analyzeSentiment("Flatfile is an amazing tool!");
    // analysis -> { score: 6, category: 'positive' }
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import { analyzeSentiment } from '@flatfile/plugin-enrich-sentiment';

    const analysis = analyzeSentiment("Flatfile is an amazing tool!");
    // analysis -> { score: 6, category: 'positive' }
    ```
  </Tab>
</Tabs>

## API Reference

### `enrichSentiment(config: EnrichSentimentConfig)`

The main entry point for the plugin. It creates and registers a `recordHook` with Flatfile that performs sentiment analysis on incoming records based on the provided configuration.

**Parameters:**

* `config` (EnrichSentimentConfig): An object containing configuration options
  * `sheetSlug` (string): The slug of the sheet to target
  * `textFields` (string\[]): An array of field keys to analyze. Defaults to `['description']`
  * `automaticValidation` (boolean): Must be `true` to enable analysis

**Returns:** A function of type `(listener: FlatfileListener) => void` which is used to install the plugin into a Flatfile listener.

### `analyzeSentiment(text: string)`

A pure function that takes a string of text and returns its sentiment analysis.

**Parameters:**

* `text` (string): The input text to analyze

**Returns:** An object with the following properties:

* `score` (number): A numerical score representing the sentiment. Positive values are positive, negative values are negative
* `category` ('positive' | 'negative' | 'neutral'): A string category for the sentiment

### `performEnrichSentiment(value: string, field: string)`

A wrapper around `analyzeSentiment` that includes basic validation and formats the output for use within a record hook. It checks for empty input and structures the return value with either an error or a result object.

**Parameters:**

* `value` (string): The text value from the record field
* `field` (string): The name of the field being analyzed, used for creating informative messages

**Returns:** An object with one of two shapes:

* On success: `{ error: null, result: { score: number, category: string, message: string } }`
* On error (empty input): `{ error: string, result: null }`

## Troubleshooting

### Empty Text Fields

If a text field specified in the `textFields` configuration is empty or null, the plugin will not throw an error. Instead, it will add a warning to the record using `record.addWarning()` with a message like "No text found for sentiment analysis in field: \[field\_name]".

### Analysis Not Running

If `automaticValidation` is set to `false` or is omitted, the plugin will not perform any analysis or add any warnings. It will only add an informational message to the record stating that automatic analysis is disabled.

## Notes

### Default Behavior

* The plugin is **disabled by default**. You must explicitly set `automaticValidation: true` to enable sentiment analysis.
* If no `textFields` are specified, the plugin will default to analyzing the `description` field.

### Generated Fields

The plugin adds two new fields to each processed record for each analyzed text field:

* `<field_name>_sentiment_score`: Contains the numerical sentiment score
* `<field_name>_sentiment_category`: Contains the sentiment category ('positive', 'negative', or 'neutral')

Ensure your Sheet configuration or downstream systems can handle these new fields.

### Configuration Considerations

* The configuration property to enable the plugin is `automaticValidation` (boolean), not `autoAnalysis`.
* The `errorMessages` property in the `EnrichSentimentConfig` type is defined but not currently implemented in the plugin's logic.
* The plugin relies on the `sentiment` npm package for its analysis logic.


# Flatfile Space Configuration Plugin
Source: https://flatfile.com/docs/plugins/space-configure

Programmatically set up and configure a new Flatfile Space with workbooks, sheets, documents, and metadata from a single configuration object.

The `@flatfile/plugin-space-configure` plugin is designed to programmatically set up and configure a new Flatfile Space. It operates within a server-side listener, typically responding to the 'space:configure' event which is triggered when a new Space is created.

Its primary purpose is to define the entire structure of a Space from a single configuration object. This includes creating one or more Workbooks, defining their Sheets with specific fields and actions, setting Space-level properties like metadata and themes, and adding initial documents such as a welcome guide.

The plugin also includes a secondary utility, `dataChecklistPlugin`, which can automatically generate and maintain a "Data Checklist" document within the Space. This document provides a summary of all the fields and data types defined in the Space's workbooks, serving as a handy reference for users.

This plugin is essential for developers who want to create templatized, repeatable, or dynamically generated Space configurations for their users.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-space-configure
```

## Configuration & Parameters

The `configureSpace` function takes a `setup` object as its primary configuration. This can be a static object or a function that returns an object.

The `setup` object has the following properties:

### workbooks

* **Type:** `Partial<Flatfile.CreateWorkbookConfig>[]`
* **Required:** Yes
* **Description:** An array of workbook configuration objects. Each object defines a workbook to be created in the Space. You can specify its name, sheets, actions, labels, etc. This is the core of the Space's data structure.
* **Default:** There is no default; you must provide at least an empty array `[]`.

### space

* **Type:** `Partial<Flatfile.SpaceConfig>`
* **Required:** No
* **Description:** An object to configure the Space itself. You can set metadata (like themes), the primary workbook ID (though the plugin handles this automatically), and other space-level settings.
* **Default:** The plugin will automatically set the `primaryWorkbookId` to the ID of the first workbook created. Other properties are unset by default.

### documents

* **Type:** `Flatfile.DocumentConfig[]`
* **Required:** No
* **Description:** An array of document configuration objects. Each object creates a document in the Space's sidebar. This is useful for providing welcome text, instructions, or guides.
* **Default:** No documents are created by default.

### config

* **Type:** `object`
* **Required:** No
* **Description:** An object for plugin-specific configurations.
  * `maintainWorkbookOrder` (boolean): If set to `true`, the plugin will configure the Space's sidebar to display the workbooks in the same order they are defined in the `workbooks` array.
* **Default:** `{ maintainWorkbookOrder: false }`

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { configureSpace } from '@flatfile/plugin-space-configure'

  export default function (listener) {
    listener.use(
      configureSpace({
        workbooks: [],
        space: {
          metadata: {
            name: 'My Empty Space',
          },
        },
      })
    )
  }
  ```

  ```typescript TypeScript theme={null}
  import type { FlatfileListener } from '@flatfile/listener'
  import { configureSpace } from '@flatfile/plugin-space-configure'

  export default function (listener: FlatfileListener) {
    listener.use(
      configureSpace({
        workbooks: [],
        space: {
          metadata: {
            name: 'My Empty Space',
          },
        },
      })
    )
  }
  ```
</CodeGroup>

### Configuration with Workbook and Sheets

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { configureSpace } from '@flatfile/plugin-space-configure'

  export default function (listener) {
    listener.use(
      configureSpace({
        workbooks: [
          {
            name: 'My First Workbook',
            sheets: [
              {
                name: 'Contacts',
                slug: 'contacts',
                fields: [
                  { key: 'firstName', type: 'string', label: 'First Name' },
                  { key: 'lastName', type: 'string', label: 'Last Name' },
                  { key: 'email', type: 'string', label: 'Email' },
                ],
              },
            ],
          },
        ],
        space: {
          metadata: {
            theme: {
              root: { primaryColor: 'blue' },
            },
          },
        },
      })
    )
  }
  ```

  ```typescript TypeScript theme={null}
  import type { FlatfileListener } from '@flatfile/listener'
  import { configureSpace } from '@flatfile/plugin-space-configure'

  export default function (listener: FlatfileListener) {
    listener.use(
      configureSpace({
        workbooks: [
          {
            name: 'My First Workbook',
            sheets: [
              {
                name: 'Contacts',
                slug: 'contacts',
                fields: [
                  { key: 'firstName', type: 'string', label: 'First Name' },
                  { key: 'lastName', type: 'string', label: 'Last Name' },
                  { key: 'email', type: 'string', label: 'Email' },
                ],
              },
            ],
          },
        ],
        space: {
          metadata: {
            theme: {
              root: { primaryColor: 'blue' },
            },
          },
        },
      })
    )
  }
  ```
</CodeGroup>

### Advanced Usage with Callback

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { configureSpace } from '@flatfile/plugin-space-configure'
  import api from '@flatfile/api'

  export default function (listener) {
    listener.use(
      configureSpace(
        {
          workbooks: [
            {
              name: 'Onboarding Workbook',
              sheets: [{ name: 'Contacts', fields: [{ key: 'email', type: 'string' }] }],
            },
          ],
          documents: [
            {
              title: 'Welcome Guide',
              body: '<h1>Welcome!</h1><p>Follow the steps to get started.</p>',
            },
          ],
        },
        async (event, workbookIds, tick) => {
          // This code runs after the Space and Workbooks are created.
          const { spaceId } = event.context
          const workbookId = workbookIds[0]

          await tick(60, 'Callback started')
          console.log(`Space ${spaceId} and Workbook ${workbookId} are ready.`)

          // You can now perform additional API calls, like adding records.
          await api.records.insert(workbookId, [
            { email: { value: 'john.doe@example.com' } },
          ])

          await tick(100, 'Callback complete')
        }
      )
    )
  }
  ```

  ```typescript TypeScript theme={null}
  import type { FlatfileListener } from '@flatfile/listener'
  import { configureSpace } from '@flatfile/plugin-space-configure'
  import api from '@flatfile/api'

  export default function (listener: FlatfileListener) {
    listener.use(
      configureSpace(
        {
          workbooks: [
            {
              name: 'Onboarding Workbook',
              sheets: [{ name: 'Contacts', fields: [{ key: 'email', type: 'string' }] }],
            },
          ],
          documents: [
            {
              title: 'Welcome Guide',
              body: '<h1>Welcome!</h1><p>Follow the steps to get started.</p>',
            },
          ],
        },
        async (event, workbookIds, tick) => {
          // This code runs after the Space and Workbooks are created.
          const { spaceId } = event.context
          const workbookId = workbookIds[0]

          await tick(60, 'Callback started')
          console.log(`Space ${spaceId} and Workbook ${workbookId} are ready.`)

          // You can now perform additional API calls, like adding records.
          await api.records.insert(workbookId, [
            { email: { value: 'john.doe@example.com' } },
          ])

          await tick(100, 'Callback complete')
        }
      )
    )
  }
  ```
</CodeGroup>

### Using Data Checklist Plugin

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { configureSpace, dataChecklistPlugin } from '@flatfile/plugin-space-configure'

  export default function (listener) {
    // Use configureSpace to set up the initial structure
    listener.use(
      configureSpace({
        workbooks: [{ name: 'Contacts', sheets: [/* ... */] }],
      })
    )

    // Use dataChecklistPlugin to generate a summary document
    // This will run after the workbook is created by configureSpace
    listener.use(dataChecklistPlugin())
  }
  ```

  ```typescript TypeScript theme={null}
  import type { FlatfileListener } from '@flatfile/listener'
  import { configureSpace, dataChecklistPlugin } from '@flatfile/plugin-space-configure'

  export default function (listener: FlatfileListener) {
    // Use configureSpace to set up the initial structure
    listener.use(
      configureSpace({
        workbooks: [{ name: 'Contacts', sheets: [/* ... */] }],
      })
    )

    // Use dataChecklistPlugin to generate a summary document
    // This will run after the workbook is created by configureSpace
    listener.use(dataChecklistPlugin())
  }
  ```
</CodeGroup>

### Error Handling Example

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { configureSpace } from '@flatfile/plugin-space-configure'

  export default function (listener) {
    listener.use(
      configureSpace(
        { workbooks: [{ name: 'My Workbook' }] },
        async (event, workbookIds, tick) => {
          try {
            await tick(75, 'Running custom logic');
            // Simulate a failing operation
            throw new Error('Custom API call failed!');
          } catch (e) {
            console.error('Error in callback:', e.message);
            // Rethrow the error to fail the job
            throw e;
          }
        }
      )
    )
  }
  ```

  ```typescript TypeScript theme={null}
  import type { FlatfileListener } from '@flatfile/listener'
  import { configureSpace } from '@flatfile/plugin-space-configure'

  export default function (listener: FlatfileListener) {
    listener.use(
      configureSpace(
        { workbooks: [{ name: 'My Workbook' }] },
        async (event, workbookIds, tick) => {
          try {
            await tick(75, 'Running custom logic');
            // Simulate a failing operation
            throw new Error('Custom API call failed!');
          } catch (e) {
            console.error('Error in callback:', e.message);
            // Rethrow the error to fail the job
            throw e;
          }
        }
      )
    )
  }
  ```
</CodeGroup>

## API Reference

### configureSpace(setupFactory, callback)

Creates a Flatfile listener plugin that listens for the `job:ready` event with the topic `space:configure`. When triggered, it configures the Space according to the provided setup. It handles creating workbooks, updating the space with a primary workbook, and creating documents.

**Parameters:**

1. `setupFactory`: `Setup | (event: FlatfileEvent) => Setup | Promise<Setup>`
   * The configuration for the space. This can be a static object or an async function that receives the event context and returns a configuration object.

2. `callback`: `(event: FlatfileEvent, workbookIds: string[], tick: TickFunction) => any | Promise<any>` (Optional)
   * An optional async function that is executed after the space and workbooks have been successfully configured. The job progress will be at 50% when the callback is invoked.
   * `event`: The original FlatfileEvent that triggered the job.
   * `workbookIds`: An array of strings containing the IDs of the workbooks that were created.
   * `tick`: A function to update the job's progress percentage and message.

**Returns:** `(listener: FlatfileListener) => void`

### dataChecklistPlugin()

A utility plugin that creates and maintains a "Data Checklist" document in a Space. It listens for `workbook:created` and `workbook:updated` events, then inspects all workbooks and sheets to generate an HTML document summarizing the data model.

**Parameters:** None

**Returns:** `(listener: FlatfileListener) => void`

## Troubleshooting

If a Space fails to configure, check the "Jobs" log in the Flatfile Dashboard for the specific Space. The `space:configure` job will show an error message detailing what went wrong.

Common issues include:

* Malformed configuration objects (e.g., incorrect field types in a sheet definition)
* API permission errors - ensure your agent has the necessary permissions to create workbooks, documents, and update spaces

## Notes

### Special Considerations

* This plugin is designed to be used in a server-side listener environment
* The `configureSpace` plugin is specifically tied to the `space:configure` job topic. It will not run on other events
* The `dataChecklistPlugin` listens for `workbook:created` and `workbook:updated` events. It will automatically update its document if you add or change workbooks in the Space after the initial configuration

### Error Handling Patterns

* The plugin is built on top of `@flatfile/plugin-job-handler`, which provides robust job management. If any of the API calls made by the plugin fail, the job handler will catch the error, mark the job as 'failed', and provide the error message in the Flatfile UI
* For custom logic inside the optional `callback` function, you are responsible for your own error handling. It is best practice to use `try/catch` blocks. If you rethrow an error from the callback, the job will be marked as 'failed'

### Default Behavior

* The plugin will automatically set the `primaryWorkbookId` to the ID of the first workbook created
* Workbooks are displayed in the sidebar in the order they are created unless `maintainWorkbookOrder` is set to `true`
* No documents are created by default unless specified in the configuration


# Space Configure from Template
Source: https://flatfile.com/docs/plugins/space-configure-from-template

Automatically configure new Flatfile Spaces by cloning an existing Space Template, including workbooks, documents, and settings.

This plugin automates the setup of a new Flatfile Space by cloning an existing Space Template. It listens for the `space:configure` event, which is triggered when a new Space is created. Upon activation, the plugin finds the oldest Space Template associated with the current Flatfile App, and then copies its workbooks, documents, and settings (including metadata, actions, labels, etc.) into the new Space.

This is particularly useful for scenarios where you need to consistently provision new Spaces with a predefined structure and configuration, ensuring a uniform starting point for all users. It should be deployed in a server-side listener.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-space-configure-from-template
```

## Configuration & Parameters

The plugin is configured through a single optional parameter passed to the `configureSpaceFromTemplate` function.

### Parameters

| Parameter  | Type                  | Description                                                                                                                                                                                                                       | Default                      |
| ---------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- |
| `callback` | `Function` (optional) | An asynchronous function executed after the Space and its Workbooks have been successfully created from the template. Receives the original `event`, an array of the new `workbookIds`, and a `tick` function to report progress. | No additional logic executed |

#### Callback Function Signature

```typescript theme={null}
(event: FlatfileEvent, workbookIds: string[], tick: TickFunction) => any | Promise<any>
```

* `event`: The FlatfileEvent object that triggered the job
* `workbookIds`: An array of strings, where each string is the ID of a workbook created in the new Space
* `tick`: A function to update the job's progress with signature `(progress: number, message?: string) => Promise<void>`

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { configureSpaceFromTemplate } from "@flatfile/plugin-space-configure-from-template";

  export default function (listener) {
    listener.use(configureSpaceFromTemplate());
  }
  ```

  ```typescript TypeScript theme={null}
  import { configureSpaceFromTemplate } from "@flatfile/plugin-space-configure-from-template";
  import type { FlatfileListener } from "@flatfile/listener";

  export default function (listener: FlatfileListener) {
    listener.use(configureSpaceFromTemplate());
  }
  ```
</CodeGroup>

### Usage with Callback

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { configureSpaceFromTemplate } from "@flatfile/plugin-space-configure-from-template";

  export default function (listener) {
    listener.use(
      configureSpaceFromTemplate(
        async (event, workbookIds, tick) => {
          const { spaceId } = event.context;
          
          // Report progress
          await tick(60, "Callback started.");

          // Example: Log the IDs of the newly created workbooks
          console.log(`Space ${spaceId} configured with workbooks: ${workbookIds.join(', ')}`);

          // Perform other custom actions here...

          await tick(99, "Callback complete!");
        }
      )
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import type { FlatfileListener } from "@flatfile/listener";
  import { configureSpaceFromTemplate } from "@flatfile/plugin-space-configure-from-template";

  export default function (listener: FlatfileListener) {
    listener.use(
      configureSpaceFromTemplate(
        async (event, workbookIds, tick) => {
          const { spaceId } = event.context;
          
          // Report progress
          await tick(60, "Callback started.");

          // Example: Log the IDs of the newly created workbooks
          console.log(`Space ${spaceId} configured with workbooks: ${workbookIds.join(', ')}`);

          // Perform other custom actions here...

          await tick(99, "Callback complete!");
        }
      )
    );
  }
  ```
</CodeGroup>

## Troubleshooting

### Common Errors

**"Space configuration failed"**

* Check the logs of your listener for a more detailed error message
* This could be due to missing API permissions, network issues, or the absence of a Space Template in your App

**"No space template found"**

* This error occurs if the plugin cannot find any Space Templates associated with the `appId` from the event context
* Ensure you have at least one Space configured as a template for the App you are using

## Notes

### Default Behavior

If no callback is provided, the plugin will configure the Space from the template and mark the job as complete. No additional custom logic is executed. When a callback is provided, it is invoked after the space configuration is complete, with the job's progress at 50%.

### Important Considerations

* **Deployment Environment**: This plugin must be deployed in a server-side listener environment, as it makes direct calls to the Flatfile API
* **Template Selection Logic**: The plugin automatically selects the oldest Space Template associated with the App (sorted by `createdAt` date). It is not possible to specify which template to use if multiple exist. The first one created will always be chosen
* **Permissions**: The agent or token used by the listener must have sufficient permissions to perform the following API actions: `spaces:list`, `spaces:update`, `workbooks:list`, `workbooks:create`, `documents:list`, and `documents:create`
* **Idempotency**: The plugin operates on the `space:configure` event, which typically runs only once for a given Space. Re-running the job may lead to duplicate workbooks or unexpected behavior

### Error Handling

Errors are caught within the `jobHandler`. A generic error is thrown to fail the job, and the specific error is logged to the server console for debugging. There is no built-in mechanism for custom error handling; failures are managed by the Flatfile Job system.


# SQL DDL to Flatfile Blueprint Converter
Source: https://flatfile.com/docs/plugins/sql-ddl-converter

Automatically create Flatfile Blueprints from SQL Data Definition Language (DDL) files to streamline database schema imports.

The SQL DDL Converter plugin automates the creation of a Flatfile Blueprint from a SQL Data Definition Language (DDL) file. Its primary purpose is to streamline the setup of a Flatfile Space by translating existing database schemas directly into Flatfile Workbooks and Sheets.

The plugin reads a provided SQL file (e.g., from a `CREATE TABLE` script), parses it to identify table structures, and converts each table into a corresponding Sheet configuration with appropriate fields. This is ideal for use cases where a data import process needs to match an existing database schema, saving significant manual configuration time. It is designed to be used in a server-side listener, typically on the `space:configure` event.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-convert-sql-ddl
```

## Configuration & Parameters

The plugin is configured through the `setupFactory` object passed to the `configureSpaceWithSqlDDL` function.

### setupFactory: SqlSetupFactory (required)

The main configuration object containing:

#### workbooks: PartialWorkbookConfig\[] (required)

An array of workbook configurations to create in the Space.

Each `PartialWorkbookConfig` object contains:

* **name** (string): The display name of the Workbook
* **source** (string, required): The relative file path to the SQL DDL file (relative to project root)
* **sheets** (PartialSheetConfig\[], required): An array of sheet configurations

Each `PartialSheetConfig` object contains:

* **name** (string): The display name of the Sheet
* **slug** (string, required): The identifier for the sheet that MUST exactly match the table name from the SQL DDL file

#### space (object, optional)

An object containing additional configuration for the Space itself, such as metadata. Follows the structure of `Flatfile.spaces.SpaceConfig`.

### Default Behavior

The plugin has no default configuration and requires the `setupFactory` object to be fully defined. It will:

* Read the file specified in the `source` property
* Parse it as MySQL DDL
* Map each table to a sheet configuration where the table name matches the sheet's `slug`
* Log errors to console and skip sheets if the `slug` doesn't correspond to any table found in the SQL file
* Continue processing remaining valid sheets

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from "@flatfile/listener";
  import { configureSpaceWithSqlDDL } from "@flatfile/plugin-convert-sql-ddl";

  listener.use(
    configureSpaceWithSqlDDL({
      workbooks: [
        {
          name: "Database Import",
          source: "src/data/schema.sql",
          sheets: [
            {
              name: "Users",
              slug: "users", // Must match a table name in schema.sql
            },
            {
              name: "Products",
              slug: "products", // Must match a table name in schema.sql
            },
          ],
        },
      ],
    })
  );
  ```

  ```typescript TypeScript theme={null}
  import { listener } from "@flatfile/listener";
  import { configureSpaceWithSqlDDL, SqlSetupFactory } from "@flatfile/plugin-convert-sql-ddl";

  const setupFactory: SqlSetupFactory = {
    workbooks: [
      {
        name: "Database Import",
        source: "src/data/schema.sql",
        sheets: [
          {
            name: "Users",
            slug: "users", // Must match a table name in schema.sql
          },
          {
            name: "Products",
            slug: "products", // Must match a table name in schema.sql
          },
        ],
      },
    ],
  };

  listener.use(configureSpaceWithSqlDDL(setupFactory));
  ```
</CodeGroup>

### Advanced Configuration with Callback

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from "@flatfile/listener";
  import { configureSpaceWithSqlDDL } from "@flatfile/plugin-convert-sql-ddl";

  listener.use(
    configureSpaceWithSqlDDL(
      {
        workbooks: [
          {
            name: "Database Import",
            source: "src/data/schema.sql",
            sheets: [
              {
                name: "Users",
                slug: "users",
              },
            ],
          },
        ],
        space: {
          metadata: {
            sidebarTitle: "SQL Import Tool",
          },
        },
      },
      (event, workbookIds, tick) => {
        // This function runs after the space is configured
        console.log("Space configured successfully!");
        console.log("Created workbook IDs:", workbookIds);
        tick(100, "Configuration complete");
      }
    )
  );
  ```

  ```typescript TypeScript theme={null}
  import { listener } from "@flatfile/listener";
  import { configureSpaceWithSqlDDL, SqlSetupFactory } from "@flatfile/plugin-convert-sql-ddl";
  import { FlatfileEvent } from "@flatfile/listener";

  const setupFactory: SqlSetupFactory = {
    workbooks: [
      {
        name: "Database Import",
        source: "src/data/schema.sql",
        sheets: [
          {
            name: "Users",
            slug: "users",
          },
        ],
      },
    ],
    space: {
      metadata: {
        sidebarTitle: "SQL Import Tool",
      },
    },
  };

  listener.use(
    configureSpaceWithSqlDDL(
      setupFactory,
      (event: FlatfileEvent, workbookIds: string[], tick: any) => {
        // This function runs after the space is configured
        console.log("Space configured successfully!");
        console.log("Created workbook IDs:", workbookIds);
        tick(100, "Configuration complete");
      }
    )
  );
  ```
</CodeGroup>

### Custom Event Handler Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from "@flatfile/listener";
  import { configureSpaceWithSqlDDL } from "@flatfile/plugin-convert-sql-ddl";

  listener.on("space:configure", async (event) => {
    const { spaceId } = event.context;
    
    const configure = configureSpaceWithSqlDDL({
      workbooks: [
        {
          name: "Customer Data",
          source: "data/my_schema.sql",
          sheets: [
            { name: "Customers", slug: "customers" },
            { name: "Orders", slug: "orders" },
          ],
        },
      ],
      space: {
        metadata: {
          theme: {
            root: {
              primaryColor: "blue"
            }
          }
        }
      }
    });
    
    await configure(listener);
  });
  ```

  ```typescript TypeScript theme={null}
  import { listener } from "@flatfile/listener";
  import { configureSpaceWithSqlDDL, SqlSetupFactory } from "@flatfile/plugin-convert-sql-ddl";
  import { FlatfileEvent } from "@flatfile/listener";

  listener.on("space:configure", async (event: FlatfileEvent) => {
    const { spaceId } = event.context;
    
    const setupFactory: SqlSetupFactory = {
      workbooks: [
        {
          name: "Customer Data",
          source: "data/my_schema.sql",
          sheets: [
            { name: "Customers", slug: "customers" },
            { name: "Orders", slug: "orders" },
          ],
        },
      ],
      space: {
        metadata: {
          theme: {
            root: {
              primaryColor: "blue"
            }
          }
        }
      }
    };
    
    const configure = configureSpaceWithSqlDDL(setupFactory);
    await configure(listener);
  });
  ```
</CodeGroup>

## Troubleshooting

### Schema Not Found Error

If a sheet's `slug` doesn't match any table in the SQL file, you'll see a console error:

Schema not found for table name accounts
**Solution**: Ensure the `slug` property exactly matches the table name in your SQL DDL file.

### File Not Found Error

If the SQL file specified in the `source` property cannot be read, a hard error will be thrown.

**Solution**:

* Verify the file path is correct and relative to your project root
* Ensure the file exists and the process has read permissions
* Check that the file contains valid MySQL DDL syntax

### Example Error Handling

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from "@flatfile/listener";
  import { configureSpaceWithSqlDDL } from "@flatfile/plugin-convert-sql-ddl";

  // Example: schema.sql contains 'users' table but not 'accounts'
  listener.use(
    configureSpaceWithSqlDDL({
      workbooks: [
        {
          name: "App Data",
          source: "data/schema.sql",
          sheets: [
            {
              name: "Users",
              slug: "users", // This will succeed
            },
            {
              name: "Accounts", 
              slug: "accounts", // This will fail and be skipped
            },
          ],
        },
      ],
    })
  );
  ```

  ```typescript TypeScript theme={null}
  import { listener } from "@flatfile/listener";
  import { configureSpaceWithSqlDDL, SqlSetupFactory } from "@flatfile/plugin-convert-sql-ddl";

  // Example: schema.sql contains 'users' table but not 'accounts'
  const setupFactory: SqlSetupFactory = {
    workbooks: [
      {
        name: "App Data",
        source: "data/schema.sql",
        sheets: [
          {
            name: "Users",
            slug: "users", // This will succeed
          },
          {
            name: "Accounts",
            slug: "accounts", // This will fail and be skipped
          },
        ],
      },
    ],
  };

  listener.use(configureSpaceWithSqlDDL(setupFactory));
  ```
</CodeGroup>

## Notes

### Important Considerations

* **Server-Side Only**: This plugin must be deployed in a server-side listener environment as it requires file system access to read SQL files
* **File Path**: The `source` property must be a file path relative to the project's root directory
* **SQL Dialect**: Currently only supports MySQL-compatible DDL syntax due to the hardcoded parser configuration
* **Exact Matching**: The `slug` property must exactly match table names in your SQL DDL file for proper field generation
* **Automatic Constraints**: The plugin automatically identifies `NOT NULL` columns and other constraints from DDL and translates them into Flatfile field constraints
* **Bundled Dependencies**: Uses `sql-ddl-to-json-schema` and `@flatfile/plugin-convert-json-schema` internally (no separate installation required)

### Default Behavior

* Reads and parses SQL files as MySQL DDL
* Maps tables to sheets based on exact slug matching
* Logs errors for unmatched slugs but continues processing valid sheets
* Automatically applies field constraints based on SQL column definitions
* Requires fully defined configuration with no built-in defaults


# Stored Constraints Plugin
Source: https://flatfile.com/docs/plugins/stored-constraints

Automatically applies server-side validation rules to records during the data import process using reusable JavaScript functions defined at the App level.

The Stored Constraints plugin automatically applies server-side validation rules to records during the data import process. These rules, called "stored constraints," are defined as reusable JavaScript functions at the App level within your Flatfile account. You can then apply these constraints to specific fields in your Sheet configurations.

When a user submits data, this plugin triggers. It fetches the stored constraint functions from the Flatfile API, finds the corresponding fields in the submitted data, and executes the validation logic for each record.

This plugin is ideal for enforcing complex or reusable business logic, such as validating a SKU against an external database, checking for valid country-state combinations, or implementing custom data quality rules that are shared across multiple Sheets or Blueprints.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-stored-constraints
```

## Configuration & Parameters

The `storedConstraint()` function itself does not accept any configuration parameters. The plugin's behavior is configured entirely within the Flatfile platform.

Configuration is managed in two places in Flatfile:

### App Constraints

* **Location**: In your Flatfile App's settings
* **Purpose**: This is where you define the reusable validation logic. Each stored constraint consists of a unique name (the "validator") and a JavaScript function body (the "function")
* **Example**: You could create a constraint named "is-valid-email" with the function body `(value, key, { record }) => { if (!/\S+@\S+\.\S+/.test(value)) { record.addError(key, 'Invalid email format.') } }`

### Sheet Field Constraints

* **Location**: In your Sheet or Blueprint configuration
* **Purpose**: You apply a stored constraint to a specific field by adding a constraint of type "stored" and referencing its name
* **`validator`** (string): The name of the App-level constraint to apply (e.g., "is-valid-email")
* **`config`** (object, optional): An arbitrary configuration object that gets passed to your validation function. This allows you to make a single stored constraint adaptable to different contexts

### Default Behavior

By default, the plugin does nothing if no fields in a Sheet are configured with a `type: 'stored'` constraint. When such constraints are present, the plugin will automatically trigger on every data submission (`commit:created` event) for all sheets and execute the corresponding validation logic.

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  // listener.js
  import { storedConstraint } from '@flatfile/plugin-stored-constraints';

  export default function (listener) {
    // Registers the plugin to run on data submission
    listener.use(storedConstraint());
  }
  ```

  ```typescript TypeScript theme={null}
  // listener.ts
  import type { FlatfileListener } from "@flatfile/listener";
  import { storedConstraint } from '@flatfile/plugin-stored-constraints';

  export default function (listener: FlatfileListener) {
    // Registers the plugin to run on data submission
    listener.use(storedConstraint());
  }
  ```
</CodeGroup>

### Configuration Example

<CodeGroup>
  ```javascript JavaScript theme={null}
  // In your Flatfile Sheet/Blueprint configuration (JSON or UI):
  /*
  {
    "key": "email",
    "label": "Email",
    "constraints": [
      {
        "type": "stored",
        "validator": "is-valid-email-domain",
        "config": {
          "allowedDomain": "example.com"
        }
      }
    ]
  }
  */

  // In your Flatfile App's Stored Constraints section (UI):
  /*
    Validator Name: "is-valid-email-domain"
    Function:
    function(value, key, { record, config }) {
      if (!value.endsWith('@' + config.allowedDomain)) {
        record.addError(key, 'Email must be from the ' + config.allowedDomain + ' domain.');
      }
    }
  */

  // Your listener code remains simple:
  // listener.js
  import { storedConstraint } from '@flatfile/plugin-stored-constraints';

  export default function (listener) {
    listener.use(storedConstraint());
  }
  ```

  ```typescript TypeScript theme={null}
  // In your Flatfile Sheet/Blueprint configuration (JSON or UI):
  /*
  {
    "key": "email",
    "label": "Email",
    "constraints": [
      {
        "type": "stored",
        "validator": "is-valid-email-domain",
        "config": {
          "allowedDomain": "example.com"
        }
      }
    ]
  }
  */

  // In your Flatfile App's Stored Constraints section (UI):
  /*
    Validator Name: "is-valid-email-domain"
    Function:
    function(value, key, { record, config }) {
      if (!value.endsWith('@' + config.allowedDomain)) {
        record.addError(key, 'Email must be from the ' + config.allowedDomain + ' domain.');
      }
    }
  */

  // Your listener code remains simple:
  // listener.ts
  import type { FlatfileListener } from "@flatfile/listener";
  import { storedConstraint } from '@flatfile/plugin-stored-constraints';

  export default function (listener: FlatfileListener) {
    listener.use(storedConstraint());
  }
  ```
</CodeGroup>

### Advanced Usage with Dependencies

Stored constraint functions have access to a `deps` object containing helpful libraries. This example shows a constraint using the `validator` library to check for a valid email format.

<CodeGroup>
  ```javascript JavaScript theme={null}
  // In your Flatfile App's Stored Constraints section (UI):
  /*
    Validator Name: "is-valid-email-advanced"
    Function:
    function(value, key, { record, deps }) {
      // Access the 'validator' library from the deps object
      if (!deps.validator.isEmail(value)) {
        record.addError(key, 'Please enter a valid email address.');
      }
    }
  */

  // Your listener code does not change:
  // listener.js
  import { storedConstraint } from '@flatfile/plugin-stored-constraints';

  export default function (listener) {
    listener.use(storedConstraint());
  }
  ```

  ```typescript TypeScript theme={null}
  // In your Flatfile App's Stored Constraints section (UI):
  /*
    Validator Name: "is-valid-email-advanced"
    Function:
    function(value, key, { record, deps }) {
      // Access the 'validator' library from the deps object
      if (!deps.validator.isEmail(value)) {
        record.addError(key, 'Please enter a valid email address.');
      }
    }
  */

  // Your listener code does not change:
  // listener.ts
  import type { FlatfileListener } from "@flatfile/listener";
  import { storedConstraint } from '@flatfile/plugin-stored-constraints';

  export default function (listener: FlatfileListener) {
    listener.use(storedConstraint());
  }
  ```
</CodeGroup>

### Error Handling Examples

#### User-facing Error

```javascript theme={null}
// Stored constraint function defined in the Flatfile App UI
function(value, key, { record }) {
  if (value === null || value === '') {
    // This adds an error to the cell in the UI
    record.addError(key, 'This field is required.');
  }
}
```

#### Server-side Error

```javascript theme={null}
// Stored constraint function defined in the Flatfile App UI
function(value, key, { record }) {
  // This will throw an exception because 'badProperty' does not exist
  const x = null;
  x.badProperty = 'test';
}

// When the above constraint runs, the plugin will catch the TypeError
// and log a message like "Error executing constraint: Cannot set properties of null..."
// to the server console. No error will appear in the Flatfile UI.
```

## Troubleshooting

If your constraints are not running, verify that:

1. The `storedConstraint()` plugin is registered in your listener
2. The field in your Sheet configuration has a constraint with `type: 'stored'`
3. The `validator` name in the Sheet constraint exactly matches the name of a Stored Constraint defined in your App
4. The listener's environment has a valid Flatfile API key with sufficient permissions

If you see "Error executing constraint" in your server logs, check the corresponding stored constraint function for runtime errors.

## Notes

### Security Considerations

* The plugin uses `eval()` to execute the function strings stored in your App constraints. This means the constraint logic is executed dynamically. Ensure that only trusted administrators can create or edit stored constraint functions to avoid security risks.

### Requirements

* The plugin requires an active connection to the Flatfile API to fetch sheet configurations and app-level constraint definitions. Ensure your listener environment is configured with a valid Flatfile API key.
* The plugin triggers on the `commit:created` event, which runs server-side after a user submits their data.

### Error Handling

* The plugin includes a `try...catch` block around the execution of each constraint. If a constraint function throws an unhandled exception, the error is logged to the server-side console, and processing continues with the next record or field.
* For user-facing validation errors, the constraint function logic itself is responsible for calling `record.addError(fieldKey, 'Your error message')`. The plugin does not automatically convert thrown exceptions into record errors.


# String Validator Plugin
Source: https://flatfile.com/docs/plugins/string

A comprehensive plugin for validating and transforming string data during the Flatfile import process with configurable validation rules, pattern matching, and automatic data corrections.

The String Validator plugin for Flatfile provides a comprehensive way to validate and transform string data during the import process. It allows you to configure multiple validation rules for specified fields in a single, easy-to-use configuration object.

Its main purpose is to enforce data quality and consistency for string-based fields. Key features include validating strings against regular expression patterns (both common predefined patterns like email and URL, and custom ones), enforcing length constraints (min, max, or exact), and ensuring proper casing (lowercase, uppercase, or titlecase). The plugin can also automatically trim leading or trailing whitespace.

Use cases include cleaning user-submitted data, ensuring identifiers match a specific format (e.g., 'ABC-123'), validating contact information like emails and phone numbers, and standardizing the case of names or categories before they are imported into a system. If a value doesn't meet a validation rule but can be corrected (e.g., wrong case, extra whitespace), the plugin will automatically transform the value and notify the user of the change.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-validate-string
```

## Configuration & Parameters

The plugin is configured with a single `StringValidationConfig` object with the following properties:

### Required Parameters

<ParamField type="string[]">
  An array of field keys (column names) to which the validation rules should be applied.
</ParamField>

### Optional Parameters

<ParamField type="string">
  The slug of a specific sheet to apply the validations to. Defaults to '\*\*' (applies to all sheets in the workbook).
</ParamField>

<ParamField type="'email' | 'phone' | 'url' | RegExp">
  A regular expression to validate the string against. You can use one of the predefined string keys for common patterns ('email', 'phone', 'url') or provide your own custom RegExp object.
</ParamField>

<ParamField type="number">
  The minimum allowed length for the string.
</ParamField>

<ParamField type="number">
  The maximum allowed length for the string.
</ParamField>

<ParamField type="number">
  The exact required length for the string.
</ParamField>

<ParamField type="'lowercase' | 'uppercase' | 'titlecase'">
  Enforces a specific character case. If the input string does not match, it will be transformed, and a validation message will be added.
</ParamField>

<ParamField type="{ leading?: boolean, trailing?: boolean }">
  An object to control whitespace trimming. If `leading` is true, it trims whitespace from the start. If `trailing` is true, it trims from the end. If the string is trimmed, the value is transformed, and a message is added.
</ParamField>

<ParamField type="boolean">
  Controls whether an empty string is considered a valid value. By default, empty strings will generate a "Field cannot be empty" error.
</ParamField>

<ParamField type="{ pattern?: string, length?: string, case?: string, trim?: string }">
  An object to provide custom error messages for different validation types, overriding the default messages. The plugin provides default messages for each validation type (e.g., "Invalid format", "Minimum length is X").
</ParamField>

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateString } from '@flatfile/plugin-validate-string';

  const listener = new FlatfileListener();

  listener.use(validateString({
    fields: ['firstName', 'lastName'],
    minLength: 2,
    maxLength: 50,
    caseType: 'titlecase'
  }));
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateString } from '@flatfile/plugin-validate-string';

  const listener = new FlatfileListener();

  listener.use(validateString({
    fields: ['firstName', 'lastName'],
    minLength: 2,
    maxLength: 50,
    caseType: 'titlecase'
  }));
  ```
</CodeGroup>

### Email Validation

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateString } from '@flatfile/plugin-validate-string';

  const listener = new FlatfileListener();

  listener.use(validateString({
    fields: ['email'],
    pattern: 'email',
    emptyStringAllowed: false,
    errorMessages: {
      pattern: 'Please provide a valid email address.',
      length: 'The email address is too long.'
    }
  }));
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateString } from '@flatfile/plugin-validate-string';

  const listener = new FlatfileListener();

  listener.use(validateString({
    fields: ['email'],
    pattern: 'email',
    emptyStringAllowed: false,
    errorMessages: {
      pattern: 'Please provide a valid email address.',
      length: 'The email address is too long.'
    }
  }));
  ```
</CodeGroup>

### Advanced Custom Pattern Validation

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateString } from '@flatfile/plugin-validate-string';

  const listener = new FlatfileListener();

  // Example: Validate a product SKU that must be in the format 'SKU-12345'
  listener.use(validateString({
    fields: ['product_sku'],
    pattern: /^SKU-\d{5}$/,
    caseType: 'uppercase',
    errorMessages: {
      pattern: 'SKU must be in the format SKU-XXXXX, where X is a digit.',
      case: 'SKU must be in uppercase.'
    }
  }));
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { validateString } from '@flatfile/plugin-validate-string';

  const listener = new FlatfileListener();

  // Example: Validate a product SKU that must be in the format 'SKU-12345'
  listener.use(validateString({
    fields: ['product_sku'],
    pattern: /^SKU-\d{5}$/,
    caseType: 'uppercase',
    errorMessages: {
      pattern: 'SKU must be in the format SKU-XXXXX, where X is a digit.',
      case: 'SKU must be in uppercase.'
    }
  }));
  ```
</CodeGroup>

### Using the Utility Function

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { validateAndTransformString } from '@flatfile/plugin-validate-string';

  const config = {
    fields: ['email'],
    pattern: 'email'
  };

  const result1 = validateAndTransformString('test@example.com', config);
  // result1 -> { value: 'test@example.com', error: null }

  const result2 = validateAndTransformString('not-an-email', config);
  // result2 -> { value: 'not-an-email', error: 'Invalid format' }
  ```

  ```typescript TypeScript theme={null}
  import { validateAndTransformString } from '@flatfile/plugin-validate-string';

  const config = {
    fields: ['email'],
    pattern: 'email'
  };

  const result1 = validateAndTransformString('test@example.com', config);
  // result1 -> { value: 'test@example.com', error: null }

  const result2 = validateAndTransformString('not-an-email', config);
  // result2 -> { value: 'not-an-email', error: 'Invalid format' }
  ```
</CodeGroup>

### Error Handling Example

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { validateAndTransformString } from '@flatfile/plugin-validate-string';

  const config = {
    fields: ['code'],
    exactLength: 5
  };

  const result = validateAndTransformString('ABC', config);
  if (result.error) {
    console.log(`Validation failed: ${result.error}`);
    // Logs: "Validation failed: Exact length must be 5"
  }
  ```

  ```typescript TypeScript theme={null}
  import { validateAndTransformString } from '@flatfile/plugin-validate-string';

  const config = {
    fields: ['code'],
    exactLength: 5
  };

  const result = validateAndTransformString('ABC', config);
  if (result.error) {
    console.log(`Validation failed: ${result.error}`);
    // Logs: "Validation failed: Exact length must be 5"
  }
  ```
</CodeGroup>

## API Reference

### validateString(config)

The main entry point for the plugin. It configures and registers a `recordHook` that listens for new commits and applies the specified string validations to each record.

**Parameters:**

* `config` (StringValidationConfig): An object that defines the validation and transformation rules.

**Returns:** A function that takes a `FlatfileListener` instance and attaches the validation logic to it.

### validateAndTransformString(value, config)

An exported utility function that runs the validation and transformation logic on a single string value. It is used internally by the plugin but can be used for custom validation logic if needed.

**Parameters:**

* `value` (string): The input string to validate and transform.
* `config` (StringValidationConfig): The configuration object defining the rules to apply.

**Returns:** An object of type `ValidationResult` with two properties:

* `value` (string): The original or transformed string value.
* `error` (string | null): An error message string if any validation fails, otherwise `null`.

## Notes

### Default Behavior

* **Empty strings**: By default, empty strings are considered invalid. To allow them, you must explicitly set `emptyStringAllowed: true` in the configuration.
* **Sheet targeting**: When no `sheetSlug` is specified, the plugin applies to all sheets in the workbook using the default value '\*\*'.
* **Error messages**: The plugin provides default error messages for each validation type (e.g., "Invalid format", "Minimum length is X") which can be customized using the `errorMessages` configuration option.

### Important Considerations

* The plugin operates using the `@flatfile/plugin-record-hook`, which is a dependency. It processes records individually during the `commit:created` event.
* The plugin can modify data. When a transformation is applied (e.g., changing case or trimming whitespace), the record's value is updated with `record.set()`.
* By default, when a transformation is applied, the plugin also adds a validation message to the cell (e.g., "Field value must be in titlecase"). This informs the user that their original data was automatically corrected.
* The plugin handles `null` and `undefined` values by simply skipping them. Validation is only applied to defined string values.
* The plugin does not throw exceptions. Instead, it captures validation failures and adds them as errors to the corresponding record and field using `record.addError(field, message)`. These errors are then visible to the user in the Flatfile UI.


# Text Summarization and Key Phrase Extraction Plugin
Source: https://flatfile.com/docs/plugins/summarize

Automatically enriches Flatfile data by performing text summarization and key phrase extraction on specified text fields using natural language processing.

This plugin automatically enriches Flatfile data by performing text summarization and key phrase extraction on a specified text field. It uses the 'compromise' natural language processing library to analyze the content of a source field, generate a concise summary, and identify important phrases. The generated summary and key phrases are then populated into their designated fields.

This is useful for processing large blocks of text, such as product descriptions, user feedback, article content, or support tickets, to quickly get a high-level overview and identify key topics without manual review. The plugin is configured per sheet and can be customized to control the length of the generated summary.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-enrich-summarize
```

## Configuration & Parameters

The `summarize` function accepts a configuration object with the following parameters:

| Parameter           | Type   | Required | Default | Description                                                                                                                                                                                                                                              |
| ------------------- | ------ | -------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sheetSlug`         | string | Yes      | -       | The slug of the sheet that the plugin should operate on                                                                                                                                                                                                  |
| `contentField`      | string | Yes      | -       | The API key of the field that contains the source text to be summarized                                                                                                                                                                                  |
| `summaryField`      | string | Yes      | -       | The API key of the field where the generated summary should be stored                                                                                                                                                                                    |
| `keyPhrasesField`   | string | Yes      | -       | The API key of the field where the extracted key phrases (as a comma-separated string) should be stored                                                                                                                                                  |
| `summaryLength`     | number | No       | 2       | An optional integer specifying the number of sentences the final summary should contain. This is overridden if `summaryPercentage` is also set                                                                                                           |
| `summaryPercentage` | number | No       | -       | An optional number specifying the desired summary length as a percentage of the total sentences in the content. For example, a value of 30 would create a summary using 30% of the original sentences. This option takes precedence over `summaryLength` |

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { summarize } from "@flatfile/plugin-enrich-summarize";

  const listener = new FlatfileListener();

  listener.use(
    summarize({
      sheetSlug: "articles",
      contentField: "full_text",
      summaryField: "summary",
      keyPhrasesField: "key_phrases",
    })
  );
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { summarize } from "@flatfile/plugin-enrich-summarize";

  const listener = new FlatfileListener();

  listener.use(
    summarize({
      sheetSlug: "articles",
      contentField: "full_text",
      summaryField: "summary",
      keyPhrasesField: "key_phrases",
    })
  );
  ```
</CodeGroup>

### Custom Summary Length

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { summarize } from "@flatfile/plugin-enrich-summarize";

  const listener = new FlatfileListener();

  // Configure the plugin to create a 3-sentence summary
  listener.use(
    summarize({
      sheetSlug: "articles",
      contentField: "full_text",
      summaryField: "summary",
      keyPhrasesField: "key_phrases",
      summaryLength: 3
    })
  );
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { summarize } from "@flatfile/plugin-enrich-summarize";

  const listener = new FlatfileListener();

  // Configure the plugin to create a 3-sentence summary
  listener.use(
    summarize({
      sheetSlug: "articles",
      contentField: "full_text",
      summaryField: "summary",
      keyPhrasesField: "key_phrases",
      summaryLength: 3
    })
  );
  ```
</CodeGroup>

### Percentage-Based Summary

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { summarize } from "@flatfile/plugin-enrich-summarize";

  const listener = new FlatfileListener();

  // Configure the plugin to create a summary using 25% of the original sentences
  listener.use(
    summarize({
      sheetSlug: "product_reviews",
      contentField: "review_text",
      summaryField: "review_summary",
      keyPhrasesField: "review_tags",
      summaryPercentage: 25
    })
  );
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from "@flatfile/listener";
  import { summarize } from "@flatfile/plugin-enrich-summarize";

  const listener = new FlatfileListener();

  // Configure the plugin to create a summary using 25% of the original sentences
  listener.use(
    summarize({
      sheetSlug: "product_reviews",
      contentField: "review_text",
      summaryField: "review_summary",
      keyPhrasesField: "review_tags",
      summaryPercentage: 25
    })
  );
  ```
</CodeGroup>

## Troubleshooting

* **Plugin not triggering**: Double-check that the `sheetSlug` in your configuration exactly matches the slug of your Sheet in Flatfile
* **Summaries not appearing**: Verify that the `contentField`, `summaryField`, and `keyPhrasesField` names are correct and that the `contentField` actually contains text
* **Data not being processed**: Ensure the `summaryField` is empty for records you expect to be processed, as the plugin will not overwrite existing summaries

## Notes

### Default Behavior

By default, the plugin generates a summary containing 2 sentences. The summarization logic takes sentences from the beginning and end of the source text to form the summary. The plugin will only generate a summary and key phrases if the target `summaryField` is empty; it will not overwrite existing data. If the source `contentField` for a record is empty, the plugin will add an error to that field and skip processing for that record.

### Limitations and Considerations

* The plugin's summarization algorithm is basic: it combines a number of sentences from the beginning and end of the text, separated by "...". It is not an abstractive or AI-based summarization
* The plugin will not overwrite data. If the `summaryField` already contains a value for a given record, that record will be skipped
* Key phrase extraction is based on a fixed grammatical pattern (`#Adjective? #Adjective? #Noun+`), which identifies phrases consisting of up to two optional adjectives followed by one or more nouns
* The plugin depends on the `compromise` NLP library, and the quality of the output is determined by its capabilities
* The plugin is designed to work on `string` field types

### Error Handling

The main error handling pattern is to check for preconditions on a per-record basis. If a precondition fails (e.g., the source text field is empty), the plugin adds a field-level error to the record using `record.addError()`. This provides direct feedback to the user in the Flatfile UI without halting the entire import process.


# Field Translation using Google Translate API
Source: https://flatfile.com/docs/plugins/translate

Automatically translate text content in specified fields using Google Translate API and create new fields with translated content

The Translate plugin for Flatfile integrates with the Google Translate API to automatically translate the text content of specified fields within a sheet. When records are processed, the plugin takes the values from designated source fields, translates them from a specified source language to a target language, and then adds the translated text into newly created fields. The new fields are automatically named by appending the target language code to the original field name (e.g., 'description' becomes 'description\_es'). This is useful for data localization, preparing multilingual product catalogs, or standardizing user-submitted content into a single language.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-convert-translate
```

## Configuration & Parameters

The plugin requires a configuration object with the following properties. All options are required and there are no default values:

| Parameter           | Type       | Description                                                                                                      |
| ------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------- |
| `sourceLanguage`    | `string`   | The IETF language code of the source text (e.g., 'en' for English)                                               |
| `targetLanguage`    | `string`   | The IETF language code for the target language (e.g., 'es' for Spanish). This is also used to name the new field |
| `sheetSlug`         | `string`   | The slug of the sheet that the plugin should listen to. Records from other sheets will be ignored                |
| `fieldsToTranslate` | `string[]` | An array of field keys (names) that should be translated. The plugin will only process these fields              |
| `projectId`         | `string`   | Your Google Cloud project ID associated with the Google Translate API                                            |
| `keyFilename`       | `string`   | The absolute or relative path to your Google Cloud service account key file (JSON)                               |

### Default Behavior

By default, the plugin does not do anything until it is configured and registered with a Flatfile listener. Once configured, it will listen for record processing events on the specified `sheetSlug`. For each record, it will read the text from the `fieldsToTranslate`, call the Google Translate API, and write the results to new fields. If a field to be translated is empty or null, it is skipped.

## Usage Examples

### Basic Usage

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    import { FlatfileListener } from '@flatfile/listener';
    import { convertTranslatePlugin } from '@flatfile/plugin-convert-translate';

    export default function (listener) {
      listener.use(
        convertTranslatePlugin({
          sourceLanguage: 'en',
          targetLanguage: 'fr',
          sheetSlug: 'contacts',
          fieldsToTranslate: ['notes', 'comments'],
          projectId: 'your-gcp-project-id',
          keyFilename: './gcp-credentials.json',
        })
      );
    }
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import { FlatfileListener } from '@flatfile/listener';
    import { convertTranslatePlugin } from '@flatfile/plugin-convert-translate';

    export default function (listener: FlatfileListener) {
      listener.use(
        convertTranslatePlugin({
          sourceLanguage: 'en',
          targetLanguage: 'fr',
          sheetSlug: 'contacts',
          fieldsToTranslate: ['notes', 'comments'],
          projectId: 'your-gcp-project-id',
          keyFilename: './gcp-credentials.json',
        })
      );
    }
    ```
  </Tab>
</Tabs>

### Multiple Language Translation

To translate to multiple languages, use the plugin multiple times with different configurations:

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    import { FlatfileListener } from '@flatfile/listener';
    import { convertTranslatePlugin } from '@flatfile/plugin-convert-translate';

    export default function (listener) {
      // Translate to Spanish
      listener.use(
        convertTranslatePlugin({
          sourceLanguage: 'en',
          targetLanguage: 'es',
          sheetSlug: 'products',
          fieldsToTranslate: ['name', 'description'],
          projectId: 'your-gcp-project-id',
          keyFilename: './gcp-credentials.json',
        })
      );

      // Translate to German
      listener.use(
        convertTranslatePlugin({
          sourceLanguage: 'en',
          targetLanguage: 'de',
          sheetSlug: 'products',
          fieldsToTranslate: ['name', 'description'],
          projectId: 'your-gcp-project-id',
          keyFilename: './gcp-credentials.json',
        })
      );
    }
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import { FlatfileListener } from '@flatfile/listener';
    import { convertTranslatePlugin } from '@flatfile/plugin-convert-translate';

    export default function (listener: FlatfileListener) {
      // Translate to Spanish
      listener.use(
        convertTranslatePlugin({
          sourceLanguage: 'en',
          targetLanguage: 'es',
          sheetSlug: 'products',
          fieldsToTranslate: ['name', 'description'],
          projectId: 'your-gcp-project-id',
          keyFilename: './gcp-credentials.json',
        })
      );

      // Translate to German
      listener.use(
        convertTranslatePlugin({
          sourceLanguage: 'en',
          targetLanguage: 'de',
          sheetSlug: 'products',
          fieldsToTranslate: ['name', 'description'],
          projectId: 'your-gcp-project-id',
          keyFilename: './gcp-credentials.json',
        })
      );
    }
    ```
  </Tab>
</Tabs>

### Manual Translation with translateRecord

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    import { translateRecord, convertTranslatePlugin } from '@flatfile/plugin-convert-translate';

    const config = {
      sourceLanguage: 'en',
      targetLanguage: 'ja',
      sheetSlug: 'articles',
      fieldsToTranslate: ['title'],
      projectId: 'your-gcp-project-id',
      keyFilename: './gcp-credentials.json',
    };

    // Initialize the client (required for translateRecord to work)
    convertTranslatePlugin(listener, config);

    // Manually process a record
    record.set('title', 'Hello World');
    const { record: updatedRecord, error } = translateRecord(record, config);

    if (error) {
      console.error('Translation failed:', error);
    } else {
      const translatedTitle = updatedRecord.get('title_ja');
      console.log(translatedTitle); // Expected to be the Japanese translation
    }
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import { FlatfileRecord } from '@flatfile/plugin-record-hook';
    import { translateRecord, convertTranslatePlugin, TranslationConfig } from '@flatfile/plugin-convert-translate';

    const config: TranslationConfig = {
      sourceLanguage: 'en',
      targetLanguage: 'ja',
      sheetSlug: 'articles',
      fieldsToTranslate: ['title'],
      projectId: 'your-gcp-project-id',
      keyFilename: './gcp-credentials.json',
    };

    // Initialize the client (required for translateRecord to work)
    convertTranslatePlugin(listener, config);

    // Manually process a record
    record.set('title', 'Hello World');
    const { record: updatedRecord, error } = translateRecord(record, config);

    if (error) {
      console.error('Translation failed:', error);
    } else {
      const translatedTitle = updatedRecord.get('title_ja');
      console.log(translatedTitle); // Expected to be the Japanese translation
    }
    ```
  </Tab>
</Tabs>

## API Reference

### convertTranslatePlugin

The main entry point for the plugin. This function registers a `recordHook` with the provided Flatfile listener, which will automatically translate specified fields on records belonging to the configured sheet.

**Signature:**

```typescript theme={null}
convertTranslatePlugin(listener: FlatfileListener, config: TranslationConfig): void
```

**Parameters:**

* `listener`: FlatfileListener - An instance of a Flatfile listener to attach the hook to
* `config`: TranslationConfig - The configuration object for the plugin

**Returns:**
`void` - This function does not return a value. It modifies the listener instance directly.

### translateRecord

A standalone function that applies the translation logic to a single `FlatfileRecord`. It reads values from the fields specified in the config, translates them, and sets the translated values on new fields in the record.

**Signature:**

```typescript theme={null}
translateRecord(record: FlatfileRecord, config: TranslationConfig): { record: FlatfileRecord; error?: string }
```

**Parameters:**

* `record`: FlatfileRecord - The record to process
* `config`: TranslationConfig - A configuration object specifying languages, fields, and credentials

**Returns:**
An object containing:

* `record`: FlatfileRecord - The record with new fields containing translated text
* `error?`: string - An error message if an issue occurred during processing

## Troubleshooting

### Error Handling

The plugin includes comprehensive error handling:

* **API Errors**: If the Google Translate API call fails, the error is logged to the console and the record won't be updated with translated values
* **Record-level Errors**: If an error occurs during processing of a single record, the plugin will attach a general error message to that specific record using `record.addError()`, making it visible in the Flatfile UI
* **No Text to Translate**: If a record has no text in any of the `fieldsToTranslate`, it is skipped without error

<Tabs>
  <Tab title="JavaScript">
    ```javascript theme={null}
    // Error handling example
    const { record: updatedRecord, error } = translateRecord(record, config);

    if (error) {
      // This adds the error to the record, which is visible to the user
      updatedRecord.addError('general', error);
    }

    return updatedRecord;
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    // Error handling example
    const { record: updatedRecord, error } = translateRecord(record, config);

    if (error) {
      // This adds the error to the record, which is visible to the user
      updatedRecord.addError('general', error);
    }

    return updatedRecord;
    ```
  </Tab>
</Tabs>

## Notes

### Requirements

* **Google Cloud Account**: A Google Cloud project with the Translate API enabled is required
* **Credentials**: You must provide a valid `projectId` and a path to a service account `keyFilename` with appropriate permissions

### Field Naming Convention

The plugin creates new fields for translated content by appending `_{targetLanguage}` to the original field name (e.g., `name` becomes `name_es`). Ensure this does not conflict with existing fields in your sheet configuration.

### Limitations

* **Field Types**: The plugin is designed to work with `string` fields. Behavior with other field types is undefined
* **Initialization**: The Google Translate client is initialized globally within the plugin's scope when `convertTranslatePlugin` is first called. Subsequent calls with different credentials will re-initialize the client


# View Mapped Data Plugin
Source: https://flatfile.com/docs/plugins/view-mapped

Automatically hides unmapped columns after the mapping stage to provide a cleaner data review experience

The View Mapped Data Plugin enhances the user experience during the data import process by automatically modifying the data grid view after users complete the column matching step. It hides all columns in the destination sheet that were not mapped by the user, providing a cleaner and more focused view for data validation.

The primary use case is to simplify the "Review" stage of an import by showing only columns that contain mapped data. This reduces visual clutter and helps users concentrate on the data that will actually be imported. The plugin also includes an option to retain required fields in the view, even if they were not mapped.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-view-mapped
```

## Configuration & Parameters

The plugin accepts an optional configuration object with the following parameter:

### `keepRequiredFields`

* **Type:** `boolean`
* **Default:** `false`
* **Description:** Controls whether fields marked as "required" in the sheet's configuration are kept visible after mapping, even if the user did not map any source data to them.

### Default Behavior

By default, the plugin will hide all unmapped fields, including those that are required. Only fields that the user explicitly mapped will remain visible in the data review step.

## Usage Examples

<CodeGroup>
  ```javascript Basic Usage theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { viewMappedPlugin } from '@flatfile/plugin-view-mapped'

  export default function(listener) {
    listener.use(viewMappedPlugin())
  }
  ```

  ```typescript Basic Usage theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { viewMappedPlugin } from '@flatfile/plugin-view-mapped'

  export default function(listener: FlatfileListener) {
    listener.use(viewMappedPlugin())
  }
  ```
</CodeGroup>

<CodeGroup>
  ```javascript With Configuration theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { viewMappedPlugin } from '@flatfile/plugin-view-mapped'

  export default function(listener) {
    listener.use(viewMappedPlugin({
      keepRequiredFields: true
    }))
  }
  ```

  ```typescript With Configuration theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { viewMappedPlugin } from '@flatfile/plugin-view-mapped'

  export default function(listener: FlatfileListener) {
    listener.use(viewMappedPlugin({
      keepRequiredFields: true
    }))
  }
  ```
</CodeGroup>

<CodeGroup>
  ```javascript Complete Example theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { viewMappedPlugin } from '@flatfile/plugin-view-mapped'

  export default function(listener) {
    // Add the plugin to the listener
    listener.use(viewMappedPlugin({ keepRequiredFields: true }))

    // Define other listeners
    listener.on('**', (event) => {
      // Your other event handlers
    })
  }
  ```

  ```typescript Complete Example theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { viewMappedPlugin } from '@flatfile/plugin-view-mapped'

  export default function(listener: FlatfileListener) {
    // Add the plugin to the listener
    listener.use(viewMappedPlugin({ keepRequiredFields: true }))

    // Define other listeners
    listener.on('**', (event) => {
      // Your other event handlers
    })
  }
  ```
</CodeGroup>

## Troubleshooting

### Error Handling

The plugin includes built-in error handling. If an issue occurs while updating the sheet view, the job will fail and log the error to the console:

```javascript theme={null}
try {
  // The plugin's internal job handler logic runs here
} catch (error) {
  // If an error occurs, it is logged and a generic error is thrown to the UI
  logError('@flatfile/plugin-view-mapped', JSON.stringify(error, null, 2))
  throw new Error('plugins.viewMapped.error')
}
```

## Notes

### Requirements

* **`trackChanges` Required:** The workbook being processed must have the `trackChanges` setting enabled. The plugin checks for this setting and will abort its operation if it is not enabled. This prevents race conditions between different data processing hooks and the workbook update.

### Behavior Details

* **Foreground Job:** The plugin creates a `foreground` job to update the sheet view. This means the user interface will be blocked with a progress indicator while columns are being hidden, preventing user interaction with a stale view of the data.

* **Commit Synchronization:** The plugin includes logic to wait for all pending sheet commits to complete before attempting to update the workbook's field configuration. This prevents race conditions and ensures data integrity.

### API Reference

#### `viewMappedPlugin(options?: ViewMappedOptions)`

Initializes the View Mapped Data plugin and returns a configured listener that should be passed to `listener.use()`. The plugin listens for the completion of a `workbook:map` job and then creates a new job to update the destination sheet's configuration, hiding any unmapped columns.

**Parameters:**

* `options` (optional): Configuration object with `keepRequiredFields` boolean property

**Returns:** A listener function of type `(listener: FlatfileListener) => void`


# Webhook Egress
Source: https://flatfile.com/docs/plugins/webhook-egress

Send workbook data from Flatfile to external webhook endpoints for seamless integration with your systems

The Webhook Egress plugin acts as a bridge between Flatfile and external systems by sending workbook data to webhook endpoints. When a user triggers a specific action in the Flatfile UI (like a "Submit" button), this plugin gathers all data from all sheets within the current workbook, packages it into a single JSON payload, and sends it via an HTTP POST request to a pre-configured webhook URL.

This plugin is ideal for integrating Flatfile with custom backends, serverless functions, or third-party services that accept data via webhooks. Common use cases include triggering data processing pipelines, updating databases, or pushing data into CRM systems once users have finished cleaning and preparing their data in Flatfile.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-webhook-egress
```

## Configuration & Parameters

### Parameters

<ParamField type="string">
  The name of the job or event that will trigger the egress process. This must correspond to an `operation` name defined in an action in your workbook configuration (e.g., 'workbook:submitActionFg').
</ParamField>

<ParamField type="string">
  The URL of the webhook endpoint where the workbook data will be sent. If not provided, the plugin will use the value from the `WEBHOOK_SITE_URL` environment variable.
</ParamField>

### Default Behavior

When the `webhookUrl` parameter is not provided during initialization, the plugin will look for an environment variable named `WEBHOOK_SITE_URL` and use its value as the destination for the data. If neither the parameter nor the environment variable is set, the request will fail. The plugin sends data as a JSON payload in an HTTP POST request with `Content-Type: application/json`.

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  // listener.js
  import { listener } from '@flatfile/listener';
  import { webhookEgress } from '@flatfile/plugin-webhook-egress';

  listener.use(webhookEgress('workbook:submitActionFg'));

  // You also need to define the action in your workbook config
  // workbook.config.js
  const workbookConfig = {
    name: 'My Workbook',
    sheets: { /* ... */ },
    actions: [
      {
        operation: 'submitActionFg',
        mode: 'foreground',
        label: 'Submit Data',
        description: 'Send data to our system.',
        primary: true,
      },
    ],
  };
  ```

  ```typescript TypeScript theme={null}
  // listener.ts
  import { listener } from '@flatfile/listener';
  import { webhookEgress } from '@flatfile/plugin-webhook-egress';

  listener.use(webhookEgress('workbook:submitActionFg'));

  // You also need to define the action in your workbook config
  // workbook.config.ts
  const workbookConfig = {
    name: 'My Workbook',
    sheets: { /* ... */ },
    actions: [
      {
        operation: 'submitActionFg',
        mode: 'foreground',
        label: 'Submit Data',
        description: 'Send data to our system.',
        primary: true,
      },
    ],
  };
  ```
</CodeGroup>

### Explicit Webhook URL Configuration

<CodeGroup>
  ```javascript JavaScript theme={null}
  // listener.js
  import { listener } from '@flatfile/listener';
  import { webhookEgress } from '@flatfile/plugin-webhook-egress';

  const MY_WEBHOOK_URL = 'https://api.myapp.com/data-ingest';

  listener.use(webhookEgress('workbook:submitActionFg', MY_WEBHOOK_URL));
  ```

  ```typescript TypeScript theme={null}
  // listener.ts
  import { listener } from '@flatfile/listener';
  import { webhookEgress } from '@flatfile/plugin-webhook-egress';

  const MY_WEBHOOK_URL: string = 'https://api.myapp.com/data-ingest';

  listener.use(webhookEgress('workbook:submitActionFg', MY_WEBHOOK_URL));
  ```
</CodeGroup>

### Webhook Response with Rejections

The plugin can process responses from your webhook that contain data rejections. If your webhook performs validation and finds errors, it can return them in a specific JSON format, and the plugin will display these errors in the Flatfile UI.

<CodeGroup>
  ```javascript JavaScript theme={null}
  // Example webhook endpoint response format
  // POST https://api.myapp.com/data-ingest
  /*
  {
    "rejections": {
      "id": "wb_abc123", // The workbook ID
      "sheets": [
        {
          "sheetId": "us_sh_xyz456", // The sheet ID
          "rejectedRecords": [
            {
              "id": "dev_rc_123", // The record ID
              "values": [
                {
                  "field": "email",
                  "message": "This email address is already in use."
                }
              ]
            }
          ]
        }
      ]
    }
  }
  */

  // No special client-side configuration needed for rejections
  import { listener } from '@flatfile/listener';
  import { webhookEgress } from '@flatfile/plugin-webhook-egress';

  listener.use(webhookEgress('workbook:submit', 'https://webhook.site/your-unique-url'));
  ```

  ```typescript TypeScript theme={null}
  // Example webhook endpoint response format
  // POST https://api.myapp.com/data-ingest
  /*
  {
    "rejections": {
      "id": "wb_abc123", // The workbook ID
      "sheets": [
        {
          "sheetId": "us_sh_xyz456", // The sheet ID
          "rejectedRecords": [
            {
              "id": "dev_rc_123", // The record ID
              "values": [
                {
                  "field": "email",
                  "message": "This email address is already in use."
                }
              ]
            }
          ]
        }
      ]
    }
  }
  */

  // No special client-side configuration needed for rejections
  import { listener } from '@flatfile/listener';
  import { webhookEgress } from '@flatfile/plugin-webhook-egress';

  listener.use(webhookEgress('workbook:submit', 'https://webhook.site/your-unique-url'));
  ```
</CodeGroup>

## Troubleshooting

### Common Error Messages

**"Failed to submit data to \[webhook]"**

* Check that the webhook URL is correct
* Verify the server is running and accessible
* Ensure the webhook is not returning error status codes (4xx or 5xx)

**"Error posting data to webhook"**

* Check for network connectivity issues
* Verify DNS resolution for the webhook URL
* Confirm the webhook URL is properly formatted

**"Request succeeded but failed to parse response from webhook"**

* Your webhook returns a 200 OK status but the response body is not valid JSON
* Ensure your endpoint returns `Content-Type: application/json`
* Return at least an empty JSON object `{}` if no specific response is needed

### Error Handling Patterns

The plugin handles errors internally and reports them as job outcomes:

* **Non-200 HTTP status codes**: The job completes with a failure message showing the status code
* **Network errors**: The job fails with a generic "Error posting data to webhook" message
* **Invalid JSON responses**: The job completes with a success status but warns about parsing issues

## Notes

### Requirements and Limitations

* An action must be configured in the workbook with an `operation` that matches the `job` string passed to the plugin
* The webhook endpoint must accept POST requests with `Content-Type: application/json`
* The entire workbook's data (all sheets and all records) is sent in a single request, which may result in large payloads for extensive workbooks

### Dependencies

* The plugin depends on `@flatfile/plugin-job-handler` for job lifecycle management (bundled with the plugin)
* Uses `@flatfile/util-response-rejection` for processing webhook rejection responses
* Utilizes `@flatfile/util-common` for error logging

### Validation Flow

The plugin supports a two-way validation flow where your webhook can return validation errors in the `rejections` format. These errors are automatically processed and displayed in the Flatfile UI, allowing users to see and correct data issues identified by your backend systems.


# Webhook Event Forwarder
Source: https://flatfile.com/docs/plugins/webhook-event-forwarder

Forward all Flatfile events to a webhook URL for language-agnostic integration with external systems

The Webhook Event Forwarder plugin for Flatfile is designed to capture all events within a Flatfile listener and forward them to a specified webhook URL. This allows developers to process Flatfile events in external systems, regardless of the programming language used at the endpoint.

Its primary purpose is to enable language-agnostic integration with backend services. Common use cases include:

* Triggering custom workflows in services like Zapier or Make
* Storing event data in an external database or data warehouse for analytics
* Notifying external systems of user actions within Flatfile (e.g., data submission, file upload)
* Implementing custom validation or data processing logic on a separate server

The plugin listens for all events (`**`), packages the full event object as a JSON payload, and sends it via an HTTP POST request.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-webhook-event-forwarder
```

## Configuration & Parameters

### webhookEventForward(url, callback?, options?)

| Parameter       | Type       | Required | Description                                                                                                                                                                 |
| --------------- | ---------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url`           | `string`   | Yes      | The webhook endpoint URL where the plugin will send all Flatfile events via an HTTP POST request                                                                            |
| `callback`      | `function` | No       | An optional callback function that is executed after the webhook request is complete. Receives `data` (webhook response) and `event` (original Flatfile event) as arguments |
| `options`       | `object`   | No       | Additional configuration options                                                                                                                                            |
| `options.debug` | `boolean`  | No       | When set to `true`, logs any errors that occur during the webhook forwarding process to the console. Default: `false`                                                       |

### Default Behavior

By default, the plugin listens for every event (`**`) emitted by the Flatfile listener. For each event, it sends an HTTP POST request to the provided `url` with the event object serialized as a JSON string in the request body. If the webhook call fails, the error is suppressed unless `options.debug` is set to `true`. No action is taken on the webhook's response unless a `callback` function is provided.

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from './listener'; // Your listener instance
  import { webhookEventForward } from '@flatfile/plugin-webhook-event-forwarder';

  listener.use(webhookEventForward('https://webhook.site/your-unique-id'));
  ```

  ```typescript TypeScript theme={null}
  import { listener } from './listener'; // Your listener instance
  import { webhookEventForward } from '@flatfile/plugin-webhook-event-forwarder';

  listener.use(webhookEventForward('https://webhook.site/your-unique-id'));
  ```
</CodeGroup>

### Configuration with Callback and Debug

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from './listener'; // Your listener instance
  import { webhookEventForward } from '@flatfile/plugin-webhook-event-forwarder';

  const myCallback = (response, event) => {
    console.log(`Received response for event: ${event.topic}`);
    console.log('Webhook response data:', response);
  };

  const options = {
    debug: true
  };

  listener.use(webhookEventForward('https://webhook.site/your-unique-id', myCallback, options));
  ```

  ```typescript TypeScript theme={null}
  import { listener } from './listener'; // Your listener instance
  import { webhookEventForward } from '@flatfile/plugin-webhook-event-forwarder';
  import type { FlatfileEvent } from '@flatfile/listener';

  const myCallback = (response: any, event: FlatfileEvent) => {
    console.log(`Received response for event: ${event.topic}`);
    console.log('Webhook response data:', response);
  };

  const options = {
    debug: true
  };

  listener.use(webhookEventForward('https://webhook.site/your-unique-id', myCallback, options));
  ```
</CodeGroup>

### Advanced Usage with Async Callback

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from './listener'; // Your listener instance
  import { webhookEventForward } from '@flatfile/plugin-webhook-event-forwarder';

  const handleWebhookResponse = async (response, event) => {
    if (response?.error) {
      console.error(`Failed to process event ${event.id} at webhook. Error: ${response.message}`);
      // You could add logic here to retry the operation or send a notification
    } else {
      console.log(`Event ${event.id} successfully processed by webhook.`);
      // You could update a record in your own database with the response
    }
  };

  listener.use(webhookEventForward('https://api.my-service.com/flatfile-hook', handleWebhookResponse));
  ```

  ```typescript TypeScript theme={null}
  import { listener } from './listener'; // Your listener instance
  import { webhookEventForward } from '@flatfile/plugin-webhook-event-forwarder';
  import type { FlatfileEvent } from '@flatfile/listener';

  const handleWebhookResponse = async (response: any, event: FlatfileEvent) => {
    if (response?.error) {
      console.error(`Failed to process event ${event.id} at webhook. Error: ${response.message}`);
      // You could add logic here to retry the operation or send a notification
    } else {
      console.log(`Event ${event.id} successfully processed by webhook.`);
      // You could update a record in your own database with the response
    }
  };

  listener.use(webhookEventForward('https://api.my-service.com/flatfile-hook', handleWebhookResponse));
  ```
</CodeGroup>

### Error Handling Example

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from './listener'; // Your listener instance
  import { webhookEventForward } from '@flatfile/plugin-webhook-event-forwarder';

  const myCallback = (response, event) => {
    // Check for the error object passed by the plugin on failure
    if (response && response.error === true) {
      console.error(`Webhook forwarding failed for event: ${event.topic}`);
      console.error(`Message: ${response.message}`);
      console.error(`Details: ${response.data}`);
    } else {
      console.log(`Webhook for event ${event.topic} succeeded.`);
    }
  };

  // Use a URL that is expected to fail to demonstrate error handling
  // and enable debug logging to see the error in the console.
  listener.use(webhookEventForward('https://httpstat.us/500', myCallback, { debug: true }));
  ```

  ```typescript TypeScript theme={null}
  import { listener } from './listener'; // Your listener instance
  import { webhookEventForward } from '@flatfile/plugin-webhook-event-forwarder';
  import type { FlatfileEvent } from '@flatfile/listener';

  const myCallback = (response: any, event: FlatfileEvent) => {
    // Check for the error object passed by the plugin on failure
    if (response && response.error === true) {
      console.error(`Webhook forwarding failed for event: ${event.topic}`);
      console.error(`Message: ${response.message}`);
      console.error(`Details: ${response.data}`);
    } else {
      console.log(`Webhook for event ${event.topic} succeeded.`);
    }
  };

  // Use a URL that is expected to fail to demonstrate error handling
  // and enable debug logging to see the error in the console.
  listener.use(webhookEventForward('https://httpstat.us/500', myCallback, { debug: true }));
  ```
</CodeGroup>

## Troubleshooting

To troubleshoot issues:

1. **Enable debug mode**: Set `debug: true` in the options object to see detailed error messages in your console logs.

2. **Verify URL accessibility**: Ensure that the `url` provided is correct and is publicly accessible from the environment where your Flatfile listener is running.

3. **Check webhook endpoint logs**: Review the server logs of your webhook endpoint to see if it's receiving requests and if it's returning any errors.

4. **Test with webhook.site**: Use a service like `webhook.site` to test if events are being sent correctly from the plugin.

## Notes

### Important Considerations

* **Event Volume**: The plugin listens for `**` (all events), which can generate a high volume of HTTP requests to your endpoint. Ensure your webhook receiver can handle the potential load.

* **Synchronous vs. Asynchronous**: The plugin sends the webhook request and continues. While the callback function can be `async` and is `await`ed, the plugin does not block the Flatfile event lifecycle.

* **Security**: The plugin sends the full event payload, which may contain sensitive data. Ensure your webhook endpoint is secure (uses HTTPS) and properly authenticated if necessary. The plugin itself does not add any authentication headers; this would need to be handled by the receiving endpoint or by wrapping the fetch call.

### Error Handling Patterns

The plugin uses a `try...catch` block to handle errors during the `fetch` call. If an error occurs (including non-ok HTTP responses like 4xx or 5xx), it will not crash the listener process. Instead, it constructs a standardized error object:

```json theme={null}
{
  "error": true,
  "message": "Error received, please try again",
  "data": "error details"
}
```

This object is then passed as the first argument to the `callback` function, if provided. If `options.debug` is `true`, the original error is also logged to the console.


# What3Words Converter Plugin
Source: https://flatfile.com/docs/plugins/what3words

Automatically convert What3Words addresses into standard geographic data including country codes, nearest places, and latitude/longitude coordinates using the official What3Words API.

The What3Words plugin for Flatfile automatically converts What3Words (W3W) addresses into standard geographic data. It listens for data commits, takes a W3W address from a specified field in a sheet, and uses the official What3Words API to fetch the corresponding country, nearest place, and latitude/longitude coordinates. The plugin then populates this information into other specified fields within the same record.

This is useful for any data import workflow where users provide locations using the What3Words system, and the application requires standard address components or geographic coordinates for mapping, logistics, or data normalization.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-convert-what3words
```

## Configuration & Parameters

The plugin is configured through the `convertWhat3words` function, which accepts a configuration object with the following parameters:

### Required Parameters

<ParamField type="string">
  The key of the field in your sheet that contains the What3Words address to be converted.
</ParamField>

<ParamField type="string">
  The key of the field where the resulting country code (e.g., "GB") will be stored.
</ParamField>

<ParamField type="string">
  The key of the field where the name of the nearest place (e.g., "London") will be stored.
</ParamField>

<ParamField type="string">
  The key of the field where the resulting latitude coordinate will be stored.
</ParamField>

<ParamField type="string">
  The key of the field where the resulting longitude coordinate will be stored.
</ParamField>

### Optional Parameters

<ParamField type="string">
  The slug of the specific sheet you want this plugin to apply to. If not provided, the plugin will apply to all sheets in the workbook that have a field matching the `what3wordsField` configuration.
</ParamField>

## Usage Examples

### Basic Usage

Apply the What3Words converter to a specific sheet:

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { convertWhat3words } from '@flatfile/plugin-convert-what3words';

  export default function (listener) {
    listener.use(
      convertWhat3words({
        sheetSlug: 'locations',
        what3wordsField: 'w3w_address',
        countryField: 'country_code',
        nearestPlaceField: 'city',
        latField: 'latitude',
        longField: 'longitude'
      })
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { convertWhat3words } from '@flatfile/plugin-convert-what3words';

  export default function (listener: FlatfileListener) {
    listener.use(
      convertWhat3words({
        sheetSlug: 'locations',
        what3wordsField: 'w3w_address',
        countryField: 'country_code',
        nearestPlaceField: 'city',
        latField: 'latitude',
        longField: 'longitude'
      })
    );
  }
  ```
</CodeGroup>

### Apply to All Sheets

Configure the plugin to run on all sheets by omitting the `sheetSlug` option:

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { convertWhat3words } from '@flatfile/plugin-convert-what3words';

  export default function (listener) {
    // Applies to all sheets with a 'what3words' field
    listener.use(
      convertWhat3words({
        what3wordsField: 'what3words',
        countryField: 'country',
        nearestPlaceField: 'nearestPlace',
        latField: 'latitude',
        longField: 'longitude'
      })
    );
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { convertWhat3words } from '@flatfile/plugin-convert-what3words';

  export default function (listener: FlatfileListener) {
    // Applies to all sheets with a 'what3words' field
    listener.use(
      convertWhat3words({
        what3wordsField: 'what3words',
        countryField: 'country',
        nearestPlaceField: 'nearestPlace',
        latField: 'latitude',
        longField: 'longitude'
      })
    );
  }
  ```
</CodeGroup>

## Troubleshooting

### Conversions Not Working

* **API Key**: Verify that the `W3W_API_KEY` secret is correctly set in your Flatfile Space
* **Configuration**: Double-check that the `sheetSlug` and all field keys (`what3wordsField`, `countryField`, etc.) in your configuration exactly match the schema of your sheet
* **Data Format**: Ensure the field specified in `what3wordsField` contains valid, correctly formatted What3Words addresses

### Error Messages

If the What3Words API call fails for any reason (e.g., invalid W3W address, network error, invalid API key), the plugin will add a record-level error to the `what3wordsField` with the message "Invalid What3Words address". This error will be visible to the user in the Flatfile UI.

## Notes

### Environment Setup

You must add your What3Words API key as an environment secret named `W3W_API_KEY` in your Flatfile Space before using this plugin.

### Default Behavior

* **Sheet Targeting**: When `sheetSlug` is not provided, the plugin applies to all sheets in the workbook that contain a field matching the `what3wordsField` configuration
* **Event Trigger**: The plugin operates using `bulkRecordHook`, which runs when the user clicks "Continue" or "Submit" on a sheet (during the `commit:created` event)
* **Processing**: The conversion is not real-time as the user types; it occurs during the commit process

### Error Handling

* The plugin processes records individually within a batch
* A failure on one record will not stop the processing of other records
* Errors are logged to the console for debugging purposes
* User-facing errors are attached directly to the record and field that caused the issue

### Idempotency

The plugin processes records on each commit. If data is re-committed, the conversion will run again, potentially overwriting previously converted data.


# Excel Extractor
Source: https://flatfile.com/docs/plugins/xlsx-extractor

Parse various Excel file formats (.xls, .xlsx, .xlsm, .xlsb, .xltx, .xltm) and extract structured data with support for header detection, merged cells, and hierarchical spreadsheets.

The Excel Extractor plugin is designed to parse various Excel file formats (.xls, .xlsx, .xlsm, .xlsb, .xltx, .xltm) and extract structured data from them. When a user uploads a supported Excel file, this plugin automatically processes it, detects headers, and transforms the sheet data into a format that Flatfile can use. It offers extensive configuration for handling complex Excel files, including options for header detection, processing merged cells, and cascading data in hierarchical spreadsheets. This plugin is intended to be used in a server-side listener within the Flatfile platform.

## Installation

Install the Excel Extractor plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-xlsx-extractor
```

## Configuration & Parameters

The Excel Extractor accepts the following configuration options:

### Basic Options

| Parameter        | Type    | Default     | Description                                                   |
| ---------------- | ------- | ----------- | ------------------------------------------------------------- |
| `raw`            | boolean | `false`     | Extract raw, underlying cell values instead of formatted text |
| `rawNumbers`     | boolean | `false`     | Extract raw numeric values instead of formatted numbers       |
| `dateNF`         | string  | `undefined` | Specific date format string for interpreting dates            |
| `chunkSize`      | number  | `10000`     | Number of records to process in each batch                    |
| `parallel`       | number  | `1`         | Number of chunks to process concurrently                      |
| `skipEmptyLines` | boolean | `false`     | Skip rows that are entirely empty                             |
| `debug`          | boolean | `false`     | Enable verbose logging for troubleshooting                    |

### Advanced Options

| Parameter                | Type    | Default     | Description                                                   |
| ------------------------ | ------- | ----------- | ------------------------------------------------------------- |
| `cascadeRowValues`       | boolean | `false`     | Fill empty cells with values from the cell above              |
| `cascadeHeaderValues`    | boolean | `false`     | Fill empty header cells with values from the cell to the left |
| `headerDetectionOptions` | object  | See below   | Configure header row detection                                |
| `mergedCellOptions`      | object  | `undefined` | Define how to handle merged cells                             |

### Header Detection Options

Default configuration:

```javascript theme={null}
{
  algorithm: 'default',
  rowsToSearch: 10
}
```

Available algorithms:

* `'default'` - Scans first 10 rows and selects the one with most non-empty cells
* `'explicitHeaders'` - Use when headers are explicitly defined
* `'specificRows'` - Define specific row numbers as headers
* `'dataRowAndSubHeaderDetection'` - Advanced detection for complex header structures

### Merged Cell Options

Configure treatment of merged cells with these options:

| Treatment        | Description                                      |
| ---------------- | ------------------------------------------------ |
| `applyToAll`     | Copy merged cell value to all cells in the range |
| `applyToTopLeft` | Keep value only in top-left cell                 |
| `coalesce`       | Keep first row/column and remove others          |
| `concatenate`    | Combine values with a separator                  |

## Usage Examples

<CodeGroup>
  ```javascript JavaScript - Basic Usage theme={null}
  import { listener } from '@flatfile/listener';
  import { ExcelExtractor } from '@flatfile/plugin-xlsx-extractor';

  export default function (listener) {
    listener.use(ExcelExtractor());
  }
  ```

  ```typescript TypeScript - Basic Usage theme={null}
  import { listener } from '@flatfile/listener';
  import { ExcelExtractor } from '@flatfile/plugin-xlsx-extractor';
  import type { FlatfileListener } from '@flatfile/listener';

  export default function (listener: FlatfileListener) {
    listener.use(ExcelExtractor());
  }
  ```
</CodeGroup>

### Configuration Example

<CodeGroup>
  ```javascript JavaScript - Configuration theme={null}
  import { listener } from '@flatfile/listener';
  import { ExcelExtractor } from '@flatfile/plugin-xlsx-extractor';

  export default function (listener) {
    listener.use(
      ExcelExtractor({
        rawNumbers: true,
        skipEmptyLines: true,
        chunkSize: 50000,
        debug: true,
      })
    );
  }
  ```

  ```typescript TypeScript - Configuration theme={null}
  import { listener } from '@flatfile/listener';
  import { ExcelExtractor } from '@flatfile/plugin-xlsx-extractor';
  import type { FlatfileListener } from '@flatfile/listener';

  export default function (listener: FlatfileListener) {
    listener.use(
      ExcelExtractor({
        rawNumbers: true,
        skipEmptyLines: true,
        chunkSize: 50000,
        debug: true,
      })
    );
  }
  ```
</CodeGroup>

### Advanced Header Detection

<CodeGroup>
  ```javascript JavaScript - Specific Rows theme={null}
  import { listener } from '@flatfile/listener';
  import { ExcelExtractor } from '@flatfile/plugin-xlsx-extractor';

  export default function (listener) {
    listener.use(
      ExcelExtractor({
        headerDetectionOptions: {
          algorithm: 'specificRows',
          rowNumbers: [2], // 0-based index for the third row
        },
      })
    );
  }
  ```

  ```typescript TypeScript - Specific Rows theme={null}
  import { listener } from '@flatfile/listener';
  import { ExcelExtractor } from '@flatfile/plugin-xlsx-extractor';
  import type { FlatfileListener } from '@flatfile/listener';

  export default function (listener: FlatfileListener) {
    listener.use(
      ExcelExtractor({
        headerDetectionOptions: {
          algorithm: 'specificRows',
          rowNumbers: [2], // 0-based index for the third row
        },
      })
    );
  }
  ```
</CodeGroup>

### Merged Cell Handling

<CodeGroup>
  ```javascript JavaScript - Merged Cells theme={null}
  import { listener } from '@flatfile/listener';
  import { ExcelExtractor } from '@flatfile/plugin-xlsx-extractor';

  export default function (listener) {
    listener.use(
      ExcelExtractor({
        mergedCellOptions: {
          acrossColumns: {
            treatment: 'concatenate',
            separator: ' ',
          },
          acrossRows: {
            treatment: 'applyToAll',
          },
        },
      })
    );
  }
  ```

  ```typescript TypeScript - Merged Cells theme={null}
  import { listener } from '@flatfile/listener';
  import { ExcelExtractor } from '@flatfile/plugin-xlsx-extractor';
  import type { FlatfileListener } from '@flatfile/listener';

  export default function (listener: FlatfileListener) {
    listener.use(
      ExcelExtractor({
        mergedCellOptions: {
          acrossColumns: {
            treatment: 'concatenate',
            separator: ' ',
          },
          acrossRows: {
            treatment: 'applyToAll',
          },
        },
      })
    );
  }
  ```
</CodeGroup>

### Direct Parser Usage

<CodeGroup>
  ```javascript JavaScript - Direct Parser theme={null}
  import * as fs from 'fs';
  import { excelParser } from '@flatfile/plugin-xlsx-extractor';

  async function parseMyFile(filePath) {
    try {
      const fileBuffer = fs.readFileSync(filePath);
      const workbookData = await excelParser(fileBuffer, {
        skipEmptyLines: true,
      });
      console.log(workbookData['Sheet1'].data);
    } catch (error) {
      console.error('Failed to parse file:', error);
    }
  }
  ```

  ```typescript TypeScript - Direct Parser theme={null}
  import * as fs from 'fs';
  import { excelParser } from '@flatfile/plugin-xlsx-extractor';
  import type { WorkbookCapture } from '@flatfile/plugin-xlsx-extractor';

  async function parseMyFile(filePath: string): Promise<WorkbookCapture | null> {
    try {
      const fileBuffer = fs.readFileSync(filePath);
      const workbookData = await excelParser(fileBuffer, {
        skipEmptyLines: true,
      });
      console.log(workbookData['Sheet1'].data);
      return workbookData;
    } catch (error) {
      console.error('Failed to parse file:', error);
      return null;
    }
  }
  ```
</CodeGroup>

## Troubleshooting

### Large File Handling

<CodeGroup>
  ```javascript JavaScript - Error Handling theme={null}
  import { excelParser } from '@flatfile/plugin-xlsx-extractor';

  async function parseLargeFile(buffer) {
    try {
      const workbookData = await excelParser(buffer);
      return workbookData;
    } catch (error) {
      if (error.message === 'plugins.extraction.fileTooLarge') {
        console.error('The file is too large to be processed. Please convert it to CSV first.');
      } else {
        console.error('An unexpected error occurred:', error);
      }
    }
  }
  ```

  ```typescript TypeScript - Error Handling theme={null}
  import { excelParser } from '@flatfile/plugin-xlsx-extractor';
  import type { WorkbookCapture } from '@flatfile/plugin-xlsx-extractor';

  async function parseLargeFile(buffer: Buffer): Promise<WorkbookCapture | null> {
    try {
      const workbookData = await excelParser(buffer);
      return workbookData;
    } catch (error: any) {
      if (error.message === 'plugins.extraction.fileTooLarge') {
        console.error('The file is too large to be processed. Please convert it to CSV first.');
      } else {
        console.error('An unexpected error occurred:', error);
      }
      return null;
    }
  }
  ```
</CodeGroup>

### Debug Mode

Enable debug mode for detailed logging:

```javascript theme={null}
ExcelExtractor({
  debug: true
})
```

This provides detailed logs about the extraction process, including detected headers, rows processed, and configuration options being applied.

## Notes

### Default Behavior

* **Header Detection**: By default, the plugin scans the first 10 rows and selects the one with the most non-empty cells as the header row
* **Empty Rows**: Empty rows are included as empty records unless `skipEmptyLines` is set to `true`
* **Merged Cells**: Handled by the underlying library's default behavior unless custom options are provided
* **Chunk Processing**: Data is processed in batches of 10,000 records by default

### Important Considerations

* **Server-Side Only**: This plugin is designed to run in a server-side environment and should be used within a Flatfile listener
* **Duplicate Headers**: If a sheet contains duplicate column headers, the plugin automatically makes them unique by appending a suffix (e.g., 'Name', 'Name\_1', 'Name\_2')
* **Empty Headers**: Empty header cells are renamed to 'empty' (e.g., 'empty', 'empty\_1')
* **Trailing Empty Rows**: The parser automatically trims any fully empty rows from the end of a sheet before processing
* **Memory Limitations**: The plugin has built-in handling for extremely large files that can cause memory issues, throwing a user-friendly error when files are too large

### Cascading Behavior

* **Row Cascading**: When `cascadeRowValues` is enabled, empty cells are filled with values from the cell above. The cascade resets on a completely blank row or a new value in the column
* **Header Cascading**: When `cascadeHeaderValues` is enabled, empty header cells are filled with values from the cell to the left. The cascade resets on a blank column or a new value


# XML Extractor
Source: https://flatfile.com/docs/plugins/xml-extractor

A Flatfile plugin for parsing XML files, flattening nested structures and attributes, and converting them into tabular format for easy import into Flatfile Sheets.

The XML Extractor plugin processes .xml files uploaded to Flatfile by parsing XML data, flattening its nested structure and attributes, and converting it into a tabular format. The plugin automatically detects the main collection of records within the XML file and transforms each record into a row, making it ideal for importing structured XML data from legacy systems, APIs, or data exports.

## Installation

Install the XML Extractor plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-xml-extractor
```

## Configuration & Parameters

The XML Extractor accepts the following configuration options:

| Parameter         | Type     | Default     | Description                                                           |
| ----------------- | -------- | ----------- | --------------------------------------------------------------------- |
| `separator`       | string   | `"/"`       | Character used to separate keys when flattening nested XML elements   |
| `attributePrefix` | string   | `"#"`       | Prefix added to keys derived from XML element attributes              |
| `transform`       | function | `undefined` | Function to transform each parsed data row before passing to Flatfile |
| `chunkSize`       | number   | `10000`     | Number of records to process in each batch                            |
| `parallel`        | number   | `1`         | Number of chunks to process concurrently                              |
| `debug`           | boolean  | `false`     | Enable verbose logging for debugging                                  |

### Default Behavior

By default, the plugin:

* Flattens nested elements using `"/"` as a separator (e.g., `<country><name>USA</name></country>` becomes `"country/name"`)
* Prefixes attributes with `"#"` (e.g., `<country code="US">` becomes `"country#code"`)
* Processes files in chunks of 10,000 records with sequential processing
* Automatically infers headers from all unique keys found across records

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from '@flatfile/listener';
  import { XMLExtractor } from '@flatfile/plugin-xml-extractor';

  export default listener.on('file:created', (event) => {
    // Only handle .xml files
    if (event.context.file.ext !== 'xml') {
      return;
    }
    return XMLExtractor()(event);
  });
  ```

  ```typescript TypeScript theme={null}
  import { listener } from '@flatfile/listener';
  import { XMLExtractor } from '@flatfile/plugin-xml-extractor';
  import type { FlatfileEvent } from '@flatfile/listener';

  export default listener.on('file:created', (event: FlatfileEvent) => {
    // Only handle .xml files
    if (event.context.file.ext !== 'xml') {
      return;
    }
    return XMLExtractor()(event);
  });
  ```
</CodeGroup>

### Custom Configuration

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { listener } from '@flatfile/listener';
  import { XMLExtractor } from '@flatfile/plugin-xml-extractor';

  export default listener.on('file:created', (event) => {
    if (event.context.file.ext !== 'xml') {
      return;
    }
    return XMLExtractor({
      separator: '_',
      attributePrefix: '@',
      transform: (row) => {
        if (row.age) {
          row.age = parseInt(row.age, 10);
        }
        return row;
      },
      chunkSize: 5000,
      parallel: 2,
    })(event);
  });
  ```

  ```typescript TypeScript theme={null}
  import { listener } from '@flatfile/listener';
  import { XMLExtractor } from '@flatfile/plugin-xml-extractor';
  import type { FlatfileEvent } from '@flatfile/listener';

  export default listener.on('file:created', (event: FlatfileEvent) => {
    if (event.context.file.ext !== 'xml') {
      return;
    }
    return XMLExtractor({
      separator: '_',
      attributePrefix: '@',
      transform: (row: Record<string, any>) => {
        if (row.age) {
          row.age = parseInt(row.age, 10);
        }
        return row;
      },
      chunkSize: 5000,
      parallel: 2,
    })(event);
  });
  ```
</CodeGroup>

### Using the Standalone Parser

<CodeGroup>
  ```javascript JavaScript theme={null}
  import * as fs from 'fs';
  import { xmlParser } from '@flatfile/plugin-xml-extractor';

  const xmlBuffer = fs.readFileSync('path/to/your/file.xml');
  const workbookData = xmlParser(xmlBuffer, { separator: '.' });

  console.log(workbookData.Sheet1.headers);
  console.log(workbookData.Sheet1.data[0]);
  ```

  ```typescript TypeScript theme={null}
  import * as fs from 'fs';
  import { xmlParser } from '@flatfile/plugin-xml-extractor';

  const xmlBuffer: Buffer = fs.readFileSync('path/to/your/file.xml');
  const workbookData = xmlParser(xmlBuffer, { separator: '.' });

  console.log(workbookData.Sheet1.headers);
  console.log(workbookData.Sheet1.data[0]);
  ```
</CodeGroup>

### Error Handling

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { xmlParser } from '@flatfile/plugin-xml-extractor';

  try {
    const emptyXml = Buffer.from('<?xml version="1.0"?>');
    xmlParser(emptyXml);
  } catch (e) {
    console.error(e.message); // "No root xml object found"
  }
  ```

  ```typescript TypeScript theme={null}
  import { xmlParser } from '@flatfile/plugin-xml-extractor';

  try {
    const emptyXml: Buffer = Buffer.from('<?xml version="1.0"?>');
    xmlParser(emptyXml);
  } catch (e: any) {
    console.error(e.message); // "No root xml object found"
  }
  ```
</CodeGroup>

## Troubleshooting

### Fields Not Appearing as Expected

Check the `separator` and `attributePrefix` configuration. The default flattening behavior might not match your desired output format.

### Handling Repeated Tags

If tags within a record are repeated (e.g., multiple `<email>` tags), the extractor creates indexed headers. The first email tag becomes the `email` field, the second becomes `email/0`, the third `email/1`, and so on.

### Debugging Row Data

Use the `transform` function to inspect intermediate parsed output:

<CodeGroup>
  ```javascript JavaScript theme={null}
  XMLExtractor({
    transform: (row) => {
      console.log(row);
      return row;
    }
  })
  ```

  ```typescript TypeScript theme={null}
  XMLExtractor({
    transform: (row: Record<string, any>) => {
      console.log(row);
      return row;
    }
  })
  ```
</CodeGroup>

## Notes

* The plugin is designed for server-side environments and should be used in Flatfile listeners deployed on the Platform
* XML files must have a single root element containing an array of similar child elements (records)
* Not suitable for arbitrary or document-style XML with highly irregular structures
* Headers are automatically inferred from all unique keys found across all records
* The primary error condition is invalid XML structure - if no root element containing records is found, the parser throws "No root xml object found"


# YAML Schema to Flatfile Blueprint Converter
Source: https://flatfile.com/docs/plugins/yaml-schema

Automates the creation of a Flatfile Blueprint (Workbook and Sheets) from one or more YAML schema definitions, streamlining the setup of a Flatfile Space by converting existing data models into a Flatfile-compatible format.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-convert-yaml-schema
```

## Configuration & Parameters

### ModelToSheetConfig

Configuration object for each Sheet to be created from a YAML schema:

| Parameter             | Type   | Required | Description                                                                                      |
| --------------------- | ------ | -------- | ------------------------------------------------------------------------------------------------ |
| `sourceUrl`           | string | Yes      | The URL where the YAML schema file is hosted                                                     |
| `name`                | string | No       | A custom name for the generated Sheet. If not provided, the `title` from the YAML schema is used |
| Additional properties | -      | No       | Other `Flatfile.SheetConfig` properties (e.g., `actions`) can be included to customize the Sheet |

### Options

Optional configuration object for the Workbook and plugin behavior:

| Parameter        | Type                  | Default | Description                                                                            |
| ---------------- | --------------------- | ------- | -------------------------------------------------------------------------------------- |
| `workbookConfig` | PartialWorkbookConfig | -       | Configuration for the generated Workbook (name, actions, etc.)                         |
| `debug`          | boolean               | `false` | When set to `true`, prints the generated Blueprint configuration object to the console |

### PartialWorkbookConfig

Configuration for the parent Workbook:

| Parameter             | Type   | Description                                                                      |
| --------------------- | ------ | -------------------------------------------------------------------------------- |
| `name`                | string | Custom name for the Workbook. Defaults to "YAML Schema Workbook" if not provided |
| `actions`             | array  | Custom actions for the Workbook                                                  |
| Additional properties | -      | Other properties from `Flatfile.CreateWorkbookConfig`                            |

### Callback Function

Optional function executed after Space and Workbook configuration:

```typescript theme={null}
(event: FlatfileEvent, workbookIds: string[], tick: TickFunction) => any | Promise<any>
```

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { configureSpaceWithYamlSchema } from '@flatfile/plugin-convert-yaml-schema'

  export default function (listener) {
    listener.use(
      configureSpaceWithYamlSchema([
        { sourceUrl: 'http://example.com/schemas/my-schema.yml' },
      ])
    )
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { configureSpaceWithYamlSchema } from '@flatfile/plugin-convert-yaml-schema'

  export default function (listener: FlatfileListener) {
    listener.use(
      configureSpaceWithYamlSchema([
        { sourceUrl: 'http://example.com/schemas/my-schema.yml' },
      ])
    )
  }
  ```
</CodeGroup>

### Advanced Configuration

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import { configureSpaceWithYamlSchema } from '@flatfile/plugin-convert-yaml-schema'

  export default function (listener) {
    listener.use(
      configureSpaceWithYamlSchema(
        [
          {
            sourceUrl: 'https://example.com/schemas/person.yml',
            name: 'Custom Person Sheet',
          },
          {
            sourceUrl: 'https://example.com/schemas/product.yml',
          },
        ],
        {
          workbookConfig: {
            name: 'My Custom Workbook',
            actions: [
              {
                operation: 'submitActionFg',
                mode: 'foreground',
                label: 'Submit all data',
                primary: true,
              },
            ],
          },
          debug: true,
        }
      )
    )
  }
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener'
  import {
    configureSpaceWithYamlSchema,
    ModelToSheetConfig,
    PartialWorkbookConfig,
  } from '@flatfile/plugin-convert-yaml-schema'

  export default function (listener: FlatfileListener) {
    listener.use(
      configureSpaceWithYamlSchema(
        [
          {
            sourceUrl: 'https://example.com/schemas/person.yml',
            name: 'Custom Person Sheet',
          },
          {
            sourceUrl: 'https://example.com/schemas/product.yml',
          },
        ] as ModelToSheetConfig[],
        {
          workbookConfig: {
            name: 'My Custom Workbook',
            actions: [
              {
                operation: 'submitActionFg',
                mode: 'foreground',
                label: 'Submit all data',
                primary: true,
              },
            ],
          } as PartialWorkbookConfig,
          debug: true,
        }
      )
    )
  }
  ```
</CodeGroup>

### Using Callback Function

<CodeGroup>
  ```javascript JavaScript theme={null}
  import api from '@flatfile/api'
  import { FlatfileListener } from '@flatfile/listener'
  import { configureSpaceWithYamlSchema } from '@flatfile/plugin-convert-yaml-schema'

  export default function (listener) {
    const callback = async (event, workbookIds, tick) => {
      const { spaceId } = event.context
      await tick(50, 'Workbook configured, creating welcome document...')
      await api.documents.create(spaceId, {
        title: 'Getting Started',
        body: '<h1>Welcome!</h1><p>Your data templates are ready.</p>',
      })
      await tick(100, 'Document created')
    }

    listener.use(
      configureSpaceWithYamlSchema(
        [{ sourceUrl: 'https://example.com/schemas/person.yml' }],
        { debug: true },
        callback
      )
    )
  }
  ```

  ```typescript TypeScript theme={null}
  import api from '@flatfile/api'
  import { FlatfileEvent, FlatfileListener } from '@flatfile/listener'
  import { TickFunction } from '@flatfile/plugin-job-handler'
  import { configureSpaceWithYamlSchema } from '@flatfile/plugin-convert-yaml-schema'

  export default function (listener: FlatfileListener) {
    const callback = async (
      event: FlatfileEvent,
      workbookIds: string[],
      tick: TickFunction
    ) => {
      const { spaceId } = event.context
      await tick(50, 'Workbook configured, creating welcome document...')
      await api.documents.create(spaceId, {
        title: 'Getting Started',
        body: '<h1>Welcome!</h1><p>Your data templates are ready.</p>',
      })
      await tick(100, 'Document created')
    }

    listener.use(
      configureSpaceWithYamlSchema(
        [{ sourceUrl: 'https://example.com/schemas/person.yml' }],
        { debug: true },
        callback
      )
    )
  }
  ```
</CodeGroup>

## Troubleshooting

### Debug Mode

The most effective troubleshooting tool is the `debug: true` option. When enabled, the complete generated Blueprint object is printed to the console where the listener is running. This allows you to inspect the exact Workbook, Sheet, and Field configuration that the plugin produced.

```javascript theme={null}
listener.use(
  configureSpaceWithYamlSchema(
    [{ sourceUrl: 'https://example.com/schemas/person.yml' }],
    { debug: true }
  )
)
```

### Common Error Scenarios

**Network/URL Issues**: If the plugin fails to fetch a schema from a `sourceUrl` (e.g., due to a network error, 404 Not Found, or invalid URL), it will throw an error with a message like:

Error: Error fetching external reference: API returned status 404: Not Found

**YAML Parsing Errors**: Errors during YAML parsing will bubble up and cause the configuration to fail. Ensure your YAML schema files are valid and properly formatted.

## Notes

### Requirements and Limitations

* This plugin must be used in a server-side listener environment
* It is designed to run on the `space:configure` event
* The YAML schema must be publicly accessible via the provided `sourceUrl`
* The plugin internally converts YAML to JSON and uses `@flatfile/plugin-convert-json-schema`, so supported YAML schema features are limited to what the underlying JSON Schema converter supports

### Default Behavior

* If no `name` is provided for a Sheet, the `title` from the YAML schema is used
* If no `name` is provided for the Workbook, it defaults to "YAML Schema Workbook"
* The `debug` option defaults to `false`

### Error Handling

The plugin's external fetching logic includes error handling for network failures and HTTP errors. When errors occur, they are caught by the Flatfile listener runner and reported in the corresponding Job's execution history.


# Zip Extractor
Source: https://flatfile.com/docs/plugins/zip-extractor

Automatically decompress .zip files uploaded to a Flatfile Space and extract their contents as individual files

The Zip Extractor plugin for Flatfile is designed to automatically decompress `.zip` files uploaded to a Flatfile Space. When a user uploads a zip archive, this plugin intercepts the file, extracts its contents, and uploads each individual file back into the same Space.

This is particularly useful for workflows where users need to bulk-upload multiple files (like CSVs, Excel files, or JSON files) at once. After this plugin runs, other extractor plugins (e.g., `@flatfile/plugin-xlsx-extractor`, `@flatfile/plugin-csv-extractor`) can then process the individual extracted files. The plugin intelligently ignores common system files and directories found in zip archives, such as those starting with a dot (`.`) or `__MACOSX`. It should be deployed in a server-side listener.

## Installation

Install the plugin using npm:

```bash theme={null}
npm install @flatfile/plugin-zip-extractor
```

## Configuration & Parameters

The ZipExtractor plugin accepts an optional configuration object with the following parameter:

| Parameter | Type      | Default | Description                                                                                                                                           |
| --------- | --------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `debug`   | `boolean` | `false` | Enables verbose logging for troubleshooting. When set to `true`, it prints helpful information such as file paths of extracted files being processed. |

### Default Behavior

By default, the plugin runs silently without printing extra diagnostic messages to the console. The plugin automatically filters out system files and directories, including files whose names start with a `.` (e.g., `.DS_Store`) and files within the `__MACOSX` directory.

## Usage Examples

### Basic Usage

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { ZipExtractor } from '@flatfile/plugin-zip-extractor';

  const listener = new FlatfileListener();

  // Register the Zip Extractor with default options
  listener.use(ZipExtractor());

  // When a .zip file is uploaded to Flatfile, the plugin will automatically
  // extract its contents and upload them back to the Space.
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { ZipExtractor } from '@flatfile/plugin-zip-extractor';

  const listener = new FlatfileListener();

  // Register the Zip Extractor with default options
  listener.use(ZipExtractor());

  // When a .zip file is uploaded to Flatfile, the plugin will automatically
  // extract its contents and upload them back to the Space.
  ```
</CodeGroup>

### With Debug Enabled

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { ZipExtractor } from '@flatfile/plugin-zip-extractor';

  const listener = new FlatfileListener();

  // Register the Zip Extractor with debugging enabled
  listener.use(ZipExtractor({ debug: true }));
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { ZipExtractor } from '@flatfile/plugin-zip-extractor';

  const listener = new FlatfileListener();

  // Register the Zip Extractor with debugging enabled
  listener.use(ZipExtractor({ debug: true }));
  ```
</CodeGroup>

### Combined with Other Extractors

<CodeGroup>
  ```javascript JavaScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { ZipExtractor } from '@flatfile/plugin-zip-extractor';
  import { XlsxExtractor } from '@flatfile/plugin-xlsx-extractor';

  const listener = new FlatfileListener();

  // First, use the Zip Extractor to decompress any uploaded .zip files.
  listener.use(ZipExtractor());

  // Then, use the XLSX Extractor to process any .xlsx files,
  // including those that were just extracted from a zip archive.
  listener.use(XlsxExtractor());
  ```

  ```typescript TypeScript theme={null}
  import { FlatfileListener } from '@flatfile/listener';
  import { ZipExtractor } from '@flatfile/plugin-zip-extractor';
  import { XlsxExtractor } from '@flatfile/plugin-xlsx-extractor';

  const listener = new FlatfileListener();

  // First, use the Zip Extractor to decompress any uploaded .zip files.
  listener.use(ZipExtractor());

  // Then, use the XLSX Extractor to process any .xlsx files,
  // including those that were just extracted from a zip archive.
  listener.use(XlsxExtractor());
  ```
</CodeGroup>

## Troubleshooting

The primary tool for troubleshooting is the `debug` option. Setting `debug: true` during initialization will cause the plugin to output detailed logs, including the paths of files it is extracting and uploading, which can help diagnose issues with the extraction process.

If an error occurs during the unzipping or file upload process, the plugin will catch the exception, log it, and fail the associated Flatfile Job with the message "Extraction failed" followed by the original error message. This provides clear feedback in the Flatfile UI.

## Notes

### Server-Side Operation

This plugin is intended to be run in a server-side listener environment, not in the browser.

### Event Handling

The plugin operates on the `file:created` event in Flatfile and specifically targets files with a `.zip` extension, ignoring all other file types.

### File Filtering

The plugin automatically filters out certain files from the archive to avoid clutter:

* Files whose names start with a `.` (e.g., `.DS_Store`)
* Files within the `__MACOSX` directory (common in archives created on macOS)

### Job Tracking

The plugin utilizes `@flatfile/plugin-job-handler` to provide feedback on the extraction progress within the Flatfile UI.

### Deprecated Export

The `zipExtractorPlugin` export is deprecated and kept for backward compatibility. Use `ZipExtractor` in all new implementations.


# CLI Reference
Source: https://flatfile.com/docs/reference/cli

Command line interface for developing, deploying, and managing Flatfile Agents

The Flatfile Command Line Interface (CLI) provides tools to develop, deploy, and manage [Listeners](/core-concepts/listeners) in your Flatfile environment.

<Note>
  Once listeners are deployed and hosted on Flatfile's secure cloud, they are
  called Agents.
</Note>

## Installation

```bash theme={null}
npx flatfile@latest <command>
```

## Configuration

### Authentication

The CLI requires your Flatfile API key and Environment ID, provided either in Environment variables (ideally in a `.env` file) or as command flags. You can find your API key and Environment ID in your Flatfile dashboard under "[API Keys and Secrets](https://platform.flatfile.com/dashboard/keys-and-secrets)".

<Note>
  **Recommended approach:** Use a `.env` file in your project root for secure,
  convenient, and consistent authentication. If you're using Git, make sure to
  add `.env` to your `.gitignore` file.
</Note>

**Using `.env` file**

Create a `.env` file in your project root:

```bash theme={null}
# .env file
FLATFILE_API_KEY="your_api_key_here"
FLATFILE_ENVIRONMENT_ID=your_environment_id_here
```

This approach keeps credentials out of your command history and makes it easy to switch between environments.

**Using command flags**

For one-off commands or CI/CD environments:

```bash theme={null}
npx flatfile develop --token YOUR_API_KEY --env YOUR_ENV_ID
```

### Regional Servers

For improved performance and compliance, Flatfile supports regional deployments:

| Region | API URL                      |
| ------ | ---------------------------- |
| US     | platform.flatfile.com/api    |
| UK     | platform.uk.flatfile.com/api |
| EU     | platform.eu.flatfile.com/api |
| AU     | platform.au.flatfile.com/api |
| CA     | platform.ca.flatfile.com/api |

Set your regional URL in `.env`:

```bash theme={null}
FLATFILE_API_URL=platform.eu.flatfile.com/api
```

<Tip>
  Contact support to enable regional server deployment for your account.
</Tip>

## Development Workflow

<Steps>
  <Step title="Develop Locally">
    Use `develop` to run your listener locally with live reloading
  </Step>

  <Step title="Deploy to Production">
    Use `deploy` to push your listener to Flatfile's cloud as an Agent
  </Step>

  <Step title="Manage Agents">
    Use `agents` commands to list, download, or delete deployed agents
  </Step>
</Steps>

<Warning>
  Use separate environments for development and production to avoid conflicts.
  The CLI will warn you when working in an environment with existing agents.
</Warning>

## Commands

### develop

Run your listener locally with automatic file watching and live reloading.

```bash theme={null}
npx flatfile develop [file-path]
```

**Options**

| Option        | Description                                          |
| ------------- | ---------------------------------------------------- |
| `[file-path]` | Path to listener file (auto-detects if not provided) |
| `--token`     | Flatfile API key                                     |
| `--env`       | Environment ID                                       |

**Features**

* Live reloading on file changes
* Real-time HTTP request logging
* Low-latency event streaming (10-50ms)
* Event handler visibility

**Example output**

```bash theme={null}
> npx flatfile develop
✔ 1 environment(s) found for these credentials
✔ Environment "development" selected
ncc: Version 0.36.1
ncc: Compiling file index.js into CJS
  ✓ 427ms      GET    200 https://platform.flatfile.com/api/v1/subscription 12345

 File change detected. 🚀
  ✓ Connected to event stream for scope us_env_1234
 ▶ commit:created  10:13:05.159 AM us_evt_1234
 ↳ on(**, {})
 ↳ on(commit:created, {"sheetSlug":"contacts"})
```

***

### deploy

Deploy your listener as a Flatfile Agent.

```bash theme={null}
npx flatfile deploy [file-path] [options]
```

**Options**

| Option         | Description                                          |
| -------------- | ---------------------------------------------------- |
| `[file-path]`  | Path to listener file (auto-detects if not provided) |
| `--slug`, `-s` | Unique identifier for the agent                      |
| `--ci`         | Disable interactive prompts for CI/CD                |
| `--token`      | Flatfile API key                                     |
| `--env`        | Environment ID                                       |

**File detection order**

1. `./index.js`
2. `./index.ts`
3. `./src/index.js`
4. `./src/index.ts`

**Examples**

```bash theme={null}
# Basic deployment
npx flatfile deploy

# Deploy with custom slug
npx flatfile deploy --slug my-agent

# CI/CD deployment
npx flatfile deploy ./src/listener.ts --ci
```

**Multiple agents**

Deploy multiple agents to the same environment using unique slugs:

```bash theme={null}
npx flatfile deploy --slug agent-one
npx flatfile deploy --slug agent-two
```

<Note>
  Without a slug, the CLI updates your existing agent or creates one with slug
  `default`.
</Note>

***

### agents list

Display all deployed agents in your environment.

```bash theme={null}
npx flatfile agents list
```

Shows each agent's:

* Agent ID
* Slug
* Deployment status
* Last activity

***

### agents download

Download a deployed agent's source code.

```bash theme={null}
npx flatfile agents download <slug>
```

**Use cases**

* Examine deployed code
* Modify existing agents
* Back up source code
* Debug deployment issues

<Tip>Use `agents list` to find the agent slug you need.</Tip>

***

### agents delete

Remove a deployed agent.

```bash theme={null}
npx flatfile agents delete <slug>
```

**Options**

| Option             | Description                  |
| ------------------ | ---------------------------- |
| `--agentId`, `-ag` | Use agent ID instead of slug |

***

## Related Resources

* [Listeners](/core-concepts/listeners) - Core concept documentation
* [Events](/reference/events) - Event system reference


# Event Reference
Source: https://flatfile.com/docs/reference/events

Complete reference for all Flatfile events, their payloads, and when they are triggered

Flatfile emits events throughout the data import lifecycle, allowing your applications to respond to user actions, system changes, and processing results. This reference documents all available events, their payloads, and when they are triggered.

## Event Structure

All Flatfile events follow a consistent structure. Optional fields may be included depending on the event's Domain ([workbook-level](#workbook-events) events, for instance, won't have a sheetId)

```typescript theme={null}
interface FlatfileEvent {
  id: string;
  topic: string;
  domain: string;
  context: {
    environmentId: string;
    spaceId?: string;
    workbookId?: string;
    sheetId?: string;
    jobId?: string;
    fileId?: string;
    [key: string]: any;
  };
  payload: any;
  attributes?: any;
  createdAt: string;
}
```

## Listening and Reacting to Events

To respond to these events, you'll need to create a [Listener](/core-concepts/listeners) that subscribes to the specific events your application needs to handle.

## Job Events

Job events are triggered when background tasks and operations change state.

### job:created

<ResponseField name="Description">
  Triggered when a new job is first created. Some jobs will enter an optional
  planning state at this time. A job with 'immediate' set to true will skip the
  planning step and transition directly to 'ready.'
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    domain: string       // event domain (e.g., "space", "workbook")
    operation: string    // operation name (e.g., "configure")
    job: string          // domain:operation format (e.g., "space:configure")
    status: string       // job status
    info?: string        // optional info message
    isPart: boolean      // whether this is a sub-job
    input?: any          // job input data
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId?: string
    jobId: string
    actorId: string
  }
  ```
</ResponseField>

### job:ready

<ResponseField name="Description">
  Triggered when a job is ready for execution by your listener. Either the job
  has a complete plan of work or the job is configured to not need a plan. This
  is the only event most job implementations will care about. Once a ready job
  is acknowledged by a listener, it transitions into an executing state.
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    domain: string       // event domain (e.g., "space", "workbook")
    operation: string    // operation name (e.g., "configure")
    job: string          // domain:operation format (e.g., "space:configure")
    status: string       // job status
    info?: string        // optional info message
    isPart: boolean      // whether this is a sub-job
    input?: any          // job input data
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId?: string
    jobId: string
    actorId: string
  }
  ```
</ResponseField>

<ResponseField name="Usage Example">
  ```typescript theme={null}
  listener.filter({ job: "*" }, (configure) => {
    configure.on("job:ready", async (event) => {
      const { jobId } = event.context
      // Handle any job that becomes ready
      await processJob(jobId)
    })
  })
  ```
</ResponseField>

### job:scheduled

<ResponseField name="Description">
  Triggered when a job is scheduled to run at a future time
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    domain: string       // event domain (e.g., "space", "workbook")
    operation: string    // operation name (e.g., "configure")
    job: string          // domain:operation format (e.g., "space:configure")
    status: string       // job status (scheduled)
    info?: string        // optional info message
    isPart: boolean      // whether this is a sub-job
    input?: any          // job input data
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId?: string
    jobId: string
    actorId: string
  }
  ```
</ResponseField>

### job:updated

<ResponseField name="Description">
  Triggered when a job is updated. For example, when a listener updates the
  state or progress of the job. The event will emit many times as the listener
  incrementally completes work and updates the job.
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    domain: string       // event domain (e.g., "space", "workbook")
    operation: string    // operation name (e.g., "configure")
    job: string          // domain:operation format (e.g., "space:configure")
    status: string       // job status
    info?: string        // optional info message
    isPart: boolean      // whether this is a sub-job
    input?: any          // job input data
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId?: string
    jobId: string
    actorId: string
  }
  ```
</ResponseField>

### job:completed

<ResponseField name="Description">
  Triggered when a job has completed successfully
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    domain: string       // event domain (e.g., "space", "workbook")
    operation: string    // operation name (e.g., "configure")
    job: string          // domain:operation format (e.g., "space:configure")
    status: string       // job status
    info?: string        // optional info message
    isPart: boolean      // whether this is a sub-job
    input?: any          // job input data
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId?: string
    jobId: string
    actorId: string
  }
  ```
</ResponseField>

### job:outcome-acknowledged

<ResponseField name="Description">
  Triggered to trigger workflow actions after the user has acknowledged that the
  job has completed or failed. Background jobs will skip this step.
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    domain: string       // event domain (e.g., "space", "workbook")
    operation: string    // operation name (e.g., "configure")
    job: string          // domain:operation format (e.g., "space:configure")
    status: string       // job status
    info?: string        // optional info message
    isPart: boolean      // whether this is a sub-job
    input?: any          // job input data
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId?: string
    workbookId?: string
    sheetId?: string
    jobId: string
    actorId: string
  }
  ```
</ResponseField>

### job:parts-completed

<ResponseField name="Description">
  Triggered when all parts of a multi-part job have completed processing
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    domain: string       // event domain (e.g., "space", "workbook")
    operation: string    // operation name (e.g., "configure")
    job: string          // domain:operation format (e.g., "space:configure")
    status: string       // job status (parts-completed)
    info?: string        // optional info message
    isPart: boolean      // whether this is a sub-job
    input?: any          // job input data
    parts: Array<{       // completed parts information
      partId: string
      status: string
      completedAt: string
    }>
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId?: string
    jobId: string
    actorId: string
  }
  ```
</ResponseField>

### job:failed

<ResponseField name="Description">Triggered when a job fails</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    domain: string       // event domain (e.g., "space", "workbook")
    operation: string    // operation name (e.g., "configure")
    job: string          // domain:operation format (e.g., "space:configure")
    status: string       // job status
    info?: string        // optional info message
    isPart: boolean      // whether this is a sub-job
    input?: any          // job input data
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId?: string
    workbookId?: string
    sheetId?: string
    jobId: string
    actorId: string
  }
  ```
</ResponseField>

### job:deleted

<ResponseField name="Description">
  Triggered when a job is deleted
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId?: string
    jobId: string
    actorId: string
  }
  ```
</ResponseField>

## Program Events

Program events are triggered when mapping programs and transformations change state.

### program:created

<ResponseField name="Description">
  Triggered when a new mapping program is created
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {}
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    workbookId: string
    sheetId: string
    actorId: string
  }
  ```
</ResponseField>

### program:updated

<ResponseField name="Description">
  Triggered when a mapping program is updated
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {}
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    workbookId: string
    sheetId: string
    actorId: string
  }
  ```
</ResponseField>

### program:recomputing

<ResponseField name="Description">
  Triggered when a mapping program begins recomputing its transformations
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {}
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    workbookId: string
    sheetId: string
    actorId: string
  }
  ```
</ResponseField>

### program:recomputed

<ResponseField name="Description">
  Triggered when a mapping program has finished recomputing its transformations
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {}
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    workbookId: string
    sheetId: string
    actorId: string
  }
  ```
</ResponseField>

## File Events

File events are triggered when files are uploaded, processed, or modified.

### file:created

<ResponseField name="Description">
  Triggered when a file upload begins or a new file is created
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    fileId: string
    actorId: string
  }
  ```
</ResponseField>

### file:updated

<ResponseField name="Description">
  Triggered when a file is updated. For example, when a file has been extracted
  into a workbook
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    fileId: string
    actorId: string
  }
  ```
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    status: string
    workbookId?: string
  }
  ```
</ResponseField>

### file:deleted

<ResponseField name="Description">
  Triggered when a file is deleted
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    fileId: string
    actorId: string
  }
  ```
</ResponseField>

### file:expired

<ResponseField name="Description">
  Triggered when a file is expired
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    fileId: string
  }
  ```
</ResponseField>

## Record Events

Record events are triggered when data records are created, updated, or deleted.

### records:created

<ResponseField name="Description">
  Triggered when new records are added to a sheet
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    sheetId: string
    recordIds: string[]
    recordCount: number
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    environmentId: string
    spaceId: string
    workbookId: string
    sheetId: string
  }
  ```
</ResponseField>

### records:updated

<Info>
  **A note on `commit:created` vs `records:updated`**

  You might think you want to listen to `records:updated` for data processing, but **[`commit:created`](#commit-events) is the recommended choice for most automation**.

  A **commit** is like a "git commit" but for data - a versioned snapshot of all changes that occurred together in a single operation. This provides complete transaction context and better performance when processing multiple record changes.

  Flatfile's own Plugins ([Record Hooks](/plugins/record-hook), [Constraints](/plugins/constraints), [Autocast](/plugins/autocast), etc.) all use `commit:created`. Reserve `records:updated` only for real-time per-record feedback scenarios.
</Info>

<ResponseField name="Description">
  Triggered when existing records are modified
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    sheetId: string
    recordIds: string[]
    changes: Array<{
      recordId: string
      fieldKey: string
      previousValue: any
      newValue: any
    }>
  }
  ```
</ResponseField>

### records:deleted

<ResponseField name="Description">
  Triggered when records are deleted from a sheet
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    sheetId: string
    recordIds: string[]
    recordCount: number
  }
  ```
</ResponseField>

## Sheet Events

Sheet events are triggered when sheets are created, modified, or when sheet-level operations occur.

### sheet:calculation-updated

<ResponseField name="Description">
  Triggered when sheet calculations or formulas are updated
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    sheetId: string
    workbookId: string  
    calculations: Array<{
      field: string
      formula: string
      updated: boolean
    }>
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    workbookId: string
    sheetId: string
    actorId: string
  }
  ```
</ResponseField>

### sheet:counts-updated

<ResponseField name="Description">
  Triggered when record counts for a sheet are updated
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    sheetId: string
    workbookId: string
    counts: {
      total: number
      valid: number
      error: number
      updated: number
    }
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    workbookId: string
    sheetId: string
    actorId: string
  }
  ```
</ResponseField>

### sheet:created

<ResponseField name="Description">
  Triggered when a new sheet is created
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    sheetId: string
    workbookId: string
    name: string
    slug: string
    fieldCount: number
  }
  ```
</ResponseField>

### sheet:updated

<ResponseField name="Description">
  Triggered when a sheet Blueprint or configuration is modified
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    sheetId: string
    workbookId: string
    changes: Array<{
      type: "field_added" | "field_removed" | "field_updated" | "config_updated"
      details: any
    }>
  }
  ```
</ResponseField>

### sheet:deleted

<ResponseField name="Description">
  Triggered when a sheet is deleted
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    sheetId: string
    workbookId: string
    name: string
    slug: string
  }
  ```
</ResponseField>

## Workbook Events

Workbook events are triggered for workbook-level operations and changes.

### workbook:created

<ResponseField name="Description">
  Triggered when a new workbook is created
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    workbookId: string
    spaceId: string
    name: string
    namespace?: string
    sheetCount: number
  }
  ```
</ResponseField>

### workbook:updated

<ResponseField name="Description">
  Triggered when a workbook is modified
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    workbookId: string
    spaceId: string
    changes: Array<{
      type: "sheet_added" | "sheet_removed" | "config_updated"
      details: any
    }>
  }
  ```
</ResponseField>

### workbook:deleted

<ResponseField name="Description">
  Triggered when a workbook is deleted
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    workbookId: string
    spaceId: string
  }
  ```
</ResponseField>

### workbook:expired

<ResponseField name="Description">
  Triggered when a workbook expires
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    workbookId: string
    spaceId: string
  }
  ```
</ResponseField>

## Space Events

Space (Project) events are triggered for project lifecycle changes.

### space:created

<ResponseField name="Description">
  Triggered when a new project (space) is created
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    spaceId: string
    environmentId: string
    name: string
    appId?: string
  }
  ```
</ResponseField>

### space:updated

<ResponseField name="Description">
  Triggered when a project is modified
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    spaceId: string
    environmentId: string
    changes: Array<{
      field: string
      previousValue: any
      newValue: any
    }>
  }
  ```
</ResponseField>

### space:deleted

<ResponseField name="Description">
  Triggered when a space is deleted
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    workbookId: string
    actorId: string
  }
  ```
</ResponseField>

### space:expired

<ResponseField name="Description">
  Triggered when a space is expired
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    workbookId: string
  }
  ```
</ResponseField>

### space:archived

<ResponseField name="Description">
  Triggered when a space is archived
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
  }
  ```
</ResponseField>

### space:guestAdded

<ResponseField name="Description">
  Triggered when a guest is added
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    actorId: string
    accountId: string
    environmentId: string
    spaceId: string
  }
  ```
</ResponseField>

### space:guestRemoved

<ResponseField name="Description">
  Triggered when a guest's access is revoked from a space
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    actorId: string
    accountId: string
    environmentId: string
    spaceId: string
  }
  ```
</ResponseField>

### space:unarchived

<ResponseField name="Description">
  Triggered when a space is unarchived and restored to active status
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {}
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    actorId: string
    accountId: string
    environmentId: string
    spaceId: string
  }
  ```
</ResponseField>

## Environment Events

Environment events are triggered for organization-level changes.

### environment:autobuild-created

<ResponseField name="Description">
  Triggered when an autobuild configuration is created for an environment
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {}
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    environmentId: string
    accountId: string
    appId?: string
    actorId: string
  }
  ```
</ResponseField>

### environment:created

<ResponseField name="Description">
  Triggered when a new environment is created
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    environmentId: string
    name: string
    slug: string
    isProd: boolean
  }
  ```
</ResponseField>

### environment:updated

<ResponseField name="Description">
  Triggered when an environment is modified
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    environmentId: string
    changes: Array<{
      field: string
      previousValue: any
      newValue: any
    }>
  }
  ```
</ResponseField>

### environment:deleted

<ResponseField name="Description">
  Triggered when an environment is deleted
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    environmentId: string
    deletedAt: string
    deletedBy: string
  }
  ```
</ResponseField>

## Action Events

Action events are triggered when custom actions are created, updated, or deleted.

### action:created

<ResponseField name="Description">
  Triggered when a new custom action is created
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    actionId: string
    name: string
    label: string
    description?: string
    type: string
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    workbookId?: string
    sheetId?: string
    actorId: string
  }
  ```
</ResponseField>

### action:updated

<ResponseField name="Description">
  Triggered when a custom action is updated
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    actionId: string
    name: string
    label: string
    description?: string
    type: string
    changes: Array<{
      field: string
      previousValue: any
      newValue: any
    }>
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    workbookId?: string
    sheetId?: string
    actorId: string
  }
  ```
</ResponseField>

### action:deleted

<ResponseField name="Description">
  Triggered when a custom action is deleted
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    actionId: string
    name: string
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    workbookId?: string
    sheetId?: string
    actorId: string
  }
  ```
</ResponseField>

## Document Events

Document events are triggered when documents are created, updated, or deleted within workbooks.

### document:created

<ResponseField name="Description">
  Triggered when a document is created on a workbook
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    actorId: string
    spaceId: string
    accountId: string
    documentId: string
    environmentId: string
  }
  ```
</ResponseField>

### document:updated

<ResponseField name="Description">
  Triggered when a document is updated on a workbook
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    actorId: string
    spaceId: string
    accountId: string
    documentId: string
    environmentId: string
  }
  ```
</ResponseField>

### document:deleted

<ResponseField name="Description">
  Triggered when a document is deleted on a workbook
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    actorId: string
    spaceId: string
    accountId: string
    documentId: string
    environmentId: string
  }
  ```
</ResponseField>

## Commit Events

Commit events are triggered when data changes are made to records.

### commit:created

<ResponseField name="Description">
  Triggered when a cell in a record is created or updated
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    sheetId: string
    workbookId: string
    versionId: string
    sheetSlug: string
  }
  ```
</ResponseField>

### commit:updated

<ResponseField name="Description">
  Triggered when commit metadata or details are updated
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    sheetId: string
    workbookId: string
    commitId: string
    versionId: string
    changes: Array<{
      field: string
      previousValue: any
      newValue: any
    }>
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    workbookId: string
    sheetId: string
    actorId: string
  }
  ```
</ResponseField>

### commit:completed

<ResponseField name="Description">
  Triggered when a commit has completed (only when trackChanges is enabled)
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    sheetId: string
    workbookId: string
    versionId: string
    commitId: string
  }
  ```
</ResponseField>

### layer:created

<ResponseField name="Description">
  Triggered when a new layer is created within a commit
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    sheetId: string
    workbookId: string
    layerId: string
    commitId: string
  }
  ```
</ResponseField>

## Snapshot Events

Snapshot events are triggered when snapshots of sheet data are created.

### snapshot:created

<ResponseField name="Description">
  Triggered when a snapshot is created of a sheet
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    snapshotId: string
    sheetId: string
    workbookId: string
    spaceId: string
  }
  ```
</ResponseField>

## Agent Events

Agent events are triggered when agents are created, updated, or deleted.

### agent:created

<ResponseField name="Description">
  Triggered when a new agent is deployed
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    agentId: string
    environmentId: string
  }
  ```
</ResponseField>

### agent:updated

<ResponseField name="Description">
  Triggered when an agent is updated
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    agentId: string
    environmentId: string
  }
  ```
</ResponseField>

### agent:deleted

<ResponseField name="Description">
  Triggered when an agent is deleted
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    agentId: string
    environmentId: string
  }
  ```
</ResponseField>

## Secret Events

Secret events are triggered when secrets are managed.

### secret:created

<ResponseField name="Description">
  Triggered when a new secret is created
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    secretId: string
    spaceId: string
    environmentId: string
  }
  ```
</ResponseField>

### secret:updated

<ResponseField name="Description">
  Triggered when a secret is updated
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    secretId: string
    spaceId: string
    environmentId: string
  }
  ```
</ResponseField>

### secret:deleted

<ResponseField name="Description">
  Triggered when a secret is deleted
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    secretId: string
    spaceId: string
    environmentId: string
  }
  ```
</ResponseField>

## Data Clip Events

Data clip events are triggered when data clips are managed.

### data-clip:collaborator-updated

<ResponseField name="Description">
  Triggered when collaborators are added or removed from a data clip
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    dataClipId: string
    collaborators: string[]
    changes: Array<{
      action: "added" | "removed"
      userId: string
    }>
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    dataClipId: string
  }
  ```
</ResponseField>

### data-clip:created

<ResponseField name="Description">
  Triggered when a new data clip is created
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    dataClipId: string
    accountId: string
    status: string
  }
  ```
</ResponseField>

### data-clip:updated

<ResponseField name="Description">
  Triggered when a data clip's details are updated
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    dataClipId: string
    accountId: string
    status: string
  }
  ```
</ResponseField>

### data-clip:deleted

<ResponseField name="Description">
  Triggered when a data clip is deleted
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    dataClipId: string
    accountId: string
    status: string
  }
  ```
</ResponseField>

### data-clip:resolutions-created

<ResponseField name="Description">
  Triggered when new conflict resolutions are created for a data clip
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    dataClipId: string
    resolutions: Array<{
      conflictId: string
      resolution: any
      resolvedBy: string
    }>
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    dataClipId: string
  }
  ```
</ResponseField>

### data-clip:resolutions-refreshed

<ResponseField name="Description">
  Triggered when conflict resolutions are refreshed or recalculated for a data clip
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {}
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    dataClipId: string
  }
  ```
</ResponseField>

### data-clip:resolutions-updated

<ResponseField name="Description">
  Triggered when existing conflict resolutions are updated for a data clip
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    dataClipId: string
    resolutions: Array<{
      conflictId: string
      resolution: any
      resolvedBy: string
      updatedAt: string
    }>
    changes: Array<{
      conflictId: string
      previousResolution: any
      newResolution: any
    }>
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    dataClipId: string
  }
  ```
</ResponseField>

## Canvas Events

Canvas events are triggered when canvases are created, updated, or deleted.

### canvas:created

<ResponseField name="Description">
  Triggered when a new canvas is created
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    // Full canvas object with all properties
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    canvasId: string
    spaceId: string
    environmentId: string
    accountId: string
  }
  ```
</ResponseField>

### canvas:updated

<ResponseField name="Description">
  Triggered when a canvas is updated
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    // Full canvas object with all properties
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    canvasId: string
    spaceId: string
    environmentId: string
    accountId: string
  }
  ```
</ResponseField>

### canvas:deleted

<ResponseField name="Description">
  Triggered when a canvas is deleted
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    // Full canvas object with all properties
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    canvasId: string
    spaceId: string
    environmentId: string
    accountId: string
  }
  ```
</ResponseField>

## Canvas Area Events

Canvas area events are triggered when canvas areas are created, updated, or deleted.

### canvas-area:created

<ResponseField name="Description">
  Triggered when a new canvas area is created
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    // Full canvas area object with all properties
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    canvasAreaId: string
    canvasId: string
  }
  ```
</ResponseField>

### canvas-area:updated

<ResponseField name="Description">
  Triggered when a canvas area is updated
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    // Full canvas area object with all properties
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    canvasAreaId: string
    canvasId: string
  }
  ```
</ResponseField>

### canvas-area:deleted

<ResponseField name="Description">
  Triggered when a canvas area is deleted
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    // Full canvas area object with all properties
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    canvasAreaId: string
    canvasId: string
  }
  ```
</ResponseField>

## Thread Events

Thread events are triggered when AI conversation threads are created, updated, or deleted.

### thread:created

<ResponseField name="Description">
  Triggered when a new AI conversation thread is created
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    threadId: string
    title?: string
    status: string
    createdAt: string
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    threadId: string
    actorId: string
  }
  ```
</ResponseField>

### thread:updated

<ResponseField name="Description">
  Triggered when an AI conversation thread is updated
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {
    threadId: string
    title?: string
    status: string
    changes: Array<{
      field: string
      previousValue: any
      newValue: any
    }>
  }
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    threadId: string
    actorId: string
  }
  ```
</ResponseField>

### thread:deleted

<ResponseField name="Description">
  Triggered when an AI conversation thread is deleted
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {}
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    accountId: string
    environmentId: string
    spaceId: string
    threadId: string
    actorId: string
  }
  ```
</ResponseField>

## Cron Events

<Warning>
  \*\* Deployed Agents Required \*\*

  Cron events are only created for environments that have deployed agents subscribed to the specific cron topics. These events will not fire in localhost development environments unless you have deployed agents running in that environment.
</Warning>

Cron events are system events triggered at scheduled intervals for automated processes.

### cron:5-minutes

<ResponseField name="Description">
  Triggered every 5 minutes for system maintenance and periodic tasks
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {}
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    environmentId: string
  }
  ```
</ResponseField>

### cron:hourly

<ResponseField name="Description">
  Triggered every hour for scheduled maintenance and cleanup tasks
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {}
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    environmentId: string
  }
  ```
</ResponseField>

### cron:daily

<ResponseField name="Description">
  Triggered once daily for daily maintenance and reporting tasks
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {}
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    environmentId: string
  }
  ```
</ResponseField>

### cron:weekly

<ResponseField name="Description">
  Triggered once weekly for weekly cleanup and archival tasks
</ResponseField>

<ResponseField name="Payload">
  ```typescript theme={null}
  {}
  ```
</ResponseField>

<ResponseField name="Context">
  ```typescript theme={null}
  {
    environmentId: string
  }
  ```
</ResponseField>


# Flatfile Query Language
Source: https://flatfile.com/docs/reference/ffql

Learn how to filter data in Sheets with FFQL.

FFQL (Flatfile Query Language) is Flatfile's custom query language used for filtering data in [Sheets](/core-concepts/blueprints). It's logical, composable, and human-readable.

For example, to find records where the first name is "Bender":

```
first_name eq Bender
```

## Syntax

The basic syntax of FFQL is:

```
[field name] <operator> <value>
```

### Field Name

`field name` is optional and excluding a field name will search across all fields. For example: `eq "Planet Express"` will search across all fields for that value.

`field name` can be the field key or the field label.

Labels or values with spaces should be wrapped in quotes. For example: `name eq "Bender Rodriguez"`, or `"First Name" eq Bender`.

### Operators

FFQL operators are:

* `eq` - Equals (exact match)
* `ne` - Not Equal
* `lt` - Less Than
* `gt` - Greater Than
* `lte` - Less Than or Equal To
* `gte` - Greater Than or Equal To
* `like` - (Case Sensitive) Like
* `ilike` - (Case Insensitive) Like
* `contains` - Contains (for columns of type `string-list` and `enum-list`)

Both `like` and `ilike` support the following wildcards:

* `%` - Matches any number of characters
* `_` - Matches a single character

So, for instance, `like "Bender%"` would match "Bender Rodriguez" and "Bender Bending Rodriguez".

### Logical Operators

FFQL supports two logical operators:

* `and` - Returns rows meeting both conditions
* `or` - Returns rows meeting either conditions (inclusive)

### The `is` Operator

You can query for message statuses by using the `is` operator. For example: `is error` returns all the rows in an `error` state. `is valid` returns all the rows in a `valid` state. `first_name is error` returns the rows where First Name is in an `error` state.

### Escaping Quotes and Backslashes

When you need to include quotes or backslashes within quoted values, you can escape them using a backslash (`\`). Here are examples of how escaping works:

| Query              | Value           |
| ------------------ | --------------- |
| `"hello \" world"` | `hello " world` |
| `'hello \' world'` | `hello ' world` |
| `"hello world \\"` | `hello world \` |
| `"hello \ world"`  | `hello \ world` |
| `"hello \\ world"` | `hello \ world` |

For example, to search for a field containing a quote:

```
first_name eq "John \"Johnny\" Doe"
```

This would match a first name value of: `John "Johnny" Doe`

***

## Constructing Queries

Complex queries are possible using a combination of operators:

```
(
    email like "@gmail.com"
    and (
        "Subscription Status" eq "On-Hold"
        or "Subscription Status" eq "Pending"
    )
    and login_attempts gte 5
)
or is warning
```

This query would return all the rows that:

1. Have a Gmail email address,
2. Have a Subscription Status of "On-Hold" or "Pending",
3. And have more than 5 login attempts.

It will also include any rows that have a "Warning" message status.

***

## Usage

### Via search bar

From the search bar in a Workbook, prepend **filter:** to your FFQL query.

**type in search bar**

```
filter: first_name eq Bender and last_name eq Rodriguez
```

### Via API

FFQL queries can be passed to any [REST API](https://reference.flatfile.com/overview/welcome) endpoint that supports the `q` parameter.

Here's an example **cURL** request using the `sheets/<sheetId>/records` endpoint:

**Shell / cURL**

```bash theme={null}
curl --location 'https://platform.flatfile.com/api/sheets/us_sh_12345678/records?q="Subscription Status" eq "On-Hold" or "Subscription Status" eq "Pending"' \
    --header 'Accept: */*' \
    --header 'Authorization: Bearer <bearer token>'
```

Make sure to [encode](https://en.wikipedia.org/wiki/URL_encoding) percent characters if you use them.


# For LLMs
Source: https://flatfile.com/docs/reference/for-llms

AI-optimized documentation formats and tools for better LLM integration

The Flatfile documentation is optimized for use with Large Language Models (LLMs) and AI tools. We provide several features to help you get faster, more accurate responses when using our documentation as context.

## Contextual Menu

We provide a contextual menu on every documentation page with quick access to AI-optimized content and direct integrations with popular AI tools:

<img alt="Contextual menu showing AI integration options" />

<img alt="Contextual menu showing AI integration options" />

* **Copy page** - Copies the current page as Markdown for pasting as context into AI tools
* **View as Markdown** - Opens the current page in a clean Markdown format
* **Open in ChatGPT** - Creates a ChatGPT conversation with the current page as context
* **Open in Claude** - Creates a Claude conversation with the current page as context

Access this menu by right-clicking on any page or using the contextual menu button (when available).

## LLM-Optimized File Formats

### /llms.txt

The `/llms.txt` file follows the [industry standard](https://llmstxt.org) that helps general-purpose LLMs index documentation more efficiently, similar to how a sitemap helps search engines.

**What it contains:**

* Complete list of all available pages in our documentation
* Page titles and URLs for easy navigation
* Structured format that AI tools can quickly parse

**How to use:**

* Access at: [https://flatfile.com/docs/llms.txt](https://flatfile.com/docs/llms.txt)
* Reference this file when asking AI tools to understand our documentation structure
* Use it to help LLMs find relevant content for specific topics

### /llms-full.txt

The `/llms-full.txt` file combines our entire documentation site into a single file, optimized for use as comprehensive context in AI tools.

**What it contains:**

* Full text content of all documentation pages
* Structured with clear page boundaries and headers
* Optimized formatting for LLM consumption

**How to use:**

* Access at: [https://flatfile.com/docs/llms-full.txt](https://flatfile.com/docs/llms-full.txt)
* Download and use as context for comprehensive questions about Flatfile
* Ideal for complex queries that might span multiple documentation areas

**Note:** This file is large and may consume significant tokens. You might want to use `/llms.txt` first to identify specific pages, then use individual page URLs or the contextual menu for targeted questions.

## Markdown Versions of Pages

All documentation pages are available in Markdown format, which provides structured text that AI tools can process more efficiently than HTML.

### .md Extension

Add `.md` to any page's URL to view the Markdown version:

```
https://flatfile.com/docs/getting-started/welcome.md
https://flatfile.com/docs/embedding/overview.md
https://flatfile.com/docs/core-concepts/workbooks.md
```

## Best Practices for AI Tool Usage

### For Specific Questions

1. **Start with targeted pages** - Use the contextual menu or `.md` extension for specific topics
2. **Reference multiple related pages** - Copy 2-3 relevant pages for comprehensive context
3. **Include the page URL** - Help the AI tool understand the source and context

### For Comprehensive Questions

1. **Use `/llms.txt` first** - Help the AI understand our documentation structure
2. **Follow up with specific pages** - Use targeted content based on the structure overview
3. **Consider `/llms-full.txt`** - For complex questions spanning multiple areas (token usage permitting)

### Example Prompts

**For specific integration help:**

```
I'm trying to embed Flatfile in my React app. Here's the relevant documentation:
[paste content from /embedding/react.md]

How do I configure it for my use case where...
```

**For comprehensive understanding:**

```
I need to understand Flatfile's architecture. Here's their documentation structure:
[paste content from /llms.txt]

Can you explain how Environments, Apps, and Spaces work together?
```

## Technical Implementation

These AI-optimization features are built into our documentation platform and automatically maintained:

* **Auto-generated** - Files are updated automatically when documentation changes
* **Optimized formatting** - Content is structured for optimal LLM processing
* **Consistent structure** - All pages follow the same format for predictable parsing

## Feedback

These AI optimization features are continuously improved based on usage patterns and feedback. If you have suggestions for better LLM integration or notice issues with the generated formats, please [join our Slack community](https://flatfile.com/join-slack/) and share your thoughts in the #docs channel.


# Icon Reference
Source: https://flatfile.com/docs/reference/icons

Reference of all available icons

Icons may be set for custom action buttons. Here's a list of all available icons that Flatfile supports.

## Usage Example

Icons can be used in [Actions](/core-concepts/actions) to provide visual cues for different operations:

```javascript theme={null}
const action = {
  operation: "export-data",
  label: "Export",
  icon: "download",
  description: "Export your data",
  mode: "background",
};
```

Icons help users quickly identify the purpose of different actions and improve the overall user experience in your Flatfile implementation.

Icons may be set for custom action buttons. Here's a list of all available icons
that Flatfile supports.

## All icons

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>add</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>alertCircle</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>alertTriangle</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>arrowLeft</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>arrowProgress</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>arrowRight</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>arrowRotateLeft</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>arrowsMaximize</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>arrowUpRight</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>at</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>ban</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>boxLogo</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>boxOpen</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>building</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>calendar</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>calendarLight</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>chat</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>checkmark</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>checkmarkCircle</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>checkmarkShield</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>chevronDoubleDown</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>chevronDown</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>chevronRight</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>clipboardCheck</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>code</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>columns</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>connection</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>connectionSlash</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>cross</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>cubeTransparent</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>database</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>diff</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>documentAdd</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>documentCopy</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>dotsVertical</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>download</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>edit</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>enter</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>eye</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>eyeSlash</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>fileCircleInfo</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>fileCode</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>filterList</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>folder</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>gear</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>github</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />

        <path />

        <path />

        <path />
      </svg>

      <code>googleDrive</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>grid</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>history</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>hourglass</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>info</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>key</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>lightningBolt</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>link</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>list</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>listTimeline</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>lock</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>mail</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>objectGroup</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>palette</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>pen</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>penSlash</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>pin</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>planetRinged</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>plus</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>questionCircle</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>search</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>selector</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>sidebar</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>sink</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>spaceShip</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>sparkles</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>table</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />

        <path />
      </svg>

      <code>tableDot</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>trash</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>truck</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>ufo</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>upload</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>user</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>userAdd</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>userGroup</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>userRemove</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>userSecret</code>
    </div>
  </Card>

  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>wandSparkles</code>
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title>
    <div>
      <svg>
        <path />
      </svg>

      <code>xCircle</code>
    </div>
  </Card>
</CardGroup>


# V2 Records API
Source: https://flatfile.com/docs/reference/v2-records-api

Using the V2 Records API with the Flatfile TypeScript API package

The V2 Records API provides an interface for reading and writing records from Flatfile sheets.
It is optimized for performance when working with large numbers of records.

<Note>Support for the V2 Records API was added to the `@flatfile/api` package in version 1.18.0.</Note>

## Overview

The V2 Records API is accessible through the `records.v2` property on the Flatfile client.

```typescript theme={null}
import { FlatfileClient } from "@flatfile/api";
const client = new FlatfileClient({ token: "YOUR_TOKEN" });
await client.records.v2.get("us_sh_1234abcd")
```

The V2 API uses a specialized JSON Lines (JSONL) format that is optimized for performance and
usage with streaming requests.

## Reading Records

### getRaw()

Retrieve records from a sheet in raw JSONL format.

```typescript theme={null}
getRaw(
  sheetId: Flatfile.SheetId,
  options?: GetRecordsRequestOptions,
  requestOptions?: RequestOptions
): Promise<JsonlRecord[]>
```

Returns an array of `JsonlRecord` objects containing the raw data structure from the API including special fields like `__k` (record ID), `__v` (version), etc.

**Example:**

```typescript theme={null}
const rawRecords = await api.records.v2.getRaw('us_sh_123', {
  fields: ['firstName', 'lastName'],
  pageSize: 1000
});

rawRecords.forEach(record => {
  console.log(`Record ID: ${record.__k}`);
  console.log(`Field values:`, record);
});
```

### getRawStreaming()

Stream records from a sheet in raw JSONL format.

```typescript theme={null}
getRawStreaming(
  sheetId: Flatfile.SheetId,
  options?: GetRecordsRequestOptions,
  requestOptions?: RequestOptions
): AsyncGenerator<JsonlRecord>
```

The most memory-efficient way to process large datasets as records are yielded individually.

**Example:**

```typescript theme={null}
for await (const rawRecord of api.records.v2.getRawStreaming('us_sh_123', {
  includeTimestamps: true
})) {
  console.log(`Record ID: ${rawRecord.__k}`);
  console.log(`Updated at: ${rawRecord.__u}`);
  // Process each record as it streams in
}
```

## Writing Records

### writeRaw()

Write records to a sheet in raw JSONL format.

```typescript theme={null}
writeRaw(
  records: JsonlRecord[],
  options?: WriteRecordsRequestOptions,
  requestOptions?: RequestOptions
): Promise<WriteRecordsResponse>
```

**Parameters:**

* `records` - Array of JsonlRecord objects to write
* `options` - Write configuration options
* `requestOptions` - Optional request configuration

**Example:**

```typescript theme={null}
const records: JsonlRecord[] = [
  { firstName: 'John', lastName: 'Doe', __s: 'us_sh_123' },
  { __k: 'us_rc_456', firstName: 'Jane', lastName: 'Smith' }  // Update existing
];

const result = await api.records.v2.writeRaw(records, {
  sheetId: 'us_sh_123',
  truncate: false
});

console.log(`Created: ${result.created}, Updated: ${result.updated}`);
```

### writeRawStreaming()

Stream records to a sheet in raw JSONL format.

```typescript theme={null}
writeRawStreaming(
  recordsStream: AsyncIterable<JsonlRecord>,
  options?: WriteStreamingOptions,
  requestOptions?: RequestOptions
): Promise<WriteRecordsResponse>
```

Efficiently write large datasets by streaming records to the server.

**Example:**

```typescript theme={null}
async function* generateRecords() {
  for (let i = 0; i < 100000; i++) {
    yield {
      firstName: `User${i}`,
      email: `user${i}@example.com`,
      __s: 'us_sh_123'
    };
  }
}

const result = await api.records.v2.writeRawStreaming(generateRecords(), {
  sheetId: 'us_sh_123',
  chunkSize: 1000,
  useBodyStreaming: 'auto'
});
```

## JSONL Record Format Reference

The V2 Records API uses JSONL (JSON Lines) format for raw record operations. JSONL is a text format where each line is a valid JSON object representing a single record.

### Special Fields

Flatfile uses special fields prefixed with `__` (double underscore) to store metadata and system information:

| Field  | Description                   | Example                                                       |
| ------ | ----------------------------- | ------------------------------------------------------------- |
| `__k`  | Record ID (unique identifier) | `"us_rc_123456"`                                              |
| `__nk` | New record ID (for creation)  | `"temp_123"`                                                  |
| `__v`  | Record version ID             | `"v_789"`                                                     |
| `__s`  | Sheet ID                      | `"us_sh_123"`                                                 |
| `__n`  | Sheet slug                    | `"contacts"`                                                  |
| `__c`  | Record configuration          | `{"locked": true}`                                            |
| `__m`  | Record metadata               | `{"source": "import"}`                                        |
| `__i`  | Validation messages           | `{"email": [{"type": "error", "message": "Invalid format"}]}` |
| `__d`  | Deleted flag                  | `true`                                                        |
| `__e`  | Valid flag                    | `false`                                                       |
| `__l`  | Linked records                | `[{...}]`                                                     |
| `__u`  | Updated timestamp             | `"2023-11-20T16:59:40.286Z"`                                  |

Note that many special fields will not be included unless the corresponding query parameter is included,
such as `includeMessages` to include the `__i` field with validation messages.

### Field Values

Regular field values are stored directly as properties on the JSONL record:

```json theme={null}
{
  "__k": "us_rc_123456",
  "__s": "us_sh_123",
  "firstName": "John",
  "lastName": "Doe",
  "email": "john.doe@example.com",
  "age": 30,
  "isActive": true
}
```

### Validation Messages

If included, the `__i` field contains validation messages for fields that have errors or warnings.
The messages are structured as an array of objects. Each validation message object will be structured
as `{"x": <field key of message>, "m": <validation message>}`.

```json theme={null}
{
  "__k": "us_rc_123456",
  "__s": "us_sh_123",
  "email": "invalid-email",
  "__i": [
    {"x": "firstName", "m": "First name is required"},
    {"x": "email", "m": "Not a valid email address"}
  ]
}
```

### Complete Example

A complete JSONL record with all common fields:

```json theme={null}
{
  "__k": "us_rc_123456",
  "__v": "v_789",
  "__s": "us_sh_123",
  "__n": "contacts",
  "__c": {
    "locked": false,
    "priority": "high"
  },
  "__m": {
    "source": "api_import",
    "batch_id": "batch_001"
  },
  "__i": [
    {"x": "firstName", "m": "First name is required"},
    {"x": "email", "m": "Not a valid email address"}
  ],
  "__e": true,
  "__u": "2023-11-20T16:59:40.286Z",
  "firstName": "John",
  "lastName": "Doe",
  "email": "john.doe@example.com",
  "company": "Acme Corp",
  "phone": "+1-555-123-4567"
}
```

### Record Operations

The V2 write records endpoint can be used to create, update, or delete records into a sheet
all in a single request. The operation performed depends on which special keys are provided.

#### Create New Record

To create a new record, the record can simply be passed with a sheet ID in `__s` and no record ID.

```json theme={null}
{
  "__s": "us_sh_123",
  "firstName": "New",
  "lastName": "User"
}
```

#### Update Existing Record

To update an existing record, provide the record ID in the `__k` field.

```json theme={null}
{
  "__k": "us_rc_123456",
  "__s": "us_sh_123",
  "firstName": "Updated",
  "lastName": "Name"
}
```

#### Delete Record

To delete a record, provide the record ID in the `__k` field and pass the delete flag
in the `__d` field.

```json theme={null}
{
  "__k": "us_rc_123456",
  "__s": "us_sh_123",
  "__d": true
}
```