# Unstructured

## Docs

- [Overview](https://docs.unstructured.io/api-reference/api-services/accessing-unstructured-api)
- [API parameters](https://docs.unstructured.io/api-reference/api-services/api-parameters): Unstructured API services provide parameters to customize the processing of documents. Below are the details for these parameters.
- [API validation errors](https://docs.unstructured.io/api-reference/api-services/api-validation-errors): This section details the structure of HTTP validation errors returned by the API.
- [Unstructured API on AWS](https://docs.unstructured.io/api-reference/api-services/aws): Follow these steps to deploy the Unstructured API service into your AWS account.
- [Unstructured API on Azure](https://docs.unstructured.io/api-reference/api-services/azure)
- [Chunking strategies](https://docs.unstructured.io/api-reference/api-services/chunking)
- [Document elements and metadata](https://docs.unstructured.io/api-reference/api-services/document-elements)
- [Examples](https://docs.unstructured.io/api-reference/api-services/examples): This page provides some examples of accessing Unstructured API via different methods.
- [Free Unstructured API](https://docs.unstructured.io/api-reference/api-services/free-api): This page describes how to obtain an API key to use with the free Unstructured API, the limitations of the free Unstructured API, and provides a quickstart example.
- [Unstructured Serverless API services](https://docs.unstructured.io/api-reference/api-services/overview)
- [Process an individual file by using the Unstructured open source Python library](https://docs.unstructured.io/api-reference/api-services/partition-via-api)
- [Partitioning strategies](https://docs.unstructured.io/api-reference/api-services/partitioning)
- [Process an individual file by making a direct POST request](https://docs.unstructured.io/api-reference/api-services/post-requests)
- [Unstructured Serverless API](https://docs.unstructured.io/api-reference/api-services/saas-api-development-guide): This page describes how to get started with the Unstructured Serverless API. Learn how to obtain an API key to use with the Unstructured Serverless API, and get started in no time.
- [Process an individual file by using the Unstructured JavaScript/TypeScript SDK](https://docs.unstructured.io/api-reference/api-services/sdk-jsts)
- [Process an individual file by using the Unstructured Python SDK](https://docs.unstructured.io/api-reference/api-services/sdk-python)
- [Supported file types](https://docs.unstructured.io/api-reference/api-services/supported-file-types)
- [Chunking](https://docs.unstructured.io/api-reference/best-practices/chunking)
- [Embedding](https://docs.unstructured.io/api-reference/best-practices/embedding)
- [Speed up processing of large files and batches](https://docs.unstructured.io/api-reference/best-practices/speed-up-large-files-batches)
- [Pipeline](https://docs.unstructured.io/api-reference/general/pipeline-1): Description
- [Set chunking behavior](https://docs.unstructured.io/api-reference/how-to/change-chunking-strategy)
- [Change element coordinate systems](https://docs.unstructured.io/api-reference/how-to/change-element-coordinate-system)
- [Set partitioning behavior](https://docs.unstructured.io/api-reference/how-to/change-partitioning-strategy)
- [Choose a hi-res model](https://docs.unstructured.io/api-reference/how-to/choose-hi-res-model)
- [Choose a partitioning strategy](https://docs.unstructured.io/api-reference/how-to/choose-partitioning-strategy)
- [Set embedding behavior](https://docs.unstructured.io/api-reference/how-to/embedding)
- [Extract images and tables from documents](https://docs.unstructured.io/api-reference/how-to/extract-image-block-types)
- [Process a subset of files](https://docs.unstructured.io/api-reference/how-to/filter-files)
- [Generate a JSON schema for a file](https://docs.unstructured.io/api-reference/how-to/generate-schema)
- [Get chunked elements](https://docs.unstructured.io/api-reference/how-to/get-chunked-elements)
- [Get element contents](https://docs.unstructured.io/api-reference/how-to/get-elements)
- [Output bounding box coordinates](https://docs.unstructured.io/api-reference/how-to/output-bounding-box-coordinates)
- [Output unique element IDs](https://docs.unstructured.io/api-reference/how-to/output-unique-element-ids)
- [Parse simple PDFs and HTML](https://docs.unstructured.io/api-reference/how-to/parse-simple-pdf-html)
- [Work with PowerPoint files](https://docs.unstructured.io/api-reference/how-to/powerpoint)
- [Set document language for better OCR](https://docs.unstructured.io/api-reference/how-to/set-document-language-ocr)
- [Extract tables as HTML](https://docs.unstructured.io/api-reference/how-to/text-as-html)
- [Transform a JSON file into a different schema](https://docs.unstructured.io/api-reference/how-to/transform-schemas)
- [Use LangChain and Llama 3](https://docs.unstructured.io/api-reference/how-to/use-langchain-llama-3)
- [Use LangChain and Ollama](https://docs.unstructured.io/api-reference/how-to/use-langchain-ollama)
- [Astra DB](https://docs.unstructured.io/api-reference/ingest/destination-connector/astradb)
- [Azure](https://docs.unstructured.io/api-reference/ingest/destination-connector/azure)
- [Azure AI Search](https://docs.unstructured.io/api-reference/ingest/destination-connector/azure-ai-search)
- [Box](https://docs.unstructured.io/api-reference/ingest/destination-connector/box)
- [Chroma](https://docs.unstructured.io/api-reference/ingest/destination-connector/chroma)
- [Clarifai](https://docs.unstructured.io/api-reference/ingest/destination-connector/clarifai)
- [Couchbase](https://docs.unstructured.io/api-reference/ingest/destination-connector/couchbase)
- [Databricks Volumes](https://docs.unstructured.io/api-reference/ingest/destination-connector/databricks-volumes)
- [Delta Table](https://docs.unstructured.io/api-reference/ingest/destination-connector/delta-table)
- [Dropbox](https://docs.unstructured.io/api-reference/ingest/destination-connector/dropbox)
- [Elasticsearch](https://docs.unstructured.io/api-reference/ingest/destination-connector/elasticsearch)
- [Google Cloud Storage](https://docs.unstructured.io/api-reference/ingest/destination-connector/google-cloud-service)
- [Kafka](https://docs.unstructured.io/api-reference/ingest/destination-connector/kafka)
- [KDB.AI](https://docs.unstructured.io/api-reference/ingest/destination-connector/kdbai)
- [LanceDB](https://docs.unstructured.io/api-reference/ingest/destination-connector/lancedb)
- [Local](https://docs.unstructured.io/api-reference/ingest/destination-connector/local)
- [Milvus](https://docs.unstructured.io/api-reference/ingest/destination-connector/milvus)
- [MongoDB](https://docs.unstructured.io/api-reference/ingest/destination-connector/mongodb)
- [OneDrive](https://docs.unstructured.io/api-reference/ingest/destination-connector/onedrive)
- [OpenSearch](https://docs.unstructured.io/api-reference/ingest/destination-connector/opensearch)
- [Overview](https://docs.unstructured.io/api-reference/ingest/destination-connector/overview)
- [Pinecone](https://docs.unstructured.io/api-reference/ingest/destination-connector/pinecone)
- [PostgreSQL](https://docs.unstructured.io/api-reference/ingest/destination-connector/postgresql)
- [Qdrant](https://docs.unstructured.io/api-reference/ingest/destination-connector/qdrant)
- [S3](https://docs.unstructured.io/api-reference/ingest/destination-connector/s3)
- [SFTP](https://docs.unstructured.io/api-reference/ingest/destination-connector/sftp)
- [SingleStore](https://docs.unstructured.io/api-reference/ingest/destination-connector/singlestore)
- [Snowflake](https://docs.unstructured.io/api-reference/ingest/destination-connector/snowflake)
- [SQLite](https://docs.unstructured.io/api-reference/ingest/destination-connector/sqlite)
- [Vectara](https://docs.unstructured.io/api-reference/ingest/destination-connector/vectara)
- [Weaviate](https://docs.unstructured.io/api-reference/ingest/destination-connector/weaviate)
- [Process files in batches by using the Unstructured Ingest CLI](https://docs.unstructured.io/api-reference/ingest/ingest-cli)
- [Chunking configuration](https://docs.unstructured.io/api-reference/ingest/ingest-configuration/chunking-configuration)
- [Embedding configuration](https://docs.unstructured.io/api-reference/ingest/ingest-configuration/embedding-configuration)
- [Fsspec configuration](https://docs.unstructured.io/api-reference/ingest/ingest-configuration/fsspec-configuration)
- [Overview](https://docs.unstructured.io/api-reference/ingest/ingest-configuration/overview)
- [Partition configuration](https://docs.unstructured.io/api-reference/ingest/ingest-configuration/partition-configuration)
- [Permissions configuration](https://docs.unstructured.io/api-reference/ingest/ingest-configuration/permissions-configuration)
- [Processor configuration](https://docs.unstructured.io/api-reference/ingest/ingest-configuration/processor-configuration)
- [Read configuration](https://docs.unstructured.io/api-reference/ingest/ingest-configuration/read-configuration)
- [Retry strategy configuration](https://docs.unstructured.io/api-reference/ingest/ingest-configuration/retry-strategy-configuration)
- [Ingest dependencies](https://docs.unstructured.io/api-reference/ingest/ingest-dependencies)
- [Overview](https://docs.unstructured.io/api-reference/ingest/overview)
- [Process files in batches by using the Unstructured Ingest Python library](https://docs.unstructured.io/api-reference/ingest/python-ingest)
- [Airtable](https://docs.unstructured.io/api-reference/ingest/source-connectors/airtable)
- [Astra DB](https://docs.unstructured.io/api-reference/ingest/source-connectors/astradb)
- [Azure](https://docs.unstructured.io/api-reference/ingest/source-connectors/azure)
- [Biomed](https://docs.unstructured.io/api-reference/ingest/source-connectors/biomed)
- [Box](https://docs.unstructured.io/api-reference/ingest/source-connectors/box)
- [Confluence](https://docs.unstructured.io/api-reference/ingest/source-connectors/confluence)
- [Couchbase](https://docs.unstructured.io/api-reference/ingest/source-connectors/couchbase)
- [Databricks Volumes](https://docs.unstructured.io/api-reference/ingest/source-connectors/databricks-volumes)
- [Delta Table](https://docs.unstructured.io/api-reference/ingest/source-connectors/delta-table)
- [Discord](https://docs.unstructured.io/api-reference/ingest/source-connectors/discord)
- [Dropbox](https://docs.unstructured.io/api-reference/ingest/source-connectors/dropbox)
- [Elasticsearch](https://docs.unstructured.io/api-reference/ingest/source-connectors/elastic-search)
- [GitHub](https://docs.unstructured.io/api-reference/ingest/source-connectors/github)
- [GitLab](https://docs.unstructured.io/api-reference/ingest/source-connectors/gitlab)
- [Google Cloud Storage](https://docs.unstructured.io/api-reference/ingest/source-connectors/google-cloud-storage)
- [Google Drive](https://docs.unstructured.io/api-reference/ingest/source-connectors/google-drive)
- [HubSpot](https://docs.unstructured.io/api-reference/ingest/source-connectors/hubspot)
- [Jira](https://docs.unstructured.io/api-reference/ingest/source-connectors/jira)
- [Kafka](https://docs.unstructured.io/api-reference/ingest/source-connectors/kafka)
- [Local](https://docs.unstructured.io/api-reference/ingest/source-connectors/local)
- [MongoDB](https://docs.unstructured.io/api-reference/ingest/source-connectors/mongodb)
- [Notion](https://docs.unstructured.io/api-reference/ingest/source-connectors/notion)
- [OneDrive](https://docs.unstructured.io/api-reference/ingest/source-connectors/one-drive)
- [OpenSearch](https://docs.unstructured.io/api-reference/ingest/source-connectors/opensearch)
- [Outlook](https://docs.unstructured.io/api-reference/ingest/source-connectors/outlook)
- [Overview](https://docs.unstructured.io/api-reference/ingest/source-connectors/overview)
- [PostgreSQL](https://docs.unstructured.io/api-reference/ingest/source-connectors/postgresql)
- [Reddit](https://docs.unstructured.io/api-reference/ingest/source-connectors/reddit)
- [S3](https://docs.unstructured.io/api-reference/ingest/source-connectors/s3)
- [Salesforce](https://docs.unstructured.io/api-reference/ingest/source-connectors/salesforce)
- [SFTP](https://docs.unstructured.io/api-reference/ingest/source-connectors/sftp)
- [SharePoint](https://docs.unstructured.io/api-reference/ingest/source-connectors/sharepoint)
- [SingleStore](https://docs.unstructured.io/api-reference/ingest/source-connectors/singlestore)
- [Slack](https://docs.unstructured.io/api-reference/ingest/source-connectors/slack)
- [Snowflake](https://docs.unstructured.io/api-reference/ingest/source-connectors/snowflake)
- [SQLite](https://docs.unstructured.io/api-reference/ingest/source-connectors/sqlite)
- [Wikipedia](https://docs.unstructured.io/api-reference/ingest/source-connectors/wikipedia)
- [Troubleshooting Unstructured API keys and API URLs](https://docs.unstructured.io/api-reference/troubleshooting/api-key-url)
- [Query processed PDF with HuggingChat](https://docs.unstructured.io/examplecode/codesamples/api/huggingchat)
- [Table extraction from PDF](https://docs.unstructured.io/examplecode/codesamples/apioss/table-extraction-from-pdf): This section describes two methods for extracting tables from PDF files.
- [Multi-file API processing](https://docs.unstructured.io/examplecode/codesamples/oss/multi-files-api-processing)
- [Delta Table source connector](https://docs.unstructured.io/examplecode/codesamples/oss/table-source-connector)
- [Vector database ingestion](https://docs.unstructured.io/examplecode/codesamples/oss/vector-database)
- [Notebooks](https://docs.unstructured.io/examplecode/notebooks): Notebooks contain complete working sample code for end-to-end solutions.
- [Langflow](https://docs.unstructured.io/examplecode/tools/langflow)
- [Frequently asked questions](https://docs.unstructured.io/faq/faq)
- [Glossary](https://docs.unstructured.io/glossary/glossary)
- [Overview](https://docs.unstructured.io/ingestion/overview)
- [Chunking](https://docs.unstructured.io/open-source/best-practices/chunking)
- [Embedding](https://docs.unstructured.io/open-source/best-practices/embedding)
- [Document elements and metadata](https://docs.unstructured.io/open-source/concepts/document-elements)
- [Models](https://docs.unstructured.io/open-source/concepts/models): Depending on your need, `Unstructured` provides OCR-based and Transformer-based models to detect elements in the documents. The models are useful to detect the complex layout in the documents and predict the element types.
- [Partitioning strategies](https://docs.unstructured.io/open-source/concepts/partitioning-strategies)
- [Chunking](https://docs.unstructured.io/open-source/core-functionality/chunking): Chunking functions in `unstructured` use metadata and document elements detected with `partition` functions to post-process elements into more useful "chunks" for uses cases such as Retrieval Augmented Generation (RAG).
- [Cleaning](https://docs.unstructured.io/open-source/core-functionality/cleaning): As part of data preparation for an NLP model, it’s common to need to clean up your data prior to passing it into the model. If there’s unwanted content in your output, for example, it could impact the quality of your NLP model. To help with this, the `unstructured` library includes cleaning functions to help users sanitize output before sending it to downstream applications.
- [Embedding](https://docs.unstructured.io/open-source/core-functionality/embedding)
- [Extracting](https://docs.unstructured.io/open-source/core-functionality/extracting)
- [Core Functionality](https://docs.unstructured.io/open-source/core-functionality/overview): The `unstructured` library includes functions to partition, chunk, clean, and stage raw source documents. These functions serve as the primary public interfaces within the library.
- [Partitioning](https://docs.unstructured.io/open-source/core-functionality/partitioning): Partitioning functions in `unstructured` allow users to extract structured content from a raw unstructured document. These functions break a document down into elements such as `Title`, `NarrativeText`, and `ListItem`, enabling users to decide what content they’d like to keep for their particular application. If you’re training a summarization model, for example, you may only be interested in `NarrativeText`.
- [Staging](https://docs.unstructured.io/open-source/core-functionality/staging)
- [Set the OCR agent](https://docs.unstructured.io/open-source/how-to/set-ocr-agent)
- [Astra DB](https://docs.unstructured.io/open-source/ingest/destination-connectors/astradb)
- [Azure](https://docs.unstructured.io/open-source/ingest/destination-connectors/azure)
- [Azure AI Search](https://docs.unstructured.io/open-source/ingest/destination-connectors/azure-ai-search)
- [Box](https://docs.unstructured.io/open-source/ingest/destination-connectors/box)
- [Chroma](https://docs.unstructured.io/open-source/ingest/destination-connectors/chroma)
- [Clarifai](https://docs.unstructured.io/open-source/ingest/destination-connectors/clarifai)
- [Couchbase](https://docs.unstructured.io/open-source/ingest/destination-connectors/couchbase)
- [Databricks Volumes](https://docs.unstructured.io/open-source/ingest/destination-connectors/databricks-volumes)
- [Delta Table](https://docs.unstructured.io/open-source/ingest/destination-connectors/delta-table)
- [Dropbox](https://docs.unstructured.io/open-source/ingest/destination-connectors/dropbox)
- [Elasticsearch](https://docs.unstructured.io/open-source/ingest/destination-connectors/elasticsearch)
- [Google Cloud Storage](https://docs.unstructured.io/open-source/ingest/destination-connectors/google-cloud-service)
- [Kafka](https://docs.unstructured.io/open-source/ingest/destination-connectors/kafka)
- [KDB.AI](https://docs.unstructured.io/open-source/ingest/destination-connectors/kdbai)
- [LanceDB](https://docs.unstructured.io/open-source/ingest/destination-connectors/lancedb)
- [Local](https://docs.unstructured.io/open-source/ingest/destination-connectors/local)
- [Milvus](https://docs.unstructured.io/open-source/ingest/destination-connectors/milvus)
- [MongoDB](https://docs.unstructured.io/open-source/ingest/destination-connectors/mongodb)
- [OneDrive](https://docs.unstructured.io/open-source/ingest/destination-connectors/onedrive)
- [OpenSearch](https://docs.unstructured.io/open-source/ingest/destination-connectors/opensearch)
- [Overview](https://docs.unstructured.io/open-source/ingest/destination-connectors/overview)
- [Pinecone](https://docs.unstructured.io/open-source/ingest/destination-connectors/pinecone)
- [PostgreSQL](https://docs.unstructured.io/open-source/ingest/destination-connectors/postgresql)
- [Qdrant](https://docs.unstructured.io/open-source/ingest/destination-connectors/qdrant)
- [S3](https://docs.unstructured.io/open-source/ingest/destination-connectors/s3)
- [SFTP](https://docs.unstructured.io/open-source/ingest/destination-connectors/sftp)
- [SingleStore](https://docs.unstructured.io/open-source/ingest/destination-connectors/singlestore)
- [Snowflake](https://docs.unstructured.io/open-source/ingest/destination-connectors/snowflake)
- [SQLite](https://docs.unstructured.io/open-source/ingest/destination-connectors/sqlite)
- [Vectara](https://docs.unstructured.io/open-source/ingest/destination-connectors/vectara)
- [Weaviate](https://docs.unstructured.io/open-source/ingest/destination-connectors/weaviate)
- [Chunking configuration](https://docs.unstructured.io/open-source/ingest/ingest-configuration/chunking-configuration)
- [Embedding configuration](https://docs.unstructured.io/open-source/ingest/ingest-configuration/embedding-configuration)
- [Fsspec configuration](https://docs.unstructured.io/open-source/ingest/ingest-configuration/fsspec-configuration)
- [Overview](https://docs.unstructured.io/open-source/ingest/ingest-configuration/overview)
- [Partition configuration](https://docs.unstructured.io/open-source/ingest/ingest-configuration/partition-configuration)
- [Permissions configuration](https://docs.unstructured.io/open-source/ingest/ingest-configuration/permissions-configuration)
- [Processor configuration](https://docs.unstructured.io/open-source/ingest/ingest-configuration/processor-configuration)
- [Read configuration](https://docs.unstructured.io/open-source/ingest/ingest-configuration/read-configuration)
- [Retry strategy configuration](https://docs.unstructured.io/open-source/ingest/ingest-configuration/retry-strategy-configuration)
- [Ingest](https://docs.unstructured.io/open-source/ingest/overview): The Unstructured Python Ingest library is a powerful tool designed to coordinate the process of pulling data from data providers, partitioning the content, and pushing that new content to a desired location. This technical documentation will provide an in-depth understanding of the Python Ingest library, including its features, architecture, installation, configuration, usage, API reference, troubleshooting, examples, and more.
- [Airtable](https://docs.unstructured.io/open-source/ingest/source-connectors/airtable)
- [Astra DB](https://docs.unstructured.io/open-source/ingest/source-connectors/astradb)
- [Azure](https://docs.unstructured.io/open-source/ingest/source-connectors/azure)
- [Biomed](https://docs.unstructured.io/open-source/ingest/source-connectors/biomed)
- [Box](https://docs.unstructured.io/open-source/ingest/source-connectors/box)
- [Confluence](https://docs.unstructured.io/open-source/ingest/source-connectors/confluence)
- [Couchbase](https://docs.unstructured.io/open-source/ingest/source-connectors/couchbase)
- [Databricks Volumes](https://docs.unstructured.io/open-source/ingest/source-connectors/databricks-volumes)
- [Delta Table](https://docs.unstructured.io/open-source/ingest/source-connectors/delta-table)
- [Discord](https://docs.unstructured.io/open-source/ingest/source-connectors/discord)
- [Dropbox](https://docs.unstructured.io/open-source/ingest/source-connectors/dropbox)
- [Elasticsearch](https://docs.unstructured.io/open-source/ingest/source-connectors/elastic-search)
- [GitHub](https://docs.unstructured.io/open-source/ingest/source-connectors/github)
- [GitLab](https://docs.unstructured.io/open-source/ingest/source-connectors/gitlab)
- [Google Cloud Storage](https://docs.unstructured.io/open-source/ingest/source-connectors/google-cloud-storage)
- [Google Drive](https://docs.unstructured.io/open-source/ingest/source-connectors/google-drive)
- [HubSpot](https://docs.unstructured.io/open-source/ingest/source-connectors/hubspot)
- [Jira](https://docs.unstructured.io/open-source/ingest/source-connectors/jira)
- [Kafka](https://docs.unstructured.io/open-source/ingest/source-connectors/kafka)
- [Local](https://docs.unstructured.io/open-source/ingest/source-connectors/local)
- [MongoDB](https://docs.unstructured.io/open-source/ingest/source-connectors/mongodb)
- [Notion](https://docs.unstructured.io/open-source/ingest/source-connectors/notion)
- [OneDrive](https://docs.unstructured.io/open-source/ingest/source-connectors/one-drive)
- [OpenSearch](https://docs.unstructured.io/open-source/ingest/source-connectors/opensearch)
- [Outlook](https://docs.unstructured.io/open-source/ingest/source-connectors/outlook)
- [Overview](https://docs.unstructured.io/open-source/ingest/source-connectors/overview)
- [PostgreSQL](https://docs.unstructured.io/open-source/ingest/source-connectors/postgresql)
- [Reddit](https://docs.unstructured.io/open-source/ingest/source-connectors/reddit)
- [S3](https://docs.unstructured.io/open-source/ingest/source-connectors/s3)
- [Salesforce](https://docs.unstructured.io/open-source/ingest/source-connectors/salesforce)
- [SFTP](https://docs.unstructured.io/open-source/ingest/source-connectors/sftp)
- [SharePoint](https://docs.unstructured.io/open-source/ingest/source-connectors/sharepoint)
- [SingleStore](https://docs.unstructured.io/open-source/ingest/source-connectors/singlestore)
- [Slack](https://docs.unstructured.io/open-source/ingest/source-connectors/slack)
- [Snowflake](https://docs.unstructured.io/open-source/ingest/source-connectors/snowflake)
- [SQLite](https://docs.unstructured.io/open-source/ingest/source-connectors/sqlite)
- [Wikipedia](https://docs.unstructured.io/open-source/ingest/source-connectors/wikipedia)
- [Docker Installation](https://docs.unstructured.io/open-source/installation/docker-installation): The instructions below guide you on how to use the unstructured library inside a Docker container.
- [Full Installation](https://docs.unstructured.io/open-source/installation/full-installation)
- [Overview](https://docs.unstructured.io/open-source/installation/overview): Unstructured open source library offers flexible options for installation.
- [Integrations](https://docs.unstructured.io/open-source/integrations): Integrate your model development pipeline with your favorite machine learning frameworks and libraries, and prepare your data for ingestion into downstream systems. Most of our integrations come in the form of [staging functions](/open-source/core-functionality/staging), which take a list of `Element` objects as input and return formatted dictionaries as output.
- [Unstructured Open Source](https://docs.unstructured.io/open-source/introduction/overview)
- [Quickstart](https://docs.unstructured.io/open-source/introduction/quick-start): Using Unstructured Open Source.
- [Supported file types](https://docs.unstructured.io/open-source/introduction/supported-file-types)
- [Chunking](https://docs.unstructured.io/platform/chunking)
- [Supported connectors](https://docs.unstructured.io/platform/connectors)
- [Astra DB](https://docs.unstructured.io/platform/destinations/astradb)
- [Azure AI Search](https://docs.unstructured.io/platform/destinations/azure-ai-search)
- [Couchbase](https://docs.unstructured.io/platform/destinations/couchbase)
- [Databricks Volumes](https://docs.unstructured.io/platform/destinations/databricks-volumes)
- [Delta Table](https://docs.unstructured.io/platform/destinations/delta-table)
- [Elasticsearch](https://docs.unstructured.io/platform/destinations/elasticsearch)
- [Google Cloud Storage](https://docs.unstructured.io/platform/destinations/google-cloud)
- [Milvus](https://docs.unstructured.io/platform/destinations/milvus)
- [MongoDB](https://docs.unstructured.io/platform/destinations/mongodb)
- [OneDrive](https://docs.unstructured.io/platform/destinations/onedrive)
- [Overview](https://docs.unstructured.io/platform/destinations/overview): Destination connectors in the Unstructured Platform are designed to specify the endpoint for data processed within the platform. These connectors ensure that the transformed and analyzed data is securely and efficiently transferred to a storage system for future use, often to a vector database for tasks that involve high-speed retrieval and advanced data analytics operations.
- [Pinecone](https://docs.unstructured.io/platform/destinations/pinecone)
- [S3](https://docs.unstructured.io/platform/destinations/s3)
- [Weaviate](https://docs.unstructured.io/platform/destinations/weaviate)
- [Document elements and metadata](https://docs.unstructured.io/platform/document-elements)
- [Embedding](https://docs.unstructured.io/platform/embedding)
- [Jobs](https://docs.unstructured.io/platform/jobs)
- [Overview](https://docs.unstructured.io/platform/overview)
- [Partitioning](https://docs.unstructured.io/platform/partitioning)
- [Unstructured Platform quickstart](https://docs.unstructured.io/platform/quickstart)
- [Azure](https://docs.unstructured.io/platform/sources/azure-blob-storage)
- [Confluence](https://docs.unstructured.io/platform/sources/confluence)
- [Databricks Volumes](https://docs.unstructured.io/platform/sources/databricks-volumes)
- [Google Cloud Storage](https://docs.unstructured.io/platform/sources/google-cloud)
- [MongoDB](https://docs.unstructured.io/platform/sources/mongodb)
- [OneDrive](https://docs.unstructured.io/platform/sources/onedrive)
- [Outlook](https://docs.unstructured.io/platform/sources/outlook)
- [Overview](https://docs.unstructured.io/platform/sources/overview): Source connectors are essential components in data integration systems that establish a link between your files and the data ingestion process. They facilitate the batch processing of files, allowing for the systematic retrieval and ingestion of data stored in various file formats.
- [S3](https://docs.unstructured.io/platform/sources/s3)
- [SharePoint](https://docs.unstructured.io/platform/sources/sharepoint)
- [Summarizing](https://docs.unstructured.io/platform/summarizing)
- [Supported file types](https://docs.unstructured.io/platform/supported-file-types)
- [Workflows](https://docs.unstructured.io/platform/workflows)
- [Unstructured](https://docs.unstructured.io/welcome)


## Optional

- [Support](mailto:support@unstructured.io)
- [Community](https://short.unstructured.io/pzw05l7)
- [Feedback](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4)
- [Blog](https://unstructured.io/blog)
- [Product](https://unstructured.io/product)


# Overview



To process an individual file, you can choose from several available methods, including a direct `POST` request, Python code, and JavaScript/TypeScript code.
Whether you're using the Free Unstructured API, the Unstructured Serverless API, the Unstructured API on Azure/AWS, or your local deployment of the Unstructured API, the functionality is the same.

Choose your preferred method:

*   [Use the Unstructured Python SDK](/api-reference/api-services/sdk-python)
*   [Use the Unstructured JavaScript/TypeScript SDK](/api-reference/api-services/sdk-jsts)
*   [Use the Unstructured open source Python library](/api-reference/api-services/partition-via-api)
*   [Make a direct POST request](/api-reference/api-services/post-requests)

The API parameters for all these methods are documented on the [API parameters](/api-reference/api-services/api-parameters) page.

<Info>
  Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
  [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
  to you:

  *   You need to work with documents in cloud storage.
  *   You want to cache the results of processing multiple files in batches.
  *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
</Info>

If you'd like to try out the Unstructured API interactively by using the Free Unstructured API to process a single file, you can do so by using the [Swagger UI](https://api.unstructured.io/general/docs#/default/pipeline_1_general_v0_general_post).

1.  Go to the [Swagger UI](https://api.unstructured.io/general/docs#/default/pipeline_1_general_v0_general_post).
2.  For **Servers**, select **[https://api.unstructured.io](https://api.unstructured.io) - Hosted API Free**.
3.  Click **Authorize**.
4.  In the **Available authorizations** dialog box, for **Value**, enter your Free Unstructured API key. [Get a Free Unstructured API key](/api-reference/api-services/free-api#get-an-api-key).
5.  Click **Authorize**.
6.  Click **Close**.
7.  Expand the **POST** section.
8.  In the **Request body** section, next to **files**, click **Choose File**.
9.  Browse to and select a file for the Free Unstructured API to process.
10. Enter any other settings as desired. [Learn how](/api-reference/api-services/api-parameters).
11. At the end of the list of settings, click **Execute**.
12. See the results in the **Responses** section below the **Execute** button.


# API parameters

Unstructured API services provide parameters to customize the processing of documents. Below are the details for these parameters.

<Warning>
  The following parameters apply only to POST requests, the Unstructured Python SDK, and the Unstructured JavaScript/TypeScript SDK.

  Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
  [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) instead.

  The Ingest CLI and Ingest Python library provide faster processing of larger individual files,
  and faster and easier processing of multiple files at a time in batches.
</Warning>

## Parameters

The only required parameter is `files` -  the file you wish to process.

[//]: # "TODO: In the future we should automatically generate this table to make sure it's up to date"

| POST, Python                               | JavaScript/TypeScript                    | Description                                                                                                                                                                                                                                                                                                                                    |
| ------------------------------------------ | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `files` (*shared.Files*)                   | `files` (*File*, *Blob*, *shared.Files*) | The file to process.                                                                                                                                                                                                                                                                                                                           |
| `chunking_strategy` (*str*)                | `chunkingStrategy` (*string*)            | Use one of the supported strategies to chunk the returned elements after partitioning. When no chunking strategy is specified, no chunking is performed and any other chunking parameters provided are ignored. Supported strategies: `basic`, `by_title`, `by_page`, and `by_similarity`. [Learn more](/api-reference/api-services/chunking). |
| `content_type` (*str*)                     | `contentType` (*string*)                 | A hint to Unstructured about the content type to use (such as `text/markdown`), when there are problems processing a specific file. This value is a MIME type in the format `type/subtype`. For available MIME types, see [model.py](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/file_utils/model.py).              |
| `coordinates` (*bool*)                     | `coordinates` (*boolean*)                | True to return bounding box coordinates for each element extracted with OCR. Default: false. [Learn more](/api-reference/api-services/examples#saving-bounding-box-coordinates).                                                                                                                                                               |
| `encoding` (*str*)                         | `encoding` (*string*)                    | The encoding method used to decode the text input. Default: `utf-8`.                                                                                                                                                                                                                                                                           |
| `extract_image_block_types` (*List\[str]*) | `extractImageBlockTypes` (*string\[]*)   | The types of elements to extract, for use in extracting image blocks as Base64 encoded data stored in element metadata fields, for example: `["Image","Table"]`. Supported filetypes are image and PDF. [Learn more](/api-reference/how-to/extract-image-block-types).                                                                         |
| `gz_uncompressed_content_type` (*str*)     | `gzUncompressedContentType` (*string*)   | If file is gzipped, use this content type after unzipping. Example: `application/pdf`                                                                                                                                                                                                                                                          |
| `hi_res_model_name` (*str*)                | `hiResModelName`  (*string*)             | The name of the inference model used when strategy is `hi_res`. Options are `layout_v1.1.0` and `yolox`. Default: `layout_v1.1.0`. [Learn more](/api-reference/api-services/examples#changing-partition-strategy-for-a-pdf).                                                                                                                   |
| `include_page_breaks` (*bool*)             | `includePageBreaks` (*boolean*)          | True for the output to include page breaks if the filetype supports it. Default: false.                                                                                                                                                                                                                                                        |
| `languages` (*List\[str]*)                 | `languages` (*string\[]*)                | The languages present in the document, for use in partitioning and OCR. [View the list of available languages](https://github.com/tesseract-ocr/tessdata). [Learn more](/api-reference/api-services/examples#specifying-the-language-of-a-document-for-better-ocr-results).                                                                    |
| `output_format` (*str*)                    | `outputFormat` (*string*)                | The format of the response. Supported formats are `application/json` and `text/csv`. Default: `application/json`.                                                                                                                                                                                                                              |
| `pdf_infer_table_structure` (*bool*)       | `pdfInferTableStructure` (*boolean*)     | **Deprecated!** If true and `strategy` is `hi_res`, any `Table` elements extracted from a PDF will include an additional metadata field, `text_as_html`, where the value (string) is a just a transformation of the data into an HTML table.                                                                                                   |
| `skip_infer_table_types` (*List\[str]*)    | `skipInferTableTypes` (*string\[]*)      | The document types that you want to skip table extraction for. Default: `[]`.                                                                                                                                                                                                                                                                  |
| `starting_page_number` (*int*)             | `startingPageNumber` (*number*)          | The page number to be be assigned to the first page in the document. This information will be included in elements' metadata and can be be especially useful when partitioning a document that is part of a larger document.                                                                                                                   |
| `strategy` (*str*)                         | `strategy` (*string*)                    | The strategy to use for partitioning PDF and image files. Options are `fast`, `hi_res`, `ocr_only`, and `auto`. Default: `auto`. [Learn more](/api-reference/api-services/partitioning).                                                                                                                                                       |
| `unique_element_ids` (*bool*)              | `uniqueElementIds` (*boolean*)           | True to assign UUIDs to element IDs, which guarantees their uniqueness (useful when using them as primary keys in database). Otherwise a SHA-256 of the element's text is used. Default: false.                                                                                                                                                |
| `xml_keep_tags` (*bool*)                   | `xmlKeepTags` (*boolean*)                | True to retain the XML tags in the output. Otherwise it will just extract the text from within the tags. Only applies to XML documents.                                                                                                                                                                                                        |

The following parameters only apply when a chunking strategy is specified. Otherwise, they are ignored. [Learn more](/api-reference/api-services/chunking).

| POST, Python                     | JavaScript/TypeScript             | Description                                                                                                                                                                                                                                                                                                                                                                                    |
| -------------------------------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `combine_under_n_chars` (*int*)  | `combineUnderNChars` (*number*)   | Applies only when the chunking strategy is set to `by_title`. Use this parameter to combines small chunks until the combined chunk reaches a length of `n` characters. This can mitigate the appearance of small chunks created by short paragraphs, not intended as section headings, being identified as `Title` elements in certain documents. Default: the same value as `max_characters`. |
| `include_orig_elements` (*bool*) | `includeOrigElements` (*boolean*) | True (the default) to have the elements that are used to form a chunk appear in `.metadata.orig_elements` for that chunk.                                                                                                                                                                                                                                                                      |
| `max_characters` (*int*)         | `maxCharacters` (*number*)        | Cut off new sections after reaching a length of `n` characters. (This is a hard maximum.) Default: 500.                                                                                                                                                                                                                                                                                        |
| `multipage_sections` (*bool*)    | `multipageSections` (*boolean*)   | Applies only when the chunking strategy is set to `by_title`. Determines if a chunk can include elements from more than one page. Default: true.                                                                                                                                                                                                                                               |
| `new_after_n_chars` (*int*)      | `newAfterNChars` (*number*)       | Applies only when the chunking strategy is specified. Cuts off new sections after reaching a length of `n` characters. (This is a soft maximum.) Default: 1500.                                                                                                                                                                                                                                |
| `overlap` (*int*)                | `overlap` (*number*)              | A prefix of this many trailing characters from the prior text-split chunk is applied to second and later chunks formed from oversized elements by text-splitting. Default: none.                                                                                                                                                                                                               |
| `overlap_all` (*bool*)           | `overlapAll` (*boolean*)          | True to have an overlap also applied to "normal" chunks formed by combining whole elements. Use with caution, as this can introduce noise into otherwise clean semantic units. Default: none.                                                                                                                                                                                                  |
| `similarity_threshold` (*float*) | `similarityThreshold` (*number*)  | Applies only when the chunking strategy is set to `by_similarity`. The minimum similarity text in consecutive elements must have to be included in the same chunk. Must be between 0.0 and 1.0, exclusive (0.01 to 0.99, inclusive). Default: 0.5.                                                                                                                                             |

The following parameters are specific to the Python and JavaScript/TypeScript clients and are not sent to the server. [Learn more](/api-reference/api-services/sdk-python#page-splitting).

| POST, Python                          | JavaScript/TypeScript                 | Description                                                                                                                                                                                                                                          |
| ------------------------------------- | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `split_pdf_page` (*bool*)             | `splitPdfPage` (*boolean*)            | True to split the PDF file client-side. [Learn more](/api-reference/api-services/sdk-python#page-splitting).                                                                                                                                         |
| `split_pdf_allow_failed` (*bool*)     | `splitPdfAllowFailed` (*boolean*)     | When `true`, a failed split request will not stop the processing of the rest of the document. The affected page range will be ignored in the results. When `false`, a failed split request will cause the entire document to fail. Default: `false`. |
| `split_pdf_concurrency_level` (*int*) | `splitPdfConcurrencyLevel` (*number*) | The number of split files to be sent concurrently. Default: 5. Maximum: 15.                                                                                                                                                                          |
| `split_pdf_page_range` (*List\[int]*) | `splitPdfPageRange` (*number\[]*)     | A list of 2 integers within the range `[1, length_of_pdf]`. When pdf splitting is enabled, this will send only the specified page range to the API.                                                                                                  |

Need help getting started? Check out the [Examples page](/api-reference/api-services/examples) for some inspiration.


# API validation errors

This section details the structure of HTTP validation errors returned by the API.

## HTTPValidationError

**Type**: object

**Title**: HTTPValidationError

**Detail**

*   **Type**: array

*   **Description**: An array of ValidationError items, providing detailed information about the validation errors encountered.

## ValidationError

**Type**: object

**Title**: ValidationError

**Required Fields**: loc, msg, type

*   **Location (loc)**

    *   **Type**: array

    *   **Description**: The location of the validation error in the request. Each item in the array can be either a string (e.g., field name) or an integer (e.g., array index).
*   **Message (msg)**

    *   **Type**: string

    *   **Description**: A descriptive message about the validation error.
*   **Error Type (type)**

    *   **Type**: string

    *   **Description**: The type of validation error, categorizing the nature of the error.


# Unstructured API on AWS

Follow these steps to deploy the Unstructured API service into your AWS account.

<Warning>
  This article describes how to create several interrelated resources in your AWS account.
  Your AWS account will be charged on an ongoing basis for these resources, even if you are not actively using them.<br /><br />
  Manually stopping or terminating the associated Amazon EC2 instances alone will not reduce these ongoing charges.<br /><br />
  To stop accruing all related ongoing charges, you must delete all of the associated AWS resources.
  To do this, see [Manage related AWS account costs](#manage-related-aws-account-costs).
</Warning>

*Estimated time to complete: 30 minutes*

You will need:

1.  **An AWS account**:

    *   If you have an existing account, log in: [https://aws.amazon.com/](https://aws.amazon.com) > **Sign In to the Console**.

    *   If you do not have an existing account, create one: [https://aws.amazon.com/free](https://aws.amazon.com/free) > **Create a Free Account**.

2.  **IAM permissions**: In a later step, AWS CloudFormation creates required infrastructure in your account. To learn how to create the associated role, see [Creating IAM Roles with AWS CloudFormation](https://blog.awsfundamentals.com/aws-iam-roles-with-aws-cloudformation#heading-creating-iam-roles-with-aws-cloudformation).

3.  **SSH key pair**: For secure access to the Amazon EC2 instance that CloudFormation creates in a later step, create an SSH key pair. To learn how, see [Create a key pair for your Amazon EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-key-pairs.html).

## Part I: Setting up the Virtual Private Cloud (VPC)

*Note: If you have already configured a Virtual Private Cloud (VPC) for your organization that meets the requirements for deploying the Unstructured API, you may skip this part and proceed to the Part II. Ensure that your existing VPC setup includes the necessary subnets, internet gateway, and route tables as outlined in this guide.*

In Part I, you will construct a resilient and secure infrastructure within AWS by setting up a Virtual Private Cloud (VPC). Your VPC will encompass a dual-tiered subnet model consisting of both **public** and **private** subnets across multiple Availability Zones (AZs).

You will establish the foundational network structure for deploying the Unstructured API by creating two public subnets and one private subnet within your VPC. The public subnets will host resources that require direct access to the internet, such as a load balancer, enabling them to communicate with external users. The private subnet is designed for resources that should not be directly accessible from the internet, like EC2 Compute Engine.

![Infrastructure Diagram](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Infrastructure_Diagram.png)

1.  **Access the VPC dashboard**:

    a.   In the AWS Management Console, in the top menu bar, click **Services > Networking & Content Delivery > VPC**.<br />

    b.   In the sidebar, click **Your VPCs**, and then click **Create VPC**.<br />

2.  **Create the VPC**:

    a.   Select **VPC only**.<br />

    b.   Enter a **Name tag** for your VPC.<br />

    c.   Specify the **IPv4 CIDR block** (for example, `10.0.0.0/16`).<br />

    d.   You may leave **IPv6 CIDR block**, **Tenancy**, and **Tags** settings at their defaults.<br />

    e.   Click **Create VPC**.

![create vpc](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/VPC_Step2.png)

3.  **Create the subnets**:

    a.   After creating the VPC, in the sidebar, click **Subnets**.<br />

    b.   Click **Create subnet**.<br />

    c.   In the **VPC ID** dropdown menu. select the VPC that you just created.<br />

    d.   For the first public subnet:<br />

    *   Enter a **Subnet name**.

    *   Select an **Availability Zone**.

    *   Specify the **IPv4 CIDR block** (for example, `10.0.0.0/16`).

    *   Specify the **IPv4 subnet CIDR block** (for example, `10.0.1.0/24`).

    *   You may leave the **Tags** setting at its default.

    *   Click **Add new subnet**. (Do not click **Create subnet** yet.)

    e.  Repeat the process for the second public subnet with a different **Availability Zone** and **IPv4 subnet CIDR block** (for example, `10.0.2.0/24`).<br />

    *   *Note: Each subnet must reside entirely within one Availability Zone and cannot span zones. If you specify the same Availability Zone or IPv4 subnet CIDR block as the first public subnet, AWS CloudFormation might fail in a later step*.

    *   To learn more, see [Subnet basics](https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html#subnet-basics).

    *   Click **Add new subnet**. (Do not click **Create subnet** yet.)

    f.  Repeat the process for the private subnet with a different **Availability Zone** and **IPv4 subnet CIDR block** (for example, `10.0.3.0/24`).<br />

    *   *Note: Each subnet must reside entirely within one Availability Zone and cannot span zones. If you specify the same Availability Zone or IPv4 subnet CIDR block as the first or second public subnets, AWS CloudFormation might fail in a later step*.

    g.  Click **Create subnet**.

![create subnet](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/VPC_Step3.png)

4.  **Create the internet gateway (for the public subnets)**:

    a.   In the sidebar, click **Internet gateways**.<br />

    b.   Click **Create internet gateway**, enter a **Name tag**, and click **Create internet gateway**.<br />

    c.   In the sidebar, click **Internet gateways** again.<br />

    d.   Click the **Internet gateway ID** for the internet gateway that you just created.<br />

    e.   Click **Actions > Attach to VPC**.<br />

    f.   In the **Available VPCs** dropdown list, select the VPC from *Step 2 - Create the VPC*.<br />

    g.   Click **Attach internet gateway**.<br />

![create internet gateway](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/VPC_Step4.png)

5.  **Set up route tables (for the public subnets)**:

    AWS automatically created a default route table in *Step 3 - Create the subnets*. To tailor your network architecture, you will create a new route table specifically for your public subnets, which will include a route to the internet gateway from *Step 4 - Create the internet gateway (for the public subnets)*.

    a.   In the sidebar, click *Route tables*.

    b.   Click **Create route table**.

    c.   Enter a **Name**.

    d.   Select the **VPC** from *Step 2 - Create the VPC*.

    e.   Click **Create route table**.

![create route table](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/VPC_Step5.png)

6.  **Associate public subnets to the route table and internet gateway**:

    a.  Connect the **public subnets** to the **route table** from *Step 5 - Set up route tables (for the public subnets)*:<br />

    *   In the sidebar, click **Subnets**.

    *   Select the first public subnet from *Step 3 - Create the subnets*.

    *   Click **Actions > Edit route table association**.

    *   In the **Route table ID** dropdown list, select the route table from *Step 5 - Set up route tables (for the public subnets)*, and then click **Save**.

    *   Repeat the process for the second public subnet.

    b.   Now, you’ll ensure that the two public subnets can access the internet by connecting the route table to the internet gateway:<br />

    *   In the sidebar, click **Route tables**.

    *   Select the route table from *Step 5 - Set up route tables (for the public subnets)*.

    *   Click **Actions > Edit routes**.

    *   Click **Add route**, in the destination box, enter `0.0.0.0/0`, which represents all IP addresses. In the **Target** dropdown list, select **Internet Gateway**, and select the internet gateway from *Step 4 - Create the internet gateway (for the public subnets)*.

    *   Click **Save changes** to establish the route, granting internet access to the first and second public subnets at the same time.

    c.  For the **private subnet**:

    *   In the sidebar, click **Subnets**.

    *   Select the private subnet from *Step 3 - Create the subnets*.

    *   Click **Actions > Edit route table association**.

    *   In the **Route table ID** dropdown list, select the main route table, or create and then select a new route table without a route to the internet gateway.

    *   Click **Save**.

![connect public subnet to route table](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/VPC_Step6.png) ![edit routes](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/VPC_Step7.png)

7.  **Inspect the VPC resource map**:

    You can check the configurations from the resource maps on the VPC details dashboard by clicking **Your VPCs** in the sidebar, clicking the **VPC ID** for your VPC, and then clicking the **Resource map** tab.

![VPC Resource Maps](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/VPC_Step8.png)

## Part II: Deploying the Unstructured API from the AWS Marketplace

8.  **Go to the Unstructured API page on AWS Marketplace**:

    a.   Leaving the VPC dashboard from Part I open, in a separate web browser tab, go to the [Unstructured API](http://aws.amazon.com/marketplace/pp/prodview-fuvslrofyuato) product page in the AWS Marketplace.

    b.   Click **Continue to Subscribe**.

    c.   Review the terms and conditions.

    d.   Click **Continue to Configuration**.

![Unstructured API on AWS Marketplace](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Marketplace_Step8.png)

9.  **Configure the CloudFormation template**:

    a.   In the **Fulfillment option** dropdown list, select **CloudFormation Template**.

    b.   For **Fulfillment option** and **Software version**, leave the default `UnstructuredAPI` template and software version.

    c.   In the **Region** dropdown list, select the Region that corresponds to the VPC from Part I.

    *   *Note: You must select the same Region where you set up the VPC in Part I. To find the Region, on the VPC dashboard tab from Part I that you left open, with your VPC displayed, find the VPC's Region name next to your username in the top navigation bar.*

    d.   Click **Continue to Launch**.

    e.   In the **Choose Action** dropdown list, select **Launch CloudFormation**.

    f.   Click **Launch**.

![CloudFormation Configuration](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Marketplace_Step9.png)

10. **Create the CloudFormation stack**:

After you click **Launch**, the **Create stack** page appears in CloudFormation.

**Step 1: Create the stack**

a.   Leave **Choose an existing template** selected.

b.   Leave **Amazon S3 URL** selected and the default **Amazon S3 URL** value unchanged.

c.   Click **Next**.

![Create Stack](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Marketplace_Step10a.png)

**Step 2: Specify the stack's details**

a.   Enter some unique **Stack name**.

b.   In the **Parameters** section, in the **InstanceType** drop-down list, select **m5.xlarge**.

c.   In the **KeyName** drop-down list, select the name of the SSH key pair from the beginning of this article.

d.   In the **LoadBalancerScheme** dropdown list, select **internet-facing**.

e.   For **SSHLocation**, enter `0.0.0.0/0`, but only if you allow public access on the internet.

*   **Note**: It is generally recommended to limit SSH access to a specific IP range for enhanced security. This can be done by setting the `SSHLocation` to the IP address or range associated with your organization. Please consult your IT department or VPN vendor to obtain the correct IP information for these settings.

*   AWS provides `AWS Client VPN`, which is a managed client-based VPN service that enables secure access AWS resources and resources in your on-premises network. To learn more, see [Getting started with AWS Client VPN](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/cvpn-getting-started.html).

f.   In the **Subnets** dropdown multiselect list, select the two public subnets and the private subnet from Part I.

g.   In the **VPC** dropdown list, select the VPC from Part I.

h.   You can leave the default values for all of the other **Parameters** fields.

i.   Click **Next**.

![Specify stack details](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Marketplace_Step10b.png)

**Step 3: Configure the stack's options**

a.   You can leave the default values, or specify any non-default stack options.

b.   Click **Next**.

![Specify stack options](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Marketplace_Step10c.png)

**Step 4: Review**

a.   Review the stack's settings.

b.   Click **Submit**.

![Review stack](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Marketplace_Step10d.png)

11. **Get the Unstructured API endpoint**:

a.   The CloudFormation details page for the stack appears. If you do not see it, on the sidebar, click **Stacks**, and then click the name of your stack.

b.   Check the status of the CloudFormation stack. A successful deployment will show a **CREATE\_COMPLETE** value for the **Status** field on the **Stack Info** tab on this stack's details page. The deployment can take several minutes.

c.   After a successful deployment, click the **Resources** tab on this stack's details page. Then click the **Physical ID** link next to **ApplicationLoadBalancer** on this tab.

d.   On the **EC2 > Load balancers > (Load balancer ID)** page that appears, copy the **DNS Name** value, which is shown as an **(A Record)** and ends with `.elb.amazonaws.com`.

*   Note: You will use this **DNS Name** to replace the `<application-load-balancer-dns-name>` for the following healthcheck and data processing steps.

![Unstructured API Endpoint](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Marketplace_Step11.png)

## Healthcheck

Perform a health check by running this [curl](https://curl.se/) command from a terminal on your local machine, replacing `<application-load-balancer-dns-name>` with your application load balancer's DNS name. This health check can take several minutes:

```bash
curl http://<application-load-balancer-dns-name>/healthcheck

```

![Healthcheck](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/healthcheck.png)

## Data processing

For example, run one of the following, setting the following environment variables to make your code more portable:

*   Set `UNSTRUCTURED_API_URL` to `http://`, followed by your load balancer's DNS name, followed by `/general/v0/general`.

    <Info>You can now use this value (`http://`, followed by your load balancer's DNS name, followed by `/general/v0/general`) in place of
    calling the [Unstructured Serverless API](/api-reference/api-services/saas-api-development-guide) URL or the [Free Unstructured API](/api-reference/api-services/free-api) URL as described elsewhere in the Unstructured API documentation.</Info>

*   Set `LOCAL_FILE_INPUT_DIR` to the path on your local machine to the files for the Unstructured API to process. If you do not have any input files available, you can download any of the ones from the [example-docs](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/example-docs) folder in GitHub.

*   Set `LOCAL_FILE_OUTPUT_DIR` to the path on your local machine for Unstructured API to send the processed output in JSON format:

<AccordionGroup>
  <Accordion title="Ingest CLI">
    You must first [install the Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli).

    Because you are calling a private API and therefore do not need an Unstructured API key, you can omit the command-line option `--api-key` Or, for better code portability, it is recommended that you first set the environment variable `UNSTRUCTURED_API_KEY` to an empty string and then include the command-line option `--api-key`.

    ```bash CLI
    unstructured-ingest \
      local \
        --input-path $LOCAL_FILE_INPUT_DIR \
        --output-dir $LOCAL_FILE_OUTPUT_DIR \
        --partition-by-api \
        --api-key $UNSTRUCTURED_API_KEY \
        --partition-endpoint $UNSTRUCTURED_API_URL \
        --strategy hi_res \
        --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
    ```
  </Accordion>

  <Accordion title="Ingest Python library">
    You must first [install the Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library).

    Because you are calling a private API and therefore do not need an Unstructured API key, you can omit the parameter `api_key`. Or, for better code portability, it is recommended that you first set the environment variable `UNSTRUCTURED_API_KEY` to an empty string and then include the parameter `api_key`.

    ```python Python Ingest v2
    import os

    from unstructured_ingest.v2.pipeline.pipeline import Pipeline
    from unstructured_ingest.v2.interfaces import ProcessorConfig
    from unstructured_ingest.v2.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.v2.processes.partitioner import PartitionerConfig

    if __name__ == "__main__":
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                partition_by_api=True,
                api_key=os.getenv("UNSTRUCTURED_API_KEY"),
                partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
                strategy="hi_res",
                additional_partition_args={
                    "split_pdf_page": True,
                    "split_pdf_allow_failed": True,
                    "split_pdf_concurrency_level": 15
                }
            ),
            uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
        ).run()
    ```
  </Accordion>
</AccordionGroup>

## Accessing the hosting EC2 instance

If you need to access the Amazon EC2 instance that hosts the Unstructured API, do the following:

1.  In the CloudFormation console, open the details page for the stack from Part II. If you do not see it, on the CloudFormation console's sidebar, click **Stacks**, and then click the name of your stack.

2.  Click the **Resources** tab on this stack's details page. Then click the **Physical ID** link next to **EC2TargetGroup** on this tab.

3.  On the **EC2 > Target groups > (CloudFormation stack name)** page that appears, on the **Targets** tab, click the **Instance ID** link.

4.  In the list of instances that appears, click the **Instance ID** link.

5.  Click **Connect**, and then follow any of the on-screen options to access the EC2 instance.

## Manage related AWS account costs

After you run the CloudFormation stack that you created in Part II, charges will begin accruing to your AWS account on an ongoing basis for related AWS resources.
The amounts of these charges vary based on where these resource are located, which resources are covered by AWS Free Tier offerings, the extent to
which you customize these resources' settings, how much you use these resources, and other factors. Stopping or terminating
the related Amazon EC2 instances alone will not eliminate these ongoing charges.

To stop these charges from accruing,
[delete the CloudFormation stack](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-delete-stack.html)
that you created and ran in Part II. This stops and deletes all of the related AWS resources.

Before you delete the stack, note the following:

*   You should click the **Resources** tab on the stack's details page to be aware of the associated resources that will be deleted.
*   You should note any resource dependencies, resources with deletion protection or termination protection enabled, or nested stacks
    that might prevent stack deletion, and resolve these issues that could prevent stack deletion. To find these kinds of issues:

    *   On the **Template** tab on the stack's details page, look for occurences of the `DependsOn` attribute, which are set to the name
        of the resource dependency.
    *   On the **Template** tab on the stack's details page, look for occurences of the `DeletionPolicy` attribute set to `Retain` or the
        `UpdateReplacePolicy` attribute set to `Retain`. The associated resources have deletion protection enabled.
    *   On the **Stack info** tab on the stack's details page, look for the **Termination protection** field. If it is set to **Activated**,
        termination protection is enabled.
    *   On the **Resources** tab on the stack's details page, look for resources with their **Type** set to `AWS::CloudFormation::Stack`. These indicate nested stacks.

After you delete the stack, you should check your [AWS Billing and Cost Management dashboard](https://docs.aws.amazon.com/awsconsolehelpdocs/latest/gsg/billinginfo.html) to confirm that associated charges are no longer accruing.


# Unstructured API on Azure



Follow these steps to deploy the Unstructured API service into your Azure account.

<Warning>
  This article describes how to create several interrelated resources in your Azure account.
  Your Azure account will be charged on an ongoing basis for these resources, even if you are not actively using them.<br /><br />
  Manually shutting down the associated Azure virtual machine when you are not using it can help reduce—but not fully eliminate—these ongoing charges.<br /><br />
  To stop accruing all related ongoing charges, you must delete all of the associated Azure resources.
</Warning>

<Steps>
  <Step title=" Log in to the Azure Portal">
    Go to [https://portal.azure.com](https://portal.azure.com/).
  </Step>

  <Step title="Access the Azure Marketplace">
    Go to the [Unstructured Data Preprocessing - Customer Hosted API](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/unstructured1691024866136.customer_api_v1?tab=Overview/) offering in the Azure Marketplace.

    ![Azure Marketplace](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Azure_Step2.png)
  </Step>

  <Step title="Start the deployment process">
    1.  Click **Get It Now** and fill out the form.
    2.  Read the terms and click **Continue**.
    3.  Click **Create**.

    ![Deployment Process](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Azure_Step3.png)
  </Step>

  <Step title=" Configure the deployment options">
    1.  On the **Create a virtual machine** page, click the **Basics** tab.

    2.  In the **Project details** section, select an existing **Subscription**, and select an **Resource group** from the dropdown menus, or create a new resource group by clicking **Create new**.

        ![project details](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Azure_Step4a.png)

    3.  In the **Instance details** section, enter a name in the **Virtual machine name** field. Note this name, as you will need it later steps.

    4.  Select a **Region** from the dropdown menu.

    5.  For **Image**, select **Unstructured Customer Hosted API Hourly - x64 Gen2** (*default*).

    6.  For **Size**, select a VM size from the dropdown menu, or leave the default VM size selection. To learn more, see [Azure VM comparisons](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/).

        ![instance details](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Azure_Step4b.png)

    7.  In the **Administrator account** section, for **Authentication type**, select **SSH public key** or **Password**.

    8.  Enter the credential settings, depending on the authentication type.

        ![administrator account](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Azure_Step4c.png)

    <Note>Do not click **Review + create** yet. You must still set up the load balancer.</Note>
  </Step>

  <Step title="Set up the load balancer">
    1.  Click the **Networking** tab.

    2.  In the **Networking interface** section, fill out the following fields:

        *   **Virtual network**: Click **Create new**, or select an existing virtual network from the dropdown menu. To learn more, see [Quickstart: Use the Azure portal to create a virtual network](https://learn.microsoft.com/en-us/azure/virtual-network/quick-create-portal).
        *   **Subnet**: Click **Manage subnet configuration**, or select a new or existing subnet from the dropdown menu. To learn more, see [Add, change, or delete a virtual network subnet](https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-manage-subnet?tabs=azure-portal).
        *   **Configure network security group**: Click **Create new**, or select an existing security group from the dropdown menu. To learn more, see [Create, change, or delete a network security group](https://learn.microsoft.com/en-us/azure/virtual-network/manage-network-security-group?tabs=network-security-group-portal).

    3.  In the **Load balancing** section, fill out the following fields:

        *   **Load balancing options**: Select **Azure load balancer**.
        *   **Select a load balancer**: Click **Create a load balancer** and fill out the following fields in the pop-up window, or select an existing load balancer from the dropdown menu and note this name as you will need it in later steps:

            *   Enter a **Load balancer name**. Note this name, as you will need it in later steps.
            *   For **Type**, select **Public** or **Internal**.
            *   For **Protocol**, select **TCP** or **UDP**.
            *   Set both **Port** and **Backend port** to **80**.

        ![load balancer](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Azure_Step5.png)

    4.  Click **Create**.
  </Step>

  <Step title="Finalize and deploy">
    1.  Click **Review + create**.
    2.  Wait for validation.
    3.  Click **Create**.

        ![deployment](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Azure_Step6.png)
  </Step>

  <Step title="Post-deployment: additional load balancer configuration">
    1.  Go to your load balancer: in the Azure portal's **Search resources, services and docs** box, enter and then select **Load balancers**.
    2.  Search for and open the new load balancer that you created earlier, or the existing load balancer that you chose earlier.
    3.  Make any necessary settings updates to your new or existing load balancer, based on the recommended configurations in the [Load balancer network settings](#load-balancer-network-settings) section, later on this page.
  </Step>

  <Step title="Post-deployment: get the deployed endpoint URL">
    1.  Go to your virtual machine: in the Azure portal's **Search resources, services and docs** box, enter and then select **Virtual machines**.

    2.  Search for and open the new virtual machine that you created earlier, using the name that you entered earlier.

    3.  On the **Overview** tab, under **Properties**, note the **Public IP address** for the **Load balancer**.

        ![retrieve public ip](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Azure_Step7.png)

    4.  The deployed endpoint URL is **http\://\<load-balancer-public-IP-address>/general/v0/general**. Note this endpoint URL, as you will need it later to call the Unstructured API.
  </Step>

  <Step title="Post-deployment: set API environment variables">
    Note the API environment variables in the [API environment variables](#api-environment-variables) section, later on this page. If you need to set any of these in the Docker container on the virtual machine, do the following:

    1.  If the virtual machine is not already running from earlier, click the **Start** icon.

    2.  After the virtual machine starts, click the **Connect** icon, and then click **Connect** from the drop-down list.

    3.  Follow the on-screen directions for one of the available options to connect to the virtual machine and display a connected terminal.

    4.  Stop the running container in the virtual machine, so that you can restart it later with the environment variables set: In the connected terminal, run the following command: `sudo docker container ls`.

    5.  Note the `CONTAINER ID` value for the running container.

    6.  Run the following command, replacing `<CONTAINER ID>` with the `CONTAINER ID` value:

        ```bash
        sudo docker container rm --force <CONTAINER ID>
        ```

    7.  Now run the container again, setting the environment variables at the same time: Run the following command: `sudo docker image ls`.

    8.  Note the `REPOSITORY` and `TAG` value for the Docker image.

    9.  Run the following command, replacing `<REPOSITORY>` and `<TAG>` with the `REPOSITORY` and `TAG` values for the Docker image, and replacing
        `<VAR1>=<value1>`, `<VAR2>=<value2>` and so on with the environment variable name and value pairs:

        ```bash
        sudo docker run -d --restart unless-stopped \
        -p 80:5000 \
        -e <VAR1>=<value1> -e <VAR2>=<value2> -e <VAR3>=<value3> \
        <REPOSITORY>:<TAG>
        ```

    10. Verify that the environment variables were set correctly: Run the following command:

        ```bash
        sudo docker container ls
        ```

    11. Note the `CONTAINER ID` value for the running container.

    12. Run the following command, replacing `<CONTAINER ID>` with the `CONTAINER ID` value:

    ```bash
    sudo docker exec <CONTAINER ID> bash -c 'printenv'
    ```

    14. The environment variables should be in the list that appears.

    <Note>To help manage your overall costs, you should click the **Stop** icon whenever you are not using this virtual machine to call the Unstructured API.</Note>
  </Step>

  <Step title="Call the Unstructured API">
    You can now use the running virtual machine to call the Unstructured API. For example, run one of the following, setting the following environment variables to make your code more portable:

    *   Set `UNSTRUCTURED_API_URL` to `http://`, followed by your load balancer's public IP address, followed by `/general/v0/general`.
    *   Set `LOCAL_FILE_INPUT_DIR` to the path on your local machine to the files for the Unstructured API to process. If you do not have any input files available, you can download any of the ones from the [example-docs](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/example-docs) folder in GitHub.
    *   Set `LOCAL_FILE_OUTPUT_DIR` to the path on your local machine for Unstructured API to send the processed output in JSON format.
  </Step>
</Steps>

<AccordionGroup>
  <Accordion title="Ingest CLI">
    You must first [install the Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli).

    Because you are calling a private API and therefore do not need an Unstructured API key, you can omit the command-line option `--api-key` Or, for better code portability, it is recommended that you first set the environment variable `UNSTRUCTURED_API_KEY` to an empty string and then include the command-line option `--api-key`.

    ```bash CLI
    unstructured-ingest \
      local \
        --input-path $LOCAL_FILE_INPUT_DIR \
        --output-dir $LOCAL_FILE_OUTPUT_DIR \
        --partition-by-api \
        --api-key $UNSTRUCTURED_API_KEY \
        --partition-endpoint $UNSTRUCTURED_API_URL \
        --strategy hi_res \
        --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
    ```
  </Accordion>

  <Accordion title="Ingest Python library">
    You must first [install the Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library).

    Because you are calling a private API and therefore do not need an Unstructured API key, you can omit the parameter `api_key`. Or, for better code portability, it is recommended that you first set the environment variable `UNSTRUCTURED_API_KEY` to an empty string and then include the parameter `api_key`.

    ```python Python Ingest v2
    import os

    from unstructured_ingest.v2.pipeline.pipeline import Pipeline
    from unstructured_ingest.v2.interfaces import ProcessorConfig
    from unstructured_ingest.v2.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.v2.processes.partitioner import PartitionerConfig

    if __name__ == "__main__":
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                partition_by_api=True,
                api_key=os.getenv("UNSTRUCTURED_API_KEY"),
                partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
                strategy="hi_res",
                additional_partition_args={
                    "split_pdf_page": True,
                    "split_pdf_allow_failed": True,
                    "split_pdf_concurrency_level": 15
                }
            ),
            uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
        ).run()
    ```
  </Accordion>
</AccordionGroup>

<Note>To help manage your overall costs, you should stop running the associated virtual machine whenever you are not using it to call the Unstructured API.</Note>

## Load balancer network settings

Unstructured recommends the following load balancer settings, which you should set on your deployment's load balancer soon after you finalize and deploy it.

On the load balancer's **Overview** tab in the Azure portal:

*   **SKU**: **Standard**

On the load balancer's **Settings** tab in the Azure portal:

*   **Frontend IP configuration**: Private IP

*   **Backend pools**: VMSS

*   **Health probes**:

    *   **Protocol**: **HTTP**, or **HTTPS** (this requires setting up a reverse proxy on the VMSS set to do TLS termination)
    *   **Port**: `80` or `443` (this can be any port that the backend VMs are listening on)
    *   **Path**: `/healthcheck`
    *   **Interval (seconds)**: `5`

*   **Load balancing rules**:

    *   **Protocol**: **TCP**
    *   **Port**: `443` for HTTPS, or `80` for HTTP
    *   **Backend port**: `443` for HTTPS, or `80` for HTTP
    *   **Idle timeout (minutes)**: `60`
    *   **Enable TCP Reset** box: Checked

*   **Inbound NAT rules**:

    *   **Frontend Port**: `443` for HTTPS, or `80` for HTTP
    *   **Backend port**: `443` for HTTPS, or `80` for HTTP
    *   **Protocol**: **TCP**
    *   **Enable TCP Reset** box: Checked
    *   **Idle timeout (minutes)**: `60`

## API environment variables

Unstructured supports the following environment variables, which you can set in the Docker image on the virtual machine, as needed:

*   `ALLOW_ORIGINS`: CORS-allowed origins.
*   `UNSTRUCTURED_ALLOWED_MIMETYPE`: The list of allowed MIME types, if you want to limit the file types that can be processed.
*   `UNSTRUCTURED_API_KEY`: The default Unstructured API key to use.
*   `UNSTRUCTURED_MEMORY_FREE_MINIMUM_MB`: The minimum amount of free memory in MB to allow for processing a file. If this memory is too low, the server will return a `503` error.
*   `UNSTRUCTURED_PDF_HI_RES_MAX_PAGES`: The maximum number of pages in a PDF file that the Unstructured API will not reject, if the `hi_res` strategy is used. The default is `300`.
*   `UNSTRUCTURED_REDIRECT_ROOT_URL`: If this is set, redirect a `GET` request to the Unstructured API to use this URL instead.


# Chunking strategies



Chunking functions use metadata and document elements detected with partition functions to split a document into
appropriately-sized chunks for uses cases such as Retrieval Augmented Generation (RAG).

If you are familiar with chunking methods that split long text documents into smaller chunks, you'll notice that
Unstructured methods slightly differ, since the partitioning step already divides an entire document into its structural elements.

Individual elements will only be split if they exceed the desired maximum chunk size. Two or more consecutive text elements
that will together fit within `max_characters` will be combined. After chunking, you will only have elements of the
following types:

*   `CompositeElement`: Any text element will become a `CompositeElement` after chunking. A composite element can be a
    combination of two or more original text elements that together fit within the maximum chunk size. It can also be a single
    element that doesn't leave room in the chunk for any others but fits by itself. Or it can be a fragment of an original
    text element that was too big to fit in one chunk and required splitting.
*   `Table`:  A table element is not combined with other elements and if it fits within `max_characters` it will remain as is.
*   `TableChunk`: large tables that exceed `max_characters` chunk size are split into special `TableChunk` elements.

### "basic" chunking strategy

*   The basic strategy combines sequential elements to maximally fill each chunk while respecting both the specified `max_characters` (hard-max) and `new_after_n_chars` (soft-max) option values.

*   A single element that by itself exceeds the hard-max is isolated (never combined with another element) and then divided into two or more chunks using text-splitting.

*   A `Table` element is always isolated and never combined with another element. A `Table` can be oversized, like any other text element, and in that case is divided into two or more `TableChunk` elements using text-splitting.

*   If specified, `overlap` is applied between chunks formed by splitting oversized elements and is also applied between other chunks when `overlap_all` is `True`.

### "by\_title" chunking strategy

The `by_title` chunking strategy preserves section boundaries and optionally page boundaries as well. “Preserving” here means that a single chunk will never contain text that occurred in two different sections. When a new section starts, the existing chunk is closed and a new one started, even if the next element would fit in the prior chunk.

In addition to the behaviors of the `basic` strategy above, the `by_title` strategy has the following behaviors:

*   **Detect section headings.** A `Title` element is considered to start a new section. When a `Title` element is encountered, the prior chunk is closed and a new chunk started, even if the `Title` element would fit in the prior chunk.

*   **Respect page boundaries.** Page boundaries can optionally also be respected using the `multipage_sections` argument. This defaults to `True` meaning that a page break does *not* start a new chunk. Setting this to `False` will separate elements that occur on different pages into distinct chunks.

*   **Combine small sections.** In certain documents, partitioning may identify a list-item or other short paragraph as a `Title` element even though it does not serve as a section heading. This can produce chunks substantially smaller than desired. This behavior can be mitigated using the `combine_text_under_n_chars` argument. This defaults to the same value as `max_characters` such that sequential small sections are combined to maximally fill the chunking window. Setting this to `0` will disable section combining.

### "by\_page" chunking strategy

Only available in Unstructured API and Platform.

The `by_page` chunking strategy ensures the content from different pages do not end up in the same chunk.
When a new page is detected, the existing chunk is completed and a new one is started, even if the next element would fit in the
prior chunk.

### "by\_similarity" chunking strategy

Only available in Unstructured API and Platform.

The `by_similarity` chunking strategy employs the `sentence-transformers/multi-qa-mpnet-base-dot-v1` embedding model to
identify topically similar sequential elements and combine them into chunks.

As with other strategies, chunks will never exceed the hard-maximum chunk size set by `max_characters`. For this reason,
not all elements that share a topic will necessarily appear in the same chunk. However, with this strategy you can
guarantee that two elements with low similarity will not be combined in a single chunk.

You can control the level of topic similarity you require for elements to have by setting the `similarity_threshold` parameter.
`similarity_threshold` expects a value between 0.0 and 1.0 specifying the minimum similarity text in consecutive elements
must have to be included in the same chunk. The default is 0.5.

###

## Learn more

<Icon icon="blog" />  [Chunking for RAG: best practices](https://unstructured.io/blog/chunking-for-rag-best-practices)


# Document elements and metadata



`unstructured` simplifies and streamline the preprocessing of structured and unstructured documents for downstream tasks.
What that means is no matter where your data is and no matter what format that data is in, Unstructured's toolkit will
transform and preprocess that data into an easily digestible and usable format that is uniform across data formats.

When you partition a document with Unstructured, the result is a list of document `Element` objects.
These element objects represent different components of the source document.

## Element example

Here's an example of what an element might look like:

```
{'type': 'NarrativeText',
 'element_id': '5ef1d1117721f0472c1ad825991d7d37',
 'text': 'The Unstructured API documentation covers the following API services:',
 'metadata': {'last_modified': '2024-05-01T14:15:22',
  'page_number': 1,
  'languages': ['eng'],
  'parent_id': '56f24319ae258b735cac3ec2a271b1d9',
  'file_directory': '/content',
  'filename': 'Unstructured API services - Unstructured.html',
  'filetype': 'text/html'}}
```

As you can see, every element will have a `type`, an `element_id`, the extracted `text`, and some `metadata` which may
vary depending on the element type, document structure, and some additional parameters used during partitioning and/or chunking.

Let's explore some of these document element components in more detail.

## Element type

Instead of treating all documents like a wall of plain text, Unstructured preserves the semantic structure of the documents.
This gives you more control and flexibility over how you further use the processed documents and allows you to take their
structure into consideration. At the same time, normalizing data from various file formats to the Unstructured element
type scheme lets you treat all documents the same in your downstream processing, regardless of source format.
For example, if you plan to summarize a document, you may only be interested in the narrative
of the document, and not care about footers and headers. You can easily filter out the elements you don't need using their type.

Here are some examples of the element types your document may contain:

| Element type        | Description                                                                                                                                        |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Formula`           | An element containing formulas in a document.                                                                                                      |
| `FigureCaption`     | An element for capturing text associated with figure captions.                                                                                     |
| `NarrativeText`     | NarrativeText is an element consisting of multiple, well-formulated sentences. This excludes elements such titles, headers, footers, and captions. |
| `ListItem`          | ListItem is a NarrativeText element that is part of a list.                                                                                        |
| `Title`             | A text element for capturing titles.                                                                                                               |
| `Address`           | A text element for capturing physical addresses.                                                                                                   |
| `EmailAddress`      | A text element for capturing email addresses.                                                                                                      |
| `Image`             | A text element for capturing image metadata.                                                                                                       |
| `PageBreak`         | An element for capturing page breaks.                                                                                                              |
| `Table`             | An element for capturing tables.                                                                                                                   |
| `Header`            | An element for capturing document headers.                                                                                                         |
| `Footer`            | An element for capturing document footers.                                                                                                         |
| `CodeSnippet`       | An element for capturing code snippets.                                                                                                            |
| `PageNumber`        | An element for capturing page numbers.                                                                                                             |
| `UncategorizedText` | Base element for capturing free text from within document.                                                                                         |

If you apply chunking during partitioning of a document or later, you will also see the `CompositeElement` type.
`CompositeElement` is a chunk formed from text (non-Table) elements. It is only produced by chunking.
A composite element may be formed by combining one or more sequential elements produced by partitioning. For example,
several individual list items may be combined into a single chunk.

## Element ID

By default, the element ID is a SHA-256 hash of the element's text, its position on the page, page number it's on,
and the name of the document file - this is to ensure that the ID is deterministic and unique at the document level.
To obtain globally unique IDs in the output (UUIDs), you can pass `unique_element_ids=True` into any of the partition
functions. This can be helpful if you'd like to use the IDs as a primary key in a database, for example.

## Metadata

Unstructured tracks a variety of metadata about the elements extracted from documents.
Here are a couple of examples of what element metadata enables you to do:

*   filter document elements based on an element metadata value. For instance, you may want to limit your scope to elements from a certain page, or you may want to use only elements that have an email matching a regular expression in their metadata.
*   map an element to the document page where it occurred so that original page can be retrieved when that element matches search criteria.

Metadata is tracked at the element level. You can access the metadata for a given document element with
`element.metadata`. For a dictionary representation, use `element.metadata.to_dict()`.

### Common metadata fields

All document types return the following metadata fields when the information is available from the source file:

| Metadata field name        | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `filename`                 | Filename                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `file_directory`           | File directory                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `last_modified`            | Last modified Date                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `filetype`                 | File type                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `coordinates`              | XY Bounding Box Coordinates. See notes below for further details about the bounding box.                                                                                                                                                                                                                                                                                                                                                                                           |
| `parent_id`                | Element Hierarchy. `parent_id` may be used to infer where an element resides within the overall hierarchy of a document. For instance, a NarrativeText element may have a Title element as a parent (a “sub-title”), which in turn may have another Title element as its parent (a "title").                                                                                                                                                                                       |
| `category_depth`           | Element depth relative to other elements of the same category.  Category depth is the depth of an element relative to other elements of the same category. It’s set by a document partitioner and enables the hierarchy post-processor to compute more accurate hierarchies. Category depth may be set using native document hierarchies, e.g. reflecting \<H1>, \<H2>, or \<H3> tags within an HTML document or the indentation level of a bulleted list item in a Word document. |
| `text_as_html`             | HTML representation of extracted tables. Only applicable to table elements.                                                                                                                                                                                                                                                                                                                                                                                                        |
| `languages`                | Document Languages.  At document level or element level. List is ordered by probability of being the primary language of the text.                                                                                                                                                                                                                                                                                                                                                 |
| `emphasized_text_contents` | Emphasized text (bold or italic) in the original document.                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `emphasized_text_tags`     | Tags on text that is emphasized in the original document.                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `is_continuation`          | True if element is a continuation of a previous element.  Only relevant for chunking, if an element was divided into two due to max\_characters.                                                                                                                                                                                                                                                                                                                                   |
| `detection_class_prob`     | Detection model class probabilities.  From unstructured-inference, hi-res strategy.                                                                                                                                                                                                                                                                                                                                                                                                |

Notes on common metadata fields:

#### Metadata for document hierarchy

`parent_id` and `category_depth` enhance hierarchy detection to identify the document
structure in various file formats by measuring relative depth of an element within its category. This is especially
useful in documents with native hierarchies like HTML or Word files, where elements like headings or list items inherently define structure.

#### Element's coordinates

Some document types support location data for the elements, usually in the form of bounding boxes.
If it exists, an element's location data is available with `element.metadata.coordinates`. The `coordinates` property of an `ElementMetadata` stores:

*   `points` : These specify the corners of the bounding box starting from the top left corner and proceeding counter-clockwise. The points represent pixels, the origin is in the top left and the `y` coordinate increases in the downward direction.
*   `system`: The points have an associated coordinate system. A typical example of a coordinate system is `PixelSpace`, which is used for representing the coordinates of images. The coordinate system has a name, orientation, layout width, and layout height.

The Unstructured Open Source library offers a way to change the coordinates of an element to a new coordinate system by
using the `Element.convert_coordinates_to_new_system` method. If the `in_place` flag is `True`, the coordinate system
and points of the element are updated in place and the new coordinates are returned. If the `in_place` flag is `False`,
only the altered coordinates are returned.

```python
from unstructured.documents.elements import Element
from unstructured.documents.coordinates import PixelSpace, RelativeCoordinateSystem

coordinates = ((10, 10), (10, 100), (200, 100), (200, 10))
coordinate_system = PixelSpace(width=850, height=1100)

element = Element(coordinates=coordinates, coordinate_system=coordinate_system)
print(element.metadata.coordinates.to_dict())
print(element.metadata.coordinates.system.orientation)
print(element.metadata.coordinates.system.width)
print(element.metadata.coordinates.system.height)

element.convert_coordinates_to_new_system(RelativeCoordinateSystem(), in_place=True)
# Should now be in terms of new coordinate system
print(element.metadata.coordinates.to_dict())
print(element.metadata.coordinates.system.orientation)
print(element.metadata.coordinates.system.width)
print(element.metadata.coordinates.system.height)

```

### Additional metadata fields by document type

| Field Name             | Applicable Doc Types | Description                                                                     |
| ---------------------- | -------------------- | ------------------------------------------------------------------------------- |
| `page_number`          | DOCX, PDF, PPT, XLSX | Page number                                                                     |
| `page_name`            | XLSX                 | Sheet name in an Excel document                                                 |
| `sent_from`            | EML                  | Email sender                                                                    |
| `sent_to`              | EML                  | Email recipient                                                                 |
| `subject`              | EML                  | Email subject                                                                   |
| `attached_to_filename` | MSG                  | filename that attachment file is attached to                                    |
| `header_footer_type`   | Word Doc             | Pages a header or footer applies to: "primary", "even\_only", and "first\_page" |
| `link_urls`            | HTML                 | The url associated with a link in a document.                                   |
| `link_texts`           | HTML                 | The text associated with a link in a document.                                  |
| `section`              | EPUB                 | Book section title corresponding to table of contents                           |

Notes on additional metadata by document type:

#### Email

Emails will include `sent_from`, `sent_to`, and `subject` metadata. `sent_from` is a list of strings because
the [RFC 822](https://www.rfc-editor.org/rfc/rfc822) spec for emails allows for multiple sent from email addresses.

#### Microsoft Excel documents

For Excel documents, `ElementMetadata` will contain a `page_name` element, which corresponds to the sheet name in the Excel
document.

#### Microsoft Word documents

Headers and footers in Word documents include a `header_footer_type` indicating which page a header or footer applies to.
Valid values are `"primary"`, `"even_only"`, and `"first_page"`.

### Table-specific metadata

For `Table` elements, the raw text of the table will be stored in the `text` attribute for the Element, and HTML representation
of the table will be available in the element metadata under `element.metadata.text_as_html`. By default,
Unstructured will automatically extract all tables for all doc types unless you set `skip_infer_table_types` parameter.

Here's an example of a table element. The `text` of the element will look like this:

```
Dataset Base Model1 Large Model Notes PubLayNet [38] F / M M Layouts of modern scientific documents PRImA [3] M - Layouts of scanned modern magazines and scientific reports Newspaper [17] F - Layouts of scanned US newspapers from the 20th century TableBank [18] F F Table region on modern scientific and business document HJDataset [31] F / M - Layouts of history Japanese documents
```

And the `text_as_html` metadata for the same element will look like this:

```py
<table><thead><th>Dataset</th><th>| Base Model’</th><th>| Notes</th></thead><tr><td>PubLayNet</td><td>[38] F/M</td><td>Layouts of modern scientific documents</td></tr><tr><td>PRImA [3]</td><td>M</td><td>Layouts of scanned modern magazines and scientific reports</td></tr><tr><td>Newspaper</td><td>F</td><td>Layouts of scanned US newspapers from the 20th century</td></tr><tr><td>TableBank</td><td>F</td><td>Table region on modern scientific and business document</td></tr><tr><td>HJDataset [31]</td><td>F/M</td><td>Layouts of history Japanese documents</td></tr></table>
```

### Data connector metadata fields

Documents processed through source connectors include additional document metadata. These additional fields only ever
appear if the source document was processed by a connector.

#### Common data connector metadata fields

*   Data Source metadata (on json output):
    *   url
    *   version
    *   date created
    *   date modified
    *   date processed
    *   record locator
*   Record locator is specific to each connector

#### Additional metadata fields by connector type (via record locator)

| Source connector      | Additional metadata              |
| --------------------- | -------------------------------- |
| airtable              | base id, table id, view id       |
| azure (from fsspec)   | protocol, remote file path       |
| box (from fsspec)     | protocol, remote file path       |
| confluence            | url, page id                     |
| discord               | channel                          |
| dropbox (from fsspec) | protocol, remote file path       |
| elasticsearch         | url, index name, document id     |
| fsspec                | protocol, remote file path       |
| google drive          | drive id, file id                |
| gcs (from fsspec)     | protocol, remote file path       |
| jira                  | base url, issue key              |
| onedrive              | user pname, server relative path |
| outlook               | message id, user email           |
| s3 (from fsspec)      | protocol, remote file path       |
| sharepoint            | server path, site url            |
| wikipedia             | page title, age url              |


# Examples

This page provides some examples of accessing Unstructured API via different methods.

For each of these examples, you'll need:

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

<Info>
  If you do not specify the API URL, your [Unstructured Serverless API](/api-reference/api-services/saas-api-development-guide) pay-as-you-go account will be used by default. You must always specify your Serverless API key.<br /><br />
  To use the [Free Unstructured API](/api-reference/api-services/free-api), you must always specify your Free API key, and the Free API URL which is `https://api.unstructured.io/general/v0/general`<br /><br />
  To use the pay-as-you-go Unstructured API on Azure or AWS with the SDKs, you must always specify the corresponding API URL. See the [Azure](/api-reference/api-services/azure) or [AWS](/api-reference/api-services/aws) instructions.
</Info>

### Changing partition strategy for a PDF

Here's how you can modify partition strategy for a PDF file, and select an alternative model to use with Unstructured API.
The `hi_res` strategy supports different models, and the default is `layout_v1.1.0`.

<iframe width="560" height="315" src="https://www.youtube.com/embed/SwJVB_kPqTc" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

<AccordionGroup>
  <Accordion title="Ingest CLI">
    ```bash CLI
    unstructured-ingest \
      local \
        --input-path $LOCAL_FILE_INPUT_DIR \
        --output-dir $LOCAL_FILE_OUTPUT_DIR \
        --strategy hi_res \
        --hi-res-model-name layout_v1.1.0 \
        --partition-by-api \
        --api-key $UNSTRUCTURED_API_KEY \
        --partition-endpoint $UNSTRUCTURED_API_URL \
        --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
    ```
  </Accordion>

  <Accordion title="Ingest Python">
    ```python Python
    import os

    from unstructured_ingest.v2.pipeline.pipeline import Pipeline
    from unstructured_ingest.v2.interfaces import ProcessorConfig
    from unstructured_ingest.v2.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.v2.processes.partitioner import PartitionerConfig

    if __name__ == "__main__":
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                strategy="hi_res",
                hi_res_model_name="layout_v1.0.0",
                partition_by_api=True,
                api_key=os.getenv("UNSTRUCTURED_API_KEY"),
                partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
                additional_partition_args={
                    "split_pdf_page": True,
                    "split_pdf_allow_failed": True,
                    "split_pdf_concurrency_level": 15
                }
            ),
            uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
        ).run()
    ```
  </Accordion>

  <Accordion title="POST">
    <Info>
      Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
      [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
      to you:

      *   You need to work with documents in cloud storage.
      *   You want to cache the results of processing multiple files in batches.
      *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
    </Info>

    ```bash POST
    curl -X 'POST' $UNSTRUCTURED_API_URL \
    -H 'accept: application/json' \
    -H 'Content-Type: multipart/form-data' \
    -H 'unstructured-api-key: $UNSTRUCTURED_API_KEY' \
    -F 'files=@sample-docs/layout-parser-paper.pdf' \
    -F 'strategy=hi_res' \
    -F 'hi_res_model_name=layout_v1.1.0'
    ```
  </Accordion>

  <Accordion title="Python SDK">
    <Info>
      Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
      [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
      to you:

      *   You need to work with documents in cloud storage.
      *   You want to cache the results of processing multiple files in batches.
      *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
    </Info>

    ```python Python
    import asyncio
    import os
    import json
    import unstructured_client
    from unstructured_client.models import shared

    client = unstructured_client.UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
        server_url=os.getenv("UNSTRUCTURED_API_URL"),
    )

    async def call_api(filename, input_dir, output_dir):
        req = {
            "partition_parameters": {
                "files": {
                    "content": open(filename, "rb"),
                    "file_name": os.path.basename(filename),
                },
                "strategy": shared.Strategy.HI_RES,
                "hi_res_model_name": "layout_v1.1.0",
                "split_pdf_page": True,
                "split_pdf_allow_failed": True,
                "split_pdf_concurrency_level": 15
            }
        }

        try:
            res = await client.general.partition_async(request=req)
            element_dicts = [element for element in res.elements]
            json_elements = json.dumps(element_dicts, indent=2)

            # Create the output directory structure.
            relative_path = os.path.relpath(os.path.dirname(filename), input_dir)
            output_subdir = os.path.join(output_dir, relative_path)
            os.makedirs(output_subdir, exist_ok=True)

            # Write the output file.
            output_filename = os.path.join(output_subdir, os.path.basename(filename) + ".json")
            with open(output_filename, "w") as file:
                file.write(json_elements)

        except Exception as e:
            print(f"Error processing {filename}: {e}")

    async def process_files(input_directory, output_directory):
        tasks = []

        for root, _, files in os.walk(input_directory):
            for file in files:
                if not file.endswith('.json'):
                    full_path = os.path.join(root, file)
                    tasks.append(call_api(full_path, input_directory, output_directory))

        await asyncio.gather(*tasks)

    if __name__ == "__main__":
        asyncio.run(process_files(
            input_directory=os.getenv("LOCAL_FILE_INPUT_DIR"), 
            output_directory=os.getenv("LOCAL_FILE_OUTPUT_DIR")
        ))
    ```
  </Accordion>

  <Accordion title="JavaScript/TypeScript SDK">
    <Info>
      Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
      [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
      to you:

      *   You need to work with documents in cloud storage.
      *   You want to cache the results of processing multiple files in batches.
      *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
    </Info>

    ```typescript TypeScript
    import { UnstructuredClient } from "unstructured-client";
    import * as fs from "fs";
    import * as path from "path";
    import { Strategy } from "unstructured-client/sdk/models/shared/index.js";
    import { PartitionResponse } from "unstructured-client/sdk/models/operations";

    // Send all files in the source path to Unstructured for processing.
    // Send the processed data to the destination path.
    function processFiles(
        client: UnstructuredClient,
        sourcePath: string,
        destinationPath: string
    ): void {

        // If an output directory does not exist for the corresponding input
        // directory, then create it.
        if (!fs.existsSync(destinationPath)) {
            fs.mkdirSync(destinationPath, { recursive: true });
        }

        // Get all folders and files at the current level of the input directory.
        const items = fs.readdirSync(sourcePath);

        // For each folder and file in the input directory...
        for (const item of items) {
            const inputPath = path.join(sourcePath, item);
            const outputPath = path.join(destinationPath, item)

            // If it's a folder, call this function recursively.
            if (fs.statSync(inputPath).isDirectory()) {
                processFiles(client, inputPath, outputPath);
            } else {
                // If it's a file, send it to Unstructured for processing.
                const data = fs.readFileSync(inputPath);

                client.general.partition({
                    partitionParameters: {
                        files: {
                            content: data,
                            fileName: inputPath
                        },
                        strategy: Strategy.HiRes,
                        hiResModelName: "layout_v1.1.0", 
                        splitPdfPage: true,
                        splitPdfConcurrencyLevel: 15,
                        splitPdfAllowFailed: true
                    }
                }).then((res: PartitionResponse) => {
                    // If successfully processed, write the processed data to
                    // the destination directory.
                    if (res.statusCode == 200) {
                        const jsonElements = JSON.stringify(res.elements, null, 2)
                        fs.writeFileSync(outputPath + ".json", jsonElements)
                    }
                }).catch((e) => {
                    if (e.statusCode) {
                        console.log(e.statusCode);
                        console.log(e.body);
                    } else {
                        console.log(e);
                    }
                });
            }
        }
    }

    const client = new UnstructuredClient({
        security: { apiKeyAuth: process.env.UNSTRUCTURED_API_KEY },
        serverURL: process.env.UNSTRUCTURED_API_URL
    });

    processFiles(
        client,
        process.env.LOCAL_FILE_INPUT_DIR,
        process.env.LOCAL_FILE_OUTPUT_DIR
    );
    ```
  </Accordion>
</AccordionGroup>

If you have a local deployment of the Unstructured API, you can use other supported models, such as `yolox`.

### Specifying the language of a document for better OCR results

For better OCR results, you can specify what languages your document is in using the `languages` parameter.
[View the list of available languages](https://github.com/tesseract-ocr/tessdata).

<AccordionGroup>
  <Accordion title="Ingest CLI">
    ```bash CLI
    unstructured-ingest \
      local \
        --input-path $LOCAL_FILE_INPUT_DIR \
        --output-dir $LOCAL_FILE_OUTPUT_DIR \
        --strategy ocr_only \
        --ocr-languages kor \
        --partition-by-api \
        --api-key $UNSTRUCTURED_API_KEY \
        --partition-endpoint $UNSTRUCTURED_API_URL \
        --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
    ```
  </Accordion>

  <Accordion title="Ingest Python">
    ```python Python
    import os

    from unstructured_ingest.v2.pipeline.pipeline import Pipeline
    from unstructured_ingest.v2.interfaces import ProcessorConfig
    from unstructured_ingest.v2.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.v2.processes.partitioner import PartitionerConfig

    if __name__ == "__main__":
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                strategy="ocr_only",
                ocr_languages=["kor"],
                partition_by_api=True,
                api_key=os.getenv("UNSTRUCTURED_API_KEY"),
                partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
                additional_partition_args={
                    "split_pdf_page": True,
                    "split_pdf_allow_failed": True,
                    "split_pdf_concurrency_level": 15
                }
            ),
            uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
        ).run()
    ```
  </Accordion>

  <Accordion title="POST">
    <Info>
      Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
      [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
      to you:

      *   You need to work with documents in cloud storage.
      *   You want to cache the results of processing multiple files in batches.
      *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
    </Info>

    ```bash POST
    curl -X 'POST' $UNSTRUCTURED_API_URL \
    -H 'accept: application/json' \
    -H 'Content-Type: multipart/form-data' \
    -H 'unstructured-api-key: $UNSTRUCTURED_API_KEY' \
    -F 'files=@sample-docs/korean.png' \
    -F 'strategy=ocr_only' \
    -F 'languages=kor'
    ```
  </Accordion>

  <Accordion title="Python SDK">
    <Info>
      Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
      [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
      to you:

      *   You need to work with documents in cloud storage.
      *   You want to cache the results of processing multiple files in batches.
      *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
    </Info>

    ```python Python
    import asyncio
    import os
    import json
    import unstructured_client
    from unstructured_client.models import shared

    client = unstructured_client.UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
        server_url=os.getenv("UNSTRUCTURED_API_URL"),
    )

    async def call_api(filename, input_dir, output_dir):
        req = {
            "partition_parameters": {
                "files": {
                    "content": open(filename, "rb"),
                    "file_name": os.path.basename(filename),
                },
                "strategy": shared.Strategy.OCR_ONLY,
                "languages": ["kor"],
                "split_pdf_page": True,
                "split_pdf_allow_failed": True,
                "split_pdf_concurrency_level": 15
            }
        }

        try:
            res = await client.general.partition_async(request=req)
            element_dicts = [element for element in res.elements]
            json_elements = json.dumps(element_dicts, indent=2)

            # Create the output directory structure.
            relative_path = os.path.relpath(os.path.dirname(filename), input_dir)
            output_subdir = os.path.join(output_dir, relative_path)
            os.makedirs(output_subdir, exist_ok=True)

            # Write the output file.
            output_filename = os.path.join(output_subdir, os.path.basename(filename) + ".json")
            with open(output_filename, "w") as file:
                file.write(json_elements)

        except Exception as e:
            print(f"Error processing {filename}: {e}")

    async def process_files(input_directory, output_directory):
        tasks = []

        for root, _, files in os.walk(input_directory):
            for file in files:
                if not file.endswith('.json'):
                    full_path = os.path.join(root, file)
                    tasks.append(call_api(full_path, input_directory, output_directory))

        await asyncio.gather(*tasks)

    if __name__ == "__main__":
        asyncio.run(process_files(
            input_directory=os.getenv("LOCAL_FILE_INPUT_DIR"), 
            output_directory=os.getenv("LOCAL_FILE_OUTPUT_DIR")
        ))
    ```
  </Accordion>

  <Accordion title="JavaScript/TypeScript SDK">
    <Info>
      Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
      [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
      to you:

      *   You need to work with documents in cloud storage.
      *   You want to cache the results of processing multiple files in batches.
      *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
    </Info>

    ```typescript TypeScript
    import { UnstructuredClient } from "unstructured-client";
    import * as fs from "fs";
    import * as path from "path";
    import { Strategy } from "unstructured-client/sdk/models/shared/index.js";
    import { PartitionResponse } from "unstructured-client/sdk/models/operations";

    // Send all files in the source path to Unstructured for processing.
    // Send the processed data to the destination path.
    function processFiles(
        client: UnstructuredClient,
        sourcePath: string,
        destinationPath: string
    ): void {

        // If an output directory does not exist for the corresponding input
        // directory, then create it.
        if (!fs.existsSync(destinationPath)) {
            fs.mkdirSync(destinationPath, { recursive: true });
        }

        // Get all folders and files at the current level of the input directory.
        const items = fs.readdirSync(sourcePath);

        // For each folder and file in the input directory...
        for (const item of items) {
            const inputPath = path.join(sourcePath, item);
            const outputPath = path.join(destinationPath, item)

            // If it's a folder, call this function recursively.
            if (fs.statSync(inputPath).isDirectory()) {
                processFiles(client, inputPath, outputPath);
            } else {
                // If it's a file, send it to Unstructured for processing.
                const data = fs.readFileSync(inputPath);

                client.general.partition({
                    partitionParameters: {
                        files: {
                            content: data,
                            fileName: inputPath
                        },
                        strategy: Strategy.OcrOnly,
                        languages: ["kor"],
                        splitPdfPage: true,
                        splitPdfConcurrencyLevel: 15,
                        splitPdfAllowFailed: true
                    }
                }).then((res: PartitionResponse) => {
                    // If successfully processed, write the processed data to
                    // the destination directory.
                    if (res.statusCode == 200) {
                        const jsonElements = JSON.stringify(res.elements, null, 2)
                        fs.writeFileSync(outputPath + ".json", jsonElements)
                    }
                }).catch((e) => {
                    if (e.statusCode) {
                        console.log(e.statusCode);
                        console.log(e.body);
                    } else {
                        console.log(e);
                    }
                });
            }
        }
    }

    const client = new UnstructuredClient({
        security: { apiKeyAuth: process.env.UNSTRUCTURED_API_KEY },
        serverURL: process.env.UNSTRUCTURED_API_URL
    });

    processFiles(
        client,
        process.env.LOCAL_FILE_INPUT_DIR,
        process.env.LOCAL_FILE_OUTPUT_DIR
    );
    ```
  </Accordion>
</AccordionGroup>

### Saving bounding box coordinates

When elements are extracted from PDFs or images, it may be useful to get their bounding boxes as well.
Set the `coordinates` parameter to `true` to add this field to the elements in the response.

<AccordionGroup>
  <Accordion title="Ingest CLI">
    ```bash CLI
    unstructured-ingest \
      local \
        --input-path $LOCAL_FILE_INPUT_DIR \
        --output-dir $LOCAL_FILE_OUTPUT_DIR \
        --partition-by-api \
        --api-key $UNSTRUCTURED_API_KEY \
        --partition-endpoint $UNSTRUCTURED_API_URL \
        --strategy hi_res \
        --additional-partition-args="{\"coordinates\":\"true\", \"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
    ```
  </Accordion>

  <Accordion title="Ingest Python">
    ```python Python
    import os

    from unstructured_ingest.v2.pipeline.pipeline import Pipeline
    from unstructured_ingest.v2.interfaces import ProcessorConfig
    from unstructured_ingest.v2.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.v2.processes.partitioner import PartitionerConfig

    if __name__ == "__main__":
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                partition_by_api=True,
                api_key=os.getenv("UNSTRUCTURED_API_KEY"),
                partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
                strategy="hi_res",
                additional_partition_args={
                    "coordinates": True,
                    "split_pdf_page": True,
                    "split_pdf_allow_failed": True,
                    "split_pdf_concurrency_level": 15
                }
            ),
            uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
        ).run()
    ```
  </Accordion>

  <Accordion title="POST">
    <Info>
      Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
      [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
      to you:

      *   You need to work with documents in cloud storage.
      *   You want to cache the results of processing multiple files in batches.
      *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
    </Info>

    ```bash POST
    curl -X 'POST' $UNSTRUCTURED_API_URL \
    -H 'accept: application/json' \
    -H 'Content-Type: multipart/form-data' \
    -H 'unstructured-api-key: $UNSTRUCTURED_API_KEY' \
    -F 'files=@sample-docs/layout-parser-paper.pdf' \
    -F 'coordinates=true' \
    -F 'strategy=hi_res'
    ```
  </Accordion>

  <Accordion title="Python SDK">
    <Info>
      Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
      [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
      to you:

      *   You need to work with documents in cloud storage.
      *   You want to cache the results of processing multiple files in batches.
      *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
    </Info>

    ```python Python
    import asyncio
    import os
    import json
    import unstructured_client
    from unstructured_client.models import shared

    client = unstructured_client.UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
        server_url=os.getenv("UNSTRUCTURED_API_URL"),
    )

    async def call_api(filename, input_dir, output_dir):
        req = {
            "partition_parameters": {
                "files": {
                    "content": open(filename, "rb"),
                    "file_name": os.path.basename(filename),
                },
                "strategy": shared.Strategy.HI_RES,
                "coordinates": True,
                "split_pdf_page": True,
                "split_pdf_allow_failed": True,
                "split_pdf_concurrency_level": 15
            }
        }

        try:
            res = await client.general.partition_async(request=req)
            element_dicts = [element for element in res.elements]
            json_elements = json.dumps(element_dicts, indent=2)

            # Create the output directory structure.
            relative_path = os.path.relpath(os.path.dirname(filename), input_dir)
            output_subdir = os.path.join(output_dir, relative_path)
            os.makedirs(output_subdir, exist_ok=True)

            # Write the output file.
            output_filename = os.path.join(output_subdir, os.path.basename(filename) + ".json")
            with open(output_filename, "w") as file:
                file.write(json_elements)

        except Exception as e:
            print(f"Error processing {filename}: {e}")

    async def process_files(input_directory, output_directory):
        tasks = []

        for root, _, files in os.walk(input_directory):
            for file in files:
                if not file.endswith('.json'):
                    full_path = os.path.join(root, file)
                    tasks.append(call_api(full_path, input_directory, output_directory))

        await asyncio.gather(*tasks)

    if __name__ == "__main__":
        asyncio.run(process_files(
            input_directory=os.getenv("LOCAL_FILE_INPUT_DIR"), 
            output_directory=os.getenv("LOCAL_FILE_OUTPUT_DIR")
        ))
    ```
  </Accordion>

  <Accordion title="JavaScript/TypeScript SDK">
    <Info>
      Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
      [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
      to you:

      *   You need to work with documents in cloud storage.
      *   You want to cache the results of processing multiple files in batches.
      *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
    </Info>

    ```typescript TypeScript
    import { UnstructuredClient } from "unstructured-client";
    import * as fs from "fs";
    import * as path from "path";
    import { Strategy } from "unstructured-client/sdk/models/shared/index.js";
    import { PartitionResponse } from "unstructured-client/sdk/models/operations";

    // Send all files in the source path to Unstructured for processing.
    // Send the processed data to the destination path.
    function processFiles(
        client: UnstructuredClient,
        sourcePath: string,
        destinationPath: string
    ): void {

        // If an output directory does not exist for the corresponding input
        // directory, then create it.
        if (!fs.existsSync(destinationPath)) {
            fs.mkdirSync(destinationPath, { recursive: true });
        }

        // Get all folders and files at the current level of the input directory.
        const items = fs.readdirSync(sourcePath);

        // For each folder and file in the input directory...
        for (const item of items) {
            const inputPath = path.join(sourcePath, item);
            const outputPath = path.join(destinationPath, item)

            // If it's a folder, call this function recursively.
            if (fs.statSync(inputPath).isDirectory()) {
                processFiles(client, inputPath, outputPath);
            } else {
                // If it's a file, send it to Unstructured for processing.
                const data = fs.readFileSync(inputPath);

                client.general.partition({
                    partitionParameters: {
                        files: {
                            content: data,
                            fileName: inputPath
                        },
                        strategy: Strategy.HiRes,
                        coordinates: true,
                        splitPdfPage: true,
                        splitPdfConcurrencyLevel: 15,
                        splitPdfAllowFailed: true
                    }
                }).then((res: PartitionResponse) => {
                    // If successfully processed, write the processed data to
                    // the destination directory.
                    if (res.statusCode == 200) {
                        const jsonElements = JSON.stringify(res.elements, null, 2)
                        fs.writeFileSync(outputPath + ".json", jsonElements)
                    }
                }).catch((e) => {
                    if (e.statusCode) {
                        console.log(e.statusCode);
                        console.log(e.body);
                    } else {
                        console.log(e);
                    }
                });
            }
        }
    }

    const client = new UnstructuredClient({
        security: { apiKeyAuth: process.env.UNSTRUCTURED_API_KEY },
        serverURL: process.env.UNSTRUCTURED_API_URL
    });

    processFiles(
        client,
        process.env.LOCAL_FILE_INPUT_DIR,
        process.env.LOCAL_FILE_OUTPUT_DIR
    );
    ```
  </Accordion>
</AccordionGroup>

### Returning unique element IDs

By default, the element ID is a SHA-256 hash of the element text. This is to ensure that
the ID is deterministic. One downside is that the ID is not guaranteed to be unique.
Different elements with the same text will have the same ID, and there could also be hash collisions.
To use UUIDs in the output instead, set `unique_element_ids=true`. Note: this means that the element IDs
will be random, so with every partition of the same file, you will get different IDs.
This can be helpful if you'd like to use the IDs as a primary key in a database, for example.

<AccordionGroup>
  <Accordion title="Ingest CLI">
    ```bash CLI
    unstructured-ingest \
      local \
        --input-path $LOCAL_FILE_INPUT_DIR \
        --output-dir $LOCAL_FILE_OUTPUT_DIR \
        --partition-by-api \
        --api-key $UNSTRUCTURED_API_KEY \
        --partition-endpoint $UNSTRUCTURED_API_URL \
        --strategy hi_res \
        --additional-partition-args="{\"unique_element_ids\":\"true\", \"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
    ```
  </Accordion>

  <Accordion title="Ingest Python">
    ```python Python
    import os

    from unstructured_ingest.v2.pipeline.pipeline import Pipeline
    from unstructured_ingest.v2.interfaces import ProcessorConfig
    from unstructured_ingest.v2.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.v2.processes.partitioner import PartitionerConfig

    if __name__ == "__main__":
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                partition_by_api=True,
                api_key=os.getenv("UNSTRUCTURED_API_KEY"),
                partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
                strategy="hi_res",
                additional_partition_args={
                    "unique_element_ids": True,
                    "split_pdf_page": True,
                    "split_pdf_allow_failed": True,
                    "split_pdf_concurrency_level": 15
                }
            ),
            uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
        ).run()
    ```
  </Accordion>

  <Accordion title="POST">
    <Info>
      Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
      [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
      to you:

      *   You need to work with documents in cloud storage.
      *   You want to cache the results of processing multiple files in batches.
      *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
    </Info>

    ```bash POST
    curl -X 'POST' $UNSTRUCTURED_API_URL \
    -H 'accept: application/json'  \
    -H 'Content-Type: multipart/form-data' \
    -H 'unstructured-api-key: $UNSTRUCTURED_API_KEY' \
    -F 'files=@sample-docs/layout-parser-paper-fast.pdf' \
    -F 'unique_element_ids=true'
    ```
  </Accordion>

  <Accordion title="Python SDK">
    <Info>
      Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
      [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
      to you:

      *   You need to work with documents in cloud storage.
      *   You want to cache the results of processing multiple files in batches.
      *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
    </Info>

    ```python Python
    import asyncio
    import os
    import json
    import unstructured_client
    from unstructured_client.models import shared

    client = unstructured_client.UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
        server_url=os.getenv("UNSTRUCTURED_API_URL"),
    )

    async def call_api(filename, input_dir, output_dir):
        req = {
            "partition_parameters": {
                "files": {
                    "content": open(filename, "rb"),
                    "file_name": os.path.basename(filename),
                },
                "strategy": shared.Strategy.HI_RES,
                "unique_element_ids": True,
                "split_pdf_page": True,
                "split_pdf_allow_failed": True,
                "split_pdf_concurrency_level": 15
            }
        }

        try:
            res = await client.general.partition_async(request=req)
            element_dicts = [element for element in res.elements]
            json_elements = json.dumps(element_dicts, indent=2)

            # Create the output directory structure.
            relative_path = os.path.relpath(os.path.dirname(filename), input_dir)
            output_subdir = os.path.join(output_dir, relative_path)
            os.makedirs(output_subdir, exist_ok=True)

            # Write the output file.
            output_filename = os.path.join(output_subdir, os.path.basename(filename) + ".json")
            with open(output_filename, "w") as file:
                file.write(json_elements)

        except Exception as e:
            print(f"Error processing {filename}: {e}")

    async def process_files(input_directory, output_directory):
        tasks = []

        for root, _, files in os.walk(input_directory):
            for file in files:
                if not file.endswith('.json'):
                    full_path = os.path.join(root, file)
                    tasks.append(call_api(full_path, input_directory, output_directory))

        await asyncio.gather(*tasks)

    if __name__ == "__main__":
        asyncio.run(process_files(
            input_directory=os.getenv("LOCAL_FILE_INPUT_DIR"), 
            output_directory=os.getenv("LOCAL_FILE_OUTPUT_DIR")
        ))
    ```
  </Accordion>

  <Accordion title="JavaScript/TypeScript SDK">
    <Info>
      Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
      [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
      to you:

      *   You need to work with documents in cloud storage.
      *   You want to cache the results of processing multiple files in batches.
      *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
    </Info>

    ```typescript TypeScript
    import { UnstructuredClient } from "unstructured-client";
    import * as fs from "fs";
    import * as path from "path";
    import { Strategy } from "unstructured-client/sdk/models/shared/index.js";
    import { PartitionResponse } from "unstructured-client/sdk/models/operations";

    // Send all files in the source path to Unstructured for processing.
    // Send the processed data to the destination path.
    function processFiles(
        client: UnstructuredClient,
        sourcePath: string,
        destinationPath: string
    ): void {

        // If an output directory does not exist for the corresponding input
        // directory, then create it.
        if (!fs.existsSync(destinationPath)) {
            fs.mkdirSync(destinationPath, { recursive: true });
        }

        // Get all folders and files at the current level of the input directory.
        const items = fs.readdirSync(sourcePath);

        // For each folder and file in the input directory...
        for (const item of items) {
            const inputPath = path.join(sourcePath, item);
            const outputPath = path.join(destinationPath, item)

            // If it's a folder, call this function recursively.
            if (fs.statSync(inputPath).isDirectory()) {
                processFiles(client, inputPath, outputPath);
            } else {
                // If it's a file, send it to Unstructured for processing.
                const data = fs.readFileSync(inputPath);

                client.general.partition({
                    partitionParameters: {
                        files: {
                            content: data,
                            fileName: inputPath
                        },
                        uniqueElementIds: true,
                        strategy: Strategy.HiRes,
                        splitPdfPage: true,
                        splitPdfConcurrencyLevel: 15,
                        splitPdfAllowFailed: true
                    }
                }).then((res: PartitionResponse) => {
                    // If successfully processed, write the processed data to
                    // the destination directory.
                    if (res.statusCode == 200) {
                        const jsonElements = JSON.stringify(res.elements, null, 2)
                        fs.writeFileSync(outputPath + ".json", jsonElements)
                    }
                }).catch((e) => {
                    if (e.statusCode) {
                        console.log(e.statusCode);
                        console.log(e.body);
                    } else {
                        console.log(e);
                    }
                });
            }
        }
    }

    const client = new UnstructuredClient({
        security: { apiKeyAuth: process.env.UNSTRUCTURED_API_KEY },
        serverURL: process.env.UNSTRUCTURED_API_URL
    });

    processFiles(
        client,
        process.env.LOCAL_FILE_INPUT_DIR,
        process.env.LOCAL_FILE_OUTPUT_DIR
    );
    ```
  </Accordion>
</AccordionGroup>

### Adding the chunking step after partitioning

You can combine partitioning and subsequent chunking in a single request by setting the `chunking_strategy` parameter.
By default, the `chunking_strategy` is set to `None`, and no chunking is performed.

[//]: # "TODO: add a link to the concepts section about chunking strategies. Need to create the shared Concepts section first"

<AccordionGroup>
  <Accordion title="Ingest CLI">
    ```bash CLI
    unstructured-ingest \
      local \
        --input-path $LOCAL_FILE_INPUT_DIR \
        --output-dir $LOCAL_FILE_OUTPUT_DIR \
        --chunking-strategy by_title \
        --chunk-max-characters 1024 \
        --partition-by-api \
        --api-key $UNSTRUCTURED_API_KEY \
        --partition-endpoint $UNSTRUCTURED_API_URL \
        --strategy hi_res \
        --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
    ```
  </Accordion>

  <Accordion title="Ingest Python">
    ```python Python
    import os

    from unstructured_ingest.v2.pipeline.pipeline import Pipeline
    from unstructured_ingest.v2.interfaces import ProcessorConfig
    from unstructured_ingest.v2.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
    from unstructured_ingest.v2.processes.chunker import ChunkerConfig

    if __name__ == "__main__":
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                partition_by_api=True,
                api_key=os.getenv("UNSTRUCTURED_API_KEY"),
                partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
                strategy="hi_res",
                additional_partition_args={
                    "split_pdf_page": True,
                    "split_pdf_allow_failed": True,
                    "split_pdf_concurrency_level": 15
                }
            ),
            chunker_config=ChunkerConfig(
                chunking_strategy="by_title",
                chunk_max_characters=1024
            ),
            uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
        ).run()
    ```
  </Accordion>

  <Accordion title="POST">
    <Info>
      Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
      [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
      to you:

      *   You need to work with documents in cloud storage.
      *   You want to cache the results of processing multiple files in batches.
      *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
    </Info>

    ```bash POST
    curl -X 'POST' $UNSTRUCTURED_API_URL \
    -H 'accept: application/json'  \
    -H 'Content-Type: multipart/form-data' \
    -H 'unstructured-api-key: $UNSTRUCTURED_API_KEY' \
    -F 'files=@sample-docs/layout-parser-paper-fast.pdf' \
    -F 'chunking_strategy=by_title' \
    -F 'max_characters=1024' \
    -F 'strategy=hi_res'
    ```
  </Accordion>

  <Accordion title="Python SDK">
    <Info>
      Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
      [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
      to you:

      *   You need to work with documents in cloud storage.
      *   You want to cache the results of processing multiple files in batches.
      *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
    </Info>

    ```python Python
    import asyncio
    import os
    import json
    import unstructured_client
    from unstructured_client.models import shared

    client = unstructured_client.UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
        server_url=os.getenv("UNSTRUCTURED_API_URL"),
    )

    async def call_api(filename, input_dir, output_dir):
        req = {
            "partition_parameters": {
                "files": {
                    "content": open(filename, "rb"),
                    "file_name": os.path.basename(filename),
                },
                "chunking_strategy": "by_title",
                "max_characters": 1024,
                "strategy": shared.Strategy.HI_RES,
                "split_pdf_page": True,
                "split_pdf_allow_failed": True,
                "split_pdf_concurrency_level": 15
            }
        }

        try:
            res = await client.general.partition_async(request=req)
            element_dicts = [element for element in res.elements]
            json_elements = json.dumps(element_dicts, indent=2)

            # Create the output directory structure.
            relative_path = os.path.relpath(os.path.dirname(filename), input_dir)
            output_subdir = os.path.join(output_dir, relative_path)
            os.makedirs(output_subdir, exist_ok=True)

            # Write the output file.
            output_filename = os.path.join(output_subdir, os.path.basename(filename) + ".json")
            with open(output_filename, "w") as file:
                file.write(json_elements)

        except Exception as e:
            print(f"Error processing {filename}: {e}")

    async def process_files(input_directory, output_directory):
        tasks = []

        for root, _, files in os.walk(input_directory):
            for file in files:
                if not file.endswith('.json'):
                    full_path = os.path.join(root, file)
                    tasks.append(call_api(full_path, input_directory, output_directory))

        await asyncio.gather(*tasks)

    if __name__ == "__main__":
        asyncio.run(process_files(
            input_directory=os.getenv("LOCAL_FILE_INPUT_DIR"), 
            output_directory=os.getenv("LOCAL_FILE_OUTPUT_DIR")
        ))
    ```
  </Accordion>

  <Accordion title="JavaScript/TypeScript SDK">
    <Info>
      Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
      [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
      to you:

      *   You need to work with documents in cloud storage.
      *   You want to cache the results of processing multiple files in batches.
      *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
    </Info>

    ```typescript TypeScript
    import { UnstructuredClient } from "unstructured-client";
    import * as fs from "fs";
    import * as path from "path";
    import { ChunkingStrategy, Strategy } from "unstructured-client/sdk/models/shared/index.js";
    import { PartitionResponse } from "unstructured-client/sdk/models/operations";

    // Send all files in the source path to Unstructured for processing.
    // Send the processed data to the destination path.
    function processFiles(
        client: UnstructuredClient,
        sourcePath: string,
        destinationPath: string
    ): void {

        // If an output directory does not exist for the corresponding input
        // directory, then create it.
        if (!fs.existsSync(destinationPath)) {
            fs.mkdirSync(destinationPath, { recursive: true });
        }

        // Get all folders and files at the current level of the input directory.
        const items = fs.readdirSync(sourcePath);

        // For each folder and file in the input directory...
        for (const item of items) {
            const inputPath = path.join(sourcePath, item);
            const outputPath = path.join(destinationPath, item)

            // If it's a folder, call this function recursively.
            if (fs.statSync(inputPath).isDirectory()) {
                processFiles(client, inputPath, outputPath);
            } else {
                // If it's a file, send it to Unstructured for processing.
                const data = fs.readFileSync(inputPath);

                client.general.partition({
                    partitionParameters: {
                        files: {
                            content: data,
                            fileName: inputPath
                        },
                        strategy: Strategy.HiRes,
                        chunkingStrategy: ChunkingStrategy.ByTitle,
                        maxCharacters: 1024,
                        splitPdfPage: true,
                        splitPdfConcurrencyLevel: 15,
                        splitPdfAllowFailed: true
                    }
                }).then((res: PartitionResponse) => {
                    // If successfully processed, write the processed data to
                    // the destination directory.
                    if (res.statusCode == 200) {
                        const jsonElements = JSON.stringify(res.elements, null, 2)
                        fs.writeFileSync(outputPath + ".json", jsonElements)
                    }
                }).catch((e) => {
                    if (e.statusCode) {
                        console.log(e.statusCode);
                        console.log(e.body);
                    } else {
                        console.log(e);
                    }
                });
            }
        }
    }

    const client = new UnstructuredClient({
        security: { apiKeyAuth: process.env.UNSTRUCTURED_API_KEY },
        serverURL: process.env.UNSTRUCTURED_API_URL
    });

    processFiles(
        client,
        process.env.LOCAL_FILE_INPUT_DIR,
        process.env.LOCAL_FILE_OUTPUT_DIR
    );
    ```
  </Accordion>
</AccordionGroup>


# Free Unstructured API

This page describes how to obtain an API key to use with the free Unstructured API, the limitations of the free Unstructured API, and provides a quickstart example.

<Info>The Free Unstructured API is different than the free 14-day trial for the [Unstructured Serverless API](/api-reference/api-services/saas-api-development-guide).</Info>

## Get an API key

The Free Unstructured API requires authentication via an API key. Here's how you can obtain your API key:

1.  Go to [https://unstructured.io/api-key-free](https://unstructured.io/api-key-free).
2.  Fill out the registration form. Make sure your contact information (especially your **Email**) is valid.
3.  Check the **I agree** box if you consent to Unstructured contacting you about our products and services.
4.  Click the **Terms and Conditions** link, read it, and check the related box to agree.
5.  Click **Submit**. You will receive a Free Unstructured API key at the **Email** you provided. Store your API key in a secure location. Do not share it with others.
6.  For the Free Unstructured API, the API URL is `https://api.unstructured.io/general/v0/general`

<Warning>
  Free Unstructured API keys do not work with the [Unstructured Serverless API](/api-reference/api-services/saas-api-development-guide). If you try to use a Free Unstructured API key with an Unstructured Serverless API URL, the call will fail. Use your Free Unstructured API URL instead.
</Warning>

[Try the quickstart](#quickstart).

## Free Unstructured API limitations

The Free Unstructured API is designed for prototyping purposes, and not for production use:

*   The API usage is limited to 1000 pages per month.
*   Unlike the users of Unstructured Serverless API, users of the Free Unstructured API do not get their own dedicated infrastructure.
*   The data sent over the Free Unstructured API can be used for model training purposes, and other service improvements.

If you require a production-ready API, consider using the [Unstructured Serverless API](/api-reference/api-services/saas-api-development-guide) instead.

We calculate a page as follows:

*   For these file types, a page is a page, slide, or image: .pdf, .pptx, and .tiff.
*   For .docx files that have page metadata, we calculate the number of pages based on that metadata.
*   For all other file types, we calculate the number of pages as the file's size divided by 100 KB.

## Quickstart

These examples use your local machine. They send source (input) files from your local machine to the Unstructured Serverless API which delivers the processed data to a destination (output) location, also on your local machine. Data is processed on Unstructured-hosted compute resources.

### Unstructured Ingest CLI

To work with the Free Unstructured API by using the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli), you will need to:

*   Install Python, and then install the CLI package:

    ```bash
    pip install unstructured
    ```

*   Set the `UNSTRUCTURED_API_KEY` environment variable to your Free Unstructured API key.

*   Set the `UNSTRUCTURED_API_URL` environment variable to your Free Unstructured API URL, which is `https://api.unstructured.io/general/v0/general`

*   Have some compatible files on your local machine to be processed. [See the list of supported file types](/api-reference/api-services/supported-file-types). If you do not have any files available, you can download some from the [example-docs](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/example-docs) folder in the Unstructured repo on GitHub.

Now, use the CLI to call the API, replacing:

*   `<path/to/input>` with the source (input) path to the directory on your local machine that contains the compatible files for Unstructured to process on its hosted compute resources.
*   `<path/to/output>` with the destination (output) path to the directory on your local machine that will contain the processed data that Unstructured returns from its hosted compute resources.

```bash CLI
unstructured-ingest \
  local \
    --input-path <path/to/input> \
    --output-dir <path/to/output> \
    --partition-by-api \
    --api-key $UNSTRUCTURED_API_KEY \
    --partition-endpoint $UNSTRUCTURED_API_URL \
    --strategy hi_res \
    --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
```

After the command successfully runs, see the results in the specified output path on your local machine.

### Unstructured Ingest Python library

To work with the Unstructured Serverless API by using the [Unstructured Python library](/ingestion/overview#unstructured-ingest-python-library), you will need to:

*   Install Python, and then install the CLI package:

    ```bash
    pip install unstructured-ingest
    ```

*   Set the following environment variables:

    *   Set `UNSTRUCTURED_API_KEY` to your API key.
    *   Set `UNSTRUCTURED_API_URL` to your API URL.

    [Get your API key and API URL](#get-started).

*   Have some compatible files on your local machine to be processed. [See the list of supported file types](/api-reference/api-services/supported-file-types). If you do not have any files available, you can download some from the [example-docs](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/example-docs) folder in the Unstructured repo on GitHub.

Now, use the CLI to call the API, replacing:

*   `<path/to/input>` with the source (input) path to the directory on your local machine that contains the compatible files for Unstructured to process on its hosted compute resources.
*   `<path/to/output>` with the destination (output) path to the directory on your local machine that will contain the processed data that Unstructured returns from its hosted compute resources.

```python Python Ingest v2
import os

from unstructured_ingest.v2.pipeline.pipeline import Pipeline
from unstructured_ingest.v2.interfaces import ProcessorConfig
from unstructured_ingest.v2.processes.connectors.local import (
    LocalIndexerConfig,
    LocalDownloaderConfig,
    LocalConnectionConfig,
    LocalUploaderConfig
)
from unstructured_ingest.v2.processes.partitioner import PartitionerConfig

if __name__ == "__main__":
    Pipeline.from_configs(
        context=ProcessorConfig(),
        indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
        downloader_config=LocalDownloaderConfig(),
        source_connection_config=LocalConnectionConfig(),
        partitioner_config=PartitionerConfig(
            partition_by_api=True,
            api_key=os.getenv("UNSTRUCTURED_API_KEY"),
            partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
            strategy="hi_res",
            additional_partition_args={
                "split_pdf_page": True,
                "split_pdf_allow_failed": True,
                "split_pdf_concurrency_level": 15
            }
        ),
        uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
    ).run()
```

After the command successfully runs, see the results in the specified output path on your local machine.

## Telemetry

We’ve partnered with [Scarf](https://scarf.sh) to collect anonymized user statistics to understand which features our community is using and how to prioritize product decision-making in the future.

To learn more about how we collect and use this data, please read our [Privacy Policy](https://unstructured.io/privacy-policy).

To opt out of this data collection, you can set the environment variable `SCARF_NO_ANALYTICS=true` before running any commands that call Unstructured Serverless API services.


# Unstructured Serverless API services



Unstructured Serverless API services include these offerings:

<CardGroup cols={2}>
  <Card title="Unstructured Serverless API" icon="computer" href="/api-reference/api-services/saas-api-development-guide">
    Serverless API hosted by Unstructured. Scalable and secure. Your data remains private. Start with a 14-day free trial, then pay as you go.<br /><br />
    [Try the quickstart](/api-reference/api-services/saas-api-development-guide#quickstart).<br /><br />
    [Learn more](/api-reference/api-services/saas-api-development-guide).
  </Card>

  <Card title="Free Unstructured API" icon="gift" href="/api-reference/api-services/free-api">
    Hosted by Unstructured. Free to use, but data processing is limited to 1000 pages per month. Documents submitted to the Free Unstructured API may be utilized for Unstructured's proprietary model training and evaluation purposes.<br /><br />
    [Try the quickstart](/api-reference/api-services/free-api#quickstart).<br /><br />
    [Learn more](/api-reference/api-services/free-api).
  </Card>

  <Card title="Unstructured API on Azure" icon="windows" href="/api-reference/api-services/azure">
    The functionality and security of the Unstructured Serverless API, but deployed through the Azure Marketplace, in your preferred infrastructure. Pay as you go.<br /><br />
    [Learn more](/api-reference/api-services/azure).
  </Card>

  <Card title="Unstructured API on AWS" icon="aws" href="/api-reference/api-services/aws">
    The functionality and security of the Unstructured Serverless API, but deployed through the AWS Marketplace, in your preferred infrastructure. Pay as you go.<br /><br />
    [Learn more](/api-reference/api-services/aws).
  </Card>
</CardGroup>

<Icon icon="blog" />  [Read the launch announcement](https://unstructured.io/blog/introducing-unstructured-serverless-api).

## Benefits over open source

Unstructured Serverless API services provide the following benefits beyond the [Unstructured open source library](/open-source/introduction/overview) offering:

*   Designed for production scenarios.
*   Significantly increased performance on document and table extraction.
*   Access to newer and more sophisticated vision transformer models.
*   Access to Unstructured's fine-tuned OCR models.
*   Access to Unstructured's by-page and by-similarity chunking strategies.
*   Adherence to security and SOC2 Type 1, SOC2 Type 2, and HIPAA compliance standards.
*   Authentication and identity management.
*   Incremental data loading.
*   Image extraction from documents.
*   More sophisticated document hierarchy detection.
*   Unstructured manages code dependencies, for instance for libraries such as Tesseract.
*   Unstructured manages its own infrastructure, including parallelization and other performance optimizations.

## Supported file types

Unstructured supports processing of the following file types:

By file extension:

| File extension |
| -------------- |
| `.bmp`         |
| `.csv`         |
| `.doc`         |
| `.docx`        |
| `.eml`         |
| `.epub`        |
| `.heic`        |
| `.html`        |
| `.jpeg`        |
| `.png`         |
| `.md`          |
| `.msg`         |
| `.odt`         |
| `.org`         |
| `.p7s`         |
| `.pdf`         |
| `.png`         |
| `.ppt`         |
| `.pptx`        |
| `.rst`         |
| `.rtf`         |
| `.tiff`        |
| `.txt`         |
| `.tsv`         |
| `.xls`         |
| `.xlsx`        |
| `.xml`         |

By file type:

| Category          | File types                                |
| ----------------- | ----------------------------------------- |
| CSV               | `.csv`                                    |
| E-mail            | `.eml`, `.msg`, `.p7s`                    |
| EPUB              | `.epub`                                   |
| Excel             | `.xls`, `.xlsx`                           |
| HTML              | `.html`                                   |
| Image             | `.bmp`, `.heic`, `.jpeg`, `.png`, `.tiff` |
| Markdown          | `.md`                                     |
| Org Mode          | `.org`                                    |
| Open Office       | `.odt`                                    |
| PDF               | `.pdf`                                    |
| Plain text        | `.txt`                                    |
| PowerPoint        | `.ppt`, `.pptx`                           |
| reStructured Text | `.rst`                                    |
| Rich Text         | `.rtf`                                    |
| TSV               | `.tsv`                                    |
| Word              | `.doc`, `.docx`                           |
| XML               | `.xml`                                    |

## Data ingestion

Unstructured Serverless API services support ingesting data from various sources. [Learn how](/ingestion/overview).

## Billing

We calculate a page as follows:

*   For these file types, a page is a page, slide, or image: .pdf, .pptx, and .tiff.
*   For .docx files that have page metadata, we calculate the number of pages based on that metadata.
*   For all other file types, we calculate the number of pages as the file's size divided by 100 KB.

## Get support

Should you require any assistance or have any questions regarding the Unstructured API, please contact our support team at [support@unstructured.io](mailto:support%40unstructured.io).


# Process an individual file by using the Unstructured open source Python library



If you're used to working the [Unstructured open source library](https://github.com/Unstructured-IO/unstructured) but
would like to leverage the advanced capabilities of Unstructured API services, you can do so without leaving your familiarity with the open source library.
Whether you're using the Free Unstructured API, the Unstructured Serverless API,
the Unstructured API on Azure/AWS, or your local deployment of the Unstructured API, you can use the open source library to send an individual file through `partition_via_api` for processing with Unstructured API services.

<Info>
  Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
  [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
  to you:

  *   You need to work with documents in cloud storage.
  *   You want to cache the results of processing multiple files in batches.
  *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
</Info>

To use the open source library, you'll also need:

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

<Info>
  If you do not specify the API URL, your [Unstructured Serverless API](/api-reference/api-services/saas-api-development-guide) pay-as-you-go account will be used by default. You must always specify your Serverless API key.<br /><br />
  To use the [Free Unstructured API](/api-reference/api-services/free-api), you must always specify your Free API key, and the Free API URL which is `https://api.unstructured.io/general/v0/general`<br /><br />
  To use the pay-as-you-go Unstructured API on Azure or AWS with the SDKs, you must always specify the corresponding API URL. See the [Azure](/api-reference/api-services/azure) or [AWS](/api-reference/api-services/aws) instructions.
</Info>

## Installation

Make sure you have the Unstructured open source library installed.
Refer to the [Unstructured open source library quickstart guide](/open-source/introduction/quick-start) for instructions.

## Basics

Here's a basic example in which you send an email file to partition via Unstructured API using `partition_via_api` available
in the Unstructured open source library:

```python
import os, json

from unstructured.partition.api import partition_via_api

elements = partition_via_api(
    filename="example-docs/pdf/DA-1p.pdf",
    api_key=os.getenv("UNSTRUCTURED_API_KEY"),
    api_url=os.getenv("UNSTRUCTURED_API_URL"),
    strategy="hi_res",
    split_pdf_page=True,
    split_pdf_concurrency_level=15
)

element_dicts = [element.to_dict() for element in elements]

# Print the processed data's first element only.
print(element_dicts[0])

# Write the processed data to a local file.
json_elements = json.dumps(element_dicts, indent=2)

with open("example-docs/pdf/DA-1p.pdf.json", "w") as file:
    file.write(json_elements)
```

When sending a request to the Free Unstructured API, you only need to provide your individual API key.
When sending requests to the Unstructured Serverless API, you need to supply your unique API URL in addition to your API key.
For Unstructured API on Azure/AWS, use the API URL that you configured through those services.

## Parameters & examples

The API parameters are the same across all methods of accessing the Unstructured API.

*   Refer to the [API parameters](/api-reference/api-services/api-parameters) page for the full list of available parameters.
*   Refer to the [Examples](/api-reference/api-services/examples) page for some inspiration on using the parameters.

[//]: # "TODO: when we have the concepts page shared across products, link it from here for the users to learn about partition strategies, chunking strategies and other important shared concepts"


# Partitioning strategies



For certain document types, such as images and PDFs, for example, Unstructured products offer a variety of different
ways to preprocess them, controlled by the `strategy` parameter.

PDF documents, for example, vary in quality and complexity. In simple cases, traditional NLP extraction techniques may
be enough to extract all the text out of a document. In other cases, advanced image-to-text models are required
to process a PDF. You can think of the strategies as being "rule-based" workflows (thus they are "fast"), or
"model-based" workflows (slower workflow because it requires model inference, but you get "higher resolution", thus "hi\_res").
When choosing a partitioning strategy for your files, you have to be mindful of the quality/speed trade-off.
To give you an example, the `fast` strategy is roughly 100x faster than leading image-to-text models.

**Available options:**

*   `auto` (default strategy): The "auto" strategy will choose the partitioning strategy based on document characteristics and the function kwargs.
*   `fast`:  The "rule-based" strategy leverages traditional NLP extraction techniques to quickly pull all the text elements. "Fast" strategy is not recommended for image-based file types.
*   `hi_res`: The "model-based" strategy identifies the layout of the document. The advantage of "hi\_res" is that it uses the document layout to gain additional information about document elements. We recommend using this strategy if your use case is highly sensitive to correct classifications for document elements.
*   `ocr_only`: Another "model-based" strategy that leverages Optical Character Recognition to extract text from the image-based files.

**These strategies are available on the following partition functions:**

| Document Type            | Partition Function | Strategies                             | Table Support | Options                                                                                      |
| ------------------------ | ------------------ | -------------------------------------- | ------------- | -------------------------------------------------------------------------------------------- |
| Images (.png/.jpg/.heic) | partition\_image   | “auto”, “hi\_res”, “ocr\_only”         | Yes           | Encoding; Include Page Breaks; Infer Table Structure; OCR Languages, Strategy                |
| PDFs (.pdf)              | partition\_pdf     | “auto”, “fast”, “hi\_res”, “ocr\_only” | Yes           | Encoding; Include Page Breaks; Infer Table Structure; Max Partition; OCR Languages, Strategy |


# Process an individual file by making a direct POST request



Whether you're using the free Unstructured API, the Unstructured Serverless API, Unstructured API on Azure/AWS, or your local
deployment of Unstructured API, you can work with the API by sending single-file POST requests to it.

<Info>
  Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
  [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
  to you:

  *   You need to work with documents in cloud storage.
  *   You want to cache the results of processing multiple files in batches.
  *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
</Info>

To make POST requests, you will need:

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Let's start with a simple example in which you send a PDF file (`*.pdf`) to partition via Unstructured API.

```bash
 curl -X 'POST' \
  '$UNSTRUCTURED_API_URL' \
  -H 'accept: application/json' \
  -H 'Content-Type: multipart/form-data' \
  -H 'unstructured-api-key: $UNSTRUCTURED_API_KEY' \
  -F 'files=@sample-docs/report.pdf' \
  -F 'strategy=hi_res' \
  -o 'sample-docs/report.json'
```

In the example above we're sending the request to the Free Unstructured API, in which case the API URL is the same for all
users: `https://api.unstructured.io/general/v0/general` and is represented by the environment variable `UNSTRUCTURED_API_URL`. Note, however, that you still need to authenticate yourself with
your individual API Key, represented by the environment variable `UNSTRUCTURED_API_KEY`. Learn how to obtain an API key in the Getting Started section of the documentation:
[Free Unstructured API guide](/api-reference/api-services/free-api) and [Unstructured Serverless API guide](/api-reference/api-services/saas-api-development-guide).

If you are using the Unstructured Serverless API, you need to replace the URL in this example with the unique API URL that you have
received in the same email as your API key. For Unstructured API on Azure/AWS, use the API URL that you
configured through those services.

## Parameters & examples

The API parameters are the same across all methods of accessing the Unstructured API.

*   Refer to the [API parameters](/api-reference/api-services/api-parameters) page for the full list of available parameters.
*   Refer to the [Examples](/api-reference/api-services/examples) page for some inspiration on using the parameters.

[//]: # "TODO: when we have the concepts page shared across products, link it from here for the users to learn about partition strategies, chunking strategies and other important shared concepts"

## Postman collection

Unstructured offers a [Postman collection](https://learning.postman.com/docs/collections/collections-overview/) that you can import into Postman to make POST requests through a graphical user interface.

1.  [Install Postman](https://learning.postman.com/docs/getting-started/installation/installation-and-updates/).

2.  [Sign in to Postman](https://learning.postman.com/docs/getting-started/installation/postman-account/#signing-in-to-postman).

3.  In your workspace, click **Import**.

    ![Import a Postman collection](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/post/import.png)

4.  In the **Paste cURL, Raw text or URL** box, enter the following URL, and then press `Enter`:

    ```
    https://raw.githubusercontent.com/Unstructured-IO/docs/main/examplecode/codesamples/api/Unstructured-POST.postman_collection.json
    ```

5.  On the sidebar, click **Collections**.

    ![Collections icon in the sidebar](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/post/collection.png)

6.  Expand **Unstructured POST**.

7.  Click one of the following:

    *   To call the Unstructured Serverless API, click **(Serverless API) Basic Request**.
    *   To call the Free Unstructured API, click **(Free API) Basic Request**.

8.  On the **Headers** tab, next to `unstructured-api-key`, enter your Unstructured API key in the **Value** column.

    ![Headers tab in the request area](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/post/headers.png)

    <Warning>
      Unstructured Serverless API keys do not work with the [Free Unstructured API](/api-reference/api-services/free-api). If you try to use a Serverless API key with a Free API URL, the call will fail. [Get a Serverless API key and its related Serverless API URL](/api-reference/api-services/saas-api-development-guide#get-started).

      Free Unstructured API keys do not work with the [Unstructured Serverless API](/api-reference/api-services/saas-api-development-guide). If you try to use a Free API key with a Serverless API URL, the call will fail. [Get a Free API key and its related Free API URL](/api-reference/api-services/free-api#get-an-api-key).
    </Warning>

9.  On the **Body** tab, next to `files`, click the **Select files** box in the **Value** column.

    ![Body tab in the request area](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/post/body.png)

10. Click **New file from local machine**.

11. Browse to and select the file that you want Unstructured to process.

12. Click **Send**.

To download the processed data to your local machine, in the response area, click the ellipses, and then click **Save response to file**.

![Save response to file in the response area](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/post/save.png)


# Unstructured Serverless API

This page describes how to get started with the Unstructured Serverless API. Learn how to obtain an API key to use with the Unstructured Serverless API, and get started in no time.

## Get started

To call the Unstructured Serverless API, you need an API key and API URL:

1.  Go to [https://app.unstructured.io](https://app.unstructured.io).
2.  Enter your email address and then click **Sign In** to receive a magic link to sign in to your personalized dashboard, or authenticate yourself with your Google or GitHub account.
3.  Once you authenticate with the magic link or with your Google or GitHub account, your dashboard appears.
4.  On the sidebar, click **API Keys**, if it is not already selected.
5.  To get your API key, click the copy icon in the **Actions** column for your API key. Store your copied API key in a secure location. Do not share it with others.
6.  To get your API URL, click the copy icon next to the URL next to **API URL**. Store your copied API URL in a secure location. Do not share it with others.

<Warning>
  Unstructured Serverless API keys do not work with the [Free Unstructured API](/api-reference/api-services/free-api). If you try to use an Unstructured Serverless API key with a Free Unstructured API URL, the call will fail. Use your Unstructured Serverless API URL instead.
</Warning>

[Try the quickstart](#quickstart).

## Set up billing

<iframe width="560" height="315" src="https://www.youtube.com/embed/DPITllKgYvA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

Once you sign up for the Unstructured Serverless API, you can enjoy a free 14-day trial with usage capped at 1000 pages per day.

<Info>The free 14-day trial for the Unstructured Serverless API is different than the [Free Unstructured API](/api-reference/api-services/free-api).</Info>

At the end of the 14-day free trial, or if you need to go past the trial's page processing limits during the 14-day free trial, you must set up your billing information to keep using
the Unstructured Serverless API:

1.  Go to [https://app.unstructured.io](https://app.unstructured.io) and sign in.
2.  On the sidebar, click **Billing**.
3.  Click **Manage Payment Method**, follow the on-screen instructions to enter your payment details through Stripe, and then click **Save card**.

Your card is billed monthly based on your usage. The **Billing** page shows a billing overview for the current month and a list of your billing invoices.

We calculate a page as follows:

*   For these file types, a page is a page, slide, or image: .pdf, .pptx, and .tiff.
*   For .docx files that have page metadata, we calculate the number of pages based on that metadata.
*   For all other file types, we calculate the number of pages as the file's size divided by 100 KB.

<iframe width="560" height="315" src="https://www.youtube.com/embed/VyZqGxBMbnE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

## Quickstart

These examples use your local machine. They send source (input) files from your local machine to the Unstructured Serverless API which delivers the processed data to a destination (output) location, also on your local machine. Data is processed on Unstructured-hosted compute resources.

### Unstructured Ingest CLI

To work with the Unstructured Serverless API by using the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli), you will need to:

*   Install Python, and then install the CLI package:

    ```bash
    pip install unstructured-ingest
    ```

*   Set the following environment variables:

    *   Set `UNSTRUCTURED_API_KEY` to your API key.
    *   Set `UNSTRUCTURED_API_URL` to your API URL.

    [Get your API key and API URL](#get-started).

*   Have some compatible files on your local machine to be processed. [See the list of supported file types](/api-reference/api-services/supported-file-types). If you do not have any files available, you can download some from the [example-docs](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/example-docs) folder in the Unstructured repo on GitHub.

Now, use the CLI to call the API, replacing:

*   `<path/to/input>` with the source (input) path to the directory on your local machine that contains the compatible files for Unstructured to process on its hosted compute resources.
*   `<path/to/output>` with the destination (output) path to the directory on your local machine that will contain the processed data that Unstructured returns from its hosted compute resources.

```bash CLI
unstructured-ingest \
  local \
    --input-path <path/to/input> \
    --output-dir <path/to/output> \
    --partition-by-api \
    --api-key $UNSTRUCTURED_API_KEY \
    --partition-endpoint $UNSTRUCTURED_API_URL \
    --strategy hi_res \
    --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
```

After the command successfully runs, see the results in the specified output path on your local machine.

### Unstructured Ingest Python library

To work with the Unstructured Serverless API by using the [Unstructured Python library](/ingestion/overview#unstructured-ingest-python-library), you will need to:

*   Install Python, and then install the CLI package:

    ```bash
    pip install unstructured-ingest
    ```

*   Set the following environment variables:

    *   Set `UNSTRUCTURED_API_KEY` to your API key.
    *   Set `UNSTRUCTURED_API_URL` to your API URL.

    [Get your API key and API URL](#get-started).

*   Have some compatible files on your local machine to be processed. [See the list of supported file types](/api-reference/api-services/supported-file-types). If you do not have any files available, you can download some from the [example-docs](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/example-docs) folder in the Unstructured repo on GitHub.

Now, use the CLI to call the API, replacing:

*   `<path/to/input>` with the source (input) path to the directory on your local machine that contains the compatible files for Unstructured to process on its hosted compute resources.
*   `<path/to/output>` with the destination (output) path to the directory on your local machine that will contain the processed data that Unstructured returns from its hosted compute resources.

```python Python Ingest v2
import os

from unstructured_ingest.v2.pipeline.pipeline import Pipeline
from unstructured_ingest.v2.interfaces import ProcessorConfig
from unstructured_ingest.v2.processes.connectors.local import (
    LocalIndexerConfig,
    LocalDownloaderConfig,
    LocalConnectionConfig,
    LocalUploaderConfig
)
from unstructured_ingest.v2.processes.partitioner import PartitionerConfig

if __name__ == "__main__":
    Pipeline.from_configs(
        context=ProcessorConfig(),
        indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
        downloader_config=LocalDownloaderConfig(),
        source_connection_config=LocalConnectionConfig(),
        partitioner_config=PartitionerConfig(
            partition_by_api=True,
            api_key=os.getenv("UNSTRUCTURED_API_KEY"),
            partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
            strategy="hi_res",
            additional_partition_args={
                "split_pdf_page": True,
                "split_pdf_allow_failed": True,
                "split_pdf_concurrency_level": 15
            }
        ),
        uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
    ).run()
```

After the command successfully runs, see the results in the specified output path on your local machine.

## Manage your account

**To manage your account**: Begin by going to [https://app.unstructured.io](https://app.unstructured.io) and signing in.

**To manage your API keys**:

*   On the sidebar, click **API Keys**.
*   To create a key, click **Generate New Key**, and follow the on-screen instructions.
*   To enable or disable a key, switch **On/Off** in the column for that key to on or off.
*   To delete a key, click the trash can in the **Actions** column for that key.

**To view your usage**: On the sidebar, click **Usage**.

<iframe width="560" height="315" src="https://www.youtube.com/embed/VyZqGxBMbnE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

**To view your billing costs and invoices and to manage your payment method**: On the sidebar, click **Billing**.

<iframe width="560" height="315" src="https://www.youtube.com/embed/DPITllKgYvA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

**To log out of your account**: On the sidebar, click your email address, and then click **Logout**.

If you need direct assistance, our support team is just an email away. Contact us at [support@unstructured.io](mailto:support@unstructured.io).

## Telemetry

We’ve partnered with [Scarf](https://scarf.sh) to collect anonymized user statistics to understand which features our community is using and how to prioritize product decision-making in the future.

To learn more about how we collect and use this data, please read our [Privacy Policy](https://unstructured.io/privacy-policy).

To opt out of this data collection, you can set the environment variable `SCARF_NO_ANALYTICS=true` before running any commands that call Unstructured Serverless API services.


# Process an individual file by using the Unstructured JavaScript/TypeScript SDK



The [Unstructured JavaScript/TypeScript SDK](https://github.com/Unstructured-IO/unstructured-js-client) client allows you to send an individual file for processing by Unstructured API services. Whether you're using the
Free Unstructured API, the Unstructured Serverless API, the Unstructured API on Azure/AWS, or your local
deployment of the Unstructured API, you can access the API using the JavaScript/TypeScript SDK.

<Info>
  Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
  [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
  to you:

  *   You need to work with documents in cloud storage.
  *   You want to cache the results of processing multiple files in batches.
  *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
</Info>

To use the JavaScript/TypeScript SDK, you'll need:

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

<Info>
  If you do not specify the API URL, your [Unstructured Serverless API](/api-reference/api-services/saas-api-development-guide) pay-as-you-go account will be used by default. You must always specify your Serverless API key.<br /><br />
  To use the [Free Unstructured API](/api-reference/api-services/free-api), you must always specify your Free API key, and the Free API URL which is `https://api.unstructured.io/general/v0/general`<br /><br />
  To use the pay-as-you-go Unstructured API on Azure or AWS with the SDKs, you must always specify the corresponding API URL. See the [Azure](/api-reference/api-services/azure) or [AWS](/api-reference/api-services/aws) instructions.
</Info>

## Installation

Before using the SDK to interact with Unstructured API services, install the library:

<CodeGroup>
  ```bash JavaScript/TypeScript
  npm install unstructured-client
  ```
</CodeGroup>

<Note>
  The SDK uses semantic versioning and major bumps could bring breaking changes. It is advised to
  pin your installed version.
</Note>

## Basics

Let's start with a simple example in which you send a PDF document to be partitioned with the free Unstructured API:

<Warning>
  The JavaScript/TypeScript SDK has the following breaking changes in v0.11.0:

  *   Imports under the `dist` path have moved up a level
  *   Enums are now used for parameters with a set of options
      *   This includes `chunkingStrategy`, `outputFormat`, and `strategy`
  *   All parameters to `partition` have moved to a `partitionParameters` object
</Warning>

<CodeGroup>
  ```typescript TypeScript
  import { UnstructuredClient } from "unstructured-client";
  import { PartitionResponse } from "unstructured-client/sdk/models/operations";
  import { Strategy } from "unstructured-client/sdk/models/shared";
  import * as fs from "fs";

  const key = process.env.UNSTRUCTURED_API_KEY;
  const url = process.env.UNSTRUCTURED_API_URL;

  const client = new UnstructuredClient({
      serverURL: url,
      security: {
          apiKeyAuth: key,
      },
  });

  const filename = "PATH_TO_INPUT_FILE";
  const data = fs.readFileSync(filename);

  client.general.partition({
      partitionParameters: {
          files: {
              content: data,
              fileName: filename,
          },
          strategy: Strategy.HiRes,
          splitPdfPage: true,
          splitPdfAllowFailed: true,
          splitPdfConcurrencyLevel: 15
          languages: ['eng']
      }
  }).then((res: PartitionResponse) => {
      if (res.statusCode == 200) {
          // Print the processed data's first element only.
          console.log(res.elements?.[0])

          // Write the processed data to a local file.
          const jsonElements = JSON.stringify(res.elements, null, 2)

          fs.writeFileSync("PATH_TO_OUTPUT_FILE", jsonElements)
      }
  }).catch((e) => {
      if (e.statusCode) {
        console.log(e.statusCode);
        console.log(e.body);
      } else {
        console.log(e);
      }
  });
  ```

  ```typescript TypeScript (SDK <=v0.10.6)
  import { UnstructuredClient } from "unstructured-client";
  import { PartitionResponse } from "unstructured-client/dist/sdk/models/operations";
  import * as fs from "fs";

  const key = process.env.UNSTRUCTURED_API_KEY;
  const url = process.env.UNSTRUCTURED_API_URL;

  const client = new UnstructuredClient({
      serverURL: url,
      security: {
          apiKeyAuth: key,
      },
  });

  const filename = "PATH_TO_FILE";
  const data = fs.readFileSync(filename);

  client.general.partition({
      files: {
          content: data,
          fileName: filename,
      },
      strategy: 'hi_res',
      languages: ['eng']
  }).then((res: PartitionResponse) => {
      if (res.statusCode == 200) {
          console.log(res.elements?.[0]);
      }
  }).catch((e) => {
    if (e.statusCode) {
      console.log(e.statusCode);
      console.log(e.body);
    } else {
      console.log(e);
    }
  });
  ```
</CodeGroup>

For a code example that works with an entire directory of files instead of just a single PDF, see the [Processing multiple files](#processing-multiple-files) section.

## Page splitting

In order to speed up processing of large PDF files, the `splitPdfPage`[\*](#parameter-names) parameter is `true` by default. This
causes the PDF to be split into small batches of pages before sending requests to the API. The client
awaits all parallel requests and combines the responses into a single response object. This is specific to PDF files and other
filetypes are ignored.

The number of parallel requests is controlled by `splitPdfConcurrencyLevel`[\*](#parameter-names).
The default is 8 and the max is set to 15 to avoid high resource usage and costs.

If at least one request is successful, the responses are combined into a single response object. An
error is returned only if all requests failed or there was an error during splitting.

<Note>
  This feature may lead to unexpected results when chunking because the server does not see the entire
  document context at once. If you'd like to chunk across the whole document and still get the speedup from
  parallel processing, you can:

  *   Partition the PDF with `splitPdfPage` set to `true`, without any chunking parameters.
  *   Store the returned elements in `results.json`.
  *   Partition this JSON file with the desired chunking parameters.
</Note>

```typescript TypeScript
client.general.partition({
    partitionParameters: {
        files: {
            content: data,
            fileName: filename,
        },
        strategy: Strategy.HiRes,
        // Set to `false` to disable PDF splitting
        splitPdfPage: true,
        // Continue PDF splitting even if some earlier split operations fail.
        splitPdfAllowFailed: true,
        // Modify splitPdfConcurrencyLevel to set the number of parallel requests
        splitPdfConcurrencyLevel: 10,
    }
})
```

## Customizing the client

### Retries

You can also change the defaults for retries through the `retryConfig`[\*](#parameter-names)
when initializing the client. If a request to the API fails, the client will retry the
request with an exponential backoff strategy up to a maximum interval of one minute. The
function keeps retrying until the total elapsed time exceeds `maxElapsedTime`[\*](#parameter-names),
which defaults to one hour:

```typescript TypeScript
const key = process.env.UNSTRUCTURED_API_KEY;
const url = process.env.UNSTRUCTURED_API_URL;    

const client = new UnstructuredClient({
    security: { apiKeyAuth: key },
    serverURL: url,
    retryConfig: {
        strategy: "backoff",
        retryConnectionErrors: true,
        backoff: {
            initialInterval: 500,
            maxInterval: 60000,
            exponent: 1.5,
            maxElapsedTime: 900000, // 15min*60sec*1000ms = 15 minutes
        },
    };
});
```

## Processing multiple files

The code example in the [Basics](#basics) section processes a single PDF file. But what if you want to process
multiple files inside a directory with a mixture of subdirectories and files with different file types?

The following example takes an input directory path to read files from and an output directory path to
write the processed data to, processing one file at a time.

```typescript TypeScript
import { UnstructuredClient } from "unstructured-client";
import * as fs from "fs";
import * as path from "path";
import { Strategy } from "unstructured-client/sdk/models/shared/index.js";
import { PartitionResponse } from "unstructured-client/sdk/models/operations";

// Send all files in the source path to Unstructured for processing.
// Send the processed data to the destination path.
function processFiles(
    client: UnstructuredClient,
    sourcePath: string,
    destinationPath: string
): void {

    // If an output directory does not exist for the corresponding input
    // directory, then create it.
    if (!fs.existsSync(destinationPath)) {
        fs.mkdirSync(destinationPath, { recursive: true });
    }

    // Get all folders and files at the current level of the input directory.
    const items = fs.readdirSync(sourcePath);

    // For each folder and file in the input directory...
    for (const item of items) {
        const inputPath = path.join(sourcePath, item);
        const outputPath = path.join(destinationPath, item)

        // If it's a folder, call this function recursively.
        if (fs.statSync(inputPath).isDirectory()) {
            processFiles(client, inputPath, outputPath);
        } else {
            // If it's a file, send it to Unstructured for processing.
            const data = fs.readFileSync(inputPath);

            client.general.partition({
                partitionParameters: {
                    files: {
                        content: data,
                        fileName: inputPath
                    },
                    strategy: Strategy.HiRes,
                    splitPdfPage: true,
                    splitPdfConcurrencyLevel: 15,
                    splitPdfAllowFailed: true
                }
            }).then((res: PartitionResponse) => {
                // If successfully processed, write the processed data to
                // the destination directory.
                if (res.statusCode == 200) {
                    const jsonElements = JSON.stringify(res.elements, null, 2)
                    fs.writeFileSync(outputPath + ".json", jsonElements)
                }
            }).catch((e) => {
                if (e.statusCode) {
                    console.log(e.statusCode);
                    console.log(e.body);
                } else {
                    console.log(e);
                }
            });
        }
    }
}

const client = new UnstructuredClient({
    security: { apiKeyAuth: process.env.UNSTRUCTURED_API_KEY },
    serverURL: process.env.UNSTRUCTURED_API_URL
});

processFiles(
    client,
    process.env.LOCAL_FILE_INPUT_DIR,
    process.env.LOCAL_FILE_OUTPUT_DIR
);
```

## Parameters & examples

The parameter names used in this document are for the JavaScript/TypeScript SDK, which follow camelCase
convention. The Python SDK uses snake\_case convention. Other than this difference in naming convention,
the names used in the SDKs are the same across all methods.

*   Refer to the [API parameters](/api-reference/api-services/api-parameters) page for the full list of available parameters.
*   Refer to the [Examples](/api-reference/api-services/examples) page for some inspiration on using the parameters.


# Process an individual file by using the Unstructured Python SDK



The [Unstructured Python SDK](https://github.com/Unstructured-IO/unstructured-python-client) client allows you to send an individual file for processing by Unstructured API services. Whether you're using the
Free Unstructured API, the Unstructured Serverless API, the Unstructured API on Azure/AWS, or your local
deployment of the Unstructured API, you can access the API using the Python SDK.

<Info>
  Unstructured recommends that you use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
  [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) if any of the following apply
  to you:

  *   You need to work with documents in cloud storage.
  *   You want to cache the results of processing multiple files in batches.
  *   You want more precise control over document-processing pipeline stages such as partitioning, chunking, filtering, staging, and embedding.
</Info>

To use the Python SDK, you'll need:

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

<Info>
  If you do not specify the API URL, your [Unstructured Serverless API](/api-reference/api-services/saas-api-development-guide) pay-as-you-go account will be used by default. You must always specify your Serverless API key.<br /><br />
  To use the [Free Unstructured API](/api-reference/api-services/free-api), you must always specify your Free API key, and the Free API URL which is `https://api.unstructured.io/general/v0/general`<br /><br />
  To use the pay-as-you-go Unstructured API on Azure or AWS with the SDKs, you must always specify the corresponding API URL. See the [Azure](/api-reference/api-services/azure) or [AWS](/api-reference/api-services/aws) instructions.
</Info>

## Installation

Before using the SDK to interact with Unstructured API services, install the library:

```bash Python
pip install unstructured-client
```

<Note>
  The SDK uses semantic versioning and major bumps could bring breaking changes. It is advised to
  pin your installed version. See the [migration guide](#migration-guide), later on this page, for breaking change announcements.
</Note>

## Basics

Let's start with a simple example in which you send a PDF document to be partitioned with the Unstructured API. Note that the Python SDK gives you a choice of two interfaces. The functionality is identical, but the Pydantic model syntax may provide completion hints in your IDE, for instance.

<CodeGroup>
  ```python Python (TypedDict example)
  import os, json

  import unstructured_client
  from unstructured_client.models import shared

  client = unstructured_client.UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  filename = "PATH_TO_INPUT_FILE"

  req = {
      "partition_parameters": {
          "files": {
              "content": open(filename, "rb"),
              "file_name": filename,
          },
          "strategy": shared.Strategy.HI_RES,
          "languages": ['eng'],
          "split_pdf_page": True,            # If True, splits the PDF file into smaller chunks of pages.
          "split_pdf_allow_failed": True,    # If True, the partitioning continues even if some pages fail.
          "split_pdf_concurrency_level": 15  # Set the number of concurrent request to the maximum value: 15.
      }
  }

  try:
      res = client.general.partition(request=req)
      element_dicts = [element for element in res.elements]

      # Print the processed data's first element only.
      print(element_dicts[0])

      # Write the processed data to a local file.
      json_elements = json.dumps(element_dicts, indent=2)

      with open("PATH_TO_OUTPUT_FILE", "w") as file:
          file.write(json_elements)
  except Exception as e:
      print(e)
  ```

  ```python Python (Pydantic model example)
  import os, json

  import unstructured_client
  from unstructured_client.models import operations, shared

  client = unstructured_client.UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  filename = "PATH_TO_INPUT_FILE"

  req = operations.PartitionRequest(
      partition_parameters=shared.PartitionParameters(
          files=shared.Files(
              content=open(filename, "rb"),
              file_name=filename,
          ),
          strategy=shared.Strategy.HI_RES,
          languages=['eng'],
          split_pdf_page=True,            # If True, splits the PDF file into smaller chunks of pages.
          split_pdf_allow_failed=True,    # If True, the partitioning continues even if some pages fail.
          split_pdf_concurrency_level=15  # Set the number of concurrent request to the maximum value: 15.
      ),
  )

  try:
      res = client.general.partition(request=req)
      element_dicts = [element for element in res.elements]
      
      # Print the processed data's first element only.
      print(element_dicts[0])

      # Write the processed data to a local file.
      json_elements = json.dumps(element_dicts, indent=2)

      with open("PATH_TO_OUTPUT_FILE", "w") as file:
          file.write(json_elements)
  except Exception as e:
      print(e)
  ```
</CodeGroup>

## Async partitioning

The Python SDK also has a `partition_async`. This call is equivalent to `partition` except that it can be used in a non blocking context. For instance, `asyncio.gather` can be used to concurrently process multiple files inside of a directory hierarchy, as demonstrated here:

<CodeGroup>
  ```python Python
  import asyncio
  import os
  import json
  import unstructured_client
  from unstructured_client.models import shared

  client = unstructured_client.UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  async def call_api(filename, input_dir, output_dir):
      req = {
          "partition_parameters": {
              "files": {
                  "content": open(filename, "rb"),
                  "file_name": os.path.basename(filename),
              },
              "strategy": shared.Strategy.HI_RES,
          }
      }

      try:
          res = await client.general.partition_async(request=req)
          element_dicts = [element for element in res.elements]
          json_elements = json.dumps(element_dicts, indent=2)

          # Create the output directory structure.
          relative_path = os.path.relpath(os.path.dirname(filename), input_dir)
          output_subdir = os.path.join(output_dir, relative_path)
          os.makedirs(output_subdir, exist_ok=True)

          # Write the output file.
          output_filename = os.path.join(output_subdir, os.path.basename(filename) + ".json")
          with open(output_filename, "w") as file:
              file.write(json_elements)

      except Exception as e:
          print(f"Error processing {filename}: {e}")

  async def process_files(input_directory, output_directory):
      tasks = []

      for root, _, files in os.walk(input_directory):
          for file in files:
              if not file.endswith('.json'):
                  full_path = os.path.join(root, file)
                  tasks.append(call_api(full_path, input_directory, output_directory))

      await asyncio.gather(*tasks)

  if __name__ == "__main__":
      asyncio.run(process_files(
          input_directory=os.getenv("LOCAL_FILE_INPUT_DIR"), 
          output_directory=os.getenv("LOCAL_FILE_OUTPUT_DIR")
      ))
  ```
</CodeGroup>

## Page splitting

In order to speed up processing of large PDF files, the `split_pdf_page`[\*](#parameter-names) parameter is `True` by default. This
causes the PDF to be split into small batches of pages before sending requests to the API. The client
awaits all parallel requests and combines the responses into a single response object. This is specific to PDF files and other
filetypes are ignored.

The number of parallel requests is controlled by `split_pdf_concurrency_level`[\*](#parameter-names).
The default is 8 and the max is set to 15 to avoid high resource usage and costs.

If at least one request is successful, the responses are combined into a single response object. An
error is returned only if all requests failed or there was an error during splitting.

<Note>
  This feature may lead to unexpected results when chunking because the server does not see the entire
  document context at once. If you'd like to chunk across the whole document and still get the speedup from
  parallel processing, you can:

  *   Partition the PDF with `split_pdf_page` set to `True`, without any chunking parameters.
  *   Store the returned elements in `results.json`.
  *   Partition this JSON file with the desired chunking parameters.
</Note>

```python Python
req = operations.PartitionRequest(
    partition_parameters=shared.PartitionParameters(
        files=shared.Files(
            content=file.read(),
            file_name=filename,
        ),
        strategy=shared.Strategy.HI_RES,
        split_pdf_page=True,            # If True, splits the PDF file into smaller chunks of pages.
        split_pdf_allow_failed=True,    # If True, the partitioning continues even if some pages fail.
        split_pdf_concurrency_level=15  # Set the number of concurrent request to the maximum value: 15.
    )
)
res = client.general.partition(req)
```

## Customizing the client

### Retries

You can also change the defaults for retries through the `retry_config`[\*](#parameter-names)
when initializing the client. If a request to the API fails, the client will retry the
request with an exponential backoff strategy up to a maximum interval of one minute. The
function keeps retrying until the total elapsed time exceeds `max_elapsed_time`[\*](#parameter-names),
which defaults to one hour:

```python Python
import os 

client = UnstructuredClient(
    api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
    server_url=os.getenv("UNSTRUCTURED_API_URL"),
    retry_config=RetryConfig(
        strategy="backoff",
        retry_connection_errors=True,
        backoff=BackoffStrategy(
            # time intervals are defined in milliseconds
            initial_interval=500,
            max_interval=60000,
            exponent=1.5,
            max_elapsed_time=900000,  # 15min*60sec*1000ms = 15 minutes
        ),
    )
)
```

### Disabling SSL validation

If you disable SSL validation, requests will accept any TLS certificate
presented by the server and ignore hostname mismatches and/or expired certificates,
which will make your application vulnerable to man-in-the-middle (MitM) attacks.
Only set this to `False` for testing.

```python Python
http_client = requests.Session()
http_client.verify = False

client = UnstructuredClient(
    client=http_client,
    ...
)
```

## Handling the response

The partition response defaults to a dict format that can be converted to Unstructured elements with
the `elements_from_dicts` utility function as seen below. Otherwise, the API response can be sent directly
to your vector store or another destination.

```python Python
from unstructured.staging.base import elements_from_dicts

# ...

if res.elements is not None:
    elements = elements_from_dicts(response.elements)
```

## Parameters & examples

The parameter names used in this document are for the Python SDK, which follow snake\_case convention. The JavaScript/TypeScript SDK follows camelCase
convention. Other than this difference in naming convention,
the names used in the SDKs are the same across all methods.

*   Refer to the [API parameters](/api-reference/api-services/api-parameters) page for the full list of available parameters.
*   Refer to the [Examples](/api-reference/api-services/examples) page for some inspiration on using the parameters.

## Migration guide

There are minor breaking changes in 0.26.0. If you encounter any errors when upgrading, please find the solution below.

**If you see the error: `AttributeError: 'PartitionParameters' object has no attribute 'partition_parameters'`**

Previously, the SDK accepted a `PartitionParameters` object as input to the `sdk.general.partition` function. Now, this object must be wrapped in a `PartitionRequest` object. The old behavior was deprecated in 0.23.0 and removed in 0.26.0.

```python
# Instead of:
from unstructured_client.models import shared

req = shared.PartitionParameters(
    files=files,
)

resp = s.general.partition(request=req)


# Switch to:
from unstructured_client.models import shared, operations

req = operations.PartitionRequest(
    partition_parameters=shared.PartitionParameters(
        files=files,
    )
)

resp = s.general.partition(request=req)
```

**If you see the error: `TypeError: BaseModel.__init__() takes 1 positional argument but 2 were given`**

In 0.26.0, the `PartitionRequest` constructor no longer allows for positional arguments. You must specify `partition_parameters` by name.

```python
# Instead of:
req = operations.PartitionRequest(
    shared.PartitionParameters(
        files=files,
    )
)

# Switch to:
req = operations.PartitionRequest(
    partition_parameters=shared.PartitionParameters(
        files=files,
    )
)
```

**If you see the error: `TypeError: General.partition() takes 1 positional argument but 2 were given`**

In 0.26.0, the `partition` function no longer allows for positional arguments. You must specify `request` by name.

```python
# Instead of:
resp = s.general.partition(req)

# Switch to:
resp = s.general.partition(request=req)
```


# Supported file types



Unstructured supports processing of the following file types:

By file extension:

| File extension |
| -------------- |
| `.bmp`         |
| `.csv`         |
| `.doc`         |
| `.docx`        |
| `.eml`         |
| `.epub`        |
| `.heic`        |
| `.html`        |
| `.jpeg`        |
| `.png`         |
| `.md`          |
| `.msg`         |
| `.odt`         |
| `.org`         |
| `.p7s`         |
| `.pdf`         |
| `.png`         |
| `.ppt`         |
| `.pptx`        |
| `.rst`         |
| `.rtf`         |
| `.tiff`        |
| `.txt`         |
| `.tsv`         |
| `.xls`         |
| `.xlsx`        |
| `.xml`         |

By file type:

| Category          | File types                                |
| ----------------- | ----------------------------------------- |
| CSV               | `.csv`                                    |
| E-mail            | `.eml`, `.msg`, `.p7s`                    |
| EPUB              | `.epub`                                   |
| Excel             | `.xls`, `.xlsx`                           |
| HTML              | `.html`                                   |
| Image             | `.bmp`, `.heic`, `.jpeg`, `.png`, `.tiff` |
| Markdown          | `.md`                                     |
| Org Mode          | `.org`                                    |
| Open Office       | `.odt`                                    |
| PDF               | `.pdf`                                    |
| Plain text        | `.txt`                                    |
| PowerPoint        | `.ppt`, `.pptx`                           |
| reStructured Text | `.rst`                                    |
| Rich Text         | `.rtf`                                    |
| TSV               | `.tsv`                                    |
| Word              | `.doc`, `.docx`                           |
| XML               | `.xml`                                    |


# Chunking





# Embedding





# Speed up processing of large files and batches



When you use Unstructured API services, here are some techniques that you can try to help speed up the processing of large files and large batches of files.

*   Choose your partitioning strategy wisely. For example, if you have simple PDFs that don't have images and tables, you might be able to use the `fast` strategy. Try the `fast` strategy on a few of your documents before you try using the `hi_res` strategy. [Learn more](/api-reference/api-services/partitioning).
*   For processing large numbers of documents, use [ingestion](/ingestion/overview) and [add CPUs](#adding-cpus).
*   For processing large individual PDF files with the Unstructured SDKs, [use PDF splitting parameters](#pdf-files).

## Adding CPUs

For speeding up file processing during [ingestion](/ingestion/overview), the Unstructured CLI and Unstructured Python Ingest enable you to instruct Unstructured to use additional local CPUs where applicable.

Using additional local CPUs applies only to pipeline steps that Unstructured logs as being processed across CPUs. It does not apply to pipeline steps that are logged as being processed asynchronously. To get a list of which operations are processed where, look for the following log messages when you run an ingest pipeline:

*   Steps that are processed across CPUs correspond to log messages that read: `processing content across processes`. These steps might benefit by setting a higher number of local CPUs to be used.
*   Steps that are processed asynchronously correspond to log messages that read: `processing content across processes`. Any settings to use a higher number of local CPUs are ignored for these steps.

For the Unstructured CLI, you can set `--num-processes` to the maximum number of available local CPUs that you want to use where applicable, for example:

```bash
unstructured-ingest \
  local \
    --num-processes <number>
    # ...
```

To get the maximum number of available local logical CPUs that can be used where applicable, see your operating system's documentation.

For Unstructured Python Ingest, you can set the `ProcessorConfig` object's `num_processes` parameter to the maximum number of available local CPUs that you want to use where applicable, for example:

<CodeGroup>
  ```python Python Ingest v2
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  # ...

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(
              num_processes=<number>,
              # ...  
          ),
          # ...
      ).run()
  ```

  ```python Python Ingest v1
  from unstructured_ingest.interfaces import (
      ProcessorConfig,
      # ...
  )
  from unstructured_ingest.runner import LocalRunner

  # ...

  if __name__ == "__main__":
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              num_processes=<number>,
              # ...
          ),
          # ...
      ).run()
  ```
</CodeGroup>

In Python, to specify the maximum number of available local logical CPUs that can be used where applicable, you can call functions such as [os.cpu\_count](https://docs.python.org/3/library/os.html#os.cpu_count) and [multiprocessing.cpu\_count](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count).

## PDF files

To speed up PDF file processing, the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli), the [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library), the [Unstructured SDK for Python](/api-reference/api-services/sdk-python), and the [Unstructured SDK for JavaScript/TypeScript](/api-reference/api-services/sdk-jsts) provide the following parameters to help speed up processing a large PDF file:

*   `split_pdf_page` (CLI/Python) or `splitPdfPage` (JavaScript/TypeScript), when set to true, splits the PDF file on the client side before sending it as batches to Unstructured for processing. The number of pages in each batch is determined internally. Batches can contain between 2 and 20 pages.
*   `split_pdf_concurrency_level` (CLI/Python) or `splitPdfConcurrencyLevel` (JavaScript/TypeScript) is an integer that specifies the number of parallel requests. The default is 5. The maximum is 15. This behavior is ignored unless `split_pdf_page` (CLI/Python) or `splitPdfPage` (JavaScript/TypeScript) is also set to true.
*   `split_pdf_allow_failed` (CLI/Python) or splitPdfAllowFailed\` (JavaScript/TypeScript), when set to true, allows partitioning to continue even if some pages fail.
*   `split_pdf_page_range` (CLI/Python only) is a list of two integers that specify the beginning and ending page numbers of the PDF file to be sent. A `ValueError` is raised if the specified range is not valid. This behavior is ignored unless `split_pdf_page` is also set to true.

[Learn more](/api-reference/api-services/sdk#page-splitting).


# Pipeline

post /general/v0/general
Description



# Set chunking behavior





# Change element coordinate systems





# Set partitioning behavior





# Choose a hi-res model



## Task

You want to specify a high-resolution object detection model to be used when processing image files, or PDFs with embedded images or tables, but you are not sure which model to specify.

## Approach

Use the following decision-maker to help you determine which model to specify.

<Steps>
  <Step title="Are you processing image files, or PDFs with embedded images or tables?">
    *   If **Yes**, then continue with **Step 2**.
    *   If **No**, then Unstructured will not use a high-resolution object detection model when processing your files. Set the command's `--strategy` option (CLI) or `strategy` parameter (Python/JavaScript/TypeScript) to `fast`. See also [Choose a partitioning strategy](/api-reference/how-to/choose-partitioning-strategy).
  </Step>

  <Step title="To process image files, or PDFs with embedded images or tables...">
    If you already have your scripts or code in place and just need help in choosing a model, then skip ahead to **Step 3**. Otherwise:

    *   To have Unstructured make its best choice on your behalf about the model to use, set the command's `--strategy` option (CLI) or `strategy` parameter (Python/JavaScript/TypeScript) to `auto`. You have completed this decision-maker. See also [Auto partitioning strategy logic](/api-reference/how-to/choose-partitioning-strategy#auto-partitioning-strategy-logic).
    *   To specify a specific model, set `--strategy` or `strategy` to `hi_res`. Then set `--hi-res-model-name` (CLI), `hi_res_model_name` (Python), or `hiResModelName` (JavaScript/TypeScript) to one of the models in **Step 3**.
  </Step>

  <Step title="Specify one of the following models...">
    *   `layout_v1.1.0` generally performs better than `yolox` at bounding box definitions and element classification. `layout_v1.1.0` is a proprietary Unstructured object detection model and is used by default, as applicable, if `--hi-res-model-name`, `hi_res_model_name`, or `hiResModelName` is not specified.
    *   `yolox` is also provided for backwards compatibility and originally was the replacement for `detectron2_onnx`.
    *   `detectron2_onnx` generally underperforms the preceding models. However, it is still accessible to maintain backwards compatibility.
  </Step>
</Steps>

<iframe width="560" height="315" src="https://www.youtube.com/embed/SwJVB_kPqTc" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

## Code examples

See [Changing partition strategy for a PDF](/api-reference/api-services/examples#changing-partition-strategy-for-a-pdf).


# Choose a partitioning strategy



## Task

You want to use the fastest, highest-precision, yet most cost-effective [partitioning strategy](/api-reference/api-services/partitioning) overall, but you are not sure which one to specify for your particular files.

## Approach

The `--strategy` command option (CLI) or `strategy` parameter (Python/JavaScript/TypeScript) specifies the partitioning strategy to use.

Use the following decision-maker to help you determine what to set `--strategy` or `strategy` to.

<Steps>
  <Step title="How many of the files to be processed are image files, or PDF files with embedded images or tables in them?">
    *   If **All**, you can increase precision for processing images and tables by specifying `hi_res` for `--strategy` or `strategy`. For additional benefits, skip ahead to **Step 2**.
    *   If **Some**, you can increase precision for processing images and tables by specifying `auto` for `--strategy` or `strategy`. This will leave the decision up to Unstructured on a file-by-file basis about the best partitioning strategy to use. You have completed this decision-maker. See also [Auto partitioning strategy logic](#auto-partitioning-strategy-logic).
    *   If **None**, you can increase performance and decrease cost by specifying `fast` for `--strategy` or `strategy`. You have completed this decision-maker.

    <Note>For OCR, you can also set `--strategy` or `strategy` to `ocr_only`. Unstructured will use the Tesseract OCR agent as applicable. You cannot specify any other OCR agent to be used instead.</Note>
  </Step>

  <Step title="For object detection, do you know which particular high-resolution object detection model you want to use?">
    *   If **No**, then specify `auto` for `--strategy` or `strategy`. This will decrease performance compared to answering **Yes**. You have completed this decision-maker. See also [Auto partitioning strategy logic](#auto-partitioning-strategy-logic).
    *   If **Yes**, then set `--hi-res-model-name` (CLI), `hi_res_model_name` (Python), or `hiResModelName`(JavaScript/TypeScript) to the available model's name. [Learn about the available models](/api-reference/how-to/choose-hi-res-model).
  </Step>
</Steps>

<iframe width="560" height="315" src="https://www.youtube.com/embed/SwJVB_kPqTc" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

## Code example

See [Changing partition strategy for a PDF](/api-reference/api-services/examples#changing-partition-strategy-for-a-pdf).

## Auto partitioning strategy logic

Setting `--strategy` or `strategy` to `auto` leaves the decision up to Unstructured on a file-by-file basis about which partitioning strategy to use. Specifically:

*   If the file is an image, the `hi_res` strategy is used for that file. The `layout_v1.0.0` high-resolution object detection model is used.

*   If the file is a PDF, the local processing logic or Unstructured tries to detect whether there are any embedded tables or images in that file.

    *   If no embedded tables or images are detected, the `fast` strategy is used for that file. No high-resolution object detection model is used.
    *   If at least one embedded table or image is found, the `hi_res` strategy is used for that file. The `layout_v1.0.0` high-resolution object detection model is used.

*   If `--strategy` or `strategy` is not specified, the `auto` strategy is used by default.


# Set embedding behavior



<Note>
  The following information applies only to the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) and the [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library).

  The Unstructured SDKs for Python and JavaScript/TypeScript, and the Unstructured open-source library, do not support this functionality.
</Note>

## Concepts

You can use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library)
to generate *embeddings* after the partitioning and chunking steps in an ingest pipeline. The chunking step is particularly important to ensure that the text pieces
(also known as the *documents* or *elements*) can fit the input limits of an *embedding model*.

You generate embeddings by specifying an embedding model that is provided or used by an *embedding provider*.
An [embedding model](https://python.langchain.com/v0.2/docs/concepts/#embedding-models) creates arrays of numbers
known as *vectors*, representing the text that is extracted by Unstructured.
These vectors are stored or *embedded* next to the data itself.

These vector embeddings allow *vector databases* to more quickly and efficiently analyze and process these inherent
properties and relationships between data. For example, you can save the extracted text along with its embeddings in a *vector store*.
When a user queries a retrieval augmented generation (RAG) application, the application can use a vector database to perform a similarity search in that vector store
and then return the documents whose embeddings are the closest to that user's query.

Learn more about [chunking](https://unstructured.io/blog/chunking-for-rag-best-practices) and
[embedding](https://unstructured.io/blog/understanding-embedding-models-make-an-informed-choice-for-your-rag).

## Generate embeddings

<iframe width="560" height="315" src="https://www.youtube.com/embed/ulMrzxNGc7A" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

To use the Ingest CLI or Ingest Python library to generate embeddings, do the following:

1.  Choose an embedding provider that you want to use from among the following allowed providers, and note the provider's ID:

    *   The provider ID `aws-bedrock` for [Amazon Bedrock](https://aws.amazon.com/bedrock/). [Learn more](https://python.langchain.com/v0.2/docs/integrations/text_embedding/bedrock/).
    *   `huggingface` for [Hugging Face](https://huggingface.co/). [Learn more](https://python.langchain.com/v0.2/docs/integrations/text_embedding/huggingfacehub/).
    *   `mixedbread-ai` for [Mixedbread](https://www.mixedbread.ai/). [Learn more](https://www.mixedbread.ai/docs/embeddings/overview).
    *   `octoai` for [Octo AI](https://octo.ai/). [Learn more](https://octo.ai/docs/text-gen-solution/using-unstructured-io-for-embedding-documents).
    *   `openai` for [OpenAI](https://openai.com/). [Learn more](https://python.langchain.com/v0.2/docs/integrations/text_embedding/openai/).
    *   `togetherai` for [Together.ai](https://www.together.ai/). [Learn more](https://docs.together.ai/docs/embedding-models).
    *   `vertexai` for [Google Vertex AI PaLM](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview). [Learn more](https://python.langchain.com/v0.2/docs/integrations/text_embedding/google_vertex_ai_palm/).
    *   `voyageai` for [Voyage AI](https://www.voyageai.com/). [Learn more](https://python.langchain.com/v0.2/docs/integrations/text_embedding/voyageai/).

2.  Run the following command to install the required Python package for the embedding provider:

    *   For `aws-bedrock`, run `pip install "unstructured-ingest[bedrock]"`.
    *   For `huggingface`, run `pip install "unstructured-ingest[embed-huggingface]"`.
    *   For `mixedbread-ai`, run `pip install "unstructured-ingest[embed-mixedbreadai]"`.
    *   For `octoai`, run `pip install "unstructured-ingest[embed-octoai]"`.
    *   For `openai`, run `pip install "unstructured-ingest[openai]"`.
    *   For `togetherai`, run `pip install "unstructured-ingest[togetherai]"`.
    *   For `vertexai`, run `pip install "unstructured-ingest[embed-vertexai]"`.
    *   For `voyageai`, run `pip install "unstructured-ingest[embed-voyageai]"`.

3.  For the following embedding providers, you can choose the model that you want to use. If you do choose a model, note the model's name:

    *   `aws-bedrock`. [Choose a model](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html). No default model is provided. [Learn more about the supported models](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html).
    *   `huggingface`. [Choose a model](https://huggingface.co/models?other=embeddings), or use the default model [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2).
    *   `mixedbread-ai`. [Choose a model](https://www.mixedbread.ai/docs/embeddings/models), or use the default model [mixedbread-ai/mxbai-embed-large-v1](https://www.mixedbread.ai/docs/embeddings/mxbai-embed-large-v1).
    *   `octoai`. [Choose a model](https://octo.ai/blog/supercharge-rag-performance-using-octoai-and-unstructured-embeddings/), or use the default model `thenlper/gte-large`.
    *   `openai`. [Choose a model](https://platform.openai.com/docs/guides/embeddings/embedding-models), or use the default model `text-embedding-ada-002`.
    *   `togetherai`. [Choose a model](https://docs.together.ai/docs/embedding-models), or use the default model `togethercomputer/m2-bert-80M-8k-retrieval`.
    *   `vertexai`. [Choose a model](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text-embeddings-api), or use the default model `textembedding-gecko@001`.
    *   `voyageai`.  [Choose a model](https://docs.voyageai.com/docs/embeddings). No default model is provided.

4.  Note the special settings to connect to the provider:

    *   For `aws-bedrock`, you'll need an AWS access key value, the corresponding AWS secret access key value, and the corresponding AWS Region identifier. [Get an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
    *   For `huggingface`, if you use a gated model (a model with special conditions that you must accept before you can use it, or a privately published model), you'll need an HF inference API key value, beginning with `hf_`. [Get an HF inference API key](https://huggingface.co/docs/api-inference/en/quicktour#get-your-api-token). To learn whether your model requires an HF inference API key, see your model provider's documentation.
    *   For `mixedbread-ai`, you'll need a Mixedbread API key value. [Get a Mixedbread API key](https://www.mixedbread.ai/dashboard?next=api-keys).
    *   For `octoai`, you'll need an Octo AI API token value. [Get an Octo AI API token](https://octo.ai/docs/getting-started/how-to-create-octoai-access-token).
    *   For `openai`, you'll need an OpenAI API key value. [Get an OpenAI API key](https://platform.openai.com/docs/quickstart/create-and-export-an-api-key).
    *   For `togetherai`, you'll need a together.ai API key value. [Get a together.ai API key](https://docs.together.ai/reference/authentication-1).
    *   For `vertexai`, you'll need the path to a Google Cloud credentials JSON file. Learn more [here](https://cloud.google.com/docs/authentication/application-default-credentials#GAC) and [here](https://googleapis.dev/python/google-auth/latest/reference/google.auth.html#module-google.auth).
    *   For `voyageai`, you'll need a Voyage AI API key value. [Get a Voyage AI API key](https://docs.voyageai.com/docs/api-key-and-installation#authentication-with-api-keys).

5.  Now, apply all of this information as follows, and then run your command or code:

<AccordionGroup>
  <Accordion title="Ingest CLI">
    For the [source connector](/api-reference/ingest/source-connectors/overview) command:

    *   Set the command's `--embedding-provider` to the provider's ID, for example `huggingface`.
    *   Set `--embedding-model-name` to the model name, as applicable, for example `sentence-transformers/sentence-t5-xl`. Or omit this to use the default model, as applicable.
    *   Set `--embedding-api-key` to the provider's required API key value or credentials JSON file path, as appropriate.
    *   For `aws-bedrock`:

        *   Set `--embedding-aws-access-key-id` to the AWS access key value.
        *   Set `--embedding-aws-secret-access-key` to the corresponding AWS secret access key value.
        *   Set `--embedding-aws-region` to the corresponding AWS Region identifier.
  </Accordion>

  <Accordion title="Ingest Python library">
    For the [source connector's](/api-reference/ingest/source-connectors/overview) `EmbedderConfig` object:

    *   Set the `embedding_provider` parameter to the provider's ID, for example `huggingface`.
    *   Set `embedding_model_name` to the model name, as applicable, for example `sentence-transformers/sentence-t5-xl`. Or omit this to use the default model, as applicable.
    *   Set `embedding_api_key` to the provider's required API key value or credentials JSON file path, as appropriate.
    *   For `aws-bedrock`:

        *   Set `embedding_aws_access_key_id` to the AWS access key value.
        *   Set `embedding_aws_secret_access_key` to the corresponding AWS secret access key value.
        *   Set `embedding_aws_region` to the corresponding AWS Region identifier.
  </Accordion>
</AccordionGroup>


# Extract images and tables from documents



## Task

You want to get, decode, and show elements, such as images and tables, that are embedded in a PDF document.

## Approach

Extract the Base64-encoded representation of specific elements, such as images and tables, in the document.
For each of these extracted elements, decode the Base64-encoded representation of the element into its original visual representation
and then show it.

## To run this example

You will need a document that is one of the document types supported by the `extract_image_block_types` argument.
See the `extract_image_block_types` entry in [API Parameters](/api-reference/api-services/api-parameters).
This example uses a PDF file with embedded images and tables.

## Code

<AccordionGroup>
  <Accordion title="Ingest Python library">
    For the [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library), you can use the standard Python
    [json.load](https://docs.python.org/3/library/json.html#json.load) function to load into a Python dictionary the contents of a JSON
    file that the Ingest Python library outputs after the processing is complete.

    ```python Python
    import json, base64, io
    from PIL import Image

    def get_image_block_types(input_json_file_path: str):
        with open(input_json_file_path, 'r') as file:
            file_elements = json.load(file)

        for element in file_elements:
            if "image_base64" in element["metadata"]:
                # Decode the Base64-encoded representation of the 
                # processed "Image" or "Table" element into its original
                # visual representation, and then show it.
                image_data = base64.b64decode(element["metadata"]["image_base64"])
                image = Image.open(io.BytesIO(image_data))
                image.show()

    if __name__ == "__main__":
        # Source: https://github.com/Unstructured-IO/unstructured-ingest/blob/main/example-docs/pdf/embedded-images-tables.pdf

        # Specify where to get the local file, relative to this .py file.
        get_image_block_types(
            input_json_file_path="local-ingest-output/embedded-images-tables.json"
        )
    ```
  </Accordion>

  <Accordion title="Python SDK">
    For the [Unstructured Python SDK](/api-reference/api-services/sdk-python), you'll need:

    These environment variables:

    *   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
    *   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

    ```python Python SDK
    from unstructured_client import UnstructuredClient
    from unstructured_client.models import operations, shared
    from unstructured.staging.base import elements_from_dicts, elements_to_json

    import os
    import base64
    from PIL import Image
    import io

    if __name__ == "__main__":
        client = UnstructuredClient(
            api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
            server_url=os.getenv("UNSTRUCTURED_API_URL")
        )

        # Source: https://github.com/Unstructured-IO/unstructured/blob/main/example-docs/embedded-images-tables.pdf
        
        # Where to get the input file and store the processed data, relative to this .py file.
        local_input_filepath = "local-ingest-input-pdf/embedded-images-tables.pdf"
        local_output_filepath = "local-ingest-output/embedded-images-tables.json"

        with open(local_filepath, "rb") as f:
            files = shared.Files(
                content=f.read(),
                file_name=local_input_filepath
            )

        request = operations.PartitionRequest(
            shared.PartitionParameters(
                files=files,
                strategy=shared.Strategy.HI_RES,
                split_pdf_page=True,
                split_pdf_allow_failed=True,
                split_pdf_concurrency_level=15,
                # Extract the Base64-encoded representation of each
                # processed "Image" and "Table" element. Extract each into
                # an "image_base64" object, as a child of the
                # "metadata" object, for that element in the result.
                # Element type names, such as "Image" and "Table" here,
                # are case-insensitive.
                # Any available Unstructured element type is allowed.
                extract_image_block_types=["Image", "Table"]
            )
        )

        try:
            result = await client.general.partition_async(request)

            for element in result.elements:
                if "image_base64" in element["metadata"]:
                    # Decode the Base64-encoded representation of the 
                    # processed "Image" or "Table" element into its original
                    # visual representation, and then show it.
                    image_data = base64.b64decode(element["metadata"]["image_base64"])
                    image = Image.open(io.BytesIO(image_data))
                    image.show()

        except Exception as e:
            print(e)
    ```
  </Accordion>
</AccordionGroup>

## See also

*   [Extract text as HTML](/api-reference/how-to/text-as-html)
*   [Table extraction from PDF](/examplecode/codesamples/apioss/table-extraction-from-pdf)


# Process a subset of files



<Note>
  The following information applies only to the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) and the [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library).

  The Unstructured SDKs for Python and JavaScript/TypeScript and the Unstructured open-source library do not support this functionality.
</Note>

## Task

You want to process only files with specified extensions, only files at or below a specified size, or both.

## Approach

For the Ingest CLI, use the following command options. For the Ingest Python library, use the following parameters for the `FiltererConfig` object.

*   Use `--file-glob` (CLI) or `file_glob` (Python) to specify the list of file extensions to process.
*   Use `--max-file-size` (CLI) or `max_file_size` (Python) to specify the maximum size of files to process, in bytes.

## To run this example

The following example processes only `.pdf` and `.eml` files that have a file size of 100 KB or less. To run this example, you should have a directory
with a mixture of files, including at least one `.pdf` file and one `.eml` file, and with at least one of these files having a file size of 100 KB or less.

## Code

<CodeGroup>
  ```bash CLI Ingest v2
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --file-glob "*.pdf,*.eml" \
      --max-file-size 100000 \
      --partition-by-api \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --api-key $UNSTRUCTURED_API_KEY \
      --strategy hi_res
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig,
      LocalUploaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.filter import FiltererConfig

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          filterer_config=FiltererConfig(
              file_glob=["*.pdf","*.eml"],
              max_file_size=100000
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "unique_element_ids": True,
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>


# Generate a JSON schema for a file



## Task

You want to generate a schema for a JSON file that Unstructured produces, so that you can validate, test,
and document related JSON files across your systems.

## Approach

Use a Python package such as [genson](https://pypi.org/project/genson/) to generate schemas for your
JSON files.

<Info>The `genson` package is not owned or supported by Unstructured. For questions and
requests, see the [Issues](https://github.com/wolverdude/genson/issues) tab of the
`genson` repository in GitHub.</Info>

## Generate a schema from the terminal

<Steps>
  <Step title="Install genson">
    Use [pip](https://pip.pypa.io/en/stable/installation/) to install the [genson](https://pypi.org/project/genson/) package.

    ```bash
    pip install genson
    ```
  </Step>

  <Step title="Install jq">
    By default, `genson` generates the JSON schema as a single string without any line breaks or indented whitespace.

    To pretty-print the schema that `genson` produces, install the [jq](https://jqlang.github.io/jq/) utility.

    <Info>The `jq` utility is not owned or supported by Unstructured. For questions and
    requests, see the [Issues](https://github.com/jqlang/jq/issues) tab of the
    `jq` repository in GitHub.</Info>
  </Step>

  <Step title="Generate the schema">
    1.  Run the `genson` command, specifying the path to the input (source) JSON file, and the path to
        the output (target) JSON schema file to be generated. Use `jq` to pretty-print the schema's content
        into the file to be generated.

        ```bash
        genson "/path/to/input/file.json" | jq '.' > "/path/to/output/schema.json"
        ```

    2.  You can find the generated JSON schema file in the output path that you specified.
  </Step>
</Steps>

## Generate a schema from Python code

<Steps>
  <Step title="Install dependencies">
    In your Python project, install the [genson](https://pypi.org/project/genson/) package.

    ```bash
    pip install genson
    ```
  </Step>

  <Step title="Add and run the schema generation code">
    1.  Set the following local environment variables:

        *   Set `LOCAL_FILE_INPUT_PATH` to the local path to the input (source) JSON file.
        *   Set `LOCAL_FILE_OUTPUT_PATH` to the local path to the output (target) JSON schema file to be generated.

    2.  Add the following Python code file to your project:

        ```python
        import os, json
        from genson import SchemaBuilder

        def json_schema_from_file(
            input_file_path: str,
            output_schema_path: str
        ) -> None:
            try:
                with open(input_file_path, "r") as file:
                    json_data = json.load(file)

                builder = SchemaBuilder()
                builder.add_object(json_data)

                schema = builder.to_schema()

                try:
                    with open(output_schema_path, "w") as schema_file:
                        json.dump(schema, schema_file, indent=2)
                except IOError as e:
                    raise IOError(f"Error writing to output file: {e}")
                
                print(f"JSON schema successfully generated and saved to '{output_schema_path}'.")
            except FileNotFoundError:
                print(f"Error: Input file '{input_file_path}' not found.")
            except IOError as e:
                print(f"I/O error occurred: {e}")
            except Exception as e:
                print(f"An unexpected error occurred: {e}")

        if __name__ == "__main__":
            json_schema_from_file(
                input_file_path=os.getenv("LOCAL_FILE_INPUT_PATH"),
                output_schema_path=os.getenv("LOCAL_FILE_OUTPUT_PATH")
            )
        ```

    3.  Run the Python code file.

    4.  Check the path specified by `LOCAL_FILE_OUTPUT_PATH` for the generated JSON schema file.
  </Step>
</Steps>


# Get chunked elements



## Task

You want to get, and print or save, the contents of elements that have been chunked.

## Approach

Chunked elements are typically represented in this format:

```json
{
  "type": "CompositeElement",
  "element_id": "3800f7ff047e97cbf0a5f7df7ad52c80",
  "text": "4,5 Exception noted in \u201cSpecial Considerations for Promotional Labeling and Advertising Material\u201d\n\n3",
  "metadata": {
    "filetype": "application/pdf",
    "languages": ["eng"],
    "page_number": 6,
    "orig_elements": "eJy ... Full content omitted for brevity ... x8=",
    "filename": "Portable-Document-Format-Specifications.pdf"
  }
}
```

```json
{
  "type": "Table",
  "element_id": "758127b42c51b93b59abf08640d1ccab",
  "text": "Gels and karyotypes High pressure liquid chromatography 600 dpi (8 bit grayscale depth) 300 dpi",
  "metadata": {
    "text_as_html": "<table><tbody><tr><td>Gels and karyotypes</td><td>600 dpi (8 bit grayscale depth)</td></tr><tr><td>High pressure liquid chromatography</td><td>300</td></tr></tbody></table>",
    "filetype": "application/pdf",
    "languages": ["eng"],
    "page_number": 8,
    "orig_elements": "eJy ... Full content omitted for brevity ... MnD",
    "filename": "Portable-Document-Format-Specifications.pdf"
}
```

To get the element's chunked content, extract the contents of the element's `text` field.

The chunked content might not contain all of its associated content. To get all of the elements that were used to derive this chunked content, extract the contents of the element's `orig_elements` field, which is nested inside of its parent `metadata` field.

The contents of the `orig_elements` field is in compressed Base64 gzipped format. To get the content, Base64-decode the bytes, decompress them, and then decode them using UTF-8.

## To run this example

You will need to chunk a document during processing. This example uses a PDF file chunked into 200- to 300-character elements.

## Code

<AccordionGroup>
  <Accordion title="Ingest Python library">
    For the [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library), you can use the standard Python
    [json.load](https://docs.python.org/3/library/json.html#json.load) function to load into a Python dictionary the contents of a JSON
    file that the Ingest Python library outputs after the processing is complete.

    ```python Python
    import json, base64, zlib
    from typing import List, Dict, Any

    # Extract the contents of an orig_elements field.
    def extract_orig_elements(orig_elements):
        decoded_orig_elements = base64.b64decode(orig_elements)
        decompressed_orig_elements = zlib.decompress(decoded_orig_elements)
        return decompressed_orig_elements.decode('utf-8')

    def get_chunked_elements(input_json_file_path: str) -> List[Dict[str, Any]]:
        # Create a dictionary that will hold only
        # a transposed version of the returned elements. 
        # For instance, we just want to capture each element's ID,
        # the chunk's text, and the chunk's associated elements in context.
        orig_elements_dict: List[Dict[str, Any]] = []

        with open(input_json_file_path, 'r') as file:
            file_elements = json.load(file)

        for element in file_elements:
            # For each chunk that has an "orig_elements" field...
            if "orig_elements" in element["metadata"]:
                # ...get the chunk's associated elements in context...
                orig_elements = extract_orig_elements(element["metadata"]["orig_elements"])
                # ...and then transpose it and other associated fields into a separate dictionary.
                orig_elements_dict.append({
                    "element_id": element["element_id"],
                    "text": element["text"],
                    "orig_elements": json.loads(orig_elements)
                })
        
        return orig_elements_dict

    if __name__ == "__main__":
        # Source file: https://www.fda.gov/files/drugs/published/Portable-Document-Format-Specifications.pdf
        input_filepath = "local-ingest-output-json/Portable-Document-Format-Specifications.json"
        output_filepath = "local-ingest-output-orig/Portable-Document-Format-Specifications-Orig-Elements-Only.json"

        orig_elements_dict = get_chunked_elements(input_json_file_path = input_filepath)

        # Convert the elements into a JSON object.
        orig_elements_json = json.dumps(orig_elements_dict, indent=2)

        # Write the JSON to a file.
        with open(output_filepath, "w") as file:
            file.write(orig_elements_json)
    ```
  </Accordion>

  <Accordion title="Python SDK">
    For the [Unstructured Python SDK](/api-reference/api-services/sdk-python), you'll need:

    These environment variables:

    *   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
    *   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

    ```python
    from unstructured_client import UnstructuredClient
    from unstructured_client.models import operations, shared
    from unstructured.staging.base import elements_from_dicts, elements_to_json

    import os, base64, zlib
    from typing import List, Dict, Any

    # Extract the contents of an orig_elements field.
    def extract_orig_elements(orig_elements):
        decoded_orig_elements = base64.b64decode(orig_elements)
        decompressed_orig_elements = zlib.decompress(decoded_orig_elements)
        return decompressed_orig_elements.decode('utf-8')

    # Source file: https://www.fda.gov/files/drugs/published/Portable-Document-Format-Specifications.pdf
    input_filepath = "local-ingest-input-pdf/Portable-Document-Format-Specifications.pdf"
    output_filepath = "local-ingest-output-json/Portable-Document-Format-Specifications.json"

    client = UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
        server_url=os.getenv("UNSTRUCTURED_API_URL"),
    )

    with open(input_filepath, "rb") as f:
        files = shared.Files(
            content=f.read(),
            file_name=input_filepath
        )

    # Chunk the document with a basic chunking strategy.
    # Create chunks that are at least around 200 characters (soft limit)
    # but never more than 300 characters (hard maximum).
    req = operations.PartitionRequest(
        shared.PartitionParameters(
            files=files,
            strategy=shared.Strategy.HI_RES,
            split_pdf_page=True,
            split_pdf_allow_failed=True,
            split_pdf_concurrency_level=15,
            chunking_strategy="basic",
            new_after_n_chars=200,
            max_characters=300
        )
    )

    try:
        res = await client.general.partition_async(request=req)

        # Create a dictionary that will hold only
        # a transposed version of the returned elements. 
        # For instance, we just want to capture each element's ID,
        # the chunk's text, and the chunk's associated elements in context.
        orig_elements_dict: List[Dict[str, Any]] = []

        for element in res.elements:
            # For each chunk that has an "orig_elements" field...
            if "orig_elements" in element["metadata"]:
                # ...get the chunk's associated elements in context...
                orig_elements = extract_orig_elements(element["metadata"]["orig_elements"])
                # ...and then transpose it and other associated fields into a separate dictionary.
                orig_elements_dict.append({
                    "element_id": element["element_id"],
                    "text": element["text"],
                    "orig_elements": json.loads(orig_elements)
                })
        
        # Convert the elements into a JSON object.
        orig_elements_json = json.dumps(orig_elements_dict, indent=2)

        # Write the JSON to a file.
        with open(output_filepath, "w") as file:
            file.write(orig_elements_json)
    except Exception as e:
        print(e)
    ```
  </Accordion>
</AccordionGroup>

## See also

*   [Recovering chunk elements](/open-source/core-functionality/chunking#recovering-chunk-elements)
*   [Chunking strategies](/api-reference/api-services/chunking)


# Get element contents



## Task

You want to get, manipulate, and print or save, the contents of the [document elements and metadata](/api-reference/api-services/document-elements) from the processed data that Unstructured returns.

## Approach

Each element in the document elements contains fields for that element's type, its ID, the extracted text, and associated metadata.

The programmatic approach you take to get these document elements will depend on which tool, SDK, or library you use:

<AccordionGroup>
  <Accordion title="Ingest CLI">
    For the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli), you can use a tool such as [jq](https://jqlang.github.io/jq/)
    to work with a JSON file that the CLI outputs after the processing is complete.

    For example, the following script uses `jq` to access and print each element's ID, text, and originating file name:

    ```bash Shell
    #!/usr/bin/env bash

    JSON_FILE="local-ingest-output/my-file.json"

    jq -r '.[] | "ID: \(.element_id)\nText: \(.text)\nFilename: \(.metadata.filename)\n"' \
        "$JSON_FILE"
    ```
  </Accordion>

  <Accordion title="Ingest Python library">
    For the [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library), you can use the standard Python
    [json.load](https://docs.python.org/3/library/json.html#json.load) function to load into a Python dictionary the contents of a JSON
    file that the Ingest Python library outputs after the processing is complete.

    For example, the following code example uses standard Python to access and print each element's ID, text, and originating file name:

    ```python Python
    import json

    def parse_json_file(input_file_path: str):
        with open(input_file_path, 'r') as file:
            file_elements = json.load(file)

        for element in file_elements:
            print(f"ID: {element["element_id"]}")
            print(f"Text: {element["text"]}")
            print(f"Filename: {element["metadata"]["filename"]}\n")

    if __name__ == "__main__":
        parse_json_file(
            input_file_path="local-ingest-output/my-file.json"
        )
    ```
  </Accordion>

  <Accordion title="Python SDK">
    For the [Unstructured Python SDK](/api-reference/api-services/sdk-python), calling an `UnstructuredClient` object's `general.partition_async` method returns a `PartitionResponse` object.

    This `PartitionResponse` object's `elements` variable contains a list of key-value dictionaries (`List[Dict[str, Any]]`). For example:

    ```python Python
    # ...

    res = await client.general.partition_async(request=req)

    # Do something with the elements, for example:
    save_elements_to_file(res.elements)

    # ...
    ```

    You can use standard Python [list operations](https://docs.python.org/3/tutorial/datastructures.html#more-on-lists) on this list.

    You can also use standard Python [looping techniques](https://docs.python.org/3/tutorial/datastructures.html#looping-techniques) on this list to access each element in this list.

    To work with an individual element's contents, you can use standard [dictionary operations](https://docs.python.org/3/library/stdtypes.html#dict) on the element.

    For example:

    ```python Python
    # ...

    res = await client.general.partition_async(request=req)

    for element in res.elements:
        # Do something with each element, for example:
        save_element_to_database(f"{element["element_id"]}")
        save_element_to_database(f"{element["text"]}")
        save_element_to_database(f"{element["metadata"]["filename"]}\n")

    # ...
    ```

    To serialize this list as JSON, you can:

    1.  Use the `elements_from_dicts` function to convert the list of key-value dictionaries (`Iterable[Dict[str, Any]]`) into a list of elements (`Iterable[Element]`).
    2.  Use the `elements_to_json` function to convert the list of elements into a JSON-formatted string and then print or save that string.

    For example:

    ```python Python
    from unstructured.staging.base import elements_from_dicts, elements_to_json

    # ...

    res = await client.general.partition_async(request=req)

    dict_elements = elements_from_dicts(
        element_dicts=res.elements
    )

    elements_to_json(
        elements=dict_elements,
        indent=2,
        filename=output_filepath
    )
    # ...
    ```
  </Accordion>

  <Accordion title="JavaScript/TypeScript SDK">
    For the [Unstructured JavaScript/TypeScript SDK](/api-reference/api-services/sdk-jsts), calling an `UnstructuredClient` object's `general.partition` method returns a `Promise<PartitionResponse>` object.

    This `PartitionResponse` object's `elements` property contains an `Array` of string-value objects (`{ [k: string]: any; }[]`). For example:

    ```typescript TypeScript
    // ...

    client.general.partition({
        partitionParameters: {
            files: {
                content: data,
                fileName: inputFilepath
            },
            strategy: Strategy.HiRes,
            splitPdfPage: true,
            splitPdfAllowFailed: true,
            splitPdfConcurrencyLevel: 15
        }
    }).then((res) => {
        if (res.statusCode == 200) {
            // Do something with the elements, for example:
            saveElementsToFile(res.elements)
        }
    } // ...
    ```

    You can use standard [Array operations](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array) on this array.

    You can also use standard `Array` techniques such as [forEach](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/forEach) to access each object in this array. For example:

    ```typescript TypeScript
    // ...

    client.general.partition({
        partitionParameters: {
            files: {
                content: data,
                fileName: inputFilepath
            },
            strategy: Strategy.HiRes,
            splitPdfPage: true,
            splitPdfAllowFailed: true,
            splitPdfConcurrencyLevel: 15
        }
    }).then((res) => {
        if (res.statusCode == 200) {
            res.elements?.forEach(element => {
                // Do something with each element, for example:
                saveElementToDatabase(`${element["element_id"]}`)
                saveElementToDatabase(`${element["text"]}`)
                saveElementToDatabase(`${element["metadata"]["filename"]}`)
            }
        }
    } // ...
    ```

    To serialize this list as JSON, you can use the standard [JSON.stringify](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify) function to serialize it to JSON-formatted string and the Node.js [fs.WriteFileSync](https://nodejs.org/api/fs.html#fswritefilesyncfile-data-options) function to save it as a file. For example:

    ```typescript TypeScript
    // ...

    client.general.partition({
        partitionParameters: {
            files: {
                content: data,
                fileName: inputFilepath
            },
            strategy: Strategy.HiRes,
            splitPdfPage: true,
            splitPdfAllowFailed: true,
            splitPdfConcurrencyLevel: 15
        }
    }).then((res) => {
        if (res.statusCode == 200) {    
            const jsonElements = JSON.stringify(res.elements, null, 2)

            fs.writeFileSync(outputFilepath, jsonElements)
        }
    } // ...
    ```
  </Accordion>

  <Accordion title="Open-source library">
    For the [Unstructured open-source library](/api-reference/api-services/partition-via-api), calling the `partition_via_api` function returns a list of elements (`list[Element]`). For example:

    ```python Python
    # ...

    elements = partition_via_api(
        filename=input_filepath,
        api_key=os.getenv("UNSTRUCTURED_API_KEY"),
        api_url=os.getenv("UNSTRUCTURED_API_URL"),
        strategy="hi_res"
    )

    # ...
    ```

    You can use standard Python [list operations](https://docs.python.org/3/tutorial/datastructures.html#more-on-lists) on this list.

    You can also use standard Python [looping techniques](https://docs.python.org/3/tutorial/datastructures.html#looping-techniques) on this list to access each element in this list.

    Each individual element has the following attributes:

    *   `.text` provides the element's `text` field value as a `str`. See [Element example](/open-source/concepts/document-elements#element-example).
    *   `.metadata` provides the element's `metadata` field as an `ElementMetadata` object. See [Metadata](/open-source/concepts/document-elements#metadata).
    *   `.category` provides the element's `type` field value as a `str`. See [Element type](/open-source/concepts/document-elements#element-type).
    *   `.id` provides the element's `element_id` value as a `str`. See [Element ID](/open-source/concepts/document-elements#element-id).

    In addition, the following methods are available:

    *   `.convert_coordinates_to_new_system()` converts the element's location coordinates, if any, to a new coordinate system. See [Element's coordinates](/open-source/concepts/document-elements#elements-coordinates).
    *   `.to_dict()` gets the element's content as a standard Python key-value dictionary (`dict[str, Any]`).

    For example:

    ```python Python
    # ...

    elements = partition_via_api(
        filename=input_filepath,
        api_key=os.getenv("UNSTRUCTURED_API_KEY"),
        api_url=os.getenv("UNSTRUCTURED_API_URL"),
        strategy="hi_res"
    )

    for element in elements:
        # Do something with each element, for example:
        save_element_to_database(f"{element.id}")
        save_element_to_database(f"{element.text}")
        save_element_to_database(f"{element.metadata.filename}")

    ```

    To serialize this list as a Python dictionary, you can use the `elements_to_dicts` method, for example:

    ```python Python
    from unstructured.staging.base import elements_to_dicts

    # ...

    elements = partition_via_api(
        filename=input_filepath,
        api_key=os.getenv("UNSTRUCTURED_API_KEY"),
        api_url=os.getenv("UNSTRUCTURED_API_URL"),
        strategy="hi_res"
    )

    elements_dicts = elements_to_dicts(elements)
    ```

    To serialize this list as JSON, you can use the `elements_to_json` function to convert the list of elements (`Iterable[Element]`) into a JSON-formatted string and then print or save that string. For example:

    ```python Python
    from unstructured.staging.base import elements_to_json

    # ...

    elements = partition_via_api(
        filename=input_filepath,
        api_key=os.getenv("UNSTRUCTURED_API_KEY"),
        api_url=os.getenv("UNSTRUCTURED_API_URL"),
        strategy="hi_res"
    )

    json_elements = elements_to_json(
        elements=elements,
        indent=2
    )

    elements_to_json(
        elements=elements,
        indent=2,
        filename=output_filepath
    )
    ```
  </Accordion>
</AccordionGroup>


# Output bounding box coordinates





# Output unique element IDs





# Parse simple PDFs and HTML





# Work with PowerPoint files





# Set document language for better OCR





# Extract tables as HTML



## Task

You want to get, save, or show the contents of elements that are represented as HTML, such as tables that are embedded in a PDF document.

## Approach

Extract the contents of an element's `text_as_html` JSON object, which is nested inside of its parent `metadata` object.

## To run this example

You will need a document that is one of the document types that can output the `text_as_html` JSON object. For the list of applicable document types, see the entries in the table at the beginning of [Partitioning](/open-source/core-functionality/partitioning) where "Table Support" is "Yes."

This example uses a PDF file with an embedded table.

## Code

<AccordionGroup>
  <Accordion title="Ingest Python library">
    For the [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library), you can use the standard Python
    [json.load](https://docs.python.org/3/library/json.html#json.load) function to load into a Python dictionary the contents of a JSON
    file that the Ingest Python library outputs after the processing is complete.

    ```python Python
    import json, os, webbrowser

    def get_tables_as_html(
            input_json_file_path: str,
            output_html_dir_path: str
        ):
        with open(input_json_file_path, 'r') as file:
            file_elements = json.load(file)

        # Provide some minimal CSS for better table readability.
        table_css = "<head><style>table, th, td { border: 1px solid; }</style></head>"

        for element in file_elements:
            if "text_as_html" in element["metadata"]:
                # Surround the element's HTML with basic <html> and <body> tags, and add the minimal CSS.
                html_string = f"<!DOCTYPE html><html>{table_css}<body>{element["metadata"]["text_as_html"]}</body></html>"

                # Save the element's HTML to a local file.
                save_path = f"{output_html_dir_path}/{element["element_id"]}.html"
                file = open(save_path, 'w')
                file.write(html_string)
                file.close()

                # View the locally saved file in the local default web browser.
                webbrowser.open_new(f"file:///{os.getcwd()}/{save_path}")

    if __name__ == "__main__":
        # Source: https://github.com/Unstructured-IO/unstructured-ingest/blob/main/example-docs/pdf/embedded-images-tables.pdf

        # Specify where to get the local file, relative to this .py file, and
        # where to store the retrieved HTML, relative to this .py file.
        get_tables_as_html(
            input_json_file_path="local-ingest-output/embedded-images-tables.json",
            output_html_dir_path="local-ingest-output/html/"
        )
    ```
  </Accordion>

  <Accordion title="Python SDK">
    For the [Unstructured Python SDK](/api-reference/api-services/sdk-python), you'll need:

    These environment variables:

    *   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
    *   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

    ```python Python SDK
    from unstructured_client import UnstructuredClient
    from unstructured_client.models import operations, shared
    from unstructured.staging.base import elements_from_dicts, elements_to_json

    import os, webbrowser

    if __name__ == "__main__":
        client = UnstructuredClient(
            api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
            server_url=os.getenv("UNSTRUCTURED_API_URL")
        )

        # Source: https://github.com/Unstructured-IO/unstructured-ingest/blob/main/example-docs/pdf/embedded-images-tables.pdf

        # Where to get the local file, relative to this .py file.
        local_input_filepath = "local-ingest-input-pdf/embedded-images-tables.pdf"

        # Where to store the retrieved HTML (and the processed JSON), relative to this .py file.
        local_output_filepath = "local-ingest-output"

        with open(local_input_filepath, "rb") as f:
            files = shared.Files(
                content=f.read(),
                file_name=local_input_filepath
            )

        request = operations.PartitionRequest(
            shared.PartitionParameters(
                files=files,
                strategy=shared.Strategy.HI_RES,
                split_pdf_page=True,
                split_pdf_allow_failed=True,
                split_pdf_concurrency_level=15
            )
        )

        try:
            result = await client.general.partition_async(request)

            # Provide some minimal CSS for better table readability.
            table_css = "<head><style>table, th, td { border: 1px solid; }</style></head>"

            for element in result.elements:
                if "text_as_html" in element["metadata"]:
                    # Surround the element's HTML with basic <html> and <body> tags, and add the minimal CSS.
                    html_string = f"<!DOCTYPE html><html>{table_css}<body>{element["metadata"]["text_as_html"]}</body></html>"

                    # Save the element's HTML to a local file.
                    save_path = f"{local_output_path}/{element["element_id"]}.html"
                    file = open(save_path, 'w')
                    file.write(html_string)
                    file.close()

                    # View the locally saved file in the local default web browser.
                    webbrowser.open_new(f"file:///{os.getcwd()}/{save_path}")

            # Also get the elements for inspection and validation.
            dict_elements = elements_from_dicts(
                element_dicts=result.elements
            )

            # Save the elements as JSON.
            elements_to_json(
                elements=dict_elements,
                indent=2,
                filename=f"{local_output_filepath}/embedded-images-tables.json"
            )
        except Exception as e:
            print(e)
    ```
  </Accordion>
</AccordionGroup>

## See also

*   [Extract images and tables from documents](/api-reference/how-to/extract-image-block-types)
*   [Table Extraction from PDF](/examplecode/codesamples/apioss/table-extraction-from-pdf)


# Transform a JSON file into a different schema



## Task

You want to convert a JSON file that Unstructured produces into a separate JSON file that uses
a different JSON schema than the one that Unstructured uses.

## Approach

Use a Python package such as [json-converter](https://pypi.org/project/json-converter/) in your Python code project
to transform your source JSON file into a target JSON file that conforms to your own schema.

<Info>The `json-converter` package is not owned or supported by Unstructured. For questions and
requests, see the [Issues](https://github.com/ebi-ait/json-converter/issues) tab of the
`json-converter` repository in GitHub.</Info>

## Code

<Steps>
  <Step title="Install dependencies">
    In your local Python code project, install the [json-converter](https://pypi.org/project/json-converter/)
    package.

    ```bash
    pip install json-converter
    ```
  </Step>

  <Step title="Identify the JSON file to transform">
    1.  Find the local source JSON file that you want to transform.
    2.  Note the JSON field names and structures that you want to transform. For example, the JSON file might
        look like the following (the ellipses indicate content omitted for brevity):

        ```json
        [
          {
            "type": "...",
            "element_id": "...",
            "text": "...",
            "metadata": {
              "filetype": "...",
              "languages": [
                "eng"
              ],
              "page_number": 1,
              "filename": "..."
            }
          },
          {
            "type": "...",
            "element_id": "...",
            "text": "...",
            "metadata": {
              "filetype": "...",
              "languages": [
                "eng"
              ],
              "page_number": 1,
              "filename": "..."
            }
          },
          {
            "...": "..."
          }
        ]
        ```
  </Step>

  <Step title="Create the JSON field mappings file">
    1.  Decide what you want the JSON schema in the transformed file to look like. For example, the
        transformed JSON file might look like the following (the ellipses indicate content omitted for brevity):

        ```json
        [
          {
            "content_type": "...",
            "content_id": "...",
            "content": "...",
            "content_properties": {
              "page": 1
            }
          },
          {
            "content_type": "...",
            "content_id": "...",
            "content": "...",
            "content_properties": {
              "page": 1
            }
          },
          {
            "...": "..."
          }
        ]
        ```

    2.  Create the JSON field mappings file, for example:

        ```json
        {
          "content_type": ["type"],
          "content_id": ["element_id"],
          "content": ["text"]
          "content_properties.page": ["metadata.page_number"]
        }
        ```

        This file declares the following mappings:

        *   The `type` field is renamed to `content_type`.
        *   The `element_id` field is renamed to `content_id`.
        *   The `text` field is renamed to `content`.
        *   The `page_number` field nested inside `metadata` is renamed to `page` and is nested inside `content_properties`.
        *   All of the other fields (`filetype`, `languages`, and `filename`) are dropped.

        For more information about the format of this JSON field mappings file, see the
        [Project Description](https://pypi.org/project/json-converter) in the `json-converter` page on PyPI or the
        [README](https://github.com/ebi-ait/json-converter) in the `json-converter` repository in GitHub.
  </Step>

  <Step title="Add and run the transform code">
    1.  Set the following local environment variables:

        *   Set `LOCAL_FILE_INPUT_PATH` to the local path to the source JSON file.
        *   Set `LOCAL_FILE_OUTPUT_PATH` to the local path to the target JSON file.
        *   Set `LOCAL_FIELD_MAPPINGS_PATH` to the local path to the JSON field mappings file.

    2.  Add the following Python code file to your project:

        ```python
        import os, json
        from json_converter.json_mapper import JsonMapper

        # Converts one JSON file with one schema to another 
        # JSON file with a different schema. 
        # Provide the path to the input (source) and output 
        # (target) JSON files and the path to the JSON schema 
        # field mappings file.
        def convert_json_with_schemas(
            input_file_path: str,
            output_file_path: str,
            field_mappings_path: str
        ) -> None:

            output_data = []

            try:
                with open(field_mappings_path, 'r') as f:
                    element_mappings = json.load(f)
            except FileNotFoundError:
                print(f"Error: Input JSON schema field mappings file '{input_file_path}' not found.")
            except IOError as e:
                print(f"I/O error occurred: {e}")
            except Exception as e:
                print(f"An unexpected error occurred: {e}")

            try:
                with open(input_file_path, 'r') as f:
                    input_data = json.load(f)
                    
                    for item in input_data:
                        converted_data = JsonMapper(item).map(element_mappings)
                        output_data.append(converted_data)
            except FileNotFoundError:
                print(f"Error: Input JSON file '{input_file_path}' not found.")
            except IOError as e:
                print(f"I/O error occurred: {e}")
            except Exception as e:
                print(f"An unexpected error occurred: {e}")

            try:
                with open(output_file_path, 'w') as f:
                    json.dump(output_data, f, indent=2)
            
                print(f"Transformation complete. Output written to '{output_file_path}'.")
            except IOError as e:
                print(f"I/O error occurred: {e}")
            except Exception as e:
                print(f"An unexpected error occurred: {e}")

        if __name__ == "__main__":
            convert_json_with_schemas(
                input_file_path=os.getenv("LOCAL_FILE_INPUT_PATH"),
                output_file_path=os.getenv("LOCAL_FIELD_MAPPINGS_PATH"),
                field_mappings_path=os.getenv("LOCAL_FILE_OUTPUT_PATH")
            )
        ```

    3.  Run the Python code file.

    4.  Check the path specified by `LOCAL_FILE_OUTPUT_PATH` for the transformed JSON file.
  </Step>
</Steps>

## Troubleshooting

### Error when trying to import Mapping from collections

**Issue**: When you run your Python code file, the following error message appears: "ImportError: cannot import name 'Mapping' from 'collections'".

**Cause**: When you use the `json-converter` package with newer versions of Python such as 3.11 and later,
Python tries to use an outdated import in this `json-converter` package.

**Solution**: Update the `json-converter` package's source code to use a different import, as follows:

1.  In your Python project, find the `json-converter` package's source location, by running the `pip show` command:

    ```bash
    pip show json-converter
    ```

    Note the path in the **Location** field.

2.  Use your code editor to the open the path to the `json-converter` package's source code.

3.  In the source code, open the file named `json_mapper.py`.

4.  Change the following line of code...

    ```python
    from collections import Mapping
    ```

    ...to the following line of code, by adding `.abc`:

    ```python
    from collections.abc import Mapping
    ```

5.  Save this source code file.

6.  Run your Python code file again.


# Use LangChain and Llama 3





# Use LangChain and Ollama





# Astra DB



Batch process all your records to store structured outputs in an Astra DB account.

You will need:

The Astra DB connector prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/kupQrnCQjPw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An Astra account. [Create or sign in to an Astra account](https://astra.datastax.com/).
*   A database in the Astra account. [Create a database in an account](https://docs.datastax.com/en/astra-db-classic/databases/manage-create.html).
*   An application token for the database. [Create a database application token](https://docs.datastax.com/en/astra-db-serverless/administration/manage-application-tokens.html).
*   A namespace in the database. [Create a namespace in a database](https://docs.datastax.com/en/astra-db-serverless/databases/manage-namespaces.html#create-namespace).
*   A collection in the namespace. [Create a collection in a namespace](https://docs.datastax.com/en/astra-db-serverless/databases/manage-collections.html#create-collection).

The Astra DB connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[astradb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `ASTRA_DB_API_ENDPOINT` - The API endpoint for the Astra DB database, represented by `--api-endpoint` (CLI) or `api_endpoint` (Python). To get the endpoint, see the **Database Details > API Endpoint** value on your database's **Overview** tab.
*   `ASTRA_DB_APPLICATION_TOKEN` - The database application token value for the database, represented by `--token` (CLI) or `token` (Python). To get the token, see the **Database Details > Application Tokens** box on your database's **Overview** tab.
*   `ASTRA_DB_KEYSPACE` - The name of the keyspace for the database, represented by `--keyspace` (CLI) or `keyspace` (Python).
*   `ASTRA_DB_COLLECTION` - The name of the collection for the keyspace, represented by `--collection-name` (CLI) or `collection_name` (Python).
*   `ASTRA_DB_EMBEDDING_DIMENSIONS` - The number of dimensions in the collection, represented by `--embedding-dimension` (CLI) or `embedding_dimension` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    astradb \
      --api-endpoint $ASTRA_DB_API_ENDPOINT \
      --token $ASTRA_DB_APPLICATION_TOKEN \
      --keyspace $ASTRA_DB_KEYSPACE \
      --collection-name $ASTRA_DB_COLLECTION \
      --embedding-dimension $ASTRA_DB_EMBEDDING_DIMENSIONS
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.astradb import (
      AstraDBConnectionConfig,
      AstraDBAccessConfig,
      AstraDBUploadStagerConfig,
      AstraDBUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=AstraDBConnectionConfig(
              access_config=AstraDBAccessConfig(
                  api_endpoint=os.getenv("ASTRA_DB_API_ENDPOINT"),
                  token=os.getenv("ASTRA_DB_APPLICATION_TOKEN")
              )
          ),
          stager_config=AstraDBUploadStagerConfig(),
          uploader_config=AstraDBUploaderConfig(
              keyspace=os.getenv("ASTRA_DB_KEYSPACE"),
              collection_name=os.getenv("ASTRA_DB_COLLECTION"),
              embedding_dimension=os.getenv("ASTRA_DB_EMBEDDING_DIMENSIONS")
          )
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.astradb import (
      AstraDBAccessConfig,
      AstraDBWriteConfig,
      SimpleAstraDBConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.astradb import (
      AstraDBWriter,
  )
  from unstructured_ingest.runner.writers.base_writer import Writer


  def get_writer() -> Writer:
      return AstraDBWriter(
          connector_config=SimpleAstraDBConfig(
              access_config=AstraDBAccessConfig(
                  api_endpoint=os.getenv("ASTRA_DB_API_ENDPOINT"),
                  token=os.getenv("ASTRA_DB_APPLICATION_TOKEN"),
              ),
              keyspace=os.getenv("ASTRA_DB_KEYSPACE"),
              collection_name=os.getenv("ASTRA_DB_COLLECTION"),
              embedding_dimension=os.getenv("ASTRA_DB_EMBEDDING_DIMENSIONS"),
          ),
          write_config=AstraWriteConfig(batch_size=80),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>


# Azure



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in an Azure Storage account.

You will need:

The Azure Storage account prerequisites.

The following video shows how to fulfill the minimum set of Azure Storage account prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/Vl3KCphlh9Y" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

<Note>
  If you are generating an SAS token as shown in the preceding video, be sure to set the following permissions:

  *   **Read** and **List** for reading from the container only.
  *   **Write** and **List** for writing to the container only.
  *   **Read**, **Write**, and **List** for both reading from and writing to the container.
</Note>

Here are some more details about these prerequisites:

*   An Azure account. To create one, [learn how](https://azure.microsoft.com/pricing/purchase-options/azure-account).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/2bQ6WiJ1ncA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An Azure Storage account, and a container within that account. [Create a storage account](https://learn.microsoft.com/azure/storage/common/storage-account-create). [Create a container](https://learn.microsoft.com/azure/storage/blobs/blob-containers-portal).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/AhuNgBafmUo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    <iframe width="560" height="315" src="https://www.youtube.com/embed/xmndjYnGvcs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Azure Storage remote URL, using the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

    For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
    Azure Storage remote URL would be `az://my-container/my-folder/`.

*   An SAS token (recommended), access key, or connection string for the Azure Storage account.  [Create an SAS token (recommended)](https://learn.microsoft.com/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens). [Get an access key](https://learn.microsoft.com/azure/storage/common/storage-account-keys-manage#view-account-access-keys). [Get a connection string](https://learn.microsoft.com/azure/storage/common/storage-configure-connection-string#configure-a-connection-string-for-an-azure-storage-account).

    Create an SAS token (recommended):

    <iframe width="560" height="315" src="https://www.youtube.com/embed/X6cmJ2IbVzo?start=240&end=370" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    Get an access key or connection string:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/muMmcwVfFqs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The Azure Storage account connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[azure]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `AZURE_STORAGE_REMOTE_URL` - The Azure Storage remote URL, represented by `--remote-url` (CLI) or `remote_url` (Python).

    The remote URL takes the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

    For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
    Azure Storage remote URL would be `az://my-container/my-folder/`.

*   `AZURE_STORAGE_ACCOUNT_NAME` - The name of the Azure Storage account, represented by `--account-name` (CLI) or `account_name` (Python).

*   `AZURE_STORAGE_ACCOUNT_KEY`, `AZURE_STORAGE_CONNECTION_STRING`, or `AZURE_STORAGE_SAS_TOKEN` - The name of the key, connection string, or SAS token for the Azure Storage account, depending on your security configuration, represented by `--account-key` (CLI) or `account_key` (Python), `--connection-string` (CLI) or `connection_string` (Python), and `--sas_token` (CLI) or `sas_token` (Python), respectively.

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    azure \
      --remote-url $AZURE_STORAGE_REMOTE_URL \
      --account-name $AZURE_STORAGE_ACCOUNT_NAME \
      --account-key $AZURE_STORAGE_ACCOUNT_KEY
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.azure import (
      AzureConnectionConfig,
      AzureAccessConfig,
      AzureUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=AzureConnectionConfig(
              access_config=AzureAccessConfig(
                  account_name=os.getenv("AZURE_STORAGE_ACCOUNT_NAME"),
                  account_key=os.getenv("AZURE_STORAGE_ACCOUNT_KEY")
              )
          ),
          uploader_config=AzureUploaderConfig(remote_url=os.getenv("AZURE_STORAGE_REMOTE_URL"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.azure import (
      AzureAccessConfig,
      AzureWriteConfig,
      SimpleAzureBlobStorageConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.fsspec.azure import (
      AzureWriter,
  )

  def get_writer() -> Writer:
      return AzureWriter(
          connector_config=SimpleAzureBlobStorageConfig(
              access_config=AzureAccessConfig(account_name=os.getenv("AZURE_STORAGE_ACCOUNT_NAME")),
              remote_url=os.getenv("AZURE_STORAGE_REMOTE_URL"),
          ),
          write_config=AzureWriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res"
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>


# Azure AI Search



Batch process all your records to store structured outputs in an Azure AI Search account.

You will need:

The Azure AI Search prerequisites:

The following video shows how to fulfill the minimum set of Azure AI Search prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/6ZjU5OupWE8" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

Here are some more details about these prerequisites:

*   The endpoint and API key for Azure AI Search. [Create an endpoint and API key](https://learn.microsoft.com/azure/search/search-create-service-portal).
*   The name of the index in Azure AI Search. [Create an index](https://learn.microsoft.com/rest/api/searchservice/create-index).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/WY8h8Gtyo7o" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    The Azure AI Search index that you use must have an index schema that is compatible with the schema of the documents
    that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following index schema example for your own needs:

    ```json
    {
      "@odata.context": "https://ingest-test-azure-ai-search.search.windows.net/$metadata#indexes/$entity",
      "@odata.etag": "\"0x8DCED5D96393CA9\"",
      "name": "<my-index-name>",
      "defaultScoringProfile": null,
      "fields": [
        {
          "name": "id",
          "type": "Edm.String",
          "searchable": true,
          "filterable": true,
          "retrievable": true,
          "stored": true,
          "sortable": true,
          "facetable": true,
          "key": true,
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "analyzer": null,
          "normalizer": null,
          "dimensions": null,
          "vectorSearchProfile": null,
          "vectorEncoding": null,
          "synonymMaps": []
        },
        {
          "name": "element_id",
          "type": "Edm.String",
          "searchable": true,
          "filterable": true,
          "retrievable": true,
          "stored": true,
          "sortable": true,
          "facetable": true,
          "key": false,
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "analyzer": null,
          "normalizer": null,
          "dimensions": null,
          "vectorSearchProfile": null,
          "vectorEncoding": null,
          "synonymMaps": []
        },
        {
          "name": "text",
          "type": "Edm.String",
          "searchable": true,
          "filterable": true,
          "retrievable": true,
          "stored": true,
          "sortable": true,
          "facetable": true,
          "key": false,
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "analyzer": null,
          "normalizer": null,
          "dimensions": null,
          "vectorSearchProfile": null,
          "vectorEncoding": null,
          "synonymMaps": []
        },
        {
          "name": "embeddings",
          "type": "Collection(Edm.Single)",
          "searchable": true,
          "filterable": false,
          "retrievable": true,
          "stored": true,
          "sortable": false,
          "facetable": false,
          "key": false,
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "analyzer": null,
          "normalizer": null,
          "dimensions": 3072,
          "vectorSearchProfile": "embeddings-config-profile",
          "vectorEncoding": null,
          "synonymMaps": []
        },
        {
          "name": "type",
          "type": "Edm.String",
          "searchable": true,
          "filterable": true,
          "retrievable": true,
          "stored": true,
          "sortable": true,
          "facetable": true,
          "key": false,
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "analyzer": null,
          "normalizer": null,
          "dimensions": null,
          "vectorSearchProfile": null,
          "vectorEncoding": null,
          "synonymMaps": []
        },
        {
          "name": "metadata",
          "type": "Edm.ComplexType",
          "fields": [
            {
              "name": "category_depth",
              "type": "Edm.Int32",
              "searchable": false,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "parent_id",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "attached_to_filename",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "filetype",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "last_modified",
              "type": "Edm.DateTimeOffset",
              "searchable": false,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "is_continuation",
              "type": "Edm.Boolean",
              "searchable": false,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "file_directory",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "filename",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "data_source",
              "type": "Edm.ComplexType",
              "fields": [
                {
                  "name": "url",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "version",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "date_created",
                  "type": "Edm.DateTimeOffset",
                  "searchable": false,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "date_modified",
                  "type": "Edm.DateTimeOffset",
                  "searchable": false,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "date_processed",
                  "type": "Edm.DateTimeOffset",
                  "searchable": false,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "permissions_data",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "record_locator",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                }
              ]
            },
            {
              "name": "coordinates",
              "type": "Edm.ComplexType",
              "fields": [
                {
                  "name": "system",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "layout_width",
                  "type": "Edm.Double",
                  "searchable": false,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "layout_height",
                  "type": "Edm.Double",
                  "searchable": false,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "points",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                }
              ]
            },
            {
              "name": "languages",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "page_number",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "orig_elements",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "links",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "page_name",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "url",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "link_urls",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "link_texts",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "sent_from",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "sent_to",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "subject",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "section",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "header_footer_type",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "emphasized_text_contents",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "emphasized_text_tags",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "text_as_html",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "regex_metadata",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "detection_class_prob",
              "type": "Edm.Double",
              "searchable": false,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            }
          ]
        }
      ],
      "scoringProfiles": [],
      "corsOptions": null,
      "suggesters": [],
      "analyzers": [],
      "normalizers": [],
      "tokenizers": [],
      "tokenFilters": [],
      "charFilters": [],
      "encryptionKey": null,
      "similarity": {
        "@odata.type": "#Microsoft.Azure.Search.BM25Similarity",
        "k1": null,
        "b": null
      },
      "semantic": null,
      "vectorSearch": {
        "algorithms": [
          {
            "name": "embeddings-config",
            "kind": "hnsw",
            "hnswParameters": {
              "metric": "cosine",
              "m": 4,
              "efConstruction": 400,
              "efSearch": 500
            },
            "exhaustiveKnnParameters": null
          }
        ],
        "profiles": [
          {
            "name": "embeddings-config-profile",
            "algorithm": "embeddings-config",
            "vectorizer": null,
            "compression": null
          }
        ],
        "vectorizers": [],
        "compressions": []
      }
    }
    ```

    See also:

    *   [Search indexes in Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-an-index)
    *   [Schema of a search index](https://learn.microsoft.com/azure/search/search-what-is-an-index#schema-of-a-search-index)
    *   [Example index schema](https://learn.microsoft.com/rest/api/searchservice/create-index#examples)
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

The Azure AI Search connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[azure-ai-search]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `AZURE_SEARCH_ENDPOINT` - The endpoint URL for Azure AI Search, represented by `--endpoint` (CLI) or `endpoint` (Python).
*   `AZURE_SEARCH_API_KEY` - The API key for Azure AI Search, represented by `--key` (CLI) or `key` (Python).
*   `AZURE_SEARCH_INDEX` - The name of the index for Azure AI Search, represented by `--index` (CLI) or `index` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    azure-ai-search \
      --key $AZURE_SEARCH_API_KEY \
      --endpoint $AZURE_SEARCH_ENDPOINT \
      --index $AZURE_SEARCH_INDEX
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.azure_ai_search import (
      AzureAISearchAccessConfig,
      AzureAISearchConnectionConfig,
      AzureAISearchUploadStagerConfig,
      AzureAISearchUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=AzureAISearchConnectionConfig(
              access_config=AzureAISearchAccessConfig(
                  key=os.getenv("AZURE_SEARCH_API_KEY")
              ),
              endpoint=os.getenv("AZURE_SEARCH_ENDPOINT"),
              index=os.getenv("AZURE_SEARCH_INDEX")
          ),
          stager_config=AzureAISearchUploadStagerConfig(),
          uploader_config=AzureAISearchUploaderConfig()
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.azure_ai_search import (
      AzureAISearchAccessConfig,
      AzureAISearchWriteConfig,
      SimpleAzureAISearchStorageConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.azure_ai_search import (
      AzureAISearchWriter,
  )
  from unstructured_ingest.runner.writers.base_writer import Writer


  def get_writer() -> Writer:
      return AzureAISearchWriter(
          connector_config=SimpleAzureAISearchStorageConfig(
              access_config=AzureAISearchAccessConfig(key=os.getenv("AZURE_SEARCH_API_KEY")),
              endpoint=os.getenv("AZURE_SEARCH_ENDPOINT"),
          ),
          write_config=AzureAISearchWriteConfig(index=os.getenv("AZURE_SEARCH_INDEX")),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR")",
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>


# Box



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a Box account.

You will need:

The Box prerequisites:

1.  Access to the [Developer Console](https://app.box.com/developers/console) from your [Box enterprise account](https://account.box.com/signup/enterprise-plan) or [Box developer account](https://account.box.com/signup/developer).

2.  A Box Custom App in your Box account, set up to use **Server Authentication (with JWT)**. See [Setup with JWT](https://developer.box.com/guides/authentication/jwt/jwt-setup/).

3.  The appropriate application scopes and advanced features set up for the Box Custom App, as follows:

    *   In the Box Custom App, on the **Configuration** tab, under **Application Scopes**, check the box titled **Write all files and folders stored in Box**.
    *   In the Box Custom App, on the **Configuration** tab, under **Advanced Features**, check the box titled **Make API calls using the as-user header**.
    *   Click **Save Changes** before continuing.

4.  Authorization by a Box Admin in your Box account for the Box Custom App. See [App Authorization](https://developer.box.com/guides/authentication/jwt/jwt-setup/#app-authorization).

5.  Access by your Box account's source or target [folder](https://app.box.com/folder/0) to your Box Custom App, as follows:

    *   In the Box Custom App, on the **General Settings** tab, copy the **Service Account ID** (which takes the form `AutomationUser_<your-app-service-id>_<a-random-string@boxdevedition.com`).
    *   **Share** your Box account's target folder with the copied service account's email address as a **Co-owner** or **Editor**.
    *   Note the remote URL to the target folder, which takes the format `box://<path/to/folder/in/account>`.

6.  The private key configuration JSON file for the Box Custom App. To download this file, in the Box Custom App, on the **Configuration** tab, under **Add and Manage Public Keys**, click **Generate a Public/Private Keypair**. Store the downloaded private key configuration JSON file in a secure location. Do not share it with others.

The Box connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[box]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `BOX_APP_CONFIG_PATH` - The local path to the downloaded private key configuration JSON file for the Box Custom App, represented by `--box-app-config` (CLI) or `box_app_config` (Python).
*   `BOX_REMOTE_URL` - The remote URL to the target folder, represented by `--remote-url` (CLI) or `remote_url` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    box \
      --box-app-config $BOX_APP_CONFIG_PATH \
      --remote-url $BOX_REMOTE_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.box import (
      BoxAccessConfig,
      BoxConnectionConfig,
      BoxIndexerConfig,
      BoxUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=BoxConnectionConfig(
              access_config=BoxAccessConfig(
                  box_app_config=os.getenv("BOX_APP_CONFIG_PATH")
              )
          ),
          uploader_config=BoxUploaderConfig(
              remote_url=os.getenv("BOX_REMOTE_URL")
          )
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.box import (
      BoxAccessConfig,
      BoxWriteConfig,
      SimpleBoxConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.fsspec.box import (
      BoxWriter,
  )


  def get_writer() -> Writer:
      return BoxWriter(
          connector_config=SimpleBoxConfig(
              access_config=BoxAccessConfig(box_app_config=os.getenv("BOX_APP_CONFIG_PATH")),
              remote_url=os.getenv("BOX_REMOTE_URL"),
          ),
          write_config=BoxWriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>


# Chroma



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a Chroma account.

You will need:

The Chroma prerequisites:

*   A Chroma server. See [Deployment](https://docs.trychroma.com/deployment).

    For example, here is a video about how to deploy a Chroma server to AWS:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/4ApW6ZwqW20" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Chroma server's hostname or IP address, and the server's port number.

*   If you are not connecting to the server through HTTP, the path to the server instance.

*   The name of the tenant that you want to access on the server.

*   The name of the database that you want to access in the tenant.

*   The name of the collection that you want to access in the database.

The Chroma connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[chroma]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `CHROMA_HOST` - The , represented by `--host` (CLI) or `host` (Python).
*   `CHROMA_PORT` - The , represented by `--port` (CLI) or `port` (Python).
*   `CHROMA_TENANT` - The name of the tenant that you want to access on the Chroma server, represented by `--tenant` (CLI) or `tenant` (Python).
*   `CHROMA_DATABASE` - The name of the database that you want to access in the tenant, represented by `--database` (CLI) or `database` (Python).
*   `CHROMA_COLLECTION` - The name of the collection that you want to access in the database, represented by `--collection-name` (CLI) or `collection_name` (Python).

Additional settings include:

*   `--path` (CLI) or `path` (Python): The location where Chroma is persisted if you are not connecting through HTTP.
*   `--settings` (CLI) or `settings` (Python): A dictionary of settings to communicate with the Chroma server, for example: `'{"persist_directory":"./chroma-persist"}'`.
*   `--headers` (CLI) or `headers` (Python): A dictionary of headers to send to the Chroma server, for example: `'{"Authorization":"Basic()"}'`.
*   `--ssl` (CLI) or `ssl` (Python): True to use SSL for the connection.

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --work-dir $WORK_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    chroma \
      --host $CHROMA_HOST \
      --port $CHROMA_PORT \
      --collection-name $CHROMA_COLLECTION \
      --tenant $CHROMA_TENANT \
      --database $CHROMA_DATABASE \
      --settings "{\"persist_directory\":\"./chroma-persist\"}" \
      --headers "{\"Authorization\":\"Basic()\"}" \
      --batch-size 80 \
      --ssl
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.chroma import (
      ChromaAccessConfig,
      ChromaConnectionConfig,
      ChromaUploadStagerConfig,
      ChromaUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=ChromaConnectionConfig(
              access_config=ChromaAccessConfig(
                  settings={"persist_directory":"./chroma-persist"},
                  headers={"Authorization":"Basic()"}
              ),
              host=os.getenv("CHROMA_HOST"),
              port=os.getenv("CHROMA_PORT"),
              tenant=os.getenv("CHROMA_TENANT"),
              database=os.getenv("CHROMA_DATABASE"),
              collection_name=os.getenv("CHROMA_COLLECTION")
          ),
          stager_config=ChromaUploadStagerConfig(),
          uploader_config=ChromaUploaderConfig()
      ).run()

  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.chroma import (
      ChromaAccessConfig,
      ChromaWriteConfig,
      SimpleChromaConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.chroma import (
      ChromaWriter,
  )


  def get_writer() -> Writer:
      return ChromaWriter(
          connector_config=SimpleChromaConfig(
              access_config=ChromaAccessConfig(),
              host=os.getenv("CHROMA_HOST"),
              port=os.getenv("CHROMA_PORT"),
              collection_name=os.getenv("CHROMA_COLLECTION"),
              tenant=os.getenv("CHROMA_TENANT"),
              database=os.getenv("CHROMA_DATABASE"),
          ),
          write_config=ChromaWriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>


# Clarifai



Batch process all your records using `unstructured-ingest` to store unstructured outputs locally on your filesystem and upload those to Clarifai apps.

First start with the installation of clarifai dependencies as shown here.

```bash
pip install "unstructured-ingest[clarifai]"
```

Create a clarifai app with base workflow. Find more information in the [create clarifai app](https://docs.clarifai.com/clarifai-basics/applications/create-an-application/).

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  # Chunking is optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --num-processes 2 \
      --verbose \
    clarifai \
      --app-id $CLARIFAI_APP_ID \
      --user-id $CLARIFAI_USER_ID \
      --api-key $CLARIFAI_PAT_KEY \
      --batch-size 100
  ```

  ```python Python
  import os

  from unstructured_ingest.connector.clarifai import (
      ClarifaiAccessConfig,
      ClarifaiWriteConfig,
      SimpleClarifaiConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.clarifai import (
      ClarifaiWriter,
  )


  def get_writer() -> Writer:
      return ClarifaiWriter(
          connector_config=SimpleClarifaiConfig(
              access_config=ClarifaiAccessConfig(api_key=os.getenv("CLARIFAI_PAT_KEY")),
              app_id=os.getenv("CLARIFAI_APP_ID"),
              user_id=os.getenv("CLARIFAI_USER_ID"),
          ),
          write_config=ClarifaiWriteConfig(),
      )

  # Chunking is optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

The upstream connector can be any of the ones supported, but for the convenience here, showing a sample command using the upstream local connector.

For a full list of the options the Unstructured Ingest CLI accepts check `unstructured-ingest <upstream connector> clarifai --help`.

NOTE: Keep in mind that you will need to have all the appropriate extras and dependencies for the file types of the documents contained in your data storage platform if you’re running this locally. You can find more information about this in the [installation guide](/open-source/installation/overview).


# Couchbase



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

[Couchbase](https://couchbase.com) is a Distributed NoSQL Cloud Database. Couchbase embraces AI with coding assistance for developers, and vector search for their applications.

Batch process all your records to store structured outputs in a Couchbase database.

You will need:

The Couchbase database prerequisites.

*   For the [Unstructured Platform](/platform/overview), only Couchbase Capella clusters are supported.
*   For [Unstructured Ingest](/ingestion/overview), Couchbase Capella clusters and local Couchbase server deployments are supported.

<iframe width="560" height="315" src="https://www.youtube.com/embed/9-RIBmIdi70" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

For Couchbase Capella, you will need:

*   A Couchbase Capella account. [Create an account](https://docs.couchbase.com/cloud/get-started/create-account.html).
*   A Couchbase Capella cluster. [Create a cluster](https://docs.couchbase.com/cloud/clusters/create-database.html).
*   A [bucket, scope, and collection](https://docs.couchbase.com/cloud/clusters/data-service/about-buckets-scopes-collections.html).
*   A cluster connection string. [Learn how](https://docs.couchbase.com/cloud/get-started/connect.html).
*   Incoming IP address allowance. [Learn how](https://docs.couchbase.com/cloud/clusters/allow-ip-address.html).
*   A username and password. [Learn how](https://docs.couchbase.com/cloud/clusters/manage-database-users.html).

For a local Couchbase server, you will need:

*   Installation of a local Couchbase server. [Learn how](https://docs.couchbase.com/server/current/getting-started/start-here.html).
*   Connection details to the local Couchbase server. [Learn how](https://docs.couchbase.com/server/current/guides/connect.html).

To learn more about how to set up a Couchbase cluster and play with data, refer to this [tutorial](https://developer.couchbase.com/tutorial-quickstart-flask-python).

The Couchbase DB connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[couchbase]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables are required for the Couchbase Connector:

*   `CB_CONN_STR` - The Connection String for the Couchbase server, represented by `--connection-string` (CLI) or `connection_string` (Python).
*   `CB_USERNAME` - The username for the Couchbase server, represented by `--username` (CLI) or `username` (Python).
*   `CB_PASSWORD` - The password for the Couchbase server, represented by `--password` (CLI) or `password` (Python).
*   `CB_BUCKET` - The name of the bucket in the Couchbase server, represented by `--bucket` (CLI) or `bucket` (Python).
*   `CB_SCOPE` - The name of the scope in the bucket, represented by `--scope` (CLI) or `scope` (Python).
*   `CB_COLLECTION` - The name of the collection in the scope, represented by `--collection` (CLI) or `collection` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    couchbase \
      --connection-string $CB_CONN_STR \
      --username $CB_USERNAME \
      --password $CB_PASSWORD \
      --bucket $CB_BUCKET \
      --scope $CB_SCOPE \
      --collection $CB_COLLECTION \
      --num-processes 2 \
      --batch-size 80
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.couchbase import (
      CouchbaseAccessConfig,
      CouchbaseConnectionConfig,
      CouchbaseUploadStagerConfig,
      CouchbaseUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=CouchbaseConnectionConfig(
              access_config=CouchbaseAccessConfig(
                  password=os.getenv("CB_PASSWORD"),
              ),
              connection_string=os.getenv("CB_CONN_STR"),
              username=os.getenv("CB_USERNAME"),
              bucket=os.getenv("CB_BUCKET"),
              scope=os.getenv("CB_SCOPE"),
              collection=os.getenv("CB_COLLECTION")
          ),
          stager_config=CouchbaseUploadStagerConfig(),
          uploader_config=CouchbaseUploaderConfig(batch_size=100)
      ).run()
  ```
</CodeGroup>

To understand how [vector search](https://www.couchbase.com/products/vector-search/) works in Couchbase, refer to this [tutorial](https://developer.couchbase.com/tutorial-python-langchain-pdf-chat) and the [Couchbase docs](https://docs.couchbase.com/cloud/vector-search/vector-search.html).


# Databricks Volumes



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in Databricks Volumes.

You will need:

The Databricks Volumes prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/rNZpwa1-g7M" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The preceding video shows how to use Databricks personal access tokens (PATs), which are supported only for [Unstructured Ingest](/ingestion/overview).

To learn how to use Databricks-managed service principals, which are supported by both the [Unstructured Platform](/platform/overview) and Unstructured Ingest,
see the additional videos later on this page.

*   The Databricks workspace URL. Get the workspace URL for
    [AWS](https://docs.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids),
    [Azure](https://learn.microsoft.com/azure/databricks/workspace/workspace-details#workspace-instance-names-urls-and-ids),
    or [GCP](https://docs.gcp.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids).

    Examples:

    *   AWS: `https://<workspace-id>.cloud.databricks.com`
    *   Azure: `https://adb-<workspace-id>.<random-number>.azuredatabricks.net`
    *   GCP: `https://<workspace-id>.<random-number>.gcp.databricks.com`

*   The Databricks authentication details. For more information, see the documentation for
    [AWS](https://docs.databricks.com/dev-tools/auth/index.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/),
    or [GCP](https://docs.gcp.databricks.com/dev-tools/auth/index.html).

    The following videos show how to create a Databricks-managed service principal and then grant it access to a Databricks volume:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/wBmqv5DaA1E" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    <iframe width="560" height="315" src="https://www.youtube.com/embed/DykQRxgh2aQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    For the [Unstructured Platform](/platform/overview), only the following Databricks authentication type is supported:

    *   For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.
        Note that for Azure, only Databricks-managed service principals are supported. Microsoft Entra ID-managed service principals are not supported.

    For [Unstructured Ingest](/ingestion/overview), the following Databricks authentication types are supported:

    *   For Databricks personal access token authentication (AWS, Azure, and GCP): The personal access token's value.
    *   For username and password (basic) authentication (AWS only): The user's name and password values.
    *   For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.
    *   For OAuth user-to-machine (U2M) authentication (AWS, Azure, and GCP): No additional values.
    *   For Azure managed identities (MSI) authentication (Azure only): The client ID value for the corresponding managed identity.
    *   For Microsoft Entra ID service principal authentication (Azure only): The tenant ID, client ID, and client secret values for the corresponding service principal.
    *   For Azure CLI authentication (Azure only): No additional values.
    *   For Microsoft Entra ID user authentication (Azure only): The Entra ID token for the corresponding Entra ID user.
    *   For Google Cloud Platform credentials authentication (GCP only): The local path to the corresponding Google Cloud service account's credentials file.
    *   For Google Cloud Platform ID authentication (GCP only): The Google Cloud service account's email address.

*   The Databricks catalog name for the volume. Get the catalog name for [AWS](https://docs.databricks.com/catalogs/manage-catalog.html), [Azure](https://learn.microsoft.com/azure/databricks/catalogs/manage-catalog), or [GCP](https://docs.gcp.databricks.com/catalogs/manage-catalog.html).

*   The Databricks schema name for the volume. Get the schema name for [AWS](https://docs.databricks.com/schemas/manage-schema.html), [Azure](https://learn.microsoft.com/azure/databricks/schemas/manage-schema), or [GCP](https://docs.gcp.databricks.com/schemas/manage-schema.html).

*   The Databricks volume name, and optionally any path in that volume that you want to access directly. Get the volume information for [AWS](https://docs.databricks.com/files/volumes.html), [Azure](https://learn.microsoft.com/azure/databricks/files/volumes), or [GCP](https://docs.gcp.databricks.com/files/volumes.html).

*   Make sure that the target user or service principal has access to the target volume. To learn more, see the documentation for
    [AWS](https://docs.databricks.com/volumes/utility-commands.html#change-permissions-on-a-volume),
    [Azure](https://learn.microsoft.com/azure/databricks/volumes/utility-commands#change-permissions-on-a-volume),
    or [GCP](https://docs.gcp.databricks.com/volumes/utility-commands.html#change-permissions-on-a-volume).

The Databricks Volumes connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[databricks-volumes]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `DATABRICKS_HOST` - The Databricks host URL, represented by `--host` (CLI) or `host` (Python).
*   `DATABRICKS_CATALOG` - The Databricks catalog name for the Volume, represented by `--catalog` (CLI) or `catalog` (Python).
*   `DATABRICKS_SCHEMA` - The Databricks schema name for the Volume, represented by `--schema` (CLI) or `schema` (Python). If not specified, `default` is used.
*   `DATABRICKS_VOLUME` - The Databricks Volume name, represented by `--volume` (CLI) or `volume` (Python).
*   `DATABRICKS_VOLUME_PATH` - Any optional path to access within the volume, specified by `--volume-path` (CLI) or `volume_path` (Python).

Environment variables based on your authentication type, depending on your cloud provider:

*   For Databricks personal access token authentication (AWS, Azure, and GCP):

    *   `DATABRICKS_TOKEN` - The personal access token, represented by `--token` (CLI) or `token` (Python).

*   For username and password (basic) authentication (AWS only): The user's name and password values.

    *   `DATABRICKS_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
    *   `DATABRICKS_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

*   For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.

    *   `DATABRICKS_CLIENT_ID` - The client ID value for the corresponding service principal, represented by `--client-id` (CLI) or `client_id` (Python).
    *   `DATABRICKS_CLIENT_SECRET` - The client ID and OAuth secret values for the corresponding service principal, represented by `--client-secret` (CLI) or `client_secret` (Python).

*   For OAuth user-to-machine (U2M) authentication (AWS, Azure, and GCP): No additional environment variables.

*   For Azure managed identities (MSI) authentication (Azure only):

    *   `ARM_CLIENT_ID` - The client ID value for the corresponding managed identity, represented by `--azure-client-id` (CLI) or `azure_client_id` (Python).
    *   If the target identity has not already been added to the workspace, then you must also specify the
        `DATABRICKS_AZURE_RESOURCE_ID`, represented by `--azure-workspace-resource-id` (CLI) or `azure_workspace_resource_id` (Python).

*   For Microsoft Entra ID service principal authentication (Azure only):

    *   `ARM_TENANT_ID` - The tenant ID value for the corresponding service principal, represented by `--azure-tenant-id` (CLI) or `azure_tenant_id` (Python).
    *   `ARM_CLIENT_ID` - The client ID value for the corresponding service principal, represented by `--azure-client-id` (CLI) or `azure_client_id` (Python).
    *   `ARM_CLIENT_SECRET` - The client secret value for the corresponding service principal, represented by `--azure-client-secret` (CLI) or `azure_client_secret` (Python).
    *   If the service principal has not already been added to the workspace, then you must also specify the
        `DATABRICKS_AZURE_RESOURCE_ID`, represented by `--azure-workspace-resource-id` (CLI) or `azure_workspace_resource_id` (Python).

*   For Azure CLI authentication (Azure only): No additional environment variables.

*   For Microsoft Entra ID user authentication (Azure only):

    *   `DATABRICKS_TOKEN` - The Entra ID token for the corresponding Entra ID user, represented by `--token` (CLI) or `token` (Python).

*   For Google Cloud Platform credentials authentication (GCP only):

    *   `GOOGLE_CREDENTIALS` - The local path to the corresponding Google Cloud service account's credentials file, represented by `--google-credentials` (CLI) or `google_credentials`

*   For Google Cloud Platform ID authentication (GCP only):

    *   `GOOGLE_SERVICE_ACCOUNT` - The Google Cloud service account's email address, represented by `--google-service-account` (CLI) or `google_service_account` (Python).

*   Alternatively, you can store the preceding settings in a local
    [Databricks configuration profile](https://docs.databricks.com/en/dev-tools/auth/config-profiles.html) and then just
    refer to the profile's name:

    *   `DATABRICKS_PROFILE` - The name of the Databricks configuration profile, represented by `--profile` (CLI) or `profile` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
    databricks-volumes \
      --profile $DATABRICKS_PROFILE \
      --host $DATABRICKS_HOST \
      --catalog $DATABRICKS_CATALOG \
      --schema $DATABRICKS_SCHEMA \
      --volume $DATABRICKS_VOLUME \
      --volume-path $DATABRICKS_VOLUME_PATH
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  # For all supported Databricks authentication types, you can import this:
  from unstructured_ingest.v2.processes.connectors.databricks_volumes import (
      DatabricksVolumesConnectionConfig,
      DatabricksVolumesAccessConfig,
      DatabricksVolumesUploaderConfig
  )

  # Alternatively, for supported Databricks on AWS authentication types only, you can import this:
  # from unstructured_ingest.v2.processes.connectors.databricks.volumes_aws import (
  #     DatabricksAWSVolumesConnectionConfig,
  #     DatabricksAWSVolumesAccessConfig,
  #     DatabricksAWSVolumesUploaderConfig
  # )

  # Alternatively, for supported Azure Databricks authentication types only, you can import this:
  # from unstructured_ingest.v2.processes.connectors.databricks.volumes_azure import (
  #     DatabricksAzureVolumesConnectionConfig,
  #     DatabricksAzureVolumesAccessConfig,
  #     DatabricksAzureVolumesUploaderConfig
  # )

  # Alternatively, for supported Databricks on Google Cloud authentication types only, you can import this:
  # from unstructured_ingest.v2.processes.connectors.databricks.volumes_gcp import (
  #     DatabricksGoogleVolumesConnectionConfig,
  #     DatabricksGoogleVolumesAccessConfig,
  #     DatabricksGoogleVolumesUploaderConfig
  # )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          # For specifying a Databricks configuration profile:
          destination_connection_config=DatabricksVolumesConnectionConfig(
              access_config=DatabricksVolumesAccessConfig(profile=os.getenv("DATABRICKS_PROFILE")),
              host=os.getenv("DATABRICKS_HOST"),
              catalog=os.getenv("DATABRICKS_CATALOG"),
              schema=os.getenv("DATABRICKS_SCHEMA"),
              volume=os.getenv("DATABRICKS_VOLUME"),
              volume_path=os.getenv("DATABRICKS_VOLUME_PATH")
          ),
          uploader_config=DatabricksVolumesUploaderConfig(overwrite=True)
          # Other examples:
          #
          # For Databricks on AWS, with Databricks personal access token authentication:
          # destination_connection_config=DatabricksAWSVolumesConnectionConfig(
          #     access_config=DatabricksAWSVolumesAccessConfig(token=os.getenv("DATABRICKS_TOKEN")),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          # uploader_config=DatabricksAWSVolumesUploaderConfig(
          #     catalog=os.getenv("DATABRICKS_CATALOG"),
          #     schema=os.getenv("DATABRICKS_SCHEMA"),
          #     volume=os.getenv("DATABRICKS_VOLUME"),
          #     volume_path=os.getenv("DATABRICKS_VOLUME_PATH"),
          #     overwrite=True
          # )
          #
          # For Azure Databricks, with Microsoft Entra ID service principal authentication:
          # destination_connection_config=DatabricksAzureVolumesConnectionConfig(
          #     access_config=DatabricksAzureVolumesAccessConfig(
          #         azure_client_id=os.getenv("ARM_CLIENT_ID"),
          #         azure_client_secret=os.getenv("ARM_CLIENT_SECRET"),
          #         azure_tenant_id=os.getenv("ARM_TENANT_ID")   
          #     ),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          # uploader_config=DatabricksAzureVolumesUploaderConfig(
          #     catalog=os.getenv("DATABRICKS_CATALOG"),
          #     schema=os.getenv("DATABRICKS_SCHEMA"),
          #     volume=os.getenv("DATABRICKS_VOLUME"),
          #     volume_path=os.getenv("DATABRICKS_VOLUME_PATH"),
          #     overwrite=True
          # )
          #
          # For Databricks on Google Cloud, with Google Cloud Platform credentials authentication:
          # destination_connection_config=DatabricksGoogleVolumesConnectionConfig(
          #     access_config=DatabricksGoogleVolumesAccessConfig(
          #         google_service_account=os.getenv("GOOGLE_CREDENTIALS")
          #     ),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          # uploader_config=DatabricksAWSVolumesUploaderConfig(
          #     catalog=os.getenv("DATABRICKS_CATALOG"),
          #     schema=os.getenv("DATABRICKS_SCHEMA"),
          #     volume=os.getenv("DATABRICKS_VOLUME"),
          #     volume_path=os.getenv("DATABRICKS_VOLUME_PATH"),
          #     overwrite=True
          # )
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.databricks_volumes import (
      DatabricksVolumesAccessConfig,
      DatabricksVolumesWriteConfig,
      SimpleDatabricksVolumesConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.databricks_volumes import (
      DatabricksVolumesWriter,
  )


  def get_writer() -> Writer:
      return DatabricksVolumesWriter(
          connector_config=SimpleDatabricksVolumesConfig(
              host=os.getenv("DATABRICKS_HOST"),
              access_config=DatabricksVolumesAccessConfig(
                  token=os.getenv("DATABRICKS_TOKEN")
              ),
          ),
          write_config=DatabricksVolumesWriteConfig(
              catalog=os.getenv("DATABRICKS_CATALOG"),
              schema=os.getenv("DATABRICKS_SCHEMA"),
              volume=os.getenv("DATABRICKS_VOLUME"),
              volume_path=os.getenv("DATABRICKS_VOLUME_PATH")
          ),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(
              chunk_elements=True
              chunking_strategy="by_title",
          ),
          embedding_config=EmbeddingConfig(
              provider="huggingface"
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>


# Delta Table



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a Delta Table in an Amazon S3 bucket.

You will need:

The Delta Table prerequisites for Amazon S3:

The following video shows how to fulfill the minimum set of S3 prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/_W4565dcUGI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The preceding video does not show how to create an AWS account or an S3 bucket.

For more information about prerequisites, see the following:

*   An AWS account. [Create an AWS account](https://aws.amazon.com/free).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/lIdh92JmWtg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An S3 bucket. [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
    Additional approaches are in the following video and in the how-to sections at the end of this page.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/e6w9LwZJFIA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/y4SfQoJpipo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at
    minimum the permission of `s3:PutObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

*   For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account.
    [Create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/MoFTaGJE65Q" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   If the target files are in the root of the bucket, the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
    If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

*   If the target files are in a folder, make sure the authenticated AWS IAM user has
    authenticated access to the folder as well. [Enable authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2.  Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3.  Browse to the existing bucket and open it.

4.  Click the **Permissions** tab.

5.  In the **Bucket policy** area, click **Edit**.

6.  In the **Policy** text area, copy the following JSON-formatted policy.
    To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this policy, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-bucket-name>` in two places with the name of your bucket.

    ```json
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::<my-account-id>:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::<my-bucket-name>",
                    "arn:aws:s3:::<my-bucket-name>/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    ```

7.  Click **Save changes**.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
    the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    ```yaml
    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

    Parameters:
      BucketName:
        Type: String
        Description: 'Name of the S3 bucket to create'

    Resources:
      MyS3Bucket:
        Type: 'AWS::S3::Bucket'
        Properties:
          BucketName: !Ref BucketName
          PublicAccessBlockConfiguration:
            BlockPublicAcls: true
            BlockPublicPolicy: false
            IgnorePublicAcls: true
            RestrictPublicBuckets: true

      BucketPolicy:
        Type: 'AWS::S3::BucketPolicy'
        Properties:
          Bucket: !Ref MyS3Bucket
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Sid: AllowAllAuthenticatedUsersInAccount
                Effect: Allow
                Principal:
                  AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
                Action:
                  - 's3:GetObject'
                  - 's3:PutObject'
                  - 's3:ListBucket'
                  - 's3:DeleteObject'
                Resource:
                  - !Sub 'arn:aws:s3:::${BucketName}'
                  - !Sub 'arn:aws:s3:::${BucketName}/*'

    Outputs:
      BucketName:
        Description: 'Name of the created S3 bucket'
        Value: !Ref MyS3Bucket
    ```

2.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3.  Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4.  Click **Create stack > With new resources (standard)**.

5.  On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6.  Click **Choose file**, and browse to and select the YAML file from your local machine.

7.  Click **Next**.

8.  Enter a unique **Stack name** and **BucketName**.

9.  Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2.  [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3.  Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
    To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this script, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-unique-bucket-name>` with the name of your bucket.
    *   Replace `<us-east-1>` with your AWS Region.

    ```bash
    #!/bin/bash

    # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
    ACCOUNT_ID="<my-account-id>"
    BUCKET_NAME="<my-unique-bucket-name>"
    REGION="<us-east-1>"

    # Temporary filename for the bucket policy.
    # Do not change this variable.
    POLICY_FILE="bucket_policy.json"

    # Create the bucket.
    aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

    # Wait for the bucket to exist.
    echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
    aws s3api wait bucket-exists --bucket $BUCKET_NAME

    # Check if the wait command was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket '$BUCKET_NAME' has been fully created."
    else
        echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
        exit 1
    fi

    # Remove the "block public policy" bucket access setting.
    aws s3api put-public-access-block \
        --bucket $BUCKET_NAME \
        --public-access-block-configuration \
        '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

    # Check if the operation was successful.
    if [ $? -eq 0 ]; then
        echo "The block public policy access setting was removed from '$BUCKET_NAME'."
    else
        echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
        exit 1
    fi

    # Create the bucket policy.
    cat << EOF > $POLICY_FILE
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::$BUCKET_NAME",
                    "arn:aws:s3:::$BUCKET_NAME/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    EOF

    # Apply the bucket policy.
    aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

    # Check if the policy application was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket policy was applied to '$BUCKET_NAME'."
    else
        echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
        exit 1
    fi

    # Verify the applied policy.
    echo "Verifying the applied policy:"
    aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

    # Remove the temporary bucket policy file.
    rm $POLICY_FILE
    ```

4.  Run the script, for example:

    ```bash
    sh create-s3-bucket.sh
    ```

5.  After the bucket is created, you can delete the script file, if you want.

The Delta Table connector dependencies for Amazon S3:

```bash CLI, Python
pip install "unstructured-ingest[delta-table]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `AWS_S3_URL` - The path to the S3 bucket or folder, formatted as `s3://my-bucket/` (if the files are in the bucket's root) or `s3://my-bucket/my-folder/`, represented by `--table-uri` (CLI) or `table_uri` (Python).
*   `AWS_ACCESS_KEY_ID` - The AWS access key ID for the authenticated AWS IAM user, represented by `--aws-access-key-id` (CLI) or `aws_access_key` (Python).
*   `AWS_SECRET_ACCESS_KEY` - The corresponding AWS secret access key, represented by `--aws-secret-access-key` (CLI) or `aws_secret_access_key` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
    delta-table \
      --aws-access-key-id $AWS_ACCESS_KEY_ID \
      --aws-secret-access-key $AWS_SECRET_ACCESS_KEY \
      --table-uri $AWS_S3_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.delta_table import (
      DeltaTableConnectionConfig,
      DeltaTableAccessConfig,
      DeltaTableUploadStagerConfig,
      DeltaTableUploaderConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=DeltaTableConnectionConfig(
              access_config=DeltaTableAccessConfig(
                  aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
                  aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY")
              ),
              table_uri=os.getenv("AWS_S3_URL")
          ),
          stager_config=DeltaTableUploadStagerConfig(),
          uploader_config=DeltaTableUploaderConfig()
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.delta_table import DeltaTableWriteConfig, SimpleDeltaTableConfig
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.delta_table import (
      DeltaTableWriter,
  )


  def get_writer() -> Writer:
      return DeltaTableWriter(
          connector_config=SimpleDeltaTableConfig(
              table_uri=os.getenv("AWS_S3_URL"),
              storage_options={
                  "AWS_ACCESS_KEY_ID": os.getenv("AWS_ACCESS_KEY_ID"),
                  "AWS_SECRET_ACCESS_KEY": os.getenv("AWS_SECRET_ACCESS_KEY"),
              },
          ),
          write_config=DeltaTableWriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>


# Dropbox



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a Dropbox account.

You will need:

The Dropbox prerequisites:

1.  A Dropbox account. [Get an account](https://www.dropbox.com/try/teams).

2.  A target source or destination folder in your Dropbox account. [Create a folder](https://help.dropbox.com/create-upload/creating-using-folders-on-dropbox).

3.  A Dropbox app for your Dropbox account. To learn how to create an app, click the **App Console** tab on the [Getting Started](https://www.dropbox.com/developers/reference/getting-started) page.

4.  Permission for your Dropbox app to read from, and write to, the target folder in your Dropbox account as needed. To do this:

    *   On the **Permissions** tab of your Dropbox app, check the boxes **files.content.read** or **files.content.write** or both. [Learn more](https://developers.dropbox.com/oauth-guide).

    *   On the **Settings** tab of your Dropbox app, for **App folder name**, set the name of the target folder in your Dropbox account for your Dropbox app to have access to.

    *   Note the remote URL to the target folder, which takes the format `dropbox://<path/to/folder/in/account>`.

5.  An access token for your Dropbox account. [Get a token](https://dropbox.tech/developers/generate-an-access-token-for-your-own-account). Save this token in a secure location. Do not share it with others.

The Dropbox connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[dropbox]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `DROPBOX_ACCESS_TOKEN` - The value of your access token, represented by `--token` (CLI) or `token` (Python).
*   `DROPBOX_REMOTE_URL` - The remote URL to the target folder, represented by `--remote-url` (CLI) or `remote_url` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY\
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    dropbox \
      --token $DROPBOX_ACCESS_TOKEN \
      --remote-url $DROPBOX_REMOTE_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.dropbox import (
      DropboxAccessConfig,
      DropboxConnectionConfig,
      DropboxUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=DropboxConnectionConfig(
              access_config=DropboxAccessConfig(
                  token=os.getenv("DROPBOX_ACCESS_TOKEN")
              )
          ),
          uploader_config=DropboxUploaderConfig(
              remote_url=os.getenv("DROPBOX_REMOTE_URL")
          )
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.dropbox import (
      DropboxAccessConfig,
      DropboxWriteConfig,
      SimpleDropboxConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.fsspec.dropbox import (
      DropboxWriter,
  )


  def get_writer() -> Writer:
      return DropboxWriter(
          connector_config=SimpleDropboxConfig(
              access_config=DropboxAccessConfig(token=os.getenv("DROPBOX_ACCESS_TOKEN")),
              remote_url=os.getenv("DROPBOX_REMOTE_URL"),
          ),
          write_config=DropboxWriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>


# Elasticsearch



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in Elasticsearch.

You will need:

The Elasticsearch prerequisites.

*   For the [Unstructured Platform](/platform/overview), only Elastic Cloud instances are supported.

*   For [Unstructured Ingest](/ingestion/overview), Elastic Cloud instances and self-manged Elasticsearch instances are supported.

*   For Elastic Cloud, you will need an [Elastic Cloud service instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#hosted-elasticsearch-service).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/HEYIFC7uOus" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For self-managed Elasticsearch, you will need a [self-managed Elasticsearch instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#elasticsearch-deployment-options).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/9znbAkNG1Ok" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   You will need the name of the index on the instance. See [Create index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html) and [Get index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html).

    The Elasticsearch index that you use must have a schema that is compatible with the schema of the documents
    that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following index schema example for your own needs:

    ```json
    {
        "settings": {
            "index": {
                "knn": true,
                "knn.algo_param.ef_search": 100
            }
        },
        "mappings": {
            "properties": {
                "element_id": {
                    "type": "keyword"
                },
                "text": {
                    "type": "text"
                },
                "embeddings": {
                    "type": "dense_vector",
                    "dims": 384,
                    "index": true,
                    "similarity": "cosine"
                },
                "metadata": {
                    "type": "object",
                    "properties": {
                        "parent_id": {
                            "type": "text"
                        },
                        "page_number": {
                            "type": "integer"
                        },
                        "is_continuation": {
                            "type": "boolean"
                        },
                        "orig_elements": {
                            "type": "text"
                        }
                    }
                }
            }
        }
    }
    ```

    See also:

    *   [An Introduction to Elasticsearch Mapping](https://www.elastic.co/blog/found-elasticsearch-mapping-introduction)
    *   [Explicit mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html)
    *   [Dynamic field mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html)
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

    <iframe width="560" height="315" src="https://www.youtube.com/embed/0f0tEcrvP8g" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For Elastic Cloud, you will need the Elastic Cloud service instance's API key. If you are using Unstructured Ingest, you will also need the instance's Cloud ID. To get these, see your Elasticsearch Service web console.

*   For self-managed Elasticsearch, you will need:

    *   The self-managed instance's hostname and port number. See [Networking](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html).
    *   If you're using basic authentication to the self-managed instance, the user's name and password.
    *   If you're using token-based authentication to the self-managed instance, the bearer token or API key for the instance. See [Token-based authentication services](https://www.elastic.co/guide/en/elasticsearch/reference/current/token-authentication-services.html) and [Create API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html).
    *   If you're using certificate authentication to the self-managed instance, the path to the Certificate Authority (CA) file on the instance, and the certificate fingerprint. See [SSL certificate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-ssl.html) and [Where can I see my Certificate Fingerprint?](https://discuss.elastic.co/t/where-can-i-see-my-certificate-fingerprint/319335/3).

The Elasticsearch connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[elasticsearch]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `ELASTICSEARCH_HOST` - The hostname and port number, defined as `<hostname>:<port-number>` and represented by `--hosts` (CLI) or `hosts` (Python).
*   `ELASTICSEARCH_INDEX_NAME` - The name of the search index, represented by `--index-name` (CLI) or `index_name` (Python).

If you're using Elastic Cloud to connect to ElasticSearch:

*   `ELASTIC_CLOUD_ID` - The instance's Cloud ID, represented by `--cloud-id` (CLI) or `cloud_id` (Python).
*   `ELASTIC_CLOUD_API_KEY` - The corresponding Cloud API key value, represented by `--api-key-id` (CLI) or `api_key_id` (Python).

If you're using basic authentication to the instance:

*   `ELASTICSEARCH_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
*   `ELASTICSEARCH_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

If you're using token-based authentication to the instance instead:

*   `ELASTICSEARCH_BEARER_TOKEN` - The bearer token's value, represented by `--bearer-auth` (CLI) or `bearer_auth` (Python).
*   `ELASTIC_CLOUD_API_KEY_ID` - The API key's value, represented by `--api-key` (CLI) or `api_key` (Python).

If you're using certificates:

*   `ELASTICSEARCH_CA_CERTS` - The path to the Certificate Authority (CA) file, represented by `--ca-certs` (CLI) or `ca_certs` (Python).
*   `ELASTICSEARCH_SSL_ASSERT_FINGERPRINT` - The certificate's fingerprint, represented by `--ssl-assert-fingerprint` or `ssl_assert_fingerprint` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 4 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
    elasticsearch \
      --hosts $ELASTICSEARCH_HOST \
      --username $ELASTICSEARCH_USERNAME \
      --password $ELASTICSEARCH_PASSWORD \
      --index-name $ELASTICSEARCH_INDEX_NAME \
      --ca-certs $ELASTICSEARCH_CA_CERTS \
      --ssl-assert-fingerprint $ELASTICSEARCH_SSL_ASSERT_FINGERPRINT \
      --num-processes 2
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.elasticsearch import (
      ElasticsearchConnectionConfig,
      ElasticsearchAccessConfig,
      ElasticsearchUploadStagerConfig,
      ElasticsearchUploaderConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=ElasticsearchConnectionConfig(
              access_config=ElasticsearchAccessConfig(
                  password=os.getenv("ELASTICSEARCH_PASSWORD"),
                  ssl_assert_fingerprint=os.getenv("ELASTICSEARCH_SSL_ASSERT_FINGERPRINT")
              ),
              hosts=[os.getenv("ELASTICSEARCH_HOST")],
              username=os.getenv("ELASTICSEARCH_USERNAME"),
              ca_certs=os.getenv("ELASTICSEARCH_CA_CERTS")
          ),
          stager_config=ElasticsearchUploadStagerConfig(index_name=os.getenv("ELASTICSEARCH_INDEX_NAME")),
          uploader_config=ElasticsearchUploaderConfig(index_name=os.getenv("ELASTICSEARCH_INDEX_NAME"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.elasticsearch import (
      ElasticsearchAccessConfig,
      ElasticsearchWriteConfig,
      SimpleElasticsearchConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.elasticsearch import (
      ElasticsearchWriter,
  )


  def get_writer() -> Writer:
      return ElasticsearchWriter(
          connector_config=SimpleElasticsearchConfig(
              access_config=ElasticsearchAccessConfig(
                  password=os.getenv("ELASTICSEARCH_PASSWORD"),
                  ssl_assert_fingerprint=os.getenv("ELASTICSEARCH_SSL_ASSERT_FINGERPRINT")
              ),
              hosts=[os.getenv("ELASTICSEARCH_HOST")],
              username=os.getenv("ELASTICSEARCH_USERNAME"),
              ca_certs=os.getenv("ELASTICSEARCH_CA_CERTS"),
          ),
          write_config=ElasticsearchWriteConfig(
              batch_size_bytes=15_000_000,
              num_processes=2,
          ),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>


# Google Cloud Storage



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in Google Cloud Service.

You will need:

The Google Cloud Storage prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/HYaALQ0F-L4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A Google Cloud service account. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#console).

*   A service account key for the service account. See [Create a service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating) in
    [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete).

    To ensure maximum compatibility across Unstructured service offerings, you should give the service account key information to Unstructured as
    a single-line string that contains the contents of the downloaded service account key file (and not the service account key file itself).
    To print this single-line string without line breaks, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
    In this command, replace `<path-to-downloaded-key-file>` with the path to the service account key file that you downloaded by following the preceding instructions.

    *   For macOS or Linux:
        ```text
        tr -d '\n' < <path-to-downloaded-key-file>
        ```
    *   For Windows:
        ```text
        (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
        ```

*   The URI for a Google Cloud Storage bucket. This URI consists of the target bucket name, plus any target folder within the bucket, expressed as `gs://<bucket-name>[/folder-name]`. [Create a bucket](https://cloud.google.com/storage/docs/creating-buckets#console).

    This bucket must have, at minimum, one of the following roles applied to the target Google Cloud service account:

    *   `Storage Object Viewer` for bucket read access.
    *   `Storage Object Creator` for bucket write access.
    *   The `Storage Object Admin` role provides read and write access, plus access to additional bucket operations.

    To apply one of these roles to a service account for a bucket, see [Add a principal to a bucket-level policy](https://cloud.google.com/storage/docs/access-control/using-iam-permissions#bucket-add)
    in [Set and manage IAM policies on buckets](https://cloud.google.com/storage/docs/access-control/using-iam-permissions).

The Google Cloud Storage connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[gcs]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `GCS_SERVICE_ACCOUNT_KEY` - The Google Cloud service account key for Google Cloud Storage, represented by `--service-account-key` (CLI) or `service_account_key` (Python).
*   `GCS_REMOTE_URL` - The Google Cloud Storage bucket URL, represented by `--remote-url` (CLI) or `remote_url` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
    gcs \
      --service-account-key $GCS_SERVICE_ACCOUNT_KEY \
      --remote-url $GCS_REMOTE_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.gcs import (
      GcsConnectionConfig,
      GcsAccessConfig,
      GcsUploaderConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=GcsConnectionConfig(
              access_config=GcsAccessConfig(
                  service_account_key=os.getenv("GCS_SERVICE_ACCOUNT_KEY")
              )
          ),
          uploader_config=GcsUploaderConfig(remote_url=os.getenv("GCS_REMOTE_URL"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.gcs import (
      GcsAccessConfig,
      GcsWriteConfig,
      SimpleGcsConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.fsspec.gcs import (
      GcsWriter,
  )


  def get_writer() -> Writer:
      return GcsWriter(
          connector_config=SimpleGcsConfig(
              access_config=GcsAccessConfig(token=os.getenv("GCS_SERVICE_ACCOUNT_KEY")),
              remote_url=os.getenv("GCS_REMOTE_URL"),
          ),
          write_config=GcsWriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>


# Kafka



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in Kafka.

You will need:

The Kafka prerequisites:

*   A Kafka cluster, such as ones provided by [Confluent Cloud](https://www.confluent.io/confluent-cloud), [Amazon Managed Streaming for Apache Kafka (Amazon MSK)](https://aws.amazon.com/msk/), or [Google Cloud Managed Service for Apache Kafka](https://cloud.google.com/products/managed-service-for-apache-kafka).
*   The hostname of the bootstrap Kafka cluster to connect to.
*   The port number of the cluster.
*   The name of the topic to read messages from and write messages to on the cluster.
*   If you use Kafka API keys and secrets for authentication, the key and secret values.

The Kafka connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[kafka]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `KAFKA_BOOTSTRAP_SERVER` - The hostname of the bootstrap Kafka cluster to connect to, represented by `--bootstrap-server` (CLI) or `bootstrap_server` (Python).
*   `KAFKA_PORT` - The port number of the cluster, represented by `--port` (CLI) or `port` (Python).
*   `KAFKA_TOPIC` - The unique name of the topic to read messages from and write messages to on the cluster, represented by `--topic` (CLI) or `topic` (Python).

If you use Kafka API keys and secrets for authentication:

*   `KAFKA_API_KEY` - The Kafka API key value, represented by `--kafka-api-key` (CLI) or `kafka_api_key` (Python).
*   `KAFKA_SECRET` - The secret value for the Kafka API key, represented by `--secret` (CLI) or `secret` (Python).

Additional settings include:

*   `--confluent` (CLI) or `confluent` (Python): True to indicate that the cluster is running Confluent Kafka.
*   `--num-messages-to-consume` (CLI) or `num_messages_to_consume` (Python): The maximum number of messages to get from the topic. The default is `1`.
*   `--timeout` (CLI) or `timeout` (Python): The maximum amount of time to wait for the response of a request to the topic, expressed in seconds. The default is `1.0`.

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY\
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    kafka \
      --bootstrap-server $KAFKA_BOOTSTRAP_SERVER \ 
      --port $KAFKA_PORT \
      --topic $KAFKA_TOPIC \
      --kafka-api-key $KAFKA_API_KEY \
      --secret $KAFKA_API_KEY \
      --confluent true \
      --num-messages-to-consume 1 \
      --timeout 1.0
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  from unstructured_ingest.v2.processes.connectors.kafka.cloud import (
      CloudKafkaConnectionConfig,
      CloudKafkaAccessConfig,
      CloudKafkaUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=CloudKafkaConnectionConfig(
              access_config=CloudKafkaAccessConfig(
                  kafka_api_key=os.getenv("KAFKA_API_KEY"),
                  secret=os.getenv("KAFKA_SECRET")
              ),
              bootstrap_server=os.getenv("KAFKA_BOOTSTRAP_SERVER"),
              port=os.getenv("KAFKA_PORT")
          ),
          uploader_config=CloudKafkaUploaderConfig(
              batch_size=100,
              topic=os.getenv("KAFKA_TOPIC"),
              timeout=10
          )
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.kafka import (
      KafkaAccessConfig,
      KafkaWriteConfig,
      SimpleKafkaConfig
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.kafka import (
      KafkaWriter
  )
  from unstructured_ingest.runner.writers.base_writer import Writer

  def get_writer() -> Writer:
      return KafkaWriter(
          connector_config=SimpleKafkaConfig(
              bootstrap_server=os.getenv("KAFKA_BOOTSTRAP_SERVER"),
              port=os.getenv("KAFKA_PORT"),
              topic=os.getenv("KAFKA_TOPIC"),
              access_config=KafkaAccessConfig(
                  kafka_api_key=os.getenv("KAFKA_API_KEY"),
                  secret=os.getenv("KAFKA_SECRET"),
              ),
              confluent=False,
              num_messages_to_consume=1,
              timeout=1.0
          ),
          write_config=KafkaWriteConfig()
      )


  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>


# KDB.AI



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in KDB.AI.

You will need:

The KDB.AI prerequisites:

*   A KDB.AI Cloud or server instance. [Sign Up for KDB.AI Cloud: Starter Edition](https://trykdb.kx.com/kdbai/signup/). [Set up KDB.AI Server](https://code.kx.com/kdbai/latest/gettingStarted/kdb-ai-server-setup.html).
*   The instance's endpoint URL. [Get the KDB.AI Cloud endpoint URL](https://code.kx.com/kdbai/latest/gettingStarted/kdb-ai-cloud-setup.html#connect-to-your-database). [Get the KDB.AI Server endpoint URL](https://code.kx.com/kdbai/latest/gettingStarted/kdb-ai-server-setup.html).
*   An API key. [Create the API key](https://code.kx.com/kdbai/latest/gettingStarted/kdb-ai-cloud-setup.html#create-an-api-key).
*   The name of the target table to access. [Create the table](https://code.kx.com/kdbai/latest/gettingStarted/quickstart.html#create-a-new-table).

The KDB.AI connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[kdbai]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `KDBAI_ENDPOINT` - The KDB.AI instance's endpoint URL, represented by `--endpoint` (CLI) or `endpoint` (Python).
*   `KDBAI_API_KEY` - The KDB.AI API key, represented by `--api-key` (CLI) or `api_key` (Python).
*   `KDBAI_TABLE` - The name of the target table, represented by `--table-name` (CLI) or `table_name` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    kdbai \
      --endpoint $KDBAI_API_KEY \
      --api-key $KDBAI_API_KEY \
      --table-name $KDBAI_TABLE
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.kdbai import (
      KdbaiConnectionConfig,
      KdbaiAccessConfig,
      KdbaiUploadStagerConfig,
      KdbaiUploaderConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=KdbaiConnectionConfig(
              access_config=KdbaiAccessConfig(
                  api_key=os.getenv("KDBAI_API_KEY")
              ),
              endpoint=os.getenv("KDBAI_ENDPOINT")
          ),
          stager_config=KdbaiUploadStagerConfig(),
          uploader_config=KdbaiUploaderConfig(table_name=os.getenv("KDBAI_TABLE"))
      ).run()
  ```
</CodeGroup>


# LanceDB



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in LanceDB.

You will need:

The LanceDB prerequisites:

*   A [LanceDB open source software (OSS) installation](https://lancedb.github.io/lancedb/basic/#installation) on a local machine, a server, or a virtual machine.
    (LanceDB Cloud is not supported.)

*   For LanceDB OSS with local data storage:

    *   The local path to the folder where the LanceDB data is (or will be) stored.
        See [Connect to a database](https://lancedb.github.io/lancedb/basic/#connect-to-a-database) in the LanceDB documentation.
    *   The name of the target [LanceDB table](https://lancedb.github.io/lancedb/basic/#create-an-empty-table) within the local data folder.

*   For LanceDB OSS with data storage in an Amazon S3 bucket:

    *   The URI for the target Amazon S3 bucket and any target folder path within that bucket. Use the format `s3://<bucket-name>[/<folder-name>]`.
    *   The name of the target [LanceDB table](https://lancedb.github.io/lancedb/guides/storage/#object-stores) within the Amazon S3 bucket.
    *   The AWS access key ID and AWS secret access key for the AWS IAM entity that has access to the Amazon S3 bucket.

    For more information, see [AWS S3](https://lancedb.github.io/lancedb/guides/storage/#aws-s3) in the LanceDB documentation, along with the following video:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/hyDHfhVVAhs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For LanceDB OSS with data storage in an Azure Blob Storage account:

    *   The name of the target Azure Blob Storage account.
        \= The URI for the target container within that Azure Blob Storage account and any target folder path within that container. Use the format `az://<container-name>[/<folder-name>]`.
    *   The name of the target [LanceDB table](https://lancedb.github.io/lancedb/guides/storage/#object-stores) within the Azure Blob Storage account.
    *   The access key for the Azure Blob Storage account.

    For more information, see [Azure Blob Storage](https://lancedb.github.io/lancedb/guides/storage/#azure-blob-storage) in the LanceDB documentation, along with the following video:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/Vl3KCphlh9Y" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For LanceDB OSS with data storage in a Google Cloud Storage bucket:

    *   The URI for the target Google Cloud Storage bucket and any target folder path within that bucket. Use the format `gs://<bucket-name>[/<folder-name>]`.
    *   The name of the target [LanceDB table](https://lancedb.github.io/lancedb/guides/storage/#object-stores) within the Google Cloud Storage bucket.
    *   A single-line string that contains the contents of the downloaded service account key file for the Google Cloud service account that has access to the
        Google Cloud Storage bucket.

    For more information, see [Google Cloud Storage](https://lancedb.github.io/lancedb/guides/storage/#google-cloud-storage) in the LanceDB documentation, along with the following video:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/HYaALQ0F-L4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The LanceDB connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[lancedb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   For LanceDB OSS with local data storage:

    *   `LANCEDB_URI` - The local path to the folder where the LanceDB data is stored, represented by `--uri` (CLI) or `uri` (Python).
    *   `LANCEDB_TABLE` - The name of the target LanceDB table within the local data folder, represented by `--table-name` (CLI) or `table_name` (Python).

*   For LanceDB OSS with data storage in an Amazon S3 bucket:

    *   `LANCEDB_URI` - The URI for the target Amazon S3 bucket and any target folder path within that bucket. Use the format `s3://<bucket-name>[/<folder-name>]`. This is represented by `--uri` (CLI) or `uri` (Python).
    *   `LANCEDB_TABLE` - The name of the target LanceDB table within the Amazon S3 bucket, rrepresented by `--table-name` (CLI) or `table_name` (Python).
    *   `AWS_ACCESS_KEY_ID` - The AWS access key ID for the AWS IAM entity that has access to the Amazon S3 bucket, represented by `--aws-access-key-id` (CLI) or `aws_access_key_id` (Python).
    *   `AWS_SECRET_ACCESS_KEY` - The AWS secret access key for the AWS IAM entity that has access to the Amazon S3 bucket, represented by `--aws-secret-access-key` (CLI) or `aws_secret_access_key` (Python).

*   For LanceDB OSS with data storage in an Azure Blob Storage account:

    *   `LANCEDB_URI` - The URI for the target container within that Azure Blob Storage account and any target folder path within that container. Use the format `az://<container-name>[/<folder-name>]`. This is represented by `--uri` (CLI) or `uri` (Python).
    *   `LANCEDB_TABLE` - The name of the target LanceDB table within the Azure Blob Storage account, represented by `--table-name` (CLI) or `table_name` (Python).
    *   `AZURE_STORAGE_ACCOUNT_NAME` - The name of the target Azure Blob Storage account, represented by `--azure-storage-account-name` (CLI) or `azure_storage_account_name` (Python).
    *   `AZURE_STORAGE_ACCOUNT_KEY` - The access key for the Azure Blob Storage account, represented by `--azure-storage-account-key` (CLI) or `azure_storage_account_key` (Python).

*   For LanceDB OSS with data storage in a Google Cloud Storage bucket:

    *   `LANCEDB_URI` - The URI for the target Google Cloud Storage bucket and any target folder path within that bucket. Use the format `gs://<bucket-name>[/<folder-name>]`. This is represented by `--uri` (CLI) or `uri` (Python).
    *   `LANCEDB_TABLE` - The name of the target LanceDB table within the Google Cloud Storage bucket, represented by `--table-name` (CLI) or `table_name` (Python).
    *   `GCS_SERVICE_ACCOUNT_KEY` - A single-line string that contains the contents of the downloaded service account key file for the Google Cloud service account
        that has access to the Google Cloud Storage bucket, represented by `--google-service-account-key` (CLI) or `google_service_account_key` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  # For LanceDB OSS with local data storage:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    lancedb-local \
      --uri $LANCEDB_URI \
      --table-name $LANCEDB_TABLE

  # For LanceDB OSS with data storage in an Amazon S3 bucket:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    lancedb-aws \
      --aws-access-key-id $AWS_ACCESS_KEY_ID \
      --aws-secret-access-key $AWS_SECRET_ACCESS_KEY \
      --uri $LANCEDB_URI \
      --table-name $LANCEDB_TABLE \
      --timeout 30s

  # For LanceDB OSS with data storage in an Azure Blob Storage account:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    lancedb-azure \
      --azure-storage-account-name $AZURE_STORAGE_ACCOUNT_NAME \
      --azure-storage-account-key $AZURE_STORAGE_ACCOUNT_KEY \
      --uri $LANCEDB_URI \
      --table-name $LANCEDB_TABLE \
      --timeout 30s

  # For LanceDB OSS with data storage in a Google Cloud Storage bucket:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    lancedb-gcs \
      --google-service-account-key $GCS_SERVICE_ACCOUNT_KEY \
      --uri $LANCEDB_URI \
      --table-name $LANCEDB_TABLE \
      --timeout 30s
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # For LanceDB OSS with local data storage:
  # from unstructured_ingest.v2.processes.connectors.lancedb.local import (
  #     LanceDBLocalConnectionConfig,
  #     LanceDBLocalAccessConfig,
  #     LanceDBUploadStagerConfig,
  #     LanceDBUploaderConfig
  # )

  # For LanceDB OSS with data storage in an Amazon S3 bucket:
  from unstructured_ingest.v2.processes.connectors.lancedb.aws import (
      LanceDBS3ConnectionConfig,
      LanceDBS3AccessConfig,
      LanceDBUploadStagerConfig,
      LanceDBUploaderConfig
  )

  # For LanceDB OSS with data storage in an Azure Blob Storage account:
  # from unstructured_ingest.v2.processes.connectors.lancedb.azure import (
  #     LanceDBAzureConnectionConfig,
  #     LanceDBAzureAccessConfig,
  #     LanceDBUploadStagerConfig,
  #     LanceDBUploaderConfig
  # )

  # For LanceDB OSS with data storage in a Google Cloud Storage bucket:
  # from unstructured_ingest.v2.processes.connectors.lancedb.gcp import (
  #     LanceDBGCSConnectionConfig,
  #     LanceDBGCSAccessConfig,
  #     LanceDBUploadStagerConfig,
  #     LanceDBUploaderConfig
  # )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),

          # For LanceDB OSS with local data storage:
          # destination_connection_config=LanceDBLocalConnectionConfig(
          #     access_config=LanceDBLocalAccessConfig(),
          #     uri=os.getenv("LANCEDB_URI")
          # ),

          # For LanceDB OSS with data storage in an Amazon S3 bucket:
          destination_connection_config=LanceDBS3ConnectionConfig(
              access_config=LanceDBS3AccessConfig(
                  aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
                  aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY")
              ),
              uri=os.getenv("LANCEDB_URI"),
              timeout="30s"
          ),

          # For LanceDB OSS with data storage in an Azure Blob Storage account:
          # destination_connection_config=LanceDBAzureConnectionConfig(
          #     access_config=LanceDBAzureAccessConfig(
          #         azure_storage_account_name=os.getenv("AZURE_STORAGE_ACCOUNT_NAME"),
          #         azure_storage_account_key=os.getenv("AZURE_STORAGE_ACCOUNT_KEY")
          #     ),
          #     uri=os.getenv("LANCEDB_URI"),
          #     timeout="30s"
          # ),

          # For LanceDB OSS with data storage in a Google Cloud Storage bucket:
          # destination_connection_config=LanceDBGCSConnectionConfig(
          #     access_config=LanceDBGCSAccessConfig(
          #         google_service_account_key=os.getenv("GCS_SERVICE_ACCOUNT_KEY")
          #     ),
          #     uri=os.getenv("LANCEDB_URI"),
          #     timeout="30s"
          # ),

          stager_config=LanceDBUploadStagerConfig(),
          uploader_config=LanceDBUploaderConfig(table_name=os.gentenv("LANCEDB_TABLE"))
      ).run()
  ```
</CodeGroup>


# Local



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs locally.

You will need the local destination connector dependencies:

```bash CLI, Python
pip install unstructured-ingest
```

To use the local destination connector, you must set `--output-dir` (CLI) or `output_dir` (Python) to the path in the local filesystem which will contain the structured output.

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported.

The following 3-minute video shows how to use the Unstructured Ingest Python library to send multiple PDFs from a local directory in batches to be ingested by Unstructured API services for processing:

<iframe width="560" height="315" src="https://www.youtube.com/embed/tSKHFXsBQ-c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig,
      LocalUploaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import LocalRunner

  if __name__ == "__main__":
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
              recursive=True,
          ),
      )
      runner.run()
  ```
</CodeGroup>


# Milvus



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in Milvus.

You will need:

The Milvus prerequisites:

*   A [Milvus instance](https://milvus.io/docs/install-overview.md).
*   The [URI](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md) of the instance.
*   The [username and password, or token](https://milvus.io/docs/authenticate.md) to access the instance.
*   The name of the [database](https://milvus.io/docs/manage_databases.md) in the instance.
*   The name of the [collection](https://milvus.io/docs/manage-collections.md) in the database.

<iframe width="560" height="315" src="https://www.youtube.com/embed/tl9eV-_nAHI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The Milvus connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[milvus]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `MILVUS_URI` - The Milvus instance's URI, represented by `--uri` (CLI) or `uri` (Python).
*   `MILVUS_USER` and `MILVUS_PASSWORD`, or `MILVUS_TOKEN` - The username and password, or token, to access the instance. This is represented by `--user` and `--password`, or `--token` (CLI); or `user` and `password`, or `token` (Python).
*   `MILVUS_DB` - The database's name, represented by `--db-name` (CLI) or `db_name` (Python).
*   `MILVUS_COLLECTION` - The collection's name, represented by `--collection-name` (CLI) or `collection_name` (Python).
*   `MILVUS_FIELDS_TO_INCLUDE` - A list of fields to include a comma-separated list (CLI) or an array of strings (Python), represented by `--field-to-include` (CLI) or `fields_to_include` (Python).

Additional settings include:

*   To emit the `metadata` field's child fields directly into the output, include `--flatten-metadata` (CLI) or `flatten_metadata=True` (Python). This is the default if not specified.
*   To keep the `metadata` field with its child fields intact in the output, include `--no-flatten-metadata` (CLI) or `flatten_metadata=False` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    milvus \
      --uri $MILVUS_URI \
      --user $MILVUS_USER \
      --password $MILVUS_PASSWORD \
      --db-name $MILVUS_DB \
      --collection-name $MILVUS_COLLECTION \
      --fields-to-include type,element_id,text,embeddings
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.milvus import (
      MilvusConnectionConfig,
      MilvusAccessConfig,
      MilvusUploadStagerConfig,
      MilvusUploaderConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=MilvusConnectionConfig(
              access_config=MilvusAccessConfig(
                  password=os.getenv("MILVUS_PASSWORD")
              ),
              uri=os.getenv("MILVUS_URI"),
              user=os.getenv("MILVUS_USER"),
              db_name=os.getenv("MILVUS_DB")
          ),
          stager_config=MilvusUploadStagerConfig(fields_to_include=["type", "element_id", "text", "embeddings"]),
          uploader_config=MilvusUploaderConfig(collection_name=os.getenv("MILVUS_COLLECTION"))
      ).run()
  ```
</CodeGroup>


# MongoDB



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in MongoDB.

You will need:

The MongoDB prerequisites for a MongoDB Atlas deployment:

<iframe width="560" height="315" src="https://www.youtube.com/embed/g6qDfbg808M" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A MongoDB Atlas account. [Create an account](https://www.mongodb.com/cloud/atlas/register).
*   A MongoDB Atlas cluster. [Create a cluster](https://www.mongodb.com/docs/atlas/tutorial/deploy-free-tier-cluster).
*   The cluster must be reachable from your application environment. [Learn how](https://www.mongodb.com/docs/atlas/setup-cluster-security/#network-and-firewall-requirements).
*   The cluster must be configured to use your IP address. [Learn how](https://www.mongodb.com/docs/atlas/setup-cluster-security/#ip-access-list).
*   The cluster must have at least one database. [Create a database](https://www.mongodb.com/docs/compass/current/databases/#create-a-database).
*   The database must have at least one user, and that user must have sufficient access to the database. [Create a database user](https://www.mongodb.com/docs/atlas/security-add-mongodb-users/#add-database-users). [Give the user database access](https://www.mongodb.com/docs/manual/core/authorization/).
*   The database must have at least one collection. [Create a collection](https://www.mongodb.com/docs/compass/current/collections/#create-a-collection).
*   The connection string for the cluster. For MongoDB Atlas, this connection string must include the protocol, username, password, host, and cluster name. For example:

    ```text
    mongodb+srv://<db_user>:<db_password>@<host>/?retryWrites=true&w=majority&appName=<cluster>
    ```

    To get the connection string in MongoDB Atlas, do the following:

    1.  Log in to your MongoDB Atlas console.
    2.  In the sidebar, under **Databases**, click **Clusters**.
    3.  Click on the cluster you want to connect to.
    4.  Click **Connect**, or click the **Cmd Line Tools** tab and then click **Connect Instructions**.
    5.  Click **Drivers**.
    6.  Under **Add your connection string into your application code**, copy the connection string.
        You can then close the **Connect** dialog in MongoDB Atlas.

        Before you use this connection string, be sure to fill in any placeholders in the string, such as your MongoDB Atlas database user's password value.

    [Learn more](https://www.mongodb.com/resources/products/fundamentals/mongodb-connection-string).

The MongoDB connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[mongodb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

For a MongoDB Atlas deployment, the following environment variables:

*   `MONGODB_DATABASE` - The name of the database, represented by `--database` (CLI) or `database` (Python).

*   `MONGODB_COLLECTION` - The name of the collection in the database, represented by `--collection` (CLI) or `collection` (Python).

*   `MONGODB_URI` - The connection string for the cluster, represented by `--uri` (CLI) or `uri` (Python).

For a local MongoDB server, the following environment variables:

*   `MONGODB_HOST` - The host for the local MongoDB server, represented by `--host` (CLI) or `host` (Python).

*   `MONGODB_PORT` - The port for the local MongoDB server, represented by `--port` (CLI) or `port` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    mongodb \
      --uri $MONGODB_URI \
      --database $MONGODB_DATABASE \
      --collection $MONGODB_COLLECTION
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.mongodb import (
      MongoDBAccessConfig,
      MongoDBConnectionConfig,
      MongoDBUploadStagerConfig,
      MongoDBUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=MongoDBConnectionConfig(
              access_config=MongoDBAccessConfig(
                  uri=os.getenv("MONGODB_URI")
              ),
              database=os.getenv("MONGODB_DATABASE"),
              collection=os.getenv("MONGODB_COLLECTION")
          ),
          stager_config=MongoDBUploadStagerConfig(),
          uploader_config=MongoDBUploaderConfig(batch_size=100)
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.connector.mongodb import SimpleMongoDBConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
      WriteConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.mongodb import (
      MongodbWriter,
  )


  def get_writer() -> Writer:
      return MongodbWriter(
          connector_config=SimpleMongoDBConfig(
              uri=os.getenv("MONGODB_URI"),
              database=os.getenv("MONGODB_DATABASE"),
              collection=os.getenv("MONGODB_COLLECTION"),
          ),
          write_config=WriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>


# OneDrive



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a OneDrive account.

You will need:

The OneDrive prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/7uSu0LsbcEQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A OneDrive account.
*   The User Principal Name (UPN) for the OneDrive account. This is typically the OneDrive account user's email address.
*   The path to the target OneDrive folder, starting from the OneDrive account's root folder, for example `my-folder/my-subfolder`.
*   The client ID, tenant ID, and client secret for the Microsoft Entra ID app registration that has access to the target OneDrive account and
    also has the correct set of Microsoft Graph authentication scopes. These scopes include:

    *   `Files.ReadWrite.All` (if both reading and writing are needed)
    *   `Sites.ReadWrite.All` (if both reading and writing are needed)
    *   `User.Read.All`
    *   `Directory.Read.All`

See also:

*   [Registering your app for Microsoft Graph](https://learn.microsoft.com/onedrive/developer/rest-api/getting-started/app-registration)
*   [OneDrive authentication and sign-in](https://learn.microsoft.com/onedrive/developer/rest-api/getting-started/msa-oauth)
*   [OneDrive API documentation](https://docs.microsoft.com/onedrive/developer/rest-api/)

The OneDrive connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[onedrive]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `ONEDRIVE_PATH` - The path to the target folder in the OneDrive account, starting with the account's root folder (for example `my-folder/my-subfolder`), represented by `--path` (CLI) or `path` (Python).
*   `ONEDRIVE_CLIENT_ID` - The application (client) ID of the Microsoft Entra ID app registration that has access to the OneDrive account, represented by `--client-id` (CLI) or `client_id` (Python).
*   `ONEDRIVE_CLIENT_CRED` - The client secret for the Entra ID app registration, represented by `--client-cred` (CLI) or `client_cred` (Python).
*   `ONEDRIVE_TENANT` - The directory (tenant) ID of the Entra ID app registration, represented by `--tenant` (CLI) or `tenant` (Python).
*   `ONEDRIVE_USER_PNAME` - The User Principal Name (UPN) for the OneDrive user account in Entra ID (typically the user's email address), represented by `--user-pname` (CLI) or `user_pname` (Python).
*   `ONEDRIVE_AUTHORITY_URL` - The authentication token provider URL for the Entra ID app registration (the default is `https://login.microsoftonline.com` if not otherwise specified), represented by `--authority-url` (CLI) or `authority_url` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    onedrive \
      --client-cred $ONEDRIVE_CLIENT_CRED \
      --client-id $ONEDRIVE_CLIENT_ID \
      --user-pname $ONEDRIVE_USER_PNAME \
      --tenant $ONEDRIVE_TENANT \
      --authority-url $ONEDRIVE_AUTHORITY_URL \
      --remote-url $ONEDRIVE_PATH \
      --prefix "onedrive://"
  ```

  ```bash Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  from unstructured_ingest.v2.processes.connectors.onedrive import (
      OnedriveConnectionConfig,
      OnedriveAccessConfig,
      OnedriveUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=OnedriveConnectionConfig(
              access_config=OnedriveAccessConfig(client_cred=os.getenv("ONEDRIVE_CLIENT_CRED")),
              client_id=os.getenv("ONEDRIVE_CLIENT_ID"),
              user_pname=os.getenv("ONEDRIVE_USER_PNAME"),
              tenant=os.getenv("ONEDRIVE_TENANT"),
              authority_url=os.getenv("ONEDRIVE_AUTHORITY_URL")
          ),
          uploader_config=OnedriveUploaderConfig(
              remote_url=os.getenv("ONEDRIVE_PATH"),
              prefix="onedrive://"
          )
      ).run()
  ```
</CodeGroup>


# OpenSearch



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in OpenSearch.

You will need:

The OpenSearch prerequisites:

*   An OpenSearch instance, such as an [AWS OpenSearch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/setting-up.html) instance...

    <iframe width="560" height="315" src="https://www.youtube.com/embed/BNOYTbRbaFQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    ...or a [local](https://opensearch.org/downloads.html) instance.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/Rew3_pNnYIs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    In all cases, the [OpenSearch version](https://github.com/opensearch-project/opensearch-py/blob/main/COMPATIBILITY.md) must be compatible with the client version of the [opensearch-py](https://github.com/opensearch-project/opensearch-py/blob/main/COMPATIBILITY.md) package on PyPI.

*   The instance's hostname and port number. To learn how, see:

    *   [Creating and managing Amazon OpenSearch Service domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html) for AWS OpenSearch instances.
    *   [Communicate with OpenSearch](https://opensearch.org/docs/latest/getting-started/communicate/) for local OpenSearch instances.

*   The name of the search index on the instance. [Create an index](https://opensearch.org/docs/latest/api-reference/index-apis/create-index/).

    The OpenSearch index that you use must have a schema that is compatible with the schema of the documents
    that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following index schema example for your own needs:

    ```json
    {
        "settings": {
            "index": {
                "knn": true,
                "knn.algo_param.ef_search": 100
            }
        },
        "mappings": {
            "properties": {
                "element_id": {
                    "type": "keyword"
                },
                "text": {
                    "type": "text"
                },
                "embeddings": {
                    "type": "knn_vector",
                    "dimension": 384
                },
                "metadata": {
                    "type": "object",
                    "properties": {
                        "parent_id": {
                            "type": "text"
                        },
                        "page_number": {
                            "type": "integer"
                        },
                        "is_continuation": {
                           "type": "boolean"
                        },
                        "orig_elements": {
                           "type": "text"
                        }
                    }
                }
            }
        }
    }
    ```

    See also:

    *   [Mappings and field types](https://opensearch.org/docs/latest/field-types/)
    *   [Explicit mapping](https://opensearch.org/docs/latest/field-types/#explicit-mapping)
    *   [Dynamic mapping](https://opensearch.org/docs/latest/field-types/#dynamic-mapping)
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

*   If you're using basic authentication to the instance, the user's name and password.

*   If you're using certificates for authentication instead:

    *   The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA.
    *   The path to the combined private key and certificate file, or
    *   The paths to the separate private key and certificate file.

    To learn more, see:

    *   For AWS OpenSearch instances, see [Tutorial: Configure a domain with the internal user database and HTTP basic authentication](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac-http-auth.html) and [Creating a custom endpoint for Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/customendpoint.html).
    *   For local OpenSearch instances, see [Authentication backends](https://opensearch.org/docs/latest/security/authentication-backends/authc-index/), [HTTP basic authentication](https://opensearch.org/docs/latest/security/authentication-backends/basic-authc/), and [Client certificate authentication](https://opensearch.org/docs/latest/security/authentication-backends/client-auth/).

The OpenSearch connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[opensearch]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `OPENSEARCH_HOST` - The hostname and port number, defined as `<hostname>:<port-number>` and represented by `--hosts` (CLI) or `hosts` (Python).
*   `OPENSEARCH_INDEX_NAME` - The name of the search index, represented by `--index-name` (CLI) or `index_name` (Python).

If you're using basic authentication to the instance:

*   `OPENSEARCH_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
*   `OPENSEARCH_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

If you're using certificates for authentication instead:

*   `OPENSEARCH_CA_CERTS` - The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA. This is represented by `--ca-certs` (CLI) or `ca_certs` (Python).
*   `OPENSEARCH_CLIENT_CERT` - The path to the combined private key and certificate file, or the path to just the certificate file. This is represented by `--client-cert` (CLI) or `client_cert` (Python).
*   `OPENSEARCH_CLIENT_KEY` - The path to the private key file, if `OPENSEARCH_CLIENT_CERT` refers to just the certificate file. This is represented by `--client-key` (CLI) or `client_key` (Python).

Additional related settings include:

*   `--use-ssl` (CLI) or `use_ssl=True` (Python) to use SSL for the connection.
*   `--verify-certs` (CLI) or `verify_certs=True` (Python) to verify SSL certificates.
*   `--ssl-show-warn` (CLI) or `ssl_show_warn=True` (Python) to show a warning when verifying SSL certificates is disabled.

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    opensearch \
      --hosts $OPENSEARCH_HOST \
      --username $OPENSEARCH_USERNAME \
      --password $OPENSEARCH_PASSWORD \
      --index-name $OPENSEARCH_INDEX_NAME
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.opensearch import (
      OpenSearchConnectionConfig,
      OpenSearchAccessConfig,
      OpensearchUploadStagerConfig,
      OpensearchUploaderConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=OpenSearchConnectionConfig(
              access_config=OpenSearchAccessConfig(
                  password=os.getenv("OPENSEARCH_PASSWORD"),
                  use_ssl=True
              ),
              hosts=[os.getenv("OPENSEARCH_HOST")],
              username=os.getenv("OPENSEARCH_USERNAME")
          ),
          stager_config=OpensearchUploadStagerConfig(
              index_name=os.getenv("OPENSEARCH_INDEX_NAME")
          ),
          uploader_config=OpensearchUploaderConfig(
              index_name=os.getenv("OPENSEARCH_INDEX_NAME")
          )
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.elasticsearch import (
      ElasticsearchWriteConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.connector.opensearch import (
      OpenSearchAccessConfig,
      SimpleOpenSearchConfig,
  )
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.opensearch import (
      OpenSearchWriter,
  )


  def get_writer() -> Writer:
      return OpenSearchWriter(
          connector_config=SimpleOpenSearchConfig(
              access_config=OpenSearchAccessConfig(
                  hosts=os.getenv("OPENSEARCH_HOST"),
                  username=os.getenv("OPENSEARCH_USERNAME"),
                  password=os.getenv("OPENSEARCH_PASSWORD"),
              ),
              index_name=os.getenv("OPENSEARCH_INDEX_NAME"),
          ),
          write_config=ElasticsearchWriteConfig(
              batch_size_bytes=15_000_000,
              num_processes=2,
          ),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>


# Overview



Ingest and process data from a storage with a source connector, and move the results to the data storage platform of
your choice using a destination connector.

All the examples for destination connector use local source connector. You can replace it with any other source connector.

We are constantly adding new destination connectors and if you don't see your favorite platform let us know in our community Slack.

<Info>All connectors can be used by the Unstructured Ingest CLI and the Unstructured Ingest Python library. The Unstructured JavaScript/TypeScript SDK supports only local sources/destinations on the same machine.</Info>


# Pinecone



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in Pinecone.

You will need:

The Pinecone prerequisites:

*   A Pinecone account. [Get an account](https://app.pinecone.io/).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/uPoa7PLcuZ4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A Pinecone API key. [Get an API key](https://docs.pinecone.io/guides/get-started/authentication#find-your-pinecone-api-key).

*   A Pinecone serverless index. [Create a serverless index](https://docs.pinecone.io/guides/indexes/create-an-index).

The Pinecone connector dependencies.

```bash
pip install "unstructured-ingest[pinecone]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `PINECONE_API_KEY` - The Pinecone API, represented by `--api-key` (CLI) or `api_key` (Python, in the `PineconeAccessConfig` object).
*   `PINECONE_INDEX_NAME` - The Pinecone serverless index name, represented by `--index-name` (CLI) or `index_name` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported.

<iframe width="560" height="315" src="https://www.youtube.com/embed/FZNgYi-fxdg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    pinecone \
      --api-key "$PINECONE_API_KEY" \
      --index-name "$PINECONE_INDEX_NAME" \
      --batch-size 80
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.pinecone import (
      PineconeConnectionConfig,
      PineconeAccessConfig,
      PineconeUploaderConfig,
      PineconeUploadStagerConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=PineconeConnectionConfig(
              access_config=PineconeAccessConfig(
                  api_key=os.getenv("PINECONE_API_KEY")
              ),
              index_name=os.getenv("PINECONE_INDEX_NAME")
          ),
          stager_config=PineconeUploadStagerConfig(),
          uploader_config=PineconeUploaderConfig()
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.connector.pinecone import (
      PineconeAccessConfig,
      PineconeWriteConfig,
      SimplePineconeConfig,
  )
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.pinecone import (
      PineconeWriter,
  )


  def get_writer() -> Writer:
      return PineconeWriter(
          connector_config=SimplePineconeConfig(
              access_config=PineconeAccessConfig(api_key=os.getenv("PINECONE_API_KEY")),
              index_name=os.getenv("PINECONE_INDEX_NAME")
          ),
          write_config=PineconeWriteConfig(batch_size=80),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>


# PostgreSQL



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a PostgreSQL schema.

Insert query is currently limited to append.

You will need:

The PostgreSQL prerequisites, which include the following settings.

The following video shows for example how to get these settings by using [Amazon RDS for PostgreSQL](https://aws.amazon.com/rds/postgresql/).

<iframe width="560" height="315" src="https://www.youtube.com/embed/QuIlEimejDs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A PostgreSQL instance. [Install PostgreSQL](https://www.postgresql.org/docs/current/tutorial-install.html).

*   The host name and port number for the instance. These values are in the `postgresql.conf` file's `listen_addresses` and `port` settings. This file should be on the same machine as the instance. These values might also already be set as environment variables named `PGHOST` and `PGPORT` on the same machine as the instance.

*   A database in the instance. [Create a database](https://www.postgresql.org/docs/current/tutorial-createdb.html).

*   A table in the database. [Create a table](https://www.postgresql.org/docs/current/tutorial-table.html).

    The table's schema must match the schema of the documents that Unstructured produces.
    Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following table schema example for your own needs:

    <CodeGroup>
      ```sql PostgreSQL
      CREATE TABLE elements (
          id UUID PRIMARY KEY,
          element_id VARCHAR,
          text TEXT,
          embeddings DECIMAL [],
          parent_id VARCHAR,
          page_number INTEGER,
          is_continuation BOOLEAN,
          orig_elements TEXT
      );
      ```

      ```sql PostgreSQL with pgvector 
      CREATE EXTENSION vector;

      CREATE TABLE elements (
          id UUID PRIMARY KEY,
          element_id VARCHAR,
          text TEXT,
          embeddings vector(384),
          parent_id VARCHAR,
          page_number INTEGER,
          is_continuation BOOLEAN,
          orig_elements TEXT
      );
      ```
    </CodeGroup>

    See also:

    *   [CREATE TABLE](https://www.postgresql.org/docs/current/sql-createtable.html) for PostgreSQL
    *   [CREATE TABLE](https://github.com/pgvector/pgvector) for PostrgreSQL with pgvector
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

*   A user in the database, and a password for the user. [Create a user](https://www.postgresql.org/docs/current/sql-createuser.html).

*   Database access for the user. [Give database access to a user](https://www.postgresql.org/docs/current/sql-grant.html).

The PostgreSQL connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[postgres]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `PGHOST` - The host name, represented by `--host` (CLI) or `host` (Python).
*   `PGPORT` - The port number, represented by `--port` (CLI) or `port` (Python).
*   `PGUSER` - The username, represented by `--username` (CLI) or `username` (Python).
*   `PGPASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).
*   `PGDATABASE` - The name of the database, represented by `--database` (CLI) or `database` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Specify which fields to output in the processed data. This can help prevent
  # database record insert issues, where a particular field in the processed data
  # does not match a column in the database table on insert.
  metadata_includes="id,element_id,text,embeddings,type,system,layout_width,\
  layout_height,points,url,version,date_created,date_modified,date_processed,\
  permissions_data,record_locator,category_depth,parent_id,attached_filename,\
  filetype,last_modified,file_directory,filename,languages,page_number,links,\
  page_name,link_urls,link_texts,sent_from,sent_to,subject,section,\
  header_footer_type,emphasized_text_contents,emphasized_text_tags,\
  text_as_html,regex_metadata,detection_class_prob"

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --strategy hi_res \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --metadata-include "$metadata_includes" \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    postgres \
      --username $PGUSER \
      --password $PGPASSWORD \
      --host $PGHOST \
      --port $PGPORT \
      --database $PGDATABASE
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.sql.postgres import (
      PostgresConnectionConfig,
      PostgresAccessConfig,
      PostgresUploaderConfig,
      PostgresUploadStagerConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      # Specify which fields to output in the processed data. This can help prevent
      # database record insert issues, where a particular field in the processed data
      # does not match a column in the database table on insert.
      metadata_includes = [
          "id", "element_id", "text", "embeddings", "type", "system", "layout_width",
          "layout_height", "points", "url", "version", "date_created", "date_modified",
          "date_processed", "permissions_data", "record_locator", "category_depth",
          "parent_id", "attached_filename", "filetype", "last_modified", "file_directory",
          "filename", "languages", "page_number", "links", "page_name", "link_urls",
          "link_texts", "sent_from", "sent_to", "subject", "section", "header_footer_type",
          "emphasized_text_contents", "emphasized_text_tags", "text_as_html", "regex_metadata",
          "detection_class_prob"
      ]

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              metadata_include=metadata_includes,
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=PostgresConnectionConfig(
              access_config=PostgresAccessConfig(password=os.getenv("PGPASSWORD")),
              host=os.getenv("PGHOST"),
              port=os.getenv("PGPORT"),
              username=os.getenv("PGUSER"),
              database=os.getenv("PGDATABASE")
          ),
          stager_config=PostgresUploadStagerConfig(),
          uploader_config=PostgresUploaderConfig()
      ).run()
  ```
</CodeGroup>


# Qdrant



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in Qdrant.

You will need:

The Qdrant prerequisites are as follows.

*   The name of the target [collection](https://qdrant.tech/documentation/concepts/collections) on the Qdrant local installation,
    Qdrant server, or Qdrant Cloud cluster.
*   For [Qdrant local](https://github.com/qdrant/qdrant), the path to the local Qdrant installation, for example: `/qdrant/local`
*   For [Qdrant client-server](https://qdrant.tech/documentation/quickstart/), the Qdrant server URL, for example: `http://localhost:6333`
*   For [Qdrant Cloud](https://qdrant.tech/documentation/cloud-intro/):

    *   A [Qdrant account](https://cloud.qdrant.io/login).

    *   A [Qdrant cluster](https://qdrant.tech/documentation/cloud/create-cluster/).

    *   The cluster's URL. To get this URL, do the following:

        1.  Sign in to your Qdrant Cloud account.
        2.  On the sidebar, under **Dashboard**, click **Clusters**.
        3.  Click the cluster's name.
        4.  Note the value of the **Endpoint** field, for example: `https://<random-guid>.<region-id>.<cloud-provider>.cloud.qdrant.io`.

    *   A [Qdrant API key](https://qdrant.tech/documentation/cloud/authentication/#create-api-keys).

The Qdrant connector dependencies.

```bash
pip install "unstructured-ingest[qdrant]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `QDRANT_COLLECTION` - The name of the target collection on the Qdrant local installation,
    Qdrant server, or Qdrant Cloud cluster, represented by `--collection-name` (CLI) or `collection_name` (Python).
*   For Qdrant local, `QDRANT_PATH` - The path to the local Qdrant installation, represented by `--path` (CLI) or `path` (Python).
*   For Qdrant client-server, `QDRANT_URL` - The Qdrant server's URL, represented by `--url` (CLI) or `url` (Python).
*   For Qdrant Cloud:

    *   `QDRANT_URL` - The Qdrant cluster's URL, represented by `--url` (CLI) or `url` (Python).
    *   `QDRANT_API_KEY` - The Qdrant API key, represented by `--api-key` (CLI) or `api_key` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported.

This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  # For Qdrant local:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    qdrant-local \
      --path $QDRANT_PATH \
      --collection-name $QDRANT_COLLECTION \
      --batch-size 50 \
      --num-processes 1

  # For Qdrant client-server:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    qdrant-server \
      --url $QDRANT_URL \
      --collection-name $QDRANT_COLLECTION \
      --batch-size 50 \
      --num-processes 1

  # For Qdrant cloud:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    qdrant-cloud \
      --url $QDRANT_URL \
      --api-key $QDRANT_API_KEY \
      --collection-name $QDRANT_COLLECTION \
      --batch-size 50 \
      --num-processes 1
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # For Qdrant local:
  # from unstructured_ingest.v2.processes.connectors.qdrant.local import (
  #     LocalQdrantConnectionConfig,
  #     LocalQdrantAccessConfig,
  #     LocalQdrantUploadStagerConfig,
  #     LocalQdrantUploaderConfig
  # )

  # For Qdrant client-server:
  # from unstructured_ingest.v2.processes.connectors.qdrant.server import (
  #     ServerQdrantConnectionConfig,
  #     ServerQdrantAccessConfig,
  #     ServerQdrantUploadStagerConfig,
  #     ServerQdrantUploaderConfig
  # )

  # For Qdrant Cloud:
  from unstructured_ingest.v2.processes.connectors.qdrant.cloud import (
      CloudQdrantConnectionConfig,
      CloudQdrantAccessConfig,
      CloudQdrantUploadStagerConfig,
      CloudQdrantUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),

          # For Qdrant local:
          # destination_connection_config=LocalQdrantConnectionConfig(
          #     access_config=LocalQdrantAccessConfig(),
          #     path=os.getenv("QDRANT_PATH")
          # ),
          # stager_config=LocalQdrantUploadStagerConfig(),
          # uploader_config=LocalQdrantUploaderConfig(
          #     collection_name=os.gentenv("QDRANT_COLLECTION"),
          #     batch_size=50,
          #     num_processes=1
          # )

          # For Qdrant client-server:
          # destination_connection_config=ServerQdrantConnectionConfig(
          #     access_config=ServerQdrantAccessConfig(),
          #     url=os.getenv("QDRANT_URL")
          # ),
          # stager_config=ServerQdrantUploadStagerConfig(),
          # uploader_config=ServerQdrantUploaderConfig(
          #     collection_name=os.gentenv("QDRANT_COLLECTION"),
          #     batch_size=50,
          #     num_processes=1
          # )

          # For Qdrant cloud:
          destination_connection_config=CloudQdrantConnectionConfig(
              access_config=CloudQdrantAccessConfig(
                  api_key=os.getenv("QDRANT_API_KEY")
              ),
              url=os.getenv("QDRANT_URL")
          ),
          stager_config=CloudQdrantUploadStagerConfig(),
          uploader_config=CloudQdrantUploaderConfig(
              collection_name=os.gentenv("QDRANT_COLLECTION"),
              batch_size=50,
              num_processes=1
          )
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.connector.qdrant import (
      QdrantWriteConfig,
      SimpleQdrantConfig,
      QdrantAccessConfig,
  )
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.qdrant import QdrantWriter

  # This example uses Qdrant Cloud.

  def get_writer() -> Writer:
      return QdrantWriter(
          connector_config=SimpleQdrantConfig(
              url=os.getenv("QDRANT_URL"),
              access_config=QdrantAccessConfig(api_key=os.getenv("QDRANT_API_KEY")),
              collection_name=os.getenv("QDRANT_COLLECTION"),
          ),
          write_config=QdrantWriteConfig(batch_size=80),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>


# S3



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in an S3 bucket.

You will need:

The Amazon S3 prerequisites:

The following video shows how to fulfill the minimum set of S3 prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/hyDHfhVVAhs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but
not recommended); or generate an AWS STS session token for temporary access, if required by your organization's security
requirements. For more information about prerequisites, see the following:

*   An AWS account. [Create an AWS account](https://aws.amazon.com/free).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/lIdh92JmWtg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An S3 bucket. [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
    Additional approaches are in the following video and in the how-to sections at the end of this page.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/e6w9LwZJFIA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   Anonymous (supported but not recommended) or authenticated access to the bucket.

    *   [Enable anonymous bucket access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user).

        ![Enable anonymous bucket access](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/connectors/s3-anon-access.gif)

    *   [Enable authenticated bucket access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/walkthrough1.html).

        ![Enable authenticated bucket access](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/connectors/s3-auth-access.gif)

*   For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/y4SfQoJpipo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at
    minimum the permission of `s3:PutObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

*   For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account.
    [Create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/MoFTaGJE65Q" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For authenticated access in untrusted environments or enhanced security scenarios, an AWS STS session token for temporary access, in addition to an AWS access key and secret access key. [Create a session token](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_getsessiontoken).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/4_csSXc_GNU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   If the target files are in the root of the bucket, the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
    If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

*   If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has
    authenticated access to the folder as well. [Enable authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

The S3 connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[s3]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `AWS_S3_URL` - The path to the S3 bucket or folder, formatted as `s3://my-bucket/` (if the files are in the bucket's root) or `s3://my-bucket/my-folder/`.

*   If the bucket does not have anonymous access enabled, provide the AWS credentials:

    *   `AWS_ACCESS_KEY_ID` - The AWS access key ID for the authenticated AWS IAM user, represented by `--key` (CLI) or `key` (Python).
    *   `AWS_SECRET_ACCESS_KEY` - The corresponding AWS secret access key, represented by `--secret` (CLI) or `secret` (Python).
    *   `AWS_TOKEN` - If required, the AWS STS session token for temporary access, represented by `--token` (CLI) or `token` (Python).

*   If the bucket has anonymous access enabled for reading from the bucket, set `--anonymous` (CLI) or `anonymous=True` (Python) instead.

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    s3 \
      --remote-url $AWS_S3_URL \
      --key $AWS_ACCESS_KEY_ID \
      --secret $AWS_SECRET_ACCESS_KEY
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.fsspec.s3 import (
      S3ConnectionConfig,
      S3AccessConfig,
      S3UploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=S3ConnectionConfig(
              access_config=S3AccessConfig(
                  key=os.getenv("AWS_ACCESS_KEY_ID"),
                  secret=os.getenv("AWS_SECRET_ACCESS_KEY")
              )
          ),
          uploader_config=S3UploaderConfig(remote_url=os.getenv("AWS_S3_URL"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.s3 import S3AccessConfig, S3WriteConfig, SimpleS3Config
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.fsspec.s3 import (
      S3Writer,
  )

  def get_writer() -> Writer:
      return S3Writer(
          connector_config=SimpleS3Config(
              access_config=S3AccessConfig(anon=True),
              remote_url=os.getenv("AWS_S3_URL"),
          ),
          write_config=S3WriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2.  Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3.  Browse to the existing bucket and open it.

4.  Click the **Permissions** tab.

5.  In the **Bucket policy** area, click **Edit**.

6.  In the **Policy** text area, copy the following JSON-formatted policy.
    To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this policy, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-bucket-name>` in two places with the name of your bucket.

    ```json
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::<my-account-id>:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::<my-bucket-name>",
                    "arn:aws:s3:::<my-bucket-name>/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    ```

7.  Click **Save changes**.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2.  [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3.  Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
    To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this script, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-unique-bucket-name>` with the name of your bucket.
    *   Replace `<us-east-1>` with your AWS Region.

    ```bash
    #!/bin/bash

    # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
    ACCOUNT_ID="<my-account-id>"
    BUCKET_NAME="<my-unique-bucket-name>"
    REGION="<us-east-1>"

    # Temporary filename for the bucket policy.
    # Do not change this variable.
    POLICY_FILE="bucket_policy.json"

    # Create the bucket.
    aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

    # Wait for the bucket to exist.
    echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
    aws s3api wait bucket-exists --bucket $BUCKET_NAME

    # Check if the wait command was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket '$BUCKET_NAME' has been fully created."
    else
        echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
        exit 1
    fi

    # Remove the "block public policy" bucket access setting.
    aws s3api put-public-access-block \
        --bucket $BUCKET_NAME \
        --public-access-block-configuration \
        '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

    # Check if the operation was successful.
    if [ $? -eq 0 ]; then
        echo "The block public policy access setting was removed from '$BUCKET_NAME'."
    else
        echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
        exit 1
    fi

    # Create the bucket policy.
    cat << EOF > $POLICY_FILE
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::$BUCKET_NAME",
                    "arn:aws:s3:::$BUCKET_NAME/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    EOF

    # Apply the bucket policy.
    aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

    # Check if the policy application was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket policy was applied to '$BUCKET_NAME'."
    else
        echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
        exit 1
    fi

    # Verify the applied policy.
    echo "Verifying the applied policy:"
    aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

    # Remove the temporary bucket policy file.
    rm $POLICY_FILE
    ```

4.  Run the script, for example:

    ```bash
    sh create-s3-bucket.sh
    ```

5.  After the bucket is created, you can delete the script file, if you want.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
    the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    ```yaml
    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

    Parameters:
      BucketName:
        Type: String
        Description: 'Name of the S3 bucket to create'

    Resources:
      MyS3Bucket:
        Type: 'AWS::S3::Bucket'
        Properties:
          BucketName: !Ref BucketName
          PublicAccessBlockConfiguration:
            BlockPublicAcls: true
            BlockPublicPolicy: false
            IgnorePublicAcls: true
            RestrictPublicBuckets: true

      BucketPolicy:
        Type: 'AWS::S3::BucketPolicy'
        Properties:
          Bucket: !Ref MyS3Bucket
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Sid: AllowAllAuthenticatedUsersInAccount
                Effect: Allow
                Principal:
                  AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
                Action:
                  - 's3:GetObject'
                  - 's3:PutObject'
                  - 's3:ListBucket'
                  - 's3:DeleteObject'
                Resource:
                  - !Sub 'arn:aws:s3:::${BucketName}'
                  - !Sub 'arn:aws:s3:::${BucketName}/*'

    Outputs:
      BucketName:
        Description: 'Name of the created S3 bucket'
        Value: !Ref MyS3Bucket
    ```

2.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3.  Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4.  Click **Create stack > With new resources (standard)**.

5.  On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6.  Click **Choose file**, and browse to and select the YAML file from your local machine.

7.  Click **Next**.

8.  Enter a unique **Stack name** and **BucketName**.

9.  Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.


# SFTP



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in SFTP.

You will need:

The SFTP prerequisites:

*   The SFTP server hostname, port, username, and password.

    SFTP servers are offered by several vendors. For example, the following video shows how to create and set up an SFTP server by using AWS Transfer Family:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/3_HHSnoFsoM" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The directory path to start accessing data from, specified as `sftp://<path>/<to>/<directory>`.

See the [SFTP documentation](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol).

The SFTP connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[sftp]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `SFTP_HOST` - The SFTP hostname, represented by `--host` (CLI) or `host` (Python).
*   `SFTP_PORT` - The SFTP port number, represented by `--port` (CLI) or `port` (Python).
*   `SFTP_REMOTE_URL` - The directory path to start accessing data from, represented by `--remote-url` (CLI) or `remote_url` (Python).
*   `SFTP_USERNAME` - The SFTP username, represented by `--username` (CLI) or `username` (Python).
*   `SFTP_PASSWORD` - The SFTP password, represented by `--password` (CLI) or `password` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --strategy hi_res \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
    sftp \
      --host $SFTP_HOST \
      --port $SFTP_PORT \
      --username $SFTP_USERNAME \
      --password $SFTP_PASSWORD \
      --remote-url $SFTP_REMOTE_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.sftp import (
      SftpConnectionConfig,
      SftpAccessConfig,
      SftpUploaderConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=SftpConnectionConfig(
              access_config=SftpAccessConfig(password=os.getenv("SFTP_PASSWORD")),
              host=os.getenv("SFTP_HOST"),
              port=os.getenv("SFTP_PORT"),
              username=os.getenv("SFTP_USERNAME")
          ),
          uploader_config=SftpUploaderConfig(remote_url=os.getenv("SFTP_REMOTE_URL"))
      ).run()
  ```
</CodeGroup>


# SingleStore



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a SingleStore account.

You will need:

The SingleStore prerequisites:

*   A SingleStore deployment, database, and table. [Learn how](https://www.singlestore.com/blog/how-to-get-started-with-singlestore/).
*   The hostname for the SingleStore deployment.
*   The port for the host.
*   The username for the deployment.
*   The password for the user.
*   The name of the database in the deployment.
*   The name of the table in the database.

To get the values for the hostname, port, username, and password:

1.  In your SingleStore account's dashboard sidebar, click **Deployments**.
2.  From the drop-down list at the top of the **Deployments** page, select your deployment.
3.  On the **Overview** tab, in the **Compute** area, in the **Connect** drop-down list for your deployment, select
    **Your App**.
4.  If a **Create User** dialog box appears, note the **User** name and **Password** values.
5.  In the **Connect to Workspace** pane's **Your App** tab, note the string in the following format:

```
<user-name>:<password>@<host>:<port>
```

*   `<user-name>` is the username.
*   `<password>` is the user's password.
*   `<host>` is the workspace's hostname.
*   `<post>` is the host's port.

To get the values for the database and table names:

1.  In your SingleStore dashboard's sidebar, click **Deployments**.
2.  From the drop-down list at the top of the **Deployments** page, select your deployment.
3.  On the **Databases** tab, note **Name** of your database.
4.  Click the database and, on the **Tables** tab, note the name of your table.

The SingleStore connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[singlestore]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `SINGLESTORE_HOST` - The hostname for the SingleStore deployment, represented by `--host` (CLI) or `host` (Python).
*   `SINGLESTORE_PORT` - The port for the host, represented by `--port` (CLI) or `port` (Python).
*   `SINGLESTORE_USER` - The username for the deployment, represented by `--user` (CLI) or `user` (Python).
*   `SINGLESTORE_PASSWORD` - The password for the user, represented by `--password` (CLI) or `password` (Python).
*   `SINGLESTORE_DB` - The name of the database in the deployment, represented by `--database` (CLI) or `database` (Python).
*   `SINGLESTORE_TABLE` - The name of the table in the database, represented by `--table-name` (CLI) or `table_name` (Python).

The table's schema must match the schema of the documents that Unstructured produces.

During insertion, JSON objects are flattened, and underscores are placed between nested object names.
For example, the following JSON object matches a column in the table named `data_source_date_created`:

```json
{
    "...": "...",
    "data_source": {
        "date_created": "1719963233.949"
    },
    "...": "..."
}
```

The `metadata` object itself is also flattened in a similar way. However, for nested objects in `metadata`,
the column in the table does not start with `metadata_`. For example, the following JSON object matches a
column in the table named `last_modified`:

```json
{
    "...": "...",
    "metadata": {
        "...": "...",
        "last_modified": "2022-12-16T17:04:16-05:00",
        "...": "..."
    },
    "...": "..."
}
```

Unstructured cannot provide a table schema that is guaranteed to work in all
circumstances. This is because these schemas will vary based on your source files' types; how you
want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

You can adapt the following table schema example for your own needs:

```sql
CREATE TABLE elements (
    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
    element_id TEXT,
    text TEXT,
    embeddings Vector(384),
    parent_id TEXT,
    page_number TEXT,
    is_continuation BOOLEAN,
    orig_elements TEXT
);
```

See also:

*   [CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table/)
    in the SingleStore documentation
*   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    singlestore \
      --host $SINGLESTORE_HOST \
      --port $SINGLESTORE_PORT \
      --user $SINGLESTORE_USER \
      --password $SINGLESTORE_PASSWORD \
      --database $SINGLESTORE_DB \
      --table-name $SINGLESTORE_TABLE
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.singlestore import (
      SingleStoreConnectionConfig,
      SingleStoreAccessConfig,
      SingleStoreUploadStagerConfig,
      SingleStoreUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=SingleStoreConnectionConfig(
              host=os.getenv("SINGLESTORE_HOST"),
              port=os.getenv("SINGLESTORE_PORT"),
              user=os.getenv("SINGLESTORE_USER"),
              database=os.getenv("SINGLESTORE_DB"),
              access_config=SingleStoreAccessConfig(
                  password=os.getenv("SINGLESTORE_PASSWORD")
              )
          ),
          stager_config=SingleStoreUploadStagerConfig(drop_empty_cols=True),
          uploader_config=SingleStoreUploaderConfig(table_name=os.getenv("SINGLESTORE_TABLE"))
      ).run()
  ```
</CodeGroup>


# Snowflake



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in Snowflake.

You will need:

The Snowflake prerequisites:

*   A Snowflake [account](https://signup.snowflake.com/) and its [identifier](https://docs.snowflake.com/user-guide/admin-account-identifier).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/6HNmxQWqmgg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Snowflake [username and its password](https://docs.snowflake.com/user-guide/admin-user-management#creating-users) in the account.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/5hPrJNEa4aA?start=64&end=267" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Snowflake [hostname and its port number](https://docs.snowflake.com/sql-reference/functions/system_allowlist) in the account.

*   The name of the Snowflake [database](https://docs.snowflake.com/sql-reference/sql/create-database) in the account.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/KrXCgpsRd0A" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The name of the [schema](https://docs.snowflake.com/sql-reference/sql/create-schema) in the database.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/4SoS_c6rT_U?start=97&end=157" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The name of the [table](https://docs.snowflake.com/sql-reference/sql/create-table) in the schema.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/ZaZzqFMGe1g?end=169" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The Snowflake connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[snowflake]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `SNOWFLAKE_ACCOUNT` - The ID of the Snowflake account, represented by `--account` (CLI) or `account` (Python).
*   `SNOWFLAKE_USER`  - The name of the Snowflake user, represented by `--user` (CLI) or `user` (Python).
*   `SNOWFLAKE_PASSWORD`  - The user's password, represented by `--password` (CLI) or `password` (Python).
*   `SNOWFLAKE_HOST`  - The hostname for the Snowflake account, represented by `--host` (CLI) or `host` (Python).
*   `SNOWFLAKE_PORT`  - The host's port number, represented by `--port` (CLI) or `port` (Python).
*   `SNOWFLAKE_DATABASE`  - The name of the Snowflake database, represented by `--database` (CLI) or `database` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    snowflake \
      --account $SNOWFLAKE_ACCOUNT \
      --user $SNOWFLAKE_USER \
      --password $SNOWFLAKE_PASSWORD \
      --host $SNOWFLAKE_HOST \
      --port $SNOWFLAKE_PORT \
      --database $SNOWFLAKE_DATABASE \
      --schema PUBLIC \
      --table-name elements \
      --batch-size 50
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.sql.snowflake import (
      SnowflakeConnectionConfig,
      SnowflakeAccessConfig,
      SnowflakeUploadStagerConfig,
      SnowflakeUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=SnowflakeConnectionConfig(
              access_config=SnowflakeAccessConfig(password=os.getenv("SNOWFLAKE_PASSWORD")),
              account=os.getenv("SNOWFLAKE_ACCOUNT"),
              user=os.getenv("SNOWFLAKE_USER"),
              host=os.getenv("SNOWFLAKE_HOST"),
              port=os.getenv("SNOWFLAKE_PORT"),
              database=os.getenv("SNOWFLAKE_DATABASE"),
              schema="PUBLIC"
          ),
          stager_config=SnowflakeUploadStagerConfig(),
          uploader_config=SnowflakeUploaderConfig(
              batch_size=50,
              table_name="elements"
          )
      ).run()
  ```
</CodeGroup>


# SQLite



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a SQLite schema.

Insert query is currently limited to append.

You will need:

The SQLite prerequisites:

*   A SQLite instance. [Download and install SQLite](https://www.sqlitetutorial.net/download-install-sqlite/).
*   A SQLite database. [Create a database](https://www.sqlite.org/quickstart.html).
*   The path to the database's `.db` file.
*   A table in the database. [Create a table](https://www.sqlite.org/cli.html).

    The table's schema must match the schema of the documents that Unstructured produces.
    Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following table schema example for your own needs:

    ```sql SQLite
    CREATE TABLE elements (
        id TEXT PRIMARY KEY,
        element_id TEXT,
        text TEXT,
        embeddings TEXT,
        parent_id TEXT,
        page_number INTEGER,
        is_continuation INTEGER,
        orig_elements TEXT
    );
    ```

    See also:

    *   [CREATE TABLE](https://www.sqlite.org/lang_createtable.html) for SQLite
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `SQLITE_DB_PATH` - The path to the database's `.db` file, represented by `--database` (CLI) or `database` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Specify which fields to output in the processed data. This can help prevent
  # database record insert issues, where a particular field in the processed data
  # does not match a column in the database table on insert.
  metadata_includes="id,element_id,text,embeddings,type,system,layout_width,\
  layout_height,points,url,version,date_created,date_modified,date_processed,\
  permissions_data,record_locator,category_depth,parent_id,attached_filename,\
  filetype,last_modified,file_directory,filename,languages,page_number,links,\
  page_name,link_urls,link_texts,sent_from,sent_to,subject,section,\
  header_footer_type,emphasized_text_contents,emphasized_text_tags,\
  text_as_html,regex_metadata,detection_class_prob"

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --strategy hi_res \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --metadata-include "$metadata_includes" \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    sqlite \
      --database-path $SQLITE_DB_PATH
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.sql.sqlite import (
      SQLiteConnectionConfig,
      SQLiteAccessConfig,
      SQLiteUploaderConfig,
      SQLiteUploadStagerConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      # Specify which fields to output in the processed data. This can help prevent
      # database record insert issues, where a particular field in the processed data
      # does not match a column in the database table on insert.
      metadata_includes = [
          "id", "element_id", "text", "embeddings", "type", "system", "layout_width",
          "layout_height", "points", "url", "version", "date_created", "date_modified",
          "date_processed", "permissions_data", "record_locator", "category_depth",
          "parent_id", "attached_filename", "filetype", "last_modified", "file_directory",
          "filename", "languages", "page_number", "links", "page_name", "link_urls",
          "link_texts", "sent_from", "sent_to", "subject", "section", "header_footer_type",
          "emphasized_text_contents", "emphasized_text_tags", "text_as_html", "regex_metadata",
          "detection_class_prob"
      ]

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              metadata_include=metadata_includes,
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=SQLiteConnectionConfig(
              access_config=SQLiteAccessConfig(),
              database_path=os.getenv("SQLITE_DB_PATH")
          ),
          stager_config=SQLiteUploadStagerConfig(),
          uploader_config=SQLiteUploaderConfig()
      ).run()
  ```
</CodeGroup>


# Vectara



Process all your records using `unstructured-ingest` to store structured outputs locally on your filesystem and upload those to a Vectara corpus. If you don’t yet have a Vectara account, [sign up](https://console.vectara.com/signup/) for one now.

The upstream connector can be any of the ones supported, but for convenience here, showing a sample command using the upstream local connector.

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  # Chunking is optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --num-processes 2 \
      --verbose \
    vectara \
      --customer-id $VECTARA_CUSTOMER_ID \
      --oauth-client-id $VECTARA_OAUTH_CLIENT_ID \
      --oauth-secret $VECTARA_OAUTH_SECRET \
      --corpus-name test-corpus-vectara
  ```

  ```python Python
  import os

  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.connector.vectara import (
      SimpleVectaraConfig,
      VectaraAccessConfig,
      WriteConfig,
  )
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.vectara import (
      VectaraWriter,
  )


  def get_writer() -> Writer:
      return VectaraWriter(
          connector_config=SimpleVectaraConfig(
              access_config=VectaraAccessConfig(
                  oauth_client_id=os.getenv("VECTARA_OAUTH_CLIENT_ID"),
                  oauth_secret=os.getenv("VECTARA_OAUTH_SECRET"),
              ),
              customer_id=os.getenv("VECTARA_CUSTOMER_ID"),
              corpus_name="test-corpus-vectara",
          ),
          write_config=WriteConfig(),
      )


  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For a full list of the options the Unstructured Ingest CLI accepts check `unstructured-ingest <upstream connector> vectara --help`.

NOTE: Keep in mind that you will need to have all the appropriate extras and dependencies for the file types of the documents contained in your data storage platform if you’re running this locally. You can find more information about this in the [installation guide](/open-source/installation/overview).


# Weaviate



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a Weaviate database.

<iframe width="560" height="315" src="https://www.youtube.com/embed/uqUrH8ksI0I" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

You will need:

The Weaviate prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/Ldb7PZU-pR4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

1.  A Weaviate database instance. The following information assumes that you have a Weaviate Cloud (WCD) account with a Weaviate database cluster in that account.
    [Create a WCD account](https://weaviate.io/developers/wcs/quickstart#create-a-wcd-account). [Create a database cluster](https://weaviate.io/developers/wcs/quickstart#create-a-weaviate-cluster). For other database options, [learn more](https://weaviate.io/developers/weaviate/installation).

2.  The URL and API key for the database cluster. [Get the URL and API key](https://weaviate.io/developers/wcs/quickstart#explore-the-details-panel).

3.  A collection in the database cluster. Note the name of the collection, also known as the collection's *class name*. [Create a collection](https://weaviate.io/developers/wcs/tools/collections-tool).

    The schema of the collection that you use must match the data that Unstructured writes to it. Otherwise, you might get unexpected results or errors.
    Unstructured cannot provide a schema that is guaranteed to work for everyone in all circumstances. This is because these schemas will vary based on
    your source files' types; how you want Unstructured to partition, chunk, and generate embeddings;
    any custom post-processing code that you run; and other factors.

    You can adapt the following collection schema example for your own needs:

    ```json
    {
        "class": "Elements",
        "properties": [
            {
                "name": "element_id",
                "dataType": ["text"]
            },
            {
                "name": "text",
                "dataType": ["text"]
            },
            {
                "name": "embeddings",
                "dataType": ["number[]"]
            },
            {
                "name": "metadata",
                "dataType": ["object"],
                "nestedProperties": [
                    {
                        "name": "parent_id",
                        "dataType": ["text"]
                    },
                    {
                        "name": "page_number",
                        "dataType": ["text"]
                    },
                    {
                        "name": "is_continuation",
                        "dataType": ["boolean"]
                    },
                    {
                        "name": "orig_elements",
                        "dataType": ["text"]
                    }
                ]
            }
        ]
    }
    ```

    See also :

    *   [Collection schema](https://weaviate.io/developers/weaviate/config-refs/schema)
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

The Weaviate connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[weaviate]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `WEAVIATE_URL` - THE REST endpoint for the Weaviate database cluster, represented by `--host-url` (CLI) or `host_url` (Python).

*   `WEAVIATE_API_KEY` - The API key for the database cluster, represented by `--api-key` (CLI) or `api_key` (Python).

    <Note>For the CLI, the `--api-key` option here is part of the `weaviate` command. For Python, the `api_key` parameter here is part of the `WeaviateAccessConfig` object.</Note>

*   `WEAVIATE_COLLECTION_CLASS_NAME` - The name of the collection in the database, represented by `--class-name` (CLI) or `class_name` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding is optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --strategy fast \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    weaviate \
      --host-url $WEAVIATE_URL \
      --api-key $WEAVIATE_API_KEY \
      --class-name $WEAVIATE_COLLECTION_CLASS_NAME
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.weaviate import (
      WeaviateConnectionConfig,
      WeaviateAccessConfig,
      WeaviateUploaderConfig,
      WeaviateUploadStagerConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=WeaviateConnectionConfig(
              access_config=WeaviateAccessConfig(
                  api_key=os.getenv("WEAVIATE_API_KEY")
              ),
              host_url=os.getenv("WEAVIATE_URL"),
              class_name=os.getenv("WEAVIATE_COLLECTION_CLASS_NAME")
          ),
          stager_config=WeaviateUploadStagerConfig(),
          uploader_config=WeaviateUploaderConfig()
      ).run()
  ```

  ```python Python Ingest v1
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.connector.weaviate import (
      SimpleWeaviateConfig,
      WeaviateAccessConfig,
      WeaviateWriteConfig,
  )
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.weaviate import (
      WeaviateWriter,
  )

  import os

  def get_writer() -> Writer:
      return WeaviateWriter(
          connector_config=SimpleWeaviateConfig(
              access_config=WeaviateAccessConfig(
                  api_key=os.getenv("WEAVIATE_API_KEY")
              ),
              host_url=os.getenv("WEAVIATE_URL"),
              class_name=os.getenv("WEAVIATE_COLLECTION_CLASS_NAME"),
          ),
          write_config=WeaviateWriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")),
              strategy="hi_res",
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>


# Process files in batches by using the Unstructured Ingest CLI



The Unstructured Ingest CLI enables you to use command-line scripts to send files in batches to Unstructured API services for processing, and to tell Unstructured API services where to deliver the processed data. [Learn more](/ingestion/overview#unstructured-ingest-cli).

## Installation

One approach to get started quickly with the Unstructured Ingest CLI is to install Python and then run the following command:

```bash
pip install unstructured-ingest
```

This default installation option enables the ingestion of plain text files, HTML, XML, JSON and emails that do not require any extra dependencies. This default option also enables you to specify local source and destination locations.

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

For additional installation options, see [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) in the [Ingest](/ingestion/overview) section.

<Info>To migrate from older, deprecated versions of the Ingest CLI that used `pip install unstructured`, see the [migration guide](/ingestion/overview#migration-guide).</Info>

## Usage

To call the Unstructured Ingest CLI, follow this calling pattern, where:

*   `<source>` is the command name for one of the available [source](/api-reference/ingest/source-connectors/overview) (input) connectors, such as `local` for a local source location, `azure` for an Azure Storage account source, `s3` for an Amazon S3 bucket source, and so on.
*   `<destination>` is the command name for one of the available [destination](/api-reference/ingest/destination-connector/overview) (output) connectors, such as `local` for a local destination, `azure` for an Azure Storage account destination, `s3` for an Amazon S3 bucket destination, and so on.
*   `<setting>` is one or more command-line options for specifying how and where Unstructured API services will ingest the files from the `<source>`, or how and where to deliver the processed data to the `<destination>`.

```bash CLI
#!/usr/bin/env bash

unstructured-ingest \
  <source> \
    --<setting1> <value1> \
    --<setting2> <value2> \
    --<settingN> <valueN> \
  <destination> \
    --<setting1> <value1> \
    --<setting2> <value2> \
    --<settingN> <valueN>
```

To learn how to use the Unstructured Ingest CLI to work with a specific source (input) and destination (output) location, see the CLI script examples for the [source](/api-reference/ingest/source-connectors/overview) and [destination](/api-reference/ingest/destination-connector/overview) connectors that are available for you to choose from.

See also the [ingest configuration](/api-reference/ingest/ingest-configuration/overview) settings for command-line options that enable you to further control how batches are sent and processed.


# Chunking configuration



A common chunking configuration is a critical element in the data processing pipeline, particularly when creating embeddings and populating vector databases with the results. This configuration defines the parameters governing the segmentation of text into meaningful chunks, whether at the document, paragraph, or sentence level. It plays a pivotal role in determining the size and structure of these chunks, ensuring that they align with the specific requirements of downstream tasks, such as embedding generation and vector database population. By carefully configuring chunking parameters, users can optimize the granularity of data segments, ultimately contributing to more cohesive and contextually rich results. This is crucial for tasks like natural language processing and text analysis, as well as for the efficient storage and retrieval of embeddings in vector databases, enhancing the quality and relevance of the results.

## Configs

*   <Icon icon="v" /><Icon icon="2" />  `chunk_api_key`: If `chunk_by_api` is set to `True`, requests that are sent to the Unstructured API will use this Unstructured API key to make authenticated calls.

*   <Icon icon="v" /><Icon icon="2" />  `chunk_by_api`: Default: `False`. If set to `True`, uses Unstructured API services to run chunking. If set to `False`, runs chunking locally.

*   <Icon icon="v" /><Icon icon="2" />  `chunk_combine_text_under_n_chars`: Combine consecutive chunks when the first does not exceed length `n` and the second will fit without exceeding the hard-maximum length. Only operative for the `by_title` chunking strategy.

*   <Icon icon="v" /><Icon icon="1" />  `chunk_elements`: <Icon icon="ban" />  **Deprecated!** `False` (default) to not run chunking as part of the ingest process. This option is deprecated in favor of the `chunking_strategy` option. Setting this to `True` has the same effect as `chunking_strategy=by_title`.

*   <Icon icon="v" /><Icon icon="2" />  `chunk_include_orig_elements`: `True` when chunking to add the original elements consolidated to form each chunk to `.metadata.orig_elements` on that chunk.

*   <Icon icon="v" /><Icon icon="2" />  `chunk_max_characters`: Default: `500`. The hard-maximum chunk length. No chunk will exceed this length. An oversized element will be divided by text-splitting to fit this window.

*   <Icon icon="v" /><Icon icon="2" />  `chunk_multipage_selections`: `True` to ignore page boundaries when chunking such that elements from two different pages can appear in the same chunk. Only operative for the `by_title` chunking strategy.

*   <Icon icon="v" /><Icon icon="2" />  `chunk_new_after_n_chars`: The soft-maximum chunk length. Another element will not be added to a chunk of `n` length even when it would fit without exceeding the hard-maximum length.

*   <Icon icon="v" /><Icon icon="2" />  `chunk_overlap`: Default: `0`. Prefix each chunk's text with the last overlap of `n` characters from the prior chunk. Only applies to oversized chunks divided by text-splitting. To apply overlap to non-oversized chunks, use `chunk_overlap_all`.

*   <Icon icon="v" /><Icon icon="2" />  `chunk_overlap_all`: Applies overlap to chunks formed from whole elements as well as those formed by text-splitting oversized elements. The overlap length is taken from the `chunk_overlap` value.

*   <Icon icon="v" /><Icon icon="2" />  `chunking_endpoint`: If `chunk_by_api` is set to `True`, chunking requests are sent to this Unstructured API URL. By default, this URL is the Unstructured Serverless API URL: `https://api.unstructuredapp.io/general/v0/general`.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `chunking_strategy`: One of `basic` or `by_title`. When omitted, no chunking is performed. The `basic` strategy maximally fills each chunk with whole elements, up the specified size limits as specified by `max_characters` and `new_after_n_chars`. A single element that exceeds this length is divided into two or more chunks using text-splitting. A `Table` element is never combined with any other element and appears as a chunk of its own or as a sequence of `TableChunk` elements splitting is required. The `by_title` behaviors are the same except that section and optionally page boundaries are respected such that two consecutive elements from different sections appear in separate chunks.

*   <Icon icon="v" /><Icon icon="1" />  `combine_text_under_n_chars`: Default: `max_characters`. Combines small elements (for example a series of `Title` elements) until a section reaches a length of `n `characters. Only operative for the `by_title` chunking strategy. Defaults to max\_characters which combines chunks whenever space allows. Specifying `0` for this argument suppresses combining of small chunks.

*   <Icon icon="v" /><Icon icon="1" />  `include_orig_elements`: Default: `True`. Adds the document elements consolidated to form each chunk to the `orig_elements` field in `metadata`. Setting this to `False` allows for somewhat smaller payloads when you don't need that metadata.

*   <Icon icon="v" /><Icon icon="1" />  `max_characters`: Default: `500`. Combine elements into chunks no larger than `n` characters. This is a hard maximum: no chunk with text longer than this value will appear in the output stream.

*   <Icon icon="v" /><Icon icon="1" />  `multipage_sections`: Default: `True`. When `False`, in addition to section boundaries, page boundaries are also respected. Only operative for the `by_title` chunking strategy.

*   <Icon icon="v" /><Icon icon="1" />  `new_after_n_chars`: Default: `max_characters (off)`. Cuts off new chunks once they reach a length of `n` characters. This is a soft maximum. Defaults to `max_characters` when not specified, which effectively disables any soft window. Specifying `0` for this argument causes each element to appear in a chunk by itself (although an element with text longer than `max_characters` will be still be divided into two or more chunks using text splitting).

*   <Icon icon="v" /><Icon icon="1" />  `overlap`: Default: `0`. Prefix each chunk's text with the last overlap of `n` characters from the prior chunk. Only applies to oversized chunks divided by text-splitting. To apply overlap to non-oversized chunks, use `overlap_all`.

*   <Icon icon="v" /><Icon icon="1" />  `overlap_all`: Applies overlap to chunks formed from whole elements as well as those formed by text-splitting oversized elements. The overlap length is taken from the `overlap` value.


# Embedding configuration



A common embedding configuration is a critical component that allows for dynamic selection of embedders and their associated parameters to create vectors from data. This configuration provides the flexibility to choose from various embedding models and fine-tune parameters to optimize the quality and characteristics of the resulting vectors. It enables users to tailor the embedding process to the specific needs of their data and downstream applications, ensuring that the generated vectors effectively capture semantic relationships and contextual information within the dataset.

## Configs

*   <Icon icon="v" /><Icon icon="1" />  `api_key`: The API key to use, if one is required to generate the embeddings through an API service, such as OpenAI.

*   <Icon icon="v" /><Icon icon="1" />  `aws_access_key_id`: The AWS access key ID to be used for AWS-based embedders, such as Amazon Bedrock.

*   <Icon icon="v" /><Icon icon="1" />  `aws_region`: The AWS Region ID to be used for AWS-based embedders, such as Amazon Bedrock.

*   <Icon icon="v" /><Icon icon="1" />  `aws_secret_access_key`: The AWS secret access key to be used for AWS-based embedders, such as Amazon Bedrock.

*   <Icon icon="v" /><Icon icon="2" />  `embedding_provider`: The embedding provider to use while doing embedding. Available values include `aws-bedrock`, `huggingface`, `mixedbread-ai`, `octoai`, `openai`, `togetherai`, `vertexai`, and `voyageai`.

*   <Icon icon="v" /><Icon icon="2" />  `embedding_api_key`: The API key to use, if one is required to generate the embeddings through an API service, such as OpenAI.

*   <Icon icon="v" /><Icon icon="2" />  `embedding_aws_access_key_id`: The AWS access key ID to be used for AWS-based embedders, such as Amazon Bedrock.

*   <Icon icon="v" /><Icon icon="2" />  `embedding_aws_region`: The AWS Region ID to be used for AWS-based embedders, such as Amazon Bedrock.

*   <Icon icon="v" /><Icon icon="2" />  `embedding_aws_secret_access_key`: The AWS secret access key to be used for AWS-based embedders, such as Amazon Bedrock.

*   <Icon icon="v" /><Icon icon="2" />  `embedding_model_name`: The specific model to use for the embedding provider, if necessary.

*   <Icon icon="v" /><Icon icon="1" />  `model_name`: The specific model to use for the embedding provider, if necessary.

*   <Icon icon="v" /><Icon icon="1" />  `provider`: The embedding provider to use while doing embedding. Available values include `aws-bedrock`, `huggingface`, `mixedbread-ai`, `octoai`, `openai`, `togetherai`, `vertexai`, and `voyageai`.

<Icon icon="v" /><Icon icon="1" />  The default `model_name` values unless otherwise specified are:

*   `aws-bedrock`: None

*   `huggingface`: `sentence-transformers/all-MiniLM-L6-v2`, with 384 dimensions

*   `mixedbread-ai`: `mixedbread-ai/mxbai-embed-large-v1`, with 1024 dimensions

*   `octoai`: `thenlper/gte-large`, with 1024 dimensions

*   `openai`: `text-embedding-ada-002`, with 1536 dimensions

*   `togetherai`: `togethercomputer/m2-bert-80M-8k-retrieval`, with 768 dimensions

*   `vertexai`: `textembedding-gecko@001`, with 768 dimensions

*   `voyageai`: None


# Fsspec configuration



A common fsspec configuration is a shared set of parameters and settings utilized by connectors responsible for managing cloud-based file-system content. These configurations enable connectors to interact with cloud storage systems consistently, specifying details such as authentication credentials, access permissions, storage locations, and optimization options. By standardizing these fsspec configurations, connectors can seamlessly interface with various cloud-based file systems, promoting interoperability and efficient management of remote data storage across different cloud providers.

## File Storage Configuration

Fsspec configuration is an extension of the File Storage Config:

*   <Icon icon="v" /><Icon icon="1" />  `recursive`: Default: `False`. If a directory path is provided, whether further folders should be traversed recursively.

*   <Icon icon="v" /><Icon icon="1" />  `remote_url`: Path to the content on the remote system, prefixed with the protocol (for example, `s3://...`).

*   <Icon icon="v" /><Icon icon="1" />  `uncompress`: Default: `False`. Whether to uncompress tar and zip files when downloaded. Will ignore these otherwise.

## Configs

The following are configurations set on the fsspec config itself:

*

*   <Icon icon="v" /><Icon icon="1" />  `access_config`: A dictionary of access keyword arguments (`kwargs`) that might be needed to provide access to the cloud provider associated with the data.

The following are generated for the user after the config is initialized:

*   <Icon icon="v" /><Icon icon="1" />  `dir_path` and `file_path`: The raw remote path is parsed into the directory and file if it exists.

*   <Icon icon="v" /><Icon icon="1" />  `path_without_protocol`: The file path on the remote source without the protocol.

*   <Icon icon="v" /><Icon icon="1" />  `protocol`: The protocol is extracted from the raw remote path passed in (for example, `s3` from `s3://...`).


# Overview



The configurations listed in this section are common to all connectors.
These configurations act as instructions for how to collect, process, and store data.

<iframe width="560" height="315" src="https://www.youtube.com/embed/A_kqK2KHTdg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

Some connectors implement only version 2 (v2) of the connector specification; some connectors implement only version 1 (v1); and some connectors provide both v2 and v1 implementations.

*   Connectors that implement both v2 and v1 are prefaced by <Icon icon="v" /><Icon icon="2" />, <Icon icon="v" /><Icon icon="1" />.

*   v2-only implementations are prefaced by <Icon icon="v" /><Icon icon="2" />.

*   v1-only implementations are prefaced by <Icon icon="v" /><Icon icon="1" />.

If you're looking for parameters specific to a connector (such as authentication options)—or to determine whether a connector provides a v2 implementation, a v1 implementation, or both—browse the
corresponding connector's source code:

*   [v2 non-fsspec connectors](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/unstructured_ingest/v2/processes/connectors)
*   [v2 fsspec connectors](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/unstructured_ingest/v2/processes/connectors/fsspec)
*   [v1 non-fsspec connectors](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/unstructured_ingest/connector)
*   [v1 fsspec connectors](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/unstructured_ingest/connector/fsspec)
*   [v1 Notion connector](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/unstructured_ingest/connector/notion)


# Partition configuration



A standard partition configuration is a collection of parameters designed to oversee document partitioning, whether executed through API integration or by the unstructured library on a local system. These parameters serve a dual role, encompassing those passed to the partition method for the initial segmentation of documents and those responsible for coordinating data after processing, including the dynamic metadata associated with each element.

## Configs for Partitioning

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `additional_partition_args`: A JSON string representation of any values to pass through to the `partition` function.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `encoding`: The encoding method used to decode the text input. By default, UTF-8 will be used.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `ocr_languages`: The languages present in the document, for use in partitioning, OCR, or both. Multiple languages indicate that the text could be in any of the specified languages.

*   <Icon icon="v" /><Icon icon="1" />  `pdf_infer_table_structure`: <Icon icon="ban" />  **Deprecated!** Use `skip_infer_table_types` to opt out of table extraction for any file type. If `False` and `strategy=hi_res`, no `Table` elements will be extracted from PDF files regardless of `skip_infer_table_types` contents.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `skip_infer_table_types`: List of document types that you want to skip table extraction with.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `strategy`: Default: `auto`. The strategy to use for partitioning PDF and image files. Uses a layout detection model if set to `hi_res`. Otherwise, partitioning simply extracts the text from the document and processes it.

## Configs for the Process

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `api_key`: If `partition_by_api` is set to `True`, requests that are sent to the Unstructured API will use this Unstructured API key to make authenticated calls.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `fields_include`: Fields to include in the output JSON. By default, the following fields are included: `element_id`, `text`, `type`, `metadata`, and `embeddings`.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `flatten_metadata`: Default: `False`. If set to `True`, the hierarchical metadata structure is flattened to have all values exist at the top level.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `hi_res_model_name`: The model to use when `strategy` is set to `hi_res`. Available values are `layout_v1.0.0` (the default) and `yolox`.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `metadata_exclude`: Values from the `metadata` field to exclude from the output.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `metadata_include`: If provided, only the specified fields are preserved in the `metadata` output.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `partition_by_api`: Default: `False`. If set to `True`, uses Unstructured API services to run partitioning. If set to `False`, runs partitioning locally.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `partition_endpoint`: If `partition_by_api` is set to `True`, partitioning requests are sent to this Unstructured API URL.


# Permissions configuration



A common permissions configuration is leveraged to extract user access data associated with the content from the source data provider. Currently supported for only the SharePoint connector.

## Configs

*   <Icon icon="v" /><Icon icon="1" />  `application_id`: The SharePoint application client ID.

*   <Icon icon="v" /><Icon icon="1" />  `client_cred`: The SharePoint application client secret.

*   <Icon icon="v" /><Icon icon="1" />  `tenant`: The SharePoint tenant ID.


# Processor configuration



A common process configuration plays a pivotal role in overseeing the entire ingest process, encompassing various aspects to ensure efficiency and reliability. This configuration includes parameters for managing a pool of workers, which allows for parallelization of the ingest process to maximize throughput. Additionally, it addresses the storage and retrieval of intermediate results, supporting caching strategies that enable replayability in case of errors or interruptions. By configuring worker pools, the process can efficiently handle multiple tasks concurrently, enhancing performance. Furthermore, saving intermediate results allows for the resumption of the process from a known state, reducing data loss and ensuring robustness in the face of unexpected errors or system failures. This comprehensive configuration promotes an organized and resilient approach to data ingestion, improving overall system reliability and efficiency.

## Configs

*   <Icon icon="v" /><Icon icon="2" />  `disable_parallelism`: `True` if the `INGEST_DISABLE_PARALLELISM` environment variable is set to `True` (case-insensitive), otherwise `False` (the default).

*   <Icon icon="v" /><Icon icon="2" />  `download_only`: Default: `False`. If set to `True`, downloads any files that are not already present in the connector's specified download directory (`download_dir`), or `work_dir` if `download_dir` is not specified, or the default file path for `work_dir` if `work_dir` is not specified.

*   <Icon icon="v" /><Icon icon="2" />  `max_connections`: The maximum of connections allowed when running an asynchronous step in the ingest pipeline.

*   <Icon icon="v" /><Icon icon="2" />  `max_docs`: The maximum number of documents to be pulled in for the entire ingest process.

*   <Icon icon="v" /><Icon icon="2" />, <Icon icon="v" /><Icon icon="1" />  `num_processes`: Default: `2`. For every step that can use a pool of workers to increase throughput, how many workers to configure in the pool.

*   <Icon icon="v" /><Icon icon="1" />  `output_dir`: Where the final results will be located when the process is finished. This is regardless of whether a destination is configured. If a directory is not specified, by default a folder named `structured-output`, relative to the current working directory, is used.

*   <Icon icon="v" /><Icon icon="2" />  `preserve_downloads`: When set to `False` (the default), will remove downloaded files after they are successfully processed.

*   <Icon icon="v" /><Icon icon="2" />, <Icon icon="v" /><Icon icon="1" />  `raise_on_error`: `False `(default) for any single document that fails in the process, causes the error to be logged but allows for all other documents to proceed in the process. If `True`, causes the entire process to fail and raise the error if any one document fails.

*   <Icon icon="v" /><Icon icon="2" />  `re_download`: When set to `False` (the default), will not re-download files if they are already present in the download directory. Setting this to `True` downloads the files even if they exist in the download directory. `True` is useful in cases where for some reason you suspect that the files in the remote location might have changed, and you want to be sure that the file copies in the download directory are the most current ones.

*   <Icon icon="v" /><Icon icon="2" />, <Icon icon="v" /><Icon icon="1" />  `reprocess`: Default: `False`. If set to `True`, will ignore all content that may have been cached and rerun each step.

*   <Icon icon="v" /><Icon icon="2" />  `tqdm`: When set to `False` (the default), will not show a progress bar.

*   <Icon icon="v" /><Icon icon="2" />  `uncompress`: When set to `False` (the default), will not uncompress any archived ZIP or TAR files. If set to `True`, and the specified partitioner provides support for compressed ZIP and TAR files, these files will be uncompressed.

*   <Icon icon="v" /><Icon icon="2" />, <Icon icon="v" /><Icon icon="1" />  `verbose`: Default: `False`. If set to `True`, debug logging should be included in the output.

*   <Icon icon="v" /><Icon icon="2" />, <Icon icon="v" /><Icon icon="1" />  `work_dir`: The file path for where intermediate results should be saved. If a file path is not set, a default file path will be used relative to the user's home directory (`/home/username/.cache/unstructured/ingest/pipeline` or `C:\Users\username\.cache\unstructured\ingest\pipeline`).


# Read configuration



A shared read configuration serves as a universal set of parameters that are consistent across all source connectors, providing a standardized way to access and retrieve documents from various sources. This configuration typically includes settings such as the download directory, which specifies the location where retrieved documents are stored. By maintaining common parameters like the download directory, users can streamline their data extraction processes, making it easier to manage and organize the downloaded documents irrespective of the source connector in use. This promotes consistency, ease of maintenance, and a more straightforward integration process when working with multiple source connectors within a system.

## Configs

*   <Icon icon="v" /><Icon icon="1" />  `download_dir`: What location to download the files to. When run via the CLI, a default location will be used if one is not provided.

*   <Icon icon="v" /><Icon icon="1" />  `download_only`: Default: `False`. If set to `True`, the process will exit right after all the files are downloaded and omit any future steps such as partitioning and uploading to a destination.

*   <Icon icon="v" /><Icon icon="1" />  `max_docs`: How many documents are pulled in for a single process.

*   <Icon icon="v" /><Icon icon="1" />  `preserve_downloads`: `False` (default) for the process to delete the downloaded content at the end if everything finished without error. By setting this to `True`, those files will be preserved.

*   <Icon icon="v" /><Icon icon="1" />  `re_download`: `False` (default) for the process to skip downloads if the files already exist in the download directory. By setting this to `True`, it will force the files to be re downloaded regardless of them existing already.


# Retry strategy configuration



A common retry strategy configuration is a critical element in enhancing the robustness and resiliency of a system, especially when dealing with temporary network issues. This configuration typically includes parameters that define how the system should respond to network-related errors, such as connection timeouts or transient failures. It specifies the number of retry attempts, the time intervals between retries, and the conditions under which retries should be initiated. By configuring an effective retry strategy, a system can automatically recover from network disruptions, increasing its ability to withstand intermittent issues and ensuring a more reliable and uninterrupted operation. This strategy is vital for maintaining system stability, data integrity, and minimizing downtime in the face of unpredictable network hiccups.

## Configs

*   <Icon icon="v" /><Icon icon="1" />  `max_retries`: How many times to retry before stopping. By default, there is no limit to the number of tries.

*   <Icon icon="v" /><Icon icon="1" />  `max_retry_time`: If provided, will stop trying after this much time has elapsed.


# Ingest dependencies



When you install the [Unstructured Ingest CLI](/api-reference/ingest/ingest-cli) and the
[Unstructured Ingest Python library](/api-reference/ingest/python-ingest) by running the command
`pip install unstructured-ingest` by itself, you get the following by default:

*   The [local source connector](/api-reference/ingest/source-connectors/local) and the [local destination connector](/api-reference/ingest/destination-connector/local).
*   Support for the following file types:

| File type |
| --------- |
| `.bmp`    |
| `.eml`    |
| `.heic`   |
| `.html`   |
| `.jpg`    |
| `.jpeg`   |
| `.tiff`   |
| `.png`    |
| `.txt`    |
| `.xml`    |

To add support for additional file types, run the following:

| Command                                   | File type |
| ----------------------------------------- | --------- |
| `pip install "unstructured-ingest[csv]"`  | `.csv`    |
| `pip install "unstructured-ingest[doc]"`  | `.doc`    |
| `pip install "unstructured-ingest[docx]"` | `.docx`   |
| `pip install "unstructured-ingest[epub]"` | `.epub`   |
| `pip install "unstructured-ingest[md]"`   | `.md`     |
| `pip install "unstructured-ingest[msg]"`  | `.msg`    |
| `pip install "unstructured-ingest[odt]"`  | `.odt`    |
| `pip install "unstructured-ingest[org]"`  | `.org`    |
| `pip install "unstructured-ingest[pdf]"`  | `.pdf`    |
| `pip install "unstructured-ingest[ppt]"`  | `.ppt`    |
| `pip install "unstructured-ingest[pptx]"` | `.pptx`   |
| `pip install "unstructured-ingest[rtf]"`  | `.rtf`    |
| `pip install "unstructured-ingest[rst]"`  | `.rst`    |
| `pip install "unstructured-ingest[tsv]"`  | `.tsv`    |
| `pip install "unstructured-ingest[xlsx]"` | `.xlsx`   |

To add support for additional connectors, run the following:

| Command                                                 | Connector type       |
| ------------------------------------------------------- | -------------------- |
| `pip install "unstructured-ingest[airtable]"`           | Airtable             |
| `pip install "unstructured-ingest[astra]"`              | Astra DB             |
| `pip install "unstructured-ingest[azure]"`              | Azure Blob Storage   |
| `pip install "unstructured-ingest[azure-ai-search]"`    | Azure AI Search      |
| `pip install "unstructured-ingest[biomed]"`             | Biomed               |
| `pip install "unstructured-ingest[box]"`                | Box                  |
| `pip install "unstructured-ingest[chroma]"`             | Chroma               |
| `pip install "unstructured-ingest[clarifai]"`           | Clarifai             |
| `pip install "unstructured-ingest[confluence]"`         | Confluence           |
| `pip install "unstructured-ingest[couchbase]"`          | Couchbase            |
| `pip install "unstructured-ingest[databricks-volumes]"` | Databricks Volumes   |
| `pip install "unstructured-ingest[delta-table]"`        | Delta Tables         |
| `pip install "unstructured-ingest[discord]"`            | Discord              |
| `pip install "unstructured-ingest[dropbox]"`            | Dropbox              |
| `pip install "unstructured-ingest[elasticsearch]"`      | Elasticsearch        |
| `pip install "unstructured-ingest[gcs]"`                | Google Cloud Storage |
| `pip install "unstructured-ingest[github]"`             | GitHub               |
| `pip install "unstructured-ingest[gitlab]"`             | GitLab               |
| `pip install "unstructured-ingest[google-drive]"`       | Google Drive         |
| `pip install "unstructured-ingest[hubspot]"`            | HubSpot              |
| `pip install "unstructured-ingest[jira]"`               | JIRA                 |
| `pip install "unstructured-ingest[kafka]"`              | Apache Kafka         |
| `pip install "unstructured-ingest[milvus]"`             | Milvus               |
| `pip install "unstructured-ingest[mongodb]"`            | MongoDB              |
| `pip install "unstructured-ingest[notion]"`             | Notion               |
| `pip install "unstructured-ingest[onedrive]"`           | OneDrive             |
| `pip install "unstructured-ingest[opensearch]"`         | OpenSearch           |
| `pip install "unstructured-ingest[outlook]"`            | Outlook              |
| `pip install "unstructured-ingest[pinecone]"`           | Pinecone             |
| `pip install "unstructured-ingest[postgres]"`           | PostgreSQL, SQLite   |
| `pip install "unstructured-ingest[qdrant]"`             | Qdrant               |
| `pip install "unstructured-ingest[reddit]"`             | Reddit               |
| `pip install "unstructured-ingest[s3]"`                 | Amazon S3            |
| `pip install "unstructured-ingest[sharepoint]"`         | SharePoint           |
| `pip install "unstructured-ingest[salesforce]"`         | Salesforce           |
| `pip install "unstructured-ingest[singlestore]"`        | SingleStore          |
| `pip install "unstructured-ingest[snowflake]"`          | Snowflake            |
| `pip install "unstructured-ingest[sftp]"`               | SFTP                 |
| `pip install "unstructured-ingest[slack]"`              | Slack                |
| `pip install "unstructured-ingest[wikipedia]"`          | Wikipedia            |
| `pip install "unstructured-ingest[weaviate]"`           | Weaviate             |

To add support for available embedding libraries, run the following:

| Command                                                 | Embedding library type |
| ------------------------------------------------------- | ---------------------- |
| `pip install "unstructured-ingest[bedrock]"`            | Amazon Bedrock         |
| `pip install "unstructured-ingest[embed-huggingface]"`  | Hugging Face           |
| `pip install "unstructured-ingest[embed-octoai]"`       | OctoAI                 |
| `pip install "unstructured-ingest[embed-vertexai]"`     | Google Vertex AI       |
| `pip install "unstructured-ingest[embed-voyageai]"`     | Voyage AI              |
| `pip install "unstructured-ingest[embed-mixedbreadai]"` | Mixedbread             |
| `pip install "unstructured-ingest[openai]"`             | OpenAI                 |
| `pip install "unstructured-ingest[togetherai]"`         | together.ai            |

For details about the specific dependencies that are installed, see:

*   [Common dependencies](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/requirements/common)
*   [File type dependencies](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/requirements/local_partition)
*   [Connector dependencies](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/requirements/connectors)
*   [Embedding library dependencies](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/requirements/embed)

See also [setup.py](https://github.com/Unstructured-IO/unstructured-ingest/blob/main/setup.py).


# Overview



You can send multiple files in batches to be ingested by Unstructured API services for processing.
*Ingestion* is the term that Unstructured uses to refer to the set of activities that happens when files are input for processing. [Learn more](/ingestion/overview).

You can send batches to Unstructured API services by using the following tools:

*   The [Unstructured Ingest CLI](/api-reference/ingest/ingest-cli)
*   The [Unstructured Ingest Python](/api-reference/ingest/python-ingest) library

The following 3-minute video shows how to use the Unstructured Ingest Python library to send multiple PDFs from a local directory in batches to be ingested by Unstructured API services for processing:

<iframe width="560" height="315" src="https://www.youtube.com/embed/tSKHFXsBQ-c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The following 5-minute video goes into more detail about the various components of the Unstructured Ingest Python library:

<iframe width="560" height="315" src="https://www.youtube.com/embed/A_kqK2KHTdg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

## Ingest flow

The Unstructured ingest flow is similar to an extract, transform and load (ETL) data pipeline.
Because of this, a customer-defined implementation of the Unstructured ingest flow is sometimes referred to as an *ingest pipeline* or simply a *pipeline*.
An Unstructured ingest pipeline contains the following logical steps:

<Steps>
  <Step title="Index">
    Reaches out to the source location and pulls in metadata for each document.

    For example, this could include information such as the path to the files to be analyzed.

    *   For the Unstructured CLI, you can control this behavior, where available for a connector, through its `--input-path` command option.
    *   For the Unstructured Ingest Python library's v2 calling pattern, you can control this behavior, where available for a connector, through its `<Prefix>IndexerConfig` class (where `<Prefix>` represents the connector provider's name, such as `Azure` for Azure.)
  </Step>

  <Step title="Post-Index Filter">
    After indexing, you might not want to download everything that was indexed.

    For example, you might want to download only files that match specific types, file names, paths, or sizes.

    For the Unstructured Ingest Python library's v2 calling pattern, you can control this behavior through the `FiltererConfig` class.
  </Step>

  <Step title="Download">
    Using the information generated from the indexer and the filter, downloads the content as files on the local file system for processing. This may require manipulation of the data to prepare it for partitioning.

    For example, this could include information such as the path to a local directory to download files to.

    *   For the Unstructured CLI, you can control this behavior through a connector's `--download-dir` command option.
    *   For the Unstructured Ingest Python library's v2 calling pattern, you can control this behavior through a connector's `<Prefix>DownloaderConfig` class.
  </Step>

  <Step title="Post-Download Filter">
    After downloading, if uncompression is enabled, you might not want to uncompress everything that was downloaded. The filter that was defined at the beginning is repeated here.
  </Step>

  <Step title="Uncompress">
    If enabled, searches for any compressed files (Unstructured supports TAR and ZIP) and uncompresses them.

    *   For the Unstructured CLI, you can control this behavior through the `--uncompress` command option.
    *   For the Unstructured Ingest Python library's v2 calling pattern, you can control this behavior through the `UncompressConfig` class.
  </Step>

  <Step title="Post-Uncompress Filter">
    After downloading, and uncompressing if enabled, you might not want to partition everything that was downloaded, and uncompressed if enabled. The filter that was defined at the beginning is repeated here again.
  </Step>

  <Step title="Partition">
    Generates the structured enriched content from the local files that have been downloaded, uncompressed if enabled, and filtered. Both local-based partitioning and Unstructured API services-based partitioning is supported, with API services-based partitioning set to run asynchronously and local-based partitioning set to run through multiprocessing.

    *   For the Unstructured Ingest Python library's v2 calling pattern, you can control this behavior through the `PartitionerConfig` class.
    *   For the Unstructured Ingest Python library's v1 calling pattern, you can control this behavior through the `PartitionConfig` class.
  </Step>

  <Step title="Chunk">
    Optionally, chunks the partitioned content. Chunking can also be run locally or through Unstructured API services, with asynchronous or multiprocessing behaviors set in the same approach as the partitioner.

    *   For the Unstructured Ingest Python library's v2 calling pattern, you can control this behavior through the `ChunkerConfig` class.
    *   For the Unstructured Ingest Python library's v1 calling pattern, you can control this behavior through the `ChunkingConfig` class.
  </Step>

  <Step title="Embed">
    Optionally, generates vector embeddings for each element in the structured output. Most of the time, this is done through API calls to a third-party embedding vendor and therefore runs asynchronously. But it can also use a locally available Hugging Face model, which will run through multiprocessing.

    *   For the Unstructured Ingest Python library's v2 calling pattern, you can control this behavior through the `EmbedderConfig` class.
    *   For the Unstructured Ingest Python library's v1 calling pattern, you can control this behavior through the `EmbeddingConfig` class.
  </Step>

  <Step title="Stage">
    This is an optional step that does not apply most of the time. However, sometimes the data needs to be modified from the existing structure to better support the upload step, such as converting it to a CSV file for tabular-based destinations.

    For the Unstructured Ingest Python library's v2 calling pattern, you can control this behavior, where available for a connector, through its `<Prefix>UploadStagerConfig` class.
  </Step>

  <Step title="Upload">
    Writes the local content to the destination. If no destination is provided, the local one will be used, which writes the final result to a location on the local filesystem. If batch uploads are needed, this will run in a single process with access to all documents. If batch uploads are not supported, all documents can be uploaded concurrently by using the asynchronous approach.

    For the Unstructured Ingest Python library's v2 calling pattern, you can control this behavior through a connector's `<Prefix>UploaderConfig` class.
  </Step>
</Steps>

## Generate Python code examples

You can connect any available source connector to any available destination connector. However, the source connector code examples in the
documentation show connecting only to the local destination connector. Similarly, the destination connector code examples in the
documentation show connecting only to the local source connector.

To quickly generate an Unstructured Ingest Python library code example that connects *any* available source connector to *any* available destination connector,
do the following:

1.  Open the [Unstructured Ingest Code Generator](https://huggingface.co/spaces/MariaK/unstructured-pipeline-builder) webpage.

2.  Select your input (source) location type from the **Get unstructured documents from** drop-down list.

3.  Select your output (destination) location type from the **Upload RAG-ready documents to** drop-down list.

4.  Select your chunking strategy from the **Chunking strategy** drop-down list:

    *   **None** - Do not chunk the data elements' content.
    *   **basic** - Combine sequential data elements to maximally fill each chunk. However, do not mix `Table` and non-`Table` elements in the same chunk.
    *   **by\_title** - Use the `basic` strategy and also preserve section boundaries. Optionally preserve page boundaries as well.
    *   **by\_page** - Use the `basic` strategy and also preserve page boundaries.
    *   **by\_similarity** - Use the `sentence-transformers/multi-qa-mpnet-base-dot-v1` embedding model to identify topically similar sequential elements and combine them into chunks. This strategy is availably only when calling Unstructured API services.

    To learn more, see [Chunking strategies](/api-reference/api-services/chunking) and [Chunking configuration](/api-reference/ingest/ingest-configuration/chunking-configuration).

5.  For any chunking strategy other than **None**:

    *   Enter your chunk size in the **Chunk size (characters)** box, or leave the default of **1000** characters.
    *   If you need to apply overlapping to the chunks, enter the chunk overlap size in the **Chunk overlap (characters)** box, or leave default of **20** characters.

    To learn more, see [Chunking configuration](/api-reference/ingest/ingest-configuration/chunking-configuration).

6.  To generate vector embeddings, select the provider in the **Embedding provider** drop-down list.

    To learn more, see [Embedding configuraton](/api-reference/ingest/ingest-configuration/embedding-configuration).

7.  Click **Generate code**.

8.  Copy the example code from the **Generated Code** pane into your code project.

9.  The code example will contain one or more environment variables that you must set for the code to run correctly. To learn what to
    set these variables to, click the documentation links that are below the **Generated Code** pane.

## Learn more

*   [Ingest configuration](/api-reference/ingest/ingest-configuration/overview) settings enable you to control how batches are sent and processed.
*   [Source connectors](/api-reference/ingest/source-connectors/overview) enable you to send batches from local or remote locations to be ingested by Unstructured API services for processing.
*   [Destination connectors](/api-reference/ingest/destination-connector/overview) enable Unstructured API services to send the processed data to local or remote locations.

## See also

*   The [Unstructured Platform](/platform/overview) enables you to send batches to Unstructured API services from remote locations, and to have Unstructured API services send the processed data to remote locations, all without using code or a CLI.


# Process files in batches by using the Unstructured Ingest Python library



The Unstructured Ingest Python library enables you to use Python code to send files in batches to Unstructured API services for processing, and to tell Unstructured API services where to deliver the processed data.

The following 3-minute video shows how to use the Unstructured Ingest Python library to send multiple PDFs from a local directory in batches to be ingested by Unstructured API services for processing:

<iframe width="560" height="315" src="https://www.youtube.com/embed/tSKHFXsBQ-c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

[Learn more](/overview#unstructured-ingest-python).

## Installation

One approach to get started quickly with the Unstructured Ingest Python library is to install Python and then run the following command:

```bash
pip install unstructured-ingest
```

This default installation option enables the ingestion of plain text files, HTML, XML, JSON and emails that do not require any extra dependencies. This default option also enables you to specify local source and destination locations.

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

For additional installation options, and information about v2 and v1 implementations in this library, see the [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) in the [Ingest](/ingestion/overview) section.

<Info>To migrate from older, deprecated versions of the Ingest Python library that used `pip install unstructured`, see the [migration guide](/ingestion/overview#migration-guide).</Info>

## Usage

For example, to use the Unstructured Ingest Python library to ingest files from a local source (input) location and to deliver the processed data to an Azure Storage account destination (output) location:

<CodeGroup>
  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.azure import (
      AzureConnectionConfig,
      AzureAccessConfig,
      AzureUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=AzureConnectionConfig(
              access_config=AzureAccessConfig(
                  account_name=os.getenv("AZURE_STORAGE_ACCOUNT_NAME"),
                  account_key=os.getenv("AZURE_STORAGE_ACCOUNT_KEY")
              )
          ),
          uploader_config=AzureUploaderConfig(remote_url=os.getenv("AZURE_STORAGE_REMOTE_URL"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.azure import (
      AzureAccessConfig,
      AzureWriteConfig,
      SimpleAzureBlobStorageConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.fsspec.azure import (
      AzureWriter,
  )

  def get_writer() -> Writer:
      return AzureWriter(
          connector_config=SimpleAzureBlobStorageConfig(
              access_config=AzureAccessConfig(account_name=os.getenv("AZURE_STORAGE_ACCOUNT_NAME")),
              remote_url=os.getenv("AZURE_STORAGE_REMOTE_URL"),
          ),
          write_config=AzureWriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res"
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

To learn how to use the Unstructured Ingest Python library to work with a specific source (input) and destination (output) location, see the Python code examples for the [source](/api-reference/ingest/source-connectors/overview) and [destination](/api-reference/ingest/destination-connector/overview) connectors that are available for you to choose from.

See also the [ingest configuration](/api-reference/ingest/ingest-configuration/overview) settings that enable you to further control how batches are sent and processed.


# Airtable



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Airtable to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your data and store structured outputs locally on your filesystem.

You will need:

The Airtable connector prerequisites:

*   An [Airtable](https://www.airtable.com/) account. [Create a free Airtable account](https://airtable.com/signup).

*   An Airtable personal access token. [Create a personal access token](https://support.airtable.com/docs/creating-and-using-api-keys-and-access-tokens).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/c3gvIWAAcTY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The ID of the Airtable base to access. [Create a base](https://www.airtable.com/guides/build/create-a-base). [Get a base's ID](https://support.airtable.com/docs/finding-airtable-ids#finding-base-url-ids).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/rU347YVMm2M" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The ID of the table to access in the base. [Create a table](https://www.airtable.com/guides/build/create-a-table). [Get a table's ID](https://support.airtable.com/docs/finding-airtable-ids#finding-base-url-ids).

*   The ID of the view to access in the table. [Create a view](https://www.airtable.com/guides/build/create-custom-views-of-data). [Get a view's ID](https://support.airtable.com/docs/finding-airtable-ids#finding-base-url-ids).

By default, Unstructured processes all tables from all bases within an Airtable organization. You can limit the
tables that Unstructured ingests data from within Airtable by specifying a list of Airtable paths.
An Airtable path uses the following structure: `base_id/table_id(optional)/view_id(optional)`

For example, given the following example Airtable URL:

```text
https://airtable.com/appr9nKeXLAtg6bgn/tblZ8uT1GY7NLbWit/viwDcpzf9dP0Gqz5J
```

*   The base's ID is `appr9nKeXLAtg6bgn`. The base's path is `appr9nKeXLAtg6bgn`.
*   The table's ID is `tblZ8uT1GY7NLbWit`. The table's path is `appr9nKeXLAtg6bgn/tblZ8uT1GY7NLbWit`.
*   The view's ID is `viwDcpzf9dP0Gqz5J`. The view's path is `appr9nKeXLAtg6bgn/tblZ8uT1GY7NLbWit/viwDcpzf9dP0Gqz5J`.

You can call the Airtable API to get lists of available IDs for Airtable bases, tables, and views in bulk, as follows:

*   [Base IDs](https://airtable.com/developers/web/api/list-bases)
*   [Table and view IDs](https://airtable.com/developers/web/api/get-base-schema)
*   [Base, table, and view IDs](https://pyairtable.readthedocs.io/en/latest/metadata.html)

The Airtable connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[airtable]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `AIRTABLE_TOKEN` - The Airtable personal access token, represented by `--personal-access-token` (CLI) or `personal_access_token` (Python).
*   `AIRTABLE_PATHS` - The list of Airtable paths to process, represented by `--list-of-paths` (CLI) or `list_of_paths` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    airtable \
      --personal-access-token $AIRTABLE_PERSONAL_ACCESS_TOKEN \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --reprocess \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.airtable import (
      AirtableIndexerConfig,
      AirtableDownloaderConfig,
      AirtableConnectionConfig,
      AirtableAccessConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig

  # Chunking is optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=AirtableIndexerConfig(list_of_paths=[os.getenv("AIRTABLE_PATHS")]),
          downloader_config=AirtableDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=AirtableConnectionConfig(
              access_config=AirtableAccessConfig(
                  personal_access_token=os.getenv("AIRTABLE_TOKEN")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.airtable import AirtableAccessConfig, SimpleAirtableConfig
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import AirtableRunner

  if __name__ == "__main__":
      runner = AirtableRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
          ),
          connector_config=SimpleAirtableConfig(
              access_config=AirtableAccessConfig(
                  personal_access_token=os.getenv("AIRTABLE_PERSONAL_ACCESS_TOKEN")
              ),
              list_of_paths=[os.getenv("AIRTABLE_PATHS")]
          ),
      )
      runner.run()
  ```
</CodeGroup>


# Astra DB



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Astra DB to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Astra DB connector prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/kupQrnCQjPw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An Astra account. [Create or sign in to an Astra account](https://astra.datastax.com/).
*   A database in the Astra account. [Create a database in an account](https://docs.datastax.com/en/astra-db-classic/databases/manage-create.html).
*   An application token for the database. [Create a database application token](https://docs.datastax.com/en/astra-db-serverless/administration/manage-application-tokens.html).
*   A namespace in the database. [Create a namespace in a database](https://docs.datastax.com/en/astra-db-serverless/databases/manage-namespaces.html#create-namespace).
*   A collection in the namespace. [Create a collection in a namespace](https://docs.datastax.com/en/astra-db-serverless/databases/manage-collections.html#create-collection).

The Astra DB connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[astradb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `ASTRA_DB_API_ENDPOINT` - The API endpoint for the Astra DB database, represented by `--api-endpoint` (CLI) or `api_endpoint` (Python). To get the endpoint, see the **Database Details > API Endpoint** value on your database's **Overview** tab.
*   `ASTRA_DB_APPLICATION_TOKEN` - The database application token value for the database, represented by `--token` (CLI) or `token` (Python). To get the token, see the **Database Details > Application Tokens** box on your database's **Overview** tab.
*   `ASTRA_DB_KEYSPACE` - The name of the keyspace for the database, represented by `--keyspace` (CLI) or `keyspace` (Python).
*   `ASTRA_DB_COLLECTION` - The name of the collection for the keyspace, represented by `--collection-name` (CLI) or `collection_name` (Python).
*   `ASTRA_DB_EMBEDDING_DIMENSIONS` - The number of dimensions in the collection, represented by `--embedding-dimension` (CLI) or `embedding_dimension` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    astradb \
      --api-endpoint $ASTRA_DB_API_ENDPOINT \
      --token $ASTRA_DB_APPLICATION_TOKEN \
      --keyspace $ASTRA_DB_KEYSPACE \
      --collection-name $ASTRA_DB_COLLECTION \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.astradb import (
      AstraDBAccessConfig,
      AstraDBConnectionConfig,
      AstraDBDownloaderConfig,
      AstraDBIndexerConfig,
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=AstraDBIndexerConfig(
              collection_name=os.getenv("ASTRA_DB_COLLECTION"),
              keyspace=os.getenv("ASTRA_DB_KEYSPACE"),           
          ),
          downloader_config=AstraDBDownloaderConfig(
              collection_name=os.getenv("ASTRA_DB_COLLECTION"),
              keyspace=os.getenv("ASTRA_DB_KEYSPACE"),
          ),
          source_connection_config=AstraDBConnectionConfig(
              access_config=AstraDBAccessConfig(
                  token=os.getenv("ASTRA_DB_APPLICATION_TOKEN"),
                  api_endpoint=os.getenv("ASTRA_DB_API_ENDPOINT"),
              ),
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
          ),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR")),
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.runner import AstraDBRunner
  from unstructured_ingest.connector.astradb import (
      SimpleAstraDBConfig,
      AstraDBAccessConfig
  )
  from unstructured_ingest.interfaces import (
      ProcessorConfig,
      ReadConfig,
      PartitionConfig
  )

  if __name__ == "__main__":
      runner = AstraDBRunner(
          processor_config=ProcessorConfig(
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR")
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
          ),
          connector_config=SimpleAstraDBConfig(
              access_config=AstraDBAccessConfig(
                  token=os.getenv("ASTRA_DB_APPLICATION_TOKEN"),
                  api_endpoint=os.getenv("ASTRA_DB_API_ENDPOINT")
              ),
              keyspace=os.getenv("ASTRA_DB_KEYSPACE"),
              collection_name=os.getenv("ASTRA_DB_COLLECTION")
          )
      ).run()
  ```
</CodeGroup>


# Azure



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Azure Storage to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Azure Storage account prerequisites.

The following video shows how to fulfill the minimum set of Azure Storage account prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/Vl3KCphlh9Y" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

<Note>
  If you are generating an SAS token as shown in the preceding video, be sure to set the following permissions:

  *   **Read** and **List** for reading from the container only.
  *   **Write** and **List** for writing to the container only.
  *   **Read**, **Write**, and **List** for both reading from and writing to the container.
</Note>

Here are some more details about these prerequisites:

*   An Azure account. To create one, [learn how](https://azure.microsoft.com/pricing/purchase-options/azure-account).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/2bQ6WiJ1ncA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An Azure Storage account, and a container within that account. [Create a storage account](https://learn.microsoft.com/azure/storage/common/storage-account-create). [Create a container](https://learn.microsoft.com/azure/storage/blobs/blob-containers-portal).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/AhuNgBafmUo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    <iframe width="560" height="315" src="https://www.youtube.com/embed/xmndjYnGvcs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Azure Storage remote URL, using the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

    For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
    Azure Storage remote URL would be `az://my-container/my-folder/`.

*   An SAS token (recommended), access key, or connection string for the Azure Storage account.  [Create an SAS token (recommended)](https://learn.microsoft.com/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens). [Get an access key](https://learn.microsoft.com/azure/storage/common/storage-account-keys-manage#view-account-access-keys). [Get a connection string](https://learn.microsoft.com/azure/storage/common/storage-configure-connection-string#configure-a-connection-string-for-an-azure-storage-account).

    Create an SAS token (recommended):

    <iframe width="560" height="315" src="https://www.youtube.com/embed/X6cmJ2IbVzo?start=240&end=370" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    Get an access key or connection string:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/muMmcwVfFqs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The Azure Storage account connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[azure]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `AZURE_STORAGE_REMOTE_URL` - The Azure Storage remote URL, represented by `--remote-url` (CLI) or `remote_url` (Python).

    The remote URL takes the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

    For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
    Azure Storage remote URL would be `az://my-container/my-folder/`.

*   `AZURE_STORAGE_ACCOUNT_NAME` - The name of the Azure Storage account, represented by `--account-name` (CLI) or `account_name` (Python).

*   `AZURE_STORAGE_ACCOUNT_KEY`, `AZURE_STORAGE_CONNECTION_STRING`, or `AZURE_STORAGE_SAS_TOKEN` - The name of the key, connection string, or SAS token for the Azure Storage account, depending on your security configuration, represented by `--account-key` (CLI) or `account_key` (Python), `--connection-string` (CLI) or `connection_string` (Python), and `--sas_token` (CLI) or `sas_token` (Python), respectively.

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    azure \
      --remote-url $AZURE_STORAGE_REMOTE_URL \
      --account-name $AZURE_STORAGE_ACCOUNT_NAME \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.azure import (
      AzureIndexerConfig,
      AzureDownloaderConfig,
      AzureConnectionConfig,
      AzureAccessConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=AzureIndexerConfig(remote_url=os.getenv("AZURE_STORAGE_REMOTE_URL")),
          downloader_config=AzureDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=AzureConnectionConfig(
              access_config=AzureAccessConfig(
                  account_name=os.getenv("AZURE_STORAGE_ACCOUNT_NAME"),
                  account_key=os.getenv("AZURE_STORAGE_ACCOUNT_KEY")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()

  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.azure import (
      AzureAccessConfig,
      SimpleAzureBlobStorageConfig,
  )
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import AzureRunner

  if __name__ == "__main__":
      runner = AzureRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleAzureBlobStorageConfig(
              access_config=AzureAccessConfig(
                  account_name=os.getenv("AZURE_STORAGE_ACCOUNT_NAME"),
              ),
              remote_url=os.getenv("AZURE_STORAGE_REMOTE_URL"),
          ),
      )
      runner.run()
  ```
</CodeGroup>


# Biomed



Connect Biomed to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to
store structured outputs locally on your filesystem.

This connector allows you to extract Biomedical documents from the supported FTP directories:

*   [https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa\_pdf](https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf)

Make sure to have the Biomed dependencies installed:

```bash Shell
pip install "unstructured-ingest[biomed]"
```

You need to provide the path, from which the documents should be downloaded. For example, to download the documents in
the path: `https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/07/`, set the `path` parameter to `oa_pdf/07/`

Make sure to set the `--partition-by-api` flag and pass in your API key with `--api-key`:

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  unstructured-ingest \
    biomed \
      --path "oa_pdf/07/07/sbaa031.073.PMC7234218.pdf" \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --preserve-downloads \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --strategy hi_res
  ```

  ```python Python
  import os

  from unstructured_ingest.connector.biomed import (
      SimpleBiomedConfig,
  )
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import BiomedRunner

  if __name__ == "__main__":
      runner = BiomedRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
          ),
          connector_config=SimpleBiomedConfig(path="oa_pdf/07/07/sbaa031.073.PMC7234218.pdf"),
      )
      runner.run()
  ```
</CodeGroup>

Additionally, if you're using Unstructured Serverless API, your locally deployed Unstructured API, or an Unstructured API
deployed on Azure or AWS, you also need to specify the API URL via the `--partition-endpoint` argument.


# Box



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Box to your preprocessing pipeline, and use the Unstructured CLI or Python to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Box prerequisites:

1.  Access to the [Developer Console](https://app.box.com/developers/console) from your [Box enterprise account](https://account.box.com/signup/enterprise-plan) or [Box developer account](https://account.box.com/signup/developer).

2.  A Box Custom App in your Box account, set up to use **Server Authentication (with JWT)**. See [Setup with JWT](https://developer.box.com/guides/authentication/jwt/jwt-setup/).

3.  The appropriate application scopes and advanced features set up for the Box Custom App, as follows:

    *   In the Box Custom App, on the **Configuration** tab, under **Application Scopes**, check the box titled **Write all files and folders stored in Box**.
    *   In the Box Custom App, on the **Configuration** tab, under **Advanced Features**, check the box titled **Make API calls using the as-user header**.
    *   Click **Save Changes** before continuing.

4.  Authorization by a Box Admin in your Box account for the Box Custom App. See [App Authorization](https://developer.box.com/guides/authentication/jwt/jwt-setup/#app-authorization).

5.  Access by your Box account's source or target [folder](https://app.box.com/folder/0) to your Box Custom App, as follows:

    *   In the Box Custom App, on the **General Settings** tab, copy the **Service Account ID** (which takes the form `AutomationUser_<your-app-service-id>_<a-random-string@boxdevedition.com`).
    *   **Share** your Box account's target folder with the copied service account's email address as a **Co-owner** or **Editor**.
    *   Note the remote URL to the target folder, which takes the format `box://<path/to/folder/in/account>`.

6.  The private key configuration JSON file for the Box Custom App. To download this file, in the Box Custom App, on the **Configuration** tab, under **Add and Manage Public Keys**, click **Generate a Public/Private Keypair**. Store the downloaded private key configuration JSON file in a secure location. Do not share it with others.

The Box connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[box]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `BOX_APP_CONFIG_PATH` - The local path to the downloaded private key configuration JSON file for the Box Custom App, represented by `--box-app-config` (CLI) or `box_app_config` (Python).
*   `BOX_REMOTE_URL` - The remote URL to the target folder, represented by `--remote-url` (CLI) or `remote_url` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    box \
      --box-app-config $BOX_APP_CONFIG_PATH \
      --remote-url $BOX_REMOTE_URL \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --recursive \
      --verbose \
      --partition-by-api \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --api-key $UNSTRUCTURED_API_KEY \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.box import (
      BoxAccessConfig,
      BoxConnectionConfig,
      BoxIndexerConfig,
      BoxDownloaderConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=BoxIndexerConfig(remote_url=os.getenv("BOX_REMOTE_URL")),
          downloader_config=BoxDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=BoxConnectionConfig(
              access_config=BoxAccessConfig(
                  box_app_config=os.getenv("BOX_APP_CONFIG_PATH")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.box import BoxAccessConfig, SimpleBoxConfig
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import BoxRunner

  if __name__ == "__main__":
      runner = BoxRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleBoxConfig(
              remote_url=os.getenv("BOX_REMOTE_URL"),
              recursive=True,
              access_config=BoxAccessConfig(
                  box_app_config=os.getenv("BOX_APP_CONFIG_PATH")
              ),
          ),
      )
      runner.run()
  ```
</CodeGroup>


# Confluence



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Confluence to your preprocessing pipeline, and use the Unstructured CLI or Python to batch process all your spaces and pages and store structured outputs locally on your filesystem.

You will need:

The Confluence prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/tXu6S6Pd7Dc" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A [Confluence account](https://www.atlassian.com/software/confluence).
*   The URL for the target Confluence instance.
*   A [Confluence user](https://confluence.atlassian.com/doc/add-and-invite-users-138313.html) with
    the correct [permissions](https://support.atlassian.com/confluence-cloud/docs/what-are-confluence-cloud-permissions-and-restrictions/) to
    access the target spaces and pages in the Confluence instance.
*   A [Confluence API token](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/) for the target Confluence user.
*   Optionally, the names of the specific [spaces](https://support.atlassian.com/confluence-cloud/docs/navigate-spaces/) in the Confluence instance to access.

The Confluence connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[confluence]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `CONFLUENCE_URL` - The URL of the Confluence instance, represented by `--url` (CLI) or `url` (Python).
*   `CONFLUENCE_API_TOKEN` - The value of the Confluence API token for authenticating with the Confluence instance, represented by `--api-token` (CLI) or `api_token` (Python).
*   `CONFLUENCE_USER_EMAIL` - The user's email address for authenticating with the Confluence instance, represented by `--user-email` (CLI) or `user_email` (Python).

Additional settings include:

*   `--spaces` (CLI) or `spaces` (Python): Optionally, the list of the names of the specific spaces to access, expressed as a comma-separated list of strings (CLI) or an array of strings (Python), with each string representing a space's name. The default is no specific spaces, if not otherwise specified.
*   `--max-num-of-spaces` (CLI) or `max_num_of_spaces` (Python): Optionally, the maximum number of spaces to access, expressed as an integer. The default value is `500` if not otherwise specified.
*   `--max-num-of-docs-from-each-space` (CLI) or `max_num_of_docs_from_each_space` (Python): Optionally, the maximum number of documents to access from each space, expressed as an integer. The default value is `100` if not otherwise specified.

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    confluence \
      --url $CONFLUENCE_URL \
      --spaces luke,paul \
      --user-email $CONFLUENCE_USER_EMAIL \
      --api-token $CONFLUENCE_API_TOKEN \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --chunking-strategy by_title \
      --embedding-provider huggingface
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.confluence import (
      ConfluenceIndexerConfig,
      ConfluenceDownloaderConfig,
      ConfluenceConnectionConfig,
      ConfluenceAccessConfig
  )

  # Chunking and embedding are optional.

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=ConfluenceIndexerConfig(
              spaces=["luke", "paul"]
          ),
          downloader_config=ConfluenceDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=ConfluenceConnectionConfig(
              access_config=ConfluenceAccessConfig(api_token=os.getenv("CONFLUENCE_API_TOKEN")),
              url=os.getenv("CONFLUENCE_URL"),
              user_email=os.getenv("CONFLUENCE_USER_EMAIL")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.confluence import ConfluenceAccessConfig, SimpleConfluenceConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import ConfluenceRunner

  if __name__ == "__main__":
      runner = ConfluenceRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              metadata_exclude=["filename", "file_directory", "metadata.data_source.date_processed"],
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleConfluenceConfig(
              access_config=ConfluenceAccessConfig(
                  api_token=os.getenv("CONFLUENCE_API_TOKEN"),
              ),
              user_email=os.getenv("CONFLUENCE_USER_EMAIL"),
              url=os.getenv("CONFLUENCE_URL"),
          ),
      )
      runner.run()
  ```
</CodeGroup>


# Couchbase



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Couchbase Database to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Couchbase database prerequisites.

*   For the [Unstructured Platform](/platform/overview), only Couchbase Capella clusters are supported.
*   For [Unstructured Ingest](/ingestion/overview), Couchbase Capella clusters and local Couchbase server deployments are supported.

<iframe width="560" height="315" src="https://www.youtube.com/embed/9-RIBmIdi70" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

For Couchbase Capella, you will need:

*   A Couchbase Capella account. [Create an account](https://docs.couchbase.com/cloud/get-started/create-account.html).
*   A Couchbase Capella cluster. [Create a cluster](https://docs.couchbase.com/cloud/clusters/create-database.html).
*   A [bucket, scope, and collection](https://docs.couchbase.com/cloud/clusters/data-service/about-buckets-scopes-collections.html).
*   A cluster connection string. [Learn how](https://docs.couchbase.com/cloud/get-started/connect.html).
*   Incoming IP address allowance. [Learn how](https://docs.couchbase.com/cloud/clusters/allow-ip-address.html).
*   A username and password. [Learn how](https://docs.couchbase.com/cloud/clusters/manage-database-users.html).

For a local Couchbase server, you will need:

*   Installation of a local Couchbase server. [Learn how](https://docs.couchbase.com/server/current/getting-started/start-here.html).
*   Connection details to the local Couchbase server. [Learn how](https://docs.couchbase.com/server/current/guides/connect.html).

To learn more about how to set up a Couchbase cluster and play with data, refer to this [tutorial](https://developer.couchbase.com/tutorial-quickstart-flask-python).

The Couchbase DB connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[couchbase]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables are required for the Couchbase Connector:

*   `CB_CONN_STR` - The Connection String for the Couchbase server, represented by `--connection-string` (CLI) or `connection_string` (Python).
*   `CB_USERNAME` - The username for the Couchbase server, represented by `--username` (CLI) or `username` (Python).
*   `CB_PASSWORD` - The password for the Couchbase server, represented by `--password` (CLI) or `password` (Python).
*   `CB_BUCKET` - The name of the bucket in the Couchbase server, represented by `--bucket` (CLI) or `bucket` (Python).
*   `CB_SCOPE` - The name of the scope in the bucket, represented by `--scope` (CLI) or `scope` (Python).
*   `CB_COLLECTION` - The name of the collection in the scope, represented by `--collection` (CLI) or `collection` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    couchbase \
      --metadata-exclude filename,file_directory,metadata.data_source.date_processed \
      --connection-string "$COUCHBASE_CONNECTION_STRING" \
      --username $COUCHBASE_USERNAM \
      --password $COUCHBASE_PASSWORD \
      --bucket $COUCHBASE_BUCKET \
      --scope $COUCHBASE_SCOPE \
      --collection $COUCHBASE_COLLECTION \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.couchbase import (
      CouchbaseIndexerConfig,
      CouchbaseDownloaderConfig,
      CouchbaseConnectionConfig,
      CouchbaseAccessConfig,
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=CouchbaseIndexerConfig(batch_size=100),
          downloader_config=CouchbaseDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=CouchbaseConnectionConfig(
              access_config=CouchbaseAccessConfig(
                  password=os.getenv("CB_PASSWORD")
              ),
              username=os.getenv("CB_USERNAME"),
              connection_string=os.getenv("CB_CONN_STR"),
              bucket=os.getenv("CB_BUCKET"),
              scope=os.getenv("CB_SCOPE"),
              collection=os.getenv("CB_COLLECTION")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig()
      ).run()
  ```
</CodeGroup>


# Databricks Volumes



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Databricks Volumes to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Databricks Volumes prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/rNZpwa1-g7M" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The preceding video shows how to use Databricks personal access tokens (PATs), which are supported only for [Unstructured Ingest](/ingestion/overview).

To learn how to use Databricks-managed service principals, which are supported by both the [Unstructured Platform](/platform/overview) and Unstructured Ingest,
see the additional videos later on this page.

*   The Databricks workspace URL. Get the workspace URL for
    [AWS](https://docs.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids),
    [Azure](https://learn.microsoft.com/azure/databricks/workspace/workspace-details#workspace-instance-names-urls-and-ids),
    or [GCP](https://docs.gcp.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids).

    Examples:

    *   AWS: `https://<workspace-id>.cloud.databricks.com`
    *   Azure: `https://adb-<workspace-id>.<random-number>.azuredatabricks.net`
    *   GCP: `https://<workspace-id>.<random-number>.gcp.databricks.com`

*   The Databricks authentication details. For more information, see the documentation for
    [AWS](https://docs.databricks.com/dev-tools/auth/index.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/),
    or [GCP](https://docs.gcp.databricks.com/dev-tools/auth/index.html).

    The following videos show how to create a Databricks-managed service principal and then grant it access to a Databricks volume:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/wBmqv5DaA1E" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    <iframe width="560" height="315" src="https://www.youtube.com/embed/DykQRxgh2aQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    For the [Unstructured Platform](/platform/overview), only the following Databricks authentication type is supported:

    *   For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.
        Note that for Azure, only Databricks-managed service principals are supported. Microsoft Entra ID-managed service principals are not supported.

    For [Unstructured Ingest](/ingestion/overview), the following Databricks authentication types are supported:

    *   For Databricks personal access token authentication (AWS, Azure, and GCP): The personal access token's value.
    *   For username and password (basic) authentication (AWS only): The user's name and password values.
    *   For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.
    *   For OAuth user-to-machine (U2M) authentication (AWS, Azure, and GCP): No additional values.
    *   For Azure managed identities (MSI) authentication (Azure only): The client ID value for the corresponding managed identity.
    *   For Microsoft Entra ID service principal authentication (Azure only): The tenant ID, client ID, and client secret values for the corresponding service principal.
    *   For Azure CLI authentication (Azure only): No additional values.
    *   For Microsoft Entra ID user authentication (Azure only): The Entra ID token for the corresponding Entra ID user.
    *   For Google Cloud Platform credentials authentication (GCP only): The local path to the corresponding Google Cloud service account's credentials file.
    *   For Google Cloud Platform ID authentication (GCP only): The Google Cloud service account's email address.

*   The Databricks catalog name for the volume. Get the catalog name for [AWS](https://docs.databricks.com/catalogs/manage-catalog.html), [Azure](https://learn.microsoft.com/azure/databricks/catalogs/manage-catalog), or [GCP](https://docs.gcp.databricks.com/catalogs/manage-catalog.html).

*   The Databricks schema name for the volume. Get the schema name for [AWS](https://docs.databricks.com/schemas/manage-schema.html), [Azure](https://learn.microsoft.com/azure/databricks/schemas/manage-schema), or [GCP](https://docs.gcp.databricks.com/schemas/manage-schema.html).

*   The Databricks volume name, and optionally any path in that volume that you want to access directly. Get the volume information for [AWS](https://docs.databricks.com/files/volumes.html), [Azure](https://learn.microsoft.com/azure/databricks/files/volumes), or [GCP](https://docs.gcp.databricks.com/files/volumes.html).

*   Make sure that the target user or service principal has access to the target volume. To learn more, see the documentation for
    [AWS](https://docs.databricks.com/volumes/utility-commands.html#change-permissions-on-a-volume),
    [Azure](https://learn.microsoft.com/azure/databricks/volumes/utility-commands#change-permissions-on-a-volume),
    or [GCP](https://docs.gcp.databricks.com/volumes/utility-commands.html#change-permissions-on-a-volume).

The Databricks Volumes connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[databricks-volumes]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `DATABRICKS_HOST` - The Databricks host URL, represented by `--host` (CLI) or `host` (Python).
*   `DATABRICKS_CATALOG` - The Databricks catalog name for the Volume, represented by `--catalog` (CLI) or `catalog` (Python).
*   `DATABRICKS_SCHEMA` - The Databricks schema name for the Volume, represented by `--schema` (CLI) or `schema` (Python). If not specified, `default` is used.
*   `DATABRICKS_VOLUME` - The Databricks Volume name, represented by `--volume` (CLI) or `volume` (Python).
*   `DATABRICKS_VOLUME_PATH` - Any optional path to access within the volume, specified by `--volume-path` (CLI) or `volume_path` (Python).

Environment variables based on your authentication type, depending on your cloud provider:

*   For Databricks personal access token authentication (AWS, Azure, and GCP):

    *   `DATABRICKS_TOKEN` - The personal access token, represented by `--token` (CLI) or `token` (Python).

*   For username and password (basic) authentication (AWS only): The user's name and password values.

    *   `DATABRICKS_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
    *   `DATABRICKS_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

*   For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.

    *   `DATABRICKS_CLIENT_ID` - The client ID value for the corresponding service principal, represented by `--client-id` (CLI) or `client_id` (Python).
    *   `DATABRICKS_CLIENT_SECRET` - The client ID and OAuth secret values for the corresponding service principal, represented by `--client-secret` (CLI) or `client_secret` (Python).

*   For OAuth user-to-machine (U2M) authentication (AWS, Azure, and GCP): No additional environment variables.

*   For Azure managed identities (MSI) authentication (Azure only):

    *   `ARM_CLIENT_ID` - The client ID value for the corresponding managed identity, represented by `--azure-client-id` (CLI) or `azure_client_id` (Python).
    *   If the target identity has not already been added to the workspace, then you must also specify the
        `DATABRICKS_AZURE_RESOURCE_ID`, represented by `--azure-workspace-resource-id` (CLI) or `azure_workspace_resource_id` (Python).

*   For Microsoft Entra ID service principal authentication (Azure only):

    *   `ARM_TENANT_ID` - The tenant ID value for the corresponding service principal, represented by `--azure-tenant-id` (CLI) or `azure_tenant_id` (Python).
    *   `ARM_CLIENT_ID` - The client ID value for the corresponding service principal, represented by `--azure-client-id` (CLI) or `azure_client_id` (Python).
    *   `ARM_CLIENT_SECRET` - The client secret value for the corresponding service principal, represented by `--azure-client-secret` (CLI) or `azure_client_secret` (Python).
    *   If the service principal has not already been added to the workspace, then you must also specify the
        `DATABRICKS_AZURE_RESOURCE_ID`, represented by `--azure-workspace-resource-id` (CLI) or `azure_workspace_resource_id` (Python).

*   For Azure CLI authentication (Azure only): No additional environment variables.

*   For Microsoft Entra ID user authentication (Azure only):

    *   `DATABRICKS_TOKEN` - The Entra ID token for the corresponding Entra ID user, represented by `--token` (CLI) or `token` (Python).

*   For Google Cloud Platform credentials authentication (GCP only):

    *   `GOOGLE_CREDENTIALS` - The local path to the corresponding Google Cloud service account's credentials file, represented by `--google-credentials` (CLI) or `google_credentials`

*   For Google Cloud Platform ID authentication (GCP only):

    *   `GOOGLE_SERVICE_ACCOUNT` - The Google Cloud service account's email address, represented by `--google-service-account` (CLI) or `google_service_account` (Python).

*   Alternatively, you can store the preceding settings in a local
    [Databricks configuration profile](https://docs.databricks.com/en/dev-tools/auth/config-profiles.html) and then just
    refer to the profile's name:

    *   `DATABRICKS_PROFILE` - The name of the Databricks configuration profile, represented by `--profile` (CLI) or `profile` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    databricks-volumes \
      --profile $DATABRICKS_PROFILE \
      --host $DATABRICKS_HOST \
      --catalog $DATABRICKS_CATALOG \
      --schema $DATABRICKS_SCHEMA \
      --volume $DATABRICKS_VOLUME \
      --volume-path $DATABRICKS_VOLUME_PATH \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  # For all supported Databricks authentication types, you can import this:
  from unstructured_ingest.v2.processes.connectors.databricks_volumes import (
      DatabricksVolumesIndexerConfig,
      DatabricksVolumesConnectionConfig,
      DatabricksVolumesAccessConfig,
      DatabricksVolumesDownloaderConfig
  )

  # Alternatively, for supported Databricks on AWS authentication types only, you can import this:
  # from unstructured_ingest.v2.processes.connectors.databricks.volumes_aws import (
  #     DatabricksAWSVolumesIndexerConfig,
  #     DatabricksAWSVolumesConnectionConfig,
  #     DatabricksAWSVolumesAccessConfig,
  #     DatabricksAWSVolumesDownloaderConfig
  # )

  # Alternatively, for supported Azure Databricks authentication types only, you can import this:
  # from unstructured_ingest.v2.processes.connectors.databricks.volumes_azure import (
  #     DatabricksAzureVolumesIndexerConfig,
  #     DatabricksAzureVolumesConnectionConfig,
  #     DatabricksAzureVolumesAccessConfig,
  #     DatabricksAzureVolumesDownloaderConfig
  # )

  # Alternatively, for supported Databricks on Google Cloud authentication types only, you can import this:
  # from unstructured_ingest.v2.processes.connectors.databricks.volumes_gcp import (
  #     DatabricksGoogleVolumesIndexerConfig,
  #     DatabricksGoogleVolumesConnectionConfig,
  #     DatabricksGoogleVolumesAccessConfig,
  #     DatabricksGoogleVolumesDownloaderConfig
  # )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalConnectionConfig,
      LocalUploaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(reprocess=True),
          indexer_config=DatabricksVolumesIndexerConfig(recursive=True),
          # For specifying a Databricks configuration profile:
          downloader_config=DatabricksVolumesDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=DatabricksVolumesConnectionConfig(
              access_config=DatabricksVolumesAccessConfig(profile=os.getenv("DATABRICKS_PROFILE")),
              host=os.getenv("DATABRICKS_HOST"),
              catalog=os.getenv("DATABRICKS_CATALOG"),
              schema=os.getenv("DATABRICKS_SCHEMA"),
              volume=os.getenv("DATABRICKS_VOLUME"),
              volume_path=os.getenv("DATABRICKS_VOLUME_PATH")
          ),
          # Other examples:
          #
          # For Databricks on AWS, with Databricks personal access token authentication:
          # downloader_config=DatabricksAWSVolumesDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          # source_connection_config=DatabricksAWSVolumesConnectionConfig(
          #     access_config=DatabricksAWSVolumesAccessConfig(token=os.getenv("DATABRICKS_TOKEN")),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          #
          # For Azure Databricks, with Microsoft Entra ID service principal authentication:
          # downloader_config=DatabricksAzureVolumesDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          # source_connection_config=DatabricksAzureVolumesConnectionConfig(
          #     access_config=DatabricksAzureVolumesAccessConfig(
          #         azure_client_id=os.getenv("ARM_CLIENT_ID"),
          #         azure_client_secret=os.getenv("ARM_CLIENT_SECRET"),
          #         azure_tenant_id=os.getenv("ARM_TENANT_ID")   
          #     ),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          #
          # For Databricks on Google Cloud, with Google Cloud Platform credentials authentication:
          # downloader_config=DatabricksGoogleVolumesDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          # source_connection_config=DatabricksGoogleVolumesConnectionConfig(
          #     access_config=DatabricksGoogleVolumesAccessConfig(
          #         google_service_account=os.getenv("GOOGLE_CREDENTIALS")
          #     ),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>


# Delta Table



Connect Delta Table to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to
store structured outputs locally on your filesystem.

Make sure to have the Delta Table dependencies installed:

```bash Shell
pip install "unstructured-ingest[delta-table]"
```

AWS credentials need to be available for use with the storage options.
Specify the to the DeltaTable using the `table-uri` argument, and pass a dictionary of the options to use for the storage backend
via `storage_options`.

Make sure to set the `--partition-by-api` flag and pass in your API key with `--api-key`:

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  unstructured-ingest \
    delta-table \
      --table-uri s3://utic-dev-tech-fixtures/sample-delta-lake-data/deltatable/ \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --storage_options "AWS_REGION=us-east-2,AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY" \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --strategy hi_res
  ```

  ```python Python
  import os

  from unstructured_ingest.connector.delta_table import SimpleDeltaTableConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import DeltaTableRunner

  if __name__ == "__main__":
      runner = DeltaTableRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
          ),
          connector_config=SimpleDeltaTableConfig(
              table_uri="s3://utic-dev-tech-fixtures/sample-delta-lake-data/deltatable/",
              storage_options={
                  "AWS_REGION": "us-east-2",
                  "AWS_ACCESS_KEY_ID": os.getenv("AWS_ACCESS_KEY_ID"),
                  "AWS_SECRET_ACCESS_KEY": os.getenv("AWS_SECRET_ACCESS_KEY"),
              },
          ),
      )
      runner.run()
  ```
</CodeGroup>

Additionally, if you're using Unstructured Serverless API, your locally deployed Unstructured API, or an Unstructured API
deployed on Azure or AWS, you also need to specify the API URL via the `--partition-endpoint` argument.


# Discord



Connect Discord to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to
store structured outputs locally on your filesystem.

Make sure to have the Discord dependencies installed:

```bash Shell
pip install "unstructured-ingest[discord]"
```

To ingests the contents of Discord channels, you need to supply the following information:

*   `token`: an authentication token used to access Discord API
*   `channels`: a list of discord channel ids to ingest from

Optionally you can set the number of days to go back in history of the channels via the `period` argument.

Make sure to set the `--partition-by-api` flag and pass in your API key with `--api-key`:

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  unstructured-ingest \
    discord \
      --channels 12345678 \
      --token $DISCORD_TOKEN \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --output-dir discord-example \
      --preserve-downloads \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --strategy hi_res
  ```

  ```python Python
  import os

  from unstructured_ingest.connector.discord import DiscordAccessConfig, SimpleDiscordConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import DiscordRunner

  if __name__ == "__main__":
      runner = DiscordRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
              strategy="hi_res",
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
          ),
          connector_config=SimpleDiscordConfig(
              access_config=DiscordAccessConfig(token=os.getenv("DISCORD_TOKEN")),
              channels=["12345678"],
          ),
      )
      runner.run()
  ```
</CodeGroup>

Additionally, if you're using Unstructured Serverless API, your locally deployed Unstructured API, or an Unstructured API
deployed on Azure or AWS, you also need to specify the API URL via the `--partition-endpoint` argument.


# Dropbox



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Dropbox to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Dropbox prerequisites:

1.  A Dropbox account. [Get an account](https://www.dropbox.com/try/teams).

2.  A target source or destination folder in your Dropbox account. [Create a folder](https://help.dropbox.com/create-upload/creating-using-folders-on-dropbox).

3.  A Dropbox app for your Dropbox account. To learn how to create an app, click the **App Console** tab on the [Getting Started](https://www.dropbox.com/developers/reference/getting-started) page.

4.  Permission for your Dropbox app to read from, and write to, the target folder in your Dropbox account as needed. To do this:

    *   On the **Permissions** tab of your Dropbox app, check the boxes **files.content.read** or **files.content.write** or both. [Learn more](https://developers.dropbox.com/oauth-guide).

    *   On the **Settings** tab of your Dropbox app, for **App folder name**, set the name of the target folder in your Dropbox account for your Dropbox app to have access to.

    *   Note the remote URL to the target folder, which takes the format `dropbox://<path/to/folder/in/account>`.

5.  An access token for your Dropbox account. [Get a token](https://dropbox.tech/developers/generate-an-access-token-for-your-own-account). Save this token in a secure location. Do not share it with others.

The Dropbox connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[dropbox]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `DROPBOX_ACCESS_TOKEN` - The value of your access token, represented by `--token` (CLI) or `token` (Python).
*   `DROPBOX_REMOTE_URL` - The remote URL to the target folder, represented by `--remote-url` (CLI) or `remote_url` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    dropbox \
      --remote-url $DROPBOX_REMOTE_URL \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --token $DROPBOX_ACCESS_TOKEN \
      --num-processes 2 \
      --recursive \
      --verbose \
      --partition-by-api \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --api-key $UNSTRUCTURED_API_KEY \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.dropbox import (
      DropboxIndexerConfig,
      DropboxDownloaderConfig,
      DropboxAccessConfig,
      DropboxConnectionConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalConnectionConfig,
      LocalUploaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=DropboxIndexerConfig(remote_url=os.getenv("DROPBOX_REMOTE_URL")),
          downloader_config=DropboxDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=DropboxConnectionConfig(
              access_config=DropboxAccessConfig(
                  token=os.getenv("DROPBOX_ACCESS_TOKEN")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.dropbox import DropboxAccessConfig, SimpleDropboxConfig
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import DropboxRunner

  if __name__ == "__main__":
      runner = DropboxRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleDropboxConfig(
              access_config=DropboxAccessConfig(token=os.getenv("DROPBOX_ACCESS_TOKEN")),
              remote_url=os.getenv("DROPBOX_REMOTE_URL"),
              recursive=True,
          ),
      )
      runner.run()
  ```
</CodeGroup>


# Elasticsearch



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Elasticsearch to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Elasticsearch prerequisites.

*   For the [Unstructured Platform](/platform/overview), only Elastic Cloud instances are supported.

*   For [Unstructured Ingest](/ingestion/overview), Elastic Cloud instances and self-manged Elasticsearch instances are supported.

*   For Elastic Cloud, you will need an [Elastic Cloud service instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#hosted-elasticsearch-service).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/HEYIFC7uOus" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For self-managed Elasticsearch, you will need a [self-managed Elasticsearch instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#elasticsearch-deployment-options).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/9znbAkNG1Ok" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   You will need the name of the index on the instance. See [Create index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html) and [Get index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html).

    The Elasticsearch index that you use must have a schema that is compatible with the schema of the documents
    that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following index schema example for your own needs:

    ```json
    {
        "settings": {
            "index": {
                "knn": true,
                "knn.algo_param.ef_search": 100
            }
        },
        "mappings": {
            "properties": {
                "element_id": {
                    "type": "keyword"
                },
                "text": {
                    "type": "text"
                },
                "embeddings": {
                    "type": "dense_vector",
                    "dims": 384,
                    "index": true,
                    "similarity": "cosine"
                },
                "metadata": {
                    "type": "object",
                    "properties": {
                        "parent_id": {
                            "type": "text"
                        },
                        "page_number": {
                            "type": "integer"
                        },
                        "is_continuation": {
                            "type": "boolean"
                        },
                        "orig_elements": {
                            "type": "text"
                        }
                    }
                }
            }
        }
    }
    ```

    See also:

    *   [An Introduction to Elasticsearch Mapping](https://www.elastic.co/blog/found-elasticsearch-mapping-introduction)
    *   [Explicit mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html)
    *   [Dynamic field mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html)
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

    <iframe width="560" height="315" src="https://www.youtube.com/embed/0f0tEcrvP8g" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For Elastic Cloud, you will need the Elastic Cloud service instance's API key. If you are using Unstructured Ingest, you will also need the instance's Cloud ID. To get these, see your Elasticsearch Service web console.

*   For self-managed Elasticsearch, you will need:

    *   The self-managed instance's hostname and port number. See [Networking](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html).
    *   If you're using basic authentication to the self-managed instance, the user's name and password.
    *   If you're using token-based authentication to the self-managed instance, the bearer token or API key for the instance. See [Token-based authentication services](https://www.elastic.co/guide/en/elasticsearch/reference/current/token-authentication-services.html) and [Create API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html).
    *   If you're using certificate authentication to the self-managed instance, the path to the Certificate Authority (CA) file on the instance, and the certificate fingerprint. See [SSL certificate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-ssl.html) and [Where can I see my Certificate Fingerprint?](https://discuss.elastic.co/t/where-can-i-see-my-certificate-fingerprint/319335/3).

The Elasticsearch connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[elasticsearch]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `ELASTICSEARCH_HOST` - The hostname and port number, defined as `<hostname>:<port-number>` and represented by `--hosts` (CLI) or `hosts` (Python).
*   `ELASTICSEARCH_INDEX_NAME` - The name of the search index, represented by `--index-name` (CLI) or `index_name` (Python).

If you're using Elastic Cloud to connect to ElasticSearch:

*   `ELASTIC_CLOUD_ID` - The instance's Cloud ID, represented by `--cloud-id` (CLI) or `cloud_id` (Python).
*   `ELASTIC_CLOUD_API_KEY` - The corresponding Cloud API key value, represented by `--api-key-id` (CLI) or `api_key_id` (Python).

If you're using basic authentication to the instance:

*   `ELASTICSEARCH_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
*   `ELASTICSEARCH_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

If you're using token-based authentication to the instance instead:

*   `ELASTICSEARCH_BEARER_TOKEN` - The bearer token's value, represented by `--bearer-auth` (CLI) or `bearer_auth` (Python).
*   `ELASTIC_CLOUD_API_KEY_ID` - The API key's value, represented by `--api-key` (CLI) or `api_key` (Python).

If you're using certificates:

*   `ELASTICSEARCH_CA_CERTS` - The path to the Certificate Authority (CA) file, represented by `--ca-certs` (CLI) or `ca_certs` (Python).
*   `ELASTICSEARCH_SSL_ASSERT_FINGERPRINT` - The certificate's fingerprint, represented by `--ssl-assert-fingerprint` or `ssl_assert_fingerprint` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    elasticsearch \
      --metadata-exclude filename,file_directory,metadata.data_source.date_processed \
      --hosts $ELASTICSEARCH_HOST \
      --username $ELASTICSEARCH_USERNAME \
      --password $ELASTICSEARCH_PASSWORD \
      --index-name $ELASTICSEARCH_INDEX_NAME \
      --ca-certs $ELASTICSEARCH_CA_CERTS \
      --ssl-assert-fingerprint $ELASTICSEARCH_SSL_ASSERT_FINGERPRINT \
      --fields 'ethnicity, director, plot' \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.elasticsearch import (
      ElasticsearchIndexerConfig,
      ElasticsearchDownloaderConfig,
      ElasticsearchConnectionConfig,
      ElasticsearchAccessConfig,
      ElasticsearchUploadStagerConfig,
      ElasticsearchUploaderConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=ElasticsearchIndexerConfig(index_name=os.getenv("ELASTICSEARCH_INDEX_NAME")),
          downloader_config=ElasticsearchDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=ElasticsearchConnectionConfig(
              access_config=ElasticsearchAccessConfig(
                  password=os.getenv("ELASTICSEARCH_PASSWORD"),
                  ssl_assert_fingerprint=os.getenv("ELASTICSEARCH_SSL_ASSERT_FINGERPRINT")
              ),
              hosts=[os.getenv("ELASTICSEARCH_HOST")],
              username=os.getenv("ELASTICSEARCH_USERNAME"),
              ca_certs=os.getenv("ELASTICSEARCH_CA_CERTS")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig()
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.elasticsearch import (
      ElasticsearchAccessConfig,
      SimpleElasticsearchConfig,
  )
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import ElasticSearchRunner

  if __name__ == "__main__":
      runner = ElasticSearchRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              metadata_exclude=["filename", "file_directory", "metadata.data_source.date_processed"],
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleElasticsearchConfig(
              access_config=ElasticsearchAccessConfig(
                  password=os.getenv("ELASTICSEARCH_PASSWORD"),
                  ssl_assert_fingerprint=os.getenv("ELASTICSEARCH_SSL_ASSERT_FINGERPRINT")
              ),
              hosts=[os.getenv("ELASTICSEARCH_HOST")],
              username=os.getenv("ELASTICSEARCH_USERNAME"),
              ca_certs=os.getenv("ELASTICSEARCH_CA_CERTS"),
              index_name="movies",
              fields=["ethnicity", "director", "plot"],
          ),
      )
      runner.run()
  ```
</CodeGroup>


# GitHub



Connect GitHub to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to store structured outputs locally on your filesystem.

First, install the GitHub dependencies as shown here.

```bash
pip install "unstructured-ingest[github]"
```

Provide the GitHub repo URL (`url`) to fetch the files from, e.g. `"https://github.com/Unstructured-IO/unstructured"`
or `"Unstructured-IO/unstructured"`, and supply your GitHub access token (`git-access-token`).
Learn more about GitHub authentication [here](https://docs.github.com/en/authentication).

Optionally, specify a branch and what file types to limit the ingestion to:

*   `git-branch`: The branch to fetch files from. If not given, the default repository branch is used.
*   `git-file-glob`: A comma-separated list of file globs to limit which types of files are accepted, e.g. `'*.html,*.txt'`

Make sure to set the `--partition-by-api` flag and pass in your API key with `--api-key`:

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  unstructured-ingest \
    github \
      --url Unstructured-IO/unstructured \
      --git-branch main \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --strategy hi_res
  ```

  ```python Python
  import os

  from unstructured_ingest.connector.git import GitAccessConfig
  from unstructured_ingest.connector.github import SimpleGitHubConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import GithubRunner

  if __name__ == "__main__":
      runner = GithubRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
          ),
          connector_config=SimpleGitHubConfig(
              url="Unstructured-IO/unstructured", branch="main", access_config=GitAccessConfig()
          ),
      )
      runner.run()
  ```
</CodeGroup>

Additionally, if you're using Unstructured SaaS API, your locally deployed Unstructured API, or an Unstructured API
deployed on Azure or AWS, you also need to specify the API URL via the `--partition-endpoint` argument.


# GitLab



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect GitLab to your preprocessing pipeline, and use the Unstructured CLI or Python to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The GitLab prerequisites:

*   A [GitLab account](https://gitlab.com/users/sign_up).
*   The URL for the target [GitLab repository](https://docs.gitlab.com/ee/user/project/repository/).
*   A [GitLab access token](https://docs.gitlab.com/ee/security/tokens/) that allows access to the repository.
*   Optionally, the name of a specific branch to access in the repository. The repository's default repository branch is used if not otherwise specified.
*   Optionally, a list of file globs to limit which types of files are accepted, for example `*.html` or `*.txt`.

The GitLab connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[gitlab]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `GITLAB_REPO_URL` - The URL for the target GitLab repository, represented by `--url` (CLI) or `url` (Python).
*   `GITLAB_TOKEN` - The GitLab access token that allows access to the repository, represented by `--git-access-token` (CLI) or `access_token` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    gitlab \
    --url $GITLAB_URL \
    --git-access-token $GITLAB_TOKEN \
    --git-branch v0.0.7 \
    --git-file-glob "*.html,*.txt" \
    --output-dir $LOCAL_FILE_OUTPUT_DIR \
    --partition-by-api \
    --api-key $UNSTRUCTURED_API_KEY \
    --partition-endpoint $UNSTRUCTURED_API_URL \
    --strategy hi_res \
    --chunking-strategy by_title \
    --embedding-provider huggingface \
    --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest v2
  # pip install "unstructured-ingest[gitlab]"

  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.gitlab import (
      GitLabIndexerConfig,
      GitLabDownloaderConfig,
      GitLabConnectionConfig,
      GitLabAccessConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=GitLabIndexerConfig(
              path="/",
              recursive=True,
              git_branch="v0.0.7"
          ),
          downloader_config=GitLabDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=GitLabConnectionConfig(
              access_config=GitLabAccessConfig(access_token=os.getenv("GITLAB_TOKEN")),
              url=os.getenv("GITLAB_URL")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.git import GitAccessConfig
  from unstructured_ingest.connector.gitlab import SimpleGitlabConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import GitlabRunner

  if __name__ == "__main__":
      runner = GitlabRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleGitlabConfig(
              url=os.getenv("GITLAB_REPO_URL"),
              branch="v0.0.7",
              access_config=GitAccessConfig(),
          ),
      )
      runner.run()
  ```
</CodeGroup>


# Google Cloud Storage



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Google Cloud Storage to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Google Cloud Storage prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/HYaALQ0F-L4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A Google Cloud service account. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#console).

*   A service account key for the service account. See [Create a service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating) in
    [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete).

    To ensure maximum compatibility across Unstructured service offerings, you should give the service account key information to Unstructured as
    a single-line string that contains the contents of the downloaded service account key file (and not the service account key file itself).
    To print this single-line string without line breaks, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
    In this command, replace `<path-to-downloaded-key-file>` with the path to the service account key file that you downloaded by following the preceding instructions.

    *   For macOS or Linux:
        ```text
        tr -d '\n' < <path-to-downloaded-key-file>
        ```
    *   For Windows:
        ```text
        (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
        ```

*   The URI for a Google Cloud Storage bucket. This URI consists of the target bucket name, plus any target folder within the bucket, expressed as `gs://<bucket-name>[/folder-name]`. [Create a bucket](https://cloud.google.com/storage/docs/creating-buckets#console).

    This bucket must have, at minimum, one of the following roles applied to the target Google Cloud service account:

    *   `Storage Object Viewer` for bucket read access.
    *   `Storage Object Creator` for bucket write access.
    *   The `Storage Object Admin` role provides read and write access, plus access to additional bucket operations.

    To apply one of these roles to a service account for a bucket, see [Add a principal to a bucket-level policy](https://cloud.google.com/storage/docs/access-control/using-iam-permissions#bucket-add)
    in [Set and manage IAM policies on buckets](https://cloud.google.com/storage/docs/access-control/using-iam-permissions).

The Google Cloud Storage connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[gcs]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `GCS_SERVICE_ACCOUNT_KEY` - The Google Cloud service account key for Google Cloud Storage, represented by `--service-account-key` (CLI) or `service_account_key` (Python).
*   `GCS_REMOTE_URL` - The Google Cloud Storage bucket URL, represented by `--remote-url` (CLI) or `remote_url` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    gcs \
      --service-account-key $GCS_SERVICE_ACCOUNT_KEY \
      --remote-url $GCS_REMOTE_URL \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --recursive \
      --verbose \
      --strategy hi_res \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.gcs import (
      GcsIndexerConfig,
      GcsDownloaderConfig,
      GcsConnectionConfig,
      GcsAccessConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=GcsIndexerConfig(remote_url=os.getenv("GCS_REMOTE_URL")),
          downloader_config=GcsDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=GcsConnectionConfig(
              access_config=GcsAccessConfig(
                  service_account_key=os.getenv("GCS_SERVICE_ACCOUNT_KEY")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.gcs import GcsAccessConfig, SimpleGcsConfig
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import GCSRunner

  if __name__ == "__main__":
      runner = GCSRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
          ),
          connector_config=SimpleGcsConfig(
              access_config=GcsAccessConfig(token=os.getenv("GCS_SERVICE_ACCOUNT_KEY")),
              remote_url=os.getenv("GCS_REMOTE_URL"),
              recursive=True
          ),
      )
      runner.run()
  ```
</CodeGroup>


# Google Drive



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Google Drive to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Google Drive prerequisites:

1.  A Google Cloud service account and its related `credentials.json` key file or its contents in JSON format. [Learn how](https://developers.google.com/workspace/guides/create-credentials#service-account).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/0-cS_qI0XCE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

2.  Note the local path to the `credentials.json` key file. Or, copy the key file's contents into a compatible string—including properly escaped quotes—as required.
    To make converting a JSON object into a compatible string easier, you can search the Internet by using a search phrase such as "tools for converting a JSON object into a string."
    Before using any tool, check to make sure that the tool does not share the key file's contents with anyone you do not trust.

3.  Give the service account's email address access to the Google Drive folder. [Learn more](https://www.googlecloudcommunity.com/gc/Workspace-Q-A/Can-i-give-access-to-document-of-google-drive-to-service-account/m-p/530106).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/ykJQzEe_2dM" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

4.  Note the folder ID. This is a part of the URL for your Google Drive folder represented in the following URL as `{folder_id}`: `https://drive.google.com/drive/folders/{folder-id}`.

The Google Drive connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[google-drive]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `GOOGLE_DRIVE_FOLDER_ID` - The folder ID, represented by `--drive-id` (CLI) or `drive_id` (Python).
*   `GCP_SERVICE_ACCOUNT_KEY_FILEPATH` - The path to the `credentials.json` key file, represented by `--service-account-key-path` (CLI) or `service_account_key_path` (Python), or
*   `GCP_SERVICE_ACCOUNT_KEY_STRING` - The contents of the `credentials.json` key file as a string, represented by `--service-account-key` (CLI) or `service_account_key` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    google-drive \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --drive-id $GOOGLE_DRIVE_FOLDER_ID \
      --service-account-key-path $GCP_SERVICE_ACCOUNT_KEY_FILEPATH \ # Or
      --service-account-key $GCP_SERVICE_ACCOUNT_KEY_STRING \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.google_drive import (
      GoogleDriveConnectionConfig,
      GoogleDriveAccessConfig,
      GoogleDriveIndexerConfig,
      GoogleDriveDownloaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=GoogleDriveIndexerConfig(),
          downloader_config=GoogleDriveDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=GoogleDriveConnectionConfig(
              access_config=GoogleDriveAccessConfig(
                  service_account_key_path=os.getenv("GCP_SERVICE_ACCOUNT_KEY_FILEPATH"), # Or
                  service_account_key=os.getenv("GCP_SERVICE_ACCOUNT_KEY_STRING")
              ),
              drive_id=os.getenv("GOOGLE_DRIVE_FOLDER_ID"),
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.google_drive import (
      GoogleDriveAccessConfig,
      SimpleGoogleDriveConfig,
  )
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import GoogleDriveRunner

  if __name__ == "__main__":
      runner = GoogleDriveRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleGoogleDriveConfig(
              access_config=GoogleDriveAccessConfig(
                  service_account_key=os.getenv("GCP_SERVICE_ACCOUNT_KEY_STRING")
              ),
              recursive=True,
              drive_id=os.getenv("GOOGLE_DRIVE_FOLDER_ID"),
          ),
      )
      runner.run()
  ```
</CodeGroup>

## Output a key file's contents as a string

If you need to convert the contents of a `credentials.json` file into a string, you could use a Python script such as the following.
This script takes the local path to the key file as input and outputs the key file's contents as a string.

```python Python
# Filename: json_file_to_string.py

import json
import sys

def json_file_to_string(file_path):
    try:
        # Read the JSON file.
        with open(file_path, 'r') as file:
            # Load the JSON data.
            data = json.load(file)
        
        # Convert the JSON data back to a string, with no whitespace.
        json_string = json.dumps(data, separators=(',', ':'))
        
        return json_string
    
    except FileNotFoundError:
        print(f"Error: File '{file_path}' not found.")
        return None
    except json.JSONDecodeError:
        print(f"Error: '{file_path}' is not a valid JSON file.")
        return None
    except Exception as e:
        print(f"An unexpected error occurred: {e}")
        return None

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python json_file_to_string.py <path_to_json_file>")
        sys.exit(1)
    
    file_path = sys.argv[1]
    result = json_file_to_string(file_path)
    
    if result:
        print(result)
```


# HubSpot



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect HubSpot to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The HubSpot prerequisites:

*   A [HubSpot private app and its related access token](https://developers.hubspot.com/docs/api/private-apps).
*   The HubSpot [object types](https://developers.hubspot.com/docs/api/crm/understanding-the-crm#objects) and any of their built-in properties that you want to access. Supported object types include tickets, emails, notes, calls, products, and communications.
*   Any HubSpot params and [custom properties](https://knowledge.hubspot.com/properties/create-and-edit-properties) that you want to access for any of the supported object types.

The HubSpot connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[hubspot]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `HUBSPOT_API_TOKEN` - The access token value for the HubSpot private app, represented by `--api-token` (CLI) or `api_token` (Python).
*   `HUBSPOT_OBJECT_TYPES` - A comma-separated list of strings (CLI) or an array of strings (Python) specifying one or more of the
    object types to access. This must be a subset of
    [tickets](https://developers.hubspot.com/docs/api/crm/tickets),
    [calls](https://developers.hubspot.com/docs/api/crm/calls),
    [emails](https://developers.hubspot.com/docs/api/crm/email),
    [notes](https://developers.hubspot.com/docs/api/crm/notes),
    [tasks](https://developers.hubspot.com/docs/api/crm/tasks),
    [products](https://developers.hubspot.com/docs/api/crm/products), and
    [communications](https://developers.hubspot.com/docs/api/crm/communications). These are represented by
    `--object-types` (CLI) or `object_types` (Python); if not specified, all of these object types are accessed by default.
*   `HUBSPOT_PROPERTIES` - Any filtering to be applied to specified object types, declared as a dictionary of key-value pairs (Python) specifying one or more of the default
    [properties](https://developers.hubspot.com/docs/api/crm/properties) and their desired property values, represented by
    `properties` (Python). Not available in the CLI.
*   `HUBSPOT_CUSTOM_PROPERTIES` - Any [custom properties](https://developers.hubspot.com/docs/api/crm/properties) to access.
    For the CLI, this is specified as a JSON-like string in the form `"{\"<object_type>\":[\"<custom_property_id>\", ..., \"<custom_property_id>\"]}"`.
    For Python, this is specified as a `Dict[str, List[str]]`.
    This must map to the internal name of the variable. If the property is missing,
    it will be omitted. This is represented by `--custom-properties` (CLI) or `custom_properties` (Python).
*   `HUBSPOT_PARAMS` - For Python, an array of strings specifying any values to filter the customer properties on represented by `params`. Not available in the CLI.

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  unstructured-ingest \
    hubspot \
      --api-token $HUBSPOT_API_TOKEN \
      --object-types $HUBSPOT_OBJECT_TYPES \
      --custom-properties $HUBSPOT_CUSTOM_PROPERTIES \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.runner import HubSpotRunner
  from unstructured_ingest.connector.hubspot import (
      HubSpotAccessConfig,
      SimpleHubSpotConfig
  )
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig
  )

  if __name__ == "__main__":
      runner = HubSpotRunner(
          processor_config=ProcessorConfig(
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR")
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res"
          ),
          connector_config=SimpleHubSpotConfig(
              access_config=HubSpotAccessConfig(
                  api_token=os.getenv("HUBSPOT_API_TOKEN")
              ),
              object_types=os.getenv("HUBSPOT_OBJECT_TYPES"),
              properties=os.getenv("HUBSPOT_PROPERTIES"),
              custom_properties=os.getenv("HUBSPOT_CUSTOM_PROPERTIES"),
              params=os.getenv("HUBSPOT_PARAMS")
          )
      ).run()
  ```
</CodeGroup>


# Jira



Connect Jira to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to store structured outputs locally on your filesystem.

First, install the Jira dependencies as shown here.

```bash
pip install "unstructured-ingest[jira]"
```

To processes all the issues in projects within your Jira domain, you must specify:

*   `url`:Atlassian (Jira) domain URL
*   `api-token`: API token to authenticate into Atlassian (Jira). Check [Atlassian documentation](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/) for more info.
*   `user-email`: User email for the domain

Optionally, you can limit what issues should be processed:

*   `list-of-projects`: explicitly specify which project ids should be processed
*   `list-of-boards`: explicitly specify which board ids should be processed
*   `list-of-issues`: explicitly specify which issue ids or keys should be processed

If any of the optional arguments are provided, connector will ingest only those components, and nothing else.
When none of the optional arguments are provided, all issues in all projects will be ingested.

Make sure to set the `--partition-by-api` flag and pass in your API key with `--api-key`:

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  unstructured-ingest \
    jira \
      --metadata-exclude filename,file_directory,metadata.data_source.date_processed \
      --url https://unstructured-jira-connector-test.atlassian.net \
      --user-email 12345678@unstructured.io \
      --api-token ABCDE1234ABDE1234ABCDE1234 \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --strategy hi_res
  ```

  ```python Python
  import os

  from unstructured_ingest.connector.jira import JiraAccessConfig, SimpleJiraConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import JiraRunner

  if __name__ == "__main__":
      runner = JiraRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              metadata_exclude=["filename", "file_directory", "metadata.data_source.date_processed"],
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
          ),
          connector_config=SimpleJiraConfig(
              access_config=JiraAccessConfig(api_token=os.getenv("JIRA_API_TOKEN")),
              url=os.getenv("JIRA_URL"),
              user_email=os.getenv("JIRA_EMAIL"),
          ),
      )
      runner.run()
  ```
</CodeGroup>

Additionally, if you're using Unstructured Serverless API, your locally deployed Unstructured API, or an Unstructured API
deployed on Azure or AWS, you also need to specify the API URL via the `--partition-endpoint` argument.


# Kafka



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Kafka to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Kafka prerequisites:

*   A Kafka cluster, such as ones provided by [Confluent Cloud](https://www.confluent.io/confluent-cloud), [Amazon Managed Streaming for Apache Kafka (Amazon MSK)](https://aws.amazon.com/msk/), or [Google Cloud Managed Service for Apache Kafka](https://cloud.google.com/products/managed-service-for-apache-kafka).
*   The hostname of the bootstrap Kafka cluster to connect to.
*   The port number of the cluster.
*   The name of the topic to read messages from and write messages to on the cluster.
*   If you use Kafka API keys and secrets for authentication, the key and secret values.

The Kafka connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[kafka]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `KAFKA_BOOTSTRAP_SERVER` - The hostname of the bootstrap Kafka cluster to connect to, represented by `--bootstrap-server` (CLI) or `bootstrap_server` (Python).
*   `KAFKA_PORT` - The port number of the cluster, represented by `--port` (CLI) or `port` (Python).
*   `KAFKA_TOPIC` - The unique name of the topic to read messages from and write messages to on the cluster, represented by `--topic` (CLI) or `topic` (Python).

If you use Kafka API keys and secrets for authentication:

*   `KAFKA_API_KEY` - The Kafka API key value, represented by `--kafka-api-key` (CLI) or `kafka_api_key` (Python).
*   `KAFKA_SECRET` - The secret value for the Kafka API key, represented by `--secret` (CLI) or `secret` (Python).

Additional settings include:

*   `--confluent` (CLI) or `confluent` (Python): True to indicate that the cluster is running Confluent Kafka.
*   `--num-messages-to-consume` (CLI) or `num_messages_to_consume` (Python): The maximum number of messages to get from the topic. The default is `1`.
*   `--timeout` (CLI) or `timeout` (Python): The maximum amount of time to wait for the response of a request to the topic, expressed in seconds. The default is `1.0`.

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    kafka \
      --bootstrap-server $KAFKA_BOOTSTRAP_SERVER \ 
      --port $KAFKA_PORT \
      --topic $KAFKA_TOPIC \
      --kafka-api-key $KAFKA_API_KEY \
      --secret $KAFKA_API_KEY \
      --confluent true \
      --batch-size 100 \
      --num-messages-to-consume 1 \
      --timeout 1.0 \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.kafka.cloud import (
      CloudKafkaIndexerConfig,
      CloudKafkaDownloaderConfig,
      CloudKafkaConnectionConfig,
      CloudKafkaAccessConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=CloudKafkaIndexerConfig(
              topic=os.getenv("KAFKA_TOPIC"),
              num_messages_to_consume=100,
              timeout=1
          ),
          downloader_config=CloudKafkaDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=CloudKafkaConnectionConfig(
              access_config=CloudKafkaAccessConfig(
                  kafka_api_key=os.getenv("KAFKA_API_KEY"),
                  secret=os.getenv("KAFKA_SECRET")
              ),
              bootstrap_server=os.getenv("KAFKA_BOOTSTRAP_SERVER"),
              port=os.getenv("KAFKA_PORT")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.runner import KafkaRunner
  from unstructured_ingest.connector.kafka import (
      KafkaAccessConfig,
      SimpleKafkaConfig
  )
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig
  )

  if __name__ == "__main__":
      runner = KafkaRunner(
          processor_config=ProcessorConfig(
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR")
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res"
          ),
          connector_config=SimpleKafkaConfig(
              bootstrap_server=os.getenv("KAFKA_BOOTSTRAP_SERVER"),
              port=os.getenv("KAFKA_PORT"),
              topic=os.getenv("KAFKA_TOPIC"),
              access_config=KafkaAccessConfig(
                  kafka_api_key=os.getenv("KAFKA_API_KEY"),
                  secret=os.getenv("KAFKA_SECRET"),
              ),
              confluent=False,
              num_messages_to_consume=1,
              timeout=1.0
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface"
          )
      ).run()
  ```
</CodeGroup>


# Local



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect local files to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need the local source connector dependencies:

```bash CLI, Python
pip install unstructured-ingest
```

To use the local source connector, you must set `--input-path` (CLI) or `input_path` (Python) to the path in the local filesystem which contains
documents you wish to process. Optionally, you can limit processing to certain file types by setting `--file-glob` (CLI) or `file_glob` (Python), for example to `.docx`
to process only `.docx` files.

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported.

The following 3-minute video shows how to use the Unstructured Ingest Python library to send multiple PDFs from a local directory in batches to be ingested by Unstructured API services for processing:

<iframe width="560" height="315" src="https://www.youtube.com/embed/tSKHFXsBQ-c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig,
      LocalUploaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import LocalRunner

  if __name__ == "__main__":
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
              recursive=True,
          ),
      )
      runner.run()
  ```
</CodeGroup>


# MongoDB



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect MongoDB to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The MongoDB prerequisites for a MongoDB Atlas deployment:

<iframe width="560" height="315" src="https://www.youtube.com/embed/g6qDfbg808M" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A MongoDB Atlas account. [Create an account](https://www.mongodb.com/cloud/atlas/register).
*   A MongoDB Atlas cluster. [Create a cluster](https://www.mongodb.com/docs/atlas/tutorial/deploy-free-tier-cluster).
*   The cluster must be reachable from your application environment. [Learn how](https://www.mongodb.com/docs/atlas/setup-cluster-security/#network-and-firewall-requirements).
*   The cluster must be configured to use your IP address. [Learn how](https://www.mongodb.com/docs/atlas/setup-cluster-security/#ip-access-list).
*   The cluster must have at least one database. [Create a database](https://www.mongodb.com/docs/compass/current/databases/#create-a-database).
*   The database must have at least one user, and that user must have sufficient access to the database. [Create a database user](https://www.mongodb.com/docs/atlas/security-add-mongodb-users/#add-database-users). [Give the user database access](https://www.mongodb.com/docs/manual/core/authorization/).
*   The database must have at least one collection. [Create a collection](https://www.mongodb.com/docs/compass/current/collections/#create-a-collection).
*   The connection string for the cluster. For MongoDB Atlas, this connection string must include the protocol, username, password, host, and cluster name. For example:

    ```text
    mongodb+srv://<db_user>:<db_password>@<host>/?retryWrites=true&w=majority&appName=<cluster>
    ```

    To get the connection string in MongoDB Atlas, do the following:

    1.  Log in to your MongoDB Atlas console.
    2.  In the sidebar, under **Databases**, click **Clusters**.
    3.  Click on the cluster you want to connect to.
    4.  Click **Connect**, or click the **Cmd Line Tools** tab and then click **Connect Instructions**.
    5.  Click **Drivers**.
    6.  Under **Add your connection string into your application code**, copy the connection string.
        You can then close the **Connect** dialog in MongoDB Atlas.

        Before you use this connection string, be sure to fill in any placeholders in the string, such as your MongoDB Atlas database user's password value.

    [Learn more](https://www.mongodb.com/resources/products/fundamentals/mongodb-connection-string).

The MongoDB connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[mongodb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

For a MongoDB Atlas deployment, the following environment variables:

*   `MONGODB_DATABASE` - The name of the database, represented by `--database` (CLI) or `database` (Python).

*   `MONGODB_COLLECTION` - The name of the collection in the database, represented by `--collection` (CLI) or `collection` (Python).

*   `MONGODB_URI` - The connection string for the cluster, represented by `--uri` (CLI) or `uri` (Python).

For a local MongoDB server, the following environment variables:

*   `MONGODB_HOST` - The host for the local MongoDB server, represented by `--host` (CLI) or `host` (Python).

*   `MONGODB_PORT` - The port for the local MongoDB server, represented by `--port` (CLI) or `port` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    mongodb \
      --metadata-exclude filename,file_directory,metadata.data_source.date_processed \
      --uri $MONGODB_URI \
      --database $MONGODB_DATABASE \
      --collection $MONGODB_COLLECTION \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.mongodb import (
      MongoDBAccessConfig,
      MongoDBConnectionConfig,
      MongoDBIndexerConfig,
      MongoDBDownloaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import LocalConnectionConfig
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=MongoDBIndexerConfig(batch_size=100),
          downloader_config=MongoDBDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=MongoDBConnectionConfig(
              access_config=MongoDBAccessConfig(uri=os.getenv("MONGODB_URI")),
              database=os.getenv("MONGODB_DATABASE"),
              collection=os.getenv("MONGODB_COLLECTION")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig()
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.mongodb import (
      SimpleMongoDBConfig,
  )
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import MongoDBRunner

  if __name__ == "__main__":
      runner = MongoDBRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              metadata_exclude=["filename", "file_directory", "metadata.data_source.date_processed"],
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleMongoDBConfig(
              uri=os.getenv("MONGODB_URI"),
              database=os.getenv("MONGODB_DATABASE"),
              collection=os.getenv("MONGODB_COLLECTION"),
          ),
      )
      runner.run()
  ```
</CodeGroup>


# Notion



Connect Notion to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to store
structured outputs locally on your filesystem.

First, install the Notion dependencies as shown here:

```bash
pip install "unstructured-ingest[notion]"
```

Make sure to provide `notion-api-key`. To get the credentials for your Notion workspace, follow the steps described
in [Notion documentation](https://developers.notion.com/docs/create-a-notion-integration).

Optionally, specify the following parameters:

*   `page-ids`: Notion page IDs to extract text from.
*   `database-ids`: Notion database IDs to extract text from.

Make sure to set the `--partition-by-api` flag and pass in your API key with `--api-key`:

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  unstructured-ingest \
    notion \
      --api-key "<Notion api key>" \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --page-ids "<Comma delimited list of page ids to process>" \
      --database-ids "<Comma delimited list of database ids to process>" \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key "$UNSTRUCTURED_API_KEY" \
      --strategy hi_res
  ```

  ```python Python
  import os

  from unstructured_ingest.connector.notion.connector import NotionAccessConfig, SimpleNotionConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import NotionRunner

  if __name__ == "__main__":
      runner = NotionRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
          ),
          connector_config=SimpleNotionConfig(
              access_config=NotionAccessConfig(
                  notion_api_key="POPULATE API KEY",
              ),
              page_ids=["LIST", "OF", "PAGE", "IDS"],
              database_ids=["LIST", "OF", "DATABASE", "IDS"],
              recursive=False,
          ),
      )
      runner.run()
  ```
</CodeGroup>

Additionally, if you're using Unstructured Serverless API, your locally deployed Unstructured API, or an Unstructured API
deployed on Azure or AWS, you also need to specify the API URL via the `--partition-endpoint` argument.


# OneDrive



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect OneDrive to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The OneDrive prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/7uSu0LsbcEQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A OneDrive account.
*   The User Principal Name (UPN) for the OneDrive account. This is typically the OneDrive account user's email address.
*   The path to the target OneDrive folder, starting from the OneDrive account's root folder, for example `my-folder/my-subfolder`.
*   The client ID, tenant ID, and client secret for the Microsoft Entra ID app registration that has access to the target OneDrive account and
    also has the correct set of Microsoft Graph authentication scopes. These scopes include:

    *   `Files.ReadWrite.All` (if both reading and writing are needed)
    *   `Sites.ReadWrite.All` (if both reading and writing are needed)
    *   `User.Read.All`
    *   `Directory.Read.All`

See also:

*   [Registering your app for Microsoft Graph](https://learn.microsoft.com/onedrive/developer/rest-api/getting-started/app-registration)
*   [OneDrive authentication and sign-in](https://learn.microsoft.com/onedrive/developer/rest-api/getting-started/msa-oauth)
*   [OneDrive API documentation](https://docs.microsoft.com/onedrive/developer/rest-api/)

The OneDrive connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[onedrive]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `ONEDRIVE_PATH` - The path to the target folder in the OneDrive account, starting with the account's root folder (for example `my-folder/my-subfolder`), represented by `--path` (CLI) or `path` (Python).
*   `ONEDRIVE_CLIENT_ID` - The application (client) ID of the Microsoft Entra ID app registration that has access to the OneDrive account, represented by `--client-id` (CLI) or `client_id` (Python).
*   `ONEDRIVE_CLIENT_CRED` - The client secret for the Entra ID app registration, represented by `--client-cred` (CLI) or `client_cred` (Python).
*   `ONEDRIVE_TENANT` - The directory (tenant) ID of the Entra ID app registration, represented by `--tenant` (CLI) or `tenant` (Python).
*   `ONEDRIVE_USER_PNAME` - The User Principal Name (UPN) for the OneDrive user account in Entra ID (typically the user's email address), represented by `--user-pname` (CLI) or `user_pname` (Python).
*   `ONEDRIVE_AUTHORITY_URL` - The authentication token provider URL for the Entra ID app registration (the default is `https://login.microsoftonline.com` if not otherwise specified), represented by `--authority-url` (CLI) or `authority_url` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    onedrive \
      --client-id $ONEDRIVE_CLIENT_ID \
      --client-cred $ONEDRIVE_CLIENT_CRED \
      --authority-url $ONEDRIVE_AUTHORITY_URL \
      --tenant $ONEDRIVE_TENANT \
      --user-pname $ONEDRIVE_USER_PNAME \
      --path $ONEDRIVE_PATH \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.onedrive import (
      OnedriveIndexerConfig,
      OnedriveDownloaderConfig,
      OnedriveConnectionConfig,
      OnedriveAccessConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import LocalConnectionConfig
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=OnedriveIndexerConfig(path=os.getenv("ONEDRIVE_PATH")),
          downloader_config=OnedriveDownloaderConfig(
              download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")
          ),
          source_connection_config=OnedriveConnectionConfig(
              access_config=OnedriveAccessConfig(
                  client_cred=os.getenv("ONEDRIVE_CLIENT_CRED")
              ),
              client_id=os.getenv("ONEDRIVE_CLIENT_ID"),
              tenant=os.getenv("ONEDRIVE_TENANT"),
              user_pname=os.getenv("ONEDRIVE_USER_PNAME"),
              authority_url=os.getenv("ONEDRIVE_AUTHORITY_URL")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig()
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.onedrive import OneDriveAccessConfig, SimpleOneDriveConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import OneDriveRunner

  if __name__ == "__main__":
      runner = OneDriveRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleOneDriveConfig(
              access_config=OneDriveAccessConfig(client_cred=os.getenv("ONEDRIVE_CLIENT_CRED")),
              path=os.getenv("ONEDRIVE_PATH"),
              client_id=os.getenv("ONEDRIVE_CLIENT_ID"),
              tenant=os.getenv("ONEDRIVE_TENANT"),
              authority_url=os.getenv("ONEDRIVE_AUTHORITY_URL"),
              user_pname=os.getenv("ONEDRIVE_USER_PNAME"),
              recursive=False,
          ),
      )
      runner.run()
  ```
</CodeGroup>


# OpenSearch



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect OpenSearch to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The OpenSearch prerequisites:

*   An OpenSearch instance, such as an [AWS OpenSearch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/setting-up.html) instance...

    <iframe width="560" height="315" src="https://www.youtube.com/embed/BNOYTbRbaFQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    ...or a [local](https://opensearch.org/downloads.html) instance.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/Rew3_pNnYIs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    In all cases, the [OpenSearch version](https://github.com/opensearch-project/opensearch-py/blob/main/COMPATIBILITY.md) must be compatible with the client version of the [opensearch-py](https://github.com/opensearch-project/opensearch-py/blob/main/COMPATIBILITY.md) package on PyPI.

*   The instance's hostname and port number. To learn how, see:

    *   [Creating and managing Amazon OpenSearch Service domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html) for AWS OpenSearch instances.
    *   [Communicate with OpenSearch](https://opensearch.org/docs/latest/getting-started/communicate/) for local OpenSearch instances.

*   The name of the search index on the instance. [Create an index](https://opensearch.org/docs/latest/api-reference/index-apis/create-index/).

    The OpenSearch index that you use must have a schema that is compatible with the schema of the documents
    that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following index schema example for your own needs:

    ```json
    {
        "settings": {
            "index": {
                "knn": true,
                "knn.algo_param.ef_search": 100
            }
        },
        "mappings": {
            "properties": {
                "element_id": {
                    "type": "keyword"
                },
                "text": {
                    "type": "text"
                },
                "embeddings": {
                    "type": "knn_vector",
                    "dimension": 384
                },
                "metadata": {
                    "type": "object",
                    "properties": {
                        "parent_id": {
                            "type": "text"
                        },
                        "page_number": {
                            "type": "integer"
                        },
                        "is_continuation": {
                           "type": "boolean"
                        },
                        "orig_elements": {
                           "type": "text"
                        }
                    }
                }
            }
        }
    }
    ```

    See also:

    *   [Mappings and field types](https://opensearch.org/docs/latest/field-types/)
    *   [Explicit mapping](https://opensearch.org/docs/latest/field-types/#explicit-mapping)
    *   [Dynamic mapping](https://opensearch.org/docs/latest/field-types/#dynamic-mapping)
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

*   If you're using basic authentication to the instance, the user's name and password.

*   If you're using certificates for authentication instead:

    *   The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA.
    *   The path to the combined private key and certificate file, or
    *   The paths to the separate private key and certificate file.

    To learn more, see:

    *   For AWS OpenSearch instances, see [Tutorial: Configure a domain with the internal user database and HTTP basic authentication](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac-http-auth.html) and [Creating a custom endpoint for Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/customendpoint.html).
    *   For local OpenSearch instances, see [Authentication backends](https://opensearch.org/docs/latest/security/authentication-backends/authc-index/), [HTTP basic authentication](https://opensearch.org/docs/latest/security/authentication-backends/basic-authc/), and [Client certificate authentication](https://opensearch.org/docs/latest/security/authentication-backends/client-auth/).

The OpenSearch connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[opensearch]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `OPENSEARCH_HOST` - The hostname and port number, defined as `<hostname>:<port-number>` and represented by `--hosts` (CLI) or `hosts` (Python).
*   `OPENSEARCH_INDEX_NAME` - The name of the search index, represented by `--index-name` (CLI) or `index_name` (Python).

If you're using basic authentication to the instance:

*   `OPENSEARCH_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
*   `OPENSEARCH_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

If you're using certificates for authentication instead:

*   `OPENSEARCH_CA_CERTS` - The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA. This is represented by `--ca-certs` (CLI) or `ca_certs` (Python).
*   `OPENSEARCH_CLIENT_CERT` - The path to the combined private key and certificate file, or the path to just the certificate file. This is represented by `--client-cert` (CLI) or `client_cert` (Python).
*   `OPENSEARCH_CLIENT_KEY` - The path to the private key file, if `OPENSEARCH_CLIENT_CERT` refers to just the certificate file. This is represented by `--client-key` (CLI) or `client_key` (Python).

Additional related settings include:

*   `--use-ssl` (CLI) or `use_ssl=True` (Python) to use SSL for the connection.
*   `--verify-certs` (CLI) or `verify_certs=True` (Python) to verify SSL certificates.
*   `--ssl-show-warn` (CLI) or `ssl_show_warn=True` (Python) to show a warning when verifying SSL certificates is disabled.

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    opensearch \
      --index-name $OPENSEARCH_INDEX_NAME \
      --fields "director,plot" \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --hosts $OPENSEARCH_HOST \
      --username $OPENSEARCH_USERNAME \
      --password $OPENSEARCH_PASSWORD \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.opensearch import (
      OpensearchIndexerConfig,
      OpensearchDownloaderConfig,
      OpenSearchConnectionConfig,
      OpenSearchAccessConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import LocalConnectionConfig
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=OpensearchIndexerConfig(index_name=os.getenv("OPENSEARCH_INDEX_NAME")),
          downloader_config=OpensearchDownloaderConfig(
              download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR"),
              fields=["director", "plot"]
          ),
          source_connection_config=OpenSearchConnectionConfig(
              access_config=OpenSearchAccessConfig(
                  password=os.getenv("OPENSEARCH_PASSWORD"),
                  use_ssl=True
              ),
              hosts=[os.getenv("OPENSEARCH_HOST")],
              username=os.getenv("OPENSEARCH_USERNAME")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig()
      ).run()
  ```

  ```python Python Ingest v1
  from unstructured_ingest.connector.opensearch import (
      OpenSearchAccessConfig,
      SimpleOpenSearchConfig,
  )
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import OpenSearchRunner

  if __name__ == "__main__":
      runner = OpenSearchRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              metadata_exclude=["filename", "file_directory", "metadata.data_source.date_processed"],
          ),
          connector_config=SimpleOpenSearchConfig(
              access_config=OpenSearchAccessConfig(hosts=["http://localhost:9200"]),
              index_name="movies",
              fields=["ethnicity", "director", "plot"],
          ),
      )
      runner.run()
  ```
</CodeGroup>


# Outlook



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Outlook to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Outlook prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/9yESRp9pzv0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Outlook user's email address.
*   A Microsoft Entra ID app registration in the same Azure account as the Outlook account. You will need
    this app registration's application (client) ID, client secret, and directory (tenant) ID. [Learn how](https://learn.microsoft.com/entra/identity-platform/quickstart-register-app).
*   The Entra ID app registration must have the following Graph API permission levels of the application (not delegated) type:

    *   `Mail.Read`
    *   `Mail.ReadBasic`
    *   `User.Read.All`

    [Learn how](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions).

The Outlook connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[outlook]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `OUTLOOK_USER_EMAIL` - The Outlook user's email address, represented by `--user-email` (CLI) or `user_mail` (Python).
*   `OUTLOOK_APP_CLIENT_ID` - The application (client) ID of the Microsoft Entra ID app registration that has access to the user's email account, represented by `--client-id` (CLI) or `client_id` (Python).
*   `OUTLOOK_APP_CLIENT_SECRET` - The client secret for the Entra ID app registration, represented by `--client-cred` (CLI) or `client_cred` (Python).
*   `OUTLOOK_APP_TENANT` - The directory (tenant) ID of the Entra ID app registration, represented by `--tenant` (CLI) or `tenant` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Embedding is optional.

  unstructured-ingest \
    outlook \
      --user-email $OUTLOOK_USER_EMAIL \
      --outlook-folders Inbox \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --client-id $OUTLOOK_APP_CLIENT_ID \
      --client-cred $OUTLOOK_APP_CLIENT_SECRET \
      --tenant $OUTLOOK_APP_TENANT \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --chunking-strategy by_title \
      --embedding-provider huggingface
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.outlook import (
      OutlookIndexerConfig,
      OutlookDownloaderConfig,
      OutlookConnectionConfig,
      OutlookAccessConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Embedding is optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=OutlookIndexerConfig(
              outlook_folders=["Inbox"],
              recursive=False,
              user_email=os.getenv("OUTLOOK_USER_EMAIL")
          ),
          downloader_config=OutlookDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=OutlookConnectionConfig(
              access_config=OutlookAccessConfig(client_cred=os.getenv("OUTLOOK_APP_CLIENT_SECRET")),
              client_id=os.getenv("OUTLOOK_APP_CLIENT_ID"),
              tenant=os.getenv("OUTLOOK_APP_TENANT")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.outlook import OutlookAccessConfig, SimpleOutlookConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import OutlookRunner

  if __name__ == "__main__":
      runner = OutlookRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")
              strategy="hi_res",
          ),
          connector_config=SimpleOutlookConfig(
              access_config=OutlookAccessConfig(
                  client_credential=os.getenv("OUTLOOK_APP_CLIENT_SECRET"),
              ),
              client_id=os.getenv("OUTLOOK_APP_CLIENT_ID"),
              tenant=os.getenv("OUTLOOK_APP_TENANT"),
              user_email=os.getenv("OUTLOOK_USER_EMAIL"),
              outlook_folders=["Inbox"],
              recursive=True,
          ),
      )
      runner.run()
  ```
</CodeGroup>


# Overview



Connect to your favorite data storage platforms for effortless batch processing of your files. We are constantly adding
new data connectors and if you don’t see your favorite platform let us know in our community Slack.

When ingesting data from a storage via a source connector, you typically store the resulting `*.json` files in a
specified output directory.

To "rehydrate" elements in JSON form into in-memory objects to further use in your application, use `elements_from_json`:

```python
from unstructured.staging.base import elements_from_json

elements = elements_from_json(filename=path_to_json_file)
```

<Info>All connectors can be used by the Unstructured Ingest CLI and the Unstructured Ingest Python library. The Unstructured JavaScript/TypeScript SDK supports only local sources/destinations on the same machine.</Info>


# PostgreSQL



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect PostgreSQL to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The PostgreSQL prerequisites, which include the following settings.

The following video shows for example how to get these settings by using [Amazon RDS for PostgreSQL](https://aws.amazon.com/rds/postgresql/).

<iframe width="560" height="315" src="https://www.youtube.com/embed/QuIlEimejDs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A PostgreSQL instance. [Install PostgreSQL](https://www.postgresql.org/docs/current/tutorial-install.html).

*   The host name and port number for the instance. These values are in the `postgresql.conf` file's `listen_addresses` and `port` settings. This file should be on the same machine as the instance. These values might also already be set as environment variables named `PGHOST` and `PGPORT` on the same machine as the instance.

*   A database in the instance. [Create a database](https://www.postgresql.org/docs/current/tutorial-createdb.html).

*   A table in the database. [Create a table](https://www.postgresql.org/docs/current/tutorial-table.html).

    The table's schema must match the schema of the documents that Unstructured produces.
    Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following table schema example for your own needs:

    <CodeGroup>
      ```sql PostgreSQL
      CREATE TABLE elements (
          id UUID PRIMARY KEY,
          element_id VARCHAR,
          text TEXT,
          embeddings DECIMAL [],
          parent_id VARCHAR,
          page_number INTEGER,
          is_continuation BOOLEAN,
          orig_elements TEXT
      );
      ```

      ```sql PostgreSQL with pgvector 
      CREATE EXTENSION vector;

      CREATE TABLE elements (
          id UUID PRIMARY KEY,
          element_id VARCHAR,
          text TEXT,
          embeddings vector(384),
          parent_id VARCHAR,
          page_number INTEGER,
          is_continuation BOOLEAN,
          orig_elements TEXT
      );
      ```
    </CodeGroup>

    See also:

    *   [CREATE TABLE](https://www.postgresql.org/docs/current/sql-createtable.html) for PostgreSQL
    *   [CREATE TABLE](https://github.com/pgvector/pgvector) for PostrgreSQL with pgvector
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

*   A user in the database, and a password for the user. [Create a user](https://www.postgresql.org/docs/current/sql-createuser.html).

*   Database access for the user. [Give database access to a user](https://www.postgresql.org/docs/current/sql-grant.html).

The PostgreSQL connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[postgres]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `PGHOST` - The host name, represented by `--host` (CLI) or `host` (Python).
*   `PGPORT` - The port number, represented by `--port` (CLI) or `port` (Python).
*   `PGUSER` - The username, represented by `--username` (CLI) or `username` (Python).
*   `PGPASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).
*   `PGDATABASE` - The name of the database, represented by `--database` (CLI) or `database` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    postgres \
      --host $PGHOST \
      --port $PGPORT \
      --database $PGDATABASE \
      --table-name elements \
      --id-column id \
      --username $PGUSER \
      --password $PGPASSWORD \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.sql.postgres import (
      PostgresIndexerConfig,
      PostgresDownloaderConfig,
      PostgresConnectionConfig,
      PostgresAccessConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalConnectionConfig,
      LocalUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=PostgresIndexerConfig(
              table_name="elements",
              id_column="id"
          ),
          downloader_config=PostgresDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=PostgresConnectionConfig(
              access_config=PostgresAccessConfig(password=os.getenv("PGPASSWORD")),
              host=os.getenv("PGHOST"),
              port=os.getenv("PGPORT"),
              username=os.getenv("PGUSER"),
              database=os.getenv("PGDATABASE")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig(),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>


# Reddit



Connect Reddit to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to store structured outputs locally on your filesystem.

First, install the Reddit dependencies as shown here.

```bash
pip install "unstructured-ingest[reddit]"
```

You must provide:

*   A client ID and a client secret to authenticate yourself. Learn how to get them [here](https://github.com/reddit-archive/reddit/wiki/OAuth2-Quick-Start-Example#first-steps).
*   `user-agent`: user agent request header to use when calling Reddit API
*   `subreddit-name`: The name of a subreddit, without the "r\\", e.g. "machinelearning"

Optionally you can choose to specify:

*   `search-query`: If set, return posts using this query. Otherwise, use hot posts.
*   `num-posts`: If set, limits the number of posts to pull in.

Make sure to set the `--partition-by-api` flag and pass in your API key with `--api-key`:

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  unstructured-ingest \
    reddit \
      --subreddit-name machinelearning \
      --client-id $REDDIT_CLIENT_ID \
      --client-secret $REDDIT_CLIENT_SECRET \
      --user-agent "Unstructured Ingest Subreddit fetcher by \u\..." \
      --search-query "Unstructured" \
      --num-posts 10 \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --strategy hi_res
  ```

  ```python Python
  import os

  from unstructured_ingest.connector.reddit import RedditAccessConfig, SimpleRedditConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import RedditRunner

  if __name__ == "__main__":
      runner = RedditRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
          ),
          connector_config=SimpleRedditConfig(
              access_config=RedditAccessConfig(
                  client_secret=os.getenv("REDDIT_CLIENT_SECRET"),
              ),
              subreddit_name="machinelearning",
              client_id=os.getenv("REDDIT_CLIENT_ID"),
              user_agent=r"Unstructured Ingest Subreddit fetcher by \\u\...",
              search_query="Unstructured",
              num_posts=10,
          ),
      )
      runner.run()
  ```
</CodeGroup>

Additionally, if you're using Unstructured Serverless API, your locally deployed Unstructured API, or an Unstructured API
deployed on Azure or AWS, you also need to specify the API URL via the `--partition-endpoint` argument.


# S3



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect S3 to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Amazon S3 prerequisites:

The following video shows how to fulfill the minimum set of S3 prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/hyDHfhVVAhs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but
not recommended); or generate an AWS STS session token for temporary access, if required by your organization's security
requirements. For more information about prerequisites, see the following:

*   An AWS account. [Create an AWS account](https://aws.amazon.com/free).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/lIdh92JmWtg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An S3 bucket. [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
    Additional approaches are in the following video and in the how-to sections at the end of this page.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/e6w9LwZJFIA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   Anonymous (supported but not recommended) or authenticated access to the bucket.

    *   [Enable anonymous bucket access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user).

        ![Enable anonymous bucket access](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/connectors/s3-anon-access.gif)

    *   [Enable authenticated bucket access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/walkthrough1.html).

        ![Enable authenticated bucket access](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/connectors/s3-auth-access.gif)

*   For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/y4SfQoJpipo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at
    minimum the permission of `s3:PutObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

*   For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account.
    [Create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/MoFTaGJE65Q" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For authenticated access in untrusted environments or enhanced security scenarios, an AWS STS session token for temporary access, in addition to an AWS access key and secret access key. [Create a session token](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_getsessiontoken).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/4_csSXc_GNU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   If the target files are in the root of the bucket, the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
    If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

*   If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has
    authenticated access to the folder as well. [Enable authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

The S3 connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[s3]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `AWS_S3_URL` - The path to the S3 bucket or folder, formatted as `s3://my-bucket/` (if the files are in the bucket's root) or `s3://my-bucket/my-folder/`.

*   If the bucket does not have anonymous access enabled, provide the AWS credentials:

    *   `AWS_ACCESS_KEY_ID` - The AWS access key ID for the authenticated AWS IAM user, represented by `--key` (CLI) or `key` (Python).
    *   `AWS_SECRET_ACCESS_KEY` - The corresponding AWS secret access key, represented by `--secret` (CLI) or `secret` (Python).
    *   `AWS_TOKEN` - If required, the AWS STS session token for temporary access, represented by `--token` (CLI) or `token` (Python).

*   If the bucket has anonymous access enabled for reading from the bucket, set `--anonymous` (CLI) or `anonymous=True` (Python) instead.

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported.

<iframe width="560" height="315" src="https://www.youtube.com/embed/9qSiL0QFsBA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

This example uses the local destination connector.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    s3 \
      --remote-url $AWS_S3_URL \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --key $AWS_ACCESS_KEY_ID \
      --secret $AWS_SECRET_ACCESS_KEY \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy fast \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.fsspec.s3 import (
      S3IndexerConfig,
      S3DownloaderConfig,
      S3ConnectionConfig,
      S3AccessConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=S3IndexerConfig(remote_url=os.getenv("AWS_S3_URL")),
          downloader_config=S3DownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=S3ConnectionConfig(
              access_config=S3AccessConfig(
                  key=os.getenv("AWS_ACCESS_KEY_ID"),
                  secret=os.getenv("AWS_SECRET_ACCESS_KEY")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))        
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.s3 import S3AccessConfig, SimpleS3Config
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import S3Runner

  if __name__ == "__main__":
      runner = S3Runner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleS3Config(
              access_config=S3AccessConfig(
                  key=os.getenv("AWS_ACCESS_KEY_ID"),
                  secret=os.getenv("AWS_SECRET_ACCESS_KEY")
              ),
              remote_url=os.getenv("AWS_S3_URL"),
          ),
      )
      runner.run()
  ```
</CodeGroup>

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2.  Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3.  Browse to the existing bucket and open it.

4.  Click the **Permissions** tab.

5.  In the **Bucket policy** area, click **Edit**.

6.  In the **Policy** text area, copy the following JSON-formatted policy.
    To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this policy, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-bucket-name>` in two places with the name of your bucket.

    ```json
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::<my-account-id>:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::<my-bucket-name>",
                    "arn:aws:s3:::<my-bucket-name>/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    ```

7.  Click **Save changes**.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2.  [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3.  Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
    To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this script, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-unique-bucket-name>` with the name of your bucket.
    *   Replace `<us-east-1>` with your AWS Region.

    ```bash
    #!/bin/bash

    # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
    ACCOUNT_ID="<my-account-id>"
    BUCKET_NAME="<my-unique-bucket-name>"
    REGION="<us-east-1>"

    # Temporary filename for the bucket policy.
    # Do not change this variable.
    POLICY_FILE="bucket_policy.json"

    # Create the bucket.
    aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

    # Wait for the bucket to exist.
    echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
    aws s3api wait bucket-exists --bucket $BUCKET_NAME

    # Check if the wait command was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket '$BUCKET_NAME' has been fully created."
    else
        echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
        exit 1
    fi

    # Remove the "block public policy" bucket access setting.
    aws s3api put-public-access-block \
        --bucket $BUCKET_NAME \
        --public-access-block-configuration \
        '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

    # Check if the operation was successful.
    if [ $? -eq 0 ]; then
        echo "The block public policy access setting was removed from '$BUCKET_NAME'."
    else
        echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
        exit 1
    fi

    # Create the bucket policy.
    cat << EOF > $POLICY_FILE
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::$BUCKET_NAME",
                    "arn:aws:s3:::$BUCKET_NAME/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    EOF

    # Apply the bucket policy.
    aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

    # Check if the policy application was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket policy was applied to '$BUCKET_NAME'."
    else
        echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
        exit 1
    fi

    # Verify the applied policy.
    echo "Verifying the applied policy:"
    aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

    # Remove the temporary bucket policy file.
    rm $POLICY_FILE
    ```

4.  Run the script, for example:

    ```bash
    sh create-s3-bucket.sh
    ```

5.  After the bucket is created, you can delete the script file, if you want.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
    the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    ```yaml
    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

    Parameters:
      BucketName:
        Type: String
        Description: 'Name of the S3 bucket to create'

    Resources:
      MyS3Bucket:
        Type: 'AWS::S3::Bucket'
        Properties:
          BucketName: !Ref BucketName
          PublicAccessBlockConfiguration:
            BlockPublicAcls: true
            BlockPublicPolicy: false
            IgnorePublicAcls: true
            RestrictPublicBuckets: true

      BucketPolicy:
        Type: 'AWS::S3::BucketPolicy'
        Properties:
          Bucket: !Ref MyS3Bucket
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Sid: AllowAllAuthenticatedUsersInAccount
                Effect: Allow
                Principal:
                  AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
                Action:
                  - 's3:GetObject'
                  - 's3:PutObject'
                  - 's3:ListBucket'
                  - 's3:DeleteObject'
                Resource:
                  - !Sub 'arn:aws:s3:::${BucketName}'
                  - !Sub 'arn:aws:s3:::${BucketName}/*'

    Outputs:
      BucketName:
        Description: 'Name of the created S3 bucket'
        Value: !Ref MyS3Bucket
    ```

2.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3.  Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4.  Click **Create stack > With new resources (standard)**.

5.  On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6.  Click **Choose file**, and browse to and select the YAML file from your local machine.

7.  Click **Next**.

8.  Enter a unique **Stack name** and **BucketName**.

9.  Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.


# Salesforce



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Salesforce to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Salesforce prerequisites:

*   A Salesforce account. [Create an account](https://developer.salesforce.com/signup).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/PIVPq2J1OMA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Salesforce categories (objects) that you want to access, specified as a comma-separated list. Available categories include `Account`, `Case`, `Campaign`, `EmailMessage`, and `Lead`.

*   The OAuth consumer key (client ID) and private key (PEM) for the Salesforce connected app. The PEM is a string that begins with `—–BEGIN RSA PRIVATE KEY—–` and ends with `—–END RSA PRIVATE KEY—–`. [Create a connected app](https://help.salesforce.com/s/articleView?id=sf.connected_app_create.htm\&type=5). [Get the consumer key](https://help.salesforce.com/s/articleView?id=sf.connected_app_rotate_consumer_details.htm\&type=5). [Get the private key](https://help.salesforce.com/s/articleView?id=sf.task_create_a_certificate_and_private_key.htm\&type=5).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/GcbcMsMV4oY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    <iframe width="560" height="315" src="https://www.youtube.com/embed/hFV2Ao4aXAY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The Salesforce connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[salesforce]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `SALESFORCE_USERNAME` - The Salesforce username that has access to the required Salesforce categories, represented by `--username` (CLI) or `username` (Python).
*   `SALESFORCE_CONSUMER_KEY` - The consumer key (client ID) for the Salesforce connected app, represented by `--consumer-key` (CLI) or `consumer_key` (Python).
*   `SALESFORCE_PRIVATE_KEY` - The contents of the private key (PEM) associated with the consumer key for the Salesforce connected app, represented by `--private-key` (CLI) or `private_key` (Python), or
*   `SALESFORCE_PRIVATE_KEY_PATH` - The local path to the (PEM) associated with the consumer key for the Salesforce connected app, represented by `--private-key-path` (CLI) or `private_key_path` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    salesforce \
      --username $SALESFORCE_USERNAME \
      --consumer-key $SALESFORCE_CONSUMER_KEY \
      --private-key $SALESFORCE_PRIVATE_KEY \
      --categories "EmailMessage,Account,Lead,Case,Campaign" \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --recursive \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.salesforce import (
      SalesforceIndexerConfig,
      SalesforceDownloaderConfig,
      SalesforceConnectionConfig,
      SalesforceAccessConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SalesforceIndexerConfig(categories=["EmailMessage", "Account", "Lead", "Case", "Campaign"]),
          downloader_config=SalesforceDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=SalesforceConnectionConfig(
              access_config=SalesforceAccessConfig(
                  consumer_key=os.getenv("SALESFORCE_CONSUMER_KEY"),
                  private_key=os.getenv("SALESFORCE_PRIVATE_KEY")
              ),
              username=os.getenv("SALESFORCE_USERNAME")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.salesforce import SalesforceAccessConfig, SimpleSalesforceConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import SalesforceRunner

  if __name__ == "__main__":
      runner = SalesforceRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleSalesforceConfig(
              access_config=SalesforceAccessConfig(
                  consumer_key=os.getenv("SALESFORCE_CONSUMER_KEY"),
              ),
              username=os.getenv("SALESFORCE_USERNAME"),
              private_key=os.getenv("SALESFORCE_PRIVATE_KEY"),
              categories=["EmailMessage", "Account", "Lead", "Case", "Campaign"],
              recursive=True,
          ),
      )
      runner.run()
  ```
</CodeGroup>


# SFTP



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect SFTP to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The SFTP prerequisites:

*   The SFTP server hostname, port, username, and password.

    SFTP servers are offered by several vendors. For example, the following video shows how to create and set up an SFTP server by using AWS Transfer Family:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/3_HHSnoFsoM" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The directory path to start accessing data from, specified as `sftp://<path>/<to>/<directory>`.

See the [SFTP documentation](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol).

The SFTP connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[sftp]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `SFTP_HOST` - The SFTP hostname, represented by `--host` (CLI) or `host` (Python).
*   `SFTP_PORT` - The SFTP port number, represented by `--port` (CLI) or `port` (Python).
*   `SFTP_REMOTE_URL` - The directory path to start accessing data from, represented by `--remote-url` (CLI) or `remote_url` (Python).
*   `SFTP_USERNAME` - The SFTP username, represented by `--username` (CLI) or `username` (Python).
*   `SFTP_PASSWORD` - The SFTP password, represented by `--password` (CLI) or `password` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    sftp \
      --host $SFTP_HOST \
      --port $SFTP_PORT \
      --username $SFTP_USERNAME \
      --password $SFTP_PASSWORD \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --recursive \
      --verbose \
      --strategy hi_res \
      --remote-url $SFTP_REMOTE_URL \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.sftp import (
      SftpIndexerConfig,
      SftpDownloaderConfig,
      SftpConnectionConfig,
      SftpAccessConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SftpIndexerConfig(remote_url=os.getenv("SFTP_REMOTE_URL")),
          downloader_config=SftpDownloaderConfig(
              download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR"),
              remote_url=os.getenv("SFTP_REMOTE_URL")
          ),
          source_connection_config=SftpConnectionConfig(
              access_config=SftpAccessConfig(password=os.getenv("SFTP_PASSWORD")),
              host=os.getenv("SFTP_HOST"),
              port=os.getenv("SFTP_PORT"),
              username=os.getenv("SFTP_USERNAME"),
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()

  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.sftp import SftpAccessConfig, SimpleSftpConfig
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import SftpRunner

  if __name__ == "__main__":
      runner = SftpRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
          ),
          connector_config=SimpleSftpConfig(
              access_config=SftpAccessConfig(
                  host=os.getenv("SFTP_HOST"),
                  port=os.getenv("SFTP_PORT"),
                  username=os.getenv("SFTP_USERNAME"),
                  password=os.getenv("SFTP_PASSWORD"),
              ),
              remote_url=os.getenv("SFTP_REMOTE_URL"),
              recursive=True,
          ),
      )
      runner.run()
  ```
</CodeGroup>


# SharePoint



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect SharePoint to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The SharePoint prerequisites, which include the following settings.

<iframe width="560" height="315" src="https://www.youtube.com/embed/r4BLhGgpAWQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The SharePoint site URL.

    *   Site URLs typically have the format `https://<tenant>.sharepoint.com`.
    *   Relative site URLs typically have the format `https://<tenant>.sharepoint.com/sites/<site_name>`.
    *   To process all sites within a tenant, use a site URL of `https://<tenant>-admin.sharepoint.com`. This requires the app to be registered at a tenant level.

    [Learn more](https://learn.microsoft.com/microsoft-365/community/query-string-url-tricks-sharepoint-m365).

*   The path in the SharePoint site from which to start parsing files, for example `"Shared Documents"`. If the connector is to process all sites within the tenant, this filter will be applied to all site document libraries.

*   A SharePoint app principal with its application (client) ID, client secret, and access permissions to the SharePoint instance. [Get a client ID and client secret, and set access permissions](https://github.com/vgrem/Office365-REST-Python-Client/wiki/How-to-connect-to-SharePoint-Online-and-and-SharePoint-2013-2016-2019-on-premises--with-app-principal).

    You can create SharePoint app principals through `https://<tenant>.sharepoint.com/_layouts/15/appregnew.aspx`.

    For a SharePoint app principal with site-scoped permissions, use app permission request XML such as the following to grant
    permissions through `https://<tenant>.sharepoint.com/_layouts/15/appinv.aspx`:

    ```xml
    <AppPermissionRequests AllowAppOnlyPolicy="true">
      <AppPermissionRequest Scope="http://sharepoint/content/sitecollection" Right="FullControl" />
    </AppPermissionRequests>
    ```

    For a SharePoint app principal with tenant-scoped permissions, use app permission request XML such as the following to grant
    permissions through `https://<tenant>-admin.sharepoint.com/_layouts/15/appinv.aspx` instead:

    ```xml
    <AppPermissionRequests AllowAppOnlyPolicy="true">
      <AppPermissionRequest Scope="http://sharepoint/content/tenant" Right="FullControl" />
    </AppPermissionRequests>
    ```

    Available `Right` settings include `Read`, `Write`, `Manage`, and `FullControl`. To learn more, see
    [Add-in permissions in SharePoint](https://learn.microsoft.com/sharepoint/dev/sp-add-ins/add-in-permissions-in-sharepoint).

The SharePoint connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[sharepoint]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `SHAREPOINT_APP_CLIENT_ID` - The SharePoint application (client) ID, represented by `--client-id` (CLI) or `client_id` (Python).
*   `SHAREPOINT_APP_CLIENT_SECRET` - The client secret for the SharePoint application, represented by `--client-cred` (CLI) or `client_cred` (Python).
*   `SHAREPOINT_SITE` - The SharePoint site URL, represented by `--site` (CLI) or `site` (Python).
*   `SHAREPOINT_PATH` - The path in the SharePoint site from which to start parsing files, represented by `--path` (CLI) or `path` (Python).

{/*
- `SHAREPOINT_APP_PERMISSIONS_CLIENT_ID` - The associated Azure application (client) ID, represented by `--permissions-application-id` (CLI) or `permissions_application_id` (Python).
- `SHAREPOINT_APP_PERMISSIONS_CLIENT_SECRET` - The client secret for the Azure application, represented by `--permissions-client-cred` (CLI) or `permissions_client_cred` (Python).
- `SHAREPOINT_APP_PERMISSIONS_TENANT` - The domian name of the tenant for the Azure application, which is typically `<organization-name>.onmicrosoft.com`, and which is represented by `--permissions-tenant` (CLI) or `permissions_tenant` (Python).
 */}

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    sharepoint \
      --client-id $SHAREPOINT_APP_CLIENT_ID \
      --client-cred $SHAREPOINT_APP_CLIENT_SECRET \
      --site $SHAREPOINT_SITE \
      --path $SHAREPOINT_PATH \
      --no-omit-files \
      --omit-pages \
      --omit-lists \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from pydantic import Secret

  from unstructured_ingest.v2.processes.connectors.sharepoint import (
      SharepointIndexerConfig,
      SharepointDownloaderConfig,
      SharepointConnectionConfig,
      SharepointAccessConfig,
      SharepointPermissionsConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalConnectionConfig,
      LocalUploaderConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SharepointIndexerConfig(
              path=os.getenv("SHAREPOINT_PATH"),
              recursive=True,  # Recursively process files in their respective folders. (False (default) for no recursion.)
              omit_lists=True, # Do not process lists. (False (default) to process lists.)
              omit_pages=True, # Do not process site pages. (False (default) to process site pages.)
              omit_files=False # Process files (default). (True to not process files.)
          ),
          downloader_config=SharepointDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=SharepointConnectionConfig(
              access_config=SharepointAccessConfig(client_cred=os.getenv("SHAREPOINT_APP_CLIENT_SECRET")),
              client_id=os.getenv("SHAREPOINT_APP_CLIENT_ID"),
              site=os.getenv("SHAREPOINT_SITE"),
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "reprocess": True,
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.sharepoint import (
      SharepointAccessConfig,
      SharepointPermissionsConfig,
      SimpleSharepointConfig,
  )
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import SharePointRunner

  if __name__ == "__main__":
      runner = SharePointRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
          ),
          connector_config=SimpleSharepointConfig(
              access_config=SharepointAccessConfig(
                  client_cred=os.getenv("SHAREPOINT_APP_CLIENT_SECRET"),
              ),
              client_id=os.getenv("SHAREPOINT_APP_CLIENT_ID"),
              site=os.getenv("SHAREPOINT_SITE"),
              files_only=True,
              path=os.getenv("SHAREPOINT_PATH"),
          ),
      )
      runner.run()
  ```
</CodeGroup>


# SingleStore



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect SingleStore to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The SingleStore prerequisites:

*   A SingleStore deployment, database, and table. [Learn how](https://www.singlestore.com/blog/how-to-get-started-with-singlestore/).
*   The hostname for the SingleStore deployment.
*   The port for the host.
*   The username for the deployment.
*   The password for the user.
*   The name of the database in the deployment.
*   The name of the table in the database.

To get the values for the hostname, port, username, and password:

1.  In your SingleStore account's dashboard sidebar, click **Deployments**.
2.  From the drop-down list at the top of the **Deployments** page, select your deployment.
3.  On the **Overview** tab, in the **Compute** area, in the **Connect** drop-down list for your deployment, select
    **Your App**.
4.  If a **Create User** dialog box appears, note the **User** name and **Password** values.
5.  In the **Connect to Workspace** pane's **Your App** tab, note the string in the following format:

```
<user-name>:<password>@<host>:<port>
```

*   `<user-name>` is the username.
*   `<password>` is the user's password.
*   `<host>` is the workspace's hostname.
*   `<post>` is the host's port.

To get the values for the database and table names:

1.  In your SingleStore dashboard's sidebar, click **Deployments**.
2.  From the drop-down list at the top of the **Deployments** page, select your deployment.
3.  On the **Databases** tab, note **Name** of your database.
4.  Click the database and, on the **Tables** tab, note the name of your table.

The SingleStore connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[singlestore]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `SINGLESTORE_HOST` - The hostname for the SingleStore deployment, represented by `--host` (CLI) or `host` (Python).
*   `SINGLESTORE_PORT` - The port for the host, represented by `--port` (CLI) or `port` (Python).
*   `SINGLESTORE_USER` - The username for the deployment, represented by `--user` (CLI) or `user` (Python).
*   `SINGLESTORE_PASSWORD` - The password for the user, represented by `--password` (CLI) or `password` (Python).
*   `SINGLESTORE_DB` - The name of the database in the deployment, represented by `--database` (CLI) or `database` (Python).
*   `SINGLESTORE_TABLE` - The name of the table in the database, represented by `--table-name` (CLI) or `table_name` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    singlestore \
      --host $SINGLESTORE_HOST \
      --port $SINGLESTORE_PORT \
      --user $SINGLESTORE_USER \
      --password $SINGLESTORE_PASSWORD \
      --database $SINGLESTORE_DB \
      --table-name $SINGLESTORE_TABLE \
      --id-column id \
      --batch-size 100 \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy fast \
      --chunking-strategy by_title \
      --embedding-provider huggingface
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.sql.singlestore import (
      SingleStoreIndexerConfig,
      SingleStoreDownloaderConfig,
      SingleStoreConnectionConfig,
      SingleStoreAccessConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalUploaderConfig,
      LocalConnectionConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SingleStoreIndexerConfig(
              table_name=os.getenv("SINGLESTORE_TABLE"),
              id_column="id",
              batch_size=100
          ),
          downloader_config=SingleStoreDownloaderConfig(
              download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")
          ),
          source_connection_config=SingleStoreConnectionConfig(
              access_config=SingleStoreAccessConfig(
                  password=os.getenv("SINGLESTORE_PASSWORD")
              ),
              host=os.getenv("SINGLESTORE_HOST"),
              port=os.getenv("SINGLESTORE_PORT"),
              user=os.getenv("SINGLESTORE_USER"),
              database=os.getenv("SINGLESTORE_DB")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig(),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>


# Slack



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Slack to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Slack prerequisites:

*   A Slack app. Create a Slack app by following [Step 1: Creating an app](https://api.slack.com/quickstart#creating).
*   The app must have the `channels:history` OAuth scope. Give the app this scope by following [Step 2: Requesting scopes](https://api.slack.com/quickstart#scopes).
*   The app must be installed and authorized for the target Slack workspace. Install and authorize the app by following [Step 3: Installing and authorizing the app](https://api.slack.com/quickstart#installing).
*   The app's access token. Get this token by following [Step 3: Installing and authorizing the app](https://api.slack.com/quickstart#installing).
*   Add the app to the target channels in the Slack workspace. To do this from the channel, open the channel's details page, click the **Integrations** tab, click **Add apps**, and follow the on-screen directions to install the app.
*   The channel ID for each target channel. To get this ID, open the channel's details page, and look for the **Channel ID** field on the **About** tab.
*   The starting and ending date and time range for the channels to be processed. Supported formats include:

    *   `YYYY-MM-DD`
    *   `YYYY-MM-DDTHH:MM:SS`
    *   `YYYY-MM-DDTHH:MM:SSZ`
    *   `YYYY-MM-DD+HH:MM:SS`
    *   `YYYY-MM-DD-HH:MM:SS`

The Slack connector dependencies:

```bash
pip install "unstructured-ingest[slack]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `SLACK_BOT_USER_OAUTH_TOKEN` - The OAuth token for the Slack app, represented by `--token` (CLI) or `token` (Python).

To specify the starting and ending date and time range for the channels to be processed:

*   For the CLI, use one of the following supported formats:

    *   `YYYY-MM-DD`
    *   `YYYY-MM-DDTHH:MM:SS`
    *   `YYYY-MM-DDTHH:MM:SSZ`
    *   `YYYY-MM-DD+HH:MM:SS`
    *   `YYYY-MM-DD-HH:MM:SS`

*   For Python, use the `datetime.datetime` function.

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    slack \
      --token $SLACK_BOT_USER_OAUTH_TOKEN \
      --channels C03FVNHR70A,C03FVNRG43D \
      --start-date 2024-10-22 \
      --end-date 2024-10-23 \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest v2
  import os
  from datetime import datetime

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.slack import (
      SlackIndexerConfig,
      SlackDownloaderConfig,
      SlackConnectionConfig,
      SlackAccessConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SlackIndexerConfig(
              channels=["C03FVNHR70A", "C03FVNRG43D"],
              start_date=datetime(year=2024, month=10, day=22),
              end_date=datetime(year=2024, month=10, day=23)
          ),
          downloader_config=SlackDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=SlackConnectionConfig(
              access_config=SlackAccessConfig(token=os.getenv("SLACK_BOT_USER_OAUTH_TOKEN"))
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os
  from datetime import datetime

  from unstructured_ingest.connector.slack import SimpleSlackConfig, SlackAccessConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import SlackRunner

  if __name__ == "__main__":
      runner = SlackRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          connector_config=SimpleSlackConfig(
              access_config=SlackAccessConfig(
                  token=os.getenv("SLACK_BOT_USER_OAUTH_TOKEN"),
              ),
              channels=["C03FVNHR70A", "C03FVNRG43D"],
              start_date=datetime(year=2024, month=10, day=22),
              end_date=datetime(year=2024, month=10, day=23)
          ),
      )
      runner.run()
  ```
</CodeGroup>


# Snowflake



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Snowflake to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Snowflake prerequisites:

*   A Snowflake [account](https://signup.snowflake.com/) and its [identifier](https://docs.snowflake.com/user-guide/admin-account-identifier).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/6HNmxQWqmgg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Snowflake [username and its password](https://docs.snowflake.com/user-guide/admin-user-management#creating-users) in the account.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/5hPrJNEa4aA?start=64&end=267" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Snowflake [hostname and its port number](https://docs.snowflake.com/sql-reference/functions/system_allowlist) in the account.

*   The name of the Snowflake [database](https://docs.snowflake.com/sql-reference/sql/create-database) in the account.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/KrXCgpsRd0A" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The name of the [schema](https://docs.snowflake.com/sql-reference/sql/create-schema) in the database.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/4SoS_c6rT_U?start=97&end=157" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The name of the [table](https://docs.snowflake.com/sql-reference/sql/create-table) in the schema.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/ZaZzqFMGe1g?end=169" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The Snowflake connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[snowflake]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `SNOWFLAKE_ACCOUNT` - The ID of the Snowflake account, represented by `--account` (CLI) or `account` (Python).
*   `SNOWFLAKE_USER`  - The name of the Snowflake user, represented by `--user` (CLI) or `user` (Python).
*   `SNOWFLAKE_PASSWORD`  - The user's password, represented by `--password` (CLI) or `password` (Python).
*   `SNOWFLAKE_HOST`  - The hostname for the Snowflake account, represented by `--host` (CLI) or `host` (Python).
*   `SNOWFLAKE_PORT`  - The host's port number, represented by `--port` (CLI) or `port` (Python).
*   `SNOWFLAKE_DATABASE`  - The name of the Snowflake database, represented by `--database` (CLI) or `database` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured \
    snowflake \
      --account $SNOWFLAKE_ACCOUNT \
      --user $SNOWFLAKE_USER \
      --password $SNOWFLAKE_PASSWORD \
      --host $SNOWFLAKE_HOST \
      --port $SNOWFLAKE_PORT \
      --database $SNOWFLAKE_DATABASE \
      --schema PUBLIC \
      --table-name elements\
      --id-column id \
      --batch-size 100 \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR\
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.sql.snowflake import (
      SnowflakeIndexerConfig,
      SnowflakeConnectionConfig,
      SnowflakeAccessConfig,
      SnowflakeDownloaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SnowflakeIndexerConfig(
              table_name="elements",
              id_column="id",
              batch_size=100
          ),
          downloader_config=SnowflakeDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=SnowflakeConnectionConfig(
              access_config=SnowflakeAccessConfig(password=os.getenv("SNOWFLAKE_PASSWORD")),
              account=os.getenv("SNOWFLAKE_ACCOUNT"),
              user=os.getenv("SNOWFLAKE_USER"),
              host=os.getenv("SNOWFLAKE_HOST"),
              port=os.getenv("SNOWFLAKE_PORT"),
              database=os.getenv("SNOWFLAKE_DATABASE"),
              schema="PUBLIC"
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>


# SQLite



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect SQLite to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The SQLite prerequisites:

*   A SQLite instance. [Download and install SQLite](https://www.sqlitetutorial.net/download-install-sqlite/).
*   A SQLite database. [Create a database](https://www.sqlite.org/quickstart.html).
*   The path to the database's `.db` file.
*   A table in the database. [Create a table](https://www.sqlite.org/cli.html).

    The table's schema must match the schema of the documents that Unstructured produces.
    Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following table schema example for your own needs:

    ```sql SQLite
    CREATE TABLE elements (
        id TEXT PRIMARY KEY,
        element_id TEXT,
        text TEXT,
        embeddings TEXT,
        parent_id TEXT,
        page_number INTEGER,
        is_continuation INTEGER,
        orig_elements TEXT
    );
    ```

    See also:

    *   [CREATE TABLE](https://www.sqlite.org/lang_createtable.html) for SQLite
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `SQLITE_DB_PATH` - The path to the database's `.db` file, represented by `--database` (CLI) or `database` (Python).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    sqlite \
      --database-path $SQLITE_DB_PATH \
      --table-name elements \
      --id-column id \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.sql.sqlite import (
      SQLiteIndexerConfig,
      SQLiteDownloaderConfig,
      SQLiteConnectionConfig,
      SQLiteAccessConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalConnectionConfig,
      LocalUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SQLiteIndexerConfig(
              table_name="elements",
              id_column="id"
          ),
          downloader_config=SQLiteDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=SQLiteConnectionConfig(
              access_config=SQLiteAccessConfig(),
              database_path=os.getenv("SQLITE_DB_PATH")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig(),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>


# Wikipedia



Connect Wikipedia to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to store structured outputs locally on your filesystem.

First, install the Wikipedia dependencies as shown here.

```bash
pip install "unstructured-ingest[wikipedia]"
```

Provide the `page-title` to ingest the text from.

Make sure to set the `--partition-by-api` flag and pass in your API key with `--api-key`:

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  unstructured-ingest \
    wikipedia \
      --page-title "Open Source Software" \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --strategy hi_res
  ```

  ```python Python
  import os

  from unstructured_ingest.connector.wikipedia import SimpleWikipediaConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import WikipediaRunner

  if __name__ == "__main__":
      runner = WikipediaRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
          ),
          connector_config=SimpleWikipediaConfig(
              page_title="Open Source Software",
              auto_suggest=False,
          ),
      )
      runner.run()
  ```
</CodeGroup>

Additionally, if you're using Unstructured Serverless API, your locally deployed Unstructured API, or an Unstructured API
deployed on Azure or AWS, you also need to specify the API URL via the `--partition-endpoint` argument.


# Troubleshooting Unstructured API keys and API URLs



## Issue

When you run script or code to call Unstructured API services, you get the following error:

```
UserWarning: If intending to use the paid API, please define `server_url` in your request.
```

You also get the following related warning in your logs:

```
SDKError: API error occurred: Status 401
{"detail":"API key is malformed, please type the API key correctly in the header."}
```

## Possible causes

1.  The Unstructured API key, Unstructured API URL, or both are missing or malformed in your script or code.
2.  The API key, API URL, or both are not present in your current session.
3.  The API key is no longer valid, or the API key and API URL combination is not valid.

## Suggested solutions

For the API URL, note the following:

*   For the [Unstructured Serverless API](/api-reference/api-services/saas-api-development-guide), the API URL is typically `https://api.unstructuredapp.io/general/v0/general` (be aware of the inclusion of `app` in this API URL).
*   For the [Free Unstructured API](/api-reference/api-services/free-api), the API URL is always `https://api.unstructured.io/general/v0/general` (be aware that there is no `app` in this API URL).

<Note>The free 14-day trial of the Unstructured Serverless API is different than the Free Unstructured API.</Note>

For the API key, note the following:

*   For the [Unstructured Serverless API](/api-reference/api-services/saas-api-development-guide), the API key is in your dashboard. To access your dashboard:

    1.  Sign in to your Unstructured dashboard, at [https://app.unstructured.io](https://app.unstructured.io).
    2.  On the sidebar, click **API Keys**.

*   For the [Free Unstructured API](/api-reference/api-services/free-api), the API key is in your original email from Unstructured. If you cannot find the original email, you can regenerate it by going to [https://unstructured.io/api-key-free](https://unstructured.io/api-key-free).

<Note>The free 14-day trial of the Unstructured Serverless API is different than the Free Unstructured API.</Note>

If you still believe you have the correct API URL and API key, try the following steps:

1.  Check your script or code to see where the API URL and API key are specified, as follows:

    **For a POST request to the Unstructured API**

    *   Check the target URL. Make sure it is set to the right API URL.
    *   Check the value of the `unstructured-api-key` header. Make sure it is set to the right API key, or an environment variable representing the right API key.

    **For the Unstructured Python SDK**

    *   Check the value of the `UnstructuredClient` object's `server_url` or `server` parameter:

        *   For `server_url`, make sure it is set to the right API URL, or an environment variable representing the right API URL, or...

        *   For `server`, make sure it is set to `"saas-api"` for the Unstructured Serverless API, or `"free-api"` for the Free Unstructured API, or an environment variable representing one of these strings.

            <Note>You can use `"saas-api"` and `"free-api"` only with the Unstructured SDKs.</Note>

        *   Make sure that only `serverURL` or `server` are specified—not both.

        *   Make sure that if you specify `serverURL`, an actual API URL (or an environment variable representing an actual API URL) is specified (not `"saas-api"` or `"free-api"`, or an environment variable representing one of these strings).

        *   Make sure that if you specify `server`, an actual API URL is not specified (it must be only `"saas-api"` or `"free-api"`, or an environment variable representing one of these strings).

    *   Check the value of the `UnstructuredClient` object's `api_key_auth` parameter. Make sure it is set to the right API key, or an environment variable representing the right API key.

    **For the Unstructured JavaScript/TypeScript SDK**

    *   Check the value of the `UnstructuredClient` object's `serverURL` or `server` parameter:

        *   For `serverURL`, make sure it is set to the right API URL, or an environment variable representing the right API URL, or...

        *   For `server`, make sure it is set to `"saas-api"` for the Unstructured Serverless API, or `"free-api"` for the Free Unstructured API, or an environment variable representing one of these strings.

            <Note>You can use `"saas-api"` and `"free-api"` only with the Unstructured SDKs.</Note>

        *   Make sure that only `serverURL` or `server` are specified—not both.

        *   Make sure that if you specify `serverURL`, an actual API URL (or an environment variable representing an actual API URL) is specified (not `"saas-api"` or `"free-api"`, or an environment variable representing one of these strings).

        *   Make sure that if you specify `server`, an actual API URL is not specified (it must be only `"saas-api"` or `"free-api"`, or an environment variable representing one of these strings).

    *   Check the value of the `UnstructuredClient` object's `security: { apiKeyAuth }` parameter. Make sure it is set to the right API key, or an environment variable representing the right API key.

    **For the Unstructured Ingest CLI**

    *   Check the value of the `--partition-endpoint` command option. Make sure it is set to the right API URL, or an environment variable representing the right API URL.
    *   Check the value of the `--api-key` command option. Make sure it is set to the right API key, or an environment variable representing the right API key.

    **For the Unstructured Ingest Python library**

    For a v2 connector implementation, check the value of the `partitioner_config` parameter in the `Pipeline.from_configs` method. This parameter should be set to a `PartitionerConfig` object:

    *   Check the object's `partition_endpoint` parameter. Make sure it is set to the right API URL, or an environment variable representing the right API URL.
    *   Check the object's `api_key` parameter. Make sure it is set to the right API key, or an environment variable representing the right API key.

    For a v1 connector implementation, check the value of the `partition_config` parameter in the `Runner` object. This parameter should be set to a `PartitionConfig` object:

    *   Check the object's `partition_endpoint` parameter. Make sure it is set to the right API URL, or an environment variable representing the right API URL.
    *   Check the object's `api_key` parameter. Make sure it is set to the right API key, or an environment variable representing the right API key.

    **For the Unstructured open source library**

    *   Check the value of the `partition_by_api` command's `api_url` parameter. Make sure it is set to the right API URL, or an environment variable representing the right API URL.
    *   Check the value of the `partition_by_api` command's `api_key` parameter. Make sure it is set to the right API key, or an environment variable representing the right API key.

2.  If you follow the recommended approach to use environment variables to represent the API key and API URL instead of hard-coding them into your script or code, check your session to see if these environment variables have been loaded, for example:

    *   Run the command for your operating system to list all environment variables. Make sure the API key and API URL are listed and are set to the right ones. Make sure the names and cases of these environment variables match the ones in your script or code exactly.
    *   If you use an integrated development environment (IDE) to run your script or code, check the IDE's documentation for how to specify and load environment variables at run time or debug time. Make sure the names and cases of these environment variables match the ones in your script or code exactly.

3.  Re-verify the API key and API URL that you want to use:

    **For the Unstructured Serverless API**

    *   Sign in to your Unstructured dashboard, at [https://app.unstructured.io](https://app.unstructured.io).
    *   On the sidebar, click **API Keys**.
    *   Make sure that the API key you want to use matches the one in your script, code, or environment variable. To get the full API key value, click the copy icon, and then click **Key Only**. Paste it to a temporary location for you to compare to your the one in your script, code, or environment variable. After you make the comparison, make sure to immediately delete from that temporary location what you just pasted.
    *   Make sure the API URL matches the one in your script, code, or environment variable.

    <Note>You can use `"saas-api"` with the Unstructured SDKs, instead of an API URL. For more information, see Item 1 earlier in this section.</Note>

    **For the Free Unstructured API**

    *   Open your original email from Unstructured that contains your API key. Make sure it matches the one in your script, code, or environment variable. If you cannot find the original email, you can regenerate it by going to [https://unstructured.io/api-key-free](https://unstructured.io/api-key-free).
    *   Make sure that you are using `https://api.unstructured.io/general/v0/general` (not `api.unstructuredapp.io`) for the API URL.

    <Note>You can use `"free-api"` with the Unstructured SDKs, instead of an API URL. For more information, see Item 1 earlier in this section.</Note>

4.  If you are still getting this issue, contact Support by clicking the **Support** icon in the sidebar, or by emailing [support@unstructured.io](mailto:support@unstructured.io).


# Query processed PDF with HuggingChat



This example uses the [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) or the
[Unstructured JavaScript/TypeScript SDK](/api-reference/api-services/sdk-jsts) to send a PDF file to
[Unstructured API services](/api-reference/api-services/overview) for processing. Unstructured processes the PDF and extracts the PDF's content.
This example then sends some of the content to [HuggingChat](https://huggingface.co/chat/), Hugging Face's open-source AI chatbot,
along with some queries about this content.

To run this example, you'll need:

*   The [hugchat](https://pypi.org/project/hugchat/) package for Python, or the [huggingface-chat](https://www.npmjs.com/package/huggingface-chat) package for JavaScript/TypeScript.
*   Your Unstructured API key and API URL. [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).
*   Your Hugging Face account's email address and account password. [Get an account](https://huggingface.co/join).
*   A PDF file for Unstructured to process. This example uses a sample PDF file containing the text of the United States Constitution,
    available for download from [https://constitutioncenter.org/media/files/constitution.pdf](https://constitutioncenter.org/media/files/constitution.pdf).

These environment variables:

*   `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
*   `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

These environment variables:

*   `HUGGING_FACE_EMAIL`, representing your Hugging Face account's email address.
*   `HUGGING_FACE_PASSWORD`, representing your Hugging Face account's email address.
*   `HUGGING_FACE_COOKIE_DIR_PATH`, representing a login cookie cache path, typically `./cookies/` or `./login_cache/`.

This example's code is as follows:

<AccordionGroup>
  <Accordion title="Ingest Python library">
    ```python Python
    from unstructured_ingest.v2.pipeline.pipeline import Pipeline
    from unstructured_ingest.v2.interfaces import ProcessorConfig
    from unstructured_ingest.v2.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.v2.processes.partitioner import PartitionerConfig

    import os, json
    from hugchat import hugchat
    from hugchat.login import Login
        
    def generate_json_from_local(
            input_path: str,
            output_dir: str,
            partition_by_api: bool = False,
            api_key: str = None,
            partition_endpoint: str = None,
            split_pdf_page: bool = True,
            split_pdf_allow_failed: bool = True,
            split_pdf_concurrency_level: int = 15
        ):
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=input_path),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                partition_by_api=partition_by_api,
                api_key=api_key,
                partition_endpoint=partition_endpoint,
                additional_partition_args={
                    "split_pdf_page": split_pdf_page,
                    "split_pdf_allow_failed": split_pdf_allow_failed,
                    "split_pdf_concurrency_level": split_pdf_concurrency_level
                }
            ),
            uploader_config=LocalUploaderConfig(output_dir=output_dir)
        ).run()

    def extract_matching_texts_from_local(input_json_file_path: str, text_to_match: str) -> str:
        voting_texts = ""

        with open(input_json_file_path, 'r') as file:
            file_elements = json.load(file)

        for element in file_elements:
            if text_to_match in element["text"]:
                voting_texts += " " + element["text"]

        return voting_texts

    def log_in_to_hugging_face(email: str, passwd: str, cookie_dir_path: str) -> hugchat.ChatBot:
        sign = Login(
            email=email,
            passwd=passwd
        )

        cookies = sign.login(cookie_dir_path=cookie_dir_path)

        return hugchat.ChatBot(cookies=cookies.get_dict())

    if __name__ == "__main__":
        generate_json_from_local(
            input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
            output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
            partition_by_api=True,
            api_key=os.getenv("UNSTRUCTURED_API_KEY"),
            partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")
        )

        chatbot = log_in_to_hugging_face(
            email=os.getenv("HUGGING_FACE_EMAIL"),
            passwd=os.getenv("HUGGING_FACE_PASSWORD"),
            cookie_dir_path=os.getenv("HUGGING_FACE_COOKIE_DIR_PATH")
        )

        voting_texts = extract_matching_texts_from_local(
            input_json_file_path=f"{os.getenv("LOCAL_FILE_OUTPUT_DIR")}/constitution.json",
            text_to_match="vot"
        )

        print("\n-----\n")
        print("Querying HuggingChat...")
        print("\n-----\n")

        req = f"Given the following information, what is the minimum voting age in the United States? {voting_texts}"
        print(req)
        print("\n-----\n")
        print(chatbot.chat(text=req))

        print("\n-----\n")
        print("Querying HuggingChat again...")
        print("\n-----\n")

        follow_up = "And when were women given the right to vote in the United States?"
        print(follow_up)
        print("\n-----\n")

        print(chatbot.chat(text=follow_up))
    ```
  </Accordion>

  <Accordion title="JavaScript/TypeScript SDK">
    <Warning>
      Unstructured recommends that you use the [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) instead.

      The Ingest Python library provides faster processing of larger individual files, and faster and easier processing of multiple files at a time in batches.
    </Warning>

    ```typescript TypeScript
    import { UnstructuredClient } from "unstructured-client";
    import { Strategy } from "unstructured-client/sdk/models/shared/index.js";
    import { PartitionResponse } from "unstructured-client/sdk/models/operations";

    import * as fs from "fs";
    import * as path from 'path';
    import { fileURLToPath } from 'url';
    import { Login, ChatBot} from "huggingface-chat";

    const key = process.env.UNSTRUCTURED_API_KEY || '';
    const url = process.env.UNSTRUCTURED_API_URL || '';
    const username = process.env.HUGGING_FACE_EMAIL || '';
    const userpassword = process.env.HUGGING_FACE_PASSWORD || '';
    const cookieDirPath = process.env.HUGGING_FACE_COOKIE_DIR_PATH || '';

    const inputFilepath = path.dirname(fileURLToPath(import.meta.url)) + 
                          "/local-ingest-pdf-source/constitution.pdf"

    async function getChatBot(email: string, password: string, cachePath: string): Promise<ChatBot> {
        const signin = new Login(email, password)
        const result = await signin.login(cachePath)
        const chat = new ChatBot(result)

        return chat
    }

    async function queryAndResponse(chatbot: ChatBot, query: string) {
        let data = await chatbot.chat(query)

        if (! data) {
            console.log("No data available.");
            return;
        }
        
        let reader = data.stream?.getReader();

        if (! reader) {
            console.log("No stream available.");
            return;
        }

        while (true) {
            const { done, value } = await reader?.read();
            if (done) break;
            process.stdout.write(value);
        }    
    }

    const client = new UnstructuredClient({
        security: { apiKeyAuth: key },
        serverURL: new URL(url).origin
    });

    const data = fs.readFileSync(inputFilepath);

    client.general.partition({
        partitionParameters: {
            files: {
                content: data,
                fileName: inputFilepath
            },
            strategy: Strategy.HiRes,
            splitPdfPage: true,
            splitPdfAllowFailed: true,
            splitPdfConcurrencyLevel: 15
        }
    }).then(async (res: PartitionResponse) => {
        if (res.statusCode == 200) {
            let voting_texts = ""

            if (res.elements)
                for (const element of res.elements) {
                    if (element.text.includes("vot")) {
                        voting_texts += " " + element.text;
                    }
            }

            let chat = await getChatBot(username, userpassword, cachePath)
            
            await chat.initialize()

            const firstQuery = 'Given the following information, what is the minimum voting age in the United States?'
            const secondQuery = 'And when were women given the right to vote in the United States?'

            console.log(`${firstQuery} ${voting_texts}\n\n-----\n`)
            await queryAndResponse(chat, firstQuery)
            console.log(`\n\n-----\n`)
            console.log(`${secondQuery}\n\n-----\n`)
            await queryAndResponse(chat, secondQuery)
        }    
    }).catch((e) => {
        if (e.statusCode) {
            console.log(e.statusCode);
            console.log(e.body);
        } else {
            console.log(e);
        }
    });
    ```
  </Accordion>
</AccordionGroup>


# Table extraction from PDF

This section describes two methods for extracting tables from PDF files.

<Note>
  This sample code utilizes the [Unstructured Open Source](/open-source/introduction/overview "Open Source") library and also provides an alternative method the utilizing the [Unstructured Serverless API](api-reference/api-services/overview "Serverless API").
</Note>

## Method 1: Using partition\_pdf

To extract the tables from PDF files using the [partition\_pdf](/open-source/core-functionality/partitioning#partition-pdf), set the `infer_table_structure` parameter to `True` and `strategy` parameter to `hi_res`.

**Usage**

```python
from unstructured.partition.pdf import partition_pdf

fname = "example-docs/pdf/layout-parser-paper.pdf"

elements = partition_pdf(filename=fname,
                         infer_table_structure=True,
                         strategy='hi_res',
           )

tables = [el for el in elements if el.category == "Table"]

print(tables[0].text)
print(tables[0].metadata.text_as_html)

```

## Method 2: Using Auto Partition or Unstructured API

By default, table extraction from all file types is enabled. To extract tables from PDFs and images using [Auto Partition](/open-source/core-functionality/partitioning#partition) or [Unstructured API parameters](/api-reference/api-services/api-parameters) simply set `strategy` parameter to `hi_res`.

**Usage: Auto Partition**

```python
from unstructured.partition.auto import partition

filename = "example-docs/pdf/layout-parser-paper.pdf"

elements = partition(filename=filename,
                     strategy='hi_res',
           )

tables = [el for el in elements if el.category == "Table"]

print(tables[0].text)
print(tables[0].metadata.text_as_html)

```

**Usage: API Parameters**

```bash
curl -X 'POST' \
    'https://api.unstructured.io' \
    -H 'accept: application/json' \
    -H 'Content-Type: multipart/form-data' \
    -F 'files=@sample-docs/layout-parser-paper-with-table.jpg' \
    -F 'strategy=hi_res' \
    | jq -C . | less -R

```


# Multi-file API processing



<Note>
  This sample code utilizes the [Unstructured Serverless API](api-reference/api-services/overview "Serverless API").
</Note>

## Introduction

This guide demonstrates how to process multiple files using the Unstructured API and S3 Connector and implement context-aware chunking. The process involves installing dependencies, configuring settings, and utilizing Python scripts to manage and chunk data effectively.

## Prerequisites

Ensure you have Unstructured API key and access to an S3 bucket containing the target files.

## Step-by-Step Process

### Step 1: Install Unstructured and S3 Dependency

Install the unstructured package with S3 support.

```bash
pip install "unstructured-ingest[s3]"

```

### Step 2: Import Libraries

Import necessary libraries from the unstructured package for chunking and S3 processing.

```python
from unstructured_ingest.interfaces import (
    FsspecConfig,
    PartitionConfig,
    ProcessorConfig,
    ReadConfig,
)
from unstructured_ingest.runner import S3Runner

from unstructured.chunking.title import chunk_by_title
from unstructured.staging.base import dict_to_elements

```

### Step 3: Configuration

Set up the API key and S3 URL for accessing the data.

```python
UNSTRUCTURED_API_KEY = os.getenv('UNSTRUCTURED_API_KEY')
S3_URL = "s3://rh-financial-reports/world-development-bank-2023/"

```

### Step 4: Python Runner

Configure and run the S3Runner for processing the data.

```python
runner = S3Runner(
     processor_config=ProcessorConfig(
         verbose=True,
         output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
         num_processes=8,
     ),
     read_config=ReadConfig(),
     partition_config=PartitionConfig(
         partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
         partition_by_api=True,
         api_key=os.getenv("UNSTRUCTURED_API_KEY"),
         strategy="hi_res",
         hi_res_model_name="layout_v1.1.0",
     ),
     fsspec_config=FsspecConfig(
         remote_url=S3_URL,
     ),
 )

runner.run(anonymous=True)

```

### Step 5: Combine JSON Files from Multi-files Ingestion

Combine JSON files into a single dataset for further processing.

```json
combined_json_data = read_and_combine_json("Connector-Output/world-development-bank-2023")

```

### Step 6: Convert into Unstructured Elements for Chunking

Convert the combined JSON data into Unstructured Elements and apply chunking by title.

```python
elements = dict_to_elements(combined_json_data)
chunks = chunk_by_title(elements)

```

## Conclusion

Following these steps allows for efficient processing of multiple files using the Unstructured S3 Connector. The context-aware chunking helps in organizing and analyzing the data effectively.


# Delta Table source connector



<Note>
  This sample code utilizes the [Unstructured Open Source](/open-source/introduction/overview "Open Source") Library.
</Note>

## Objectives

1.  Extract text and metadata from a PDF file using the Unstructured.io Python SDK.

2.  Process and store this data in a Databricks Delta Table.

3.  Retrieve data from the Delta Table using the Unstructured.io Delta Table Connector.

## Prerequisites

*   Unstructured Python SDK

*   Databricks account and workspace

*   AWS S3 for Delta Table storage

## Processing and Storing into Databricks Delta Table

3.  Initialize PySpark

```python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('sparkdf').getOrCreate()

```

4.  Convert JSON output into Dataframe

```python
import pyspark

dataframe = spark.createDataFrame(res.elements)

```

5.  Store DataFrame as Delta Table

```
dataframe.write.mode("overwrite").format("delta").saveAsTable("delta_table")

```

## Conclusion

This documentation covers the essential steps for converting unstructured PDF data into structured data and storing it in a Databricks Delta Table. It also outlines how to extract this data for further use.


# Vector database ingestion



<Note> This sample code utilizes the [Unstructured Open Source](/open-source/introduction/overview "Open Source") Library. </Note>

In this guide, we demonstrate how to leverage Unstructured.IO, ChromaDB, and LangChain to summarize topics from the front page of CNN Lite. Utilizing the modern LLM stack, including Unstructured, Chroma, and LangChain, this workflow is streamlined to less than two dozen lines of code.

## Gather Links with Unstructured

First, we gather links from the CNN Lite homepage using the partition\_html function from Unstructured. When Unstructured partitions HTML pages, links are included in the metadata for each element, making link collection straightforward.

```python
from unstructured.partition.html import partition_html

cnn_lite_url = "https://lite.cnn.com/"
elements = partition_html(url=cnn_lite_url)
links = []

for element in elements:
    if element.metadata.link_urls:
        relative_link = element.metadata.link_urls[0][1:]
        if relative_link.startswith("2024"):
            links.append(f"{cnn_lite_url}{relative_link}")

```

## Ingest Individual Articles with UnstructuredURLLoader

With the links in hand, we preprocess individual news articles using UnstructuredURLLoader. This loader fetches content from the web and then uses the unstructured partition function to extract content and metadata. Here we preprocess HTML files, but it also works with other response types like application/pdf. The result is a list of LangChain Document objects.

```python
from langchain.document_loaders import UnstructuredURLLoader

loaders = UnstructuredURLLoader(urls=links, show_progress_bar=True)
docs = loaders.load()

```

## Load Documents into ChromaDB

The next step is to load the preprocessed documents into ChromaDB. This process involves vectorizing the documents using OpenAI embeddings and loading them into Chroma’s vector store. Once in Chroma, similarity search can be performed to retrieve documents related to specific topics.

```python
from langchain.vectorstores.chroma import Chroma
from langchain.embeddings import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(docs, embeddings)
query_docs = vectorstore.similarity_search("Update on the coup in Niger.", k=1)

```

## Summarize the Documents

After retrieving relevant documents from Chroma, we summarize them using LangChain. The load\_summarization\_chain function allows for easy summarization, simply requiring the selection of an LLM and summarization chain.

```python
from langchain.chat_models import ChatOpenAI
from langchain.chains.summarize import load_summarize_chain

llm = ChatOpenAI(temperature=0, model_name="gpt-3.5-turbo-16k")
chain = load_summarize_chain(llm, chain_type="stuff")
chain.run(query_docs)

```

## Jupyter Notebook

To delve deeper into this example, you can access the full Jupyter Notebook here: [News of the Day Notebook](https://github.com/Unstructured-IO/unstructured/blob/main/examples/chroma-news-of-the-day/news-of-the-day.ipynb)


# Notebooks

Notebooks contain complete working sample code for end-to-end solutions.

<CardGroup cols={2}>
  <Card title="PII removal with GLiNER in unstructured data ETL" href="https://colab.research.google.com/drive/1HwOMnGjrNbcHZ1vlhaAG0MSDBcwQfexF?usp=sharing">
    <br />

    Remove Personally Identifiable Information (PII) as a part of unstructured data preprocessing.

    <br />

    `Unstructured` `Advanced` `PII` `GLiNER`

    <br />
  </Card>

  <Card title="Custom metadata extraction and self-querying retrieval" href="https://colab.research.google.com/drive/1v2SKPEmCr0p2AFyckrYvC3heYKyKff_t?usp=sharing">
    <br />

    Extract custom metadata, and enable metadata pre-filtering in your RAG.

    <br />

    `Unstructured` `MongoDB` `Advanced` `Metadata`

    <br />
  </Card>

  <Card title="Selecting an embedding model for custom data" href="https://colab.research.google.com/drive/132oXSGSOyzZ7GO9pJhRKlvwY4F-i9Pm6?usp=sharing">
    <br />

    End-to-end data processing pipeline using Unstructured Serverless API.

    <br />

    `Unstructured`

    <br />
  </Card>

  <Card title="RAG with PDFs, LangChain and Llama 3" href="https://colab.research.google.com/drive/1BJYYyrPVe0_9EGyXqeNyzmVZDrCRZwsg?usp=sharing">
    <br />

    A RAG system with the Llama 3 model from Hugging Face.

    <br />

    `Unstructured`  `🤗 Hugging Face` `LangChain` `Llama 3`
  </Card>

  <Card title="Unstructured data ETL from S3 to SingleStore DB" href="https://colab.research.google.com/drive/1Krvn5XlYNERQe7DNIXKEz3AmESJdABLF?usp=sharing">
    <br />

    Learn to ingest, partition, chunk, embed and load data from an S3 bucket into SingleStore DB.

    <br />

    `Unstructured`  `SingleStoreDB` `AWS S3`
  </Card>

  <Card title="Google Drive to DataStax Astra DB" href="https://colab.research.google.com/drive/1Img_qGCTKavImbz7dlRtxg8mNUjiWjJT?usp=sharing">
    <br />

    Embed your Google Drive Docs in an Astra Vector Database with Unstructured Serverless API

    <br />

    `Unstructured` `Google` `DataStax`

    <br />
  </Card>

  <Card title="Weaviate RAG quickstart" href="https://colab.research.google.com/drive/1lYejikVtbPraWvh9CtxBazJuXKuJ38wu?usp=sharing">
    <br />

    Embed your local documents in an Weaviate Vector Database with Unstructured Serverless API

    <br />

    `Unstructured` `OpenAI` `Weaviate`

    <br />
  </Card>

  <Card title="Preprocess PDFs in AWS S3, load into Elasticsearch" href="https://colab.research.google.com/drive/1axADo7T_dMkeOWnZ5C4dKve16-wrtQuV?usp=sharing">
    <br />

    Ingest PDF documents from an S3 bucket, transform them into a normalized JSON with Unstructured Serverless API, chunk, embed and load into Elasticsearch.

    <br />

    `Unstructured` `AWS S3` `Elasticsearch`

    <br />
  </Card>

  <Card title="Preprocess documents in Google Drive, load into Databricks Volume" href="https://colab.research.google.com/drive/1gVd03geFUD_OTROMuhjVAHYQvgVbViq7?usp=sharing">
    <br />

    Preprocess documents from a Google Drive Unstructured Serverless API and load them into Databricks Volume.

    <br />

    `Unstructured` `Google Drive` `Databricks`

    <br />
  </Card>

  <Card title="Source references in RAG responses" href="https://colab.research.google.com/drive/1Lc8eq8P87JjzUhbYb33_c7h7njsWb-hn?usp=sharing">
    <br />

    Add document source references to RAG responses based on documents metadata.

    <br />

    `Unstructured` `RAG` `LangChain`

    <br />
  </Card>

  <Card title="Query processed PDF with HuggingChat" href="https://colab.research.google.com/drive/1rNVVX5qo7vyBwR7wTa-zS6lDMkKpTei0?usp=sharing">
    <br />

    Send a PDF to Unstructured for processing, and send a subset of the returned PDF's processed text to [HuggingChat](https://huggingface.co/chat/) for chatbot-style querying.

    <br />

    `Unstructured`  `🤗 Hugging Face` `🤗 HuggingChat`
  </Card>

  <Card title="Llama 3 Local RAG with emails" href="https://colab.research.google.com/drive/1ieDJ4LoxARrHFqxXWif8Lv8e8aZTgmtH?usp=sharing">
    <br />

    Build a local RAG app for your emails with Unstructured, LangChain and Ollama.

    <br />

    `Unstructured` `LangChain` `Ollama` `Llama 3`
  </Card>

  <Card title="Building RAG With PowerPoint presentations" href="https://colab.research.google.com/drive/1l_e7CyqfBUxBBDc6E6XIKtKV-8YWvmRX?usp=sharing">
    <br />

    A RAG solution that is based on PowerPoint files.

    <br />

    `Unstructured`  `🤗 Hugging Face` `LangChain` `Llama 3`
  </Card>

  <Card title="Synthetic test dataset generation" href="https://colab.research.google.com/drive/1VvOauC46xXeZrhh8nlTyv77yvoroMQjr?usp=sharing">
    <br />

    Build a Synthetic Test Dataset for your RAG system in 5 easy steps

    <br />

    `Unstructured` `GPT-4o` `Ragas` `LangChain`

    <br />
  </Card>

  <Card title="LLama3.1 RAG evaluation on unstructured text" href="https://colab.research.google.com/drive/1VeK-z8499WV_7wY4JAkWmxpxR2LiR2-m?usp=sharing">
    <br />

    Build a Synthetic Test Dataset for your RAG system in 5 easy steps

    <br />

    `Unstructured` `GPT-4o` `Ragas` `LangChain` `Llama3.1`

    <br />
  </Card>
</CardGroup>


# Langflow



[Langflow](https://www.langflow.org/) is a visual framework for building multi-agent and RAG applications.
It is open-source, fully customizable, and works with most LLMs and many vector stores out of the box.

![Langflow designer](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/langflow/designer.png)

This no-code, hands-on demonstration walks you through creating a Langflow project that enables you to use GPT-4o-mini to chat
in real time with a PDF document that is processed by Unstructured and has its processed data stored in an
[Astra DB](https://www.datastax.com/products/datastax-astra) vector database.

## Prerequisites

The Astra DB connector prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/kupQrnCQjPw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An Astra account. [Create or sign in to an Astra account](https://astra.datastax.com/).
*   A database in the Astra account. [Create a database in an account](https://docs.datastax.com/en/astra-db-classic/databases/manage-create.html).
*   An application token for the database. [Create a database application token](https://docs.datastax.com/en/astra-db-serverless/administration/manage-application-tokens.html).
*   A namespace in the database. [Create a namespace in a database](https://docs.datastax.com/en/astra-db-serverless/databases/manage-namespaces.html#create-namespace).
*   A collection in the namespace. [Create a collection in a namespace](https://docs.datastax.com/en/astra-db-serverless/databases/manage-collections.html#create-collection).

Also:

*   [Sign up for an OpenAI account](https://platform.openai.com/signup), and [get your OpenAI API key](https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key).
*   [Sign up for a free Langflow account](https://astra.datastax.com/signup?type=langflow).
*   [Get your Unstructured Serverless API key](/api-reference/api-services/saas-api-development-guide#get-started).

## Create and run the demonstration project

<Steps>
  <Step title="Create the Langflow project">
    1.  Sign in to your Langflow dashboard.
    2.  From your dashboard, click **New Project**.
    3.  Click **Blank Flow**.
  </Step>

  <Step title="Add the Unstructured component">
    In this step, you add a component that instructs Unstructured Serverless API services to process a local file that you specify.

    1.  On the sidebar, expand **Experimental (Beta)**, and then expand **Loaders**.

    2.  Drag the **Unstructured** component onto the designer area.

    3.  In the **Unstructured** component, click the box or icon next to **File**, and then select a local file for Unstructured to process.

        This component works only with the file extensions `.pdf`, `.docx`, and `.txt`. Although you can use any local file with one of these extensions,
        this demonstration uses [the text of the United States Constitution in PDF format](https://constitutioncenter.org/media/files/constitution.pdf),
        saved to your local development machine.

        <Tip>
          To work with multiple local files, or to work with remote files, see the suggested options in [Next steps](#next-steps).
        </Tip>

    4.  For **Unstructured.io Serverless API Key**, enter your Unstructured API key value.

        ![Unstructured component](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/langflow/unstructured-component.png)

    5.  Wait until **Saved** appears in the top navigation bar.

        ![Saved message](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/langflow/saved.png)
  </Step>

  <Step title="Add the OpenAI Embeddings component">
    In this step, you add a component that generates vector embeddings for the processed data that Unstructured outputs.

    1.  On the sidebar, expand **Embeddings**, and then drag the **OpenAI Embeddings** component onto the designer area.

    2.  In the **OpenAI Embeddings** component, for **Model**, select `text-embedding-3-large`.

    3.  For **OpenAI API Key**, enter your OpenAI API key's value.

        ![OpenAI Embeddings component](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/langflow/openai-embeddings-component.png)

    4.  Wait until **Saved** appears in the top navigation bar.
  </Step>

  <Step title="Add the Astra DB components">
    In this step, you add two components. The first component instructs Astra DB to ingest into the specified Astra DB collection the processed data that Unstructured outputs along
    with the associated generated vector embeddings. The second component instructs Astra DB to take user-supplied chat messages and perform contextual
    searches over the ingested data in the specified Astra DB collection, outputting its search results.

    1.  On the sidebar, expand **Vector Stores**, and then drag the **Astra DB** component onto the designer area.

    2.  Double-click the **Astra DB** component's title bar, and rename the component to `Astra DB Ingest`.

    3.  Repeat these previous two actions to add a second **Astra DB** component, renaming it to `Astra DB RAG`.

    4.  In both of these **Astra DB** components, in the **Database** list, select the name of your Astra DB database. Make sure this is the same database name in both components.

    5.  In the **Collection** list in both components, select the name of the collection in the database. Make sure this is the same collection name in both components.

    6.  In the **Astra DB Application Token** box in both components, enter your Astra DB application token's value. Make sure this is the same application token value in both components.

    7.  Connect the **Data** output from the **Unstructured** component to the **Ingest Data** input in the **Astra DB Ingest** component.

        To make the connection, click and hold your mouse pointer inside of the circle next to **Data** in the **Unstructured** component.
        While holding your mouse pointer, drag it over into the circle next to **Ingest Data** in the **Astra DB Ingest** component. Then
        release your mouse pointer. A line appears between these two circles.

    8.  Connect the **Embeddings** output from the **OpenAI Embeddings** component to the **Embedding or Astra Vectorize** input in the  **Astra DB Ingest** component.

        ![Astra DB component](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/langflow/astra-db-component.png)

    9.  Wait until **Saved** appears in the top navigation bar.

    10. In the title bar of the **Astra DB Ingest** component, click the play icon. This ingests the processed data
        from Unstructured and the associated generated vector embeddings into the specified Astra DB collection.

        ![Play icon](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/langflow/build.png)

    11. Wait until **Building** disppears from the top navigation bar, and a green check mark appears next to this play icon. This could take several minutes.

        <Note>
          Each time you click the play icon in the **Astra DB Ingest** component, Unstructured reprocesses the specified local
          file. If this file does not change, this could result in multiple duplicate records
          being inserted into the specified Astra DB collection. You should only click the play icon in the **Astra DB Ingest** component when you want to insert new processed data into
          the specified Astra DB collection.
        </Note>
  </Step>

  <Step title="Add the Chat Input component">
    In this step, you add a component that takes user-supplied chat messages and sends them as input to Astra DB for contextual searching.

    1.  On the sidebar, expand **Inputs**, and then drag the **Chat Input** component onto the designer area.

    2.  Connect the **Message** output from the **Chat Input** component to the **Search Input** input in the **Astra DB RAG** component.

        ![Chat Input component](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/langflow/chat-input-component.png)

    3.  Wait until **Saved** appears in the top navigation bar.
  </Step>

  <Step title="Add the Parse Data component">
    In this step, you add a component that takes the Astra DB search results and converts them into plain text, suitable for inclusion in
    a prompt to a text-based LLM.

    1.  On the sidebar, expand **Helpers**, and then drag the **Parse Data** component onto the designer area.

    2.  Connect the **Search Results** output from the **Astra DB RAG** component to the **Data** input in the **Parse Data** component.

        ![Parse Data component](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/langflow/parse-data-component.png)

    3.  Wait until **Saved** appears in the top navigation bar.
  </Step>

  <Step title="Add the Prompt component">
    In this step, you add a component that builds a prompt and then sends it to a text-based LLM.

    1.  On the sidebar, expand **Prompts**, and then drag the **Prompt** component onto the designer area.

    2.  In the **Prompt** component, next to **Template**, click the box or arrow icon.

    3.  In the **Edit Prompt** window, enter the following prompt:

        ```text
        {context}

        ---

        Given the context above, answer the question as best as possible.

        Question: {question}

        Answer: 
        ```

    4.  Click **Check & Save**.

        ![Prompt component](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/langflow/edit-prompt.png)

    5.  Connect the **Text** output from the **Parse Data** component to the **context** input in the **Prompt** component.

        ![Connect Prompt component](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/langflow/connect-prompt-component.png)

    6.  Connect the **Message** output from the **Chat Input** component to the **question** input in the **Prompt** component.

        <Note>
          You will now have two connections from the **Message** output in the **Chat Input** component:

          *   One connection was already made to the **Search Input** input in the **Astra DB RAG** component.
          *   Another connection has just now been made to the **question** input in the **Prompt** component.
        </Note>

    7.  Wait until **Saved** appears in the top navigation bar.
  </Step>

  <Step title="Add the OpenAI component">
    In this step, you create a component that sends a prompt to a text-based LLM and outputs the LLM's response.

    1.  On the sidebar, expand **Models**, and then drag the **OpenAI** component onto the designer area.

    2.  In the **Model Name** list, select **gpt-4o-mini**.

    3.  For **OpenAI API Key**, enter your OpenAI API key's value.

    4.  For **Temperature**, enter `0.1`.

    5.  Connect the **Prompt Message** output from the **Prompt** component to the **Input** input in the **OpenAI** component.

        ![OpenAI component](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/langflow/openai-component.png)

    6.  Wait until **Saved** appears in the top navigation bar.
  </Step>

  <Step title="Add the Chat Output component">
    In this step, you create a component that returns the answer to the user's original chat message.

    1.  On the sidebar, expand **Outputs**, and then drag the **Chat Output** component onto the designer area.

    2.  Connect the **Text** output from the **OpenAI** component to the **Text** input in the **Chat Output** component.

        ![Chat Output component](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/langflow/chat-output-component.png)

    3.  Wait until **Saved** appears in the top navigation bar.

    The final project should look like this:

    ![Final project results](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/langflow/final-project.png)
  </Step>

  <Step title="Run the project">
    1.  In the designer area, click **Playground**.

        ![Open Playground button](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/langflow/open-playground.png)

    2.  Enter a question into the chat box, for example, `What rights does the fifth amendment guarantee?` Then press the send button.

        ![Playground window](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/langflow/playground.png)

    3.  Wait until the answer appears.

    4.  Ask as many additional questions as you want to.
  </Step>
</Steps>

## Next steps

Now that you have your pipeline set up, here are just a few ways that you could modify it to support different requirements,
such as processing multiple files or using a different vector store.

### Process multiple files

In this demonstration, you pass to Unstructured a single local file. To pass multiple local or
non-local files to Unstructured instead, you can use the
[Unstructured Platform](/platform/overview) or
[Unstructured Ingest](/ingestion/overview) outside of Langflow.

To do this, you can:

*   [Use the Unstructured Platform to create a workflow](/platform/quickstart) that relies on any available
    [source connector](/platform/sources/overview) to connect to
    [Astra DB](/platform/destinations/astradb). Run this workflow outside of Langflow anytime you have new documents in that source location that
    you want Unstructured to process and then insert the new processed data into Astra DB. Then, back in the Langflow project,
    use the **Playground** to ask additional questions, which will now include the new data when generating answers.

*   [Use Unstructured Ingest to create a pipeline](/api-reference/ingest/overview) that relies on any available
    [source connector](/api-reference/ingest/source-connectors/overview) to connect to
    [Astra DB](/api-reference/ingest/destination-connector/astradb). Run this pipeline outside of Langflow anytime you have new documents in that non-local source location that
    you want Unstructured to process and then insert the new processed data into Astra DB. Then, back in the Langflow project,
    use the **Playground** to ask additonal questions, which will now include the new data when generating answers.

### Use a different vector store

In this demonstration, you use Astra DB as the vector store. Langflow and Unstructured support several vector stores in addition to Astra DB.

To do this, you can:

[Use the Unstructured Platform to create a workflow](/platform/quickstart) that relies on any available
[source connector](/platform/sources/overview) to connect to
one of the following available vector stores that Langflow also supports:

*   [Milvus](/platform/destinations/milvus)
*   [MongoDB](/platform/destinations/mongodb)
*   [Pinecone](/platform/destinations/pinecone)

Run this workflow outside of Langflow anytime you have new documents in the source location that
you want Unstructured to process and then insert the new processed data into the vector store. Then, back in the Langflow project,
swap out the **Astra DB RAG** component for the corresponding **Vector Stores** component that matches the new vector
store's name. Configure the new component, and then
use the **Playground** to ask additional questions, which will now use the new vector store when generating answers.

Or, [use Unstructured Ingest to create a pipeline](/api-reference/ingest/overview) that relies on any available
[source connector](/api-reference/ingest/source-connectors/overview) to connect to
one of the following available vector stores that Langflow also supports:

*   [Chroma DB](/api-reference/ingest/destination-connector/chroma)
*   [Couchbase](/api-reference/ingest/destination-connector/couchbase)
*   [Elasticsearch](/api-reference/ingest/destination-connector/elasticsearch)
*   [Milvus](/api-reference/ingest/destination-connector/milvus)
*   [MongoDB](/api-reference/ingest/destination-connector/mongodb)
*   [OpenSearch](/api-reference/ingest/destination-connector/opensearch)
*   [Pinecone](/api-reference/ingest/destination-connector/pinecone)
*   [Qdrant](/api-reference/ingest/destination-connector/qdrant)
*   [Vectara](/api-reference/ingest/destination-connector/vectara)
*   [Weaviate](/api-reference/ingest/destination-connector/weaviate)

Run this pipeline outside of Langflow anytime you have new documents in the source location that
you want Unstructured to process and then insert the new processed data into the vector store. Then, back in the Langflow project,
swap out the **Astra DB RAG** component for the corresponding **Vector Stores** component that matches the new vector
store's name. Configure the new component, and then
use the **Playground** to ask additional questions, which will now use the new vector store when generating answers.

## Learn more

See the [Langflow documentation](https://docs.langflow.org/).


# Frequently asked questions



### Get in touch

If you don't find the information you're looking for in the documentation, or require assistance,
get in touch with our Support team at [support@unstructured.io](mailto:support%40unstructured.io),
or [join our Slack](https://short.unstructured.io/pzw05l7) where our team and community can help you.

## Re-launch of Serverless API

### What is the Unstructured Serverless API, and why did you relaunch it?

The Unstructured Serverless API is a powerful tool that simplifies extracting and analyzing valuable information from
unstructured data like documents and images. We've relaunched our Serverless API with a new serverless architecture, per-page
pricing, and enhanced processing capabilities to deliver faster insights, a smoother user experience, and more predictable costs.

### What are the new Serverless API features?

Here’s a quick overview of what to expect:

*   **Simplified Onboarding and User Dashboard**: Using an intuitive dashboard, you can easily manage your keys and billing options and monitor usage.
*   **New Per-Page Pricing**: Enjoy reduced costs with a transparent and predictable pricing model.
*   **Improved Processing Throughput and Latency**: Document processing ramp-up times have been slashed from thirty minutes to under three seconds, with horizontal scaling across multiple regions.
*   **Enhanced Extraction Performance**: Our new document transformation models deliver superior extraction performance for over 25 file types.

### How is my data secured when using the Serverless API?

For the Serverless API users, your data is kept entirely private. Unstructured does not persist it, use any of it for training,
or have the ability to see it. This is in contrast to the Free API, where data may be used to refine and enhance Unstructured's
service offerings. Please refer to our [Terms and Conditions](https://unstructured.io/terms-and-conditions) for more details.

### How can I sign up for the new Serverless API?

You can sign up using your email or the Google and GitHub Single Sign-On (SSO) service. Visit our Serverless API website and get your API Key.

### Can I still use the old API keys?

Yes, you can still use your old API keys. We will migrate all the user keys to the new Serverless API.

### How can I generate and use a new API Key to process my documents?

When you log in to the Serverless API dashboard, you can access your API keys by clicking the `API Keys` link in the side navigation.
Under the `Actions` column, click the `Copy` icon to copy the key or an example code snippet to process the documents
using the Unstructured REST API, or the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli), or the [Unstructured Python SDK](https://github.com/Unstructured-IO/unstructured-python-client) or [Unstructured JavaScript/TypeScript SDK](https://github.com/Unstructured-IO/unstructured-js-client).

### How can I check my balance and make the payment?

Your usage information can be found on your API dashboard. Click the `Usage` link on the side navigation. You can view your usage by date, including detailed information, such as the number of requests, total pages processed by fast and hi-res strategy, and total cost.

### How can I update my credit card information?

You can manage your payment method by going to the `Billing` page and clicking the `Manage Payment Method` button. You will be redirected to the Stripe page, where you can enter and save your credit card information.

### How can I get the images out of my source documents and into my transformed JSON?

Unstructured will return Base64 encoded images in your transformed JSON if you set the parameter `extract_image_block_types=["Table", "Image"]`.
For more information on what parameters you can set go [here](../api-reference/api-services/api-parameters).

## API Errors

### What should be done when encountering a 401 or 404 error?

Encountering a 401 or 404 error necessitates reviewing the endpoint and API key usage. Ensuring that the endpoint alignment and API key authentication are correctly configured is crucial. This may involve verifying the inclusion or exclusion of specific path elements based on the use context, such as when employing the `partition_via_api` function or via the SDKs.

### How can I address a 504 gateway timeout error during hi\_res or ocr\_only processing?

A 504 gateway timeout error typically arises during prolonged processing tasks, particularly when employing high-resolution strategies or extensive data sets. This situation suggests the operation exceeded the allotted processing window by about 30 minutes. Adjusting processing parameters or strategies to optimize performance may be beneficial. Our support team is available to assist in identifying and implementing suitable modifications to circumvent these timeouts, ensuring efficient data processing.

## API Usage

### How does the partition\_by\_api function work with the Unstructured Serverless API?

The partition\_via\_api function within the Unstructured platform provides a streamlined way for users to segment documents via the hosted Unstructured API.
The partition\_via\_api function within the Unstructured platform provides a streamlined way for users to segment documents via the hosted Unstructured API. This automation tool is particularly valuable for those managing the API on their servers or in local containerized environments, simplifying the process of breaking down documents into more manageable components.

```Python partition
import os 

from unstructured.partition.api import partition_via_api

filename = "example-docs/eml/fake-email.eml"

elements = partition_via_api(
    filename=filename,
    api_key=os.getenv("UNSTRUCTURED_API_KEY")
    api_url=os.getenv("UNSTRUCTURED_API_URL")
)
```

### How can I specify the language for OCR document processing using Unstructured Serverless API?

To define the OCR language when processing documents, utilize the languages keyword argument in your API request. For example:

```Curl ocrlang
curl -X 'POST' \
  '$UNSTRUCTURED_API_URL' \
  -H 'unstructured-api-key: $UNSTRUCTURED_API_KEY' \
  -H 'accept: application/json' \
  -H 'Content-Type: multipart/form-data' \
  -F 'files=@sample-docs/english-and-korean.png' \
  -F 'strategy=ocr_only' \
  -F 'languages=eng' \
  -F 'languages=kor' \
  | jq -C . | less -R
```

For comprehensive language support, refer to the [Tesseract documentation](https://tesseract-ocr.github.io/tessdoc/Data-Files-in-different-versions.html), which provides a detailed list of supported languages and installation guidelines.

<Note>You can still use `ocr_languages` kwarg, but this parameter is being deprecated in favor of `languages` kwarg.</Note>

### Can I employ the basic chunking strategy with the Unstructured API?

Yes, the [Unstructured API](/api-reference/api-services/overview) supports the implementation of the [Unstructured chunking strategies](/open-source/core-functionality/chunking). This approach facilitates the efficient segmentation of content, enhancing manageability and processing. Our [API Parameter](/api-reference/api-services/api-parameters) documentation provides detailed instructions on employing the chunking\_strategy and relevant kwargs, including `multipage_sections`, `combine_under_n_chars`, `new_after_n_chars`, and `max_characters`.

### How can I extract images from PDFs via the API?

You can pass the `extract_image_block_types` and `extract_image_block_to_payload` kwargs in the [API Parameters](/api-reference/api-services/api-parameters). For more information, please refer to our [API Parameter](/api-reference/api-services/api-parameters) documentation.

### What is the difference between table\_as\_cells and table\_as\_html?

`table_as_cells` was designed for internal testing only. It has been replaced with the more descriptive name `table_as_html`.

## Terms & Conditions

### Do the Terms & Conditions allow Unstructured to use the data I send through the API?

<Note>On our paid products your data is kept entirely private. We do not persist it. We do not use any of it for training. We cannot see it.</Note>

If you decide to use our Free API, you'll see Clause #9 in our [Terms & Conditions](https://unstructured.io/terms-and-conditions). It applies solely to the Free API version. It grants us the option to use data coming through the Free API only, to refine and enhance our service offerings.

On our paid products your data is kept entirely private. We do not persist it. We do not use any of it for training. We cannot see it.

<Glossary />


# Glossary



Natural Language Processing (NLP) encompasses various tasks and methodologies. This section introduces fundamental concepts crucial for most NLP projects involving Unstructured products.

## Chunking

Vector databases often require data to be in smaller, consistent chunks for efficient storage and retrieval. Chunking involves dividing lengthy texts into smaller segments or chunks, ensuring that each piece retains enough context to be meaningful.

## Data ingestion

Unstructured `Source Connectors` make data ingestion easy. They ensure that your data is accessible, up to date, and usable for any downstream task. If you’d like to read more on our source connectors, you can find details [here](/open-source/ingest/source-connectors/overview).

## Data preprocessing

Before the core analysis, raw data often requires significant preprocessing:

*   **Partitioning**: Segregating data into smaller, manageable segments or partitions.

*   **Cleaning**: Removing anomalies, filling in missing values, and eliminating irrelevant or erroneous information.

Preprocessing ensures data integrity and can significantly influence the outcomes of subsequent tasks.

## Embeddings

Embeddings convert textual data into fixed-size vectors, preserving semantic context. These vector representations can be used for many tasks, including similarity searches, clustering, and classification. Different embeddings prioritize different aspects of the text, from semantic meaning to sentence structure.

## Ingestion strategies

Unstructured offers multiple strategies and models to extract document information. They are `fast`, `hires`, `ocr_only` and `auto`. They configure the `model` to optimize document information extraction.

High-level overview of available strategies and models in `Unstructured` library:

![strategy](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/open-source/strategy.png)

## Large language models (LLMs)

LLMs, like GPT, are trained on vast amounts of data and can comprehend and generate human-like text. They have achieved state-of-the-art results across many NLP tasks and can be fine-tuned to cater to specific domains or requirements.

## Retrieval augmented generation (RAG)

Large language models (LLMs) like OpenAI’s ChatGPT and Anthropic’s Claude have revolutionized the AI landscape with their prowess. However, they inherently suffer from significant drawbacks. One major issue is their static nature, which means they’re “frozen in time.” Despite this, LLMs might often respond to newer queries with unwarranted confidence, a phenomenon known as “hallucination.” Such errors can be highly detrimental, mainly when these models serve critical real-world applications.

Retrieval augmented generation (RAG) is a groundbreaking technique designed to counteract the limitations of foundational LLMs. By pairing an LLM with an RAG pipeline, we can enable users to access the underlying data sources that the model uses. This transparent approach ensures that an LLM’s claims can be verified for accuracy and builds a trust factor among users.

Moreover, RAG offers a cost-effective solution. Instead of bearing the extensive computational and financial burdens of training custom models or fine-tuning existing ones, RAG can, in many situations, serve as a sufficient alternative. This reduction in resource consumption is particularly beneficial for organizations that need more means to develop and deploy foundational models from scratch.

A RAG workflow can be broken down into the following steps:

1.  **Data ingestion**: The first step is acquiring data from your relevant sources. We make this easy with our [source connectors](/open-source/ingest/source-connectors/overview).

2.  **Data preprocessing and cleaning**: Once you’ve identified and collected your data sources, removing any unnecessary artifacts within the dataset is a good practice. At Unstructured, we have various tools for data processing in our [core functionalities](/open-source/core-functionality/overview).

3.  **Chunking**: The next step is to break your text into digestible pieces for your LLM to consume. We provide the basic and context-aware chunking strategies. Please refer to the documentation [here](/open-source/core-functionality/chunking).

4.  **Embedding**: After chunking, you must convert the text into a numerical representation (vector embedding) that an LLM can understand. To use the various embedding models using Unstructured tools, please refer to [this page](/open-source/core-functionality/embedding).

5.  **Vector Database**: The next step is to choose a location for storing your chunked embeddings. There are many options for your vector database (Astra DB, ChromaDB, Clarifai, Pinecone, Qdrant, Weaviate, and more). For complete list of Unstructured `Destination Connectors`, please visit [this page](/open-source/ingest/destination-connectors/overview).

6.  **User Prompt**: Take the user prompt and grab the most relevant chunks of information in the vector database via similarity search.

7.  **LLM Generation**: Once you’ve retrieved your relevant chunks, you pass the prompt + the context to the LLM for the LLM to generate a more accurate response.

For a complete guide on how to implement RAG, check out this [blog post](https://medium.com/unstructured-io/effortless-document-extraction-a-guide-to-using-unstructured-api-and-data-connectors-6c2659eda4af)

## Tokens

Tokenization decomposes texts into smaller units called tokens. A token might represent a word, part of a word, or even a single character. This process helps analyze and process the text, making it digestible for models and algorithms.

## Vector databases

Vector databases store embeddings in a manner optimized for high-speed similarity searches. Given a query embedding, these databases can quickly retrieve the most similar vectors, facilitating tasks like recommendation, anomaly detection, and clustering.

These foundational concepts provide the groundwork for more advanced NLP methodologies and pipelines. Proper understanding and implementation can vastly improve the outcomes of NLP projects.


# Overview



## Concepts

*Ingestion* is the term that Unstructured uses to refer to the set of activities that happens when files are input for processing. Ingestion enables multiple files to be processed as a batch.

You can perform ingestion with the following tools:

*   The [Unstructured Platform](/platform/overview), a no-code user interface, unlimited pay-as-you-go platform to get all of your data ready for Retrieval Augmented Generation (RAG) and model fine-tuning.
*   The [Unstructured Ingest CLI](#unstructured-ingest-cli), with unlimited pay-as-you-go and limited free options, that enable you to use command-line scripts to get all of your data ready for RAG and model fine-tuning.
*   The [Unstructured Ingest Python library](#unstructured-ingest-python-library), with unlimited pay-as-you-go and limited free options, that enable you to use Python code to get all of your data ready for RAG and model fine-tuning.

<Info>
  The [Unstructured Python SDK](/api-reference/api-services/sdk-python) and [Unstructured JavaScript/TypeScript SDK](/api-reference/api-services/sdk-jsts) can process only one file at a time.
</Info>

Files are ingested from an originating *source* location. Each batch of ingested files is processed either [all by Unstructured](#ingestion-options-for-the-unstructured-service) or [all locally](#local-ingestion-options). The processed data is sent to a target *destination* location. The kinds of locations you can specify varies:

When you use the [Unstructured Platform](/platform/overview), the source and destination must both be in cloud storage. Local source or local destination locations are not allowed. For example:

```mermaid
flowchart LR
   source[Google Drive]-->api[Processing...]-->destination[S3]
```

```mermaid
flowchart LR
    source[Azure]-->api[Processing...]-->destination[Pinecone]
```

The Unstructured Platform enables you to connect to many kinds of [sources](/platform/sources/overview) and [destinations](/platform/destinations/overview).

If you use the Unstructured Ingest CLI or the Unstructured Ingest Python library, the source or destination can be a cloud storage location or a local location. For example:

```mermaid
flowchart LR
    source[Local directory]-->api[Processing...]-->destination[Azure]
```

```mermaid
flowchart LR
   source[Google Drive]-->api[Processing...]-->destination[Local directory]
```

```mermaid
flowchart LR
    source[S3]-->api[Processing...]-->destination[Pincone]
```

```mermaid
flowchart LR
    source[Local directory]-->api[Processing...]-->destination[A different local directory]
```

Unstructured provides many [source](/api-reference/ingest/source-connectors/overview) and [destination](/api-reference/ingest/destination-connector/overview) connectors.

## Ingestion options for the Unstructured service

This is the flow for sending files to Unstructured for processing and the processed data being delivered by Unstructured:

```mermaid
flowchart LR
    source[Source]-->api[Processed by Unstructured]-->destination[Destination]
```

*   This flow always happens for the Unstructured Platform. The Platform only allows sending files from cloud storage and sending processed data to cloud storage.
*   For the Unstructured Ingest CLI or the Unstructured Ingest Python library, to use this flow:

    *   When using the Unstructured Ingest CLI, include the `--partition-by-api` option and set `--api-key` and `--partition-endpoint` to a valid, matching Unstructured API key and API URL, respectively.
    *   When using the Unstructured Ingest Python library, set `partition_by_api=True` and `api_key` and set `partition_endpoint` to a valid, matching Unstructured API key and API URL, respectively.

## Local ingestion options

This is the flow for processing files locally. No files are sent to Unstructured for processing:

```mermaid
flowchart LR
    source[Source]-->api[Processed locally]-->destination[Destination]
```

*   This flow never happens for the Unstructured Platform. The Platform does not allow sending files from a local destination to Unstructured or Unstructured sending processed data to a local destination.
*   For the Unstructured Ingest CLI or the Unstructured Ingest Python library, to use this flow:

    *   When using the Unstructured Ingest CLI, omit the `--partition-by-api`, `--api-key`, and `--partition-endpoint` options.
    *   When using the Unstructured Ingest Python library, omit `partition_by_api` or explicitly set `partition_by_api=False`. Also omit `api_key` and `partition_endpoint`.

## Unstructured Ingest CLI

The Unstructured Ingest CLI enables you to use command-line scripts to get all of your data ready for RAG and model fine-tuning.

One approach to using the CLI is installing Python and then running the following command to install the CLI:

```bash
pip install unstructured-ingest
```

This default installation option enables the ingestion of plain text files, HTML, XML, JSON and emails that do not require any extra dependencies. This default option also enables you to specify local source and destination locations.

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

For additional installation options, see:

*   [Run the library in a container](https://github.com/Unstructured-IO/unstructured/tree/main?tab=readme-ov-file#run-the-library-in-a-container)
*   [Installing the library](https://github.com/Unstructured-IO/unstructured/tree/main?tab=readme-ov-file#installing-the-library)
*   The installation commands for additional connectors for [sources](/api-reference/ingest/source-connectors/overview) and [destinations](/api-reference/ingest/destination-connector/overview)

To display the list of available source connector commands, run the following command:

```bash
unstructured-ingest --help
```

To display the list of available destination connector commands, run the following command:

```bash
unstructured-ingest local --help
```

To display help for a specific source connector command, run the following command:

```bash
unstructured-ingest <command-name> --help
```

To display help for a specific destination connector command, run the following command:

```bash
unstructured-ingest local <command-name> --help
```

To begin using the CLI, see the quickstarts for the:

*   [Unstructured Serverless API](/api-reference/api-services/saas-api-development-guide#unstructured-cli)
*   [Free Unstructured API](/api-reference/api-services/free-api#unstructured-cli)

<Info>To migrate from older, deprecated versions of the Ingest CLI that used `pip install unstructured`, see the [migration guide](#migration-guide).</Info>

## Unstructured Ingest Python library

The Unstructured Ingest Python library enable you to use Python code to get all of your data ready for RAG and model fine-tuning.

The following 3-minute video shows how to use the Unstructured Ingest Python library to send multiple PDFs from a local directory in batches to be ingested by Unstructured API services for processing:

<iframe width="560" height="315" src="https://www.youtube.com/embed/tSKHFXsBQ-c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

One approach to using the Unstructured Ingest Python library is installing Python and then running the following command to install the library and the default connectors:

```bash
pip install unstructured-ingest
```

This default installation option enables the ingestion of plain text files, HTML, XML, JSON and emails that do not require any extra dependencies. This default option also enables you to specify local source and destination locations.

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

For additional installation options, see:

*   [Run the library in a container](https://github.com/Unstructured-IO/unstructured/tree/main?tab=readme-ov-file#run-the-library-in-a-container)
*   [Installing the library](https://github.com/Unstructured-IO/unstructured/tree/main?tab=readme-ov-file#installing-the-library)
*   The installation commands for additional connectors for [sources](/api-reference/ingest/source-connectors/overview) and [destinations](/api-reference/ingest/destination-connector/overview)

Some source and destination connectors provide newer v2 and older v1 implementations, while some provide only older v1 implementations. You should use the v2 implementations wherever they are available, to help ensure better forward-compatibility of your code. For the lists of available v2 and v1 connectors, see:

*   [v2 non-fsspec connectors](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/unstructured_ingest/v2/processes/connectors)
*   [v2 fsspec connectors](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/unstructured_ingest/v2/processes/connectors/fsspec)
*   [v1 non-fsspec connectors](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/unstructured_ingest/connector)
*   [v1 fsspec connectors](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/unstructured_ingest/connector/fsspec)
*   [v1 Notion connector](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/unstructured_ingest/connector/notion)

To begin using the Unstructured Ingest Python library, see the code examples for the [source](/api-reference/ingest/source-connectors/overview) and [destination](/api-reference/ingest/destination-connector/overview) connectors.

<Info>To migrate from older, deprecated versions of the Ingest Python library that used `pip install unstructured`, see the [migration guide](#migration-guide).</Info>

### Generate Python code examples

You can connect any available source connector to any available destination connector. However, the source connector code examples in the
documentation show connecting only to the local destination connector. Similarly, the destination connector code examples in the
documentation show connecting only to the local source connector.

To quickly generate an Unstructured Ingest Python library code example that connects *any* available source connector to *any* available destination connector,
do the following:

1.  Open the [Unstructured Ingest Code Generator](https://huggingface.co/spaces/MariaK/unstructured-pipeline-builder) webpage.

2.  Select your input (source) location type from the **Get unstructured documents from** drop-down list.

3.  Select your output (destination) location type from the **Upload RAG-ready documents to** drop-down list.

4.  Select your chunking strategy from the **Chunking strategy** drop-down list:

    *   **None** - Do not chunk the data elements' content.
    *   **basic** - Combine sequential data elements to maximally fill each chunk. However, do not mix `Table` and non-`Table` elements in the same chunk.
    *   **by\_title** - Use the `basic` strategy and also preserve section boundaries. Optionally preserve page boundaries as well.
    *   **by\_page** - Use the `basic` strategy and also preserve page boundaries.
    *   **by\_similarity** - Use the `sentence-transformers/multi-qa-mpnet-base-dot-v1` embedding model to identify topically similar sequential elements and combine them into chunks. This strategy is availably only when calling Unstructured API services.

    To learn more, see [Chunking strategies](/api-reference/api-services/chunking) and [Chunking configuration](/api-reference/ingest/ingest-configuration/chunking-configuration).

5.  For any chunking strategy other than **None**:

    *   Enter your chunk size in the **Chunk size (characters)** box, or leave the default of **1000** characters.
    *   If you need to apply overlapping to the chunks, enter the chunk overlap size in the **Chunk overlap (characters)** box, or leave default of **20** characters.

    To learn more, see [Chunking configuration](/api-reference/ingest/ingest-configuration/chunking-configuration).

6.  To generate vector embeddings, select the provider in the **Embedding provider** drop-down list.

    To learn more, see [Embedding configuraton](/api-reference/ingest/ingest-configuration/embedding-configuration).

7.  Click **Generate code**.

8.  Copy the example code from the **Generated Code** pane into your code project.

9.  The code example will contain one or more environment variables that you must set for the code to run correctly. To learn what to
    set these variables to, click the documentation links that are below the **Generated Code** pane.

## Migration guide

The older `unstructured` versions of the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) and [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) have been replaced and are now deprecated.

To migrate to the newer [unstructured-ingest](https://pypi.org/project/unstructured-ingest/) versions of the Ingest CLI and Ingest Python library, do the following:

1.  If you previously ran `pip install unstructured` **only** for the purposes of using the Ingest CLI or the Ingest Python library, upgrade to the latest versions by running the following commands:

    a. `pip uninstall unstructured`<br />
    b. `pip install unstructured-ingest`

2.  If you previously installed an older version of a source or destination connector, for example `pip install "unstructured[azure]"` for the Azure Storage connector, upgrade to the latest version by running the following commands:

    a. `pip uninstall "unstructured[azure]"`<br />
    b. `pip install "unstructured-ingest[azure]"`

3.  If you were running Python code against an older version of the Ingest Python library, update your `import` statements by replacing all instances of `unstructured.ingest` with `unstructured_ingest` to run against the latest version.


# Chunking





# Embedding





# Document elements and metadata



`unstructured` simplifies and streamline the preprocessing of structured and unstructured documents for downstream tasks.
What that means is no matter where your data is and no matter what format that data is in, Unstructured's toolkit will
transform and preprocess that data into an easily digestible and usable format that is uniform across data formats.

When you partition a document with Unstructured, the result is a list of document `Element` objects.
These element objects represent different components of the source document.

## Element example

Here's an example of what an element might look like:

```
{'type': 'NarrativeText',
 'element_id': '5ef1d1117721f0472c1ad825991d7d37',
 'text': 'The Unstructured API documentation covers the following API services:',
 'metadata': {'last_modified': '2024-05-01T14:15:22',
  'page_number': 1,
  'languages': ['eng'],
  'parent_id': '56f24319ae258b735cac3ec2a271b1d9',
  'file_directory': '/content',
  'filename': 'Unstructured API services - Unstructured.html',
  'filetype': 'text/html'}}
```

As you can see, every element will have a `type`, an `element_id`, the extracted `text`, and some `metadata` which may
vary depending on the element type, document structure, and some additional parameters used during partitioning and/or chunking.

Let's explore some of these document element components in more detail.

## Element type

Instead of treating all documents like a wall of plain text, Unstructured preserves the semantic structure of the documents.
This gives you more control and flexibility over how you further use the processed documents and allows you to take their
structure into consideration. At the same time, normalizing data from various file formats to the Unstructured element
type scheme lets you treat all documents the same in your downstream processing, regardless of source format.
For example, if you plan to summarize a document, you may only be interested in the narrative
of the document, and not care about footers and headers. You can easily filter out the elements you don't need using their type.

Here are some examples of the element types your document may contain:

| Element type        | Description                                                                                                                                        |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Formula`           | An element containing formulas in a document.                                                                                                      |
| `FigureCaption`     | An element for capturing text associated with figure captions.                                                                                     |
| `NarrativeText`     | NarrativeText is an element consisting of multiple, well-formulated sentences. This excludes elements such titles, headers, footers, and captions. |
| `ListItem`          | ListItem is a NarrativeText element that is part of a list.                                                                                        |
| `Title`             | A text element for capturing titles.                                                                                                               |
| `Address`           | A text element for capturing physical addresses.                                                                                                   |
| `EmailAddress`      | A text element for capturing email addresses.                                                                                                      |
| `Image`             | A text element for capturing image metadata.                                                                                                       |
| `PageBreak`         | An element for capturing page breaks.                                                                                                              |
| `Table`             | An element for capturing tables.                                                                                                                   |
| `Header`            | An element for capturing document headers.                                                                                                         |
| `Footer`            | An element for capturing document footers.                                                                                                         |
| `CodeSnippet`       | An element for capturing code snippets.                                                                                                            |
| `PageNumber`        | An element for capturing page numbers.                                                                                                             |
| `UncategorizedText` | Base element for capturing free text from within document.                                                                                         |

If you apply chunking during partitioning of a document or later, you will also see the `CompositeElement` type.
`CompositeElement` is a chunk formed from text (non-Table) elements. It is only produced by chunking.
A composite element may be formed by combining one or more sequential elements produced by partitioning. For example,
several individual list items may be combined into a single chunk.

## Element ID

By default, the element ID is a SHA-256 hash of the element's text, its position on the page, page number it's on,
and the name of the document file - this is to ensure that the ID is deterministic and unique at the document level.
To obtain globally unique IDs in the output (UUIDs), you can pass `unique_element_ids=True` into any of the partition
functions. This can be helpful if you'd like to use the IDs as a primary key in a database, for example.

## Metadata

Unstructured tracks a variety of metadata about the elements extracted from documents.
Here are a couple of examples of what element metadata enables you to do:

*   filter document elements based on an element metadata value. For instance, you may want to limit your scope to elements from a certain page, or you may want to use only elements that have an email matching a regular expression in their metadata.
*   map an element to the document page where it occurred so that original page can be retrieved when that element matches search criteria.

Metadata is tracked at the element level. You can access the metadata for a given document element with
`element.metadata`. For a dictionary representation, use `element.metadata.to_dict()`.

### Common metadata fields

All document types return the following metadata fields when the information is available from the source file:

| Metadata field name        | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `filename`                 | Filename                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `file_directory`           | File directory                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `last_modified`            | Last modified Date                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `filetype`                 | File type                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `coordinates`              | XY Bounding Box Coordinates. See notes below for further details about the bounding box.                                                                                                                                                                                                                                                                                                                                                                                           |
| `parent_id`                | Element Hierarchy. `parent_id` may be used to infer where an element resides within the overall hierarchy of a document. For instance, a NarrativeText element may have a Title element as a parent (a “sub-title”), which in turn may have another Title element as its parent (a "title").                                                                                                                                                                                       |
| `category_depth`           | Element depth relative to other elements of the same category.  Category depth is the depth of an element relative to other elements of the same category. It’s set by a document partitioner and enables the hierarchy post-processor to compute more accurate hierarchies. Category depth may be set using native document hierarchies, e.g. reflecting \<H1>, \<H2>, or \<H3> tags within an HTML document or the indentation level of a bulleted list item in a Word document. |
| `text_as_html`             | HTML representation of extracted tables. Only applicable to table elements.                                                                                                                                                                                                                                                                                                                                                                                                        |
| `languages`                | Document Languages.  At document level or element level. List is ordered by probability of being the primary language of the text.                                                                                                                                                                                                                                                                                                                                                 |
| `emphasized_text_contents` | Emphasized text (bold or italic) in the original document.                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `emphasized_text_tags`     | Tags on text that is emphasized in the original document.                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `is_continuation`          | True if element is a continuation of a previous element.  Only relevant for chunking, if an element was divided into two due to max\_characters.                                                                                                                                                                                                                                                                                                                                   |
| `detection_class_prob`     | Detection model class probabilities.  From unstructured-inference, hi-res strategy.                                                                                                                                                                                                                                                                                                                                                                                                |

Notes on common metadata fields:

#### Metadata for document hierarchy

`parent_id` and `category_depth` enhance hierarchy detection to identify the document
structure in various file formats by measuring relative depth of an element within its category. This is especially
useful in documents with native hierarchies like HTML or Word files, where elements like headings or list items inherently define structure.

#### Element's coordinates

Some document types support location data for the elements, usually in the form of bounding boxes.
If it exists, an element's location data is available with `element.metadata.coordinates`. The `coordinates` property of an `ElementMetadata` stores:

*   `points` : These specify the corners of the bounding box starting from the top left corner and proceeding counter-clockwise. The points represent pixels, the origin is in the top left and the `y` coordinate increases in the downward direction.
*   `system`: The points have an associated coordinate system. A typical example of a coordinate system is `PixelSpace`, which is used for representing the coordinates of images. The coordinate system has a name, orientation, layout width, and layout height.

The Unstructured Open Source library offers a way to change the coordinates of an element to a new coordinate system by
using the `Element.convert_coordinates_to_new_system` method. If the `in_place` flag is `True`, the coordinate system
and points of the element are updated in place and the new coordinates are returned. If the `in_place` flag is `False`,
only the altered coordinates are returned.

```python
from unstructured.documents.elements import Element
from unstructured.documents.coordinates import PixelSpace, RelativeCoordinateSystem

coordinates = ((10, 10), (10, 100), (200, 100), (200, 10))
coordinate_system = PixelSpace(width=850, height=1100)

element = Element(coordinates=coordinates, coordinate_system=coordinate_system)
print(element.metadata.coordinates.to_dict())
print(element.metadata.coordinates.system.orientation)
print(element.metadata.coordinates.system.width)
print(element.metadata.coordinates.system.height)

element.convert_coordinates_to_new_system(RelativeCoordinateSystem(), in_place=True)
# Should now be in terms of new coordinate system
print(element.metadata.coordinates.to_dict())
print(element.metadata.coordinates.system.orientation)
print(element.metadata.coordinates.system.width)
print(element.metadata.coordinates.system.height)

```

### Additional metadata fields by document type

| Field Name             | Applicable Doc Types | Description                                                                     |
| ---------------------- | -------------------- | ------------------------------------------------------------------------------- |
| `page_number`          | DOCX, PDF, PPT, XLSX | Page number                                                                     |
| `page_name`            | XLSX                 | Sheet name in an Excel document                                                 |
| `sent_from`            | EML                  | Email sender                                                                    |
| `sent_to`              | EML                  | Email recipient                                                                 |
| `subject`              | EML                  | Email subject                                                                   |
| `attached_to_filename` | MSG                  | filename that attachment file is attached to                                    |
| `header_footer_type`   | Word Doc             | Pages a header or footer applies to: "primary", "even\_only", and "first\_page" |
| `link_urls`            | HTML                 | The url associated with a link in a document.                                   |
| `link_texts`           | HTML                 | The text associated with a link in a document.                                  |
| `section`              | EPUB                 | Book section title corresponding to table of contents                           |

Notes on additional metadata by document type:

#### Email

Emails will include `sent_from`, `sent_to`, and `subject` metadata. `sent_from` is a list of strings because
the [RFC 822](https://www.rfc-editor.org/rfc/rfc822) spec for emails allows for multiple sent from email addresses.

#### Microsoft Excel documents

For Excel documents, `ElementMetadata` will contain a `page_name` element, which corresponds to the sheet name in the Excel
document.

#### Microsoft Word documents

Headers and footers in Word documents include a `header_footer_type` indicating which page a header or footer applies to.
Valid values are `"primary"`, `"even_only"`, and `"first_page"`.

### Table-specific metadata

For `Table` elements, the raw text of the table will be stored in the `text` attribute for the Element, and HTML representation
of the table will be available in the element metadata under `element.metadata.text_as_html`. By default,
Unstructured will automatically extract all tables for all doc types unless you set `skip_infer_table_types` parameter.

Here's an example of a table element. The `text` of the element will look like this:

```
Dataset Base Model1 Large Model Notes PubLayNet [38] F / M M Layouts of modern scientific documents PRImA [3] M - Layouts of scanned modern magazines and scientific reports Newspaper [17] F - Layouts of scanned US newspapers from the 20th century TableBank [18] F F Table region on modern scientific and business document HJDataset [31] F / M - Layouts of history Japanese documents
```

And the `text_as_html` metadata for the same element will look like this:

```py
<table><thead><th>Dataset</th><th>| Base Model’</th><th>| Notes</th></thead><tr><td>PubLayNet</td><td>[38] F/M</td><td>Layouts of modern scientific documents</td></tr><tr><td>PRImA [3]</td><td>M</td><td>Layouts of scanned modern magazines and scientific reports</td></tr><tr><td>Newspaper</td><td>F</td><td>Layouts of scanned US newspapers from the 20th century</td></tr><tr><td>TableBank</td><td>F</td><td>Table region on modern scientific and business document</td></tr><tr><td>HJDataset [31]</td><td>F/M</td><td>Layouts of history Japanese documents</td></tr></table>
```

### Data connector metadata fields

Documents processed through source connectors include additional document metadata. These additional fields only ever
appear if the source document was processed by a connector.

#### Common data connector metadata fields

*   Data Source metadata (on json output):
    *   url
    *   version
    *   date created
    *   date modified
    *   date processed
    *   record locator
*   Record locator is specific to each connector

#### Additional metadata fields by connector type (via record locator)

| Source connector      | Additional metadata              |
| --------------------- | -------------------------------- |
| airtable              | base id, table id, view id       |
| azure (from fsspec)   | protocol, remote file path       |
| box (from fsspec)     | protocol, remote file path       |
| confluence            | url, page id                     |
| discord               | channel                          |
| dropbox (from fsspec) | protocol, remote file path       |
| elasticsearch         | url, index name, document id     |
| fsspec                | protocol, remote file path       |
| google drive          | drive id, file id                |
| gcs (from fsspec)     | protocol, remote file path       |
| jira                  | base url, issue key              |
| onedrive              | user pname, server relative path |
| outlook               | message id, user email           |
| s3 (from fsspec)      | protocol, remote file path       |
| sharepoint            | server path, site url            |
| wikipedia             | page title, age url              |


# Models

Depending on your need, `Unstructured` provides OCR-based and Transformer-based models to detect elements in the documents. The models are useful to detect the complex layout in the documents and predict the element types.

**Basic usage:**

```python
elements = partition(filename=filename,
                     strategy="hi_res",
                     hi_res_model_name="yolox")
```

<Note>
  *   To use any model with the partition, set the `strategy` to `hi_res` as shown above.

  *   To maintain the consistency between the `unstructured` and `unstructured-api` libraries, we are deprecating the `model_name` parameter. Please use `hi_res_model_name` parameter when specifying a model.
</Note>

The `hi_res_model_name` parameter supports the `yolox` and `detectron2_onnx` arguments.

## Using a Non-Default Model

`Unstructured` will download the model specified in `UNSTRUCTURED_HI_RES_MODEL_NAME` environment variable. If not defined, it will download the default model.

There are three ways you can use the non-default model as follows:

1.  Store the model name in the environment variable

```python
import os
from unstructured.partition.pdf import partition_pdf

os.environ["UNSTRUCTURED_HI_RES_MODEL_NAME"] = "yolox"

out_yolox = partition_pdf("example-docs/pdf/layout-parser-paper-fast.pdf", strategy="hi_res")

```

2.  Pass the model name in the `partition` function.

```python
filename = "example-docs/pdf/layout-parser-paper-fast.pdf"

elements = partition(filename=filename,
                     strategy="hi_res",
                     hi_res_model_name="yolox")

```

3.  Use [unstructured-inference](https://github.com/Unstructured-IO/unstructured-inference) library.

```python
from unstructured_inference.models.base import get_model
from unstructured_inference.inference.layout import DocumentLayout

model = get_model("yolox")
layout = DocumentLayout.from_file("sample-docs/layout-parser-paper.pdf", detection_model=model)

```


# Partitioning strategies



For certain document types, such as images and PDFs, for example, Unstructured products offer a variety of different
ways to preprocess them, controlled by the `strategy` parameter.

PDF documents, for example, vary in quality and complexity. In simple cases, traditional NLP extraction techniques may
be enough to extract all the text out of a document. In other cases, advanced image-to-text models are required
to process a PDF. You can think of the strategies as being "rule-based" workflows (thus they are "fast"), or
"model-based" workflows (slower workflow because it requires model inference, but you get "higher resolution", thus "hi\_res").
When choosing a partitioning strategy for your files, you have to be mindful of the quality/speed trade-off.
To give you an example, the `fast` strategy is roughly 100x faster than leading image-to-text models.

**Available options:**

*   `auto` (default strategy): The "auto" strategy will choose the partitioning strategy based on document characteristics and the function kwargs.
*   `fast`:  The "rule-based" strategy leverages traditional NLP extraction techniques to quickly pull all the text elements. "Fast" strategy is not recommended for image-based file types.
*   `hi_res`: The "model-based" strategy identifies the layout of the document. The advantage of "hi\_res" is that it uses the document layout to gain additional information about document elements. We recommend using this strategy if your use case is highly sensitive to correct classifications for document elements.
*   `ocr_only`: Another "model-based" strategy that leverages Optical Character Recognition to extract text from the image-based files.

**These strategies are available on the following partition functions:**

| Document Type            | Partition Function | Strategies                             | Table Support | Options                                                                                      |
| ------------------------ | ------------------ | -------------------------------------- | ------------- | -------------------------------------------------------------------------------------------- |
| Images (.png/.jpg/.heic) | partition\_image   | “auto”, “hi\_res”, “ocr\_only”         | Yes           | Encoding; Include Page Breaks; Infer Table Structure; OCR Languages, Strategy                |
| PDFs (.pdf)              | partition\_pdf     | “auto”, “fast”, “hi\_res”, “ocr\_only” | Yes           | Encoding; Include Page Breaks; Infer Table Structure; Max Partition; OCR Languages, Strategy |


# Chunking

Chunking functions in `unstructured` use metadata and document elements detected with `partition` functions to post-process elements into more useful "chunks" for uses cases such as Retrieval Augmented Generation (RAG).

## Chunking Basics

Chunking in `unstructured` differs from other chunking mechanisms you may be familiar with. Typical approaches start with the text extracted from the document and form chunks based on plain-text features, character sequences like `"\n\n"` or `"\n"` that might indicate a paragraph boundary or list-item boundary.

Because `unstructured` uses specific knowledge about each document format to partition the document into semantic units (document elements), we only need to resort to text-splitting when a single element exceeds the desired maximum chunk size. Except in that case, all chunks contain one or more whole elements, preserving the coherence of semantic units established during partitioning.

A few concepts about chunking are worth introducing before discussing the details.

*   Chunking is performed on *document elements*. It is a separate step performed *after* partitioning, on the elements produced by partitioning. (Although it can be combined with partitioning in a single step.)

*   In general, chunking *combines* consecutive elements to form chunks as large as possible without exceeding the maximum chunk size.

*   A single element that by itself exceeds the maximum chunk size is divided into two or more chunks using text-splitting.

*   Chunking produces a sequence of `CompositeElement`, `Table`, or `TableChunk` elements. Each “chunk” is an instance of one of these three types.

## Chunking Options

The following options are available to tune chunking behaviors. These are keyword arguments that can be used in a partitioning or chunking function call. All these options have defaults and need only be specified when a non-default setting is required. Specific chunking strategies (such as `by_title`) may have additional options.

*   `max_characters: int (default=500)` - the hard maximum size for a chunk. No chunk will exceed this number of characters. A single element that by itself exceeds this size will be divided into two or more chunks using text-splitting.

*   `new_after_n_chars: int (default=max_characters)` - the “soft” maximum size for a chunk. A chunk that already exceeds this number of characters will not be extended, even if the next element would fit without exceeding the specified hard maximum. This can be used in conjunction with `max_characters` to set a “preferred” size, like “I prefer chunks of around 1000 characters, but I’d rather have a chunk of 1500 (max\_characters) than resort to text-splitting”. This would be specified with `(..., max_characters=1500, new_after_n_chars=1000)`.

*   `overlap: int (default=0)` - only when using text-splitting to break up an oversized chunk, include this number of characters from the end of the prior chunk as a prefix on the next. This can mitigate the effect of splitting the semantic unit represented by the oversized element at an arbitrary position based on text length.

*   `overlap_all: bool (default=False)` - also apply overlap between “normal” chunks, not just when text-splitting to break up an oversized element. Because normal chunks are formed from whole elements that each have a clean semantic boundary, this option may “pollute” normal chunks. You’ll need to decide based on your use-case whether this option is right for you.

## Chunking

Chunking can be performed as part of partitioning or as a separate step after partitioning:

### Specifying a chunking strategy while partitioning

Chunking can be performed as part of partitioning by specifying a value for the `chunking_strategy` argument. The current options are `basic` and `by_title` (described below).

```python
from unstructured.partition.html import partition_html

chunks = partition_html(url=url, chunking_strategy="basic")

```

### Calling a chunking function

Chunking can also be performed separately from partitioning by calling a chunking function directly. This may be convenient, for example, when tuning chunking parameters. Chunking is typically faster than partitioning, especially when OCR or inference is used, so a faster feedback loop is possible by doing these separately:

```python
from unstructured.chunking.basic import chunk_elements
from unstructured.partition.html import partition_html

url = "https://understandingwar.org/backgrounder/russian-offensive-campaign-assessment-august-27-2023-0"
elements = partition_html(url=url)
chunks = chunk_elements(elements)

# -- OR --

from unstructured.chunking.title import chunk_by_title

chunks = chunk_by_title(elements)

for chunk in chunks:
    print(chunk)
    print("\n\n" + "-"*80)
    input()

```

## Chunking Strategies

There are currently two chunking strategies, *basic* and *by\_title*. The `by_title` strategy shares most behaviors with
the basic strategy so we'll describe the baseline strategy first:

### "basic" chunking strategy

*   The basic strategy combines sequential elements to maximally fill each chunk while respecting both the specified `max_characters` (hard-max) and `new_after_n_chars` (soft-max) option values.

*   A single element that by itself exceeds the hard-max is isolated (never combined with another element) and then divided into two or more chunks using text-splitting.

*   A `Table` element is always isolated and never combined with another element. A `Table` can be oversized, like any other text element, and in that case is divided into two or more `TableChunk` elements using text-splitting.

*   If specified, `overlap` is applied between chunks formed by splitting oversized elements and is also applied between other chunks when `overlap_all` is `True`.

### "by\_title" chunking strategy

The `by_title` chunking strategy preserves section boundaries and optionally page boundaries as well. “Preserving” here means that a single chunk will never contain text that occurred in two different sections. When a new section starts, the existing chunk is closed and a new one started, even if the next element would fit in the prior chunk.

In addition to the behaviors of the `basic` strategy above, the `by_title` strategy has the following behaviors:

*   **Detect section headings.** A `Title` element is considered to start a new section. When a `Title` element is encountered, the prior chunk is closed and a new chunk started, even if the `Title` element would fit in the prior chunk.

*   **Respect page boundaries.** Page boundaries can optionally also be respected using the `multipage_sections` argument. This defaults to `True` meaning that a page break does *not* start a new chunk. Setting this to `False` will separate elements that occur on different pages into distinct chunks.

*   **Combine small sections.** In certain documents, partitioning may identify a list-item or other short paragraph as a `Title` element even though it does not serve as a section heading. This can produce chunks substantially smaller than desired. This behavior can be mitigated using the `combine_text_under_n_chars` argument. This defaults to the same value as `max_characters` such that sequential small sections are combined to maximally fill the chunking window. Setting this to `0` will disable section combining.

## Recovering Chunk Elements

In general, a chunk consolidates multiple document elements to maximally fill a chunk of the desired size. Information is naturally lost in this consolidation, for example which element a portion of the text came from and certain metadata like page-number and coordinates which cannot always be resolved to a single value.

The original elements combined to make a chunk can be accessed using the .metadata.orig\_elements field on the chunk:

```python
>>> elements = [
...     Title("Lorem Ipsum"),
...     NarrativeText("Lorem ipsum dolor sit."),
... ]
>>> chunk = chunk_elements(elements)[0]
>>> print(chunk.text)
'Lorem Ipsum\n\nLorem ipsum dolor sit.'
>>> print(chunk.metadata.orig_elements)
[Title("Lorem Ipsum"), NarrativeText("Lorem ipsum dolor sit.")]

```

These elements will contain all their original metadata so can be used to access metadata that cannot reliably be consolidated, for example:

```python python
>>> {e.metadata.page_number for e in chunk.metadata.orig_elements}
{2, 3}

```

```
>>> [e.metadata.coordinates for e in chunk.metadata.orig_elements]
[<CoordinatesMetadata ...>, <CoordinatesMetadata ...>, ...]

```

```
>>> [
    e.metadata.image_path
    for e in chunk.metadata.orig_elements
    if e.metadata.image_path is not None
]
['/tmp/lorem.jpg', '/tmp/ipsum.png']

```

During serialization, `.metadata.orig_elements` is compressed into Base64 gzipped format. To deserialize `.metadata.orig_elements`, you can use the `elements_from_base64_gzipped_json`. For example:

```python
from unstructured.partition.auto import partition
from unstructured.staging.base import elements_from_base64_gzipped_json

elements = partition('local-ingest-source/fake-email.eml', chunking_strategy='basic', include_orig_elements=True)

print("Before:\n")

for element in elements:
    metadata = element.metadata.to_dict()
    print(f"Element ID: {element.id}")
    print(f"  Compressed orig_elements: {metadata["orig_elements"]}")
    print("\n")

# Output:
# -------
# Before:
#
# Element ID: 083776ca703b1925e5fef69bb2635f1f
#   Compressed orig_elements: eJztlM1uGyEUhV8Fzbq2GGbwDFZUZdNFpDaLyOomiix+Lg7tABbcaRtFfffCJFXSyiuvLbFA3HsOHD7B/XMDE3gIuHem2ZJGmLaDnmnBGTfajF0/aqm6zSi4gnEwzQfSeEBpJMrS/9xYN8HeuAQaY3qqFlPUclq5cICMqxznpKGqamOQHmqLld9hBV66aQ1+qtVJhsMsi6SU7xsIh+ZhWc2499E462A5HqOMrdoyNrt22NJ+225WlG8prR65xrAp+sXji0R8hJ/kLioXcgzkyqfX6fUcMqZZ45zArF38uGy2yDEu4tuIf/VXb/PrEPEfqY7+VTurb+UG6hF3JTb5VLM1v0sF4dfL8qPLpAxJsDYs4QlGMmcgNiYyB4dLKa9rFnw6Ljd1K1OS6H7ArvoUw/+BjZveMt5zxQSVgo2S8kGr1ggtKLOivwA7E9iNP8aEMiA5Rhcwb99j2Tmc4BQOBtyOYuwko5J3RnLRtszowYpOWj5ccJyL4y5mKK8nASnC9yg+u4w3CP4UDWWMVEpT3fZUmb63RvORgR3o2Ovy011onEnjq4sT4AsPNc1wGsjDH93mEjs=

print ("After:\n")

for element in elements:
    metadata = element.metadata.to_dict()
    print(f"Element ID: {element.id}")
    orig_elements = elements_from_base64_gzipped_json(metadata["orig_elements"])
    print(f"  Uncompressed orig_elements:")
    for orig_element in orig_elements:
        print(f"    {orig_element.category}: {orig_element.text}")
    print("\n")

# Output:
# -------
# After:
#
# Element ID: 083776ca703b1925e5fef69bb2635f1f
#   Uncompressed orig_elements:
#     NarrativeText: This is a test email to use for unit tests.
#     Title: Important points:
#     ListItem: Roses are red
#     ListItem: Violets are blue
```

## Learn more

<Icon icon="blog" />  [Chunking for RAG: best practices](https://unstructured.io/blog/chunking-for-rag-best-practices)


# Cleaning

As part of data preparation for an NLP model, it’s common to need to clean up your data prior to passing it into the model. If there’s unwanted content in your output, for example, it could impact the quality of your NLP model. To help with this, the `unstructured` library includes cleaning functions to help users sanitize output before sending it to downstream applications.

Some cleaning functions apply automatically. In the example in the **Partition** section, the output `Philadelphia Eaglesâ\x80\x99 victory` automatically gets converted to `Philadelphia Eagles' victory` in `partition_html` using the `replace_unicode_quotes` cleaning function. You can see how that works in the code snippet below:

```python
from unstructured.cleaners.core import replace_unicode_quotes

replace_unicode_quotes("Philadelphia Eaglesâ\x80\x99 victory")

```

Document elements in `unstructured` include an `apply` method that allow you to apply the text cleaning to the document element without instantiating a new element. The `apply` method expects a callable that takes a string as input and produces another string as output. In the example below, we invoke the `replace_unicode_quotes` cleaning function using the `apply` method.

```python
from unstructured.documents.elements import Text

element = Text("Philadelphia Eaglesâ\x80\x99 victory")
element.apply(replace_unicode_quotes)
print(element)

```

Since a cleaning function is just a `str -> str` function, users can also easily include their own cleaning functions for custom data preparation tasks. In the example below, we remove citations from a section of text.

```python
import re

remove_citations = lambda text: re.sub("\[\d{1,3}\]", "", text)

element = Text("[1] Geolocated combat footage has confirmed Russian gains in the Dvorichne area northwest of Svatove.")
element.apply(remove_citations)
print(element)

```

See below for a full list of cleaning functions in the `unstructured` library.

## `bytes_string_to_string`

Converts an output string that looks like a byte string to a string using the specified encoding. This happens sometimes in `partition_html` when there is a character like an emoji that isn’t expected by the HTML parser. In that case, the encoded bytes get processed.

Examples:

```python
from unstructured.cleaners.core import bytes_string_to_string

text = "Hello ð\x9f\x98\x80"
# The output should be "Hello 😀"
bytes_string_to_string(text, encoding="utf-8")

```

```python
from unstructured.cleaners.core import bytes_string_to_string
from unstructured.partition.html import partition_html

text = """\n<html charset="utf-8"><p>Hello 😀</p></html>"""
elements = partition_html(text=text)
elements[0].apply(bytes_string_to_string)
# The output should be "Hello 😀"
elements[0].text

```

For more information about the `bytes_string_to_string` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `clean`

Cleans a section of text with options including removing bullets, extra whitespace, dashes and trailing punctuation. Optionally, you can choose to lowercase the output.

Options:

*   Applies `clean_bullets` if `bullets=True`.

*   Applies `clean_extra_whitespace` if `extra_whitespace=True`.

*   Applies `clean_dashes` if `dashes=True`.

*   Applies `clean_trailing_punctuation` if `trailing_punctuation=True`.

*   Lowercases the output if `lowercase=True`.

Examples:

```python
from unstructured.cleaners.core import clean

# Returns "an excellent point!"
clean("● An excellent point!", bullets=True, lowercase=True)

# Returns "ITEM 1A: RISK FACTORS"
clean("ITEM 1A:     RISK-FACTORS", extra_whitespace=True, dashes=True)

```

For more information about the `clean` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `clean_bullets`

Removes bullets from the beginning of text. Bullets that do not appear at the beginning of the text are not removed.

Examples:

```python
from unstructured.cleaners.core import clean_bullets

# Returns "An excellent point!"
clean_bullets("● An excellent point!")

# Returns "I love Morse Code! ●●●"
clean_bullets("I love Morse Code! ●●●")

```

For more information about the `clean_bullets` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `clean_dashes`

Removes dashes from a section of text. Also handles special characters such as `\u2013`.

Examples:

```python
from unstructured.cleaners.core import clean_dashes

# Returns "ITEM 1A: RISK FACTORS"
clean_dashes("ITEM 1A: RISK-FACTORS\u2013")

```

For more information about the `clean_dashes` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `clean_non_ascii_chars`

Removes non-ascii characters from a string.

Examples:

```python
from unstructured.cleaners.core import clean_non_ascii_chars

text = "\x88This text contains ®non-ascii characters!●"

# Returns "This text contains non-ascii characters!"
clean_non_ascii_chars(text)

```

For more information about the `clean_non_ascii_chars` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `clean_ordered_bullets`

Remove alphanumeric bullets from the beginning of text up to three “sub-section” levels.

Examples:

```python
from unstructured.cleaners.core import clean_ordered_bullets

# Returns "This is a very important point"
clean_ordered_bullets("1.1 This is a very important point")

# Returns "This is a very important point ●"
clean_ordered_bullets("a.b This is a very important point ●")

```

For more information about the `clean_ordered_bullets` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `clean_postfix`

Removes the postfix from a string if they match a specified pattern.

Options:

*   Ignores case if `ignore_case` is set to `True`. The default is `False`.

*   Strips trailing whitespace is `strip` is set to `True`. The default is `True`.

Examples:

```python
from unstructured.cleaners.core import clean_postfix

text = "The end! END"

# Returns "The end!"
clean_postfix(text, r"(END|STOP)", ignore_case=True)

```

For more information about the `clean_postfix` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `clean_prefix`

Removes the prefix from a string if they match a specified pattern.

Options:

*   Ignores case if `ignore_case` is set to `True`. The default is `False`.

*   Strips leading whitespace is `strip` is set to `True`. The default is `True`.

Examples:

```python
from unstructured.cleaners.core import clean_prefix

text = "SUMMARY: This is the best summary of all time!"

# Returns "This is the best summary of all time!"
clean_prefix(text, r"(SUMMARY|DESCRIPTION):", ignore_case=True)

```

For more information about the `clean_prefix` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `clean_trailing_punctuation`

Removes trailing punctuation from a section of text.

Examples:

```python
from unstructured.cleaners.core import clean_trailing_punctuation

# Returns "ITEM 1A: RISK FACTORS"
clean_trailing_punctuation("ITEM 1A: RISK FACTORS.")

```

For more information about the `clean_trailing_punctuation` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `group_broken_paragraphs`

Groups together paragraphs that are broken up with line breaks for visual or formatting purposes. This is common in `.txt` files. By default, `group_broken_paragraphs` groups together lines split by `\n`. You can change that behavior with the `line_split` kwarg. The function considers `\n\n` to be a paragraph break by default. You can change that behavior with the `paragraph_split` kwarg.

Examples:

```python
from unstructured.cleaners.core import group_broken_paragraphs

text = """The big brown fox
was walking down the lane.

At the end of the lane, the
fox met a bear."""

group_broken_paragraphs(text)

```

```python
import re
from unstructured.cleaners.core import group_broken_paragraphs

para_split_re = re.compile(r"(\s*\n\s*){3}")

text = """The big brown fox

was walking down the lane.


At the end of the lane, the

fox met a bear."""

group_broken_paragraphs(text, paragraph_split=para_split_re)

```

For more information about the `group_broken_paragraphs` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `remove_punctuation`

Removes ASCII and unicode punctuation from a string.

Examples:

```python
from unstructured.cleaners.core import remove_punctuation

# Returns "A lovely quote"
remove_punctuation("“A lovely quote!”")

```

For more information about the `remove_punctuation` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `replace_unicode_quotes`

Replaces unicode quote characters such as `\x91` in strings.

Examples:

```
from unstructured.cleaners.core import replace_unicode_quotes

# Returns "“A lovely quote!”"
replace_unicode_characters("\x93A lovely quote!\x94")

# Returns ""‘A lovely quote!’"
replace_unicode_characters("\x91A lovely quote!\x92")

```

For more information about the `replace_unicode_quotes` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `translate_text`

The `translate_text` cleaning functions translates text between languages. `translate_text` uses the [Helsinki NLP MT models](https://huggingface.co/Helsinki-NLP) from `transformers` for machine translation. Works for Russian, Chinese, Arabic, and many other languages.

Parameters:

*   `text`: the input string to translate.

*   `source_lang`: the two letter language code for the source language of the text. If `source_lang` is not specified, the language will be detected using `langdetect`.

*   `target_lang`: the two letter language code for the target language for translation. Defaults to `"en"`.

Examples:

```python
from unstructured.cleaners.translate import translate_text

# Output is "I'm a Berliner!"
translate_text("Ich bin ein Berliner!")

# Output is "I can also translate Russian!"
translate_text("Я тоже можно переводать русский язык!", "ru", "en")

```

For more information about the `translate_text` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/translate.py).


# Embedding



The Unstructured open-source library does not offer built-in support for calling embedding providers to obtain embeddings for pieces of text.

Alternatively, the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) and the [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library)
offer built-in support for calling embedding providers as part of an ingest pipeline. [Learn how](/api-reference/how-to/embedding).

Also, you can use common third-party tools and libraries to get embeddings for document elements' text within JSON files that are
produced by calling the Unstructured open-source library. For example, the following sample Python script:

1.  Takes an Unstructured open-source library-generated JSON file as input.
2.  Reads in the JSON file's contents as a JSON object.
3.  Uses the [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)
    model on Hugging Face to generate embeddings for each `text` field of each document element in the JSON file.
4.  Adds the generated embeddings next to each corresponding `text` field in the original JSON.
5.  Saves the results back to the original JSON file.

```python Python
# Filename: embeddings.py
# pip install langchain sentence_transformers

import sys
import json
from langchain_huggingface.embeddings import HuggingFaceEmbeddings

if __name__ == "__main__":
    embeddings = HuggingFaceEmbeddings(
        model_name="sentence-transformers/all-MiniLM-L6-v2"
    )
    
    # Get the JSON file's path.
    if len(sys.argv) < 2:
        print("Error: Specify the path to the input JSON file.")
        print("For example, 'python embeddings.py myfile.json'")
        sys.exit(1)

    file_path = sys.argv[1]
    
    try:
        # Get the JSON file's contents.
        with open(file_path, 'r') as file:
            file_elements = json.load(file)

        # Process each element in the JSON file.
        for element in file_elements:
            # Get the element's "text" field.
            text = element["text"]
            # Generate the embeddings for that "text" field.
            query_result = embeddings.embed_query(text)
            # Add the embeddings to that element as an "embeddings" field.
            element["embeddings"] = query_result

        # Save the updated JSON back into the original file.
        with open(file_path, 'w') as file:
            json.dump(file_elements, file, indent=2)

        print(f"Done! Updated JSON saved to '{file_path}'.")

    except FileNotFoundError:
        print(f"Error: File '{file_path}' not found.")
    except IOError:
        print(f"Error: Unable to access file '{file_path}'.")
```

## Additional resources

For information about how to use Python scripts to call various embedding providers, see for example:

*   [Amazon Bedrock](https://docs.aws.amazon.com/code-library/latest/ug/python_3_bedrock-runtime_code_examples.html#amazon_titan_text_embeddings)
*   [Hugging Face](https://huggingface.co/blog/getting-started-with-embeddings)
*   [OctoAI](https://octo.ai/blog/introducing-octoais-embedding-api-to-power-your-rag-needs/)
*   [OpenAI](https://platform.openai.com/docs/guides/embeddings)
*   [together.ai](https://docs.together.ai/docs/embeddings-overview)
*   [Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings)
*   [Voyage AI](https://docs.voyageai.com/docs/embeddings)


# Extracting



## `extract_datetimetz`

Extracts the date, time, and timezone in the `Received` field(s) from an `.eml` file. `extract_datetimetz` takes in a string and returns a datetime.datetime object from the input string.

```python
from unstructured.cleaners.extract import extract_datetimetz

text = """from ABC.DEF.local ([ba23::58b5:2236:45g2:88h2]) by
  \n ABC.DEF.local2 ([ba23::58b5:2236:45g2:88h2%25]) with mapi id\
  n 32.88.5467.123; Fri, 26 Mar 2021 11:04:09 +1200"""

# Returns datetime.datetime(2021, 3, 26, 11, 4, 9, tzinfo=datetime.timezone(datetime.timedelta(seconds=43200)))
extract_datetimetz(text)

```

For more information about the `extract_datetimetz` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `extract_email_address`

Extracts email addresses from a string input and returns a list of all the email addresses in the input string.

```python
from unstructured.cleaners.extract import extract_email_address

text = """Me me@email.com and You <You@email.com>
    ([ba23::58b5:2236:45g2:88h2]) (10.0.2.01)"""

# Returns "['me@email.com', 'you@email.com']"
extract_email_address(text)

```

For more information about the `extract_email_address` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `extract_ip_address`

Extracts IPv4 and IPv6 IP addresses in the input string and returns a list of all IP address in input string.

```python
from unstructured.cleaners.extract import extract_ip_address

text = """Me me@email.com and You <You@email.com>
  ([ba23::58b5:2236:45g2:88h2]) (10.0.2.01)"""

# Returns "['ba23::58b5:2236:45g2:88h2', '10.0.2.01']"
extract_ip_address(text)

```

For more information about the `extract_ip_address` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `extract_ip_address_name`

Extracts the names of each IP address in the `Received` field(s) from an `.eml` file. `extract_ip_address_name` takes in a string and returns a list of all IP addresses in the input string.

```python
from unstructured.cleaners.extract import extract_ip_address_name

text = """from ABC.DEF.local ([ba23::58b5:2236:45g2:88h2]) by
  \n ABC.DEF.local2 ([ba23::58b5:2236:45g2:88h2%25]) with mapi id\
  n 32.88.5467.123; Fri, 26 Mar 2021 11:04:09 +1200"""

# Returns "['ABC.DEF.local', 'ABC.DEF.local2']"
extract_ip_address_name(text)

```

For more information about the `extract_ip_address_name` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `extract_mapi_id`

Extracts the `mapi id` in the `Received` field(s) from an `.eml` file. `extract_mapi_id` takes in a string and returns a list of a string containing the `mapi id` in the input string.

```python
from unstructured.cleaners.extract import extract_mapi_id

text = """from ABC.DEF.local ([ba23::58b5:2236:45g2:88h2]) by
  \n ABC.DEF.local2 ([ba23::58b5:2236:45g2:88h2%25]) with mapi id\
  n 32.88.5467.123; Fri, 26 Mar 2021 11:04:09 +1200"""

# Returns "['32.88.5467.123']"
extract_mapi_id(text)

```

For more information about the `extract_mapi_id` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `extract_ordered_bullets`

Extracts alphanumeric bullets from the beginning of text up to three “sub-section” levels.

Examples:

```python
from unstructured.cleaners.extract import extract_ordered_bullets

# Returns ("1", "1", None)
extract_ordered_bullets("1.1 This is a very important point")

# Returns ("a", "1", None)
extract_ordered_bullets("a.1 This is a very important point")

```

For more information about the `extract_ordered_bullets` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `extract_text_after`

Extracts text that occurs after the specified pattern.

Options:

*   If `index` is set, extract after the `(index + 1)`th occurrence of the pattern. The default is `0`.

*   Strips trailing whitespace if `strip` is set to `True`. The default is `True`.

Examples:

```python
from unstructured.cleaners.extract import extract_text_after

text = "SPEAKER 1: Look at me, I'm flying!"

# Returns "Look at me, I'm flying!"
extract_text_after(text, r"SPEAKER \d{1}:")

```

For more information about the `extract_text_after` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `extract_text_before`

Extracts text that occurs before the specified pattern.

Options:

*   If `index` is set, extract before the `(index + 1)`th occurrence of the pattern. The default is `0`.

*   Strips leading whitespace if `strip` is set to `True`. The default is `True`.

Examples:

```python
from unstructured.cleaners.extract import extract_text_before

text = "Here I am! STOP Look at me! STOP I'm flying! STOP"

# Returns "Here I am!"
extract_text_before(text, r"STOP")

```

For more information about the `extract_text_before` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `extract_us_phone_number`

Extracts a phone number from a section of text.

Examples:

```python
from unstructured.cleaners.extract import extract_us_phone_number

# Returns "215-867-5309"
extract_us_phone_number("Phone number: 215-867-5309")

```

For more information about the `extract_us_phone_number` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `group_broken_paragraphs`

Groups together paragraphs that are broken up with line breaks for visual or formatting purposes. This is common in `.txt` files. By default, `group_broken_paragraphs` groups together lines split by `\n`. You can change that behavior with the `line_split` kwarg. The function considers `\n\n` to be a paragraph break by default. You can change that behavior with the `paragraph_split` kwarg.

Examples:

```python
from unstructured.cleaners.core import group_broken_paragraphs

text = """The big brown fox
was walking down the lane.

At the end of the lane, the
fox met a bear."""

group_broken_paragraphs(text)

```

```python
import re
from unstructured.cleaners.core import group_broken_paragraphs

para_split_re = re.compile(r"(\s*\n\s*){3}")

text = """The big brown fox

was walking down the lane.


At the end of the lane, the

fox met a bear."""

group_broken_paragraphs(text, paragraph_split=para_split_re)

```

For more information about the `group_broken_paragraphs` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `remove_punctuation`

Removes ASCII and unicode punctuation from a string.

Examples:

```python
from unstructured.cleaners.core import remove_punctuation

# Returns "A lovely quote"
remove_punctuation("“A lovely quote!”")

```

For more information about the `remove_punctuation` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `replace_unicode_quotes`

Replaces unicode quote characters such as `\x91` in strings.

Examples:

```python
from unstructured.cleaners.core import replace_unicode_quotes

# Returns "“A lovely quote!”"
replace_unicode_characters("\x93A lovely quote!\x94")

# Returns ""‘A lovely quote!’"
replace_unicode_characters("\x91A lovely quote!\x92")

```

For more information about the `replace_unicode_quotes` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `translate_text`

The `translate_text` cleaning function translates text between languages. `translate_text` uses the [Helsinki NLP MT models](https://huggingface.co/Helsinki-NLP) from `transformers` for machine translation. Works for Russian, Chinese, Arabic, and many other languages.

Parameters:

*   `text`: the input string to translate.

*   `source_lang`: the two letter language code for the source language of the text. If `source_lang` is not specified, the language will be detected using `langdetect`.

*   `target_lang`: the two letter language code for the target language for translation. Defaults to `"en"`.

Examples:

```python
from unstructured.cleaners.translate import translate_text

# Output is "I'm a Berliner!"
translate_text("Ich bin ein Berliner!")

# Output is "I can also translate Russian!"
translate_text("Я тоже можно переводать русский язык!", "ru", "en")

```

For more information about the `translate_text` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/translate.py).


# Core Functionality

The `unstructured` library includes functions to partition, chunk, clean, and stage raw source documents. These functions serve as the primary public interfaces within the library.

After reading this section, you should understand the following:

*   How to partition a document into json or csv.

*   How to remove unwanted content from document elements using cleaning functions.

*   How to extract content from a document using the extraction functions.

*   How to prepare data for downstream use cases using staging functions

*   How to chunk partitioned documents for use cases such as Retrieval Augmented Generation (RAG).


# Partitioning

Partitioning functions in `unstructured` allow users to extract structured content from a raw unstructured document. These functions break a document down into elements such as `Title`, `NarrativeText`, and `ListItem`, enabling users to decide what content they’d like to keep for their particular application. If you’re training a summarization model, for example, you may only be interested in `NarrativeText`.

The easiest way to partition documents in unstructured is to use the `partition` function. If you call the `partition` function, `unstructured` will use `libmagic` to automatically determine the file type and invoke the appropriate partition function. In cases where `libmagic` is not available, filetype detection will fall back to using the file extension.

The following table shows the document types the `unstructured` library currently supports. `partition` will recognize each of these document types and route the document to the appropriate partitioning function. If you already know your document type, you can use the partitioning function listed in the table directly.

| Document Type                                                             | Partition Function | Strategies                             | Table Support | Options                                                                                      |
| ------------------------------------------------------------------------- | ------------------ | -------------------------------------- | ------------- | -------------------------------------------------------------------------------------------- |
| CSV Files (.csv)                                                          | partition\_csv     | N/A                                    | Yes           | None                                                                                         |
| E-mails (.eml)                                                            | partition\_email   | N/A                                    | No            | Encoding; Include Headers; Max Partition; Process Attachments                                |
| E-mails (.msg)                                                            | partition\_msg     | N/A                                    | No            | Encoding; Max Partition; Process Attachments                                                 |
| EPubs (.epub)                                                             | partition\_epub    | N/A                                    | Yes           | Include Page Breaks                                                                          |
| Excel Documents (.xlsx/.xls)                                              | partition\_xlsx    | N/A                                    | Yes           | None                                                                                         |
| HTML Pages (.html/.htm)                                                   | partition\_html    | N/A                                    | No            | Encoding; Include Page Breaks                                                                |
| Images (.png/.jpg/.jpeg/.tiff/.bmp/.heic)                                 | partition\_image   | “auto”, “hi\_res”, “ocr\_only”         | Yes           | Encoding; Include Page Breaks; Infer Table Structure; OCR Languages, Strategy                |
| Markdown (.md)                                                            | partition\_md      | N/A                                    | Yes           | Include Page Breaks                                                                          |
| Org Mode (.org)                                                           | partition\_org     | N/A                                    | Yes           | Include Page Breaks                                                                          |
| Open Office Documents (.odt)                                              | partition\_odt     | N/A                                    | Yes           | None                                                                                         |
| PDFs (.pdf)                                                               | partition\_pdf     | “auto”, “fast”, “hi\_res”, “ocr\_only” | Yes           | Encoding; Include Page Breaks; Infer Table Structure; Max Partition; OCR Languages, Strategy |
| Plain Text (.txt/.text/.log)                                              | partition\_text    | N/A                                    | No            | Encoding; Max Partition; Paragraph Grouper                                                   |
| PowerPoints (.ppt)                                                        | partition\_ppt     | N/A                                    | Yes           | Include Page Breaks                                                                          |
| PowerPoints (.pptx)                                                       | partition\_pptx    | N/A                                    | Yes           | Include Page Breaks                                                                          |
| ReStructured Text (.rst)                                                  | partition\_rst     | N/A                                    | Yes           | Include Page Breaks                                                                          |
| Rich Text Files (.rtf)                                                    | partition\_rtf     | N/A                                    | Yes           | Include Page Breaks                                                                          |
| TSV Files (.tsv)                                                          | partition\_tsv     | N/A                                    | Yes           | None                                                                                         |
| Word Documents (.doc)                                                     | partition\_doc     | N/A                                    | Yes           | Include Page Breaks                                                                          |
| Word Documents (.docx)                                                    | partition\_docx    | N/A                                    | Yes           | Include Page Breaks                                                                          |
| XML Documents (.xml)                                                      | partition\_xml     | N/A                                    | No            | Encoding; Max Partition; XML Keep Tags                                                       |
| Code Files (.js/.py/.java/ .cpp/.cc/.cxx/.c/.cs/ .php/.rb/.swift/.ts/.go) | partition\_text    | N/A                                    | No            | Encoding; Max Partition; Paragraph Grouper                                                   |

As shown in the examples below, the `partition` function accepts both filenames and file-like objects as input. `partition` also has some optional kwargs. For example, if you set `include_page_breaks=True`, the output will include `PageBreak` elements if the filetype supports it. Additionally you can bypass the filetype detection logic with the optional `content_type` argument which may be specified with either the `filename` or file-like object, `file`. You can find a full listing of optional kwargs in the documentation below.

```python
from unstructured.partition.auto import partition


filename = os.path.join(EXAMPLE_DOCS_DIRECTORY, "layout-parser-paper-fast.pdf")
elements = partition(filename=filename, content_type="application/pdf")
print("\n\n".join([str(el) for el in elements][:10]))

```

```python
from unstructured.partition.auto import partition


filename = os.path.join(EXAMPLE_DOCS_DIRECTORY, "layout-parser-paper-fast.pdf")
with open(filename, "rb") as f:
  elements = partition(file=f, include_page_breaks=True)
print("\n\n".join([str(el) for el in elements][5:15]))

```

The `unstructured` library also includes partitioning functions targeted at specific document types. The `partition` function uses these document-specific partitioning functions under the hood. There are a few reasons you may want to use a document-specific partitioning function instead of `partition`:

*   If you already know the document type, filetype detection is unnecessary. Using the document-specific function directly, or passing in the `content_type` will make your program run faster.

*   Fewer dependencies. You don’t need to install `libmagic` for filetype detection if you’re only using document-specific functions.

*   Additional features. The API for partition is the least common denominator for all document types. Certain document-specific function include extra features that you may want to take advantage of. For example, `partition_html` allows you to pass in a URL so you don’t have to store the `.html` file locally. See the documentation below learn about the options available in each partitioning function.

Below we see an example of how to partition a document directly with the URL using the partition\_html function.

```python
from unstructured.partition.html import partition_html

url = "https://www.cnn.com/2023/01/30/sport/empire-state-building-green-philadelphia-eagles-spt-intl/index.html"
elements = partition_html(url=url)
print("\n\n".join([str(el) for el in elements]))

```

## `partition`

The `partition` function is the simplest way to partition a document in `unstructured`. If you call the `partition` function, `unstructured` will attempt to detect the file type and route it to the appropriate partitioning function. All partitioning functions called within `partition` are called using the default kwargs. Use the document-type specific functions if you need to apply non-default settings. `partition` currently supports `.docx`, `.doc`, `.odt`, `.pptx`, `.ppt`, `.xlsx`, `.csv`, `.tsv`, `.eml`, `.msg`, `.rtf`, `.epub`, `.html`, `.xml`, `.pdf`, `.png`, `.jpg`, `.heic`, and `.txt` files. If you set the `include_page_breaks` kwarg to `True`, the output will include page breaks. This is only supported for `.pptx`, `.html`, `.pdf`, `.png`, `.heic`, and `.jpg`. The `strategy` kwarg controls the strategy for partitioning documents. Generally available strategies are “fast” for faster processing and “hi\_res” for more accurate processing.

```python
import docx

from unstructured.partition.auto import partition

document = docx.Document()
document.add_paragraph("Important Analysis", style="Heading 1")
document.add_paragraph("Here is my first thought.", style="Body Text")
document.add_paragraph("Here is my second thought.", style="Normal")
document.save("mydoc.docx")

elements = partition(filename="mydoc.docx")

with open("mydoc.docx", "rb") as f:
    elements = partition(file=f)

```

```python
from unstructured.partition.auto import partition

elements = partition(filename="example-docs/pdf/layout-parser-paper-fast.pdf")

```

The `partition` function also accepts a `url` kwarg for remotely hosted documents. If you want to force `partition` to treat the document as a particular MIME type, use the `content_type` kwarg in conjunction with `url`. Otherwise, `partition` will use the information from the `Content-Type` header in the HTTP response. The `ssl_verify` kwarg controls whether or not SSL verification is enabled for the HTTP request. By default it is on. Use `ssl_verify=False` to disable SSL verification in the request.

```python
from unstructured.partition.auto import partition

url = "https://raw.githubusercontent.com/Unstructured-IO/unstructured/main/LICENSE.md"
elements = partition(url=url)
elements = partition(url=url, content_type="text/markdown")

```

For more information about the `partition` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/auto.py).

## `partition_csv`

The `partition_csv` function pre-processes CSV files. The output is a single `Table` element. The `text_as_html` attribute in the element metadata will contain an HTML representation of the table.

Examples:

```python
from unstructured.partition.csv import partition_csv

elements = partition_csv(filename="example-docs/stanley-cups.csv")
print(elements[0].metadata.text_as_html)

```

For more information about the `partition_csv` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/csv.py).

## `partition_doc`

The `partition_doc` partitioning function pre-processes Microsoft Word documents saved in the `.doc` format. This partition function uses a combination of the styling information in the document and the structure of the text to determine the type of a text element. The `partition_doc` can take a filename or file-like object as input. `partition_doc` uses `libreoffice` to convert the file to `.docx` and then calls `partition_docx`. Ensure you have `libreoffice` installed before using `partition_doc`.

Examples:

```python
from unstructured.partition.doc import partition_doc

elements = partition_doc(filename="example-docs/fake.doc")

```

For more information about the `partition_doc` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/doc.py).

## `partition_docx`

The `partition_docx` partitioning function pre-processes Microsoft Word documents saved in the `.docx` format. This partition function uses a combination of the styling information in the document and the structure of the text to determine the type of a text element. The `partition_docx` can take a filename or file-like object as input, as shown in the two examples below.

Examples:

```python
import docx

from unstructured.partition.docx import partition_docx

document = docx.Document()
document.add_paragraph("Important Analysis", style="Heading 1")
document.add_paragraph("Here is my first thought.", style="Body Text")
document.add_paragraph("Here is my second thought.", style="Normal")
document.save("mydoc.docx")

elements = partition_docx(filename="mydoc.docx")

with open("mydoc.docx", "rb") as f:
    elements = partition_docx(file=f)

```

In Word documents, headers and footers are specified per section. In the output, the `Header` elements will appear at the beginning of a section and `Footer` elements will appear at the end. MSFT Word headers and footers have a `header_footer_type` metadata field indicating where the header or footer applies. Valid values are `"primary"`, `"first_page"` and `"even_page"`.

`partition_docx` will include page numbers in the document metadata when page breaks are present in the document. The function will detect user inserted page breaks and page breaks inserted by the Word document renderer. Some (but not all) Word document renderers insert page breaks when you save the document. If your Word document renderer does not do that, you may not see page numbers in the output even if you see them visually when you open the document. If that is the case, you can try saving the document with a different renderer.

For more information about the `partition_docx` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/docx.py).

## `partition_email`

The `partition_email` function partitions `.eml` documents and works with exports from email clients such as Microsoft Outlook and Gmail. The `partition_email` takes a filename, file-like object, or raw text as input and produces a list of document `Element` objects as output. Also `content_source` can be set to `text/html` (default) or `text/plain` to process the html or plain text version of the email, respectively. In order for `partition_email` to return the header information as elements, `include_headers` must be set to `True`. Header information is captured in element metadata regardless of the `include_headers` setting.

| Header     | Element type | Metadata name      |
| ---------- | ------------ | ------------------ |
| Subject    | Subject      | subject            |
| From       | Sender       | sent\_from         |
| To         | Recipient    | sent\_to           |
| Cc         | Recipient    | cc\_recipient      |
| Bcc        | Recipient    | bcc\_recipient     |
| Received   | ReceivedInfo | NA                 |
| Message-ID | MetaData     | email\_message\_id |
| (other)    | MetaData     | NA                 |

Examples:

```python
from unstructured.partition.email import partition_email

elements = partition_email(filename="example-docs/eml/fake-email.eml")

with open("example-docs/eml/fake-email.eml", "r") as f:
    elements = partition_email(file=f)

with open("example-docs/eml/fake-email.eml", "r") as f:
    text = f.read()
elements = partition_email(text=text)

with open("example-docs/eml/fake-email.eml", "r") as f:
    text = f.read()
elements = partition_email(text=text, content_source="text/plain")

with open("example-docs/eml/fake-email.eml", "r") as f:
    text = f.read()
elements = partition_email(text=text, include_headers=True)

```

`partition_email` includes a `max_partition` parameter that indicates the maximum character length for a document element. This parameter only applies if `"text/plain"` is selected as the `content_source`. The default value is `1500`, which roughly corresponds to the average character length for a paragraph. You can disable `max_partition` by setting it to `None`.

You can optionally partition e-mail attachments by setting `process_attachments=True`. The following is an example of what the workflow looks like:

```python
from unstructured.partition.email import partition_email

filename = "example-docs/eml/fake-email-attachment.eml"
elements = partition_email(filename=filename, process_attachments=True)
```

If the content of an email is PGP encrypted, `partition_email` will return an empty list of elements and emit a warning indicated the email is encrypted.

For more information about the `partition_email` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/email.py).

## `partition_epub`

The `partition_epub` function processes e-books in EPUB3 format. The function first converts the document to HTML using `pandocs` and then calls `partition_html`. You’ll need [pandocs](https://pandoc.org/installing.html) installed on your system to use `partition_epub`.

Examples:

```python
from unstructured.partition.epub import partition_epub

elements = partition_epub(filename="example-docs/winter-sports.epub")

```

For more information about the `partition_epub` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/epub.py).

## `partition_html`

The `partition_html` function partitions an HTML document and returns a list of document `Element` objects. `partition_html` can take a filename, file-like object, string, or url as input.

The following three invocations of partition\_html() are essentially equivalent:

```python
from unstructured.partition.html import partition_html

elements = partition_html(filename="example-docs/example-10k.html")

with open("example-docs/example-10k.html", "r") as f:
    elements = partition_html(file=f)

with open("example-docs/example-10k.html", "r") as f:
    text = f.read()
elements = partition_html(text=text)

```

The following illustrates fetching a url and partitioning the response content. The `ssl_verify` kwarg controls whether or not SSL verification is enabled for the HTTP request. By default it is on. Use `ssl_verify=False` to disable SSL verification in the request.

```python
from unstructured.partition.html import partition_html

elements = partition_html(url="https://python.org/")

# you can also provide custom headers:

elements = partition_html(url="https://python.org/",
                          headers={"User-Agent": "YourScriptName/1.0 ..."})

# and turn off SSL verification

elements = partition_html(url="https://python.org/", ssl_verify=False)

```

For more information about the `partition_html` function, you can check the [source code](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/html/partition.py).

## `partition_image`

The `partition_image` function has the same API as `partition_pdf`. The only difference is that `partition_image` does not need to convert a PDF to an image prior to processing. The `partition_image` function supports `.png`, `.heic`, and `.jpg` files.

You can also specify what languages to use for OCR with the `languages` kwarg. For example, use `languages=["eng", "deu"]` to use the English and German language packs. See the [Tesseract documentation](https://github.com/tesseract-ocr/tessdata) for a full list of languages and install instructions.

Examples:

```python
from unstructured.partition.image import partition_image

# Returns a List[Element] present in the pages of the parsed image document
elements = partition_image("example-docs/img/layout-parser-paper-fast.jpg")

# Applies the English and Swedish language pack for ocr
elements = partition_image("example-docs/img/layout-parser-paper-fast.jpg", languages=["eng", "swe"])

```

The `strategy` kwarg controls the method that will be used to process the PDF. The available strategies for images are `"auto"`, `"hi_res"` and `"ocr_only"`.

The `"auto"` strategy will choose the partitioning strategy based on document characteristics and the function kwargs. If `infer_table_structure` is passed, the strategy will be `"hi_res"` because that is the only strategy that currently extracts tables for PDFs. Otherwise, `"auto"` will choose `ocr_only`. `"auto"` is the default strategy.

The `"hi_res"` strategy will identify the layout of the document using `detectron2_onnx`. The advantage of “hi\_res” is that it uses the document layout to gain additional information about document elements. We recommend using this strategy if your use case is highly sensitive to correct classifications for document elements. If `detectron2_onnx` is not available, the `"hi_res"` strategy will fall back to the `"ocr_only"` strategy.

The `"ocr_only"` strategy runs the document through Tesseract for OCR and then runs the raw text through `partition_text`. Currently, `"hi_res"` has difficulty ordering elements for documents with multiple columns. If you have a document with multiple columns that does not have extractable text, we recommend using the `"ocr_only"` strategy.

It is helpful to use `"ocr_only"` instead of `"hi_res"` if `detectron2_onnx` does not detect a text element in the image. To run example below, ensure you have the Korean language pack for Tesseract installed on your system.

```python
from unstructured.partition.image import partition_image

filename = "example-docs/img/english-and-korean.png"
elements = partition_image(filename=filename, languages=["eng", "kor"], strategy="ocr_only")

```

For more information about the `partition_image` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/image.py).

## `partition_md`

The `partition_md` function provides the ability to parse markdown files. The following workflow shows how to use `partition_md`.

Examples:

```python
from unstructured.partition.md import partition_md

elements = partition_md(filename="README.md")

```

For more information about the `partition_md` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/md.py).

## `partition_msg`

The `partition_msg` functions processes `.msg` files, which is a filetype specific to email exports from Microsoft Outlook. Email header information (`bcc_recipient`, `cc_recipient`, `email_message_id`, `sent_from`, `sent_to`, `subject`, etc.) is captured in element metadata.

Examples:

```python
from unstructured.partition.msg import partition_msg

elements = partition_msg(filename="example-docs/fake-email.msg")

```

`partition_msg` includes a `max_partition` parameter that indicates the maximum character length for a document element. This parameter only applies if `"text/plain"` is selected as the `content_source`. The default value is `1500`, which roughly corresponds to the average character length for a paragraph. You can disable `max_partition` by setting it to `None`.

You can optionally partition e-mail attachments by setting `process_attachments=True`. The following is an example of what the workflow looks like:

```python
from unstructured.partition.msg import partition_msg

filename = "example-docs/fake-email-attachment.msg"
elements = partition_msg(filename=filename, process_attachments=True)
```

If the content of an email is PGP encrypted, `partition_msg` will return an empty list of elements and emit a warning indicated the email is encrypted.

For more information about the `partition_msg` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/msg.py).

## `partition_multiple_via_api`

`partition_multiple_via_api` is similar to `partition_via_api`, but allows you to partition multiple documents in a single REST API call. The result has the type `List[List[Element]]`, for example:

```python
[
  [NarrativeText("Narrative!"), Title("Title!")],
  [NarrativeText("Narrative!"), Title("Title!")]
]

```

Examples:

```python
from unstructured.partition.api import partition_multiple_via_api

filenames = ["example-docs/eml/fake-email.eml", "example-docs/fake.docx"]

documents = partition_multiple_via_api(filenames=filenames)

```

```python
from contextlib import ExitStack

from unstructured.partition.api import partition_multiple_via_api

filenames = ["example-docs/eml/fake-email.eml", "example-docs/fake.docx"]
files = [open(filename, "rb") for filename in filenames]

with ExitStack() as stack:
    files = [stack.enter_context(open(filename, "rb")) for filename in filenames]
    documents = partition_multiple_via_api(files=files, metadata_filenames=filenames)

```

For more information about the `partition_multiple_via_api` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/api.py).

## `partition_odt`

The `partition_odt` partitioning function pre-processes Open Office documents saved in the `.odt` format. The function first converts the document to `.docx` using `pandoc` and then processes it using `partition_docx`.

Examples:

```python
from unstructured.partition.odt import partition_odt

elements = partition_odt(filename="example-docs/fake.odt")

```

For more information about the `partition_odt` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/odt.py).

## `partition_org`

The `partition_org` function processes Org Mode (`.org`) documents. The function first converts the document to HTML using `pandoc` and then calls `partition_html`. You’ll need [pandoc](https://pandoc.org/installing.html) installed on your system to use `partition_org`.

Examples:

```python
from unstructured.partition.org import partition_org

elements = partition_org(filename="example-docs/README.org")

```

For more information about the `partition_org` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/org.py).

## `partition_pdf`

The `partition_pdf` function segments a PDF document by using a document image analysis model. If you set `url=None`, the document image analysis model will execute locally. You need to install `unstructured[local-inference]` if you’d like to run inference locally. If you set the URL, `partition_pdf` will make a call to a remote inference server. `partition_pdf` also includes a `token` function that allows you to pass in an authentication token for a remote API call.

You can also specify what languages to use for OCR with the `languages` kwarg. For example, use `languages=["eng", "deu"]` to use the English and German language packs. See the [Tesseract documentation](https://github.com/tesseract-ocr/tessdata) for a full list of languages and install instructions. OCR is only applied if the text is not already available in the PDF document.

Examples:

```python
from unstructured.partition.pdf import partition_pdf

# Returns a List[Element] present in the pages of the parsed pdf document
elements = partition_pdf("example-docs/pdf/layout-parser-paper-fast.pdf")

# Applies the English and Swedish language pack for ocr. OCR is only applied
# if the text is not available in the PDF.
elements = partition_pdf("example-docs/pdf/layout-parser-paper-fast.pdf", languages=["eng", "swe"])

```

The `strategy` kwarg controls the method that will be used to process the PDF. The available strategies for PDFs are `"auto"`, `"hi_res"`, `"ocr_only"`, and `"fast"`.

*   The `"auto"` strategy will choose the partitioning strategy based on document characteristics and the function kwargs. If `infer_table_structure` is passed, the strategy will be `"hi_res"` because that is the only strategy that currently extracts tables for PDFs. Otherwise, `"auto"` will choose `"fast"` if the PDF text is extractable and `"ocr_only"` otherwise. `"auto"` is the default strategy.

*   The `"hi_res"` strategy will identify the layout of the document using `detectron2_onnx`. The advantage of “hi\_res” is that it uses the document layout to gain additional information about document elements. We recommend using this strategy if your use case is highly sensitive to correct classifications for document elements. If `detectron2_onnx` is not available, the `"hi_res"` strategy will fall back to the `"ocr_only"` strategy.

*   The `"ocr_only"` strategy runs the document through Tesseract for OCR and then runs the raw text through `partition_text`. Currently, `"hi_res"` has difficulty ordering elements for documents with multiple columns. If you have a document with multiple columns that does not have extractable text, we recommend using the `"ocr_only"` strategy. `"ocr_only"` falls back to `"fast"` if Tesseract is not available and the document has extractable text.

*   The `"fast"` strategy will extract the text using `pdfminer` and process the raw text with `partition_text`. If the PDF text is not extractable, `partition_pdf` will fall back to `"ocr_only"`. We recommend using the `"fast"` strategy in most cases where the PDF has extractable text.

To extract images and elements as image blocks from a PDF, it is mandatory to set `strategy="hi_res"` when setting `extract_images_in_pdf=True`. With this configuration, detected images are saved in a specified directory or encoded within the file. However, keep in mind that `extract_images_in_pdf` is being phased out in favor of `extract_image_block_types`. This option allows you to specify types of images or elements, like “Image” or “Table”. If some extracted images have content clipped, you can adjust the padding by specifying two environment variables “EXTRACT\_IMAGE\_BLOCK\_CROP\_HORIZONTAL\_PAD” and “EXTRACT\_IMAGE\_BLOCK\_CROP\_VERTICAL\_PAD” (for example, EXTRACT\_IMAGE\_BLOCK\_CROP\_HORIZONTAL\_PAD = 20, EXTRACT\_IMAGE\_BLOCK\_CROP\_VERTICAL\_PAD = 10). For integrating these images directly into web applications or APIs, `extract_image_block_to_payload` can be used to convert them into `base64` format, including details about the image type, currently it’s always `image/jpeg`. Lastly, the `extract_image_block_output_dir` can be used to specify the filesystem path for saving the extracted images when not embedding them in payloads.

Examples:

```python
from unstructured.partition.pdf import partition_pdf

partition_pdf(
    filename="path/to/your/pdf_file.pdf",                  # mandatory
    strategy="hi_res",                                     # mandatory to use ``hi_res`` strategy
    extract_images_in_pdf=True,                            # mandatory to set as ``True``
    extract_image_block_types=["Image", "Table"],          # optional
    extract_image_block_to_payload=False,                  # optional
    extract_image_block_output_dir="path/to/save/images",  # optional - only works when ``extract_image_block_to_payload=False``
    )

```

If a PDF is copy protected, `partition_pdf` can process the document with the `"hi_res"` strategy (which will treat it like an image), but cannot process the document with the `"fast"` strategy. If the user chooses `"fast"` on a copy protected PDF, `partition_pdf` will fall back to the `"hi_res"` strategy. If `detectron2_onnx` is not installed, `partition_pdf` will fail for copy protected PDFs because the document will not be processable by any of the available methods.

Examples:

```python
from unstructured.partition.pdf import partition_pdf

# This will process without issue
elements = partition_pdf("example-docs/pdf/copy-protected.pdf", strategy="hi_res")

# This will output a warning and fall back to hi_res
elements = partition_pdf("example-docs/pdf/copy-protected.pdf", strategy="fast")

```

`partition_pdf` includes a `max_partition` parameter that indicates the maximum character length for a document element. This parameter only applies if the `"ocr_only"` strategy is used for partitioning. The default value is `1500`, which roughly corresponds to the average character length for a paragraph. You can disable `max_partition` by setting it to `None`.

For more information about the `partition_pdf` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/pdf.py).

## `partition_ppt`

The `partition_ppt` partitioning function pre-processes Microsoft PowerPoint documents saved in the `.ppt` format. This partition function uses a combination of the styling information in the document and the structure of the text to determine the type of a text element. The `partition_ppt` can take a filename or file-like object. `partition_ppt` uses `libreoffice` to convert the file to `.pptx` and then calls `partition_pptx`. Ensure you have `libreoffice` installed before using `partition_ppt`.

Examples:

```python
from unstructured.partition.ppt import partition_ppt

elements = partition_ppt(filename="example-docs/fake-power-point.ppt")

```

For more information about the `partition_ppt` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/ppt.py).

## `partition_pptx`

The `partition_pptx` partitioning function pre-processes Microsoft PowerPoint documents saved in the `.pptx` format. This partition function uses a combination of the styling information in the document and the structure of the text to determine the type of a text element. The `partition_pptx` can take a filename or file-like object as input, as shown in the two examples below.

Examples:

```python
from unstructured.partition.pptx import partition_pptx

elements = partition_pptx(filename="example-docs/fake-power-point.pptx")

with open("example-docs/fake-power-point.pptx", "rb") as f:
    elements = partition_pptx(file=f)

```

For more information about the `partition_pptx` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/pptx.py).

## `partition_rst`

The `partition_rst` function processes ReStructured Text (`.rst`) documents. The function first converts the document to HTML using `pandoc` and then calls `partition_html`. You’ll need [pandoc](https://pandoc.org/installing.html) installed on your system to use `partition_rst`.

Examples:

```
from unstructured.partition.rst import partition_rst

elements = partition_rst(filename="example-docs/README.rst")

```

For more information about the `partition_rst` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/rst.py).

## `partition_rtf`

The `partition_rtf` function processes rich text files. The function first converts the document to HTML using `pandocs` and then calls `partition_html`. You’ll need [pandocs](https://pandoc.org/installing.html) installed on your system to use `partition_rtf`.

Examples:

```python
from unstructured.partition.rtf import partition_rtf

elements = partition_rtf(filename="example-docs/fake-doc.rtf")

```

For more information about the `partition_rtf` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/rtf.py).

## `partition_text`

The `partition_text` function partitions text files. The `partition_text` takes a filename, file-like object, and raw text as input and produces `Element` objects as output.

Examples:

```python
from unstructured.partition.text import partition_text

elements = partition_text(filename="example-docs/fake-text.txt")

with open("example-docs/fake-text.txt", "r") as f:
  elements = partition_text(file=f)

with open("example-docs/fake-text.txt", "r") as f:
  text = f.read()
elements = partition_text(text=text)

```

If the text has extra line breaks for formatting purposes, you can group together the broken text using the `paragraph_grouper` kwarg. The `paragraph_grouper` kwarg is a function that accepts a string and returns another string.

Examples:

```python
from unstructured.partition.text import partition_text
from unstructured.cleaners.core import group_broken_paragraphs


text = """The big brown fox
was walking down the lane.

At the end of the lane, the
fox met a bear."""

partition_text(text=text, paragraph_grouper=group_broken_paragraphs)

```

`partition_text` includes a `max_partition` parameter that indicates the maximum character length for a document element. The default value is `1500`, which roughly corresponds to the average character length for a paragraph. You can disable `max_partition` by setting it to `None`.

For more information about the `partition_text` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/text.py).

## `partition_tsv`

The `partition_tsv` function pre-processes TSV files. The output is a single `Table` element. The `text_as_html` attribute in the element metadata will contain an HTML representation of the table.

Examples:

```python
from unstructured.partition.tsv import partition_tsv

elements = partition_tsv(filename="example-docs/stanley-cups.tsv")
print(elements[0].metadata.text_as_html)

```

For more information about the `partition_tsv` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/tsv.py).

## `partition_via_api`

`partition_via_api` allows users to partition documents using the hosted Unstructured API. The API partitions documents using the automatic `partition` function through the [Unstructured SDK Client](https://docs.unstructured.io/api-reference/api-services/accessing-unstructured-api). This is helpful if you’re hosting the API yourself or running it locally through a container. You can pass in your API key using the `api_key` kwarg. You can use the `content_type` kwarg to pass in the MIME type for the file. If you do not explicitly pass it, the MIME type will be inferred.

```python
from unstructured.partition.api import partition_via_api

filename = "example-docs/eml/fake-email.eml"

elements = partition_via_api(filename=filename, api_key="MY_API_KEY", content_type="message/rfc822")

with open(filename, "rb") as f:
  elements = partition_via_api(file=f, metadata_filename=filename, api_key="MY_API_KEY")

```

You can pass additional settings such as `strategy`, `languages` and `encoding` to the API through optional kwargs. These options get added to the request body when the API is called. See [the API documentation](https://api.unstructured.io/general/docs) for a full list of settings supported by the API.

```python
from unstructured.partition.api import partition_via_api

filename = "example-docs/pdf/DA-1p.pdf"

elements = partition_via_api(
  filename=filename, api_key=api_key, strategy="auto"
)

```

If you are using the [Unstructured Serverless API](/api-reference/api-services/saas-api-development-guide), you can use the `api_url` kwarg to point the `partition_via_api` function at your Unstructured Serverless API URL.

```python
import os

from unstructured.partition.api import partition_via_api

filename = "example-docs/eml/fake-email.eml"

elements = partition_via_api(
  filename=filename,
  api_key=os.getenv("UNSTRUCTURED_API_KEY"),
  api_url=os.getenv("UNSTRUCTURED_API_URL")
)

```

If you are self-hosting or running the API locally, you can use the `api_url` kwarg to point the `partition_via_api` function at your self-hosted or local API. See [here](https://github.com/Unstructured-IO/unstructured-api#dizzy-instructions-for-using-the-docker-image) for documentation on how to run the API as a container locally.

```python
import os

from unstructured.partition.api import partition_via_api

filename = "example-docs/eml/fake-email.eml"

elements = partition_via_api(
  filename=filename,
  api_key=os.getenv("UNSTRUCTURED_API_KEY"),
  api_url=os.getenv("UNSTRUCTURED_API_URL")
)

```

For more information about the `partition_via_api` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/api.py).

## `partition_xlsx`

The `partition_xlsx` function pre-processes Microsoft Excel documents. Each sheet in the Excel file will be stored as a `Table` object. The plain text of the sheet will be the `text` attribute of the `Table`. The `text_as_html` attribute in the element metadata will contain an HTML representation of the table.

Examples:

```python
from unstructured.partition.xlsx import partition_xlsx

elements = partition_xlsx(filename="example-docs/stanley-cups.xlsx")
print(elements[0].metadata.text_as_html)

```

For more information about the `partition_xlsx` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/xlsx.py).

## `partition_xml`

The `partition_xml` function processes XML documents. If `xml_keep_tags=False`, the function only returns the text attributes from the tags. You can use `xml_path` in conjunction with `xml_keep_tags=False` to restrict the text extraction to specific tags. If `xml_keep_tags=True`, the function returns tag information in addition to tag text. `xml_keep_tags` is `False` be default.

```python
from unstructured.partition.xml import partition_xml

elements = partition_xml(filename="example-docs/factbook.xml", xml_keep_tags=True)

elements = partition_xml(filename="example-docs/factbook.xml", xml_keep_tags=False)

```

For more information about the `partition_xml` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/xml.py).


# Staging



<Warning>
  The `Staging` brick is being deprecated in favor of the new and more comprehensive `Destination Connectors`. To explore the complete list and usage, please refer to [Destination Connectors documentation](../ingest/destination-connectors/overview).

  Note: We are constantly expanding our collection of destination connectors. If you wish to request a specific Destination Connector, you’re encouraged to submit a Feature Request on the [Unstructured GitHub repository](https://github.com/Unstructured-IO/unstructured/issues/new/choose).
</Warning>

Staging functions in the `unstructured` package help prepare your data for ingestion into downstream systems. A staging function accepts a list of document elements as input and return an appropriately formatted dictionary as output. In the example below, we get our narrative text samples prepared for ingestion into LabelStudio using `the stage_for_label_studio` function. We can take this data and directly upload it into LabelStudio to quickly get started with an NLP labeling task.

```python
import json
from unstructured.staging.label_studio import stage_for_label_studio

output = stage_for_label_studio(narrative_text)
print(json.dumps(output[:2], indent=4))

```

## `convert_to_csv`

Converts outputs to the initial structured data (ISD) format as a CSV string.

Examples:

```python
from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.base import convert_to_csv

elements = [Title(text="Title"), NarrativeText(text="Narrative")]
isd_csv = convert_to_csv(elements)

```

For more information about the `convert_to_csv` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/base.py).

## `convert_to_dataframe`

Converts a list of document `Element` objects to a `pandas` dataframe. The dataframe will have a `text` column with the text from the element and a `type` column indicating the element type, such as `NarrativeText` or `Title`.

Examples:

```python
from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.base import convert_to_dataframe

elements = [Title(text="Title"), NarrativeText(text="Narrative")]
df = convert_to_dataframe(elements)

```

For more information about the `convert_to_dataframe` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/base.py).

## `convert_to_dict`

Converts a list of `Element` objects to a dictionary. This is the default format for representing documents in `unstructured`.

Examples:

```python
from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.base import convert_to_dict

elements = [Title(text="Title"), NarrativeText(text="Narrative")]
isd = convert_to_dict(elements)

```

For more information about the `convert_to_dict` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/base.py).

## `dict_to_elements`

Converts a dictionary of the format produced by `convert_to_dict` back to a list of `Element` objects.

Examples:

```python
from unstructured.staging.base import dict_to_elements

isd = [
  {"text": "My Title", "type": "Title"},
  {"text": "My Narrative", "type": "NarrativeText"}
]

# elements will look like:
# [ Title(text="My Title"), NarrativeText(text="My Narrative")]
elements = dict_to_elements(isd)

```

For more information about the `dict_to_elements` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/base.py).

## `stage_csv_for_prodigy`

Formats outputs in CSV format for use with [Prodigy](https://prodi.gy/docs/api-loaders). After running `stage_csv_for_prodigy`, you can write the results to a CSV file that is ready to be used with Prodigy.

Examples:

```python
from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.prodigy import stage_csv_for_prodigy

elements = [Title(text="Title"), NarrativeText(text="Narrative")]
metadata = [{"type": "title"}, {"source": "news"}]
prodigy_csv_data = stage_csv_for_prodigy(elements, metadata)

# The resulting CSV file is ready to be used with Prodigy
with open("prodigy.csv", "w") as csv_file:
    csv_file.write(prodigy_csv_data)

```

For more information about the `stage_csv_for_prodigy` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/prodigy.py).

## `stage_for_argilla`

Convert a list of `Text` elements to an [Argilla Dataset](https://docs.argilla.io/en/latest/reference/python/python_client.html#python-ref-datasets). The type of Argilla dataset to be generated can be specified with `argilla_task` parameter. Valid values for `argilla_task` are `"text_classification"`, `"token_classification"`, and `"text2text"`. If `"token_classification"` is selected and `tokens` is not included in the optional kwargs, the `nltk` word tokenizer is used by default.

Examples:

```python
import json

from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.argilla import stage_for_argilla

elements = [Title(text="Title"), NarrativeText(text="Narrative")]
metadata = [{"type": "title"}, {"type": "text"}]

argilla_dataset = stage_for_argilla(elements, "text_classification", metadata=metadata)

```

For more information about the `stage_for_argilla` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/argilla.py).

## `stage_for_baseplate`-

The `stage_for_baseplate` staging function prepares a list of `Element` objects for ingestion into [Baseplate](https://docs.baseplate.ai/introduction), an LLM backend with a spreadsheet interface. After running the `stage_for_baseplate` function, you can use the [Baseplate API](https://docs.baseplate.ai/api-reference/documents/upsert-data-rows) to upload the documents to Baseplate. The following example code shows how to use the `stage_for_baseplate` function.

```python
from unstructured.documents.elements import ElementMetadata, NarrativeText, Title
from unstructured.staging.baseplate import stage_for_baseplate

metadata = ElementMetadata(filename="fox.epub")

elements = [
  Title("A Wonderful Story About A Fox", metadata=metadata),
  NarrativeText(
    "A fox ran into the chicken coop and the chickens flew off!",
    metadata=metadata,
  ),
]

rows = stage_for_baseplate(elements)

```

The output will look like:

```json
{
      "rows": [
          {
              "data": {
                  "element_id": "ad270eefd1cc68d15f4d3e51666d4dc8",
                  "text": "A Wonderful Story About A Fox",
                  "type": "Title",
              },
              "metadata": {"filename": "fox.epub"},
          },
          {
              "data": {
                  "element_id": "8275769fdd1804f9f2b55ad3c9b0ef1b",
                  "text": "A fox ran into the chicken coop and the chickens flew off!",
                  "type": "NarrativeText",
              },
              "metadata": {"filename": "fox.epub"},
          },
      ],
  }

```

For more information about the `stage_for_baseplate` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/baseplate.py).

## `stage_for_datasaur`

Formats a list of `Text` elements as input to token based tasks in Datasaur.

Example:

```python
from unstructured.documents.elements import Text
from unstructured.staging.datasaur import stage_for_datasaur

elements  = [Text("Text1"),Text("Text2")]
datasaur_data = stage_for_datasaur(elements)

```

The output is a list of dictionaries, each one with two keys: “text” with the content of the element and “entities” with an empty list.

You can also specify entities in the `stage_for_datasaur` function. Entities you specify in the input will be included in the entities key in the output. The list of entities is a list of dictionaries and must have all of the keys in the example below. The list of entities must be the same length as the list of elements. Use an empty list for any elements that do not have any entities.

Example:

```python
from unstructured.documents.elements import Text
from unstructured.staging.datasaur import stage_for_datasaur

elements  = [Text("Hi my name is Matt.")]
entities = [[{"text": "Matt", "type": "PER", "start_idx": 11, "end_idx": 15}]]
datasaur_data = stage_for_datasaur(elements, entities)

```

For more information about the `stage_for_datasaur` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/datasaur.py).

## `stage_for_label_box`

Formats outputs for use with [LabelBox](https://docs.labelbox.com/docs/overview). LabelBox accepts cloud-hosted data and does not support importing text directly. The `stage_for_label_box` does the following:

*   Stages the data files in the `output_directory` specified in function arguments to be uploaded to a cloud storage service.

*   Returns a config of type `List[Dict[str, Any]]` that can be written to a `json` file and imported into LabelBox.

**Note:** `stage_for_label_box` does not upload the data to remote storage such as S3. Users can upload the data to S3 using `aws s3 sync ${output_directory} ${url_prefix}` after running the `stage_for_label_box` staging function.

Examples:

The following example demonstrates generating a `config.json` file that can be used with LabelBox and uploading the staged data files to an S3 bucket.

```python
import os
import json

from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.label_box import stage_for_label_box

# The S3 Bucket name where data files should be uploaded.
S3_BUCKET_NAME = "labelbox-staging-bucket"

# The S3 key prefix (I.e. directory) where data files should be stored.
S3_BUCKET_KEY_PREFIX = "data/"

# The URL prefix where the data files will be accessed.
S3_URL_PREFIX = f"https://{S3_BUCKET_NAME}.s3.amazonaws.com/{S3_BUCKET_KEY_PREFIX}"

# The local output directory where the data files will be staged for uploading to a Cloud Storage service.
LOCAL_OUTPUT_DIRECTORY = "/tmp/labelbox-staging"

elements = [Title(text="Title"), NarrativeText(text="Narrative")]

labelbox_config = stage_for_label_box(
    elements,
    output_directory=LOCAL_OUTPUT_DIRECTORY,
    url_prefix=S3_URL_PREFIX,
    external_ids=["id1", "id2"],
    attachments=[[{"type": "RAW_TEXT", "value": "Title description"}], [{"type": "RAW_TEXT", "value": "Narrative Description"}]],
    create_directory=True,
)

# The resulting JSON config file is ready to be used with LabelBox.
with open("config.json", "w+") as labelbox_config_file:
    json.dump(labelbox_config, labelbox_config_file, indent=4)


# Upload staged data files to S3 from local output directory.
def upload_staged_files():
    from s3fs import S3FileSystem
    fs = S3FileSystem()
    for filename in os.listdir(LOCAL_OUTPUT_DIRECTORY):
        filepath = os.path.join(LOCAL_OUTPUT_DIRECTORY, filename)
        upload_key = os.path.join(S3_BUCKET_KEY_PREFIX, filename)
        fs.put_file(lpath=filepath, rpath=os.path.join(S3_BUCKET_NAME, upload_key))

upload_staged_files()

```

For more information about the `stage_for_label_box` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/label_box.py).

## `stage_for_label_studio`

Formats outputs for upload to LabelStudio. After running `stage_for_label_studio`, you can write the results to a JSON folder that is ready to be included in a new LabelStudio project.

Examples:

```python
import json

from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.label_studio import stage_for_label_studio

elements = [Title(text="Title"), NarrativeText(text="Narrative")]
label_studio_data = stage_for_label_studio(elements, text_field="my_text", id_field="my_id")

# The resulting JSON file is ready to be uploaded to LabelStudio
with open("label_studio.json", "w") as f:
    json.dump(label_studio_data, f, indent=4)

```

You can also include pre-annotations and predictions as part of your LabelStudio upload.

The `annotations` kwarg is a list of lists. If `annotations` is specified, there must be a list of annotations for each element in the `elements` list. If an element does not have any annotations, use an empty list. The following shows an example of how to upload annotations for the “Text Classification” task in LabelStudio:

```python
import json

from unstructured.documents.elements import NarrativeText
from unstructured.staging.label_studio import (
    stage_for_label_studio,
    LabelStudioAnnotation,
    LabelStudioResult,
)



elements = [NarrativeText(text="Narrative")]
annotations = [[
  LabelStudioAnnotation(
      result=[
          LabelStudioResult(
              type="choices",
              value={"choices": ["Positive"]},
              from_name="sentiment",
              to_name="text",
          )
      ]
  )
]]
label_studio_data = stage_for_label_studio(
    elements,
    annotations=annotations,
    text_field="my_text",
    id_field="my_id"
)

# The resulting JSON file is ready to be uploaded to LabelStudio
# with annotations included
with open("label_studio.json", "w") as f:
    json.dump(label_studio_data, f, indent=4)

```

Similar to annotations, the `predictions` kwarg is also a list of lists. A `prediction` is an annotation with the addition of a `score` value. If `predictions` is specified, there must be a list of predictions for each element in the `elements` list. If an element does not have any predictions, use an empty list. The following shows an example of how to upload predictions for the “Text Classification” task in LabelStudio:

```python
import json

from unstructured.documents.elements import NarrativeText
from unstructured.staging.label_studio import (
    stage_for_label_studio,
    LabelStudioPrediction,
    LabelStudioResult,
)



elements = [NarrativeText(text="Narrative")]
predictions = [[
  LabelStudioPrediction(
      result=[
          LabelStudioResult(
              type="choices",
              value={"choices": ["Positive"]},
              from_name="sentiment",
              to_name="text",
          )
      ],
      score=0.68
  )
]]
label_studio_data = stage_for_label_studio(
    elements,
    predictions=predictions,
    text_field="my_text",
    id_field="my_id"
)

# The resulting JSON file is ready to be uploaded to LabelStudio
# with annotations included
with open("label_studio.json", "w") as f:
    json.dump(label_studio_data, f, indent=4)

```

The following shows an example of how to upload annotations for the “Named Entity Recognition” task in LabelStudio:

```python
import json

from unstructured.documents.elements import NarrativeText
from unstructured.staging.label_studio import (
    stage_for_label_studio,
    LabelStudioAnnotation,
    LabelStudioResult,
)



elements = [NarrativeText(text="Narrative")]
annotations = [[
  LabelStudioAnnotation(
      result=[
          LabelStudioResult(
              type="labels",
              value={"start": 0, "end": 9, "text": "Narrative", "labels": ["MISC"]},
              from_name="label",
              to_name="text",
          )
      ]
  )
]]
label_studio_data = stage_for_label_studio(
    elements,
    annotations=annotations,
    text_field="my_text",
    id_field="my_id"
)

# The resulting JSON file is ready to be uploaded to LabelStudio
# with annotations included
with open("label_studio.json", "w") as f:
    json.dump(label_studio_data, f, indent=4)

```

See the [LabelStudio docs](https://labelstud.io/tags/labels.html) for a full list of options for labels and annotations.

For more information about the `stage_for_label_studio` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/label_studio.py).

## `stage_for_prodigy`

Formats outputs in JSON format for use with [Prodigy](https://prodi.gy/docs/api-loaders). After running `stage_for_prodigy`, you can write the results to a JSON file that is ready to be used with Prodigy.

Examples:

```python
import json

from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.prodigy import stage_for_prodigy

elements = [Title(text="Title"), NarrativeText(text="Narrative")]
metadata = [{"type": "title"}, {"type": "text"}]
prodigy_data = stage_for_prodigy(elements, metadata)

# The resulting JSON file is ready to be used with Prodigy
with open("prodigy.json", "w") as f:
    json.dump(prodigy_data, f, indent=4)

```

**Note**: Prodigy recommends `.jsonl` format for feeding data to API loaders. After running `stage_for_prodigy`, you can use the `save_as_jsonl` utility function to save the formatted data to a `.jsonl` file that is ready to be used with Prodigy.

```python
from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.prodigy import stage_for_prodigy
from unstructured.utils import save_as_jsonl

elements = [Title(text="Title"), NarrativeText(text="Narrative")]
metadata = [{"type": "title"}, {"type": "text"}]
prodigy_data = stage_for_prodigy(elements, metadata)

# The resulting jsonl file is ready to be used with Prodigy.
save_as_jsonl(prodigy_data, "prodigy.jsonl")

```

For more information about the `stage_for_prodigy` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/prodigy.py).

## `stage_for_transformers`

Prepares `Text` elements for processing in `transformers` pipelines by splitting the elements into chunks that fit into the model’s attention window.

Examples:

```python
from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline

from unstructured.documents.elements import NarrativeText
from unstructured.staging.huggingface import stage_for_transformers

model_name = "hf-internal-testing/tiny-bert-for-token-classification"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

nlp = pipeline("ner", model=model, tokenizer=tokenizer)

text = """From frost advisories this morning to a strong cold front expected later this week, the chance of fall showing up is real.

There's a refreshing crispness to the air, and it looks to get only more pronounced as the week goes on.

Frost advisories were in place this morning across portions of the Appalachians and coastal Maine as temperatures dropped into the 30s.

Temperatures this morning were in the 40s as far south as the Florida Panhandle.

And Maine even had a few reports of their first snow of the season Sunday. More cities could see their first snow later this week.

Yes, hello fall!

As temperatures moderate during the next few days, much of the east will stay right around seasonal norms, but the next blast of cold air will be strong and come with the potential for hazardous conditions.

"A more active fall weather pattern is expected to evolve by the end of this week and continuing into the weekend as a couple of cold fronts move across the central and eastern states," the Weather Prediction Center said.

The potent cold front will come in from Canada with a punch of chilly air, heavy rain and strong wind.

The Weather Prediction Center has a slight risk of excessive rainfall for much of the Northeast and New England on Thursday, including places like New York City, Buffalo and Burlington, so we will have to look out for flash flooding in these areas.

"More impactful weather continues to look likely with confidence growing that our region will experience the first real fall-like system with gusty to strong winds and a period of moderate to heavy rain along and ahead of a cold front passage," the National Weather Service office in Burlington wrote.

The potential for very heavy rain could accompany the front, bringing up to two inches of rain for much of the area, and isolated locations could see even more.

"Ensembles [forecast models] show median rainfall totals by Wednesday night around a half inch, with a potential for some spots to see around one inch, our first substantial rainfall in at least a couple of weeks," the weather service office in Grand Rapids noted, adding, "It may also get cold enough for some snow to mix in Thursday night to Friday morning, especially in the higher terrain north of Grand Rapids toward Cadillac."

There is also a chance for very strong winds to accompany the system.

The weather service is forecasting winds of 30-40 mph ahead of the cold front, which could cause some tree limbs to fall and sporadic power outages.

Behind the front, temperatures will fall.

"East Coast, with highs about 5-15 degrees below average to close out the workweek and going into next weekend, with highs only in the 40s and 50s from the Great Lakes to the Northeast on most days," the Weather Prediction Center explained.

By the weekend, a second cold front will drop down from Canada and bring a reinforcing shot of chilly air across the eastern half of the country."""

elements = stage_for_transformers([NarrativeText(text=text)], tokenizer)

```

The following optional keyword arguments can be specified in `stage_for_transformers`:

<Note>
  *   `buffer`: Indicates the number of tokens to leave as a buffer for the attention window. This is to account for special tokens like `[CLS]` that can appear at the beginning or end of an input sequence.

  *   `max_input_size`: The size of the attention window for the model. If not specified, the default is the `model_max_length` attribute on the tokenizer object.

  *   `split_function`: The function used to split the text into chunks to consider for adding to the attention window. Splits on spaces be default.

  *   `chunk_separator`: The string used to concat adjacent chunks when reconstructing the text. Uses spaces by default.

  If you need to operate on text directly instead of `unstructured` `Text` objects, use the `chunk_by_attention_window` helper function. Simply modify the example above to include the following:

  ```python
  from unstructured.staging.huggingface import chunk_by_attention_window

  chunks = chunk_by_attention_window(text, tokenizer)

  results = [nlp(chunk) for chunk in chunks]

  ```
</Note>

For more information about the `stage_for_transformers` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/huggingface.py).

## `stage_for_weaviate`

The `stage_for_weaviate` staging function prepares a list of `Element` objects for ingestion into the [Weaviate](https://weaviate.io/) vector database. You can create a schema in Weaviate for the unstructured outputs using the following workflow:

```python
from unstructured.staging.weaviate import create_unstructured_weaviate_class

import weaviate

# Change `class_name` if you want the class for unstructured documents in Weaviate
# to have a different name
unstructured_class = create_unstructured_weaviate_class(class_name="UnstructuredDocument")
schema = {"classes": [unstructured_class]}

client = weaviate.Client("http://localhost:8080")
client.schema.create(schema)

```

Once the schema is created, you can batch upload documents to Weaviate using the following workflow. See the [Weaviate documentation](https://weaviate.io/developers/weaviate) for more details on options for uploading data and querying data once it has been uploaded.

```python
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.weaviate import stage_for_weaviate

import weaviate
from weaviate.util import generate_uuid5


filename = "example-docs/pdf/layout-parser-paper-fast.pdf"
elements = partition_pdf(filename=filename, strategy="fast")
data_objects = stage_for_weaviate(elements)

client = weaviate.Client("http://localhost:8080")

with client.batch(batch_size=10) as batch:
    for data_object in tqdm.tqdm(data_objects):
        batch.add_data_object(
            data_object,
            unstructured_class_name,
            uuid=generate_uuid5(data_object),
        )

```

For more information about the `stage_for_weaviate` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/weaviate.py).


# Set the OCR agent



## Task

You want to specify the type of agent that you use when performing optical character recognition (OCR) on files, such as images and PDFs.

<Note>This task works only with the Unstructured open source library.</Note>

## Approach

Set the environment variable named `OCR_AGENT` to one of the following supported values:

*   `unstructured.partition.utils.ocr_models.tesseract_ocr.OCRAgentTesseract` to use Tesseract OCR. This is the default if not otherwise specified.
*   `unstructured.partition.utils.ocr_models.paddle_ocr.OCRAgentPaddle` to use Paddle OCR.
*   `unstructured.partition.utils.ocr_models.google_vision_ocr.OCRAgentGoogleVision` to use Google Cloud Vision OCR.

Also, be sure to install the corresponding OCR agent and its dependencies, if you have not already done so:

*   For Tesseract OCR, [see the dependency list](https://github.com/Unstructured-IO/unstructured/blob/main/requirements/extra-pdf-image.in).
*   For Paddle OCR, [see the dependency list](https://github.com/Unstructured-IO/unstructured/blob/main/requirements/extra-paddleocr.in).
*   For Google Cloud Vision OCR, [see the dependency list](https://github.com/Unstructured-IO/unstructured/blob/main/requirements/extra-pdf-image.in).

## Example code

This example uses a PNG file with an embedded combination of English and Korean text. This example uses Tesseract OCR.

Language codes will differ depending on the OCR agent you use:

*   For Tesseract OCR, [see the language codes list](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/lang.py).
*   For Paddle OCR, [see the language codes list](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/lang.py) and [language names list](https://github.com/PaddlePaddle/PaddleOCR/blob/main/doc/doc_en/multi_languages_en.md#language_abbreviations).
*   For Google Cloud Vision OCR, [see the language codes list](https://cloud.google.com/vision/docs/languages).

```python Python
import json

from unstructured.partition.image import partition_image

# Source: https://github.com/Unstructured-IO/unstructured-ingest/blob/main/example-docs/img/english-and-korean.png
# Path to the local file to process, relative to this .py file.
filename = "local-ingest-png/english-and-korean.png"

elements = partition_image(
  filename=filename,
  strategy="ocr_only",
  languages=["eng", "kor"] # Language codes differ by the OCR agent used.
)

# Convert the list of returned elements into a list of dictionaries for printing or saving.
element_dicts = [element.to_dict() for element in elements]

# Print the list.
print(json.dumps(element_dicts, indent=2))

# Or, save the list locally:
#
# file = "local-ingest-output/english-and-korean.json"
#
# with open(file, "w") as file:
#     json.dump(element_dicts, file, indent=2)
```


# Astra DB



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in an Astra DB account.

You will need:

The Astra DB connector prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/kupQrnCQjPw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An Astra account. [Create or sign in to an Astra account](https://astra.datastax.com/).
*   A database in the Astra account. [Create a database in an account](https://docs.datastax.com/en/astra-db-classic/databases/manage-create.html).
*   An application token for the database. [Create a database application token](https://docs.datastax.com/en/astra-db-serverless/administration/manage-application-tokens.html).
*   A namespace in the database. [Create a namespace in a database](https://docs.datastax.com/en/astra-db-serverless/databases/manage-namespaces.html#create-namespace).
*   A collection in the namespace. [Create a collection in a namespace](https://docs.datastax.com/en/astra-db-serverless/databases/manage-collections.html#create-collection).

The Astra DB connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[astradb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `ASTRA_DB_API_ENDPOINT` - The API endpoint for the Astra DB database, represented by `--api-endpoint` (CLI) or `api_endpoint` (Python). To get the endpoint, see the **Database Details > API Endpoint** value on your database's **Overview** tab.
*   `ASTRA_DB_APPLICATION_TOKEN` - The database application token value for the database, represented by `--token` (CLI) or `token` (Python). To get the token, see the **Database Details > Application Tokens** box on your database's **Overview** tab.
*   `ASTRA_DB_KEYSPACE` - The name of the keyspace for the database, represented by `--keyspace` (CLI) or `keyspace` (Python).
*   `ASTRA_DB_COLLECTION` - The name of the collection for the keyspace, represented by `--collection-name` (CLI) or `collection_name` (Python).
*   `ASTRA_DB_EMBEDDING_DIMENSIONS` - The number of dimensions in the collection, represented by `--embedding-dimension` (CLI) or `embedding_dimension` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    astradb \
      --api-endpoint $ASTRA_DB_API_ENDPOINT \
      --token $ASTRA_DB_APPLICATION_TOKEN \
      --keyspace $ASTRA_DB_KEYSPACE \
      --collection-name $ASTRA_DB_COLLECTION \
      --embedding-dimension $ASTRA_DB_EMBEDDING_DIMENSIONS
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.astradb import (
      AstraDBConnectionConfig,
      AstraDBAccessConfig,
      AstraDBUploadStagerConfig,
      AstraDBUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=AstraDBConnectionConfig(
              access_config=AstraDBAccessConfig(
                  api_endpoint=os.getenv("ASTRA_DB_API_ENDPOINT"),
                  token=os.getenv("ASTRA_DB_APPLICATION_TOKEN")
              )
          ),
          stager_config=AstraDBUploadStagerConfig(),
          uploader_config=AstraDBUploaderConfig(
              keyspace=os.getenv("ASTRA_DB_KEYSPACE"),
              collection_name=os.getenv("ASTRA_DB_COLLECTION"),
              embedding_dimension=os.getenv("ASTRA_DB_EMBEDDING_DIMENSIONS")
          )
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.astradb import (
      AstraDBAccessConfig,
      AstraDBWriteConfig,
      SimpleAstraDBConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.astradb import (
      AstraDBWriter,
  )
  from unstructured_ingest.runner.writers.base_writer import Writer


  def get_writer() -> Writer:
      return AstraDBWriter(
          connector_config=SimpleAstraDBConfig(
              access_config=AstraDBAccessConfig(
                  api_endpoint=os.getenv("ASTRA_DB_API_ENDPOINT"),
                  token=os.getenv("ASTRA_DB_APPLICATION_TOKEN"),
              ),
              keyspace=os.getenv("ASTRA_DB_KEYSPACE"),
              collection_name=os.getenv("ASTRA_DB_COLLECTION"),
              embedding_dimension=os.getenv("ASTRA_DB_EMBEDDING_DIMENSIONS"),
          ),
          write_config=AstraWriteConfig(batch_size=80),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Azure



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in an Azure Storage account.

You will need:

The Azure Storage account prerequisites.

The following video shows how to fulfill the minimum set of Azure Storage account prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/Vl3KCphlh9Y" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

<Note>
  If you are generating an SAS token as shown in the preceding video, be sure to set the following permissions:

  *   **Read** and **List** for reading from the container only.
  *   **Write** and **List** for writing to the container only.
  *   **Read**, **Write**, and **List** for both reading from and writing to the container.
</Note>

Here are some more details about these prerequisites:

*   An Azure account. To create one, [learn how](https://azure.microsoft.com/pricing/purchase-options/azure-account).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/2bQ6WiJ1ncA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An Azure Storage account, and a container within that account. [Create a storage account](https://learn.microsoft.com/azure/storage/common/storage-account-create). [Create a container](https://learn.microsoft.com/azure/storage/blobs/blob-containers-portal).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/AhuNgBafmUo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    <iframe width="560" height="315" src="https://www.youtube.com/embed/xmndjYnGvcs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Azure Storage remote URL, using the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

    For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
    Azure Storage remote URL would be `az://my-container/my-folder/`.

*   An SAS token (recommended), access key, or connection string for the Azure Storage account.  [Create an SAS token (recommended)](https://learn.microsoft.com/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens). [Get an access key](https://learn.microsoft.com/azure/storage/common/storage-account-keys-manage#view-account-access-keys). [Get a connection string](https://learn.microsoft.com/azure/storage/common/storage-configure-connection-string#configure-a-connection-string-for-an-azure-storage-account).

    Create an SAS token (recommended):

    <iframe width="560" height="315" src="https://www.youtube.com/embed/X6cmJ2IbVzo?start=240&end=370" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    Get an access key or connection string:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/muMmcwVfFqs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The Azure Storage account connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[azure]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `AZURE_STORAGE_REMOTE_URL` - The Azure Storage remote URL, represented by `--remote-url` (CLI) or `remote_url` (Python).

    The remote URL takes the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

    For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
    Azure Storage remote URL would be `az://my-container/my-folder/`.

*   `AZURE_STORAGE_ACCOUNT_NAME` - The name of the Azure Storage account, represented by `--account-name` (CLI) or `account_name` (Python).

*   `AZURE_STORAGE_ACCOUNT_KEY`, `AZURE_STORAGE_CONNECTION_STRING`, or `AZURE_STORAGE_SAS_TOKEN` - The name of the key, connection string, or SAS token for the Azure Storage account, depending on your security configuration, represented by `--account-key` (CLI) or `account_key` (Python), `--connection-string` (CLI) or `connection_string` (Python), and `--sas_token` (CLI) or `sas_token` (Python), respectively.

Now call the Unstructured Ingest CLI or Unstructured Ingest Python.  The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    azure \
      --remote-url $AZURE_STORAGE_REMOTE_URL \
      --account-name $AZURE_STORAGE_ACCOUNT_NAME \
      --account-key $AZURE_STORAGE_ACCOUNT_KEY
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.azure import (
      AzureConnectionConfig,
      AzureAccessConfig,
      AzureUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=AzureConnectionConfig(
              access_config=AzureAccessConfig(
                  account_name=os.getenv("AZURE_STORAGE_ACCOUNT_NAME"),
                  account_key=os.getenv("AZURE_STORAGE_ACCOUNT_KEY")
              )
          ),
          uploader_config=AzureUploaderConfig(remote_url=os.getenv("AZURE_STORAGE_REMOTE_URL"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.azure import (
      AzureAccessConfig,
      AzureWriteConfig,
      SimpleAzureBlobStorageConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.fsspec.azure import (
      AzureWriter,
  )

  def get_writer() -> Writer:
      return AzureWriter(
          connector_config=SimpleAzureBlobStorageConfig(
              access_config=AzureAccessConfig(account_name=os.getenv("AZURE_STORAGE_ACCOUNT_NAME")),
              remote_url=os.getenv("AZURE_STORAGE_REMOTE_URL"),
          ),
          write_config=AzureWriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res"
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Azure AI Search



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in an Azure AI Search account.

You will need:

The Azure AI Search prerequisites:

The following video shows how to fulfill the minimum set of Azure AI Search prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/6ZjU5OupWE8" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

Here are some more details about these prerequisites:

*   The endpoint and API key for Azure AI Search. [Create an endpoint and API key](https://learn.microsoft.com/azure/search/search-create-service-portal).
*   The name of the index in Azure AI Search. [Create an index](https://learn.microsoft.com/rest/api/searchservice/create-index).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/WY8h8Gtyo7o" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    The Azure AI Search index that you use must have an index schema that is compatible with the schema of the documents
    that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following index schema example for your own needs:

    ```json
    {
      "@odata.context": "https://ingest-test-azure-ai-search.search.windows.net/$metadata#indexes/$entity",
      "@odata.etag": "\"0x8DCED5D96393CA9\"",
      "name": "<my-index-name>",
      "defaultScoringProfile": null,
      "fields": [
        {
          "name": "id",
          "type": "Edm.String",
          "searchable": true,
          "filterable": true,
          "retrievable": true,
          "stored": true,
          "sortable": true,
          "facetable": true,
          "key": true,
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "analyzer": null,
          "normalizer": null,
          "dimensions": null,
          "vectorSearchProfile": null,
          "vectorEncoding": null,
          "synonymMaps": []
        },
        {
          "name": "element_id",
          "type": "Edm.String",
          "searchable": true,
          "filterable": true,
          "retrievable": true,
          "stored": true,
          "sortable": true,
          "facetable": true,
          "key": false,
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "analyzer": null,
          "normalizer": null,
          "dimensions": null,
          "vectorSearchProfile": null,
          "vectorEncoding": null,
          "synonymMaps": []
        },
        {
          "name": "text",
          "type": "Edm.String",
          "searchable": true,
          "filterable": true,
          "retrievable": true,
          "stored": true,
          "sortable": true,
          "facetable": true,
          "key": false,
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "analyzer": null,
          "normalizer": null,
          "dimensions": null,
          "vectorSearchProfile": null,
          "vectorEncoding": null,
          "synonymMaps": []
        },
        {
          "name": "embeddings",
          "type": "Collection(Edm.Single)",
          "searchable": true,
          "filterable": false,
          "retrievable": true,
          "stored": true,
          "sortable": false,
          "facetable": false,
          "key": false,
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "analyzer": null,
          "normalizer": null,
          "dimensions": 3072,
          "vectorSearchProfile": "embeddings-config-profile",
          "vectorEncoding": null,
          "synonymMaps": []
        },
        {
          "name": "type",
          "type": "Edm.String",
          "searchable": true,
          "filterable": true,
          "retrievable": true,
          "stored": true,
          "sortable": true,
          "facetable": true,
          "key": false,
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "analyzer": null,
          "normalizer": null,
          "dimensions": null,
          "vectorSearchProfile": null,
          "vectorEncoding": null,
          "synonymMaps": []
        },
        {
          "name": "metadata",
          "type": "Edm.ComplexType",
          "fields": [
            {
              "name": "category_depth",
              "type": "Edm.Int32",
              "searchable": false,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "parent_id",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "attached_to_filename",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "filetype",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "last_modified",
              "type": "Edm.DateTimeOffset",
              "searchable": false,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "is_continuation",
              "type": "Edm.Boolean",
              "searchable": false,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "file_directory",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "filename",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "data_source",
              "type": "Edm.ComplexType",
              "fields": [
                {
                  "name": "url",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "version",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "date_created",
                  "type": "Edm.DateTimeOffset",
                  "searchable": false,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "date_modified",
                  "type": "Edm.DateTimeOffset",
                  "searchable": false,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "date_processed",
                  "type": "Edm.DateTimeOffset",
                  "searchable": false,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "permissions_data",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "record_locator",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                }
              ]
            },
            {
              "name": "coordinates",
              "type": "Edm.ComplexType",
              "fields": [
                {
                  "name": "system",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "layout_width",
                  "type": "Edm.Double",
                  "searchable": false,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "layout_height",
                  "type": "Edm.Double",
                  "searchable": false,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "points",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                }
              ]
            },
            {
              "name": "languages",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "page_number",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "orig_elements",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "links",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "page_name",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "url",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "link_urls",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "link_texts",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "sent_from",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "sent_to",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "subject",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "section",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "header_footer_type",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "emphasized_text_contents",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "emphasized_text_tags",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "text_as_html",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "regex_metadata",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "detection_class_prob",
              "type": "Edm.Double",
              "searchable": false,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            }
          ]
        }
      ],
      "scoringProfiles": [],
      "corsOptions": null,
      "suggesters": [],
      "analyzers": [],
      "normalizers": [],
      "tokenizers": [],
      "tokenFilters": [],
      "charFilters": [],
      "encryptionKey": null,
      "similarity": {
        "@odata.type": "#Microsoft.Azure.Search.BM25Similarity",
        "k1": null,
        "b": null
      },
      "semantic": null,
      "vectorSearch": {
        "algorithms": [
          {
            "name": "embeddings-config",
            "kind": "hnsw",
            "hnswParameters": {
              "metric": "cosine",
              "m": 4,
              "efConstruction": 400,
              "efSearch": 500
            },
            "exhaustiveKnnParameters": null
          }
        ],
        "profiles": [
          {
            "name": "embeddings-config-profile",
            "algorithm": "embeddings-config",
            "vectorizer": null,
            "compression": null
          }
        ],
        "vectorizers": [],
        "compressions": []
      }
    }
    ```

    See also:

    *   [Search indexes in Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-an-index)
    *   [Schema of a search index](https://learn.microsoft.com/azure/search/search-what-is-an-index#schema-of-a-search-index)
    *   [Example index schema](https://learn.microsoft.com/rest/api/searchservice/create-index#examples)
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

The Azure AI Search connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[azure-ai-search]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `AZURE_SEARCH_ENDPOINT` - The endpoint URL for Azure AI Search, represented by `--endpoint` (CLI) or `endpoint` (Python).
*   `AZURE_SEARCH_API_KEY` - The API key for Azure AI Search, represented by `--key` (CLI) or `key` (Python).
*   `AZURE_SEARCH_INDEX` - The name of the index for Azure AI Search, represented by `--index` (CLI) or `index` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    azure-ai-search \
      --key $AZURE_SEARCH_API_KEY \
      --endpoint $AZURE_SEARCH_ENDPOINT \
      --index $AZURE_SEARCH_INDEX
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.azure_ai_search import (
      AzureAISearchAccessConfig,
      AzureAISearchConnectionConfig,
      AzureAISearchUploadStagerConfig,
      AzureAISearchUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=AzureAISearchConnectionConfig(
              access_config=AzureAISearchAccessConfig(
                  key=os.getenv("AZURE_SEARCH_API_KEY")
              ),
              endpoint=os.getenv("AZURE_SEARCH_ENDPOINT"),
              index=os.getenv("AZURE_SEARCH_INDEX")
          ),
          stager_config=AzureAISearchUploadStagerConfig(),
          uploader_config=AzureAISearchUploaderConfig()
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.azure_ai_search import (
      AzureAISearchAccessConfig,
      AzureAISearchWriteConfig,
      SimpleAzureAISearchStorageConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.azure_ai_search import (
      AzureAISearchWriter,
  )
  from unstructured_ingest.runner.writers.base_writer import Writer


  def get_writer() -> Writer:
      return AzureAISearchWriter(
          connector_config=SimpleAzureAISearchStorageConfig(
              access_config=AzureAISearchAccessConfig(key=os.getenv("AZURE_SEARCH_API_KEY")),
              endpoint=os.getenv("AZURE_SEARCH_ENDPOINT"),
          ),
          write_config=AzureAISearchWriteConfig(index=os.getenv("AZURE_SEARCH_INDEX")),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR")",
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Box



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a Box account.

You will need:

The Box prerequisites:

1.  Access to the [Developer Console](https://app.box.com/developers/console) from your [Box enterprise account](https://account.box.com/signup/enterprise-plan) or [Box developer account](https://account.box.com/signup/developer).

2.  A Box Custom App in your Box account, set up to use **Server Authentication (with JWT)**. See [Setup with JWT](https://developer.box.com/guides/authentication/jwt/jwt-setup/).

3.  The appropriate application scopes and advanced features set up for the Box Custom App, as follows:

    *   In the Box Custom App, on the **Configuration** tab, under **Application Scopes**, check the box titled **Write all files and folders stored in Box**.
    *   In the Box Custom App, on the **Configuration** tab, under **Advanced Features**, check the box titled **Make API calls using the as-user header**.
    *   Click **Save Changes** before continuing.

4.  Authorization by a Box Admin in your Box account for the Box Custom App. See [App Authorization](https://developer.box.com/guides/authentication/jwt/jwt-setup/#app-authorization).

5.  Access by your Box account's source or target [folder](https://app.box.com/folder/0) to your Box Custom App, as follows:

    *   In the Box Custom App, on the **General Settings** tab, copy the **Service Account ID** (which takes the form `AutomationUser_<your-app-service-id>_<a-random-string@boxdevedition.com`).
    *   **Share** your Box account's target folder with the copied service account's email address as a **Co-owner** or **Editor**.
    *   Note the remote URL to the target folder, which takes the format `box://<path/to/folder/in/account>`.

6.  The private key configuration JSON file for the Box Custom App. To download this file, in the Box Custom App, on the **Configuration** tab, under **Add and Manage Public Keys**, click **Generate a Public/Private Keypair**. Store the downloaded private key configuration JSON file in a secure location. Do not share it with others.

The Box connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[box]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `BOX_APP_CONFIG_PATH` - The local path to the downloaded private key configuration JSON file for the Box Custom App, represented by `--box-app-config` (CLI) or `box_app_config` (Python).
*   `BOX_REMOTE_URL` - The remote URL to the target folder, represented by `--remote-url` (CLI) or `remote_url` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    box \
      --box-app-config $BOX_APP_CONFIG_PATH \
      --remote-url $BOX_REMOTE_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.box import (
      BoxAccessConfig,
      BoxConnectionConfig,
      BoxIndexerConfig,
      BoxUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=BoxConnectionConfig(
              access_config=BoxAccessConfig(
                  box_app_config=os.getenv("BOX_APP_CONFIG_PATH")
              )
          ),
          uploader_config=BoxUploaderConfig(
              remote_url=os.getenv("BOX_REMOTE_URL")
          )
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.box import (
      BoxAccessConfig,
      BoxWriteConfig,
      SimpleBoxConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.fsspec.box import (
      BoxWriter,
  )


  def get_writer() -> Writer:
      return BoxWriter(
          connector_config=SimpleBoxConfig(
              access_config=BoxAccessConfig(box_app_config=os.getenv("BOX_APP_CONFIG_PATH")),
              remote_url=os.getenv("BOX_REMOTE_URL"),
          ),
          write_config=BoxWriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Chroma



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a Chroma account.

You will need:

The Chroma prerequisites:

*   A Chroma server. See [Deployment](https://docs.trychroma.com/deployment).

    For example, here is a video about how to deploy a Chroma server to AWS:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/4ApW6ZwqW20" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Chroma server's hostname or IP address, and the server's port number.

*   If you are not connecting to the server through HTTP, the path to the server instance.

*   The name of the tenant that you want to access on the server.

*   The name of the database that you want to access in the tenant.

*   The name of the collection that you want to access in the database.

The Chroma connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[chroma]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `CHROMA_HOST` - The , represented by `--host` (CLI) or `host` (Python).
*   `CHROMA_PORT` - The , represented by `--port` (CLI) or `port` (Python).
*   `CHROMA_TENANT` - The name of the tenant that you want to access on the Chroma server, represented by `--tenant` (CLI) or `tenant` (Python).
*   `CHROMA_DATABASE` - The name of the database that you want to access in the tenant, represented by `--database` (CLI) or `database` (Python).
*   `CHROMA_COLLECTION` - The name of the collection that you want to access in the database, represented by `--collection-name` (CLI) or `collection_name` (Python).

Additional settings include:

*   `--path` (CLI) or `path` (Python): The location where Chroma is persisted if you are not connecting through HTTP.
*   `--settings` (CLI) or `settings` (Python): A dictionary of settings to communicate with the Chroma server, for example: `'{"persist_directory":"./chroma-persist"}'`.
*   `--headers` (CLI) or `headers` (Python): A dictionary of headers to send to the Chroma server, for example: `'{"Authorization":"Basic()"}'`.
*   `--ssl` (CLI) or `ssl` (Python): True to use SSL for the connection.

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --work-dir $WORK_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    chroma \
      --host $CHROMA_HOST \
      --port $CHROMA_PORT \
      --collection-name $CHROMA_COLLECTION \
      --tenant $CHROMA_TENANT \
      --database $CHROMA_DATABASE \
      --settings "{\"persist_directory\":\"./chroma-persist\"}" \
      --headers "{\"Authorization\":\"Basic()\"}" \
      --batch-size 80 \
      --ssl
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.chroma import (
      ChromaAccessConfig,
      ChromaConnectionConfig,
      ChromaUploadStagerConfig,
      ChromaUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=ChromaConnectionConfig(
              access_config=ChromaAccessConfig(
                  settings={"persist_directory":"./chroma-persist"},
                  headers={"Authorization":"Basic()"}
              ),
              host=os.getenv("CHROMA_HOST"),
              port=os.getenv("CHROMA_PORT"),
              tenant=os.getenv("CHROMA_TENANT"),
              database=os.getenv("CHROMA_DATABASE"),
              collection_name=os.getenv("CHROMA_COLLECTION")
          ),
          stager_config=ChromaUploadStagerConfig(),
          uploader_config=ChromaUploaderConfig()
      ).run()

  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.chroma import (
      ChromaAccessConfig,
      ChromaWriteConfig,
      SimpleChromaConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.chroma import (
      ChromaWriter,
  )


  def get_writer() -> Writer:
      return ChromaWriter(
          connector_config=SimpleChromaConfig(
              access_config=ChromaAccessConfig(),
              host=os.getenv("CHROMA_HOST"),
              port=os.getenv("CHROMA_PORT"),
              collection_name=os.getenv("CHROMA_COLLECTION"),
              tenant=os.getenv("CHROMA_TENANT"),
              database=os.getenv("CHROMA_DATABASE"),
          ),
          write_config=ChromaWriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Clarifai



Batch process all your records using `unstructured-ingest` to store unstructured outputs locally on your filesystem and upload those to Clarifai apps.

First start with the installation of clarifai dependencies as shown here.

```bash
pip install "unstructured-ingest[clarifai]"
```

Create a clarifai app with base workflow. Find more information in the [create clarifai app](https://docs.clarifai.com/clarifai-basics/applications/create-an-application/).

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  # Chunking is optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --num-processes 2 \
      --verbose \
    clarifai \
      --app-id $CLARIFAI_APP_ID \
      --user-id $CLARIFAI_USER_ID \
      --api-key $CLARIFAI_PAT_KEY \
      --batch-size 100
  ```

  ```python Python
  import os

  from unstructured_ingest.connector.clarifai import (
      ClarifaiAccessConfig,
      ClarifaiWriteConfig,
      SimpleClarifaiConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.clarifai import (
      ClarifaiWriter,
  )


  def get_writer() -> Writer:
      return ClarifaiWriter(
          connector_config=SimpleClarifaiConfig(
              access_config=ClarifaiAccessConfig(api_key=os.getenv("CLARIFAI_PAT_KEY")),
              app_id=os.getenv("CLARIFAI_APP_ID"),
              user_id=os.getenv("CLARIFAI_USER_ID"),
          ),
          write_config=ClarifaiWriteConfig(),
      )

  # Chunking is optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

The upstream connector can be any of the ones supported, but for the convenience here, showing a sample command using the upstream local connector.

For a full list of the options the Unstructured Ingest CLI accepts check `unstructured-ingest <upstream connector> clarifai --help`.

NOTE: Keep in mind that you will need to have all the appropriate extras and dependencies for the file types of the documents contained in your data storage platform if you’re running this locally. You can find more information about this in the [installation guide](/open-source/installation/overview).


# Couchbase



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

[Couchbase](https://couchbase.com) is a Distributed NoSQL Cloud Database. Couchbase embraces AI with coding assistance for developers, and vector search for their applications.

Batch process all your records to store structured outputs in a Couchbase database.

You will need:

The Couchbase database prerequisites.

*   For the [Unstructured Platform](/platform/overview), only Couchbase Capella clusters are supported.
*   For [Unstructured Ingest](/ingestion/overview), Couchbase Capella clusters and local Couchbase server deployments are supported.

<iframe width="560" height="315" src="https://www.youtube.com/embed/9-RIBmIdi70" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

For Couchbase Capella, you will need:

*   A Couchbase Capella account. [Create an account](https://docs.couchbase.com/cloud/get-started/create-account.html).
*   A Couchbase Capella cluster. [Create a cluster](https://docs.couchbase.com/cloud/clusters/create-database.html).
*   A [bucket, scope, and collection](https://docs.couchbase.com/cloud/clusters/data-service/about-buckets-scopes-collections.html).
*   A cluster connection string. [Learn how](https://docs.couchbase.com/cloud/get-started/connect.html).
*   Incoming IP address allowance. [Learn how](https://docs.couchbase.com/cloud/clusters/allow-ip-address.html).
*   A username and password. [Learn how](https://docs.couchbase.com/cloud/clusters/manage-database-users.html).

For a local Couchbase server, you will need:

*   Installation of a local Couchbase server. [Learn how](https://docs.couchbase.com/server/current/getting-started/start-here.html).
*   Connection details to the local Couchbase server. [Learn how](https://docs.couchbase.com/server/current/guides/connect.html).

To learn more about how to set up a Couchbase cluster and play with data, refer to this [tutorial](https://developer.couchbase.com/tutorial-quickstart-flask-python).

The Couchbase DB connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[couchbase]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables are required for the Couchbase Connector:

*   `CB_CONN_STR` - The Connection String for the Couchbase server, represented by `--connection-string` (CLI) or `connection_string` (Python).
*   `CB_USERNAME` - The username for the Couchbase server, represented by `--username` (CLI) or `username` (Python).
*   `CB_PASSWORD` - The password for the Couchbase server, represented by `--password` (CLI) or `password` (Python).
*   `CB_BUCKET` - The name of the bucket in the Couchbase server, represented by `--bucket` (CLI) or `bucket` (Python).
*   `CB_SCOPE` - The name of the scope in the bucket, represented by `--scope` (CLI) or `scope` (Python).
*   `CB_COLLECTION` - The name of the collection in the scope, represented by `--collection` (CLI) or `collection` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library.  The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    couchbase \
      --connection-string $CB_CONN_STR \
      --username $CB_USERNAME \
      --password $CB_PASSWORD \
      --bucket $CB_BUCKET \
      --scope $CB_SCOPE \
      --collection $CB_COLLECTION \
      --num-processes 2 \
      --batch-size 80
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.couchbase import (
      CouchbaseAccessConfig,
      CouchbaseConnectionConfig,
      CouchbaseUploadStagerConfig,
      CouchbaseUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=CouchbaseConnectionConfig(
              access_config=CouchbaseAccessConfig(
                  password=os.getenv("CB_PASSWORD"),
              ),
              connection_string=os.getenv("CB_CONN_STR"),
              username=os.getenv("CB_USERNAME"),
              bucket=os.getenv("CB_BUCKET"),
              scope=os.getenv("CB_SCOPE"),
              collection=os.getenv("CB_COLLECTION")
          ),
          stager_config=CouchbaseUploadStagerConfig(),
          uploader_config=CouchbaseUploaderConfig(batch_size=100)
      ).run()
  ```
</CodeGroup>

To understand how [vector search](https://www.couchbase.com/products/vector-search/) works in Couchbase, refer to this [tutorial](https://developer.couchbase.com/tutorial-python-langchain-pdf-chat) and the [Couchbase docs](https://docs.couchbase.com/cloud/vector-search/vector-search.html)

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Databricks Volumes



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in Databricks Volumes.

You will need:

The Databricks Volumes prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/rNZpwa1-g7M" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The preceding video shows how to use Databricks personal access tokens (PATs), which are supported only for [Unstructured Ingest](/ingestion/overview).

To learn how to use Databricks-managed service principals, which are supported by both the [Unstructured Platform](/platform/overview) and Unstructured Ingest,
see the additional videos later on this page.

*   The Databricks workspace URL. Get the workspace URL for
    [AWS](https://docs.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids),
    [Azure](https://learn.microsoft.com/azure/databricks/workspace/workspace-details#workspace-instance-names-urls-and-ids),
    or [GCP](https://docs.gcp.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids).

    Examples:

    *   AWS: `https://<workspace-id>.cloud.databricks.com`
    *   Azure: `https://adb-<workspace-id>.<random-number>.azuredatabricks.net`
    *   GCP: `https://<workspace-id>.<random-number>.gcp.databricks.com`

*   The Databricks authentication details. For more information, see the documentation for
    [AWS](https://docs.databricks.com/dev-tools/auth/index.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/),
    or [GCP](https://docs.gcp.databricks.com/dev-tools/auth/index.html).

    The following videos show how to create a Databricks-managed service principal and then grant it access to a Databricks volume:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/wBmqv5DaA1E" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    <iframe width="560" height="315" src="https://www.youtube.com/embed/DykQRxgh2aQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    For the [Unstructured Platform](/platform/overview), only the following Databricks authentication type is supported:

    *   For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.
        Note that for Azure, only Databricks-managed service principals are supported. Microsoft Entra ID-managed service principals are not supported.

    For [Unstructured Ingest](/ingestion/overview), the following Databricks authentication types are supported:

    *   For Databricks personal access token authentication (AWS, Azure, and GCP): The personal access token's value.
    *   For username and password (basic) authentication (AWS only): The user's name and password values.
    *   For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.
    *   For OAuth user-to-machine (U2M) authentication (AWS, Azure, and GCP): No additional values.
    *   For Azure managed identities (MSI) authentication (Azure only): The client ID value for the corresponding managed identity.
    *   For Microsoft Entra ID service principal authentication (Azure only): The tenant ID, client ID, and client secret values for the corresponding service principal.
    *   For Azure CLI authentication (Azure only): No additional values.
    *   For Microsoft Entra ID user authentication (Azure only): The Entra ID token for the corresponding Entra ID user.
    *   For Google Cloud Platform credentials authentication (GCP only): The local path to the corresponding Google Cloud service account's credentials file.
    *   For Google Cloud Platform ID authentication (GCP only): The Google Cloud service account's email address.

*   The Databricks catalog name for the volume. Get the catalog name for [AWS](https://docs.databricks.com/catalogs/manage-catalog.html), [Azure](https://learn.microsoft.com/azure/databricks/catalogs/manage-catalog), or [GCP](https://docs.gcp.databricks.com/catalogs/manage-catalog.html).

*   The Databricks schema name for the volume. Get the schema name for [AWS](https://docs.databricks.com/schemas/manage-schema.html), [Azure](https://learn.microsoft.com/azure/databricks/schemas/manage-schema), or [GCP](https://docs.gcp.databricks.com/schemas/manage-schema.html).

*   The Databricks volume name, and optionally any path in that volume that you want to access directly. Get the volume information for [AWS](https://docs.databricks.com/files/volumes.html), [Azure](https://learn.microsoft.com/azure/databricks/files/volumes), or [GCP](https://docs.gcp.databricks.com/files/volumes.html).

*   Make sure that the target user or service principal has access to the target volume. To learn more, see the documentation for
    [AWS](https://docs.databricks.com/volumes/utility-commands.html#change-permissions-on-a-volume),
    [Azure](https://learn.microsoft.com/azure/databricks/volumes/utility-commands#change-permissions-on-a-volume),
    or [GCP](https://docs.gcp.databricks.com/volumes/utility-commands.html#change-permissions-on-a-volume).

The Databricks Volumes connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[databricks-volumes]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `DATABRICKS_HOST` - The Databricks host URL, represented by `--host` (CLI) or `host` (Python).
*   `DATABRICKS_CATALOG` - The Databricks catalog name for the Volume, represented by `--catalog` (CLI) or `catalog` (Python).
*   `DATABRICKS_SCHEMA` - The Databricks schema name for the Volume, represented by `--schema` (CLI) or `schema` (Python). If not specified, `default` is used.
*   `DATABRICKS_VOLUME` - The Databricks Volume name, represented by `--volume` (CLI) or `volume` (Python).
*   `DATABRICKS_VOLUME_PATH` - Any optional path to access within the volume, specified by `--volume-path` (CLI) or `volume_path` (Python).

Environment variables based on your authentication type, depending on your cloud provider:

*   For Databricks personal access token authentication (AWS, Azure, and GCP):

    *   `DATABRICKS_TOKEN` - The personal access token, represented by `--token` (CLI) or `token` (Python).

*   For username and password (basic) authentication (AWS only): The user's name and password values.

    *   `DATABRICKS_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
    *   `DATABRICKS_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

*   For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.

    *   `DATABRICKS_CLIENT_ID` - The client ID value for the corresponding service principal, represented by `--client-id` (CLI) or `client_id` (Python).
    *   `DATABRICKS_CLIENT_SECRET` - The client ID and OAuth secret values for the corresponding service principal, represented by `--client-secret` (CLI) or `client_secret` (Python).

*   For OAuth user-to-machine (U2M) authentication (AWS, Azure, and GCP): No additional environment variables.

*   For Azure managed identities (MSI) authentication (Azure only):

    *   `ARM_CLIENT_ID` - The client ID value for the corresponding managed identity, represented by `--azure-client-id` (CLI) or `azure_client_id` (Python).
    *   If the target identity has not already been added to the workspace, then you must also specify the
        `DATABRICKS_AZURE_RESOURCE_ID`, represented by `--azure-workspace-resource-id` (CLI) or `azure_workspace_resource_id` (Python).

*   For Microsoft Entra ID service principal authentication (Azure only):

    *   `ARM_TENANT_ID` - The tenant ID value for the corresponding service principal, represented by `--azure-tenant-id` (CLI) or `azure_tenant_id` (Python).
    *   `ARM_CLIENT_ID` - The client ID value for the corresponding service principal, represented by `--azure-client-id` (CLI) or `azure_client_id` (Python).
    *   `ARM_CLIENT_SECRET` - The client secret value for the corresponding service principal, represented by `--azure-client-secret` (CLI) or `azure_client_secret` (Python).
    *   If the service principal has not already been added to the workspace, then you must also specify the
        `DATABRICKS_AZURE_RESOURCE_ID`, represented by `--azure-workspace-resource-id` (CLI) or `azure_workspace_resource_id` (Python).

*   For Azure CLI authentication (Azure only): No additional environment variables.

*   For Microsoft Entra ID user authentication (Azure only):

    *   `DATABRICKS_TOKEN` - The Entra ID token for the corresponding Entra ID user, represented by `--token` (CLI) or `token` (Python).

*   For Google Cloud Platform credentials authentication (GCP only):

    *   `GOOGLE_CREDENTIALS` - The local path to the corresponding Google Cloud service account's credentials file, represented by `--google-credentials` (CLI) or `google_credentials`

*   For Google Cloud Platform ID authentication (GCP only):

    *   `GOOGLE_SERVICE_ACCOUNT` - The Google Cloud service account's email address, represented by `--google-service-account` (CLI) or `google_service_account` (Python).

*   Alternatively, you can store the preceding settings in a local
    [Databricks configuration profile](https://docs.databricks.com/en/dev-tools/auth/config-profiles.html) and then just
    refer to the profile's name:

    *   `DATABRICKS_PROFILE` - The name of the Databricks configuration profile, represented by `--profile` (CLI) or `profile` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library.  The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
    databricks-volumes \
      --profile $DATABRICKS_PROFILE \
      --host $DATABRICKS_HOST \
      --catalog $DATABRICKS_CATALOG \
      --schema $DATABRICKS_SCHEMA \
      --volume $DATABRICKS_VOLUME \
      --volume-path $DATABRICKS_VOLUME_PATH
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  # For all supported Databricks authentication types, you can import this:
  from unstructured_ingest.v2.processes.connectors.databricks_volumes import (
      DatabricksVolumesConnectionConfig,
      DatabricksVolumesAccessConfig,
      DatabricksVolumesUploaderConfig
  )

  # Alternatively, for supported Databricks on AWS authentication types only, you can import this:
  # from unstructured_ingest.v2.processes.connectors.databricks.volumes_aws import (
  #     DatabricksAWSVolumesConnectionConfig,
  #     DatabricksAWSVolumesAccessConfig,
  #     DatabricksAWSVolumesUploaderConfig
  # )

  # Alternatively, for supported Azure Databricks authentication types only, you can import this:
  # from unstructured_ingest.v2.processes.connectors.databricks.volumes_azure import (
  #     DatabricksAzureVolumesConnectionConfig,
  #     DatabricksAzureVolumesAccessConfig,
  #     DatabricksAzureVolumesUploaderConfig
  # )

  # Alternatively, for supported Databricks on Google Cloud authentication types only, you can import this:
  # from unstructured_ingest.v2.processes.connectors.databricks.volumes_gcp import (
  #     DatabricksGoogleVolumesConnectionConfig,
  #     DatabricksGoogleVolumesAccessConfig,
  #     DatabricksGoogleVolumesUploaderConfig
  # )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          # For specifying a Databricks configuration profile:
          destination_connection_config=DatabricksVolumesConnectionConfig(
              access_config=DatabricksVolumesAccessConfig(profile=os.getenv("DATABRICKS_PROFILE")),
              host=os.getenv("DATABRICKS_HOST"),
              catalog=os.getenv("DATABRICKS_CATALOG"),
              schema=os.getenv("DATABRICKS_SCHEMA"),
              volume=os.getenv("DATABRICKS_VOLUME"),
              volume_path=os.getenv("DATABRICKS_VOLUME_PATH")
          ),
          uploader_config=DatabricksVolumesUploaderConfig(overwrite=True)
          # Other examples:
          #
          # For Databricks on AWS, with Databricks personal access token authentication:
          # destination_connection_config=DatabricksAWSVolumesConnectionConfig(
          #     access_config=DatabricksAWSVolumesAccessConfig(token=os.getenv("DATABRICKS_TOKEN")),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          # uploader_config=DatabricksAWSVolumesUploaderConfig(
          #     catalog=os.getenv("DATABRICKS_CATALOG"),
          #     schema=os.getenv("DATABRICKS_SCHEMA"),
          #     volume=os.getenv("DATABRICKS_VOLUME"),
          #     volume_path=os.getenv("DATABRICKS_VOLUME_PATH"),
          #     overwrite=True
          # )
          #
          # For Azure Databricks, with Microsoft Entra ID service principal authentication:
          # destination_connection_config=DatabricksAzureVolumesConnectionConfig(
          #     access_config=DatabricksAzureVolumesAccessConfig(
          #         azure_client_id=os.getenv("ARM_CLIENT_ID"),
          #         azure_client_secret=os.getenv("ARM_CLIENT_SECRET"),
          #         azure_tenant_id=os.getenv("ARM_TENANT_ID")   
          #     ),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          # uploader_config=DatabricksAzureVolumesUploaderConfig(
          #     catalog=os.getenv("DATABRICKS_CATALOG"),
          #     schema=os.getenv("DATABRICKS_SCHEMA"),
          #     volume=os.getenv("DATABRICKS_VOLUME"),
          #     volume_path=os.getenv("DATABRICKS_VOLUME_PATH"),
          #     overwrite=True
          # )
          #
          # For Databricks on Google Cloud, with Google Cloud Platform credentials authentication:
          # destination_connection_config=DatabricksGoogleVolumesConnectionConfig(
          #     access_config=DatabricksGoogleVolumesAccessConfig(
          #         google_service_account=os.getenv("GOOGLE_CREDENTIALS")
          #     ),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          # uploader_config=DatabricksAWSVolumesUploaderConfig(
          #     catalog=os.getenv("DATABRICKS_CATALOG"),
          #     schema=os.getenv("DATABRICKS_SCHEMA"),
          #     volume=os.getenv("DATABRICKS_VOLUME"),
          #     volume_path=os.getenv("DATABRICKS_VOLUME_PATH"),
          #     overwrite=True
          # )
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.databricks_volumes import (
      DatabricksVolumesAccessConfig,
      DatabricksVolumesWriteConfig,
      SimpleDatabricksVolumesConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.databricks_volumes import (
      DatabricksVolumesWriter,
  )


  def get_writer() -> Writer:
      return DatabricksVolumesWriter(
          connector_config=SimpleDatabricksVolumesConfig(
              host=os.getenv("DATABRICKS_HOST"),
              access_config=DatabricksVolumesAccessConfig(
                  token=os.getenv("DATABRICKS_TOKEN")
              ),
          ),
          write_config=DatabricksVolumesWriteConfig(
              catalog=os.getenv("DATABRICKS_CATALOG"),
              schema=os.getenv("DATABRICKS_SCHEMA"),
              volume=os.getenv("DATABRICKS_VOLUME"),
              volume_path=os.getenv("DATABRICKS_VOLUME_PATH")
          ),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(
              chunk_elements=True
              chunking_strategy="by_title",
          ),
          embedding_config=EmbeddingConfig(
              provider="huggingface"
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Delta Table



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a Delta Table in an Amazon S3 bucket.

You will need:

The Delta Table prerequisites for Amazon S3:

The following video shows how to fulfill the minimum set of S3 prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/_W4565dcUGI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The preceding video does not show how to create an AWS account or an S3 bucket.

For more information about prerequisites, see the following:

*   An AWS account. [Create an AWS account](https://aws.amazon.com/free).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/lIdh92JmWtg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An S3 bucket. [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
    Additional approaches are in the following video and in the how-to sections at the end of this page.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/e6w9LwZJFIA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/y4SfQoJpipo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at
    minimum the permission of `s3:PutObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

*   For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account.
    [Create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/MoFTaGJE65Q" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   If the target files are in the root of the bucket, the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
    If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

*   If the target files are in a folder, make sure the authenticated AWS IAM user has
    authenticated access to the folder as well. [Enable authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2.  Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3.  Browse to the existing bucket and open it.

4.  Click the **Permissions** tab.

5.  In the **Bucket policy** area, click **Edit**.

6.  In the **Policy** text area, copy the following JSON-formatted policy.
    To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this policy, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-bucket-name>` in two places with the name of your bucket.

    ```json
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::<my-account-id>:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::<my-bucket-name>",
                    "arn:aws:s3:::<my-bucket-name>/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    ```

7.  Click **Save changes**.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
    the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    ```yaml
    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

    Parameters:
      BucketName:
        Type: String
        Description: 'Name of the S3 bucket to create'

    Resources:
      MyS3Bucket:
        Type: 'AWS::S3::Bucket'
        Properties:
          BucketName: !Ref BucketName
          PublicAccessBlockConfiguration:
            BlockPublicAcls: true
            BlockPublicPolicy: false
            IgnorePublicAcls: true
            RestrictPublicBuckets: true

      BucketPolicy:
        Type: 'AWS::S3::BucketPolicy'
        Properties:
          Bucket: !Ref MyS3Bucket
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Sid: AllowAllAuthenticatedUsersInAccount
                Effect: Allow
                Principal:
                  AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
                Action:
                  - 's3:GetObject'
                  - 's3:PutObject'
                  - 's3:ListBucket'
                  - 's3:DeleteObject'
                Resource:
                  - !Sub 'arn:aws:s3:::${BucketName}'
                  - !Sub 'arn:aws:s3:::${BucketName}/*'

    Outputs:
      BucketName:
        Description: 'Name of the created S3 bucket'
        Value: !Ref MyS3Bucket
    ```

2.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3.  Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4.  Click **Create stack > With new resources (standard)**.

5.  On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6.  Click **Choose file**, and browse to and select the YAML file from your local machine.

7.  Click **Next**.

8.  Enter a unique **Stack name** and **BucketName**.

9.  Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2.  [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3.  Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
    To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this script, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-unique-bucket-name>` with the name of your bucket.
    *   Replace `<us-east-1>` with your AWS Region.

    ```bash
    #!/bin/bash

    # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
    ACCOUNT_ID="<my-account-id>"
    BUCKET_NAME="<my-unique-bucket-name>"
    REGION="<us-east-1>"

    # Temporary filename for the bucket policy.
    # Do not change this variable.
    POLICY_FILE="bucket_policy.json"

    # Create the bucket.
    aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

    # Wait for the bucket to exist.
    echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
    aws s3api wait bucket-exists --bucket $BUCKET_NAME

    # Check if the wait command was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket '$BUCKET_NAME' has been fully created."
    else
        echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
        exit 1
    fi

    # Remove the "block public policy" bucket access setting.
    aws s3api put-public-access-block \
        --bucket $BUCKET_NAME \
        --public-access-block-configuration \
        '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

    # Check if the operation was successful.
    if [ $? -eq 0 ]; then
        echo "The block public policy access setting was removed from '$BUCKET_NAME'."
    else
        echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
        exit 1
    fi

    # Create the bucket policy.
    cat << EOF > $POLICY_FILE
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::$BUCKET_NAME",
                    "arn:aws:s3:::$BUCKET_NAME/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    EOF

    # Apply the bucket policy.
    aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

    # Check if the policy application was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket policy was applied to '$BUCKET_NAME'."
    else
        echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
        exit 1
    fi

    # Verify the applied policy.
    echo "Verifying the applied policy:"
    aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

    # Remove the temporary bucket policy file.
    rm $POLICY_FILE
    ```

4.  Run the script, for example:

    ```bash
    sh create-s3-bucket.sh
    ```

5.  After the bucket is created, you can delete the script file, if you want.

The Delta Table connector dependencies for Amazon S3:

```bash CLI, Python
pip install "unstructured-ingest[delta-table]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `AWS_S3_URL` - The path to the S3 bucket or folder, formatted as `s3://my-bucket/` (if the files are in the bucket's root) or `s3://my-bucket/my-folder/`, represented by `--table-uri` (CLI) or `table_uri` (Python).
*   `AWS_ACCESS_KEY_ID` - The AWS access key ID for the authenticated AWS IAM user, represented by `--aws-access-key-id` (CLI) or `aws_access_key` (Python).
*   `AWS_SECRET_ACCESS_KEY` - The corresponding AWS secret access key, represented by `--aws-secret-access-key` (CLI) or `aws_secret_access_key` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library.  The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
    delta-table \
      --aws-access-key-id $AWS_ACCESS_KEY_ID \
      --aws-secret-access-key $AWS_SECRET_ACCESS_KEY \
      --table-uri $AWS_S3_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.delta_table import (
      DeltaTableConnectionConfig,
      DeltaTableAccessConfig,
      DeltaTableUploadStagerConfig,
      DeltaTableUploaderConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=DeltaTableConnectionConfig(
              access_config=DeltaTableAccessConfig(
                  aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
                  aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY")
              ),
              table_uri=os.getenv("AWS_S3_URL")
          ),
          stager_config=DeltaTableUploadStagerConfig(),
          uploader_config=DeltaTableUploaderConfig()
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.delta_table import DeltaTableWriteConfig, SimpleDeltaTableConfig
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.delta_table import (
      DeltaTableWriter,
  )


  def get_writer() -> Writer:
      return DeltaTableWriter(
          connector_config=SimpleDeltaTableConfig(
              table_uri=os.getenv("AWS_S3_URL"),
              storage_options={
                  "AWS_ACCESS_KEY_ID": os.getenv("AWS_ACCESS_KEY_ID"),
                  "AWS_SECRET_ACCESS_KEY": os.getenv("AWS_SECRET_ACCESS_KEY"),
              },
          ),
          write_config=DeltaTableWriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Dropbox



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a Dropbox account.

You will need:

The Dropbox prerequisites:

1.  A Dropbox account. [Get an account](https://www.dropbox.com/try/teams).

2.  A target source or destination folder in your Dropbox account. [Create a folder](https://help.dropbox.com/create-upload/creating-using-folders-on-dropbox).

3.  A Dropbox app for your Dropbox account. To learn how to create an app, click the **App Console** tab on the [Getting Started](https://www.dropbox.com/developers/reference/getting-started) page.

4.  Permission for your Dropbox app to read from, and write to, the target folder in your Dropbox account as needed. To do this:

    *   On the **Permissions** tab of your Dropbox app, check the boxes **files.content.read** or **files.content.write** or both. [Learn more](https://developers.dropbox.com/oauth-guide).

    *   On the **Settings** tab of your Dropbox app, for **App folder name**, set the name of the target folder in your Dropbox account for your Dropbox app to have access to.

    *   Note the remote URL to the target folder, which takes the format `dropbox://<path/to/folder/in/account>`.

5.  An access token for your Dropbox account. [Get a token](https://dropbox.tech/developers/generate-an-access-token-for-your-own-account). Save this token in a secure location. Do not share it with others.

The Dropbox connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[dropbox]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `DROPBOX_ACCESS_TOKEN` - The value of your access token, represented by `--token` (CLI) or `token` (Python).
*   `DROPBOX_REMOTE_URL` - The remote URL to the target folder, represented by `--remote-url` (CLI) or `remote_url` (Python).

Now call the Unstructured Ingest CLI or Unstructured Ingest Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY\
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    dropbox \
      --token $DROPBOX_ACCESS_TOKEN \
      --remote-url $DROPBOX_REMOTE_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.dropbox import (
      DropboxAccessConfig,
      DropboxConnectionConfig,
      DropboxUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=DropboxConnectionConfig(
              access_config=DropboxAccessConfig(
                  token=os.getenv("DROPBOX_ACCESS_TOKEN")
              )
          ),
          uploader_config=DropboxUploaderConfig(
              remote_url=os.getenv("DROPBOX_REMOTE_URL")
          )
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.dropbox import (
      DropboxAccessConfig,
      DropboxWriteConfig,
      SimpleDropboxConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.fsspec.dropbox import (
      DropboxWriter,
  )


  def get_writer() -> Writer:
      return DropboxWriter(
          connector_config=SimpleDropboxConfig(
              access_config=DropboxAccessConfig(token=os.getenv("DROPBOX_ACCESS_TOKEN")),
              remote_url=os.getenv("DROPBOX_REMOTE_URL"),
          ),
          write_config=DropboxWriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Elasticsearch



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in Elasticsearch.

You will need:

The Elasticsearch prerequisites.

*   For the [Unstructured Platform](/platform/overview), only Elastic Cloud instances are supported.

*   For [Unstructured Ingest](/ingestion/overview), Elastic Cloud instances and self-manged Elasticsearch instances are supported.

*   For Elastic Cloud, you will need an [Elastic Cloud service instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#hosted-elasticsearch-service).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/HEYIFC7uOus" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For self-managed Elasticsearch, you will need a [self-managed Elasticsearch instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#elasticsearch-deployment-options).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/9znbAkNG1Ok" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   You will need the name of the index on the instance. See [Create index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html) and [Get index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html).

    The Elasticsearch index that you use must have a schema that is compatible with the schema of the documents
    that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following index schema example for your own needs:

    ```json
    {
        "settings": {
            "index": {
                "knn": true,
                "knn.algo_param.ef_search": 100
            }
        },
        "mappings": {
            "properties": {
                "element_id": {
                    "type": "keyword"
                },
                "text": {
                    "type": "text"
                },
                "embeddings": {
                    "type": "dense_vector",
                    "dims": 384,
                    "index": true,
                    "similarity": "cosine"
                },
                "metadata": {
                    "type": "object",
                    "properties": {
                        "parent_id": {
                            "type": "text"
                        },
                        "page_number": {
                            "type": "integer"
                        },
                        "is_continuation": {
                            "type": "boolean"
                        },
                        "orig_elements": {
                            "type": "text"
                        }
                    }
                }
            }
        }
    }
    ```

    See also:

    *   [An Introduction to Elasticsearch Mapping](https://www.elastic.co/blog/found-elasticsearch-mapping-introduction)
    *   [Explicit mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html)
    *   [Dynamic field mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html)
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

    <iframe width="560" height="315" src="https://www.youtube.com/embed/0f0tEcrvP8g" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For Elastic Cloud, you will need the Elastic Cloud service instance's API key. If you are using Unstructured Ingest, you will also need the instance's Cloud ID. To get these, see your Elasticsearch Service web console.

*   For self-managed Elasticsearch, you will need:

    *   The self-managed instance's hostname and port number. See [Networking](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html).
    *   If you're using basic authentication to the self-managed instance, the user's name and password.
    *   If you're using token-based authentication to the self-managed instance, the bearer token or API key for the instance. See [Token-based authentication services](https://www.elastic.co/guide/en/elasticsearch/reference/current/token-authentication-services.html) and [Create API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html).
    *   If you're using certificate authentication to the self-managed instance, the path to the Certificate Authority (CA) file on the instance, and the certificate fingerprint. See [SSL certificate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-ssl.html) and [Where can I see my Certificate Fingerprint?](https://discuss.elastic.co/t/where-can-i-see-my-certificate-fingerprint/319335/3).

The Elasticsearch connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[elasticsearch]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `ELASTICSEARCH_HOST` - The hostname and port number, defined as `<hostname>:<port-number>` and represented by `--hosts` (CLI) or `hosts` (Python).
*   `ELASTICSEARCH_INDEX_NAME` - The name of the search index, represented by `--index-name` (CLI) or `index_name` (Python).

If you're using Elastic Cloud to connect to ElasticSearch:

*   `ELASTIC_CLOUD_ID` - The instance's Cloud ID, represented by `--cloud-id` (CLI) or `cloud_id` (Python).
*   `ELASTIC_CLOUD_API_KEY` - The corresponding Cloud API key value, represented by `--api-key-id` (CLI) or `api_key_id` (Python).

If you're using basic authentication to the instance:

*   `ELASTICSEARCH_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
*   `ELASTICSEARCH_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

If you're using token-based authentication to the instance instead:

*   `ELASTICSEARCH_BEARER_TOKEN` - The bearer token's value, represented by `--bearer-auth` (CLI) or `bearer_auth` (Python).
*   `ELASTIC_CLOUD_API_KEY_ID` - The API key's value, represented by `--api-key` (CLI) or `api_key` (Python).

If you're using certificates:

*   `ELASTICSEARCH_CA_CERTS` - The path to the Certificate Authority (CA) file, represented by `--ca-certs` (CLI) or `ca_certs` (Python).
*   `ELASTICSEARCH_SSL_ASSERT_FINGERPRINT` - The certificate's fingerprint, represented by `--ssl-assert-fingerprint` or `ssl_assert_fingerprint` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 4 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
    elasticsearch \
      --hosts $ELASTICSEARCH_HOST \
      --username $ELASTICSEARCH_USERNAME \
      --password $ELASTICSEARCH_PASSWORD \
      --index-name $ELASTICSEARCH_INDEX_NAME \
      --ca-certs $ELASTICSEARCH_CA_CERTS \
      --ssl-assert-fingerprint $ELASTICSEARCH_SSL_ASSERT_FINGERPRINT \
      --num-processes 2
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.elasticsearch import (
      ElasticsearchConnectionConfig,
      ElasticsearchAccessConfig,
      ElasticsearchUploadStagerConfig,
      ElasticsearchUploaderConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=ElasticsearchConnectionConfig(
              access_config=ElasticsearchAccessConfig(
                  password=os.getenv("ELASTICSEARCH_PASSWORD"),
                  ssl_assert_fingerprint=os.getenv("ELASTICSEARCH_SSL_ASSERT_FINGERPRINT")
              ),
              hosts=[os.getenv("ELASTICSEARCH_HOST")],
              username=os.getenv("ELASTICSEARCH_USERNAME"),
              ca_certs=os.getenv("ELASTICSEARCH_CA_CERTS")
          ),
          stager_config=ElasticsearchUploadStagerConfig(index_name=os.getenv("ELASTICSEARCH_INDEX_NAME")),
          uploader_config=ElasticsearchUploaderConfig(index_name=os.getenv("ELASTICSEARCH_INDEX_NAME"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.elasticsearch import (
      ElasticsearchAccessConfig,
      ElasticsearchWriteConfig,
      SimpleElasticsearchConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.elasticsearch import (
      ElasticsearchWriter,
  )


  def get_writer() -> Writer:
      return ElasticsearchWriter(
          connector_config=SimpleElasticsearchConfig(
              access_config=ElasticsearchAccessConfig(
                  password=os.getenv("ELASTICSEARCH_PASSWORD"),
                  ssl_assert_fingerprint=os.getenv("ELASTICSEARCH_SSL_ASSERT_FINGERPRINT")
              ),
              hosts=[os.getenv("ELASTICSEARCH_HOST")],
              username=os.getenv("ELASTICSEARCH_USERNAME"),
              ca_certs=os.getenv("ELASTICSEARCH_CA_CERTS"),
          ),
          write_config=ElasticsearchWriteConfig(
              batch_size_bytes=15_000_000,
              num_processes=2,
          ),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Google Cloud Storage



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in Google Cloud Service.

You will need:

The Google Cloud Storage prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/HYaALQ0F-L4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A Google Cloud service account. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#console).

*   A service account key for the service account. See [Create a service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating) in
    [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete).

    To ensure maximum compatibility across Unstructured service offerings, you should give the service account key information to Unstructured as
    a single-line string that contains the contents of the downloaded service account key file (and not the service account key file itself).
    To print this single-line string without line breaks, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
    In this command, replace `<path-to-downloaded-key-file>` with the path to the service account key file that you downloaded by following the preceding instructions.

    *   For macOS or Linux:
        ```text
        tr -d '\n' < <path-to-downloaded-key-file>
        ```
    *   For Windows:
        ```text
        (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
        ```

*   The URI for a Google Cloud Storage bucket. This URI consists of the target bucket name, plus any target folder within the bucket, expressed as `gs://<bucket-name>[/folder-name]`. [Create a bucket](https://cloud.google.com/storage/docs/creating-buckets#console).

    This bucket must have, at minimum, one of the following roles applied to the target Google Cloud service account:

    *   `Storage Object Viewer` for bucket read access.
    *   `Storage Object Creator` for bucket write access.
    *   The `Storage Object Admin` role provides read and write access, plus access to additional bucket operations.

    To apply one of these roles to a service account for a bucket, see [Add a principal to a bucket-level policy](https://cloud.google.com/storage/docs/access-control/using-iam-permissions#bucket-add)
    in [Set and manage IAM policies on buckets](https://cloud.google.com/storage/docs/access-control/using-iam-permissions).

The Google Cloud Storage connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[gcs]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `GCS_SERVICE_ACCOUNT_KEY` - The Google Cloud service account key for Google Cloud Storage, represented by `--service-account-key` (CLI) or `service_account_key` (Python).
*   `GCS_REMOTE_URL` - The Google Cloud Storage bucket URL, represented by `--remote-url` (CLI) or `remote_url` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
    gcs \
      --service-account-key $GCS_SERVICE_ACCOUNT_KEY \
      --remote-url $GCS_REMOTE_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.gcs import (
      GcsConnectionConfig,
      GcsAccessConfig,
      GcsUploaderConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=GcsConnectionConfig(
              access_config=GcsAccessConfig(
                  service_account_key=os.getenv("GCS_SERVICE_ACCOUNT_KEY")
              )
          ),
          uploader_config=GcsUploaderConfig(remote_url=os.getenv("GCS_REMOTE_URL"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.gcs import (
      GcsAccessConfig,
      GcsWriteConfig,
      SimpleGcsConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.fsspec.gcs import (
      GcsWriter,
  )


  def get_writer() -> Writer:
      return GcsWriter(
          connector_config=SimpleGcsConfig(
              access_config=GcsAccessConfig(token=os.getenv("GCS_SERVICE_ACCOUNT_KEY")),
              remote_url=os.getenv("GCS_REMOTE_URL"),
          ),
          write_config=GcsWriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Kafka



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in Kafka.

You will need:

The Kafka prerequisites:

*   A Kafka cluster, such as ones provided by [Confluent Cloud](https://www.confluent.io/confluent-cloud), [Amazon Managed Streaming for Apache Kafka (Amazon MSK)](https://aws.amazon.com/msk/), or [Google Cloud Managed Service for Apache Kafka](https://cloud.google.com/products/managed-service-for-apache-kafka).
*   The hostname of the bootstrap Kafka cluster to connect to.
*   The port number of the cluster.
*   The name of the topic to read messages from and write messages to on the cluster.
*   If you use Kafka API keys and secrets for authentication, the key and secret values.

The Kafka connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[kafka]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `KAFKA_BOOTSTRAP_SERVER` - The hostname of the bootstrap Kafka cluster to connect to, represented by `--bootstrap-server` (CLI) or `bootstrap_server` (Python).
*   `KAFKA_PORT` - The port number of the cluster, represented by `--port` (CLI) or `port` (Python).
*   `KAFKA_TOPIC` - The unique name of the topic to read messages from and write messages to on the cluster, represented by `--topic` (CLI) or `topic` (Python).

If you use Kafka API keys and secrets for authentication:

*   `KAFKA_API_KEY` - The Kafka API key value, represented by `--kafka-api-key` (CLI) or `kafka_api_key` (Python).
*   `KAFKA_SECRET` - The secret value for the Kafka API key, represented by `--secret` (CLI) or `secret` (Python).

Additional settings include:

*   `--confluent` (CLI) or `confluent` (Python): True to indicate that the cluster is running Confluent Kafka.
*   `--num-messages-to-consume` (CLI) or `num_messages_to_consume` (Python): The maximum number of messages to get from the topic. The default is `1`.
*   `--timeout` (CLI) or `timeout` (Python): The maximum amount of time to wait for the response of a request to the topic, expressed in seconds. The default is `1.0`.

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY\
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    kafka \
      --bootstrap-server $KAFKA_BOOTSTRAP_SERVER \ 
      --port $KAFKA_PORT \
      --topic $KAFKA_TOPIC \
      --kafka-api-key $KAFKA_API_KEY \
      --secret $KAFKA_API_KEY \
      --confluent true \
      --num-messages-to-consume 1 \
      --timeout 1.0
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  from unstructured_ingest.v2.processes.connectors.kafka.cloud import (
      CloudKafkaConnectionConfig,
      CloudKafkaAccessConfig,
      CloudKafkaUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=CloudKafkaConnectionConfig(
              access_config=CloudKafkaAccessConfig(
                  kafka_api_key=os.getenv("KAFKA_API_KEY"),
                  secret=os.getenv("KAFKA_SECRET")
              ),
              bootstrap_server=os.getenv("KAFKA_BOOTSTRAP_SERVER"),
              port=os.getenv("KAFKA_PORT")
          ),
          uploader_config=CloudKafkaUploaderConfig(
              batch_size=100,
              topic=os.getenv("KAFKA_TOPIC"),
              timeout=10
          )
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.kafka import (
      KafkaAccessConfig,
      KafkaWriteConfig,
      SimpleKafkaConfig
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.kafka import (
      KafkaWriter
  )
  from unstructured_ingest.runner.writers.base_writer import Writer

  def get_writer() -> Writer:
      return KafkaWriter(
          connector_config=SimpleKafkaConfig(
              bootstrap_server=os.getenv("KAFKA_BOOTSTRAP_SERVER"),
              port=os.getenv("KAFKA_PORT"),
              topic=os.getenv("KAFKA_TOPIC"),
              access_config=KafkaAccessConfig(
                  kafka_api_key=os.getenv("KAFKA_API_KEY"),
                  secret=os.getenv("KAFKA_SECRET"),
              ),
              confluent=False,
              num_messages_to_consume=1,
              timeout=1.0
          ),
          write_config=KafkaWriteConfig()
      )


  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# KDB.AI



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in KDB.AI.

You will need:

The KDB.AI prerequisites:

*   A KDB.AI Cloud or server instance. [Sign Up for KDB.AI Cloud: Starter Edition](https://trykdb.kx.com/kdbai/signup/). [Set up KDB.AI Server](https://code.kx.com/kdbai/latest/gettingStarted/kdb-ai-server-setup.html).
*   The instance's endpoint URL. [Get the KDB.AI Cloud endpoint URL](https://code.kx.com/kdbai/latest/gettingStarted/kdb-ai-cloud-setup.html#connect-to-your-database). [Get the KDB.AI Server endpoint URL](https://code.kx.com/kdbai/latest/gettingStarted/kdb-ai-server-setup.html).
*   An API key. [Create the API key](https://code.kx.com/kdbai/latest/gettingStarted/kdb-ai-cloud-setup.html#create-an-api-key).
*   The name of the target table to access. [Create the table](https://code.kx.com/kdbai/latest/gettingStarted/quickstart.html#create-a-new-table).

The KDB.AI connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[kdbai]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `KDBAI_ENDPOINT` - The KDB.AI instance's endpoint URL, represented by `--endpoint` (CLI) or `endpoint` (Python).
*   `KDBAI_API_KEY` - The KDB.AI API key, represented by `--api-key` (CLI) or `api_key` (Python).
*   `KDBAI_TABLE` - The name of the target table, represented by `--table-name` (CLI) or `table_name` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    kdbai \
      --endpoint $KDBAI_API_KEY \
      --api-key $KDBAI_API_KEY \
      --table-name $KDBAI_TABLE
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.kdbai import (
      KdbaiConnectionConfig,
      KdbaiAccessConfig,
      KdbaiUploadStagerConfig,
      KdbaiUploaderConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=KdbaiConnectionConfig(
              access_config=KdbaiAccessConfig(
                  api_key=os.getenv("KDBAI_API_KEY")
              ),
              endpoint=os.getenv("KDBAI_ENDPOINT")
          ),
          stager_config=KdbaiUploadStagerConfig(),
          uploader_config=KdbaiUploaderConfig(table_name=os.getenv("KDBAI_TABLE"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# LanceDB



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in LanceDB.

You will need:

The LanceDB prerequisites:

*   A [LanceDB open source software (OSS) installation](https://lancedb.github.io/lancedb/basic/#installation) on a local machine, a server, or a virtual machine.
    (LanceDB Cloud is not supported.)

*   For LanceDB OSS with local data storage:

    *   The local path to the folder where the LanceDB data is (or will be) stored.
        See [Connect to a database](https://lancedb.github.io/lancedb/basic/#connect-to-a-database) in the LanceDB documentation.
    *   The name of the target [LanceDB table](https://lancedb.github.io/lancedb/basic/#create-an-empty-table) within the local data folder.

*   For LanceDB OSS with data storage in an Amazon S3 bucket:

    *   The URI for the target Amazon S3 bucket and any target folder path within that bucket. Use the format `s3://<bucket-name>[/<folder-name>]`.
    *   The name of the target [LanceDB table](https://lancedb.github.io/lancedb/guides/storage/#object-stores) within the Amazon S3 bucket.
    *   The AWS access key ID and AWS secret access key for the AWS IAM entity that has access to the Amazon S3 bucket.

    For more information, see [AWS S3](https://lancedb.github.io/lancedb/guides/storage/#aws-s3) in the LanceDB documentation, along with the following video:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/hyDHfhVVAhs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For LanceDB OSS with data storage in an Azure Blob Storage account:

    *   The name of the target Azure Blob Storage account.
        \= The URI for the target container within that Azure Blob Storage account and any target folder path within that container. Use the format `az://<container-name>[/<folder-name>]`.
    *   The name of the target [LanceDB table](https://lancedb.github.io/lancedb/guides/storage/#object-stores) within the Azure Blob Storage account.
    *   The access key for the Azure Blob Storage account.

    For more information, see [Azure Blob Storage](https://lancedb.github.io/lancedb/guides/storage/#azure-blob-storage) in the LanceDB documentation, along with the following video:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/Vl3KCphlh9Y" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For LanceDB OSS with data storage in a Google Cloud Storage bucket:

    *   The URI for the target Google Cloud Storage bucket and any target folder path within that bucket. Use the format `gs://<bucket-name>[/<folder-name>]`.
    *   The name of the target [LanceDB table](https://lancedb.github.io/lancedb/guides/storage/#object-stores) within the Google Cloud Storage bucket.
    *   A single-line string that contains the contents of the downloaded service account key file for the Google Cloud service account that has access to the
        Google Cloud Storage bucket.

    For more information, see [Google Cloud Storage](https://lancedb.github.io/lancedb/guides/storage/#google-cloud-storage) in the LanceDB documentation, along with the following video:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/HYaALQ0F-L4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The LanceDB connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[lancedb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   For LanceDB OSS with local data storage:

    *   `LANCEDB_URI` - The local path to the folder where the LanceDB data is stored, represented by `--uri` (CLI) or `uri` (Python).
    *   `LANCEDB_TABLE` - The name of the target LanceDB table within the local data folder, represented by `--table-name` (CLI) or `table_name` (Python).

*   For LanceDB OSS with data storage in an Amazon S3 bucket:

    *   `LANCEDB_URI` - The URI for the target Amazon S3 bucket and any target folder path within that bucket. Use the format `s3://<bucket-name>[/<folder-name>]`. This is represented by `--uri` (CLI) or `uri` (Python).
    *   `LANCEDB_TABLE` - The name of the target LanceDB table within the Amazon S3 bucket, rrepresented by `--table-name` (CLI) or `table_name` (Python).
    *   `AWS_ACCESS_KEY_ID` - The AWS access key ID for the AWS IAM entity that has access to the Amazon S3 bucket, represented by `--aws-access-key-id` (CLI) or `aws_access_key_id` (Python).
    *   `AWS_SECRET_ACCESS_KEY` - The AWS secret access key for the AWS IAM entity that has access to the Amazon S3 bucket, represented by `--aws-secret-access-key` (CLI) or `aws_secret_access_key` (Python).

*   For LanceDB OSS with data storage in an Azure Blob Storage account:

    *   `LANCEDB_URI` - The URI for the target container within that Azure Blob Storage account and any target folder path within that container. Use the format `az://<container-name>[/<folder-name>]`. This is represented by `--uri` (CLI) or `uri` (Python).
    *   `LANCEDB_TABLE` - The name of the target LanceDB table within the Azure Blob Storage account, represented by `--table-name` (CLI) or `table_name` (Python).
    *   `AZURE_STORAGE_ACCOUNT_NAME` - The name of the target Azure Blob Storage account, represented by `--azure-storage-account-name` (CLI) or `azure_storage_account_name` (Python).
    *   `AZURE_STORAGE_ACCOUNT_KEY` - The access key for the Azure Blob Storage account, represented by `--azure-storage-account-key` (CLI) or `azure_storage_account_key` (Python).

*   For LanceDB OSS with data storage in a Google Cloud Storage bucket:

    *   `LANCEDB_URI` - The URI for the target Google Cloud Storage bucket and any target folder path within that bucket. Use the format `gs://<bucket-name>[/<folder-name>]`. This is represented by `--uri` (CLI) or `uri` (Python).
    *   `LANCEDB_TABLE` - The name of the target LanceDB table within the Google Cloud Storage bucket, represented by `--table-name` (CLI) or `table_name` (Python).
    *   `GCS_SERVICE_ACCOUNT_KEY` - A single-line string that contains the contents of the downloaded service account key file for the Google Cloud service account
        that has access to the Google Cloud Storage bucket, represented by `--google-service-account-key` (CLI) or `google_service_account_key` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  # For LanceDB OSS with local data storage:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    lancedb-local \
      --uri $LANCEDB_URI \
      --table-name $LANCEDB_TABLE

  # For LanceDB OSS with data storage in an Amazon S3 bucket:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    lancedb-aws \
      --aws-access-key-id $AWS_ACCESS_KEY_ID \
      --aws-secret-access-key $AWS_SECRET_ACCESS_KEY \
      --uri $LANCEDB_URI \
      --table-name $LANCEDB_TABLE \
      --timeout 30s

  # For LanceDB OSS with data storage in an Azure Blob Storage account:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    lancedb-azure \
      --azure-storage-account-name $AZURE_STORAGE_ACCOUNT_NAME \
      --azure-storage-account-key $AZURE_STORAGE_ACCOUNT_KEY \
      --uri $LANCEDB_URI \
      --table-name $LANCEDB_TABLE \
      --timeout 30s

  # For LanceDB OSS with data storage in a Google Cloud Storage bucket:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    lancedb-gcs \
      --google-service-account-key $GCS_SERVICE_ACCOUNT_KEY \
      --uri $LANCEDB_URI \
      --table-name $LANCEDB_TABLE \
      --timeout 30s
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # For LanceDB OSS with local data storage:
  # from unstructured_ingest.v2.processes.connectors.lancedb.local import (
  #     LanceDBLocalConnectionConfig,
  #     LanceDBLocalAccessConfig,
  #     LanceDBUploadStagerConfig,
  #     LanceDBUploaderConfig
  # )

  # For LanceDB OSS with data storage in an Amazon S3 bucket:
  from unstructured_ingest.v2.processes.connectors.lancedb.aws import (
      LanceDBS3ConnectionConfig,
      LanceDBS3AccessConfig,
      LanceDBUploadStagerConfig,
      LanceDBUploaderConfig
  )

  # For LanceDB OSS with data storage in an Azure Blob Storage account:
  # from unstructured_ingest.v2.processes.connectors.lancedb.azure import (
  #     LanceDBAzureConnectionConfig,
  #     LanceDBAzureAccessConfig,
  #     LanceDBUploadStagerConfig,
  #     LanceDBUploaderConfig
  # )

  # For LanceDB OSS with data storage in a Google Cloud Storage bucket:
  # from unstructured_ingest.v2.processes.connectors.lancedb.gcp import (
  #     LanceDBGCSConnectionConfig,
  #     LanceDBGCSAccessConfig,
  #     LanceDBUploadStagerConfig,
  #     LanceDBUploaderConfig
  # )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),

          # For LanceDB OSS with local data storage:
          # destination_connection_config=LanceDBLocalConnectionConfig(
          #     access_config=LanceDBLocalAccessConfig(),
          #     uri=os.getenv("LANCEDB_URI")
          # ),

          # For LanceDB OSS with data storage in an Amazon S3 bucket:
          destination_connection_config=LanceDBS3ConnectionConfig(
              access_config=LanceDBS3AccessConfig(
                  aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
                  aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY")
              ),
              uri=os.getenv("LANCEDB_URI"),
              timeout="30s"
          ),

          # For LanceDB OSS with data storage in an Azure Blob Storage account:
          # destination_connection_config=LanceDBAzureConnectionConfig(
          #     access_config=LanceDBAzureAccessConfig(
          #         azure_storage_account_name=os.getenv("AZURE_STORAGE_ACCOUNT_NAME"),
          #         azure_storage_account_key=os.getenv("AZURE_STORAGE_ACCOUNT_KEY")
          #     ),
          #     uri=os.getenv("LANCEDB_URI"),
          #     timeout="30s"
          # ),

          # For LanceDB OSS with data storage in a Google Cloud Storage bucket:
          # destination_connection_config=LanceDBGCSConnectionConfig(
          #     access_config=LanceDBGCSAccessConfig(
          #         google_service_account_key=os.getenv("GCS_SERVICE_ACCOUNT_KEY")
          #     ),
          #     uri=os.getenv("LANCEDB_URI"),
          #     timeout="30s"
          # ),

          stager_config=LanceDBUploadStagerConfig(),
          uploader_config=LanceDBUploaderConfig(table_name=os.gentenv("LANCEDB_TABLE"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Local



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs locally.

You will need the local destination connector dependencies:

```bash CLI, Python
pip install unstructured-ingest
```

To use the local destination connector, you must set `--output-dir` (CLI) or `output_dir` (Python) to the path in the local filesystem which will contain the structured output.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library.  The source connector can be any of the ones supported.

The following 3-minute video shows how to use the Unstructured Ingest Python library to send multiple PDFs from a local directory in batches to be ingested by Unstructured API services for processing:

<iframe width="560" height="315" src="https://www.youtube.com/embed/tSKHFXsBQ-c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig,
      LocalUploaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import LocalRunner

  if __name__ == "__main__":
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
              recursive=True,
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Milvus



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in Milvus.

You will need:

The Milvus prerequisites:

*   A [Milvus instance](https://milvus.io/docs/install-overview.md).
*   The [URI](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md) of the instance.
*   The [username and password, or token](https://milvus.io/docs/authenticate.md) to access the instance.
*   The name of the [database](https://milvus.io/docs/manage_databases.md) in the instance.
*   The name of the [collection](https://milvus.io/docs/manage-collections.md) in the database.

<iframe width="560" height="315" src="https://www.youtube.com/embed/tl9eV-_nAHI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The Milvus connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[milvus]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `MILVUS_URI` - The Milvus instance's URI, represented by `--uri` (CLI) or `uri` (Python).
*   `MILVUS_USER` and `MILVUS_PASSWORD`, or `MILVUS_TOKEN` - The username and password, or token, to access the instance. This is represented by `--user` and `--password`, or `--token` (CLI); or `user` and `password`, or `token` (Python).
*   `MILVUS_DB` - The database's name, represented by `--db-name` (CLI) or `db_name` (Python).
*   `MILVUS_COLLECTION` - The collection's name, represented by `--collection-name` (CLI) or `collection_name` (Python).
*   `MILVUS_FIELDS_TO_INCLUDE` - A list of fields to include a comma-separated list (CLI) or an array of strings (Python), represented by `--field-to-include` (CLI) or `fields_to_include` (Python).

Additional settings include:

*   To emit the `metadata` field's child fields directly into the output, include `--flatten-metadata` (CLI) or `flatten_metadata=True` (Python). This is the default if not specified.
*   To keep the `metadata` field with its child fields intact in the output, include `--no-flatten-metadata` (CLI) or `flatten_metadata=False` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    milvus \
      --uri $MILVUS_URI \
      --user $MILVUS_USER \
      --password $MILVUS_PASSWORD \
      --db-name $MILVUS_DB \
      --collection-name $MILVUS_COLLECTION \
      --fields-to-include type,element_id,text,embeddings
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.milvus import (
      MilvusConnectionConfig,
      MilvusAccessConfig,
      MilvusUploadStagerConfig,
      MilvusUploaderConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=MilvusConnectionConfig(
              access_config=MilvusAccessConfig(
                  password=os.getenv("MILVUS_PASSWORD")
              ),
              uri=os.getenv("MILVUS_URI"),
              user=os.getenv("MILVUS_USER"),
              db_name=os.getenv("MILVUS_DB")
          ),
          stager_config=MilvusUploadStagerConfig(fields_to_include=["type", "element_id", "text", "embeddings"]),
          uploader_config=MilvusUploaderConfig(collection_name=os.getenv("MILVUS_COLLECTION"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# MongoDB



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in MongoDB.

You will need:

The MongoDB prerequisites for a MongoDB Atlas deployment:

<iframe width="560" height="315" src="https://www.youtube.com/embed/g6qDfbg808M" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A MongoDB Atlas account. [Create an account](https://www.mongodb.com/cloud/atlas/register).
*   A MongoDB Atlas cluster. [Create a cluster](https://www.mongodb.com/docs/atlas/tutorial/deploy-free-tier-cluster).
*   The cluster must be reachable from your application environment. [Learn how](https://www.mongodb.com/docs/atlas/setup-cluster-security/#network-and-firewall-requirements).
*   The cluster must be configured to use your IP address. [Learn how](https://www.mongodb.com/docs/atlas/setup-cluster-security/#ip-access-list).
*   The cluster must have at least one database. [Create a database](https://www.mongodb.com/docs/compass/current/databases/#create-a-database).
*   The database must have at least one user, and that user must have sufficient access to the database. [Create a database user](https://www.mongodb.com/docs/atlas/security-add-mongodb-users/#add-database-users). [Give the user database access](https://www.mongodb.com/docs/manual/core/authorization/).
*   The database must have at least one collection. [Create a collection](https://www.mongodb.com/docs/compass/current/collections/#create-a-collection).
*   The connection string for the cluster. For MongoDB Atlas, this connection string must include the protocol, username, password, host, and cluster name. For example:

    ```text
    mongodb+srv://<db_user>:<db_password>@<host>/?retryWrites=true&w=majority&appName=<cluster>
    ```

    To get the connection string in MongoDB Atlas, do the following:

    1.  Log in to your MongoDB Atlas console.
    2.  In the sidebar, under **Databases**, click **Clusters**.
    3.  Click on the cluster you want to connect to.
    4.  Click **Connect**, or click the **Cmd Line Tools** tab and then click **Connect Instructions**.
    5.  Click **Drivers**.
    6.  Under **Add your connection string into your application code**, copy the connection string.
        You can then close the **Connect** dialog in MongoDB Atlas.

        Before you use this connection string, be sure to fill in any placeholders in the string, such as your MongoDB Atlas database user's password value.

    [Learn more](https://www.mongodb.com/resources/products/fundamentals/mongodb-connection-string).

The MongoDB connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[mongodb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

For a MongoDB Atlas deployment, the following environment variables:

*   `MONGODB_DATABASE` - The name of the database, represented by `--database` (CLI) or `database` (Python).

*   `MONGODB_COLLECTION` - The name of the collection in the database, represented by `--collection` (CLI) or `collection` (Python).

*   `MONGODB_URI` - The connection string for the cluster, represented by `--uri` (CLI) or `uri` (Python).

For a local MongoDB server, the following environment variables:

*   `MONGODB_HOST` - The host for the local MongoDB server, represented by `--host` (CLI) or `host` (Python).

*   `MONGODB_PORT` - The port for the local MongoDB server, represented by `--port` (CLI) or `port` (Python).

Now call the Unstructured Ingest CLI or Unstructured Ingest Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    mongodb \
      --uri $MONGODB_URI \
      --database $MONGODB_DATABASE \
      --collection $MONGODB_COLLECTION
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.mongodb import (
      MongoDBAccessConfig,
      MongoDBConnectionConfig,
      MongoDBUploadStagerConfig,
      MongoDBUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=MongoDBConnectionConfig(
              access_config=MongoDBAccessConfig(
                  uri=os.getenv("MONGODB_URI")
              ),
              database=os.getenv("MONGODB_DATABASE"),
              collection=os.getenv("MONGODB_COLLECTION")
          ),
          stager_config=MongoDBUploadStagerConfig(),
          uploader_config=MongoDBUploaderConfig(batch_size=100)
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.connector.mongodb import SimpleMongoDBConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
      WriteConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.mongodb import (
      MongodbWriter,
  )


  def get_writer() -> Writer:
      return MongodbWriter(
          connector_config=SimpleMongoDBConfig(
              uri=os.getenv("MONGODB_URI"),
              database=os.getenv("MONGODB_DATABASE"),
              collection=os.getenv("MONGODB_COLLECTION"),
          ),
          write_config=WriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# OneDrive



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a OneDrive account.

You will need:

The OneDrive prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/7uSu0LsbcEQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A OneDrive account.
*   The User Principal Name (UPN) for the OneDrive account. This is typically the OneDrive account user's email address.
*   The path to the target OneDrive folder, starting from the OneDrive account's root folder, for example `my-folder/my-subfolder`.
*   The client ID, tenant ID, and client secret for the Microsoft Entra ID app registration that has access to the target OneDrive account and
    also has the correct set of Microsoft Graph authentication scopes. These scopes include:

    *   `Files.ReadWrite.All` (if both reading and writing are needed)
    *   `Sites.ReadWrite.All` (if both reading and writing are needed)
    *   `User.Read.All`
    *   `Directory.Read.All`

See also:

*   [Registering your app for Microsoft Graph](https://learn.microsoft.com/onedrive/developer/rest-api/getting-started/app-registration)
*   [OneDrive authentication and sign-in](https://learn.microsoft.com/onedrive/developer/rest-api/getting-started/msa-oauth)
*   [OneDrive API documentation](https://docs.microsoft.com/onedrive/developer/rest-api/)

The OneDrive connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[onedrive]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `ONEDRIVE_PATH` - The path to the target folder in the OneDrive account, starting with the account's root folder (for example `my-folder/my-subfolder`), represented by `--path` (CLI) or `path` (Python).
*   `ONEDRIVE_CLIENT_ID` - The application (client) ID of the Microsoft Entra ID app registration that has access to the OneDrive account, represented by `--client-id` (CLI) or `client_id` (Python).
*   `ONEDRIVE_CLIENT_CRED` - The client secret for the Entra ID app registration, represented by `--client-cred` (CLI) or `client_cred` (Python).
*   `ONEDRIVE_TENANT` - The directory (tenant) ID of the Entra ID app registration, represented by `--tenant` (CLI) or `tenant` (Python).
*   `ONEDRIVE_USER_PNAME` - The User Principal Name (UPN) for the OneDrive user account in Entra ID (typically the user's email address), represented by `--user-pname` (CLI) or `user_pname` (Python).
*   `ONEDRIVE_AUTHORITY_URL` - The authentication token provider URL for the Entra ID app registration (the default is `https://login.microsoftonline.com` if not otherwise specified), represented by `--authority-url` (CLI) or `authority_url` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    onedrive \
      --client-cred $ONEDRIVE_CLIENT_CRED \
      --client-id $ONEDRIVE_CLIENT_ID \
      --user-pname $ONEDRIVE_USER_PNAME \
      --tenant $ONEDRIVE_TENANT \
      --authority-url $ONEDRIVE_AUTHORITY_URL \
      --remote-url $ONEDRIVE_PATH \
      --prefix "onedrive://"
  ```

  ```bash Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  from unstructured_ingest.v2.processes.connectors.onedrive import (
      OnedriveConnectionConfig,
      OnedriveAccessConfig,
      OnedriveUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=OnedriveConnectionConfig(
              access_config=OnedriveAccessConfig(client_cred=os.getenv("ONEDRIVE_CLIENT_CRED")),
              client_id=os.getenv("ONEDRIVE_CLIENT_ID"),
              user_pname=os.getenv("ONEDRIVE_USER_PNAME"),
              tenant=os.getenv("ONEDRIVE_TENANT"),
              authority_url=os.getenv("ONEDRIVE_AUTHORITY_URL")
          ),
          uploader_config=OnedriveUploaderConfig(
              remote_url=os.getenv("ONEDRIVE_PATH"),
              prefix="onedrive://"
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# OpenSearch



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in OpenSearch.

You will need:

The OpenSearch prerequisites:

*   An OpenSearch instance, such as an [AWS OpenSearch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/setting-up.html) instance...

    <iframe width="560" height="315" src="https://www.youtube.com/embed/BNOYTbRbaFQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    ...or a [local](https://opensearch.org/downloads.html) instance.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/Rew3_pNnYIs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    In all cases, the [OpenSearch version](https://github.com/opensearch-project/opensearch-py/blob/main/COMPATIBILITY.md) must be compatible with the client version of the [opensearch-py](https://github.com/opensearch-project/opensearch-py/blob/main/COMPATIBILITY.md) package on PyPI.

*   The instance's hostname and port number. To learn how, see:

    *   [Creating and managing Amazon OpenSearch Service domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html) for AWS OpenSearch instances.
    *   [Communicate with OpenSearch](https://opensearch.org/docs/latest/getting-started/communicate/) for local OpenSearch instances.

*   The name of the search index on the instance. [Create an index](https://opensearch.org/docs/latest/api-reference/index-apis/create-index/).

    The OpenSearch index that you use must have a schema that is compatible with the schema of the documents
    that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following index schema example for your own needs:

    ```json
    {
        "settings": {
            "index": {
                "knn": true,
                "knn.algo_param.ef_search": 100
            }
        },
        "mappings": {
            "properties": {
                "element_id": {
                    "type": "keyword"
                },
                "text": {
                    "type": "text"
                },
                "embeddings": {
                    "type": "knn_vector",
                    "dimension": 384
                },
                "metadata": {
                    "type": "object",
                    "properties": {
                        "parent_id": {
                            "type": "text"
                        },
                        "page_number": {
                            "type": "integer"
                        },
                        "is_continuation": {
                           "type": "boolean"
                        },
                        "orig_elements": {
                           "type": "text"
                        }
                    }
                }
            }
        }
    }
    ```

    See also:

    *   [Mappings and field types](https://opensearch.org/docs/latest/field-types/)
    *   [Explicit mapping](https://opensearch.org/docs/latest/field-types/#explicit-mapping)
    *   [Dynamic mapping](https://opensearch.org/docs/latest/field-types/#dynamic-mapping)
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

*   If you're using basic authentication to the instance, the user's name and password.

*   If you're using certificates for authentication instead:

    *   The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA.
    *   The path to the combined private key and certificate file, or
    *   The paths to the separate private key and certificate file.

    To learn more, see:

    *   For AWS OpenSearch instances, see [Tutorial: Configure a domain with the internal user database and HTTP basic authentication](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac-http-auth.html) and [Creating a custom endpoint for Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/customendpoint.html).
    *   For local OpenSearch instances, see [Authentication backends](https://opensearch.org/docs/latest/security/authentication-backends/authc-index/), [HTTP basic authentication](https://opensearch.org/docs/latest/security/authentication-backends/basic-authc/), and [Client certificate authentication](https://opensearch.org/docs/latest/security/authentication-backends/client-auth/).

The OpenSearch connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[opensearch]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `OPENSEARCH_HOST` - The hostname and port number, defined as `<hostname>:<port-number>` and represented by `--hosts` (CLI) or `hosts` (Python).
*   `OPENSEARCH_INDEX_NAME` - The name of the search index, represented by `--index-name` (CLI) or `index_name` (Python).

If you're using basic authentication to the instance:

*   `OPENSEARCH_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
*   `OPENSEARCH_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

If you're using certificates for authentication instead:

*   `OPENSEARCH_CA_CERTS` - The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA. This is represented by `--ca-certs` (CLI) or `ca_certs` (Python).
*   `OPENSEARCH_CLIENT_CERT` - The path to the combined private key and certificate file, or the path to just the certificate file. This is represented by `--client-cert` (CLI) or `client_cert` (Python).
*   `OPENSEARCH_CLIENT_KEY` - The path to the private key file, if `OPENSEARCH_CLIENT_CERT` refers to just the certificate file. This is represented by `--client-key` (CLI) or `client_key` (Python).

Additional related settings include:

*   `--use-ssl` (CLI) or `use_ssl=True` (Python) to use SSL for the connection.
*   `--verify-certs` (CLI) or `verify_certs=True` (Python) to verify SSL certificates.
*   `--ssl-show-warn` (CLI) or `ssl_show_warn=True` (Python) to show a warning when verifying SSL certificates is disabled.

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    opensearch \
      --hosts $OPENSEARCH_HOST \
      --username $OPENSEARCH_USERNAME \
      --password $OPENSEARCH_PASSWORD \
      --index-name $OPENSEARCH_INDEX_NAME
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.opensearch import (
      OpenSearchConnectionConfig,
      OpenSearchAccessConfig,
      OpensearchUploadStagerConfig,
      OpensearchUploaderConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=OpenSearchConnectionConfig(
              access_config=OpenSearchAccessConfig(
                  password=os.getenv("OPENSEARCH_PASSWORD"),
                  use_ssl=True
              ),
              hosts=[os.getenv("OPENSEARCH_HOST")],
              username=os.getenv("OPENSEARCH_USERNAME")
          ),
          stager_config=OpensearchUploadStagerConfig(
              index_name=os.getenv("OPENSEARCH_INDEX_NAME")
          ),
          uploader_config=OpensearchUploaderConfig(
              index_name=os.getenv("OPENSEARCH_INDEX_NAME")
          )
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.elasticsearch import (
      ElasticsearchWriteConfig,
  )
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.connector.opensearch import (
      OpenSearchAccessConfig,
      SimpleOpenSearchConfig,
  )
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.opensearch import (
      OpenSearchWriter,
  )


  def get_writer() -> Writer:
      return OpenSearchWriter(
          connector_config=SimpleOpenSearchConfig(
              access_config=OpenSearchAccessConfig(
                  hosts=os.getenv("OPENSEARCH_HOST"),
                  username=os.getenv("OPENSEARCH_USERNAME"),
                  password=os.getenv("OPENSEARCH_PASSWORD"),
              ),
              index_name=os.getenv("OPENSEARCH_INDEX_NAME"),
          ),
          write_config=ElasticsearchWriteConfig(
              batch_size_bytes=15_000_000,
              num_processes=2,
          ),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Overview



Ingest and process data from a storage with a source connector, and move the results to the data storage platform of
your choice using a destination connector.

All the examples for destination connector use local source connector. You can replace it with any other source connector.

We are constantly adding new destination connectors and if you don't see your favorite platform let us know in our community Slack.


# Pinecone



<NewDocument />

Batch process all your records to store structured outputs in Pinecone.

You will need:

The Pinecone prerequisites:

*   A Pinecone account. [Get an account](https://app.pinecone.io/).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/uPoa7PLcuZ4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A Pinecone API key. [Get an API key](https://docs.pinecone.io/guides/get-started/authentication#find-your-pinecone-api-key).

*   A Pinecone serverless index. [Create a serverless index](https://docs.pinecone.io/guides/indexes/create-an-index).

The Pinecone connector dependencies.

```bash
pip install "unstructured-ingest[pinecone]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `PINECONE_API_KEY` - The Pinecone API, represented by `--api-key` (CLI) or `api_key` (Python, in the `PineconeAccessConfig` object).
*   `PINECONE_INDEX_NAME` - The Pinecone serverless index name, represented by `--index-name` (CLI) or `index_name` (Python).

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported.

<iframe width="560" height="315" src="https://www.youtube.com/embed/FZNgYi-fxdg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    pinecone \
      --api-key "$PINECONE_API_KEY" \
      --index-name "$PINECONE_INDEX_NAME" \
      --batch-size 80
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.pinecone import (
      PineconeConnectionConfig,
      PineconeAccessConfig,
      PineconeUploaderConfig,
      PineconeUploadStagerConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=PineconeConnectionConfig(
              access_config=PineconeAccessConfig(
                  api_key=os.getenv("PINECONE_API_KEY")
              ),
              index_name=os.getenv("PINECONE_INDEX_NAME")
          ),
          stager_config=PineconeUploadStagerConfig(),
          uploader_config=PineconeUploaderConfig()
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.connector.pinecone import (
      PineconeAccessConfig,
      PineconeWriteConfig,
      SimplePineconeConfig,
  )
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.pinecone import (
      PineconeWriter,
  )


  def get_writer() -> Writer:
      return PineconeWriter(
          connector_config=SimplePineconeConfig(
              access_config=PineconeAccessConfig(api_key=os.getenv("PINECONE_API_KEY")),
              index_name=os.getenv("PINECONE_INDEX_NAME")
          ),
          write_config=PineconeWriteConfig(batch_size=80),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# PostgreSQL



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a PostgreSQL schema.

Insert query is currently limited to append.

You will need:

The PostgreSQL prerequisites, which include the following settings.

The following video shows for example how to get these settings by using [Amazon RDS for PostgreSQL](https://aws.amazon.com/rds/postgresql/).

<iframe width="560" height="315" src="https://www.youtube.com/embed/QuIlEimejDs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A PostgreSQL instance. [Install PostgreSQL](https://www.postgresql.org/docs/current/tutorial-install.html).

*   The host name and port number for the instance. These values are in the `postgresql.conf` file's `listen_addresses` and `port` settings. This file should be on the same machine as the instance. These values might also already be set as environment variables named `PGHOST` and `PGPORT` on the same machine as the instance.

*   A database in the instance. [Create a database](https://www.postgresql.org/docs/current/tutorial-createdb.html).

*   A table in the database. [Create a table](https://www.postgresql.org/docs/current/tutorial-table.html).

    The table's schema must match the schema of the documents that Unstructured produces.
    Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following table schema example for your own needs:

    <CodeGroup>
      ```sql PostgreSQL
      CREATE TABLE elements (
          id UUID PRIMARY KEY,
          element_id VARCHAR,
          text TEXT,
          embeddings DECIMAL [],
          parent_id VARCHAR,
          page_number INTEGER,
          is_continuation BOOLEAN,
          orig_elements TEXT
      );
      ```

      ```sql PostgreSQL with pgvector 
      CREATE EXTENSION vector;

      CREATE TABLE elements (
          id UUID PRIMARY KEY,
          element_id VARCHAR,
          text TEXT,
          embeddings vector(384),
          parent_id VARCHAR,
          page_number INTEGER,
          is_continuation BOOLEAN,
          orig_elements TEXT
      );
      ```
    </CodeGroup>

    See also:

    *   [CREATE TABLE](https://www.postgresql.org/docs/current/sql-createtable.html) for PostgreSQL
    *   [CREATE TABLE](https://github.com/pgvector/pgvector) for PostrgreSQL with pgvector
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

*   A user in the database, and a password for the user. [Create a user](https://www.postgresql.org/docs/current/sql-createuser.html).

*   Database access for the user. [Give database access to a user](https://www.postgresql.org/docs/current/sql-grant.html).

The PostgreSQL connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[postgres]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `PGHOST` - The host name, represented by `--host` (CLI) or `host` (Python).
*   `PGPORT` - The port number, represented by `--port` (CLI) or `port` (Python).
*   `PGUSER` - The username, represented by `--username` (CLI) or `username` (Python).
*   `PGPASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).
*   `PGDATABASE` - The name of the database, represented by `--database` (CLI) or `database` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Specify which fields to output in the processed data. This can help prevent
  # database record insert issues, where a particular field in the processed data
  # does not match a column in the database table on insert.
  metadata_includes="id,element_id,text,embeddings,type,system,layout_width,\
  layout_height,points,url,version,date_created,date_modified,date_processed,\
  permissions_data,record_locator,category_depth,parent_id,attached_filename,\
  filetype,last_modified,file_directory,filename,languages,page_number,links,\
  page_name,link_urls,link_texts,sent_from,sent_to,subject,section,\
  header_footer_type,emphasized_text_contents,emphasized_text_tags,\
  text_as_html,regex_metadata,detection_class_prob"

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --strategy hi_res \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --metadata-include "$metadata_includes" \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    postgres \
      --username $PGUSER \
      --password $PGPASSWORD \
      --host $PGHOST \
      --port $PGPORT \
      --database $PGDATABASE
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.sql.postgres import (
      PostgresConnectionConfig,
      PostgresAccessConfig,
      PostgresUploaderConfig,
      PostgresUploadStagerConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      # Specify which fields to output in the processed data. This can help prevent
      # database record insert issues, where a particular field in the processed data
      # does not match a column in the database table on insert.
      metadata_includes = [
          "id", "element_id", "text", "embeddings", "type", "system", "layout_width",
          "layout_height", "points", "url", "version", "date_created", "date_modified",
          "date_processed", "permissions_data", "record_locator", "category_depth",
          "parent_id", "attached_filename", "filetype", "last_modified", "file_directory",
          "filename", "languages", "page_number", "links", "page_name", "link_urls",
          "link_texts", "sent_from", "sent_to", "subject", "section", "header_footer_type",
          "emphasized_text_contents", "emphasized_text_tags", "text_as_html", "regex_metadata",
          "detection_class_prob"
      ]

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              metadata_include=metadata_includes,
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=PostgresConnectionConfig(
              access_config=PostgresAccessConfig(password=os.getenv("PGPASSWORD")),
              host=os.getenv("PGHOST"),
              port=os.getenv("PGPORT"),
              username=os.getenv("PGUSER"),
              database=os.getenv("PGDATABASE")
          ),
          stager_config=PostgresUploadStagerConfig(),
          uploader_config=PostgresUploaderConfig()
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Qdrant



<NewDocument />

Batch process all your records to store structured outputs in Qdrant.

You will need:

The Qdrant prerequisites are as follows.

*   The name of the target [collection](https://qdrant.tech/documentation/concepts/collections) on the Qdrant local installation,
    Qdrant server, or Qdrant Cloud cluster.
*   For [Qdrant local](https://github.com/qdrant/qdrant), the path to the local Qdrant installation, for example: `/qdrant/local`
*   For [Qdrant client-server](https://qdrant.tech/documentation/quickstart/), the Qdrant server URL, for example: `http://localhost:6333`
*   For [Qdrant Cloud](https://qdrant.tech/documentation/cloud-intro/):

    *   A [Qdrant account](https://cloud.qdrant.io/login).

    *   A [Qdrant cluster](https://qdrant.tech/documentation/cloud/create-cluster/).

    *   The cluster's URL. To get this URL, do the following:

        1.  Sign in to your Qdrant Cloud account.
        2.  On the sidebar, under **Dashboard**, click **Clusters**.
        3.  Click the cluster's name.
        4.  Note the value of the **Endpoint** field, for example: `https://<random-guid>.<region-id>.<cloud-provider>.cloud.qdrant.io`.

    *   A [Qdrant API key](https://qdrant.tech/documentation/cloud/authentication/#create-api-keys).

The Qdrant connector dependencies.

```bash
pip install "unstructured-ingest[qdrant]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `QDRANT_COLLECTION` - The name of the target collection on the Qdrant local installation,
    Qdrant server, or Qdrant Cloud cluster, represented by `--collection-name` (CLI) or `collection_name` (Python).
*   For Qdrant local, `QDRANT_PATH` - The path to the local Qdrant installation, represented by `--path` (CLI) or `path` (Python).
*   For Qdrant client-server, `QDRANT_URL` - The Qdrant server's URL, represented by `--url` (CLI) or `url` (Python).
*   For Qdrant Cloud:

    *   `QDRANT_URL` - The Qdrant cluster's URL, represented by `--url` (CLI) or `url` (Python).
    *   `QDRANT_API_KEY` - The Qdrant API key, represented by `--api-key` (CLI) or `api_key` (Python).

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported.

This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  # For Qdrant local:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    qdrant-local \
      --path $QDRANT_PATH \
      --collection-name $QDRANT_COLLECTION \
      --batch-size 50 \
      --num-processes 1

  # For Qdrant client-server:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    qdrant-server \
      --url $QDRANT_URL \
      --collection-name $QDRANT_COLLECTION \
      --batch-size 50 \
      --num-processes 1

  # For Qdrant cloud:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    qdrant-cloud \
      --url $QDRANT_URL \
      --api-key $QDRANT_API_KEY \
      --collection-name $QDRANT_COLLECTION \
      --batch-size 50 \
      --num-processes 1
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # For Qdrant local:
  # from unstructured_ingest.v2.processes.connectors.qdrant.local import (
  #     LocalQdrantConnectionConfig,
  #     LocalQdrantAccessConfig,
  #     LocalQdrantUploadStagerConfig,
  #     LocalQdrantUploaderConfig
  # )

  # For Qdrant client-server:
  # from unstructured_ingest.v2.processes.connectors.qdrant.server import (
  #     ServerQdrantConnectionConfig,
  #     ServerQdrantAccessConfig,
  #     ServerQdrantUploadStagerConfig,
  #     ServerQdrantUploaderConfig
  # )

  # For Qdrant Cloud:
  from unstructured_ingest.v2.processes.connectors.qdrant.cloud import (
      CloudQdrantConnectionConfig,
      CloudQdrantAccessConfig,
      CloudQdrantUploadStagerConfig,
      CloudQdrantUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),

          # For Qdrant local:
          # destination_connection_config=LocalQdrantConnectionConfig(
          #     access_config=LocalQdrantAccessConfig(),
          #     path=os.getenv("QDRANT_PATH")
          # ),
          # stager_config=LocalQdrantUploadStagerConfig(),
          # uploader_config=LocalQdrantUploaderConfig(
          #     collection_name=os.gentenv("QDRANT_COLLECTION"),
          #     batch_size=50,
          #     num_processes=1
          # )

          # For Qdrant client-server:
          # destination_connection_config=ServerQdrantConnectionConfig(
          #     access_config=ServerQdrantAccessConfig(),
          #     url=os.getenv("QDRANT_URL")
          # ),
          # stager_config=ServerQdrantUploadStagerConfig(),
          # uploader_config=ServerQdrantUploaderConfig(
          #     collection_name=os.gentenv("QDRANT_COLLECTION"),
          #     batch_size=50,
          #     num_processes=1
          # )

          # For Qdrant cloud:
          destination_connection_config=CloudQdrantConnectionConfig(
              access_config=CloudQdrantAccessConfig(
                  api_key=os.getenv("QDRANT_API_KEY")
              ),
              url=os.getenv("QDRANT_URL")
          ),
          stager_config=CloudQdrantUploadStagerConfig(),
          uploader_config=CloudQdrantUploaderConfig(
              collection_name=os.gentenv("QDRANT_COLLECTION"),
              batch_size=50,
              num_processes=1
          )
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.connector.qdrant import (
      QdrantWriteConfig,
      SimpleQdrantConfig,
      QdrantAccessConfig,
  )
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.qdrant import QdrantWriter

  # This example uses Qdrant Cloud.

  def get_writer() -> Writer:
      return QdrantWriter(
          connector_config=SimpleQdrantConfig(
              url=os.getenv("QDRANT_URL"),
              access_config=QdrantAccessConfig(api_key=os.getenv("QDRANT_API_KEY")),
              collection_name=os.getenv("QDRANT_COLLECTION"),
          ),
          write_config=QdrantWriteConfig(batch_size=80),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# S3



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in an S3 bucket.

You will need:

The Amazon S3 prerequisites:

The following video shows how to fulfill the minimum set of S3 prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/hyDHfhVVAhs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but
not recommended); or generate an AWS STS session token for temporary access, if required by your organization's security
requirements. For more information about prerequisites, see the following:

*   An AWS account. [Create an AWS account](https://aws.amazon.com/free).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/lIdh92JmWtg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An S3 bucket. [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
    Additional approaches are in the following video and in the how-to sections at the end of this page.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/e6w9LwZJFIA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   Anonymous (supported but not recommended) or authenticated access to the bucket.

    *   [Enable anonymous bucket access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user).

        ![Enable anonymous bucket access](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/connectors/s3-anon-access.gif)

    *   [Enable authenticated bucket access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/walkthrough1.html).

        ![Enable authenticated bucket access](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/connectors/s3-auth-access.gif)

*   For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/y4SfQoJpipo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at
    minimum the permission of `s3:PutObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

*   For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account.
    [Create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/MoFTaGJE65Q" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For authenticated access in untrusted environments or enhanced security scenarios, an AWS STS session token for temporary access, in addition to an AWS access key and secret access key. [Create a session token](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_getsessiontoken).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/4_csSXc_GNU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   If the target files are in the root of the bucket, the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
    If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

*   If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has
    authenticated access to the folder as well. [Enable authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

The S3 connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[s3]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `AWS_S3_URL` - The path to the S3 bucket or folder, formatted as `s3://my-bucket/` (if the files are in the bucket's root) or `s3://my-bucket/my-folder/`.

*   If the bucket does not have anonymous access enabled, provide the AWS credentials:

    *   `AWS_ACCESS_KEY_ID` - The AWS access key ID for the authenticated AWS IAM user, represented by `--key` (CLI) or `key` (Python).
    *   `AWS_SECRET_ACCESS_KEY` - The corresponding AWS secret access key, represented by `--secret` (CLI) or `secret` (Python).
    *   `AWS_TOKEN` - If required, the AWS STS session token for temporary access, represented by `--token` (CLI) or `token` (Python).

*   If the bucket has anonymous access enabled for reading from the bucket, set `--anonymous` (CLI) or `anonymous=True` (Python) instead.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    s3 \
      --remote-url $AWS_S3_URL \
      --key $AWS_ACCESS_KEY_ID \
      --secret $AWS_SECRET_ACCESS_KEY
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.fsspec.s3 import (
      S3ConnectionConfig,
      S3AccessConfig,
      S3UploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=S3ConnectionConfig(
              access_config=S3AccessConfig(
                  key=os.getenv("AWS_ACCESS_KEY_ID"),
                  secret=os.getenv("AWS_SECRET_ACCESS_KEY")
              )
          ),
          uploader_config=S3UploaderConfig(remote_url=os.getenv("AWS_S3_URL"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.s3 import S3AccessConfig, S3WriteConfig, SimpleS3Config
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.fsspec.s3 import (
      S3Writer,
  )

  def get_writer() -> Writer:
      return S3Writer(
          connector_config=SimpleS3Config(
              access_config=S3AccessConfig(anon=True),
              remote_url=os.getenv("AWS_S3_URL"),
          ),
          write_config=S3WriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None,
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2.  Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3.  Browse to the existing bucket and open it.

4.  Click the **Permissions** tab.

5.  In the **Bucket policy** area, click **Edit**.

6.  In the **Policy** text area, copy the following JSON-formatted policy.
    To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this policy, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-bucket-name>` in two places with the name of your bucket.

    ```json
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::<my-account-id>:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::<my-bucket-name>",
                    "arn:aws:s3:::<my-bucket-name>/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    ```

7.  Click **Save changes**.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2.  [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3.  Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
    To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this script, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-unique-bucket-name>` with the name of your bucket.
    *   Replace `<us-east-1>` with your AWS Region.

    ```bash
    #!/bin/bash

    # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
    ACCOUNT_ID="<my-account-id>"
    BUCKET_NAME="<my-unique-bucket-name>"
    REGION="<us-east-1>"

    # Temporary filename for the bucket policy.
    # Do not change this variable.
    POLICY_FILE="bucket_policy.json"

    # Create the bucket.
    aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

    # Wait for the bucket to exist.
    echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
    aws s3api wait bucket-exists --bucket $BUCKET_NAME

    # Check if the wait command was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket '$BUCKET_NAME' has been fully created."
    else
        echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
        exit 1
    fi

    # Remove the "block public policy" bucket access setting.
    aws s3api put-public-access-block \
        --bucket $BUCKET_NAME \
        --public-access-block-configuration \
        '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

    # Check if the operation was successful.
    if [ $? -eq 0 ]; then
        echo "The block public policy access setting was removed from '$BUCKET_NAME'."
    else
        echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
        exit 1
    fi

    # Create the bucket policy.
    cat << EOF > $POLICY_FILE
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::$BUCKET_NAME",
                    "arn:aws:s3:::$BUCKET_NAME/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    EOF

    # Apply the bucket policy.
    aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

    # Check if the policy application was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket policy was applied to '$BUCKET_NAME'."
    else
        echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
        exit 1
    fi

    # Verify the applied policy.
    echo "Verifying the applied policy:"
    aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

    # Remove the temporary bucket policy file.
    rm $POLICY_FILE
    ```

4.  Run the script, for example:

    ```bash
    sh create-s3-bucket.sh
    ```

5.  After the bucket is created, you can delete the script file, if you want.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
    the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    ```yaml
    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

    Parameters:
      BucketName:
        Type: String
        Description: 'Name of the S3 bucket to create'

    Resources:
      MyS3Bucket:
        Type: 'AWS::S3::Bucket'
        Properties:
          BucketName: !Ref BucketName
          PublicAccessBlockConfiguration:
            BlockPublicAcls: true
            BlockPublicPolicy: false
            IgnorePublicAcls: true
            RestrictPublicBuckets: true

      BucketPolicy:
        Type: 'AWS::S3::BucketPolicy'
        Properties:
          Bucket: !Ref MyS3Bucket
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Sid: AllowAllAuthenticatedUsersInAccount
                Effect: Allow
                Principal:
                  AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
                Action:
                  - 's3:GetObject'
                  - 's3:PutObject'
                  - 's3:ListBucket'
                  - 's3:DeleteObject'
                Resource:
                  - !Sub 'arn:aws:s3:::${BucketName}'
                  - !Sub 'arn:aws:s3:::${BucketName}/*'

    Outputs:
      BucketName:
        Description: 'Name of the created S3 bucket'
        Value: !Ref MyS3Bucket
    ```

2.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3.  Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4.  Click **Create stack > With new resources (standard)**.

5.  On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6.  Click **Choose file**, and browse to and select the YAML file from your local machine.

7.  Click **Next**.

8.  Enter a unique **Stack name** and **BucketName**.

9.  Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.


# SFTP



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in SFTP.

You will need:

The SFTP prerequisites:

*   The SFTP server hostname, port, username, and password.

    SFTP servers are offered by several vendors. For example, the following video shows how to create and set up an SFTP server by using AWS Transfer Family:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/3_HHSnoFsoM" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The directory path to start accessing data from, specified as `sftp://<path>/<to>/<directory>`.

See the [SFTP documentation](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol).

The SFTP connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[sftp]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `SFTP_HOST` - The SFTP hostname, represented by `--host` (CLI) or `host` (Python).
*   `SFTP_PORT` - The SFTP port number, represented by `--port` (CLI) or `port` (Python).
*   `SFTP_REMOTE_URL` - The directory path to start accessing data from, represented by `--remote-url` (CLI) or `remote_url` (Python).
*   `SFTP_USERNAME` - The SFTP username, represented by `--username` (CLI) or `username` (Python).
*   `SFTP_PASSWORD` - The SFTP password, represented by `--password` (CLI) or `password` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --strategy hi_res \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
    sftp \
      --host $SFTP_HOST \
      --port $SFTP_PORT \
      --username $SFTP_USERNAME \
      --password $SFTP_PASSWORD \
      --remote-url $SFTP_REMOTE_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.sftp import (
      SftpConnectionConfig,
      SftpAccessConfig,
      SftpUploaderConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=SftpConnectionConfig(
              access_config=SftpAccessConfig(password=os.getenv("SFTP_PASSWORD")),
              host=os.getenv("SFTP_HOST"),
              port=os.getenv("SFTP_PORT"),
              username=os.getenv("SFTP_USERNAME")
          ),
          uploader_config=SftpUploaderConfig(remote_url=os.getenv("SFTP_REMOTE_URL"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# SingleStore



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a SingleStore account.

You will need:

The SingleStore prerequisites:

*   A SingleStore deployment, database, and table. [Learn how](https://www.singlestore.com/blog/how-to-get-started-with-singlestore/).
*   The hostname for the SingleStore deployment.
*   The port for the host.
*   The username for the deployment.
*   The password for the user.
*   The name of the database in the deployment.
*   The name of the table in the database.

To get the values for the hostname, port, username, and password:

1.  In your SingleStore account's dashboard sidebar, click **Deployments**.
2.  From the drop-down list at the top of the **Deployments** page, select your deployment.
3.  On the **Overview** tab, in the **Compute** area, in the **Connect** drop-down list for your deployment, select
    **Your App**.
4.  If a **Create User** dialog box appears, note the **User** name and **Password** values.
5.  In the **Connect to Workspace** pane's **Your App** tab, note the string in the following format:

```
<user-name>:<password>@<host>:<port>
```

*   `<user-name>` is the username.
*   `<password>` is the user's password.
*   `<host>` is the workspace's hostname.
*   `<post>` is the host's port.

To get the values for the database and table names:

1.  In your SingleStore dashboard's sidebar, click **Deployments**.
2.  From the drop-down list at the top of the **Deployments** page, select your deployment.
3.  On the **Databases** tab, note **Name** of your database.
4.  Click the database and, on the **Tables** tab, note the name of your table.

The SingleStore connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[singlestore]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `SINGLESTORE_HOST` - The hostname for the SingleStore deployment, represented by `--host` (CLI) or `host` (Python).
*   `SINGLESTORE_PORT` - The port for the host, represented by `--port` (CLI) or `port` (Python).
*   `SINGLESTORE_USER` - The username for the deployment, represented by `--user` (CLI) or `user` (Python).
*   `SINGLESTORE_PASSWORD` - The password for the user, represented by `--password` (CLI) or `password` (Python).
*   `SINGLESTORE_DB` - The name of the database in the deployment, represented by `--database` (CLI) or `database` (Python).
*   `SINGLESTORE_TABLE` - The name of the table in the database, represented by `--table-name` (CLI) or `table_name` (Python).

The table's schema must match the schema of the documents that Unstructured produces.

During insertion, JSON objects are flattened, and underscores are placed between nested object names.
For example, the following JSON object matches a column in the table named `data_source_date_created`:

```json
{
    "...": "...",
    "data_source": {
        "date_created": "1719963233.949"
    },
    "...": "..."
}
```

The `metadata` object itself is also flattened in a similar way. However, for nested objects in `metadata`,
the column in the table does not start with `metadata_`. For example, the following JSON object matches a
column in the table named `last_modified`:

```json
{
    "...": "...",
    "metadata": {
        "...": "...",
        "last_modified": "2022-12-16T17:04:16-05:00",
        "...": "..."
    },
    "...": "..."
}
```

Unstructured cannot provide a table schema that is guaranteed to work in all
circumstances. This is because these schemas will vary based on your source files' types; how you
want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

You can adapt the following table schema example for your own needs:

```sql
CREATE TABLE elements (
    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
    element_id TEXT,
    text TEXT,
    embeddings Vector(384),
    parent_id TEXT,
    page_number TEXT,
    is_continuation BOOLEAN,
    orig_elements TEXT
);
```

See also:

*   [CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table/)
    in the SingleStore documentation
*   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    singlestore \
      --host $SINGLESTORE_HOST \
      --port $SINGLESTORE_PORT \
      --user $SINGLESTORE_USER \
      --password $SINGLESTORE_PASSWORD \
      --database $SINGLESTORE_DB \
      --table-name $SINGLESTORE_TABLE
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.singlestore import (
      SingleStoreConnectionConfig,
      SingleStoreAccessConfig,
      SingleStoreUploadStagerConfig,
      SingleStoreUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=SingleStoreConnectionConfig(
              host=os.getenv("SINGLESTORE_HOST"),
              port=os.getenv("SINGLESTORE_PORT"),
              user=os.getenv("SINGLESTORE_USER"),
              database=os.getenv("SINGLESTORE_DB"),
              access_config=SingleStoreAccessConfig(
                  password=os.getenv("SINGLESTORE_PASSWORD")
              )
          ),
          stager_config=SingleStoreUploadStagerConfig(drop_empty_cols=True),
          uploader_config=SingleStoreUploaderConfig(table_name=os.getenv("SINGLESTORE_TABLE"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Snowflake



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in Snowflake.

You will need:

The Snowflake prerequisites:

*   A Snowflake [account](https://signup.snowflake.com/) and its [identifier](https://docs.snowflake.com/user-guide/admin-account-identifier).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/6HNmxQWqmgg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Snowflake [username and its password](https://docs.snowflake.com/user-guide/admin-user-management#creating-users) in the account.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/5hPrJNEa4aA?start=64&end=267" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Snowflake [hostname and its port number](https://docs.snowflake.com/sql-reference/functions/system_allowlist) in the account.

*   The name of the Snowflake [database](https://docs.snowflake.com/sql-reference/sql/create-database) in the account.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/KrXCgpsRd0A" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The name of the [schema](https://docs.snowflake.com/sql-reference/sql/create-schema) in the database.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/4SoS_c6rT_U?start=97&end=157" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The name of the [table](https://docs.snowflake.com/sql-reference/sql/create-table) in the schema.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/ZaZzqFMGe1g?end=169" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The Snowflake connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[snowflake]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `SNOWFLAKE_ACCOUNT` - The ID of the Snowflake account, represented by `--account` (CLI) or `account` (Python).
*   `SNOWFLAKE_USER`  - The name of the Snowflake user, represented by `--user` (CLI) or `user` (Python).
*   `SNOWFLAKE_PASSWORD`  - The user's password, represented by `--password` (CLI) or `password` (Python).
*   `SNOWFLAKE_HOST`  - The hostname for the Snowflake account, represented by `--host` (CLI) or `host` (Python).
*   `SNOWFLAKE_PORT`  - The host's port number, represented by `--port` (CLI) or `port` (Python).
*   `SNOWFLAKE_DATABASE`  - The name of the Snowflake database, represented by `--database` (CLI) or `database` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    snowflake \
      --account $SNOWFLAKE_ACCOUNT \
      --user $SNOWFLAKE_USER \
      --password $SNOWFLAKE_PASSWORD \
      --host $SNOWFLAKE_HOST \
      --port $SNOWFLAKE_PORT \
      --database $SNOWFLAKE_DATABASE \
      --schema PUBLIC \
      --table-name elements \
      --batch-size 50
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.sql.snowflake import (
      SnowflakeConnectionConfig,
      SnowflakeAccessConfig,
      SnowflakeUploadStagerConfig,
      SnowflakeUploaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=SnowflakeConnectionConfig(
              access_config=SnowflakeAccessConfig(password=os.getenv("SNOWFLAKE_PASSWORD")),
              account=os.getenv("SNOWFLAKE_ACCOUNT"),
              user=os.getenv("SNOWFLAKE_USER"),
              host=os.getenv("SNOWFLAKE_HOST"),
              port=os.getenv("SNOWFLAKE_PORT"),
              database=os.getenv("SNOWFLAKE_DATABASE"),
              schema="PUBLIC"
          ),
          stager_config=SnowflakeUploadStagerConfig(),
          uploader_config=SnowflakeUploaderConfig(
              batch_size=50,
              table_name="elements"
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# SQLite



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a SQLite schema.

Insert query is currently limited to append.

You will need:

The SQLite prerequisites:

*   A SQLite instance. [Download and install SQLite](https://www.sqlitetutorial.net/download-install-sqlite/).
*   A SQLite database. [Create a database](https://www.sqlite.org/quickstart.html).
*   The path to the database's `.db` file.
*   A table in the database. [Create a table](https://www.sqlite.org/cli.html).

    The table's schema must match the schema of the documents that Unstructured produces.
    Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following table schema example for your own needs:

    ```sql SQLite
    CREATE TABLE elements (
        id TEXT PRIMARY KEY,
        element_id TEXT,
        text TEXT,
        embeddings TEXT,
        parent_id TEXT,
        page_number INTEGER,
        is_continuation INTEGER,
        orig_elements TEXT
    );
    ```

    See also:

    *   [CREATE TABLE](https://www.sqlite.org/lang_createtable.html) for SQLite
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `SQLITE_DB_PATH` - The path to the database's `.db` file, represented by `--database` (CLI) or `database` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Specify which fields to output in the processed data. This can help prevent
  # database record insert issues, where a particular field in the processed data
  # does not match a column in the database table on insert.
  metadata_includes="id,element_id,text,embeddings,type,system,layout_width,\
  layout_height,points,url,version,date_created,date_modified,date_processed,\
  permissions_data,record_locator,category_depth,parent_id,attached_filename,\
  filetype,last_modified,file_directory,filename,languages,page_number,links,\
  page_name,link_urls,link_texts,sent_from,sent_to,subject,section,\
  header_footer_type,emphasized_text_contents,emphasized_text_tags,\
  text_as_html,regex_metadata,detection_class_prob"

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --strategy hi_res \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --metadata-include "$metadata_includes" \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    sqlite \
      --database-path $SQLITE_DB_PATH
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.sql.sqlite import (
      SQLiteConnectionConfig,
      SQLiteAccessConfig,
      SQLiteUploaderConfig,
      SQLiteUploadStagerConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      # Specify which fields to output in the processed data. This can help prevent
      # database record insert issues, where a particular field in the processed data
      # does not match a column in the database table on insert.
      metadata_includes = [
          "id", "element_id", "text", "embeddings", "type", "system", "layout_width",
          "layout_height", "points", "url", "version", "date_created", "date_modified",
          "date_processed", "permissions_data", "record_locator", "category_depth",
          "parent_id", "attached_filename", "filetype", "last_modified", "file_directory",
          "filename", "languages", "page_number", "links", "page_name", "link_urls",
          "link_texts", "sent_from", "sent_to", "subject", "section", "header_footer_type",
          "emphasized_text_contents", "emphasized_text_tags", "text_as_html", "regex_metadata",
          "detection_class_prob"
      ]

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              metadata_include=metadata_includes,
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=SQLiteConnectionConfig(
              access_config=SQLiteAccessConfig(),
              database_path=os.getenv("SQLITE_DB_PATH")
          ),
          stager_config=SQLiteUploadStagerConfig(),
          uploader_config=SQLiteUploaderConfig()
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Vectara



Process all your records using `unstructured-ingest` to store structured outputs locally on your filesystem and upload those to a Vectara corpus. If you don’t yet have a Vectara account, [sign up](https://console.vectara.com/signup/) for one now.

The upstream connector can be any of the ones supported, but for convenience here, showing a sample command using the upstream local connector.

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  # Chunking is optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --num-processes 2 \
      --verbose \
    vectara \
      --customer-id $VECTARA_CUSTOMER_ID \
      --oauth-client-id $VECTARA_OAUTH_CLIENT_ID \
      --oauth-secret $VECTARA_OAUTH_SECRET \
      --corpus-name test-corpus-vectara
  ```

  ```python Python
  import os

  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.connector.vectara import (
      SimpleVectaraConfig,
      VectaraAccessConfig,
      WriteConfig,
  )
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.vectara import (
      VectaraWriter,
  )


  def get_writer() -> Writer:
      return VectaraWriter(
          connector_config=SimpleVectaraConfig(
              access_config=VectaraAccessConfig(
                  oauth_client_id=os.getenv("VECTARA_OAUTH_CLIENT_ID"),
                  oauth_secret=os.getenv("VECTARA_OAUTH_SECRET"),
              ),
              customer_id=os.getenv("VECTARA_CUSTOMER_ID"),
              corpus_name="test-corpus-vectara",
          ),
          write_config=WriteConfig(),
      )


  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For a full list of the options the Unstructured Ingest CLI accepts check `unstructured-ingest <upstream connector> vectara --help`.

NOTE: Keep in mind that you will need to have all the appropriate extras and dependencies for the file types of the documents contained in your data storage platform if you’re running this locally. You can find more information about this in the [installation guide](/open-source/installation/overview).


# Weaviate



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Batch process all your records to store structured outputs in a Weaviate database.

<iframe width="560" height="315" src="https://www.youtube.com/embed/uqUrH8ksI0I" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

You will need:

The Weaviate prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/Ldb7PZU-pR4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

1.  A Weaviate database instance. The following information assumes that you have a Weaviate Cloud (WCD) account with a Weaviate database cluster in that account.
    [Create a WCD account](https://weaviate.io/developers/wcs/quickstart#create-a-wcd-account). [Create a database cluster](https://weaviate.io/developers/wcs/quickstart#create-a-weaviate-cluster). For other database options, [learn more](https://weaviate.io/developers/weaviate/installation).

2.  The URL and API key for the database cluster. [Get the URL and API key](https://weaviate.io/developers/wcs/quickstart#explore-the-details-panel).

3.  A collection in the database cluster. Note the name of the collection, also known as the collection's *class name*. [Create a collection](https://weaviate.io/developers/wcs/tools/collections-tool).

    The schema of the collection that you use must match the data that Unstructured writes to it. Otherwise, you might get unexpected results or errors.
    Unstructured cannot provide a schema that is guaranteed to work for everyone in all circumstances. This is because these schemas will vary based on
    your source files' types; how you want Unstructured to partition, chunk, and generate embeddings;
    any custom post-processing code that you run; and other factors.

    You can adapt the following collection schema example for your own needs:

    ```json
    {
        "class": "Elements",
        "properties": [
            {
                "name": "element_id",
                "dataType": ["text"]
            },
            {
                "name": "text",
                "dataType": ["text"]
            },
            {
                "name": "embeddings",
                "dataType": ["number[]"]
            },
            {
                "name": "metadata",
                "dataType": ["object"],
                "nestedProperties": [
                    {
                        "name": "parent_id",
                        "dataType": ["text"]
                    },
                    {
                        "name": "page_number",
                        "dataType": ["text"]
                    },
                    {
                        "name": "is_continuation",
                        "dataType": ["boolean"]
                    },
                    {
                        "name": "orig_elements",
                        "dataType": ["text"]
                    }
                ]
            }
        ]
    }
    ```

    See also :

    *   [Collection schema](https://weaviate.io/developers/weaviate/config-refs/schema)
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

The Weaviate connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[weaviate]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `WEAVIATE_URL` - THE REST endpoint for the Weaviate database cluster, represented by `--host-url` (CLI) or `host_url` (Python).

*   `WEAVIATE_API_KEY` - The API key for the database cluster, represented by `--api-key` (CLI) or `api_key` (Python).

    <Note>For the CLI, the `--api-key` option here is part of the `weaviate` command. For Python, the `api_key` parameter here is part of the `WeaviateAccessConfig` object.</Note>

*   `WEAVIATE_COLLECTION_CLASS_NAME` - The name of the collection in the database, represented by `--class-name` (CLI) or `class_name` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding is optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --strategy fast \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    weaviate \
      --host-url $WEAVIATE_URL \
      --api-key $WEAVIATE_API_KEY \
      --class-name $WEAVIATE_COLLECTION_CLASS_NAME
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.weaviate import (
      WeaviateConnectionConfig,
      WeaviateAccessConfig,
      WeaviateUploaderConfig,
      WeaviateUploadStagerConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=WeaviateConnectionConfig(
              access_config=WeaviateAccessConfig(
                  api_key=os.getenv("WEAVIATE_API_KEY")
              ),
              host_url=os.getenv("WEAVIATE_URL"),
              class_name=os.getenv("WEAVIATE_COLLECTION_CLASS_NAME")
          ),
          stager_config=WeaviateUploadStagerConfig(),
          uploader_config=WeaviateUploaderConfig()
      ).run()
  ```

  ```python Python Ingest v1
  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.connector.weaviate import (
      SimpleWeaviateConfig,
      WeaviateAccessConfig,
      WeaviateWriteConfig,
  )
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import LocalRunner
  from unstructured_ingest.runner.writers.base_writer import Writer
  from unstructured_ingest.runner.writers.weaviate import (
      WeaviateWriter,
  )

  import os

  def get_writer() -> Writer:
      return WeaviateWriter(
          connector_config=SimpleWeaviateConfig(
              access_config=WeaviateAccessConfig(
                  api_key=os.getenv("WEAVIATE_API_KEY")
              ),
              host_url=os.getenv("WEAVIATE_URL"),
              class_name=os.getenv("WEAVIATE_COLLECTION_CLASS_NAME"),
          ),
          write_config=WeaviateWriteConfig(),
      )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      writer = get_writer()
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")),
              strategy="hi_res",
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface",
              api_key=None
          ),
          writer=writer,
          writer_kwargs={},
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Chunking configuration



A common chunking configuration is a critical element in the data processing pipeline, particularly when creating embeddings and populating vector databases with the results. This configuration defines the parameters governing the segmentation of text into meaningful chunks, whether at the document, paragraph, or sentence level. It plays a pivotal role in determining the size and structure of these chunks, ensuring that they align with the specific requirements of downstream tasks, such as embedding generation and vector database population. By carefully configuring chunking parameters, users can optimize the granularity of data segments, ultimately contributing to more cohesive and contextually rich results. This is crucial for tasks like natural language processing and text analysis, as well as for the efficient storage and retrieval of embeddings in vector databases, enhancing the quality and relevance of the results.

## Configs

*   <Icon icon="v" /><Icon icon="2" />  `chunk_api_key`: If `chunk_by_api` is set to `True`, requests that are sent to the Unstructured API will use this Unstructured API key to make authenticated calls.

*   <Icon icon="v" /><Icon icon="2" />  `chunk_by_api`: Default: `False`. If set to `True`, uses Unstructured API services to run chunking. If set to `False`, runs chunking locally.

*   <Icon icon="v" /><Icon icon="2" />  `chunk_combine_text_under_n_chars`: Combine consecutive chunks when the first does not exceed length `n` and the second will fit without exceeding the hard-maximum length. Only operative for the `by_title` chunking strategy.

*   <Icon icon="v" /><Icon icon="1" />  `chunk_elements`: <Icon icon="ban" />  **Deprecated!** `False` (default) to not run chunking as part of the ingest process. This option is deprecated in favor of the `chunking_strategy` option. Setting this to `True` has the same effect as `chunking_strategy=by_title`.

*   <Icon icon="v" /><Icon icon="2" />  `chunk_include_orig_elements`: `True` when chunking to add the original elements consolidated to form each chunk to `.metadata.orig_elements` on that chunk.

*   <Icon icon="v" /><Icon icon="2" />  `chunk_max_characters`: Default: `500`. The hard-maximum chunk length. No chunk will exceed this length. An oversized element will be divided by text-splitting to fit this window.

*   <Icon icon="v" /><Icon icon="2" />  `chunk_multipage_selections`: `True` to ignore page boundaries when chunking such that elements from two different pages can appear in the same chunk. Only operative for the `by_title` chunking strategy.

*   <Icon icon="v" /><Icon icon="2" />  `chunk_new_after_n_chars`: The soft-maximum chunk length. Another element will not be added to a chunk of `n` length even when it would fit without exceeding the hard-maximum length.

*   <Icon icon="v" /><Icon icon="2" />  `chunk_overlap`: Default: `0`. Prefix each chunk's text with the last overlap of `n` characters from the prior chunk. Only applies to oversized chunks divided by text-splitting. To apply overlap to non-oversized chunks, use `chunk_overlap_all`.

*   <Icon icon="v" /><Icon icon="2" />  `chunk_overlap_all`: Applies overlap to chunks formed from whole elements as well as those formed by text-splitting oversized elements. The overlap length is taken from the `chunk_overlap` value.

*   <Icon icon="v" /><Icon icon="2" />  `chunking_endpoint`: If `chunk_by_api` is set to `True`, chunking requests are sent to this Unstructured API URL. By default, this URL is the Unstructured Serverless API URL: `https://api.unstructuredapp.io/general/v0/general`.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `chunking_strategy`: One of `basic` or `by_title`. When omitted, no chunking is performed. The `basic` strategy maximally fills each chunk with whole elements, up the specified size limits as specified by `max_characters` and `new_after_n_chars`. A single element that exceeds this length is divided into two or more chunks using text-splitting. A `Table` element is never combined with any other element and appears as a chunk of its own or as a sequence of `TableChunk` elements splitting is required. The `by_title` behaviors are the same except that section and optionally page boundaries are respected such that two consecutive elements from different sections appear in separate chunks.

*   <Icon icon="v" /><Icon icon="1" />  `combine_text_under_n_chars`: Default: `max_characters`. Combines small elements (for example a series of `Title` elements) until a section reaches a length of `n `characters. Only operative for the `by_title` chunking strategy. Defaults to max\_characters which combines chunks whenever space allows. Specifying `0` for this argument suppresses combining of small chunks.

*   <Icon icon="v" /><Icon icon="1" />  `include_orig_elements`: Default: `True`. Adds the document elements consolidated to form each chunk to the `orig_elements` field in `metadata`. Setting this to `False` allows for somewhat smaller payloads when you don't need that metadata.

*   <Icon icon="v" /><Icon icon="1" />  `max_characters`: Default: `500`. Combine elements into chunks no larger than `n` characters. This is a hard maximum: no chunk with text longer than this value will appear in the output stream.

*   <Icon icon="v" /><Icon icon="1" />  `multipage_sections`: Default: `True`. When `False`, in addition to section boundaries, page boundaries are also respected. Only operative for the `by_title` chunking strategy.

*   <Icon icon="v" /><Icon icon="1" />  `new_after_n_chars`: Default: `max_characters (off)`. Cuts off new chunks once they reach a length of `n` characters. This is a soft maximum. Defaults to `max_characters` when not specified, which effectively disables any soft window. Specifying `0` for this argument causes each element to appear in a chunk by itself (although an element with text longer than `max_characters` will be still be divided into two or more chunks using text splitting).

*   <Icon icon="v" /><Icon icon="1" />  `overlap`: Default: `0`. Prefix each chunk's text with the last overlap of `n` characters from the prior chunk. Only applies to oversized chunks divided by text-splitting. To apply overlap to non-oversized chunks, use `overlap_all`.

*   <Icon icon="v" /><Icon icon="1" />  `overlap_all`: Applies overlap to chunks formed from whole elements as well as those formed by text-splitting oversized elements. The overlap length is taken from the `overlap` value.

<Note>Chunking will fail if you set both `partition_by_api` to `False` and `chunking_strategy` to `by_page` or `by_similarity`. However, the rest of your data processing pipeline should be unaffected by this setting.</Note>


# Embedding configuration



A common embedding configuration is a critical component that allows for dynamic selection of embedders and their associated parameters to create vectors from data. This configuration provides the flexibility to choose from various embedding models and fine-tune parameters to optimize the quality and characteristics of the resulting vectors. It enables users to tailor the embedding process to the specific needs of their data and downstream applications, ensuring that the generated vectors effectively capture semantic relationships and contextual information within the dataset.

## Configs

*   <Icon icon="v" /><Icon icon="1" />  `api_key`: The API key to use, if one is required to generate the embeddings through an API service, such as OpenAI.

*   <Icon icon="v" /><Icon icon="1" />  `aws_access_key_id`: The AWS access key ID to be used for AWS-based embedders, such as Amazon Bedrock.

*   <Icon icon="v" /><Icon icon="1" />  `aws_region`: The AWS Region ID to be used for AWS-based embedders, such as Amazon Bedrock.

*   <Icon icon="v" /><Icon icon="1" />  `aws_secret_access_key`: The AWS secret access key to be used for AWS-based embedders, such as Amazon Bedrock.

*   <Icon icon="v" /><Icon icon="2" />  `embedding_provider`: The embedding provider to use while doing embedding. Available values include `aws-bedrock`, `huggingface`, `mixedbread-ai`, `octoai`, `openai`, `togetherai`, `vertexai`, and `voyageai`.

*   <Icon icon="v" /><Icon icon="2" />  `embedding_api_key`: The API key to use, if one is required to generate the embeddings through an API service, such as OpenAI.

*   <Icon icon="v" /><Icon icon="2" />  `embedding_aws_access_key_id`: The AWS access key ID to be used for AWS-based embedders, such as Amazon Bedrock.

*   <Icon icon="v" /><Icon icon="2" />  `embedding_aws_region`: The AWS Region ID to be used for AWS-based embedders, such as Amazon Bedrock.

*   <Icon icon="v" /><Icon icon="2" />  `embedding_aws_secret_access_key`: The AWS secret access key to be used for AWS-based embedders, such as Amazon Bedrock.

*   <Icon icon="v" /><Icon icon="2" />  `embedding_model_name`: The specific model to use for the embedding provider, if necessary.

*   <Icon icon="v" /><Icon icon="1" />  `model_name`: The specific model to use for the embedding provider, if necessary.

*   <Icon icon="v" /><Icon icon="1" />  `provider`: The embedding provider to use while doing embedding. Available values include `aws-bedrock`, `huggingface`, `mixedbread-ai`, `octoai`, `openai`, `togetherai`, `vertexai`, and `voyageai`.

<Icon icon="v" /><Icon icon="1" />  The default `model_name` values unless otherwise specified are:

*   `aws-bedrock`: None

*   `huggingface`: `sentence-transformers/all-MiniLM-L6-v2`, with 384 dimensions

*   `mixedbread-ai`: `mixedbread-ai/mxbai-embed-large-v1`, with 1024 dimensions

*   `octoai`: `thenlper/gte-large`, with 1024 dimensions

*   `openai`: `text-embedding-ada-002`, with 1536 dimensions

*   `togetherai`: `togethercomputer/m2-bert-80M-8k-retrieval`, with 768 dimensions

*   `vertexai`: `textembedding-gecko@001`, with 768 dimensions

*   `voyageai`: None


# Fsspec configuration



A common fsspec configuration is a shared set of parameters and settings utilized by connectors responsible for managing cloud-based file-system content. These configurations enable connectors to interact with cloud storage systems consistently, specifying details such as authentication credentials, access permissions, storage locations, and optimization options. By standardizing these fsspec configurations, connectors can seamlessly interface with various cloud-based file systems, promoting interoperability and efficient management of remote data storage across different cloud providers.

## File Storage Configuration

Fsspec configuration is an extension of the File Storage Config:

*   <Icon icon="v" /><Icon icon="1" />  `recursive`: Default: `False`. If a directory path is provided, whether further folders should be traversed recursively.

*   <Icon icon="v" /><Icon icon="1" />  `remote_url`: Path to the content on the remote system, prefixed with the protocol (for example, `s3://...`).

*   <Icon icon="v" /><Icon icon="1" />  `uncompress`: Default: `False`. Whether to uncompress tar and zip files when downloaded. Will ignore these otherwise.

## Configs

The following are configurations set on the fsspec config itself:

*

*   <Icon icon="v" /><Icon icon="1" />  `access_config`: A dictionary of access keyword arguments (`kwargs`) that might be needed to provide access to the cloud provider associated with the data.

The following are generated for the user after the config is initialized:

*   <Icon icon="v" /><Icon icon="1" />  `dir_path` and `file_path`: The raw remote path is parsed into the directory and file if it exists.

*   <Icon icon="v" /><Icon icon="1" />  `path_without_protocol`: The file path on the remote source without the protocol.

*   <Icon icon="v" /><Icon icon="1" />  `protocol`: The protocol is extracted from the raw remote path passed in (for example, `s3` from `s3://...`).


# Overview



The configurations listed in this section are common to all connectors.
These configurations act as instructions for how to collect, process, and store data.

<iframe width="560" height="315" src="https://www.youtube.com/embed/A_kqK2KHTdg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

Some connectors implement only version 2 (v2) of the connector specification; some connectors implement only version 1 (v1); and some connectors provide both v2 and v1 implementations.

*   Connectors that implement both v2 and v1 are prefaced by <Icon icon="v" /><Icon icon="2" />, <Icon icon="v" /><Icon icon="1" />.

*   v2-only implementations are prefaced by <Icon icon="v" /><Icon icon="2" />.

*   v1-only implementations are prefaced by <Icon icon="v" /><Icon icon="1" />.

If you're looking for parameters specific to a connector (such as authentication options)—or to determine whether a connector provides a v2 implementation, a v1 implementation, or both—browse the
corresponding connector's source code:

*   [v2 non-fsspec connectors](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/unstructured_ingest/v2/processes/connectors)
*   [v2 fsspec connectors](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/unstructured_ingest/v2/processes/connectors/fsspec)
*   [v1 non-fsspec connectors](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/unstructured_ingest/connector)
*   [v1 fsspec connectors](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/unstructured_ingest/connector/fsspec)
*   [v1 Notion connector](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/unstructured_ingest/connector/notion)


# Partition configuration



A standard partition configuration is a collection of parameters designed to oversee document partitioning, whether executed through API integration or by the unstructured library on a local system. These parameters serve a dual role, encompassing those passed to the partition method for the initial segmentation of documents and those responsible for coordinating data after processing, including the dynamic metadata associated with each element.

## Configs for Partitioning

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `additional_partition_args`: A JSON string representation of any values to pass through to the `partition` function.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `encoding`: The encoding method used to decode the text input. By default, UTF-8 will be used.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `ocr_languages`: The languages present in the document, for use in partitioning, OCR, or both. Multiple languages indicate that the text could be in any of the specified languages.

*   <Icon icon="v" /><Icon icon="1" />  `pdf_infer_table_structure`: <Icon icon="ban" />  **Deprecated!** Use `skip_infer_table_types` to opt out of table extraction for any file type. If `False` and `strategy=hi_res`, no `Table` elements will be extracted from PDF files regardless of `skip_infer_table_types` contents.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `skip_infer_table_types`: List of document types that you want to skip table extraction with.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `strategy`: Default: `auto`. The strategy to use for partitioning PDF and image files. Uses a layout detection model if set to `hi_res`. Otherwise, partitioning simply extracts the text from the document and processes it.

## Configs for the Process

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `api_key`: If `partition_by_api` is set to `True`, requests that are sent to the Unstructured API will use this Unstructured API key to make authenticated calls.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `fields_include`: Fields to include in the output JSON. By default, the following fields are included: `element_id`, `text`, `type`, `metadata`, and `embeddings`.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `flatten_metadata`: Default: `False`. If set to `True`, the hierarchical metadata structure is flattened to have all values exist at the top level.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `hi_res_model_name`: The model to use when `strategy` is set to `hi_res`. Available values are `layout_v1.0.0` (the default) and `yolox`.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `metadata_exclude`: Values from the `metadata` field to exclude from the output.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `metadata_include`: If provided, only the specified fields are preserved in the `metadata` output.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `partition_by_api`: Default: `False`. If set to `True`, uses Unstructured API services to run partitioning. If set to `False`, runs partitioning locally.

*   <Icon icon="v" /><Icon icon="2" /> ,<Icon icon="v" /><Icon icon="1" />  `partition_endpoint`: If `partition_by_api` is set to `True`, partitioning requests are sent to this Unstructured API URL.


# Permissions configuration



A common permissions configuration is leveraged to extract user access data associated with the content from the source data provider. Currently supported for only the SharePoint connector.

## Configs

*   <Icon icon="v" /><Icon icon="1" />  `application_id`: The SharePoint application client ID.

*   <Icon icon="v" /><Icon icon="1" />  `client_cred`: The SharePoint application client secret.

*   <Icon icon="v" /><Icon icon="1" />  `tenant`: The SharePoint tenant ID.


# Processor configuration



A common process configuration plays a pivotal role in overseeing the entire ingest process, encompassing various aspects to ensure efficiency and reliability. This configuration includes parameters for managing a pool of workers, which allows for parallelization of the ingest process to maximize throughput. Additionally, it addresses the storage and retrieval of intermediate results, supporting caching strategies that enable replayability in case of errors or interruptions. By configuring worker pools, the process can efficiently handle multiple tasks concurrently, enhancing performance. Furthermore, saving intermediate results allows for the resumption of the process from a known state, reducing data loss and ensuring robustness in the face of unexpected errors or system failures. This comprehensive configuration promotes an organized and resilient approach to data ingestion, improving overall system reliability and efficiency.

## Configs

*   <Icon icon="v" /><Icon icon="2" />  `disable_parallelism`: `True` if the `INGEST_DISABLE_PARALLELISM` environment variable is set to `True` (case-insensitive), otherwise `False` (the default).

*   <Icon icon="v" /><Icon icon="2" />  `download_only`: Default: `False`. If set to `True`, downloads any files that are not already present in the connector's specified download directory (`download_dir`), or `work_dir` if `download_dir` is not specified, or the default file path for `work_dir` if `work_dir` is not specified.

*   <Icon icon="v" /><Icon icon="2" />  `max_connections`: The maximum of connections allowed when running an asynchronous step in the ingest pipeline.

*   <Icon icon="v" /><Icon icon="2" />  `max_docs`: The maximum number of documents to be pulled in for the entire ingest process.

*   <Icon icon="v" /><Icon icon="2" />, <Icon icon="v" /><Icon icon="1" />  `num_processes`: Default: `2`. For every step that can use a pool of workers to increase throughput, how many workers to configure in the pool.

*   <Icon icon="v" /><Icon icon="1" />  `output_dir`: Where the final results will be located when the process is finished. This is regardless of whether a destination is configured. If a directory is not specified, by default a folder named `structured-output`, relative to the current working directory, is used.

*   <Icon icon="v" /><Icon icon="2" />  `preserve_downloads`: When set to `False` (the default), will remove downloaded files after they are successfully processed.

*   <Icon icon="v" /><Icon icon="2" />, <Icon icon="v" /><Icon icon="1" />  `raise_on_error`: `False `(default) for any single document that fails in the process, causes the error to be logged but allows for all other documents to proceed in the process. If `True`, causes the entire process to fail and raise the error if any one document fails.

*   <Icon icon="v" /><Icon icon="2" />  `re_download`: When set to `False` (the default), will not re-download files if they are already present in the download directory. Setting this to `True` downloads the files even if they exist in the download directory. `True` is useful in cases where for some reason you suspect that the files in the remote location might have changed, and you want to be sure that the file copies in the download directory are the most current ones.

*   <Icon icon="v" /><Icon icon="2" />, <Icon icon="v" /><Icon icon="1" />  `reprocess`: Default: `False`. If set to `True`, will ignore all content that may have been cached and rerun each step.

*   <Icon icon="v" /><Icon icon="2" />  `tqdm`: When set to `False` (the default), will not show a progress bar.

*   <Icon icon="v" /><Icon icon="2" />  `uncompress`: When set to `False` (the default), will not uncompress any archived ZIP or TAR files. If set to `True`, and the specified partitioner provides support for compressed ZIP and TAR files, these files will be uncompressed.

*   <Icon icon="v" /><Icon icon="2" />, <Icon icon="v" /><Icon icon="1" />  `verbose`: Default: `False`. If set to `True`, debug logging should be included in the output.

*   <Icon icon="v" /><Icon icon="2" />, <Icon icon="v" /><Icon icon="1" />  `work_dir`: The file path for where intermediate results should be saved. If a file path is not set, a default file path will be used relative to the user's home directory (`/home/username/.cache/unstructured/ingest/pipeline` or `C:\Users\username\.cache\unstructured\ingest\pipeline`).


# Read configuration



A shared read configuration serves as a universal set of parameters that are consistent across all source connectors, providing a standardized way to access and retrieve documents from various sources. This configuration typically includes settings such as the download directory, which specifies the location where retrieved documents are stored. By maintaining common parameters like the download directory, users can streamline their data extraction processes, making it easier to manage and organize the downloaded documents irrespective of the source connector in use. This promotes consistency, ease of maintenance, and a more straightforward integration process when working with multiple source connectors within a system.

## Configs

*   <Icon icon="v" /><Icon icon="1" />  `download_dir`: What location to download the files to. When run via the CLI, a default location will be used if one is not provided.

*   <Icon icon="v" /><Icon icon="1" />  `download_only`: Default: `False`. If set to `True`, the process will exit right after all the files are downloaded and omit any future steps such as partitioning and uploading to a destination.

*   <Icon icon="v" /><Icon icon="1" />  `max_docs`: How many documents are pulled in for a single process.

*   <Icon icon="v" /><Icon icon="1" />  `preserve_downloads`: `False` (default) for the process to delete the downloaded content at the end if everything finished without error. By setting this to `True`, those files will be preserved.

*   <Icon icon="v" /><Icon icon="1" />  `re_download`: `False` (default) for the process to skip downloads if the files already exist in the download directory. By setting this to `True`, it will force the files to be re downloaded regardless of them existing already.


# Retry strategy configuration



A common retry strategy configuration is a critical element in enhancing the robustness and resiliency of a system, especially when dealing with temporary network issues. This configuration typically includes parameters that define how the system should respond to network-related errors, such as connection timeouts or transient failures. It specifies the number of retry attempts, the time intervals between retries, and the conditions under which retries should be initiated. By configuring an effective retry strategy, a system can automatically recover from network disruptions, increasing its ability to withstand intermittent issues and ensuring a more reliable and uninterrupted operation. This strategy is vital for maintaining system stability, data integrity, and minimizing downtime in the face of unpredictable network hiccups.

## Configs

*   <Icon icon="v" /><Icon icon="1" />  `max_retries`: How many times to retry before stopping. By default, there is no limit to the number of tries.

*   <Icon icon="v" /><Icon icon="1" />  `max_retry_time`: If provided, will stop trying after this much time has elapsed.


# Ingest

The Unstructured Python Ingest library is a powerful tool designed to coordinate the process of pulling data from data providers, partitioning the content, and pushing that new content to a desired location. This technical documentation will provide an in-depth understanding of the Python Ingest library, including its features, architecture, installation, configuration, usage, API reference, troubleshooting, examples, and more.

The following 3-minute video shows how to use the Unstructured Ingest Python library to send multiple PDFs from a local directory in batches to be ingested by Unstructured API services for processing:

<iframe width="560" height="315" src="https://www.youtube.com/embed/tSKHFXsBQ-c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The following 5-minute video goes into more detail about the various components of the Unstructured Ingest Python library:

<iframe width="560" height="315" src="https://www.youtube.com/embed/A_kqK2KHTdg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

## Library Documentation

[Source Connectors](/open-source/ingest/source-connectors/overview)

Connect to your favorite data storage platforms for an effortless batch processing of your files.

[Destination Connectors](/open-source/ingest/destination-connectors)

Connect to your favorite data storage platforms to write you ingest results to.

[Ingest Configuration](/open-source/ingest/ingest-configuration)

Each configuration used when generating an ingest process.

## Features

The Unstructured Ingest CLI and Unstructured Ingest Python offer the following key features:

*   Data Ingestion: Facilitates the ingestion of data from various sources, such as databases, APIs, files, or streaming services.

*   Partitioning: Efficiently partitions data to extract relevant text data.

*   Customization: Allows users to define data sources, ingestion processes, and destination targets.

*   Fault Tolerance: Provides mechanisms for handling errors and retries during data ingestion.

*   Scalability: Scales horizontally to accommodate large volumes of data.

*   Logging: Offers comprehensive logging and monitoring capabilities to track the ingestion process.

## Architecture

The Unstructured Python Ingest library follows a modular architecture comprising the following components:

*   Source Connectors: These components are responsible for fetching data from external sources, which can include databases, web services, file systems, or data streams.

*   Partitioning Engine: This component optimally partitions the incoming data into dedicated `Elements` for processing and distribution.

*   Reformatters: Optional steps supported to manipulate the partitioned content output, such as chunking and adding embeddings.

*   Destination Connectors: These components send the partitioned data to the desired destination, which could be a database, data warehouse, cloud storage, or any other user-defined target.

*   The library’s modular architecture provides flexibility and extensibility, allowing users to integrate custom components and adapt the library to their specific needs.

![Unstructured ingest CLI pipeline\_diagram](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/open-source/unstructured_ingest_cli_pipeline_diagram.png)

## Installation

To install the Unstructured Ingest CLI and the Unstructured Ingest Python library, follow these steps:

1.  Run `pip install unstructured-ingest` to install the latest version of the Ingest CLI and Ingest Python library.

2.  For specific connectors, run `pip install "unstructured-ingest[CONNECTOR_DEPS]"` where `CONNECTOR_DEPS` references the extra dependency label for a particular connector. For example, `pip install "unstructured-ingest[s3]"` will install the dependencies to interact with the s3 connectors. If these aren’t installed before hand, a convenient error message will be printed for you when you run the Unstructured Ingest CLI for the first time, prompting you with the correct pip command to run.

3.  Once installed, you can run `unstructured-ingest --help` to get all the available commands.

<Info>To migrate from older, deprecated versions of the Ingest CLI or Ingest Python library that used `pip install unstructured`, see the [migration guide](/ingestion/overview#migration-guide).</Info>

## Configuration

The Unstructured Python Ingest library requires configuration to define data sources, ingestion processes, and destination targets. For the CLI, configuration is done through the various cli parameters supported. When the library is run in python, those parameters that are exposed in the CLI map to python config classes, which are described in more detail in the configs section.

## Generate Python code examples

You can connect any available source connector to any available destination connector. However, the source connector code examples in the
documentation show connecting only to the local destination connector. Similarly, the destination connector code examples in the
documentation show connecting only to the local source connector.

To quickly generate an Unstructured Ingest Python library code example that connects *any* available source connector to *any* available destination connector,
do the following:

1.  Open the [Unstructured Ingest Code Generator](https://huggingface.co/spaces/MariaK/unstructured-pipeline-builder) webpage.

2.  Select your input (source) location type from the **Get unstructured documents from** drop-down list.

3.  Select your output (destination) location type from the **Upload RAG-ready documents to** drop-down list.

4.  Select your chunking strategy from the **Chunking strategy** drop-down list:

    *   **None** - Do not chunk the data elements' content.
    *   **basic** - Combine sequential data elements to maximally fill each chunk. However, do not mix `Table` and non-`Table` elements in the same chunk.
    *   **by\_title** - Use the `basic` strategy and also preserve section boundaries. Optionally preserve page boundaries as well.
    *   **by\_page** - Use the `basic` strategy and also preserve page boundaries.
    *   **by\_similarity** - Use the `sentence-transformers/multi-qa-mpnet-base-dot-v1` embedding model to identify topically similar sequential elements and combine them into chunks. This strategy is availably only when calling Unstructured API services.

    To learn more, see [Chunking strategies](/api-reference/api-services/chunking) and [Chunking configuration](/api-reference/ingest/ingest-configuration/chunking-configuration).

5.  For any chunking strategy other than **None**:

    *   Enter your chunk size in the **Chunk size (characters)** box, or leave the default of **1000** characters.
    *   If you need to apply overlapping to the chunks, enter the chunk overlap size in the **Chunk overlap (characters)** box, or leave default of **20** characters.

    To learn more, see [Chunking configuration](/api-reference/ingest/ingest-configuration/chunking-configuration).

6.  To generate vector embeddings, select the provider in the **Embedding provider** drop-down list.

    To learn more, see [Embedding configuraton](/api-reference/ingest/ingest-configuration/embedding-configuration).

7.  Click **Generate code**.

8.  Copy the example code from the **Generated Code** pane into your code project.

9.  The code example will contain one or more environment variables that you must set for the code to run correctly. To learn what to
    set these variables to, click the documentation links that are below the **Generated Code** pane.


# Airtable



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Airtable to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your data and store structured outputs locally on your filesystem.

You will need:

The Airtable connector prerequisites:

*   An [Airtable](https://www.airtable.com/) account. [Create a free Airtable account](https://airtable.com/signup).

*   An Airtable personal access token. [Create a personal access token](https://support.airtable.com/docs/creating-and-using-api-keys-and-access-tokens).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/c3gvIWAAcTY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The ID of the Airtable base to access. [Create a base](https://www.airtable.com/guides/build/create-a-base). [Get a base's ID](https://support.airtable.com/docs/finding-airtable-ids#finding-base-url-ids).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/rU347YVMm2M" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The ID of the table to access in the base. [Create a table](https://www.airtable.com/guides/build/create-a-table). [Get a table's ID](https://support.airtable.com/docs/finding-airtable-ids#finding-base-url-ids).

*   The ID of the view to access in the table. [Create a view](https://www.airtable.com/guides/build/create-custom-views-of-data). [Get a view's ID](https://support.airtable.com/docs/finding-airtable-ids#finding-base-url-ids).

By default, Unstructured processes all tables from all bases within an Airtable organization. You can limit the
tables that Unstructured ingests data from within Airtable by specifying a list of Airtable paths.
An Airtable path uses the following structure: `base_id/table_id(optional)/view_id(optional)`

For example, given the following example Airtable URL:

```text
https://airtable.com/appr9nKeXLAtg6bgn/tblZ8uT1GY7NLbWit/viwDcpzf9dP0Gqz5J
```

*   The base's ID is `appr9nKeXLAtg6bgn`. The base's path is `appr9nKeXLAtg6bgn`.
*   The table's ID is `tblZ8uT1GY7NLbWit`. The table's path is `appr9nKeXLAtg6bgn/tblZ8uT1GY7NLbWit`.
*   The view's ID is `viwDcpzf9dP0Gqz5J`. The view's path is `appr9nKeXLAtg6bgn/tblZ8uT1GY7NLbWit/viwDcpzf9dP0Gqz5J`.

You can call the Airtable API to get lists of available IDs for Airtable bases, tables, and views in bulk, as follows:

*   [Base IDs](https://airtable.com/developers/web/api/list-bases)
*   [Table and view IDs](https://airtable.com/developers/web/api/get-base-schema)
*   [Base, table, and view IDs](https://pyairtable.readthedocs.io/en/latest/metadata.html)

The Airtable connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[airtable]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `AIRTABLE_TOKEN` - The Airtable personal access token, represented by `--personal-access-token` (CLI) or `personal_access_token` (Python).
*   `AIRTABLE_PATHS` - The list of Airtable paths to process, represented by `--list-of-paths` (CLI) or `list_of_paths` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    airtable \
      --personal-access-token $AIRTABLE_PERSONAL_ACCESS_TOKEN \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --reprocess \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.airtable import (
      AirtableIndexerConfig,
      AirtableDownloaderConfig,
      AirtableConnectionConfig,
      AirtableAccessConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig

  # Chunking is optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=AirtableIndexerConfig(list_of_paths=[os.getenv("AIRTABLE_PATHS")]),
          downloader_config=AirtableDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=AirtableConnectionConfig(
              access_config=AirtableAccessConfig(
                  personal_access_token=os.getenv("AIRTABLE_TOKEN")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.airtable import AirtableAccessConfig, SimpleAirtableConfig
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import AirtableRunner

  if __name__ == "__main__":
      runner = AirtableRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
          ),
          connector_config=SimpleAirtableConfig(
              access_config=AirtableAccessConfig(
                  personal_access_token=os.getenv("AIRTABLE_PERSONAL_ACCESS_TOKEN")
              ),
              list_of_paths=[os.getenv("AIRTABLE_PATHS")]
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Astra DB



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Astra DB to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Astra DB connector prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/kupQrnCQjPw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An Astra account. [Create or sign in to an Astra account](https://astra.datastax.com/).
*   A database in the Astra account. [Create a database in an account](https://docs.datastax.com/en/astra-db-classic/databases/manage-create.html).
*   An application token for the database. [Create a database application token](https://docs.datastax.com/en/astra-db-serverless/administration/manage-application-tokens.html).
*   A namespace in the database. [Create a namespace in a database](https://docs.datastax.com/en/astra-db-serverless/databases/manage-namespaces.html#create-namespace).
*   A collection in the namespace. [Create a collection in a namespace](https://docs.datastax.com/en/astra-db-serverless/databases/manage-collections.html#create-collection).

The Astra DB connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[astradb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `ASTRA_DB_API_ENDPOINT` - The API endpoint for the Astra DB database, represented by `--api-endpoint` (CLI) or `api_endpoint` (Python). To get the endpoint, see the **Database Details > API Endpoint** value on your database's **Overview** tab.
*   `ASTRA_DB_APPLICATION_TOKEN` - The database application token value for the database, represented by `--token` (CLI) or `token` (Python). To get the token, see the **Database Details > Application Tokens** box on your database's **Overview** tab.
*   `ASTRA_DB_KEYSPACE` - The name of the keyspace for the database, represented by `--keyspace` (CLI) or `keyspace` (Python).
*   `ASTRA_DB_COLLECTION` - The name of the collection for the keyspace, represented by `--collection-name` (CLI) or `collection_name` (Python).
*   `ASTRA_DB_EMBEDDING_DIMENSIONS` - The number of dimensions in the collection, represented by `--embedding-dimension` (CLI) or `embedding_dimension` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    astradb \
      --api-endpoint $ASTRA_DB_API_ENDPOINT \
      --token $ASTRA_DB_APPLICATION_TOKEN \
      --keyspace $ASTRA_DB_KEYSPACE \
      --collection-name $ASTRA_DB_COLLECTION \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.astradb import (
      AstraDBAccessConfig,
      AstraDBConnectionConfig,
      AstraDBDownloaderConfig,
      AstraDBIndexerConfig,
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=AstraDBIndexerConfig(
              collection_name=os.getenv("ASTRA_DB_COLLECTION"),
              keyspace=os.getenv("ASTRA_DB_KEYSPACE"),           
          ),
          downloader_config=AstraDBDownloaderConfig(
              collection_name=os.getenv("ASTRA_DB_COLLECTION"),
              keyspace=os.getenv("ASTRA_DB_KEYSPACE"),
          ),
          source_connection_config=AstraDBConnectionConfig(
              access_config=AstraDBAccessConfig(
                  token=os.getenv("ASTRA_DB_APPLICATION_TOKEN"),
                  api_endpoint=os.getenv("ASTRA_DB_API_ENDPOINT"),
              ),
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
          ),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR")),
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.runner import AstraDBRunner
  from unstructured_ingest.connector.astradb import (
      SimpleAstraDBConfig,
      AstraDBAccessConfig
  )
  from unstructured_ingest.interfaces import (
      ProcessorConfig,
      ReadConfig,
      PartitionConfig
  )

  if __name__ == "__main__":
      runner = AstraDBRunner(
          processor_config=ProcessorConfig(
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR")
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
          ),
          connector_config=SimpleAstraDBConfig(
              access_config=AstraDBAccessConfig(
                  token=os.getenv("ASTRA_DB_APPLICATION_TOKEN"),
                  api_endpoint=os.getenv("ASTRA_DB_API_ENDPOINT")
              ),
              keyspace=os.getenv("ASTRA_DB_KEYSPACE"),
              collection_name=os.getenv("ASTRA_DB_COLLECTION")
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Azure



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Azure Storage to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Azure Storage account prerequisites.

The following video shows how to fulfill the minimum set of Azure Storage account prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/Vl3KCphlh9Y" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

<Note>
  If you are generating an SAS token as shown in the preceding video, be sure to set the following permissions:

  *   **Read** and **List** for reading from the container only.
  *   **Write** and **List** for writing to the container only.
  *   **Read**, **Write**, and **List** for both reading from and writing to the container.
</Note>

Here are some more details about these prerequisites:

*   An Azure account. To create one, [learn how](https://azure.microsoft.com/pricing/purchase-options/azure-account).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/2bQ6WiJ1ncA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An Azure Storage account, and a container within that account. [Create a storage account](https://learn.microsoft.com/azure/storage/common/storage-account-create). [Create a container](https://learn.microsoft.com/azure/storage/blobs/blob-containers-portal).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/AhuNgBafmUo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    <iframe width="560" height="315" src="https://www.youtube.com/embed/xmndjYnGvcs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Azure Storage remote URL, using the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

    For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
    Azure Storage remote URL would be `az://my-container/my-folder/`.

*   An SAS token (recommended), access key, or connection string for the Azure Storage account.  [Create an SAS token (recommended)](https://learn.microsoft.com/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens). [Get an access key](https://learn.microsoft.com/azure/storage/common/storage-account-keys-manage#view-account-access-keys). [Get a connection string](https://learn.microsoft.com/azure/storage/common/storage-configure-connection-string#configure-a-connection-string-for-an-azure-storage-account).

    Create an SAS token (recommended):

    <iframe width="560" height="315" src="https://www.youtube.com/embed/X6cmJ2IbVzo?start=240&end=370" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    Get an access key or connection string:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/muMmcwVfFqs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The Azure Storage account connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[azure]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `AZURE_STORAGE_REMOTE_URL` - The Azure Storage remote URL, represented by `--remote-url` (CLI) or `remote_url` (Python).

    The remote URL takes the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

    For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
    Azure Storage remote URL would be `az://my-container/my-folder/`.

*   `AZURE_STORAGE_ACCOUNT_NAME` - The name of the Azure Storage account, represented by `--account-name` (CLI) or `account_name` (Python).

*   `AZURE_STORAGE_ACCOUNT_KEY`, `AZURE_STORAGE_CONNECTION_STRING`, or `AZURE_STORAGE_SAS_TOKEN` - The name of the key, connection string, or SAS token for the Azure Storage account, depending on your security configuration, represented by `--account-key` (CLI) or `account_key` (Python), `--connection-string` (CLI) or `connection_string` (Python), and `--sas_token` (CLI) or `sas_token` (Python), respectively.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    azure \
      --remote-url $AZURE_STORAGE_REMOTE_URL \
      --account-name $AZURE_STORAGE_ACCOUNT_NAME \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.azure import (
      AzureIndexerConfig,
      AzureDownloaderConfig,
      AzureConnectionConfig,
      AzureAccessConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=AzureIndexerConfig(remote_url=os.getenv("AZURE_STORAGE_REMOTE_URL")),
          downloader_config=AzureDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=AzureConnectionConfig(
              access_config=AzureAccessConfig(
                  account_name=os.getenv("AZURE_STORAGE_ACCOUNT_NAME"),
                  account_key=os.getenv("AZURE_STORAGE_ACCOUNT_KEY")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()

  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.azure import (
      AzureAccessConfig,
      SimpleAzureBlobStorageConfig,
  )
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import AzureRunner

  if __name__ == "__main__":
      runner = AzureRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleAzureBlobStorageConfig(
              access_config=AzureAccessConfig(
                  account_name=os.getenv("AZURE_STORAGE_ACCOUNT_NAME"),
              ),
              remote_url=os.getenv("AZURE_STORAGE_REMOTE_URL"),
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Biomed



Connect Biomed to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to
store structured outputs locally on your filesystem.

This connector allows you to extract Biomedical documents from the supported FTP directories:

*   [https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa\_pdf](https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf)

Make sure to have the Biomed dependencies installed:

```bash Shell
pip install "unstructured-ingest[biomed]"
```

You need to provide the path, from which the documents should be downloaded. For example, to download the documents in
the path: `https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/07/`, set the `path` parameter to `oa_pdf/07/`

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  unstructured-ingest \
    biomed \
      --path "oa_pdf/07/07/sbaa031.073.PMC7234218.pdf" \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --preserve-downloads \
      --strategy hi_res
  ```

  ```python Python
  from unstructured_ingest.connector.biomed import (
      SimpleBiomedConfig,
  )
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import BiomedRunner

  if __name__ == "__main__":
      runner = BiomedRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          connector_config=SimpleBiomedConfig(path="oa_pdf/07/07/sbaa031.073.PMC7234218.pdf"),
      )
      runner.run()
  ```
</CodeGroup>

For a full list of the options the Unstructured Ingest CLI accepts check `unstructured-ingest biomed --help`.


# Box



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Box to your preprocessing pipeline, and use the Unstructured CLI or Python to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Box prerequisites:

1.  Access to the [Developer Console](https://app.box.com/developers/console) from your [Box enterprise account](https://account.box.com/signup/enterprise-plan) or [Box developer account](https://account.box.com/signup/developer).

2.  A Box Custom App in your Box account, set up to use **Server Authentication (with JWT)**. See [Setup with JWT](https://developer.box.com/guides/authentication/jwt/jwt-setup/).

3.  The appropriate application scopes and advanced features set up for the Box Custom App, as follows:

    *   In the Box Custom App, on the **Configuration** tab, under **Application Scopes**, check the box titled **Write all files and folders stored in Box**.
    *   In the Box Custom App, on the **Configuration** tab, under **Advanced Features**, check the box titled **Make API calls using the as-user header**.
    *   Click **Save Changes** before continuing.

4.  Authorization by a Box Admin in your Box account for the Box Custom App. See [App Authorization](https://developer.box.com/guides/authentication/jwt/jwt-setup/#app-authorization).

5.  Access by your Box account's source or target [folder](https://app.box.com/folder/0) to your Box Custom App, as follows:

    *   In the Box Custom App, on the **General Settings** tab, copy the **Service Account ID** (which takes the form `AutomationUser_<your-app-service-id>_<a-random-string@boxdevedition.com`).
    *   **Share** your Box account's target folder with the copied service account's email address as a **Co-owner** or **Editor**.
    *   Note the remote URL to the target folder, which takes the format `box://<path/to/folder/in/account>`.

6.  The private key configuration JSON file for the Box Custom App. To download this file, in the Box Custom App, on the **Configuration** tab, under **Add and Manage Public Keys**, click **Generate a Public/Private Keypair**. Store the downloaded private key configuration JSON file in a secure location. Do not share it with others.

The Box connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[box]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `BOX_APP_CONFIG_PATH` - The local path to the downloaded private key configuration JSON file for the Box Custom App, represented by `--box-app-config` (CLI) or `box_app_config` (Python).
*   `BOX_REMOTE_URL` - The remote URL to the target folder, represented by `--remote-url` (CLI) or `remote_url` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    box \
      --box-app-config $BOX_APP_CONFIG_PATH \
      --remote-url $BOX_REMOTE_URL \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --recursive \
      --verbose \
      --partition-by-api \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --api-key $UNSTRUCTURED_API_KEY \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.box import (
      BoxAccessConfig,
      BoxConnectionConfig,
      BoxIndexerConfig,
      BoxDownloaderConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=BoxIndexerConfig(remote_url=os.getenv("BOX_REMOTE_URL")),
          downloader_config=BoxDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=BoxConnectionConfig(
              access_config=BoxAccessConfig(
                  box_app_config=os.getenv("BOX_APP_CONFIG_PATH")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.box import BoxAccessConfig, SimpleBoxConfig
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import BoxRunner

  if __name__ == "__main__":
      runner = BoxRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleBoxConfig(
              remote_url=os.getenv("BOX_REMOTE_URL"),
              recursive=True,
              access_config=BoxAccessConfig(
                  box_app_config=os.getenv("BOX_APP_CONFIG_PATH")
              ),
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Confluence



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Confluence to your preprocessing pipeline, and use the Unstructured CLI or Python to batch process all your spaces and pages and store structured outputs locally on your filesystem.

You will need:

The Confluence prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/tXu6S6Pd7Dc" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A [Confluence account](https://www.atlassian.com/software/confluence).
*   The URL for the target Confluence instance.
*   A [Confluence user](https://confluence.atlassian.com/doc/add-and-invite-users-138313.html) with
    the correct [permissions](https://support.atlassian.com/confluence-cloud/docs/what-are-confluence-cloud-permissions-and-restrictions/) to
    access the target spaces and pages in the Confluence instance.
*   A [Confluence API token](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/) for the target Confluence user.
*   Optionally, the names of the specific [spaces](https://support.atlassian.com/confluence-cloud/docs/navigate-spaces/) in the Confluence instance to access.

The Confluence connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[confluence]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `CONFLUENCE_URL` - The URL of the Confluence instance, represented by `--url` (CLI) or `url` (Python).
*   `CONFLUENCE_API_TOKEN` - The value of the Confluence API token for authenticating with the Confluence instance, represented by `--api-token` (CLI) or `api_token` (Python).
*   `CONFLUENCE_USER_EMAIL` - The user's email address for authenticating with the Confluence instance, represented by `--user-email` (CLI) or `user_email` (Python).

Additional settings include:

*   `--spaces` (CLI) or `spaces` (Python): Optionally, the list of the names of the specific spaces to access, expressed as a comma-separated list of strings (CLI) or an array of strings (Python), with each string representing a space's name. The default is no specific spaces, if not otherwise specified.
*   `--max-num-of-spaces` (CLI) or `max_num_of_spaces` (Python): Optionally, the maximum number of spaces to access, expressed as an integer. The default value is `500` if not otherwise specified.
*   `--max-num-of-docs-from-each-space` (CLI) or `max_num_of_docs_from_each_space` (Python): Optionally, the maximum number of documents to access from each space, expressed as an integer. The default value is `100` if not otherwise specified.

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    confluence \
      --url $CONFLUENCE_URL \
      --spaces luke,paul \
      --user-email $CONFLUENCE_USER_EMAIL \
      --api-token $CONFLUENCE_API_TOKEN \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --chunking-strategy by_title \
      --embedding-provider huggingface
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.confluence import (
      ConfluenceIndexerConfig,
      ConfluenceDownloaderConfig,
      ConfluenceConnectionConfig,
      ConfluenceAccessConfig
  )

  # Chunking and embedding are optional.

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=ConfluenceIndexerConfig(
              spaces=["luke", "paul"]
          ),
          downloader_config=ConfluenceDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=ConfluenceConnectionConfig(
              access_config=ConfluenceAccessConfig(api_token=os.getenv("CONFLUENCE_API_TOKEN")),
              url=os.getenv("CONFLUENCE_URL"),
              user_email=os.getenv("CONFLUENCE_USER_EMAIL")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.confluence import ConfluenceAccessConfig, SimpleConfluenceConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import ConfluenceRunner

  if __name__ == "__main__":
      runner = ConfluenceRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              metadata_exclude=["filename", "file_directory", "metadata.data_source.date_processed"],
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleConfluenceConfig(
              access_config=ConfluenceAccessConfig(
                  api_token=os.getenv("CONFLUENCE_API_TOKEN"),
              ),
              user_email=os.getenv("CONFLUENCE_USER_EMAIL"),
              url=os.getenv("CONFLUENCE_URL"),
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Couchbase



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Couchbase Database to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Couchbase database prerequisites.

*   For the [Unstructured Platform](/platform/overview), only Couchbase Capella clusters are supported.
*   For [Unstructured Ingest](/ingestion/overview), Couchbase Capella clusters and local Couchbase server deployments are supported.

<iframe width="560" height="315" src="https://www.youtube.com/embed/9-RIBmIdi70" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

For Couchbase Capella, you will need:

*   A Couchbase Capella account. [Create an account](https://docs.couchbase.com/cloud/get-started/create-account.html).
*   A Couchbase Capella cluster. [Create a cluster](https://docs.couchbase.com/cloud/clusters/create-database.html).
*   A [bucket, scope, and collection](https://docs.couchbase.com/cloud/clusters/data-service/about-buckets-scopes-collections.html).
*   A cluster connection string. [Learn how](https://docs.couchbase.com/cloud/get-started/connect.html).
*   Incoming IP address allowance. [Learn how](https://docs.couchbase.com/cloud/clusters/allow-ip-address.html).
*   A username and password. [Learn how](https://docs.couchbase.com/cloud/clusters/manage-database-users.html).

For a local Couchbase server, you will need:

*   Installation of a local Couchbase server. [Learn how](https://docs.couchbase.com/server/current/getting-started/start-here.html).
*   Connection details to the local Couchbase server. [Learn how](https://docs.couchbase.com/server/current/guides/connect.html).

To learn more about how to set up a Couchbase cluster and play with data, refer to this [tutorial](https://developer.couchbase.com/tutorial-quickstart-flask-python).

The Couchbase DB connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[couchbase]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables are required for the Couchbase Connector:

*   `CB_CONN_STR` - The Connection String for the Couchbase server, represented by `--connection-string` (CLI) or `connection_string` (Python).
*   `CB_USERNAME` - The username for the Couchbase server, represented by `--username` (CLI) or `username` (Python).
*   `CB_PASSWORD` - The password for the Couchbase server, represented by `--password` (CLI) or `password` (Python).
*   `CB_BUCKET` - The name of the bucket in the Couchbase server, represented by `--bucket` (CLI) or `bucket` (Python).
*   `CB_SCOPE` - The name of the scope in the bucket, represented by `--scope` (CLI) or `scope` (Python).
*   `CB_COLLECTION` - The name of the collection in the scope, represented by `--collection` (CLI) or `collection` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    couchbase \
      --metadata-exclude filename,file_directory,metadata.data_source.date_processed \
      --connection-string "$COUCHBASE_CONNECTION_STRING" \
      --username $COUCHBASE_USERNAM \
      --password $COUCHBASE_PASSWORD \
      --bucket $COUCHBASE_BUCKET \
      --scope $COUCHBASE_SCOPE \
      --collection $COUCHBASE_COLLECTION \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.couchbase import (
      CouchbaseIndexerConfig,
      CouchbaseDownloaderConfig,
      CouchbaseConnectionConfig,
      CouchbaseAccessConfig,
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=CouchbaseIndexerConfig(batch_size=100),
          downloader_config=CouchbaseDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=CouchbaseConnectionConfig(
              access_config=CouchbaseAccessConfig(
                  password=os.getenv("CB_PASSWORD")
              ),
              username=os.getenv("CB_USERNAME"),
              connection_string=os.getenv("CB_CONN_STR"),
              bucket=os.getenv("CB_BUCKET"),
              scope=os.getenv("CB_SCOPE"),
              collection=os.getenv("CB_COLLECTION")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig()
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Databricks Volumes



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Databricks Volumes to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Databricks Volumes prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/rNZpwa1-g7M" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The preceding video shows how to use Databricks personal access tokens (PATs), which are supported only for [Unstructured Ingest](/ingestion/overview).

To learn how to use Databricks-managed service principals, which are supported by both the [Unstructured Platform](/platform/overview) and Unstructured Ingest,
see the additional videos later on this page.

*   The Databricks workspace URL. Get the workspace URL for
    [AWS](https://docs.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids),
    [Azure](https://learn.microsoft.com/azure/databricks/workspace/workspace-details#workspace-instance-names-urls-and-ids),
    or [GCP](https://docs.gcp.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids).

    Examples:

    *   AWS: `https://<workspace-id>.cloud.databricks.com`
    *   Azure: `https://adb-<workspace-id>.<random-number>.azuredatabricks.net`
    *   GCP: `https://<workspace-id>.<random-number>.gcp.databricks.com`

*   The Databricks authentication details. For more information, see the documentation for
    [AWS](https://docs.databricks.com/dev-tools/auth/index.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/),
    or [GCP](https://docs.gcp.databricks.com/dev-tools/auth/index.html).

    The following videos show how to create a Databricks-managed service principal and then grant it access to a Databricks volume:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/wBmqv5DaA1E" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    <iframe width="560" height="315" src="https://www.youtube.com/embed/DykQRxgh2aQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    For the [Unstructured Platform](/platform/overview), only the following Databricks authentication type is supported:

    *   For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.
        Note that for Azure, only Databricks-managed service principals are supported. Microsoft Entra ID-managed service principals are not supported.

    For [Unstructured Ingest](/ingestion/overview), the following Databricks authentication types are supported:

    *   For Databricks personal access token authentication (AWS, Azure, and GCP): The personal access token's value.
    *   For username and password (basic) authentication (AWS only): The user's name and password values.
    *   For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.
    *   For OAuth user-to-machine (U2M) authentication (AWS, Azure, and GCP): No additional values.
    *   For Azure managed identities (MSI) authentication (Azure only): The client ID value for the corresponding managed identity.
    *   For Microsoft Entra ID service principal authentication (Azure only): The tenant ID, client ID, and client secret values for the corresponding service principal.
    *   For Azure CLI authentication (Azure only): No additional values.
    *   For Microsoft Entra ID user authentication (Azure only): The Entra ID token for the corresponding Entra ID user.
    *   For Google Cloud Platform credentials authentication (GCP only): The local path to the corresponding Google Cloud service account's credentials file.
    *   For Google Cloud Platform ID authentication (GCP only): The Google Cloud service account's email address.

*   The Databricks catalog name for the volume. Get the catalog name for [AWS](https://docs.databricks.com/catalogs/manage-catalog.html), [Azure](https://learn.microsoft.com/azure/databricks/catalogs/manage-catalog), or [GCP](https://docs.gcp.databricks.com/catalogs/manage-catalog.html).

*   The Databricks schema name for the volume. Get the schema name for [AWS](https://docs.databricks.com/schemas/manage-schema.html), [Azure](https://learn.microsoft.com/azure/databricks/schemas/manage-schema), or [GCP](https://docs.gcp.databricks.com/schemas/manage-schema.html).

*   The Databricks volume name, and optionally any path in that volume that you want to access directly. Get the volume information for [AWS](https://docs.databricks.com/files/volumes.html), [Azure](https://learn.microsoft.com/azure/databricks/files/volumes), or [GCP](https://docs.gcp.databricks.com/files/volumes.html).

*   Make sure that the target user or service principal has access to the target volume. To learn more, see the documentation for
    [AWS](https://docs.databricks.com/volumes/utility-commands.html#change-permissions-on-a-volume),
    [Azure](https://learn.microsoft.com/azure/databricks/volumes/utility-commands#change-permissions-on-a-volume),
    or [GCP](https://docs.gcp.databricks.com/volumes/utility-commands.html#change-permissions-on-a-volume).

The Databricks Volumes connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[databricks-volumes]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `DATABRICKS_HOST` - The Databricks host URL, represented by `--host` (CLI) or `host` (Python).
*   `DATABRICKS_CATALOG` - The Databricks catalog name for the Volume, represented by `--catalog` (CLI) or `catalog` (Python).
*   `DATABRICKS_SCHEMA` - The Databricks schema name for the Volume, represented by `--schema` (CLI) or `schema` (Python). If not specified, `default` is used.
*   `DATABRICKS_VOLUME` - The Databricks Volume name, represented by `--volume` (CLI) or `volume` (Python).
*   `DATABRICKS_VOLUME_PATH` - Any optional path to access within the volume, specified by `--volume-path` (CLI) or `volume_path` (Python).

Environment variables based on your authentication type, depending on your cloud provider:

*   For Databricks personal access token authentication (AWS, Azure, and GCP):

    *   `DATABRICKS_TOKEN` - The personal access token, represented by `--token` (CLI) or `token` (Python).

*   For username and password (basic) authentication (AWS only): The user's name and password values.

    *   `DATABRICKS_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
    *   `DATABRICKS_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

*   For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.

    *   `DATABRICKS_CLIENT_ID` - The client ID value for the corresponding service principal, represented by `--client-id` (CLI) or `client_id` (Python).
    *   `DATABRICKS_CLIENT_SECRET` - The client ID and OAuth secret values for the corresponding service principal, represented by `--client-secret` (CLI) or `client_secret` (Python).

*   For OAuth user-to-machine (U2M) authentication (AWS, Azure, and GCP): No additional environment variables.

*   For Azure managed identities (MSI) authentication (Azure only):

    *   `ARM_CLIENT_ID` - The client ID value for the corresponding managed identity, represented by `--azure-client-id` (CLI) or `azure_client_id` (Python).
    *   If the target identity has not already been added to the workspace, then you must also specify the
        `DATABRICKS_AZURE_RESOURCE_ID`, represented by `--azure-workspace-resource-id` (CLI) or `azure_workspace_resource_id` (Python).

*   For Microsoft Entra ID service principal authentication (Azure only):

    *   `ARM_TENANT_ID` - The tenant ID value for the corresponding service principal, represented by `--azure-tenant-id` (CLI) or `azure_tenant_id` (Python).
    *   `ARM_CLIENT_ID` - The client ID value for the corresponding service principal, represented by `--azure-client-id` (CLI) or `azure_client_id` (Python).
    *   `ARM_CLIENT_SECRET` - The client secret value for the corresponding service principal, represented by `--azure-client-secret` (CLI) or `azure_client_secret` (Python).
    *   If the service principal has not already been added to the workspace, then you must also specify the
        `DATABRICKS_AZURE_RESOURCE_ID`, represented by `--azure-workspace-resource-id` (CLI) or `azure_workspace_resource_id` (Python).

*   For Azure CLI authentication (Azure only): No additional environment variables.

*   For Microsoft Entra ID user authentication (Azure only):

    *   `DATABRICKS_TOKEN` - The Entra ID token for the corresponding Entra ID user, represented by `--token` (CLI) or `token` (Python).

*   For Google Cloud Platform credentials authentication (GCP only):

    *   `GOOGLE_CREDENTIALS` - The local path to the corresponding Google Cloud service account's credentials file, represented by `--google-credentials` (CLI) or `google_credentials`

*   For Google Cloud Platform ID authentication (GCP only):

    *   `GOOGLE_SERVICE_ACCOUNT` - The Google Cloud service account's email address, represented by `--google-service-account` (CLI) or `google_service_account` (Python).

*   Alternatively, you can store the preceding settings in a local
    [Databricks configuration profile](https://docs.databricks.com/en/dev-tools/auth/config-profiles.html) and then just
    refer to the profile's name:

    *   `DATABRICKS_PROFILE` - The name of the Databricks configuration profile, represented by `--profile` (CLI) or `profile` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    databricks-volumes \
      --profile $DATABRICKS_PROFILE \
      --host $DATABRICKS_HOST \
      --catalog $DATABRICKS_CATALOG \
      --schema $DATABRICKS_SCHEMA \
      --volume $DATABRICKS_VOLUME \
      --volume-path $DATABRICKS_VOLUME_PATH \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  # For all supported Databricks authentication types, you can import this:
  from unstructured_ingest.v2.processes.connectors.databricks_volumes import (
      DatabricksVolumesIndexerConfig,
      DatabricksVolumesConnectionConfig,
      DatabricksVolumesAccessConfig,
      DatabricksVolumesDownloaderConfig
  )

  # Alternatively, for supported Databricks on AWS authentication types only, you can import this:
  # from unstructured_ingest.v2.processes.connectors.databricks.volumes_aws import (
  #     DatabricksAWSVolumesIndexerConfig,
  #     DatabricksAWSVolumesConnectionConfig,
  #     DatabricksAWSVolumesAccessConfig,
  #     DatabricksAWSVolumesDownloaderConfig
  # )

  # Alternatively, for supported Azure Databricks authentication types only, you can import this:
  # from unstructured_ingest.v2.processes.connectors.databricks.volumes_azure import (
  #     DatabricksAzureVolumesIndexerConfig,
  #     DatabricksAzureVolumesConnectionConfig,
  #     DatabricksAzureVolumesAccessConfig,
  #     DatabricksAzureVolumesDownloaderConfig
  # )

  # Alternatively, for supported Databricks on Google Cloud authentication types only, you can import this:
  # from unstructured_ingest.v2.processes.connectors.databricks.volumes_gcp import (
  #     DatabricksGoogleVolumesIndexerConfig,
  #     DatabricksGoogleVolumesConnectionConfig,
  #     DatabricksGoogleVolumesAccessConfig,
  #     DatabricksGoogleVolumesDownloaderConfig
  # )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalConnectionConfig,
      LocalUploaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(reprocess=True),
          indexer_config=DatabricksVolumesIndexerConfig(recursive=True),
          # For specifying a Databricks configuration profile:
          downloader_config=DatabricksVolumesDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=DatabricksVolumesConnectionConfig(
              access_config=DatabricksVolumesAccessConfig(profile=os.getenv("DATABRICKS_PROFILE")),
              host=os.getenv("DATABRICKS_HOST"),
              catalog=os.getenv("DATABRICKS_CATALOG"),
              schema=os.getenv("DATABRICKS_SCHEMA"),
              volume=os.getenv("DATABRICKS_VOLUME"),
              volume_path=os.getenv("DATABRICKS_VOLUME_PATH")
          ),
          # Other examples:
          #
          # For Databricks on AWS, with Databricks personal access token authentication:
          # downloader_config=DatabricksAWSVolumesDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          # source_connection_config=DatabricksAWSVolumesConnectionConfig(
          #     access_config=DatabricksAWSVolumesAccessConfig(token=os.getenv("DATABRICKS_TOKEN")),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          #
          # For Azure Databricks, with Microsoft Entra ID service principal authentication:
          # downloader_config=DatabricksAzureVolumesDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          # source_connection_config=DatabricksAzureVolumesConnectionConfig(
          #     access_config=DatabricksAzureVolumesAccessConfig(
          #         azure_client_id=os.getenv("ARM_CLIENT_ID"),
          #         azure_client_secret=os.getenv("ARM_CLIENT_SECRET"),
          #         azure_tenant_id=os.getenv("ARM_TENANT_ID")   
          #     ),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          #
          # For Databricks on Google Cloud, with Google Cloud Platform credentials authentication:
          # downloader_config=DatabricksGoogleVolumesDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          # source_connection_config=DatabricksGoogleVolumesConnectionConfig(
          #     access_config=DatabricksGoogleVolumesAccessConfig(
          #         google_service_account=os.getenv("GOOGLE_CREDENTIALS")
          #     ),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Delta Table



Connect Delta Table to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to
store structured outputs locally on your filesystem.

Make sure to have the Delta Table dependencies installed:

```bash Shell
pip install "unstructured-ingest[delta-table]"
```

AWS credentials need to be available for use with the storage options.
Specify the to the DeltaTable using the `table-uri` argument, and pass a dictionary of the options to use for the storage backend
via `storage_options`.

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  unstructured-ingest \
    delta-table \
      --table-uri s3://utic-dev-tech-fixtures/sample-delta-lake-data/deltatable/ \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --storage_options "AWS_REGION=us-east-2,AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY" \
      --verbose \
      --strategy hi_res
  ```

  ```python Python
  import os

  from unstructured_ingest.connector.delta_table import SimpleDeltaTableConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import DeltaTableRunner

  if __name__ == "__main__":
      runner = DeltaTableRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          connector_config=SimpleDeltaTableConfig(
              table_uri="s3://utic-dev-tech-fixtures/sample-delta-lake-data/deltatable/",
              storage_options={
                  "AWS_REGION": "us-east-2",
                  "AWS_ACCESS_KEY_ID": os.getenv("AWS_ACCESS_KEY_ID"),
                  "AWS_SECRET_ACCESS_KEY": os.getenv("AWS_SECRET_ACCESS_KEY"),
              },
          ),
      )
      runner.run()
  ```
</CodeGroup>

For a full list of the options the Unstructured Ingest CLI accepts check `unstructured-ingest delta-table --help`.


# Discord



Connect Discord to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to
store structured outputs locally on your filesystem.

Make sure to have the Discord dependencies installed:

```bash Shell
pip install "unstructured-ingest[discord]"
```

To ingests the contents of Discord channels, you need to supply the following information:

*   `token`: an authentication token used to access Discord API
*   `channels`: a list of discord channel ids to ingest from

Optionally you can set the number of days to go back in history of the channels via the `period` argument.

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  unstructured-ingest \
    discord \
      --channels 12345678 \
      --token $DISCORD_TOKEN \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --preserve-downloads \
      --verbose \
      --strategy hi_res
  ```

  ```python Python
  import os

  from unstructured_ingest.connector.discord import DiscordAccessConfig, SimpleDiscordConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import DiscordRunner

  if __name__ == "__main__":
      runner = DiscordRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          connector_config=SimpleDiscordConfig(
              access_config=DiscordAccessConfig(token=os.getenv("DISCORD_TOKEN")),
              channels=["12345678"],
          ),
      )
      runner.run()
  ```
</CodeGroup>

For a full list of the options the Unstructured Ingest CLI accepts check `unstructured-ingest discord --help`.


# Dropbox



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Dropbox to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Dropbox prerequisites:

1.  A Dropbox account. [Get an account](https://www.dropbox.com/try/teams).

2.  A target source or destination folder in your Dropbox account. [Create a folder](https://help.dropbox.com/create-upload/creating-using-folders-on-dropbox).

3.  A Dropbox app for your Dropbox account. To learn how to create an app, click the **App Console** tab on the [Getting Started](https://www.dropbox.com/developers/reference/getting-started) page.

4.  Permission for your Dropbox app to read from, and write to, the target folder in your Dropbox account as needed. To do this:

    *   On the **Permissions** tab of your Dropbox app, check the boxes **files.content.read** or **files.content.write** or both. [Learn more](https://developers.dropbox.com/oauth-guide).

    *   On the **Settings** tab of your Dropbox app, for **App folder name**, set the name of the target folder in your Dropbox account for your Dropbox app to have access to.

    *   Note the remote URL to the target folder, which takes the format `dropbox://<path/to/folder/in/account>`.

5.  An access token for your Dropbox account. [Get a token](https://dropbox.tech/developers/generate-an-access-token-for-your-own-account). Save this token in a secure location. Do not share it with others.

The Dropbox connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[dropbox]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `DROPBOX_ACCESS_TOKEN` - The value of your access token, represented by `--token` (CLI) or `token` (Python).
*   `DROPBOX_REMOTE_URL` - The remote URL to the target folder, represented by `--remote-url` (CLI) or `remote_url` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    dropbox \
      --remote-url $DROPBOX_REMOTE_URL \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --token $DROPBOX_ACCESS_TOKEN \
      --num-processes 2 \
      --recursive \
      --verbose \
      --partition-by-api \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --api-key $UNSTRUCTURED_API_KEY \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.dropbox import (
      DropboxIndexerConfig,
      DropboxDownloaderConfig,
      DropboxAccessConfig,
      DropboxConnectionConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalConnectionConfig,
      LocalUploaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=DropboxIndexerConfig(remote_url=os.getenv("DROPBOX_REMOTE_URL")),
          downloader_config=DropboxDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=DropboxConnectionConfig(
              access_config=DropboxAccessConfig(
                  token=os.getenv("DROPBOX_ACCESS_TOKEN")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.dropbox import DropboxAccessConfig, SimpleDropboxConfig
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import DropboxRunner

  if __name__ == "__main__":
      runner = DropboxRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleDropboxConfig(
              access_config=DropboxAccessConfig(token=os.getenv("DROPBOX_ACCESS_TOKEN")),
              remote_url=os.getenv("DROPBOX_REMOTE_URL"),
              recursive=True,
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Elasticsearch



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Elasticsearch to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Elasticsearch prerequisites.

*   For the [Unstructured Platform](/platform/overview), only Elastic Cloud instances are supported.

*   For [Unstructured Ingest](/ingestion/overview), Elastic Cloud instances and self-manged Elasticsearch instances are supported.

*   For Elastic Cloud, you will need an [Elastic Cloud service instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#hosted-elasticsearch-service).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/HEYIFC7uOus" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For self-managed Elasticsearch, you will need a [self-managed Elasticsearch instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#elasticsearch-deployment-options).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/9znbAkNG1Ok" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   You will need the name of the index on the instance. See [Create index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html) and [Get index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html).

    The Elasticsearch index that you use must have a schema that is compatible with the schema of the documents
    that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following index schema example for your own needs:

    ```json
    {
        "settings": {
            "index": {
                "knn": true,
                "knn.algo_param.ef_search": 100
            }
        },
        "mappings": {
            "properties": {
                "element_id": {
                    "type": "keyword"
                },
                "text": {
                    "type": "text"
                },
                "embeddings": {
                    "type": "dense_vector",
                    "dims": 384,
                    "index": true,
                    "similarity": "cosine"
                },
                "metadata": {
                    "type": "object",
                    "properties": {
                        "parent_id": {
                            "type": "text"
                        },
                        "page_number": {
                            "type": "integer"
                        },
                        "is_continuation": {
                            "type": "boolean"
                        },
                        "orig_elements": {
                            "type": "text"
                        }
                    }
                }
            }
        }
    }
    ```

    See also:

    *   [An Introduction to Elasticsearch Mapping](https://www.elastic.co/blog/found-elasticsearch-mapping-introduction)
    *   [Explicit mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html)
    *   [Dynamic field mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html)
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

    <iframe width="560" height="315" src="https://www.youtube.com/embed/0f0tEcrvP8g" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For Elastic Cloud, you will need the Elastic Cloud service instance's API key. If you are using Unstructured Ingest, you will also need the instance's Cloud ID. To get these, see your Elasticsearch Service web console.

*   For self-managed Elasticsearch, you will need:

    *   The self-managed instance's hostname and port number. See [Networking](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html).
    *   If you're using basic authentication to the self-managed instance, the user's name and password.
    *   If you're using token-based authentication to the self-managed instance, the bearer token or API key for the instance. See [Token-based authentication services](https://www.elastic.co/guide/en/elasticsearch/reference/current/token-authentication-services.html) and [Create API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html).
    *   If you're using certificate authentication to the self-managed instance, the path to the Certificate Authority (CA) file on the instance, and the certificate fingerprint. See [SSL certificate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-ssl.html) and [Where can I see my Certificate Fingerprint?](https://discuss.elastic.co/t/where-can-i-see-my-certificate-fingerprint/319335/3).

The Elasticsearch connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[elasticsearch]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `ELASTICSEARCH_HOST` - The hostname and port number, defined as `<hostname>:<port-number>` and represented by `--hosts` (CLI) or `hosts` (Python).
*   `ELASTICSEARCH_INDEX_NAME` - The name of the search index, represented by `--index-name` (CLI) or `index_name` (Python).

If you're using Elastic Cloud to connect to ElasticSearch:

*   `ELASTIC_CLOUD_ID` - The instance's Cloud ID, represented by `--cloud-id` (CLI) or `cloud_id` (Python).
*   `ELASTIC_CLOUD_API_KEY` - The corresponding Cloud API key value, represented by `--api-key-id` (CLI) or `api_key_id` (Python).

If you're using basic authentication to the instance:

*   `ELASTICSEARCH_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
*   `ELASTICSEARCH_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

If you're using token-based authentication to the instance instead:

*   `ELASTICSEARCH_BEARER_TOKEN` - The bearer token's value, represented by `--bearer-auth` (CLI) or `bearer_auth` (Python).
*   `ELASTIC_CLOUD_API_KEY_ID` - The API key's value, represented by `--api-key` (CLI) or `api_key` (Python).

If you're using certificates:

*   `ELASTICSEARCH_CA_CERTS` - The path to the Certificate Authority (CA) file, represented by `--ca-certs` (CLI) or `ca_certs` (Python).
*   `ELASTICSEARCH_SSL_ASSERT_FINGERPRINT` - The certificate's fingerprint, represented by `--ssl-assert-fingerprint` or `ssl_assert_fingerprint` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    elasticsearch \
      --metadata-exclude filename,file_directory,metadata.data_source.date_processed \
      --hosts $ELASTICSEARCH_HOST \
      --username $ELASTICSEARCH_USERNAME \
      --password $ELASTICSEARCH_PASSWORD \
      --index-name $ELASTICSEARCH_INDEX_NAME \
      --ca-certs $ELASTICSEARCH_CA_CERTS \
      --ssl-assert-fingerprint $ELASTICSEARCH_SSL_ASSERT_FINGERPRINT \
      --fields 'ethnicity, director, plot' \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.elasticsearch import (
      ElasticsearchIndexerConfig,
      ElasticsearchDownloaderConfig,
      ElasticsearchConnectionConfig,
      ElasticsearchAccessConfig,
      ElasticsearchUploadStagerConfig,
      ElasticsearchUploaderConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=ElasticsearchIndexerConfig(index_name=os.getenv("ELASTICSEARCH_INDEX_NAME")),
          downloader_config=ElasticsearchDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=ElasticsearchConnectionConfig(
              access_config=ElasticsearchAccessConfig(
                  password=os.getenv("ELASTICSEARCH_PASSWORD"),
                  ssl_assert_fingerprint=os.getenv("ELASTICSEARCH_SSL_ASSERT_FINGERPRINT")
              ),
              hosts=[os.getenv("ELASTICSEARCH_HOST")],
              username=os.getenv("ELASTICSEARCH_USERNAME"),
              ca_certs=os.getenv("ELASTICSEARCH_CA_CERTS")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig()
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.elasticsearch import (
      ElasticsearchAccessConfig,
      SimpleElasticsearchConfig,
  )
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import ElasticSearchRunner

  if __name__ == "__main__":
      runner = ElasticSearchRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              metadata_exclude=["filename", "file_directory", "metadata.data_source.date_processed"],
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleElasticsearchConfig(
              access_config=ElasticsearchAccessConfig(
                  password=os.getenv("ELASTICSEARCH_PASSWORD"),
                  ssl_assert_fingerprint=os.getenv("ELASTICSEARCH_SSL_ASSERT_FINGERPRINT")
              ),
              hosts=[os.getenv("ELASTICSEARCH_HOST")],
              username=os.getenv("ELASTICSEARCH_USERNAME"),
              ca_certs=os.getenv("ELASTICSEARCH_CA_CERTS"),
              index_name="movies",
              fields=["ethnicity", "director", "plot"],
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# GitHub



Connect GitHub to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to store structured outputs locally on your filesystem.

First, install the GitHub dependencies as shown here.

```bash
pip install "unstructured-ingest[github]"
```

Provide the GitHub repo URL (`url`) to fetch the files from, e.g. `"https://github.com/Unstructured-IO/unstructured"`
or `"Unstructured-IO/unstructured"`, and supply your GitHub access token (`git-access-token`).
Learn more about GitHub authentication [here](https://docs.github.com/en/authentication).

Optionally, specify a branch and what file types to limit the ingestion to:

*   `git-branch`: The branch to fetch files from. If not given, the default repository branch is used.
*   `git-file-glob`: A comma-separated list of file globs to limit which types of files are accepted, e.g. `'*.html,*.txt'`

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  unstructured-ingest \
    github \
      --url Unstructured-IO/unstructured \
      --git-branch main \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --strategy hi_res
  ```

  ```python Python
  from unstructured_ingest.connector.git import GitAccessConfig
  from unstructured_ingest.connector.github import SimpleGitHubConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import GithubRunner

  if __name__ == "__main__":
      runner = GithubRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          connector_config=SimpleGitHubConfig(
              url="Unstructured-IO/unstructured", branch="main", access_config=GitAccessConfig()
          ),
      )
      runner.run()
  ```
</CodeGroup>

For a full list of the options the Unstructured Ingest CLI accepts check `unstructured-ingest github --help`.


# GitLab



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect GitLab to your preprocessing pipeline, and use the Unstructured CLI or Python to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The GitLab prerequisites:

*   A [GitLab account](https://gitlab.com/users/sign_up).
*   The URL for the target [GitLab repository](https://docs.gitlab.com/ee/user/project/repository/).
*   A [GitLab access token](https://docs.gitlab.com/ee/security/tokens/) that allows access to the repository.
*   Optionally, the name of a specific branch to access in the repository. The repository's default repository branch is used if not otherwise specified.
*   Optionally, a list of file globs to limit which types of files are accepted, for example `*.html` or `*.txt`.

The GitLab connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[gitlab]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `GITLAB_REPO_URL` - The URL for the target GitLab repository, represented by `--url` (CLI) or `url` (Python).
*   `GITLAB_TOKEN` - The GitLab access token that allows access to the repository, represented by `--git-access-token` (CLI) or `access_token` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    gitlab \
    --url $GITLAB_URL \
    --git-access-token $GITLAB_TOKEN \
    --git-branch v0.0.7 \
    --git-file-glob "*.html,*.txt" \
    --output-dir $LOCAL_FILE_OUTPUT_DIR \
    --partition-by-api \
    --api-key $UNSTRUCTURED_API_KEY \
    --partition-endpoint $UNSTRUCTURED_API_URL \
    --strategy hi_res \
    --chunking-strategy by_title \
    --embedding-provider huggingface \
    --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest v2
  # pip install "unstructured-ingest[gitlab]"

  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.gitlab import (
      GitLabIndexerConfig,
      GitLabDownloaderConfig,
      GitLabConnectionConfig,
      GitLabAccessConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=GitLabIndexerConfig(
              path="/",
              recursive=True,
              git_branch="v0.0.7"
          ),
          downloader_config=GitLabDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=GitLabConnectionConfig(
              access_config=GitLabAccessConfig(access_token=os.getenv("GITLAB_TOKEN")),
              url=os.getenv("GITLAB_URL")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.git import GitAccessConfig
  from unstructured_ingest.connector.gitlab import SimpleGitlabConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import GitlabRunner

  if __name__ == "__main__":
      runner = GitlabRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleGitlabConfig(
              url=os.getenv("GITLAB_REPO_URL"),
              branch="v0.0.7",
              access_config=GitAccessConfig(),
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Google Cloud Storage



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Google Cloud Storage to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Google Cloud Storage prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/HYaALQ0F-L4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A Google Cloud service account. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#console).

*   A service account key for the service account. See [Create a service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating) in
    [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete).

    To ensure maximum compatibility across Unstructured service offerings, you should give the service account key information to Unstructured as
    a single-line string that contains the contents of the downloaded service account key file (and not the service account key file itself).
    To print this single-line string without line breaks, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
    In this command, replace `<path-to-downloaded-key-file>` with the path to the service account key file that you downloaded by following the preceding instructions.

    *   For macOS or Linux:
        ```text
        tr -d '\n' < <path-to-downloaded-key-file>
        ```
    *   For Windows:
        ```text
        (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
        ```

*   The URI for a Google Cloud Storage bucket. This URI consists of the target bucket name, plus any target folder within the bucket, expressed as `gs://<bucket-name>[/folder-name]`. [Create a bucket](https://cloud.google.com/storage/docs/creating-buckets#console).

    This bucket must have, at minimum, one of the following roles applied to the target Google Cloud service account:

    *   `Storage Object Viewer` for bucket read access.
    *   `Storage Object Creator` for bucket write access.
    *   The `Storage Object Admin` role provides read and write access, plus access to additional bucket operations.

    To apply one of these roles to a service account for a bucket, see [Add a principal to a bucket-level policy](https://cloud.google.com/storage/docs/access-control/using-iam-permissions#bucket-add)
    in [Set and manage IAM policies on buckets](https://cloud.google.com/storage/docs/access-control/using-iam-permissions).

The Google Cloud Storage connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[gcs]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `GCS_SERVICE_ACCOUNT_KEY` - The Google Cloud service account key for Google Cloud Storage, represented by `--service-account-key` (CLI) or `service_account_key` (Python).
*   `GCS_REMOTE_URL` - The Google Cloud Storage bucket URL, represented by `--remote-url` (CLI) or `remote_url` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    gcs \
      --service-account-key $GCS_SERVICE_ACCOUNT_KEY \
      --remote-url $GCS_REMOTE_URL \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --recursive \
      --verbose \
      --strategy hi_res \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.gcs import (
      GcsIndexerConfig,
      GcsDownloaderConfig,
      GcsConnectionConfig,
      GcsAccessConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=GcsIndexerConfig(remote_url=os.getenv("GCS_REMOTE_URL")),
          downloader_config=GcsDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=GcsConnectionConfig(
              access_config=GcsAccessConfig(
                  service_account_key=os.getenv("GCS_SERVICE_ACCOUNT_KEY")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.gcs import GcsAccessConfig, SimpleGcsConfig
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import GCSRunner

  if __name__ == "__main__":
      runner = GCSRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
          ),
          connector_config=SimpleGcsConfig(
              access_config=GcsAccessConfig(token=os.getenv("GCS_SERVICE_ACCOUNT_KEY")),
              remote_url=os.getenv("GCS_REMOTE_URL"),
              recursive=True
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Google Drive



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Google Drive to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Google Drive prerequisites:

1.  A Google Cloud service account and its related `credentials.json` key file or its contents in JSON format. [Learn how](https://developers.google.com/workspace/guides/create-credentials#service-account).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/0-cS_qI0XCE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

2.  Note the local path to the `credentials.json` key file. Or, copy the key file's contents into a compatible string—including properly escaped quotes—as required.
    To make converting a JSON object into a compatible string easier, you can search the Internet by using a search phrase such as "tools for converting a JSON object into a string."
    Before using any tool, check to make sure that the tool does not share the key file's contents with anyone you do not trust.

3.  Give the service account's email address access to the Google Drive folder. [Learn more](https://www.googlecloudcommunity.com/gc/Workspace-Q-A/Can-i-give-access-to-document-of-google-drive-to-service-account/m-p/530106).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/ykJQzEe_2dM" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

4.  Note the folder ID. This is a part of the URL for your Google Drive folder represented in the following URL as `{folder_id}`: `https://drive.google.com/drive/folders/{folder-id}`.

The Google Drive connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[google-drive]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `GOOGLE_DRIVE_FOLDER_ID` - The folder ID, represented by `--drive-id` (CLI) or `drive_id` (Python).
*   `GCP_SERVICE_ACCOUNT_KEY_FILEPATH` - The path to the `credentials.json` key file, represented by `--service-account-key-path` (CLI) or `service_account_key_path` (Python), or
*   `GCP_SERVICE_ACCOUNT_KEY_STRING` - The contents of the `credentials.json` key file as a string, represented by `--service-account-key` (CLI) or `service_account_key` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    google-drive \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --drive-id $GOOGLE_DRIVE_FOLDER_ID \
      --service-account-key-path $GCP_SERVICE_ACCOUNT_KEY_FILEPATH \ # Or
      --service-account-key $GCP_SERVICE_ACCOUNT_KEY_STRING \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.google_drive import (
      GoogleDriveConnectionConfig,
      GoogleDriveAccessConfig,
      GoogleDriveIndexerConfig,
      GoogleDriveDownloaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=GoogleDriveIndexerConfig(),
          downloader_config=GoogleDriveDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=GoogleDriveConnectionConfig(
              access_config=GoogleDriveAccessConfig(
                  service_account_key_path=os.getenv("GCP_SERVICE_ACCOUNT_KEY_FILEPATH"), # Or
                  service_account_key=os.getenv("GCP_SERVICE_ACCOUNT_KEY_STRING")
              ),
              drive_id=os.getenv("GOOGLE_DRIVE_FOLDER_ID"),
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.google_drive import (
      GoogleDriveAccessConfig,
      SimpleGoogleDriveConfig,
  )
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import GoogleDriveRunner

  if __name__ == "__main__":
      runner = GoogleDriveRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleGoogleDriveConfig(
              access_config=GoogleDriveAccessConfig(
                  service_account_key=os.getenv("GCP_SERVICE_ACCOUNT_KEY_STRING")
              ),
              recursive=True,
              drive_id=os.getenv("GOOGLE_DRIVE_FOLDER_ID"),
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).

## Output a key file's contents as a string

If you need to convert the contents of a `credentials.json` file into a string, you could use a Python script such as the following.
This script takes the local path to the key file as input and outputs the key file's contents as a string.

```python Python
# Filename: json_file_to_string.py

import json
import sys

def json_file_to_string(file_path):
    try:
        # Read the JSON file.
        with open(file_path, 'r') as file:
            # Load the JSON data.
            data = json.load(file)
        
        # Convert the JSON data back to a string, with no whitespace.
        json_string = json.dumps(data, separators=(',', ':'))
        
        return json_string
    
    except FileNotFoundError:
        print(f"Error: File '{file_path}' not found.")
        return None
    except json.JSONDecodeError:
        print(f"Error: '{file_path}' is not a valid JSON file.")
        return None
    except Exception as e:
        print(f"An unexpected error occurred: {e}")
        return None

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python json_file_to_string.py <path_to_json_file>")
        sys.exit(1)
    
    file_path = sys.argv[1]
    result = json_file_to_string(file_path)
    
    if result:
        print(result)
```


# HubSpot



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect HubSpot to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The HubSpot prerequisites:

*   A [HubSpot private app and its related access token](https://developers.hubspot.com/docs/api/private-apps).
*   The HubSpot [object types](https://developers.hubspot.com/docs/api/crm/understanding-the-crm#objects) and any of their built-in properties that you want to access. Supported object types include tickets, emails, notes, calls, products, and communications.
*   Any HubSpot params and [custom properties](https://knowledge.hubspot.com/properties/create-and-edit-properties) that you want to access for any of the supported object types.

The HubSpot connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[hubspot]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `HUBSPOT_API_TOKEN` - The access token value for the HubSpot private app, represented by `--api-token` (CLI) or `api_token` (Python).
*   `HUBSPOT_OBJECT_TYPES` - A comma-separated list of strings (CLI) or an array of strings (Python) specifying one or more of the
    object types to access. This must be a subset of
    [tickets](https://developers.hubspot.com/docs/api/crm/tickets),
    [calls](https://developers.hubspot.com/docs/api/crm/calls),
    [emails](https://developers.hubspot.com/docs/api/crm/email),
    [notes](https://developers.hubspot.com/docs/api/crm/notes),
    [tasks](https://developers.hubspot.com/docs/api/crm/tasks),
    [products](https://developers.hubspot.com/docs/api/crm/products), and
    [communications](https://developers.hubspot.com/docs/api/crm/communications). These are represented by
    `--object-types` (CLI) or `object_types` (Python); if not specified, all of these object types are accessed by default.
*   `HUBSPOT_PROPERTIES` - Any filtering to be applied to specified object types, declared as a dictionary of key-value pairs (Python) specifying one or more of the default
    [properties](https://developers.hubspot.com/docs/api/crm/properties) and their desired property values, represented by
    `properties` (Python). Not available in the CLI.
*   `HUBSPOT_CUSTOM_PROPERTIES` - Any [custom properties](https://developers.hubspot.com/docs/api/crm/properties) to access.
    For the CLI, this is specified as a JSON-like string in the form `"{\"<object_type>\":[\"<custom_property_id>\", ..., \"<custom_property_id>\"]}"`.
    For Python, this is specified as a `Dict[str, List[str]]`.
    This must map to the internal name of the variable. If the property is missing,
    it will be omitted. This is represented by `--custom-properties` (CLI) or `custom_properties` (Python).
*   `HUBSPOT_PARAMS` - For Python, an array of strings specifying any values to filter the customer properties on represented by `params`. Not available in the CLI.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library.  The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  unstructured-ingest \
    hubspot \
      --api-token $HUBSPOT_API_TOKEN \
      --object-types $HUBSPOT_OBJECT_TYPES \
      --custom-properties $HUBSPOT_CUSTOM_PROPERTIES \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.runner import HubSpotRunner
  from unstructured_ingest.connector.hubspot import (
      HubSpotAccessConfig,
      SimpleHubSpotConfig
  )
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig
  )

  if __name__ == "__main__":
      runner = HubSpotRunner(
          processor_config=ProcessorConfig(
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR")
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res"
          ),
          connector_config=SimpleHubSpotConfig(
              access_config=HubSpotAccessConfig(
                  api_token=os.getenv("HUBSPOT_API_TOKEN")
              ),
              object_types=os.getenv("HUBSPOT_OBJECT_TYPES"),
              properties=os.getenv("HUBSPOT_PROPERTIES"),
              custom_properties=os.getenv("HUBSPOT_CUSTOM_PROPERTIES"),
              params=os.getenv("HUBSPOT_PARAMS")
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Jira



Connect Jira to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to store structured outputs locally on your filesystem.

First, install the Jira dependencies as shown here.

```bash
pip install "unstructured-ingest[jira]"
```

To processes all the issues in projects within your Jira domain, you must specify:

*   `url`:Atlassian (Jira) domain URL
*   `api-token`: API token to authenticate into Atlassian (Jira). Check [Atlassian documentation](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/) for more info.
*   `user-email`: User email for the domain

Optionally, you can limit what issues should be processed:

*   `list-of-projects`: explicitly specify which project ids should be processed
*   `list-of-boards`: explicitly specify which board ids should be processed
*   `list-of-issues`: explicitly specify which issue ids or keys should be processed

If any of the optional arguments are provided, connector will ingest only those components, and nothing else.
When none of the optional arguments are provided, all issues in all projects will be ingested.

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  unstructured-ingest \
    jira \
    --metadata-exclude filename,file_directory,metadata.data_source.date_processed \
    --url https://unstructured-jira-connector-test.atlassian.net \
    --user-email 12345678@unstructured.io \
    --api-token ABCDE1234ABDE1234ABCDE1234 \
    --output-dir $LOCAL_FILE_OUTPUT_DIR \
    --num-processes 2 \
    --strategy hi_res
  ```

  ```python Python
  import os

  from unstructured_ingest.connector.jira import JiraAccessConfig, SimpleJiraConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import JiraRunner

  if __name__ == "__main__":
      runner = JiraRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              metadata_exclude=["filename", "file_directory", "metadata.data_source.date_processed"],
              strategy="hi_res",
          ),
          connector_config=SimpleJiraConfig(
              access_config=JiraAccessConfig(api_token=os.getenv("JIRA_API_TOKEN")),
              url=os.getenv("JIRA_URL"),
              user_email=os.getenv("JIRA_EMAIL"),
          ),
      )
      runner.run()
  ```
</CodeGroup>

For a full list of the options the Unstructured Ingest CLI accepts check `unstructured-ingest jira --help`.


# Kafka



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Kafka to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Kafka prerequisites:

*   A Kafka cluster, such as ones provided by [Confluent Cloud](https://www.confluent.io/confluent-cloud), [Amazon Managed Streaming for Apache Kafka (Amazon MSK)](https://aws.amazon.com/msk/), or [Google Cloud Managed Service for Apache Kafka](https://cloud.google.com/products/managed-service-for-apache-kafka).
*   The hostname of the bootstrap Kafka cluster to connect to.
*   The port number of the cluster.
*   The name of the topic to read messages from and write messages to on the cluster.
*   If you use Kafka API keys and secrets for authentication, the key and secret values.

The Kafka connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[kafka]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `KAFKA_BOOTSTRAP_SERVER` - The hostname of the bootstrap Kafka cluster to connect to, represented by `--bootstrap-server` (CLI) or `bootstrap_server` (Python).
*   `KAFKA_PORT` - The port number of the cluster, represented by `--port` (CLI) or `port` (Python).
*   `KAFKA_TOPIC` - The unique name of the topic to read messages from and write messages to on the cluster, represented by `--topic` (CLI) or `topic` (Python).

If you use Kafka API keys and secrets for authentication:

*   `KAFKA_API_KEY` - The Kafka API key value, represented by `--kafka-api-key` (CLI) or `kafka_api_key` (Python).
*   `KAFKA_SECRET` - The secret value for the Kafka API key, represented by `--secret` (CLI) or `secret` (Python).

Additional settings include:

*   `--confluent` (CLI) or `confluent` (Python): True to indicate that the cluster is running Confluent Kafka.
*   `--num-messages-to-consume` (CLI) or `num_messages_to_consume` (Python): The maximum number of messages to get from the topic. The default is `1`.
*   `--timeout` (CLI) or `timeout` (Python): The maximum amount of time to wait for the response of a request to the topic, expressed in seconds. The default is `1.0`.

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    kafka \
      --bootstrap-server $KAFKA_BOOTSTRAP_SERVER \ 
      --port $KAFKA_PORT \
      --topic $KAFKA_TOPIC \
      --kafka-api-key $KAFKA_API_KEY \
      --secret $KAFKA_API_KEY \
      --confluent true \
      --batch-size 100 \
      --num-messages-to-consume 1 \
      --timeout 1.0 \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.kafka.cloud import (
      CloudKafkaIndexerConfig,
      CloudKafkaDownloaderConfig,
      CloudKafkaConnectionConfig,
      CloudKafkaAccessConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=CloudKafkaIndexerConfig(
              topic=os.getenv("KAFKA_TOPIC"),
              num_messages_to_consume=100,
              timeout=1
          ),
          downloader_config=CloudKafkaDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=CloudKafkaConnectionConfig(
              access_config=CloudKafkaAccessConfig(
                  kafka_api_key=os.getenv("KAFKA_API_KEY"),
                  secret=os.getenv("KAFKA_SECRET")
              ),
              bootstrap_server=os.getenv("KAFKA_BOOTSTRAP_SERVER"),
              port=os.getenv("KAFKA_PORT")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.runner import KafkaRunner
  from unstructured_ingest.connector.kafka import (
      KafkaAccessConfig,
      SimpleKafkaConfig
  )
  from unstructured_ingest.interfaces import (
      ChunkingConfig,
      EmbeddingConfig,
      PartitionConfig,
      ProcessorConfig,
      ReadConfig
  )

  if __name__ == "__main__":
      runner = KafkaRunner(
          processor_config=ProcessorConfig(
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR")
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res"
          ),
          connector_config=SimpleKafkaConfig(
              bootstrap_server=os.getenv("KAFKA_BOOTSTRAP_SERVER"),
              port=os.getenv("KAFKA_PORT"),
              topic=os.getenv("KAFKA_TOPIC"),
              access_config=KafkaAccessConfig(
                  kafka_api_key=os.getenv("KAFKA_API_KEY"),
                  secret=os.getenv("KAFKA_SECRET"),
              ),
              confluent=False,
              num_messages_to_consume=1,
              timeout=1.0
          ),
          chunking_config=ChunkingConfig(chunk_elements=True),
          embedding_config=EmbeddingConfig(
              provider="huggingface"
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Local



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect local files to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need the local source connector dependencies:

```bash CLI, Python
pip install unstructured-ingest
```

To use the local source connector, you must set `--input-path` (CLI) or `input_path` (Python) to the path in the local filesystem which contains
documents you wish to process. Optionally, you can limit processing to certain file types by setting `--file-glob` (CLI) or `file_glob` (Python), for example to `.docx`
to process only `.docx` files.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported.

The following 3-minute video shows how to use the Unstructured Ingest Python library to send multiple PDFs from a local directory in batches to be ingested by Unstructured API services for processing:

<iframe width="560" height="315" src="https://www.youtube.com/embed/tSKHFXsBQ-c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig,
      LocalUploaderConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.local import SimpleLocalConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import LocalRunner

  if __name__ == "__main__":
      runner = LocalRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleLocalConfig(
              input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
              recursive=True,
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# MongoDB



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect MongoDB to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The MongoDB prerequisites for a MongoDB Atlas deployment:

<iframe width="560" height="315" src="https://www.youtube.com/embed/g6qDfbg808M" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A MongoDB Atlas account. [Create an account](https://www.mongodb.com/cloud/atlas/register).
*   A MongoDB Atlas cluster. [Create a cluster](https://www.mongodb.com/docs/atlas/tutorial/deploy-free-tier-cluster).
*   The cluster must be reachable from your application environment. [Learn how](https://www.mongodb.com/docs/atlas/setup-cluster-security/#network-and-firewall-requirements).
*   The cluster must be configured to use your IP address. [Learn how](https://www.mongodb.com/docs/atlas/setup-cluster-security/#ip-access-list).
*   The cluster must have at least one database. [Create a database](https://www.mongodb.com/docs/compass/current/databases/#create-a-database).
*   The database must have at least one user, and that user must have sufficient access to the database. [Create a database user](https://www.mongodb.com/docs/atlas/security-add-mongodb-users/#add-database-users). [Give the user database access](https://www.mongodb.com/docs/manual/core/authorization/).
*   The database must have at least one collection. [Create a collection](https://www.mongodb.com/docs/compass/current/collections/#create-a-collection).
*   The connection string for the cluster. For MongoDB Atlas, this connection string must include the protocol, username, password, host, and cluster name. For example:

    ```text
    mongodb+srv://<db_user>:<db_password>@<host>/?retryWrites=true&w=majority&appName=<cluster>
    ```

    To get the connection string in MongoDB Atlas, do the following:

    1.  Log in to your MongoDB Atlas console.
    2.  In the sidebar, under **Databases**, click **Clusters**.
    3.  Click on the cluster you want to connect to.
    4.  Click **Connect**, or click the **Cmd Line Tools** tab and then click **Connect Instructions**.
    5.  Click **Drivers**.
    6.  Under **Add your connection string into your application code**, copy the connection string.
        You can then close the **Connect** dialog in MongoDB Atlas.

        Before you use this connection string, be sure to fill in any placeholders in the string, such as your MongoDB Atlas database user's password value.

    [Learn more](https://www.mongodb.com/resources/products/fundamentals/mongodb-connection-string).

The MongoDB connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[mongodb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

For a MongoDB Atlas deployment, the following environment variables:

*   `MONGODB_DATABASE` - The name of the database, represented by `--database` (CLI) or `database` (Python).

*   `MONGODB_COLLECTION` - The name of the collection in the database, represented by `--collection` (CLI) or `collection` (Python).

*   `MONGODB_URI` - The connection string for the cluster, represented by `--uri` (CLI) or `uri` (Python).

For a local MongoDB server, the following environment variables:

*   `MONGODB_HOST` - The host for the local MongoDB server, represented by `--host` (CLI) or `host` (Python).

*   `MONGODB_PORT` - The port for the local MongoDB server, represented by `--port` (CLI) or `port` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    mongodb \
      --metadata-exclude filename,file_directory,metadata.data_source.date_processed \
      --uri $MONGODB_URI \
      --database $MONGODB_DATABASE \
      --collection $MONGODB_COLLECTION \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.mongodb import (
      MongoDBAccessConfig,
      MongoDBConnectionConfig,
      MongoDBIndexerConfig,
      MongoDBDownloaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import LocalConnectionConfig
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=MongoDBIndexerConfig(batch_size=100),
          downloader_config=MongoDBDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=MongoDBConnectionConfig(
              access_config=MongoDBAccessConfig(uri=os.getenv("MONGODB_URI")),
              database=os.getenv("MONGODB_DATABASE"),
              collection=os.getenv("MONGODB_COLLECTION")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig()
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.mongodb import (
      SimpleMongoDBConfig,
  )
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import MongoDBRunner

  if __name__ == "__main__":
      runner = MongoDBRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              metadata_exclude=["filename", "file_directory", "metadata.data_source.date_processed"],
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleMongoDBConfig(
              uri=os.getenv("MONGODB_URI"),
              database=os.getenv("MONGODB_DATABASE"),
              collection=os.getenv("MONGODB_COLLECTION"),
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Notion



Connect Notion to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to store
structured outputs locally on your filesystem.

First, install the Notion dependencies as shown here:

```bash
pip install "unstructured-ingest[notion]"
```

Make sure to provide `notion-api-key`. To get the credentials for your Notion workspace, follow the steps described
in [Notion documentation](https://developers.notion.com/docs/create-a-notion-integration).

Optionally, specify the following parameters:

*   `page-ids`: Notion page IDs to extract text from.
*   `database-ids`: Notion database IDs to extract text from.

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  unstructured-ingest \
    notion \
      --api-key "<Notion api key>" \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --page-ids "<Comma delimited list of page ids to process>" \
      --database-ids "<Comma delimited list of database ids to process>" \
      --num-processes 2 \
      --verbose \
      --strategy hi_res
  ```

  ```python Python
  from unstructured_ingest.connector.notion.connector import NotionAccessConfig, SimpleNotionConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import NotionRunner

  if __name__ == "__main__":
      runner = NotionRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          connector_config=SimpleNotionConfig(
              access_config=NotionAccessConfig(
                  notion_api_key="POPULATE API KEY",
              ),
              page_ids=["LIST", "OF", "PAGE", "IDS"],
              database_ids=["LIST", "OF", "DATABASE", "IDS"],
              recursive=False,
          ),
      )
      runner.run()
  ```
</CodeGroup>

For a full list of the options the Unstructured Ingest CLI accepts check `unstructured-ingest notion --help`.


# OneDrive



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect OneDrive to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The OneDrive prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/7uSu0LsbcEQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A OneDrive account.
*   The User Principal Name (UPN) for the OneDrive account. This is typically the OneDrive account user's email address.
*   The path to the target OneDrive folder, starting from the OneDrive account's root folder, for example `my-folder/my-subfolder`.
*   The client ID, tenant ID, and client secret for the Microsoft Entra ID app registration that has access to the target OneDrive account and
    also has the correct set of Microsoft Graph authentication scopes. These scopes include:

    *   `Files.ReadWrite.All` (if both reading and writing are needed)
    *   `Sites.ReadWrite.All` (if both reading and writing are needed)
    *   `User.Read.All`
    *   `Directory.Read.All`

See also:

*   [Registering your app for Microsoft Graph](https://learn.microsoft.com/onedrive/developer/rest-api/getting-started/app-registration)
*   [OneDrive authentication and sign-in](https://learn.microsoft.com/onedrive/developer/rest-api/getting-started/msa-oauth)
*   [OneDrive API documentation](https://docs.microsoft.com/onedrive/developer/rest-api/)

The OneDrive connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[onedrive]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `ONEDRIVE_PATH` - The path to the target folder in the OneDrive account, starting with the account's root folder (for example `my-folder/my-subfolder`), represented by `--path` (CLI) or `path` (Python).
*   `ONEDRIVE_CLIENT_ID` - The application (client) ID of the Microsoft Entra ID app registration that has access to the OneDrive account, represented by `--client-id` (CLI) or `client_id` (Python).
*   `ONEDRIVE_CLIENT_CRED` - The client secret for the Entra ID app registration, represented by `--client-cred` (CLI) or `client_cred` (Python).
*   `ONEDRIVE_TENANT` - The directory (tenant) ID of the Entra ID app registration, represented by `--tenant` (CLI) or `tenant` (Python).
*   `ONEDRIVE_USER_PNAME` - The User Principal Name (UPN) for the OneDrive user account in Entra ID (typically the user's email address), represented by `--user-pname` (CLI) or `user_pname` (Python).
*   `ONEDRIVE_AUTHORITY_URL` - The authentication token provider URL for the Entra ID app registration (the default is `https://login.microsoftonline.com` if not otherwise specified), represented by `--authority-url` (CLI) or `authority_url` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    onedrive \
      --client-id $ONEDRIVE_CLIENT_ID \
      --client-cred $ONEDRIVE_CLIENT_CRED \
      --authority-url $ONEDRIVE_AUTHORITY_URL \
      --tenant $ONEDRIVE_TENANT \
      --user-pname $ONEDRIVE_USER_PNAME \
      --path $ONEDRIVE_PATH \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.onedrive import (
      OnedriveIndexerConfig,
      OnedriveDownloaderConfig,
      OnedriveConnectionConfig,
      OnedriveAccessConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import LocalConnectionConfig
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=OnedriveIndexerConfig(path=os.getenv("ONEDRIVE_PATH")),
          downloader_config=OnedriveDownloaderConfig(
              download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")
          ),
          source_connection_config=OnedriveConnectionConfig(
              access_config=OnedriveAccessConfig(
                  client_cred=os.getenv("ONEDRIVE_CLIENT_CRED")
              ),
              client_id=os.getenv("ONEDRIVE_CLIENT_ID"),
              tenant=os.getenv("ONEDRIVE_TENANT"),
              user_pname=os.getenv("ONEDRIVE_USER_PNAME"),
              authority_url=os.getenv("ONEDRIVE_AUTHORITY_URL")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig()
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.onedrive import OneDriveAccessConfig, SimpleOneDriveConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import OneDriveRunner

  if __name__ == "__main__":
      runner = OneDriveRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleOneDriveConfig(
              access_config=OneDriveAccessConfig(client_cred=os.getenv("ONEDRIVE_CLIENT_CRED")),
              path=os.getenv("ONEDRIVE_PATH"),
              client_id=os.getenv("ONEDRIVE_CLIENT_ID"),
              tenant=os.getenv("ONEDRIVE_TENANT"),
              authority_url=os.getenv("ONEDRIVE_AUTHORITY_URL"),
              user_pname=os.getenv("ONEDRIVE_USER_PNAME"),
              recursive=False,
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# OpenSearch



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect OpenSearch to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The OpenSearch prerequisites:

*   An OpenSearch instance, such as an [AWS OpenSearch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/setting-up.html) instance...

    <iframe width="560" height="315" src="https://www.youtube.com/embed/BNOYTbRbaFQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    ...or a [local](https://opensearch.org/downloads.html) instance.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/Rew3_pNnYIs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    In all cases, the [OpenSearch version](https://github.com/opensearch-project/opensearch-py/blob/main/COMPATIBILITY.md) must be compatible with the client version of the [opensearch-py](https://github.com/opensearch-project/opensearch-py/blob/main/COMPATIBILITY.md) package on PyPI.

*   The instance's hostname and port number. To learn how, see:

    *   [Creating and managing Amazon OpenSearch Service domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html) for AWS OpenSearch instances.
    *   [Communicate with OpenSearch](https://opensearch.org/docs/latest/getting-started/communicate/) for local OpenSearch instances.

*   The name of the search index on the instance. [Create an index](https://opensearch.org/docs/latest/api-reference/index-apis/create-index/).

    The OpenSearch index that you use must have a schema that is compatible with the schema of the documents
    that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following index schema example for your own needs:

    ```json
    {
        "settings": {
            "index": {
                "knn": true,
                "knn.algo_param.ef_search": 100
            }
        },
        "mappings": {
            "properties": {
                "element_id": {
                    "type": "keyword"
                },
                "text": {
                    "type": "text"
                },
                "embeddings": {
                    "type": "knn_vector",
                    "dimension": 384
                },
                "metadata": {
                    "type": "object",
                    "properties": {
                        "parent_id": {
                            "type": "text"
                        },
                        "page_number": {
                            "type": "integer"
                        },
                        "is_continuation": {
                           "type": "boolean"
                        },
                        "orig_elements": {
                           "type": "text"
                        }
                    }
                }
            }
        }
    }
    ```

    See also:

    *   [Mappings and field types](https://opensearch.org/docs/latest/field-types/)
    *   [Explicit mapping](https://opensearch.org/docs/latest/field-types/#explicit-mapping)
    *   [Dynamic mapping](https://opensearch.org/docs/latest/field-types/#dynamic-mapping)
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

*   If you're using basic authentication to the instance, the user's name and password.

*   If you're using certificates for authentication instead:

    *   The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA.
    *   The path to the combined private key and certificate file, or
    *   The paths to the separate private key and certificate file.

    To learn more, see:

    *   For AWS OpenSearch instances, see [Tutorial: Configure a domain with the internal user database and HTTP basic authentication](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac-http-auth.html) and [Creating a custom endpoint for Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/customendpoint.html).
    *   For local OpenSearch instances, see [Authentication backends](https://opensearch.org/docs/latest/security/authentication-backends/authc-index/), [HTTP basic authentication](https://opensearch.org/docs/latest/security/authentication-backends/basic-authc/), and [Client certificate authentication](https://opensearch.org/docs/latest/security/authentication-backends/client-auth/).

The OpenSearch connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[opensearch]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `OPENSEARCH_HOST` - The hostname and port number, defined as `<hostname>:<port-number>` and represented by `--hosts` (CLI) or `hosts` (Python).
*   `OPENSEARCH_INDEX_NAME` - The name of the search index, represented by `--index-name` (CLI) or `index_name` (Python).

If you're using basic authentication to the instance:

*   `OPENSEARCH_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
*   `OPENSEARCH_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

If you're using certificates for authentication instead:

*   `OPENSEARCH_CA_CERTS` - The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA. This is represented by `--ca-certs` (CLI) or `ca_certs` (Python).
*   `OPENSEARCH_CLIENT_CERT` - The path to the combined private key and certificate file, or the path to just the certificate file. This is represented by `--client-cert` (CLI) or `client_cert` (Python).
*   `OPENSEARCH_CLIENT_KEY` - The path to the private key file, if `OPENSEARCH_CLIENT_CERT` refers to just the certificate file. This is represented by `--client-key` (CLI) or `client_key` (Python).

Additional related settings include:

*   `--use-ssl` (CLI) or `use_ssl=True` (Python) to use SSL for the connection.
*   `--verify-certs` (CLI) or `verify_certs=True` (Python) to verify SSL certificates.
*   `--ssl-show-warn` (CLI) or `ssl_show_warn=True` (Python) to show a warning when verifying SSL certificates is disabled.

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    opensearch \
      --index-name $OPENSEARCH_INDEX_NAME \
      --fields "director,plot" \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --hosts $OPENSEARCH_HOST \
      --username $OPENSEARCH_USERNAME \
      --password $OPENSEARCH_PASSWORD \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.opensearch import (
      OpensearchIndexerConfig,
      OpensearchDownloaderConfig,
      OpenSearchConnectionConfig,
      OpenSearchAccessConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import LocalConnectionConfig
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=OpensearchIndexerConfig(index_name=os.getenv("OPENSEARCH_INDEX_NAME")),
          downloader_config=OpensearchDownloaderConfig(
              download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR"),
              fields=["director", "plot"]
          ),
          source_connection_config=OpenSearchConnectionConfig(
              access_config=OpenSearchAccessConfig(
                  password=os.getenv("OPENSEARCH_PASSWORD"),
                  use_ssl=True
              ),
              hosts=[os.getenv("OPENSEARCH_HOST")],
              username=os.getenv("OPENSEARCH_USERNAME")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig()
      ).run()
  ```

  ```python Python Ingest v1
  from unstructured_ingest.connector.opensearch import (
      OpenSearchAccessConfig,
      SimpleOpenSearchConfig,
  )
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import OpenSearchRunner

  if __name__ == "__main__":
      runner = OpenSearchRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              metadata_exclude=["filename", "file_directory", "metadata.data_source.date_processed"],
          ),
          connector_config=SimpleOpenSearchConfig(
              access_config=OpenSearchAccessConfig(hosts=["http://localhost:9200"]),
              index_name="movies",
              fields=["ethnicity", "director", "plot"],
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Outlook



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Outlook to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Outlook prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/9yESRp9pzv0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Outlook user's email address.
*   A Microsoft Entra ID app registration in the same Azure account as the Outlook account. You will need
    this app registration's application (client) ID, client secret, and directory (tenant) ID. [Learn how](https://learn.microsoft.com/entra/identity-platform/quickstart-register-app).
*   The Entra ID app registration must have the following Graph API permission levels of the application (not delegated) type:

    *   `Mail.Read`
    *   `Mail.ReadBasic`
    *   `User.Read.All`

    [Learn how](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions).

The Outlook connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[outlook]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `OUTLOOK_USER_EMAIL` - The Outlook user's email address, represented by `--user-email` (CLI) or `user_mail` (Python).
*   `OUTLOOK_APP_CLIENT_ID` - The application (client) ID of the Microsoft Entra ID app registration that has access to the user's email account, represented by `--client-id` (CLI) or `client_id` (Python).
*   `OUTLOOK_APP_CLIENT_SECRET` - The client secret for the Entra ID app registration, represented by `--client-cred` (CLI) or `client_cred` (Python).
*   `OUTLOOK_APP_TENANT` - The directory (tenant) ID of the Entra ID app registration, represented by `--tenant` (CLI) or `tenant` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Embedding is optional.

  unstructured-ingest \
    outlook \
      --user-email $OUTLOOK_USER_EMAIL \
      --outlook-folders Inbox \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --client-id $OUTLOOK_APP_CLIENT_ID \
      --client-cred $OUTLOOK_APP_CLIENT_SECRET \
      --tenant $OUTLOOK_APP_TENANT \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --chunking-strategy by_title \
      --embedding-provider huggingface
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.outlook import (
      OutlookIndexerConfig,
      OutlookDownloaderConfig,
      OutlookConnectionConfig,
      OutlookAccessConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Embedding is optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=OutlookIndexerConfig(
              outlook_folders=["Inbox"],
              recursive=False,
              user_email=os.getenv("OUTLOOK_USER_EMAIL")
          ),
          downloader_config=OutlookDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=OutlookConnectionConfig(
              access_config=OutlookAccessConfig(client_cred=os.getenv("OUTLOOK_APP_CLIENT_SECRET")),
              client_id=os.getenv("OUTLOOK_APP_CLIENT_ID"),
              tenant=os.getenv("OUTLOOK_APP_TENANT")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.outlook import OutlookAccessConfig, SimpleOutlookConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import OutlookRunner

  if __name__ == "__main__":
      runner = OutlookRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")
              strategy="hi_res",
          ),
          connector_config=SimpleOutlookConfig(
              access_config=OutlookAccessConfig(
                  client_credential=os.getenv("OUTLOOK_APP_CLIENT_SECRET"),
              ),
              client_id=os.getenv("OUTLOOK_APP_CLIENT_ID"),
              tenant=os.getenv("OUTLOOK_APP_TENANT"),
              user_email=os.getenv("OUTLOOK_USER_EMAIL"),
              outlook_folders=["Inbox"],
              recursive=True,
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Overview



Connect to your favorite data storage platforms for effortless batch processing of your files. We are constantly adding
new data connectors and if you don’t see your favorite platform let us know in our community Slack.

When ingesting data from a storage via a source connector, you typically store the resulting `*.json` files in a
specified output directory.

To "rehydrate" elements in JSON form into in-memory objects to further use in your application, use `elements_from_json`:

```python
from unstructured.staging.base import elements_from_json

elements = elements_from_json(filename=path_to_json_file)
```


# PostgreSQL



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect PostgreSQL to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The PostgreSQL prerequisites, which include the following settings.

The following video shows for example how to get these settings by using [Amazon RDS for PostgreSQL](https://aws.amazon.com/rds/postgresql/).

<iframe width="560" height="315" src="https://www.youtube.com/embed/QuIlEimejDs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A PostgreSQL instance. [Install PostgreSQL](https://www.postgresql.org/docs/current/tutorial-install.html).

*   The host name and port number for the instance. These values are in the `postgresql.conf` file's `listen_addresses` and `port` settings. This file should be on the same machine as the instance. These values might also already be set as environment variables named `PGHOST` and `PGPORT` on the same machine as the instance.

*   A database in the instance. [Create a database](https://www.postgresql.org/docs/current/tutorial-createdb.html).

*   A table in the database. [Create a table](https://www.postgresql.org/docs/current/tutorial-table.html).

    The table's schema must match the schema of the documents that Unstructured produces.
    Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following table schema example for your own needs:

    <CodeGroup>
      ```sql PostgreSQL
      CREATE TABLE elements (
          id UUID PRIMARY KEY,
          element_id VARCHAR,
          text TEXT,
          embeddings DECIMAL [],
          parent_id VARCHAR,
          page_number INTEGER,
          is_continuation BOOLEAN,
          orig_elements TEXT
      );
      ```

      ```sql PostgreSQL with pgvector 
      CREATE EXTENSION vector;

      CREATE TABLE elements (
          id UUID PRIMARY KEY,
          element_id VARCHAR,
          text TEXT,
          embeddings vector(384),
          parent_id VARCHAR,
          page_number INTEGER,
          is_continuation BOOLEAN,
          orig_elements TEXT
      );
      ```
    </CodeGroup>

    See also:

    *   [CREATE TABLE](https://www.postgresql.org/docs/current/sql-createtable.html) for PostgreSQL
    *   [CREATE TABLE](https://github.com/pgvector/pgvector) for PostrgreSQL with pgvector
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

*   A user in the database, and a password for the user. [Create a user](https://www.postgresql.org/docs/current/sql-createuser.html).

*   Database access for the user. [Give database access to a user](https://www.postgresql.org/docs/current/sql-grant.html).

The PostgreSQL connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[postgres]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `PGHOST` - The host name, represented by `--host` (CLI) or `host` (Python).
*   `PGPORT` - The port number, represented by `--port` (CLI) or `port` (Python).
*   `PGUSER` - The username, represented by `--username` (CLI) or `username` (Python).
*   `PGPASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).
*   `PGDATABASE` - The name of the database, represented by `--database` (CLI) or `database` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    postgres \
      --host $PGHOST \
      --port $PGPORT \
      --database $PGDATABASE \
      --table-name elements \
      --id-column id \
      --username $PGUSER \
      --password $PGPASSWORD \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.sql.postgres import (
      PostgresIndexerConfig,
      PostgresDownloaderConfig,
      PostgresConnectionConfig,
      PostgresAccessConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalConnectionConfig,
      LocalUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=PostgresIndexerConfig(
              table_name="elements",
              id_column="id"
          ),
          downloader_config=PostgresDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=PostgresConnectionConfig(
              access_config=PostgresAccessConfig(password=os.getenv("PGPASSWORD")),
              host=os.getenv("PGHOST"),
              port=os.getenv("PGPORT"),
              username=os.getenv("PGUSER"),
              database=os.getenv("PGDATABASE")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig(),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Reddit



Connect Reddit to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to store structured outputs locally on your filesystem.

First, install the Reddit dependencies as shown here.

```bash
pip install "unstructured-ingest[reddit]"
```

You must provide:

*   A client ID and a client secret to authenticate yourself. Learn how to get them [here](https://github.com/reddit-archive/reddit/wiki/OAuth2-Quick-Start-Example#first-steps).
*   `user-agent`: user agent request header to use when calling Reddit API
*   `subreddit-name`: The name of a subreddit, without the "r\\", e.g. "machinelearning"

Optionally you can choose to specify:

*   `search-query`: If set, return posts using this query. Otherwise, use hot posts.
*   `num-posts`: If set, limits the number of posts to pull in.

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  unstructured-ingest \
    reddit \
      --subreddit-name machinelearning \
      --client-id $REDDIT_CLIENT_ID \
      --client-secret $REDDIT_CLIENT_SECRET \
      --user-agent "Unstructured Ingest Subreddit fetcher by \u\..." \
      --search-query "Unstructured" \
      --num-posts 10 \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --strategy hi_res
  ```

  ```python Python
  import os

  from unstructured_ingest.connector.reddit import RedditAccessConfig, SimpleRedditConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import RedditRunner

  if __name__ == "__main__":
      runner = RedditRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          connector_config=SimpleRedditConfig(
              access_config=RedditAccessConfig(
                  client_secret=os.getenv("REDDIT_CLIENT_SECRET"),
              ),
              subreddit_name="machinelearning",
              client_id=os.getenv("REDDIT_CLIENT_ID"),
              user_agent=r"Unstructured Ingest Subreddit fetcher by \\u\...",
              search_query="Unstructured",
              num_posts=10,
          ),
      )
      runner.run()
  ```
</CodeGroup>

For a full list of the options the Unstructured Ingest CLI accepts check `unstructured-ingest reddit --help`.


# S3



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect S3 to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Amazon S3 prerequisites:

The following video shows how to fulfill the minimum set of S3 prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/hyDHfhVVAhs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but
not recommended); or generate an AWS STS session token for temporary access, if required by your organization's security
requirements. For more information about prerequisites, see the following:

*   An AWS account. [Create an AWS account](https://aws.amazon.com/free).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/lIdh92JmWtg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An S3 bucket. [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
    Additional approaches are in the following video and in the how-to sections at the end of this page.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/e6w9LwZJFIA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   Anonymous (supported but not recommended) or authenticated access to the bucket.

    *   [Enable anonymous bucket access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user).

        ![Enable anonymous bucket access](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/connectors/s3-anon-access.gif)

    *   [Enable authenticated bucket access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/walkthrough1.html).

        ![Enable authenticated bucket access](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/connectors/s3-auth-access.gif)

*   For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/y4SfQoJpipo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at
    minimum the permission of `s3:PutObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

*   For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account.
    [Create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/MoFTaGJE65Q" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For authenticated access in untrusted environments or enhanced security scenarios, an AWS STS session token for temporary access, in addition to an AWS access key and secret access key. [Create a session token](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_getsessiontoken).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/4_csSXc_GNU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   If the target files are in the root of the bucket, the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
    If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

*   If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has
    authenticated access to the folder as well. [Enable authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

The S3 connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[s3]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `AWS_S3_URL` - The path to the S3 bucket or folder, formatted as `s3://my-bucket/` (if the files are in the bucket's root) or `s3://my-bucket/my-folder/`.

*   If the bucket does not have anonymous access enabled, provide the AWS credentials:

    *   `AWS_ACCESS_KEY_ID` - The AWS access key ID for the authenticated AWS IAM user, represented by `--key` (CLI) or `key` (Python).
    *   `AWS_SECRET_ACCESS_KEY` - The corresponding AWS secret access key, represented by `--secret` (CLI) or `secret` (Python).
    *   `AWS_TOKEN` - If required, the AWS STS session token for temporary access, represented by `--token` (CLI) or `token` (Python).

*   If the bucket has anonymous access enabled for reading from the bucket, set `--anonymous` (CLI) or `anonymous=True` (Python) instead.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported.

<iframe width="560" height="315" src="https://www.youtube.com/embed/9qSiL0QFsBA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    s3 \
      --remote-url $AWS_S3_URL \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --key $AWS_ACCESS_KEY_ID \
      --secret $AWS_SECRET_ACCESS_KEY \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy fast \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from unstructured_ingest.v2.processes.connectors.fsspec.s3 import (
      S3IndexerConfig,
      S3DownloaderConfig,
      S3ConnectionConfig,
      S3AccessConfig
  )
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=S3IndexerConfig(remote_url=os.getenv("AWS_S3_URL")),
          downloader_config=S3DownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=S3ConnectionConfig(
              access_config=S3AccessConfig(
                  key=os.getenv("AWS_ACCESS_KEY_ID"),
                  secret=os.getenv("AWS_SECRET_ACCESS_KEY")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))        
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.s3 import S3AccessConfig, SimpleS3Config
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import S3Runner

  if __name__ == "__main__":
      runner = S3Runner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleS3Config(
              access_config=S3AccessConfig(
                  key=os.getenv("AWS_ACCESS_KEY_ID"),
                  secret=os.getenv("AWS_SECRET_ACCESS_KEY")
              ),
              remote_url=os.getenv("AWS_S3_URL"),
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2.  Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3.  Browse to the existing bucket and open it.

4.  Click the **Permissions** tab.

5.  In the **Bucket policy** area, click **Edit**.

6.  In the **Policy** text area, copy the following JSON-formatted policy.
    To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this policy, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-bucket-name>` in two places with the name of your bucket.

    ```json
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::<my-account-id>:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::<my-bucket-name>",
                    "arn:aws:s3:::<my-bucket-name>/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    ```

7.  Click **Save changes**.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2.  [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3.  Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
    To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this script, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-unique-bucket-name>` with the name of your bucket.
    *   Replace `<us-east-1>` with your AWS Region.

    ```bash
    #!/bin/bash

    # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
    ACCOUNT_ID="<my-account-id>"
    BUCKET_NAME="<my-unique-bucket-name>"
    REGION="<us-east-1>"

    # Temporary filename for the bucket policy.
    # Do not change this variable.
    POLICY_FILE="bucket_policy.json"

    # Create the bucket.
    aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

    # Wait for the bucket to exist.
    echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
    aws s3api wait bucket-exists --bucket $BUCKET_NAME

    # Check if the wait command was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket '$BUCKET_NAME' has been fully created."
    else
        echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
        exit 1
    fi

    # Remove the "block public policy" bucket access setting.
    aws s3api put-public-access-block \
        --bucket $BUCKET_NAME \
        --public-access-block-configuration \
        '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

    # Check if the operation was successful.
    if [ $? -eq 0 ]; then
        echo "The block public policy access setting was removed from '$BUCKET_NAME'."
    else
        echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
        exit 1
    fi

    # Create the bucket policy.
    cat << EOF > $POLICY_FILE
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::$BUCKET_NAME",
                    "arn:aws:s3:::$BUCKET_NAME/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    EOF

    # Apply the bucket policy.
    aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

    # Check if the policy application was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket policy was applied to '$BUCKET_NAME'."
    else
        echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
        exit 1
    fi

    # Verify the applied policy.
    echo "Verifying the applied policy:"
    aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

    # Remove the temporary bucket policy file.
    rm $POLICY_FILE
    ```

4.  Run the script, for example:

    ```bash
    sh create-s3-bucket.sh
    ```

5.  After the bucket is created, you can delete the script file, if you want.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
    the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    ```yaml
    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

    Parameters:
      BucketName:
        Type: String
        Description: 'Name of the S3 bucket to create'

    Resources:
      MyS3Bucket:
        Type: 'AWS::S3::Bucket'
        Properties:
          BucketName: !Ref BucketName
          PublicAccessBlockConfiguration:
            BlockPublicAcls: true
            BlockPublicPolicy: false
            IgnorePublicAcls: true
            RestrictPublicBuckets: true

      BucketPolicy:
        Type: 'AWS::S3::BucketPolicy'
        Properties:
          Bucket: !Ref MyS3Bucket
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Sid: AllowAllAuthenticatedUsersInAccount
                Effect: Allow
                Principal:
                  AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
                Action:
                  - 's3:GetObject'
                  - 's3:PutObject'
                  - 's3:ListBucket'
                  - 's3:DeleteObject'
                Resource:
                  - !Sub 'arn:aws:s3:::${BucketName}'
                  - !Sub 'arn:aws:s3:::${BucketName}/*'

    Outputs:
      BucketName:
        Description: 'Name of the created S3 bucket'
        Value: !Ref MyS3Bucket
    ```

2.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3.  Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4.  Click **Create stack > With new resources (standard)**.

5.  On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6.  Click **Choose file**, and browse to and select the YAML file from your local machine.

7.  Click **Next**.

8.  Enter a unique **Stack name** and **BucketName**.

9.  Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.


# Salesforce



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Salesforce to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Salesforce prerequisites:

*   A Salesforce account. [Create an account](https://developer.salesforce.com/signup).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/PIVPq2J1OMA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Salesforce categories (objects) that you want to access, specified as a comma-separated list. Available categories include `Account`, `Case`, `Campaign`, `EmailMessage`, and `Lead`.

*   The OAuth consumer key (client ID) and private key (PEM) for the Salesforce connected app. The PEM is a string that begins with `—–BEGIN RSA PRIVATE KEY—–` and ends with `—–END RSA PRIVATE KEY—–`. [Create a connected app](https://help.salesforce.com/s/articleView?id=sf.connected_app_create.htm\&type=5). [Get the consumer key](https://help.salesforce.com/s/articleView?id=sf.connected_app_rotate_consumer_details.htm\&type=5). [Get the private key](https://help.salesforce.com/s/articleView?id=sf.task_create_a_certificate_and_private_key.htm\&type=5).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/GcbcMsMV4oY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    <iframe width="560" height="315" src="https://www.youtube.com/embed/hFV2Ao4aXAY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The Salesforce connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[salesforce]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `SALESFORCE_USERNAME` - The Salesforce username that has access to the required Salesforce categories, represented by `--username` (CLI) or `username` (Python).
*   `SALESFORCE_CONSUMER_KEY` - The consumer key (client ID) for the Salesforce connected app, represented by `--consumer-key` (CLI) or `consumer_key` (Python).
*   `SALESFORCE_PRIVATE_KEY` - The contents of the private key (PEM) associated with the consumer key for the Salesforce connected app, represented by `--private-key` (CLI) or `private_key` (Python), or
*   `SALESFORCE_PRIVATE_KEY_PATH` - The local path to the (PEM) associated with the consumer key for the Salesforce connected app, represented by `--private-key-path` (CLI) or `private_key_path` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    salesforce \
      --username $SALESFORCE_USERNAME \
      --consumer-key $SALESFORCE_CONSUMER_KEY \
      --private-key $SALESFORCE_PRIVATE_KEY \
      --categories "EmailMessage,Account,Lead,Case,Campaign" \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --recursive \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.salesforce import (
      SalesforceIndexerConfig,
      SalesforceDownloaderConfig,
      SalesforceConnectionConfig,
      SalesforceAccessConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SalesforceIndexerConfig(categories=["EmailMessage", "Account", "Lead", "Case", "Campaign"]),
          downloader_config=SalesforceDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=SalesforceConnectionConfig(
              access_config=SalesforceAccessConfig(
                  consumer_key=os.getenv("SALESFORCE_CONSUMER_KEY"),
                  private_key=os.getenv("SALESFORCE_PRIVATE_KEY")
              ),
              username=os.getenv("SALESFORCE_USERNAME")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.salesforce import SalesforceAccessConfig, SimpleSalesforceConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import SalesforceRunner

  if __name__ == "__main__":
      runner = SalesforceRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
          ),
          connector_config=SimpleSalesforceConfig(
              access_config=SalesforceAccessConfig(
                  consumer_key=os.getenv("SALESFORCE_CONSUMER_KEY"),
              ),
              username=os.getenv("SALESFORCE_USERNAME"),
              private_key=os.getenv("SALESFORCE_PRIVATE_KEY"),
              categories=["EmailMessage", "Account", "Lead", "Case", "Campaign"],
              recursive=True,
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# SFTP



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect SFTP to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The SFTP prerequisites:

*   The SFTP server hostname, port, username, and password.

    SFTP servers are offered by several vendors. For example, the following video shows how to create and set up an SFTP server by using AWS Transfer Family:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/3_HHSnoFsoM" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The directory path to start accessing data from, specified as `sftp://<path>/<to>/<directory>`.

See the [SFTP documentation](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol).

The SFTP connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[sftp]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `SFTP_HOST` - The SFTP hostname, represented by `--host` (CLI) or `host` (Python).
*   `SFTP_PORT` - The SFTP port number, represented by `--port` (CLI) or `port` (Python).
*   `SFTP_REMOTE_URL` - The directory path to start accessing data from, represented by `--remote-url` (CLI) or `remote_url` (Python).
*   `SFTP_USERNAME` - The SFTP username, represented by `--username` (CLI) or `username` (Python).
*   `SFTP_PASSWORD` - The SFTP password, represented by `--password` (CLI) or `password` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    sftp \
      --host $SFTP_HOST \
      --port $SFTP_PORT \
      --username $SFTP_USERNAME \
      --password $SFTP_PASSWORD \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --recursive \
      --verbose \
      --strategy hi_res \
      --remote-url $SFTP_REMOTE_URL \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.fsspec.sftp import (
      SftpIndexerConfig,
      SftpDownloaderConfig,
      SftpConnectionConfig,
      SftpAccessConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SftpIndexerConfig(remote_url=os.getenv("SFTP_REMOTE_URL")),
          downloader_config=SftpDownloaderConfig(
              download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR"),
              remote_url=os.getenv("SFTP_REMOTE_URL")
          ),
          source_connection_config=SftpConnectionConfig(
              access_config=SftpAccessConfig(password=os.getenv("SFTP_PASSWORD")),
              host=os.getenv("SFTP_HOST"),
              port=os.getenv("SFTP_PORT"),
              username=os.getenv("SFTP_USERNAME"),
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()

  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.fsspec.sftp import SftpAccessConfig, SimpleSftpConfig
  from unstructured_ingest.interfaces import (
      PartitionConfig,
      ProcessorConfig,
      ReadConfig,
  )
  from unstructured_ingest.runner import SftpRunner

  if __name__ == "__main__":
      runner = SftpRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
          ),
          connector_config=SimpleSftpConfig(
              access_config=SftpAccessConfig(
                  host=os.getenv("SFTP_HOST"),
                  port=os.getenv("SFTP_PORT"),
                  username=os.getenv("SFTP_USERNAME"),
                  password=os.getenv("SFTP_PASSWORD"),
              ),
              remote_url=os.getenv("SFTP_REMOTE_URL"),
              recursive=True,
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# SharePoint



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect SharePoint to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The SharePoint prerequisites, which include the following settings.

<iframe width="560" height="315" src="https://www.youtube.com/embed/r4BLhGgpAWQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The SharePoint site URL.

    *   Site URLs typically have the format `https://<tenant>.sharepoint.com`.
    *   Relative site URLs typically have the format `https://<tenant>.sharepoint.com/sites/<site_name>`.
    *   To process all sites within a tenant, use a site URL of `https://<tenant>-admin.sharepoint.com`. This requires the app to be registered at a tenant level.

    [Learn more](https://learn.microsoft.com/microsoft-365/community/query-string-url-tricks-sharepoint-m365).

*   The path in the SharePoint site from which to start parsing files, for example `"Shared Documents"`. If the connector is to process all sites within the tenant, this filter will be applied to all site document libraries.

*   A SharePoint app principal with its application (client) ID, client secret, and access permissions to the SharePoint instance. [Get a client ID and client secret, and set access permissions](https://github.com/vgrem/Office365-REST-Python-Client/wiki/How-to-connect-to-SharePoint-Online-and-and-SharePoint-2013-2016-2019-on-premises--with-app-principal).

    You can create SharePoint app principals through `https://<tenant>.sharepoint.com/_layouts/15/appregnew.aspx`.

    For a SharePoint app principal with site-scoped permissions, use app permission request XML such as the following to grant
    permissions through `https://<tenant>.sharepoint.com/_layouts/15/appinv.aspx`:

    ```xml
    <AppPermissionRequests AllowAppOnlyPolicy="true">
      <AppPermissionRequest Scope="http://sharepoint/content/sitecollection" Right="FullControl" />
    </AppPermissionRequests>
    ```

    For a SharePoint app principal with tenant-scoped permissions, use app permission request XML such as the following to grant
    permissions through `https://<tenant>-admin.sharepoint.com/_layouts/15/appinv.aspx` instead:

    ```xml
    <AppPermissionRequests AllowAppOnlyPolicy="true">
      <AppPermissionRequest Scope="http://sharepoint/content/tenant" Right="FullControl" />
    </AppPermissionRequests>
    ```

    Available `Right` settings include `Read`, `Write`, `Manage`, and `FullControl`. To learn more, see
    [Add-in permissions in SharePoint](https://learn.microsoft.com/sharepoint/dev/sp-add-ins/add-in-permissions-in-sharepoint).

The SharePoint connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[sharepoint]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `SHAREPOINT_APP_CLIENT_ID` - The SharePoint application (client) ID, represented by `--client-id` (CLI) or `client_id` (Python).
*   `SHAREPOINT_APP_CLIENT_SECRET` - The client secret for the SharePoint application, represented by `--client-cred` (CLI) or `client_cred` (Python).
*   `SHAREPOINT_SITE` - The SharePoint site URL, represented by `--site` (CLI) or `site` (Python).
*   `SHAREPOINT_PATH` - The path in the SharePoint site from which to start parsing files, represented by `--path` (CLI) or `path` (Python).

{/*
- `SHAREPOINT_APP_PERMISSIONS_CLIENT_ID` - The associated Azure application (client) ID, represented by `--permissions-application-id` (CLI) or `permissions_application_id` (Python).
- `SHAREPOINT_APP_PERMISSIONS_CLIENT_SECRET` - The client secret for the Azure application, represented by `--permissions-client-cred` (CLI) or `permissions_client_cred` (Python).
- `SHAREPOINT_APP_PERMISSIONS_TENANT` - The domian name of the tenant for the Azure application, which is typically `<organization-name>.onmicrosoft.com`, and which is represented by `--permissions-tenant` (CLI) or `permissions_tenant` (Python).
 */}

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured-ingest \
    sharepoint \
      --client-id $SHAREPOINT_APP_CLIENT_ID \
      --client-cred $SHAREPOINT_APP_CLIENT_SECRET \
      --site $SHAREPOINT_SITE \
      --path $SHAREPOINT_PATH \
      --no-omit-files \
      --omit-pages \
      --omit-lists \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  from pydantic import Secret

  from unstructured_ingest.v2.processes.connectors.sharepoint import (
      SharepointIndexerConfig,
      SharepointDownloaderConfig,
      SharepointConnectionConfig,
      SharepointAccessConfig,
      SharepointPermissionsConfig
  )

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalConnectionConfig,
      LocalUploaderConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SharepointIndexerConfig(
              path=os.getenv("SHAREPOINT_PATH"),
              recursive=True,  # Recursively process files in their respective folders. (False (default) for no recursion.)
              omit_lists=True, # Do not process lists. (False (default) to process lists.)
              omit_pages=True, # Do not process site pages. (False (default) to process site pages.)
              omit_files=False # Process files (default). (True to not process files.)
          ),
          downloader_config=SharepointDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=SharepointConnectionConfig(
              access_config=SharepointAccessConfig(client_cred=os.getenv("SHAREPOINT_APP_CLIENT_SECRET")),
              client_id=os.getenv("SHAREPOINT_APP_CLIENT_ID"),
              site=os.getenv("SHAREPOINT_SITE"),
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "reprocess": True,
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os

  from unstructured_ingest.connector.sharepoint import (
      SharepointAccessConfig,
      SharepointPermissionsConfig,
      SimpleSharepointConfig,
  )
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import SharePointRunner

  if __name__ == "__main__":
      runner = SharePointRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              strategy="hi_res",
          ),
          connector_config=SimpleSharepointConfig(
              access_config=SharepointAccessConfig(
                  client_cred=os.getenv("SHAREPOINT_APP_CLIENT_SECRET"),
              ),
              client_id=os.getenv("SHAREPOINT_APP_CLIENT_ID"),
              site=os.getenv("SHAREPOINT_SITE"),
              files_only=True,
              path=os.getenv("SHAREPOINT_PATH"),
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# SingleStore



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect SingleStore to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The SingleStore prerequisites:

*   A SingleStore deployment, database, and table. [Learn how](https://www.singlestore.com/blog/how-to-get-started-with-singlestore/).
*   The hostname for the SingleStore deployment.
*   The port for the host.
*   The username for the deployment.
*   The password for the user.
*   The name of the database in the deployment.
*   The name of the table in the database.

To get the values for the hostname, port, username, and password:

1.  In your SingleStore account's dashboard sidebar, click **Deployments**.
2.  From the drop-down list at the top of the **Deployments** page, select your deployment.
3.  On the **Overview** tab, in the **Compute** area, in the **Connect** drop-down list for your deployment, select
    **Your App**.
4.  If a **Create User** dialog box appears, note the **User** name and **Password** values.
5.  In the **Connect to Workspace** pane's **Your App** tab, note the string in the following format:

```
<user-name>:<password>@<host>:<port>
```

*   `<user-name>` is the username.
*   `<password>` is the user's password.
*   `<host>` is the workspace's hostname.
*   `<post>` is the host's port.

To get the values for the database and table names:

1.  In your SingleStore dashboard's sidebar, click **Deployments**.
2.  From the drop-down list at the top of the **Deployments** page, select your deployment.
3.  On the **Databases** tab, note **Name** of your database.
4.  Click the database and, on the **Tables** tab, note the name of your table.

The SingleStore connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[singlestore]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `SINGLESTORE_HOST` - The hostname for the SingleStore deployment, represented by `--host` (CLI) or `host` (Python).
*   `SINGLESTORE_PORT` - The port for the host, represented by `--port` (CLI) or `port` (Python).
*   `SINGLESTORE_USER` - The username for the deployment, represented by `--user` (CLI) or `user` (Python).
*   `SINGLESTORE_PASSWORD` - The password for the user, represented by `--password` (CLI) or `password` (Python).
*   `SINGLESTORE_DB` - The name of the database in the deployment, represented by `--database` (CLI) or `database` (Python).
*   `SINGLESTORE_TABLE` - The name of the table in the database, represented by `--table-name` (CLI) or `table_name` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    singlestore \
      --host $SINGLESTORE_HOST \
      --port $SINGLESTORE_PORT \
      --user $SINGLESTORE_USER \
      --password $SINGLESTORE_PASSWORD \
      --database $SINGLESTORE_DB \
      --table-name $SINGLESTORE_TABLE \
      --id-column id \
      --batch-size 100 \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy fast \
      --chunking-strategy by_title \
      --embedding-provider huggingface
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.sql.singlestore import (
      SingleStoreIndexerConfig,
      SingleStoreDownloaderConfig,
      SingleStoreConnectionConfig,
      SingleStoreAccessConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalUploaderConfig,
      LocalConnectionConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SingleStoreIndexerConfig(
              table_name=os.getenv("SINGLESTORE_TABLE"),
              id_column="id",
              batch_size=100
          ),
          downloader_config=SingleStoreDownloaderConfig(
              download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")
          ),
          source_connection_config=SingleStoreConnectionConfig(
              access_config=SingleStoreAccessConfig(
                  password=os.getenv("SINGLESTORE_PASSWORD")
              ),
              host=os.getenv("SINGLESTORE_HOST"),
              port=os.getenv("SINGLESTORE_PORT"),
              user=os.getenv("SINGLESTORE_USER"),
              database=os.getenv("SINGLESTORE_DB")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig(),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Slack



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Slack to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Slack prerequisites:

*   A Slack app. Create a Slack app by following [Step 1: Creating an app](https://api.slack.com/quickstart#creating).
*   The app must have the `channels:history` OAuth scope. Give the app this scope by following [Step 2: Requesting scopes](https://api.slack.com/quickstart#scopes).
*   The app must be installed and authorized for the target Slack workspace. Install and authorize the app by following [Step 3: Installing and authorizing the app](https://api.slack.com/quickstart#installing).
*   The app's access token. Get this token by following [Step 3: Installing and authorizing the app](https://api.slack.com/quickstart#installing).
*   Add the app to the target channels in the Slack workspace. To do this from the channel, open the channel's details page, click the **Integrations** tab, click **Add apps**, and follow the on-screen directions to install the app.
*   The channel ID for each target channel. To get this ID, open the channel's details page, and look for the **Channel ID** field on the **About** tab.
*   The starting and ending date and time range for the channels to be processed. Supported formats include:

    *   `YYYY-MM-DD`
    *   `YYYY-MM-DDTHH:MM:SS`
    *   `YYYY-MM-DDTHH:MM:SSZ`
    *   `YYYY-MM-DD+HH:MM:SS`
    *   `YYYY-MM-DD-HH:MM:SS`

The Slack connector dependencies:

```bash
pip install "unstructured-ingest[slack]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `SLACK_BOT_USER_OAUTH_TOKEN` - The OAuth token for the Slack app, represented by `--token` (CLI) or `token` (Python).

To specify the starting and ending date and time range for the channels to be processed:

*   For the CLI, use one of the following supported formats:

    *   `YYYY-MM-DD`
    *   `YYYY-MM-DDTHH:MM:SS`
    *   `YYYY-MM-DDTHH:MM:SSZ`
    *   `YYYY-MM-DD+HH:MM:SS`
    *   `YYYY-MM-DD-HH:MM:SS`

*   For Python, use the `datetime.datetime` function.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    slack \
      --token $SLACK_BOT_USER_OAUTH_TOKEN \
      --channels C03FVNHR70A,C03FVNRG43D \
      --start-date 2024-10-22 \
      --end-date 2024-10-23 \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest v2
  import os
  from datetime import datetime

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.slack import (
      SlackIndexerConfig,
      SlackDownloaderConfig,
      SlackConnectionConfig,
      SlackAccessConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SlackIndexerConfig(
              channels=["C03FVNHR70A", "C03FVNRG43D"],
              start_date=datetime(year=2024, month=10, day=22),
              end_date=datetime(year=2024, month=10, day=23)
          ),
          downloader_config=SlackDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=SlackConnectionConfig(
              access_config=SlackAccessConfig(token=os.getenv("SLACK_BOT_USER_OAUTH_TOKEN"))
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```

  ```python Python Ingest v1
  import os
  from datetime import datetime

  from unstructured_ingest.connector.slack import SimpleSlackConfig, SlackAccessConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import SlackRunner

  if __name__ == "__main__":
      runner = SlackRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          connector_config=SimpleSlackConfig(
              access_config=SlackAccessConfig(
                  token=os.getenv("SLACK_BOT_USER_OAUTH_TOKEN"),
              ),
              channels=["C03FVNHR70A", "C03FVNRG43D"],
              start_date=datetime(year=2024, month=10, day=22),
              end_date=datetime(year=2024, month=10, day=23)
          ),
      )
      runner.run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Snowflake



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect Snowflake to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The Snowflake prerequisites:

*   A Snowflake [account](https://signup.snowflake.com/) and its [identifier](https://docs.snowflake.com/user-guide/admin-account-identifier).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/6HNmxQWqmgg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Snowflake [username and its password](https://docs.snowflake.com/user-guide/admin-user-management#creating-users) in the account.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/5hPrJNEa4aA?start=64&end=267" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Snowflake [hostname and its port number](https://docs.snowflake.com/sql-reference/functions/system_allowlist) in the account.

*   The name of the Snowflake [database](https://docs.snowflake.com/sql-reference/sql/create-database) in the account.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/KrXCgpsRd0A" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The name of the [schema](https://docs.snowflake.com/sql-reference/sql/create-schema) in the database.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/4SoS_c6rT_U?start=97&end=157" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The name of the [table](https://docs.snowflake.com/sql-reference/sql/create-table) in the schema.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/ZaZzqFMGe1g?end=169" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The Snowflake connector dependencies:

```bash CLI, Python
pip install "unstructured-ingest[snowflake]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

These environment variables:

*   `SNOWFLAKE_ACCOUNT` - The ID of the Snowflake account, represented by `--account` (CLI) or `account` (Python).
*   `SNOWFLAKE_USER`  - The name of the Snowflake user, represented by `--user` (CLI) or `user` (Python).
*   `SNOWFLAKE_PASSWORD`  - The user's password, represented by `--password` (CLI) or `password` (Python).
*   `SNOWFLAKE_HOST`  - The hostname for the Snowflake account, represented by `--host` (CLI) or `host` (Python).
*   `SNOWFLAKE_PORT`  - The host's port number, represented by `--port` (CLI) or `port` (Python).
*   `SNOWFLAKE_DATABASE`  - The name of the Snowflake database, represented by `--database` (CLI) or `database` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  unstructured \
    snowflake \
      --account $SNOWFLAKE_ACCOUNT \
      --user $SNOWFLAKE_USER \
      --password $SNOWFLAKE_PASSWORD \
      --host $SNOWFLAKE_HOST \
      --port $SNOWFLAKE_PORT \
      --database $SNOWFLAKE_DATABASE \
      --schema PUBLIC \
      --table-name elements\
      --id-column id \
      --batch-size 100 \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR\
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.sql.snowflake import (
      SnowflakeIndexerConfig,
      SnowflakeConnectionConfig,
      SnowflakeAccessConfig,
      SnowflakeDownloaderConfig
  )
  from unstructured_ingest.v2.processes.connectors.local import LocalUploaderConfig
  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SnowflakeIndexerConfig(
              table_name="elements",
              id_column="id",
              batch_size=100
          ),
          downloader_config=SnowflakeDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=SnowflakeConnectionConfig(
              access_config=SnowflakeAccessConfig(password=os.getenv("SNOWFLAKE_PASSWORD")),
              account=os.getenv("SNOWFLAKE_ACCOUNT"),
              user=os.getenv("SNOWFLAKE_USER"),
              host=os.getenv("SNOWFLAKE_HOST"),
              port=os.getenv("SNOWFLAKE_PORT"),
              database=os.getenv("SNOWFLAKE_DATABASE"),
              schema="PUBLIC"
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# SQLite



<Tip>This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).</Tip>

Connect SQLite to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need:

The SQLite prerequisites:

*   A SQLite instance. [Download and install SQLite](https://www.sqlitetutorial.net/download-install-sqlite/).
*   A SQLite database. [Create a database](https://www.sqlite.org/quickstart.html).
*   The path to the database's `.db` file.
*   A table in the database. [Create a table](https://www.sqlite.org/cli.html).

    The table's schema must match the schema of the documents that Unstructured produces.
    Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following table schema example for your own needs:

    ```sql SQLite
    CREATE TABLE elements (
        id TEXT PRIMARY KEY,
        element_id TEXT,
        text TEXT,
        embeddings TEXT,
        parent_id TEXT,
        page_number INTEGER,
        is_continuation INTEGER,
        orig_elements TEXT
    );
    ```

    See also:

    *   [CREATE TABLE](https://www.sqlite.org/lang_createtable.html) for SQLite
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).

The following environment variables:

*   `SQLITE_DB_PATH` - The path to the database's `.db` file, represented by `--database` (CLI) or `database` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured API services for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    sqlite \
      --database-path $SQLITE_DB_PATH \
      --table-name elements \
      --id-column id \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest v2
  import os

  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig

  from unstructured_ingest.v2.processes.connectors.sql.sqlite import (
      SQLiteIndexerConfig,
      SQLiteDownloaderConfig,
      SQLiteConnectionConfig,
      SQLiteAccessConfig
  )

  from unstructured_ingest.v2.processes.partitioner import PartitionerConfig
  from unstructured_ingest.v2.processes.chunker import ChunkerConfig
  from unstructured_ingest.v2.processes.embedder import EmbedderConfig

  from unstructured_ingest.v2.processes.connectors.local import (
      LocalConnectionConfig,
      LocalUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SQLiteIndexerConfig(
              table_name="elements",
              id_column="id"
          ),
          downloader_config=SQLiteDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=SQLiteConnectionConfig(
              access_config=SQLiteAccessConfig(),
              database_path=os.getenv("SQLITE_DB_PATH")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig(),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

*   To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

    Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

*   To send files to [Unstructured API services](/api-reference/api-services/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

    Unstructured API services also requires an Unstructured API key and API URL, by adding the following:

    *   `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
    *   `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
    *   The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

    [Get an API key and API URL](/api-reference/api-services/saas-api-development-guide#get-started).


# Wikipedia



Connect Wikipedia to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to store structured outputs locally on your filesystem.

First, install the Wikipedia dependencies as shown here.

```bash
pip install "unstructured-ingest[wikipedia]"
```

Provide the `page-title` to ingest the text from.

<CodeGroup>
  ```bash Shell
  #!/usr/bin/env bash

  unstructured-ingest \
    wikipedia \
      --page-title "Open Source Software" \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --strategy hi_res
  ```

  ```python Python
  from unstructured_ingest.connector.wikipedia import SimpleWikipediaConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import WikipediaRunner

  if __name__ == "__main__":
      runner = WikipediaRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          connector_config=SimpleWikipediaConfig(
              page_title="Open Source Software",
              auto_suggest=False,
          ),
      )
      runner.run()
  ```
</CodeGroup>

For a full list of the options the Unstructured Ingest CLI accepts check `unstructured-ingest wikipedia --help`.


# Docker Installation

The instructions below guide you on how to use the unstructured library inside a Docker container.

## Prerequisites

If you haven’t installed Docker on your machine, you can find the installation guide [here](https://docs.docker.com/get-docker/).

<Note>
  We build multi-platform images to support both x86\_64 and Apple silicon hardware. Using docker pull should download the appropriate image for your architecture. However, if needed, you can specify the platform with the –platform flag, e.g., –platform linux/amd64.

  We do not support GPU usage with the Unstructured library inside a Docker container.
</Note>

## Pulling the Docker Image

We create Docker images for every push to the main branch. These images are tagged with the respective short commit hash (like fbc7a69) and the application version (e.g., 0.5.5-dev1). The most recent image also receives the latest tag. To use these images, pull them from our repository:

```go
docker pull downloads.unstructured.io/unstructured-io/unstructured:latest

```

## Using the Docker Image

After pulling the image, you can create and start a container from it:

```go
# create the container
docker run -dt --name unstructured downloads.unstructured.io/unstructured-io/unstructured:latest

# start a bash shell inside the running Docker container
docker exec -it unstructured bash

```

## Building Your Own Docker Image

You can also build your own Docker image. If you only plan to parse a single type of data, you can accelerate the build process by excluding certain packages or requirements needed for other data types. Refer to the Dockerfile to determine which lines are necessary for your requirements.

```go
make docker-build

# start a bash shell inside the running Docker container
make docker-start-bash

```

## Interacting with Python Inside the Container

Once inside the running Docker container, you can directly test the library using Python’s interactive mode:

```go
python3

>>> from unstructured.partition.pdf import partition_pdf
>>> elements = partition_pdf(filename="example-docs/pdf/layout-parser-paper-fast.pdf")

>>> from unstructured.partition.text import partition_text
>>> elements = partition_text(filename="example-docs/fake-text.txt")

```


# Full Installation



**Basic Usage**

For a complete set of extras catering to every document type, use:

```bash
pip install "unstructured[all-docs]"
```

To install `unstructured`, you'll also need to install the following system dependencies:
[libmagic](https://man7.org/linux/man-pages/man3/libmagic.3.html),
[poppler](https://poppler.freedesktop.org/),
[libreoffice](https://www.libreoffice.org/discover/libreoffice/),
[pandoc](https://pandoc.org/),
and [tesseract](https://github.com/tesseract-ocr/tesseract).
Instruction details for these dependencies will vary by operating system. We recommend
running `unstructured` from the officially supported Docker image, which has these dependencies
installed already.

**Installation for Specific Document Types**

If you’re processing document types beyond the basics, you can install the necessary extras:

```bash
pip install "unstructured[docx,pptx]"

```

*Available document types:*

```bash
"csv", "doc", "docx", "epub", "image", "md", "msg", "odt", "org", "pdf", "ppt", "pptx", "rtf", "rst", "tsv", "xlsx"

```

**Installation for Specific Data Connectors**

To use any of the data connectors, you must install the specific dependency:

```bash
pip install "unstructured-ingest[s3]"

```

*Available data connectors:*

```bash
"airtable", "azure", "azure-ai-search", "biomed", "box", "confluence", "couchbase", "delta-table", "discord", "dropbox", "elasticsearch", "gcs", "github", "gitlab", "google-drive", "jira", "mongodb", "notion", "opensearch", "onedrive", "outlook", "reddit", "s3", "sharepoint", "salesforce", "slack", "wikipedia"

```


# Overview

Unstructured open source library offers flexible options for installation.

Refer to the Full installation guide to learn how to install Unstructured with all modules, or how to trim down the installation
to only what you require.

Refer to the Docker installation to learn how to run Unstructured in a Docker container.

<CardGroup cols={2}>
  <Card iconType="duotone" title="Full Installation" icon="download" href="/open-source/installation/full-installation" />

  <Card iconType="duotone" title="Docker Installation" icon="docker" href="/open-source/installation/docker-installation" />
</CardGroup>


# Integrations

Integrate your model development pipeline with your favorite machine learning frameworks and libraries, and prepare your data for ingestion into downstream systems. Most of our integrations come in the form of [staging functions](/open-source/core-functionality/staging), which take a list of `Element` objects as input and return formatted dictionaries as output.

## `Integration with Argilla`

You can convert a list of `Text` elements to an [Argilla](https://www.argilla.io/) `Dataset` using the [stage\_for\_argilla](/open-source/core-functionality/staging#stage-for-argilla) staging function. Specify the type of dataset to be generated using the `argilla_task` parameter. Valid values are `"text_classification"`, `"token_classification"`, and `"text2text"`. Follow the link for more details on usage.

## `Integration with Baseplate`

[Baseplate](https://docs.baseplate.ai/introduction) is a backend optimized for use with LLMs that has an easy to use spreadsheet interface. The `unstructured` library offers a staging function to convert a list of `Element` objects into the [rows format](https://docs.baseplate.ai/api-reference/documents/overview) required by the Baseplate API. See the [stage\_for\_baseplate](/open-source/core-functionality/staging#stage-for-baseplate) documentation for information on how to stage elements for ingestion into Baseplate.

## `Integration with Datasaur`

You can format a list of `Text` elements as input to token based tasks in [Datasaur](https://datasaur.ai/) using the [stage\_for\_datasaur](/open-source/core-functionality/staging#stage-for-datasaur) staging function. You will obtain a list of dictionaries indexed by the keys `"text"` with the content of the element, and `"entities"` with an empty list. Follow the link to learn how to customise your entities and for more details on usage.

## `Integration with Hugging Face`

You can prepare `Text` elements for processing in Hugging Face [Transformers](https://huggingface.co/docs/transformers/index) pipelines by splitting the elements into chunks that fit into the model’s attention window using the [stage\_for\_transformers](/open-source/core-functionality/staging#stage-for-transformers) staging function. You can customise the transformation by defining the `buffer` and `window_size`, the `split_function` and the `chunk_separator`. if you need to operate on text directly instead of `unstructured` `Text` objects, use the [chunk\_by\_attention\_window](/open-source/core-functionality/staging#stage-for-transformers) helper function. Follow the links for more details on usage.

## `Integration with Labelbox`

You can format your outputs for use with [LabelBox](https://labelbox.com/) using the [stage\_for\_label\_box](/open-source/core-functionality/staging#stage-for-label-box) staging function. LabelBox accepts cloud-hosted data and does not support importing text directly. With this integration you can stage the data files in the `output_directory` to be uploaded to a cloud storage service (such as S3 buckets) and get a config of type `List[Dict[str, Any]]` that can be written to a `.json` file and imported into LabelBox. Follow the link to see how to generate the `config.json` file that can be used with LabelBox, how to upload the staged data files to an S3 bucket, and for more details on usage.

## `Integration with Label Studio`

You can format your outputs for upload to [Label Studio](https://labelstud.io/) using the [stage\_for\_label\_studio](/open-source/core-functionality/staging#stage-for-label-studio) staging function. After running `stage_for_label_studio`, you can write the results to a JSON folder that is ready to be included in a new Label Studio project. You can also include pre-annotations and predictions as part of your upload.

See the [Label Studio docs](https://labelstud.io/tags/labels.html) for a full list of options for labels and annotations.

## `Integration with LangChain`

Our integration with [LangChain](https://github.com/hwchase17/langchain) makes it incredibly easy to combine language models with your data, no matter what form it is in. The [Unstructured.io File Loader](https://python.langchain.com/en/latest/modules/indexes/document_loaders/examples/unstructured_file.html) extracts the text from a variety of unstructured text files using our `unstructured` library. It is designed to be used as a way to load data into LangChain. Here is the simplest way to use the `UnstructuredFileLoader` in `langchain`.

```python
from langchain.document_loaders import UnstructuredFileLoader

loader = UnstructuredFileLoader("state_of_the_union.txt")
loader.load()

```

Checkout the [LangChain docs](https://python.langchain.com/en/latest/modules/indexes/document_loaders.html) for more examples about how to use Unstructured data loaders.

## `Integration with LlamaIndex`

To use `Unstructured.io File Loader` you will need to have [LlamaIndex](https://github.com/jerryjliu/llama_index) 🦙 (GPT Index) installed in your environment. Just `pip install llama-index` and then pass in a `Path` to a local file. Optionally, you may specify split\_documents if you want each element generated by `unstructured` to be placed in a separate document. Here is a simple example of how to use it:

```python
from pathlib import Path
from llama_index import download_loader


UnstructuredReader = download_loader("UnstructuredReader")

loader = UnstructuredReader()
documents = loader.load_data(file=Path('./10k_filing.html'))

```

See [here](https://llamahub.ai/) for more LlamaHub examples.

## `Integration with Pandas`

You can convert a list of `Element` objects to a Pandas dataframe with columns for the text from each element and their types such as `NarrativeText` or `Title` using the [convert\_to\_dataframe](/open-source/core-functionality/staging#convert-to-dataframe) staging function. Follow the link for more details on usage.

## `Integration with Prodigy`

You can format your JSON or CSV outputs for use with [Prodigy](https://prodi.gy/docs/api-loaders) using the [stage\_for\_prodigy](/open-source/core-functionality/staging#stage-for-prodigy) and [stage\_csv\_for\_prodigy](/open-source/core-functionality/staging#stage-csv-for-prodigy) staging functions. After running `stage_for_prodigy` | `stage_csv_for_prodigy`, you can write the results to a `.json` | `.jsonl` or a `.csv` file that is ready to be used with Prodigy. Follow the links for more details on usage.

## `Integration with Weaviate`

[Weaviate](https://weaviate.io/) is an open-source vector database that allows you to store data objects and vector embeddings from a variety of ML models. Storing text and embeddings in a vector database such as Weaviate is a key component of the [emerging LLM tech stack](https://medium.com/@unstructured-io/llms-and-the-emerging-ml-tech-stack-bdb189c8be5c). See the [stage\_for\_weaviate](open-source/core-functionality/staging#stage-for-weaviate) docs for details on how to upload `unstructured` outputs to Weaviate. An example notebook is also available [here](https://github.com/Unstructured-IO/unstructured/tree/main/examples/weaviate).


# Unstructured Open Source



<Note>The `unstructured` open source library is designed as a starting point for quick prototyping and has [limits](#limits). For production scenarios, see [Unstructured API services](/api-reference/api-services/overview) and [Unstructured Platform](/platform/overview).</Note>

The `unstructured` [library](https://github.com/Unstructured-IO/unstructured) offers an open-source toolkit
designed to simplify the ingestion and pre-processing of diverse data formats, including images and text-based documents
such as PDFs, HTML files, Word documents, and more. With a focus on optimizing data workflows for Large Language Models (LLMs),
`unstructured` provides modular functions and connectors that work seamlessly together. This cohesive system ensures
efficient transformation of unstructured data into structured formats, while also offering adaptability to various platforms
and use cases.

## Key functionality

*   **Precise Document Extraction**: Unstructured offers advanced capabilities in extracting elements and metadata from documents. This includes a variety of document element types and metadata. Learn more about [Document elements and metadata](../concepts/document-elements).

*   **Extensive File Support**: The platform supports a wide array of file types, ensuring versatility in handling different document formats from PDF, Images, HTML, and many more. Detailed information on supported file types can be found [here](/open-source/introduction/supported-file-types).

*   **Robust Core Functionality**: Unstructured provides a suite of core functionalities critical for efficient data processing. This includes:

    *   [Partitioning](/open-source/core-functionality/partitioning): The partitioning functions in Unstructured enable the extraction of structured content from raw, unstructured documents. This feature is crucial for transforming unorganized data into usable formats, aiding in efficient data processing and analysis.

    *   [Cleaning](/open-source/core-functionality/cleaning): Data preparation for NLP models often requires cleaning to ensure quality. The Unstructured library includes cleaning functions that assist in sanitizing output, removing unwanted content, and improving the performance of NLP models. This step is essential for maintaining the integrity of data before it is passed to downstream applications.

    *   [Extracting](/open-source/core-functionality/extracting): This functionality allows for the extraction of specific entities within documents. It is designed to identify and isolate relevant pieces of information, making it easier for users to focus on the most pertinent data in their documents.

    *   [Staging](/open-source/core-functionality/staging): Staging functions help prepare your data for ingestion into downstream systems. Please note that this functionality is being deprecated in favor of `Destination Connectors`.

    *   [Chunking](/open-source/core-functionality/chunking): The chunking process in Unstructured is distinct from conventional methods. Instead of relying solely on text-based features to form chunks, Unstructured uses a deep understanding of document formats to partition documents into semantic units (document elements).

*   **High-performant Connectors**: The platform includes optimized connectors for efficient data ingestion and output. These comprise [Source Connectors](../ingest/source-connectors/overview) for data input and [Destination Connectors](../ingest/destination-connectors/overview) for data export.

## Common use cases

*   Pretraining models
*   Fine-tuning models
*   Retrieval Augmented Generation (RAG)
*   Traditional ETL

<Note>We do not support GPU usage with the open source library.</Note>

## Limits

The open source library has the following limits as compared to [Unstructured API services](/api-reference/api-services/overview) and the [Unstructured Platform](/platform/overview):

*   Not designed for production scenarios.
*   Significantly decreased performance on document and table extraction.
*   Access only to older and less sophisticated vision transformer models.
*   No access to Unstructured's fine-tuned OCR models.
*   No access to Unstructured's by-page and by-similarity chunking strategies.
*   Lack of security and SOC2 and HIPAA compliance.
*   No authentication or identity management.
*   No incremental data loading.
*   No ETL job scheduling or monitoring.
*   No image extraction from documents.
*   Less sophisticated document hierarchy detection.
*   You must manage many of your own code dependencies, for instance for libraries such as Poppler and Tesseract.
*   You must manage your own infrastructure, including parallelization and other performance optimizations.

## Telemetry

The open source library allows you to make calls to Unstructured Serverless API services. If you do plan to make such calls, please note:

We’ve partnered with [Scarf](https://scarf.sh) to collect anonymized user statistics to understand which features our community is using and how to prioritize product decision-making in the future.

To learn more about how we collect and use this data, please read our [Privacy Policy](https://unstructured.io/privacy-policy).

To opt out of this data collection, you can set the environment variable `SCARF_NO_ANALYTICS=true` before running any commands that call Unstructured Serverless API services.


# Quickstart

Using Unstructured Open Source.

## Installation

1.  **Installing the open source library**: You can install the core SDK using pip:

    `pip install unstructured`

    Plain text files, HTML, XML, JSON, and Emails are immediately supported without any additional dependencies.

    If you need to process other document types, you can install the extras required by following the [Full Installation](open-source/installation/full-installation)

2.  **System Dependencies**: Ensure the subsequent system dependencies are installed. Your requirements might vary based on the document types you’re handling:

    *   *libmagic-dev* : Essential for filetype detection.

    *   *poppler-utils* : Needed for images and PDFs.

    *   *tesseract-ocr* : Essential for images and PDFs.

    *   *libreoffice* : For MS Office documents.

    *   *pandoc* : For EPUBs, RTFs, and Open Office documents. Please note that to handle RTF files, you need version 2.14.2 or newer. Running [this script](https://github.com/Unstructured-IO/unstructured/blob/main/scripts/install-pandoc.sh) will install the correct version for you.

## Validating Installation

After installation, confirm the setup by executing the below Python code:

```python
from unstructured.partition.auto import partition
elements = partition(filename="example-docs/eml/fake-email.eml")
```

If you’ve opted for the “local-inference” installation, you should also be able to execute:

```python
from unstructured.partition.auto import partition
elements = partition("example-docs/pdf/layout-parser-paper.pdf")

```

If these code snippets run without errors, congratulations! Your `unstructured` installation is successful and ready for use.

The following section will cover basic concepts and usage patterns in `unstructured`. After reading this section, you should be able to:

*   Partitioning a document with the `partition` function.

*   Understand how documents are structured in `unstructured`.

*   Convert a document to a dictionary and/or save it as a JSON.

The example documents in this section come from the [example-docs](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/example-docs) directory in the `unstructured` repo.

Before running the code in this make sure you’ve installed the `unstructured` library and all dependencies using the instructions in the [quickstart](../installation/overview#quick-start) section.

## Partitioning a document

In this section, we’ll cut right to the chase and get to the most important part of the library: partitioning a document. The goal of document partitioning is to read in a source document, split the document into sections, categorize those sections, and extract the text associated with those sections. Depending on the document type, unstructured uses different methods for partitioning a document. We’ll cover those in a later section. For now, we’ll use the simplest API in the library, the `partition` function. The `partition` function will detect the filetype of the source document and route it to the appropriate partitioning function. You can try out the partition function by running the cell below.

```python
from unstructured.partition.auto import partition

elements = partition(filename="example-10k.html")

```

You can also pass in a file as a file-like object using the following workflow:

```python
with open("example-10k.html", "rb") as f:
    elements = partition(file=f)

```

The `partition` function uses [libmagic](https://formulae.brew.sh/formula/libmagic) for filetype detection. If `libmagic` is not present and the user passes a filename, `partition` falls back to detecting the filetype using the file extension. `libmagic` is required if you’d like to pass a file-like object to `partition`. We highly recommend installing `libmagic` and you may observe different file detection behaviors if `libmagic` is not installed\`.

<Note>The `partition` methods and functions in the open source library process single files only. To process multiple files at a time, use the [Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) with their provided [source connectors](/open-source/ingest/source-connectors/overview) and [destination connectors](/open-source/ingest/destination-connectors/overview).</Note>

## Quickstart Tutorial

If you’re eager to dive in, head over [Getting Started](https://colab.research.google.com/drive/1U8VCjY2-x8c6y5TYMbSFtQGlQVFHCVIW#scrollTo=jZp37lfueaeZ) on Google Colab to get a hands-on introduction to the `unstructured` library. In a few minutes, you’ll have a basic workflow set up and running!

For more detailed information about specific components or advanced features, explore the rest of the documentation.


# Supported file types



Unstructured supports processing of the following file types:

By file extension:

| File extension |
| -------------- |
| `.bmp`         |
| `.csv`         |
| `.doc`         |
| `.docx`        |
| `.eml`         |
| `.epub`        |
| `.heic`        |
| `.html`        |
| `.jpeg`        |
| `.png`         |
| `.md`          |
| `.msg`         |
| `.odt`         |
| `.org`         |
| `.p7s`         |
| `.pdf`         |
| `.png`         |
| `.ppt`         |
| `.pptx`        |
| `.rst`         |
| `.rtf`         |
| `.tiff`        |
| `.txt`         |
| `.tsv`         |
| `.xls`         |
| `.xlsx`        |
| `.xml`         |

By file type:

| Category          | File types                                |
| ----------------- | ----------------------------------------- |
| CSV               | `.csv`                                    |
| E-mail            | `.eml`, `.msg`, `.p7s`                    |
| EPUB              | `.epub`                                   |
| Excel             | `.xls`, `.xlsx`                           |
| HTML              | `.html`                                   |
| Image             | `.bmp`, `.heic`, `.jpeg`, `.png`, `.tiff` |
| Markdown          | `.md`                                     |
| Org Mode          | `.org`                                    |
| Open Office       | `.odt`                                    |
| PDF               | `.pdf`                                    |
| Plain text        | `.txt`                                    |
| PowerPoint        | `.ppt`, `.pptx`                           |
| reStructured Text | `.rst`                                    |
| Rich Text         | `.rtf`                                    |
| TSV               | `.tsv`                                    |
| Word              | `.doc`, `.docx`                           |
| XML               | `.xml`                                    |

See also [Partitioning](/open-source/core-functionality/partitioning).


# Chunking



After partitioning, *chunking* rearranges the resulting document elements into manageable "chunks" to stay within
the limits of an embedding model and to improve retrieval precision. The goal is to retrieve only parts of documents
that contain only the information that is relevant to a user's query. You can specify if and how Unstructured chunks
those elements, based on your intended end use.

During chunking, Unstructured uses a basic chunking strategy that attempts to combine two or more consecutive text elements
into each chunk that fits together within **Max characters**. To determine the best **Max characters** length, see the documentation
for the embedding model that you want to use.

You can further control this behavior with by-title, by-page, or by-similarity chunking strategies.
In all cases, Unstructured will only split individual elements if they exceed the specified **Max characters** length.
After chunking, you will have document elements of only the following types:

*   `CompositeElement`: Any text element will become a `CompositeElement` after chunking. A composite element can be a
    combination of two or more original text elements that together fit within the **Max characters** length. It can also be a single
    element that doesn't leave room in the chunk for any others but fits by itself. Or it can be a fragment of an original
    text element that was too big to fit in one chunk and required splitting.
*   `Table`: A table element is not combined with other elements, and if it fits within **Max characters** it will remain as is.
*   `TableChunk`: Large tables that exceed **Max characters** are split into special `TableChunk` elements.

Here are a few examples:

```json
{
    "type": "CompositeElement",
    "element_id": "3e891e70121efac5f8374e4f1d41f25c",
    "text": "PORTABLE DOCUMENT FORMAT (PDF) SPECIFICATIONS...<full-content-redacted-for-brevity>...",
    "metadata": {
      "filetype": "application/pdf",
      "languages": [
        "eng"
      ],
      "page_number": 1,
      "orig_elements": "eJy...<full-content-redacted-for-brevity>...tY0=",
      "filename": "Portable-Document-Format-Specifications.pdf"
    }
}
```

```json
{
    "type": "Table",
    "element_id": "2abca97d9f7dde6557cda4731f5e1d01",
    "text": "Date 2005-04-08 2008-06-04 2011-12-20 2012-1-13 2014-9-26 2016-9-14...<full-content-redacted-for-brevity>...",
    "metadata": {
      "text_as_html": "<table><thead><tr><th>Date</th><th>Version</th><th>Summary of Revisions</th></tr></thead><tbody>...<full-content-redacted-for-brevity>...",
      "filetype": "application/pdf",
      "languages": [
        "eng"
      ],
      "page_number": 3,
      "orig_elements": "eJz...<full-content-redacted-for-brevity>...r8=",
      "filename": "Portable-Document-Format-Specifications.pdf"
    }
}
```

The following sections provide information about the available chunking strategies and their settings.

<Note>You can change a workflow's predefined strategy only through [Custom](/platform/workflows#create-a-custom-workflow) workflow settings.</Note>

## Basic chunking strategy

The basic chunking strategy uses only **Max characters** and **New after n characters** to combine sequential elements to maximally fill each chunk.
This strategy does not use section boundaries, page boundaries, or content similarities to determine the chunks' contents.

To use this chunking strategy, choose **Chunk by character** in the **Chunkers** section of a **Chunker** node in a workflow.

## Chunk by title strategy

The by-title chunking strategy attempts to preserve section boundaries when determining the chunks' contents.
A single chunk should not contain text that occurred in two different sections. When a new section starts, the existing
chunk is closed and a new one is started, even if the next element would fit in the prior chunk.

To use this chunking strategy, choose **Chunk by title** in the **Chunkers** section of a **Chunker** node in a workflow.

## Chunk by page strategy

The by-page chunking strategy attempts to preserve page boundaries when determining the chunks' contents.
A single chunk should not contain text that occurred in two different page. When a new page starts, the existing
chunk is closed and a new one is started, even if the next element would fit in the prior chunk.

To use this chunking strategy, choose **Chunk by page** in the **Chunkers** section of a **Chunker** node in a workflow.

## Chunk By similarity strategy

The by-similarity chunking strategy uses the
[sentence-transformers/multi-qa-mpnet-base-dot-v1](https://huggingface.co/sentence-transformers/multi-qa-mpnet-base-dot-v1) embedding model
to identify topically similar sequential elements and combines them into chunks.

As with the other chunking strategies, chunks will never exceed the absolute maximum chunk size set by **Max characters**. For this reason,
not all elements that share a topic will necessarily appear in the same chunk. However, with this strategy you can
guarantee that two elements with low similarity will not be combined in a single chunk.

To use this chunking strategy, choose **Chunk by similarity** in the **Chunkers** section of a **Chunker** node in a workflow.

You can control the level of topic similarity you require for elements to have by setting [Similarity threshold](#similarity-threshold).

## Max characters setting

Specifies the absolute maximum number of characters in a chunk.

To specify this setting, enter a number into the **Max characters** field.

This setting applies to all of the chunking strategies.

## Combine text under n characters setting

Combines elements from a section into a chunk until a section reaches a length of this many characters.

To specify this setting, enter a number into the **Combine text under n chars** field.

This setting applies only to the chunking strategy **Chunk by title**.

## Include original elements setting

If the **Include original elements** box is checked, the elements that were used to form a chunk appear in the `metadata` field's `orig_elements` field for that chunk.

This setting applies to all of the chunking strategies.

## Multipage sections setting

If the **Multipage sections** box is checked, this allows sections to span multiple pages.

This setting applies only to the chunking strategy **Chunk by title**.

## New after n characters setting

Closes new sections after reaching a length of this many characters. This is an approximate limit.

To specify this setting, enter a number into the **New after n characters** field.

This setting applies only to the chunking strategies **Chunk by character**, **Chunk by title**, and **Chunk by page**.

## Overlap setting

Applies a prefix of this many trailing characters from the prior text-split chunk to second and later chunks formed from oversized elements by text-splitting.

To specify this setting, enter a number into the **Overlap** field.

This setting applies only to the chunking strategies **Chunk by character**, **Chunk by title**, and **Chunk by page**.

## Overlap all setting

If the **Overlap all** box is checked, applies overlap to "normal" chunks formed by combining whole elements. Use with caution as this can introduce noise into otherwise clean semantic units.

This setting applies only to the chunking strategies **Chunk by character**, **Chunk by title**, and **Chunk by page**.

## Similarity threshold setting

Specifies the minimum similarity that text in consecutive elements must have to be included in the same chunk.
This must be a value between `0.0` and `1.0`, exclusive (`0.01` to `0.99`). The default is `0.5` if not otherwise specified.

To specify this setting, enter a number into the **Similarity threshold** field.

This setting applies only to the chunking strategy **Chunk by similarity**.

## Learn more

<Icon icon="blog" />  [Chunking for RAG: best practices](https://unstructured.io/blog/chunking-for-rag-best-practices).


# Supported connectors



The Unstructured Platform supports connecting to the following source and destination types.

```mermaid
  flowchart LR
    Sources-->Unstructured-->Destinations
```

## Sources

*   [Azure](/platform/sources/azure-blob-storage)
*   [Confluence](/platform/sources/confluence)
*   [Databricks Volumes](/platform/sources/databricks-volumes)
*   [Google Cloud Storage](/platform/sources/google-cloud)
*   [MongoDB](/platform/sources/mongodb)
*   [OneDrive](/platform/sources/onedrive)
*   [Outlook](/platform/sources/outlook)
*   [S3](/platform/sources/s3)
*   [SharePoint](/platform/sources/sharepoint)

If your source is not listed here, you might still be able to connect Unstructured to it through scripts or code by using the
[Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
[Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library).
[Learn more](/api-reference/ingest/source-connectors/overview).

## Destinations

*   [Astra DB](/platform/destinations/astradb)
*   [Azure AI Search](/platform/destinations/azure-ai-search)
*   [Couchbase](/platform/destinations/couchbase)
*   [Databricks Volumes](/platform/destinations/databricks-volumes)
*   [Delta Table](/platform/destinations/delta-table)
*   [Elasticsearch](/platform/destinations/elasticsearch)
*   [Google Cloud Storage](/platform/destinations/google-cloud)
*   [Milvus](/platform/destinations/milvus)
*   [MongoDB](/platform/destinations/mongodb)
*   [OneDrive](/platform/destinations/onedrive)
*   [Pinecone](/platform/destinations/pinecone)
*   [S3](/platform/destinations/s3)
*   [Weaviate](/platform/destinations/weaviate)

If your destination is not listed here, you might still be able to connect Unstructured to it through scripts or code by using the
[Unstructured Ingest CLI](/ingestion/overview#unstructured-ingest-cli) or the
[Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library).
[Learn more](/api-reference/ingest/destination-connector/overview).


# Astra DB



Send processed data from Unstructured to Astra DB.

You'll need:

The Astra DB connector prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/kupQrnCQjPw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An Astra account. [Create or sign in to an Astra account](https://astra.datastax.com/).
*   A database in the Astra account. [Create a database in an account](https://docs.datastax.com/en/astra-db-classic/databases/manage-create.html).
*   An application token for the database. [Create a database application token](https://docs.datastax.com/en/astra-db-serverless/administration/manage-application-tokens.html).
*   A namespace in the database. [Create a namespace in a database](https://docs.datastax.com/en/astra-db-serverless/databases/manage-namespaces.html#create-namespace).
*   A collection in the namespace. [Create a collection in a namespace](https://docs.datastax.com/en/astra-db-serverless/databases/manage-collections.html#create-collection).

To create the destination connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Destinations**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Astra DB**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Token** (*required*): The application token for the database.
*   **API Endpoint** (*required*): The database's associated API endpoint.
*   **Collection Name** (*required*): The name of the collection in the namespace.
*   **Embedding Dimension** (*required*): The number of dimensions in the collection.


# Azure AI Search



Send processed data from Unstructured to Azure AI Search.

You'll need:

The Azure AI Search prerequisites:

The following video shows how to fulfill the minimum set of Azure AI Search prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/6ZjU5OupWE8" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

Here are some more details about these prerequisites:

*   The endpoint and API key for Azure AI Search. [Create an endpoint and API key](https://learn.microsoft.com/azure/search/search-create-service-portal).
*   The name of the index in Azure AI Search. [Create an index](https://learn.microsoft.com/rest/api/searchservice/create-index).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/WY8h8Gtyo7o" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    The Azure AI Search index that you use must have an index schema that is compatible with the schema of the documents
    that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following index schema example for your own needs:

    ```json
    {
      "@odata.context": "https://ingest-test-azure-ai-search.search.windows.net/$metadata#indexes/$entity",
      "@odata.etag": "\"0x8DCED5D96393CA9\"",
      "name": "<my-index-name>",
      "defaultScoringProfile": null,
      "fields": [
        {
          "name": "id",
          "type": "Edm.String",
          "searchable": true,
          "filterable": true,
          "retrievable": true,
          "stored": true,
          "sortable": true,
          "facetable": true,
          "key": true,
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "analyzer": null,
          "normalizer": null,
          "dimensions": null,
          "vectorSearchProfile": null,
          "vectorEncoding": null,
          "synonymMaps": []
        },
        {
          "name": "element_id",
          "type": "Edm.String",
          "searchable": true,
          "filterable": true,
          "retrievable": true,
          "stored": true,
          "sortable": true,
          "facetable": true,
          "key": false,
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "analyzer": null,
          "normalizer": null,
          "dimensions": null,
          "vectorSearchProfile": null,
          "vectorEncoding": null,
          "synonymMaps": []
        },
        {
          "name": "text",
          "type": "Edm.String",
          "searchable": true,
          "filterable": true,
          "retrievable": true,
          "stored": true,
          "sortable": true,
          "facetable": true,
          "key": false,
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "analyzer": null,
          "normalizer": null,
          "dimensions": null,
          "vectorSearchProfile": null,
          "vectorEncoding": null,
          "synonymMaps": []
        },
        {
          "name": "embeddings",
          "type": "Collection(Edm.Single)",
          "searchable": true,
          "filterable": false,
          "retrievable": true,
          "stored": true,
          "sortable": false,
          "facetable": false,
          "key": false,
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "analyzer": null,
          "normalizer": null,
          "dimensions": 3072,
          "vectorSearchProfile": "embeddings-config-profile",
          "vectorEncoding": null,
          "synonymMaps": []
        },
        {
          "name": "type",
          "type": "Edm.String",
          "searchable": true,
          "filterable": true,
          "retrievable": true,
          "stored": true,
          "sortable": true,
          "facetable": true,
          "key": false,
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "analyzer": null,
          "normalizer": null,
          "dimensions": null,
          "vectorSearchProfile": null,
          "vectorEncoding": null,
          "synonymMaps": []
        },
        {
          "name": "metadata",
          "type": "Edm.ComplexType",
          "fields": [
            {
              "name": "category_depth",
              "type": "Edm.Int32",
              "searchable": false,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "parent_id",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "attached_to_filename",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "filetype",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "last_modified",
              "type": "Edm.DateTimeOffset",
              "searchable": false,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "is_continuation",
              "type": "Edm.Boolean",
              "searchable": false,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "file_directory",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "filename",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "data_source",
              "type": "Edm.ComplexType",
              "fields": [
                {
                  "name": "url",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "version",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "date_created",
                  "type": "Edm.DateTimeOffset",
                  "searchable": false,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "date_modified",
                  "type": "Edm.DateTimeOffset",
                  "searchable": false,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "date_processed",
                  "type": "Edm.DateTimeOffset",
                  "searchable": false,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "permissions_data",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "record_locator",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                }
              ]
            },
            {
              "name": "coordinates",
              "type": "Edm.ComplexType",
              "fields": [
                {
                  "name": "system",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "layout_width",
                  "type": "Edm.Double",
                  "searchable": false,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "layout_height",
                  "type": "Edm.Double",
                  "searchable": false,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                },
                {
                  "name": "points",
                  "type": "Edm.String",
                  "searchable": true,
                  "filterable": true,
                  "retrievable": true,
                  "stored": true,
                  "sortable": true,
                  "facetable": true,
                  "key": false,
                  "indexAnalyzer": null,
                  "searchAnalyzer": null,
                  "analyzer": null,
                  "normalizer": null,
                  "dimensions": null,
                  "vectorSearchProfile": null,
                  "vectorEncoding": null,
                  "synonymMaps": []
                }
              ]
            },
            {
              "name": "languages",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "page_number",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "orig_elements",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "links",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "page_name",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "url",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "link_urls",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "link_texts",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "sent_from",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "sent_to",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "subject",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "section",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "header_footer_type",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "emphasized_text_contents",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "emphasized_text_tags",
              "type": "Collection(Edm.String)",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": false,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "text_as_html",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "regex_metadata",
              "type": "Edm.String",
              "searchable": true,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            },
            {
              "name": "detection_class_prob",
              "type": "Edm.Double",
              "searchable": false,
              "filterable": true,
              "retrievable": true,
              "stored": true,
              "sortable": true,
              "facetable": true,
              "key": false,
              "indexAnalyzer": null,
              "searchAnalyzer": null,
              "analyzer": null,
              "normalizer": null,
              "dimensions": null,
              "vectorSearchProfile": null,
              "vectorEncoding": null,
              "synonymMaps": []
            }
          ]
        }
      ],
      "scoringProfiles": [],
      "corsOptions": null,
      "suggesters": [],
      "analyzers": [],
      "normalizers": [],
      "tokenizers": [],
      "tokenFilters": [],
      "charFilters": [],
      "encryptionKey": null,
      "similarity": {
        "@odata.type": "#Microsoft.Azure.Search.BM25Similarity",
        "k1": null,
        "b": null
      },
      "semantic": null,
      "vectorSearch": {
        "algorithms": [
          {
            "name": "embeddings-config",
            "kind": "hnsw",
            "hnswParameters": {
              "metric": "cosine",
              "m": 4,
              "efConstruction": 400,
              "efSearch": 500
            },
            "exhaustiveKnnParameters": null
          }
        ],
        "profiles": [
          {
            "name": "embeddings-config-profile",
            "algorithm": "embeddings-config",
            "vectorizer": null,
            "compression": null
          }
        ],
        "vectorizers": [],
        "compressions": []
      }
    }
    ```

    See also:

    *   [Search indexes in Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-an-index)
    *   [Schema of a search index](https://learn.microsoft.com/azure/search/search-what-is-an-index#schema-of-a-search-index)
    *   [Example index schema](https://learn.microsoft.com/rest/api/searchservice/create-index#examples)
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

To create the destination connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Destinations**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Azure AI Search**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Endpoint** (*required*): The endpoint URL for Azure AI Search.
*   **API Key** (*required*): The API key for Azure AI Search.
*   **Index Name** (*required*): The name of the index for Azure AI Search.


# Couchbase



Send processed data from Unstructured to Couchbase.

You'll need:

The Couchbase database prerequisites.

*   For the [Unstructured Platform](/platform/overview), only Couchbase Capella clusters are supported.
*   For [Unstructured Ingest](/ingestion/overview), Couchbase Capella clusters and local Couchbase server deployments are supported.

<iframe width="560" height="315" src="https://www.youtube.com/embed/9-RIBmIdi70" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

For Couchbase Capella, you will need:

*   A Couchbase Capella account. [Create an account](https://docs.couchbase.com/cloud/get-started/create-account.html).
*   A Couchbase Capella cluster. [Create a cluster](https://docs.couchbase.com/cloud/clusters/create-database.html).
*   A [bucket, scope, and collection](https://docs.couchbase.com/cloud/clusters/data-service/about-buckets-scopes-collections.html).
*   A cluster connection string. [Learn how](https://docs.couchbase.com/cloud/get-started/connect.html).
*   Incoming IP address allowance. [Learn how](https://docs.couchbase.com/cloud/clusters/allow-ip-address.html).
*   A username and password. [Learn how](https://docs.couchbase.com/cloud/clusters/manage-database-users.html).

For a local Couchbase server, you will need:

*   Installation of a local Couchbase server. [Learn how](https://docs.couchbase.com/server/current/getting-started/start-here.html).
*   Connection details to the local Couchbase server. [Learn how](https://docs.couchbase.com/server/current/guides/connect.html).

To learn more about how to set up a Couchbase cluster and play with data, refer to this [tutorial](https://developer.couchbase.com/tutorial-quickstart-flask-python).

To create the destination connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Destinations**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Couchbase**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Connection string** (*required*): The connection string for the Couchbase cluster.
*   **Bucket** (*required*): The name of the bucket to connect to on the cluster.
*   **Scope** (*required*): The name of the scope in the bucket.
*   **Collection** (*required*): The name of the collection in the scope.
*   **Batch Size**: The maximum number of documents per transmitted batch.
*   **Username** (*required*): The cluster access name (username) that has access to the cluster.
*   **Password** (*required*): The password for the corresponding username.


# Databricks Volumes



Send processed data from Unstructured to Databricks Volumes.

You'll need:

The Databricks Volumes prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/rNZpwa1-g7M" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The preceding video shows how to use Databricks personal access tokens (PATs), which are supported only for [Unstructured Ingest](/ingestion/overview).

To learn how to use Databricks-managed service principals, which are supported by both the [Unstructured Platform](/platform/overview) and Unstructured Ingest,
see the additional videos later on this page.

*   The Databricks workspace URL. Get the workspace URL for
    [AWS](https://docs.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids),
    [Azure](https://learn.microsoft.com/azure/databricks/workspace/workspace-details#workspace-instance-names-urls-and-ids),
    or [GCP](https://docs.gcp.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids).

    Examples:

    *   AWS: `https://<workspace-id>.cloud.databricks.com`
    *   Azure: `https://adb-<workspace-id>.<random-number>.azuredatabricks.net`
    *   GCP: `https://<workspace-id>.<random-number>.gcp.databricks.com`

*   The Databricks authentication details. For more information, see the documentation for
    [AWS](https://docs.databricks.com/dev-tools/auth/index.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/),
    or [GCP](https://docs.gcp.databricks.com/dev-tools/auth/index.html).

    The following videos show how to create a Databricks-managed service principal and then grant it access to a Databricks volume:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/wBmqv5DaA1E" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    <iframe width="560" height="315" src="https://www.youtube.com/embed/DykQRxgh2aQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    For the [Unstructured Platform](/platform/overview), only the following Databricks authentication type is supported:

    *   For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.
        Note that for Azure, only Databricks-managed service principals are supported. Microsoft Entra ID-managed service principals are not supported.

    For [Unstructured Ingest](/ingestion/overview), the following Databricks authentication types are supported:

    *   For Databricks personal access token authentication (AWS, Azure, and GCP): The personal access token's value.
    *   For username and password (basic) authentication (AWS only): The user's name and password values.
    *   For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.
    *   For OAuth user-to-machine (U2M) authentication (AWS, Azure, and GCP): No additional values.
    *   For Azure managed identities (MSI) authentication (Azure only): The client ID value for the corresponding managed identity.
    *   For Microsoft Entra ID service principal authentication (Azure only): The tenant ID, client ID, and client secret values for the corresponding service principal.
    *   For Azure CLI authentication (Azure only): No additional values.
    *   For Microsoft Entra ID user authentication (Azure only): The Entra ID token for the corresponding Entra ID user.
    *   For Google Cloud Platform credentials authentication (GCP only): The local path to the corresponding Google Cloud service account's credentials file.
    *   For Google Cloud Platform ID authentication (GCP only): The Google Cloud service account's email address.

*   The Databricks catalog name for the volume. Get the catalog name for [AWS](https://docs.databricks.com/catalogs/manage-catalog.html), [Azure](https://learn.microsoft.com/azure/databricks/catalogs/manage-catalog), or [GCP](https://docs.gcp.databricks.com/catalogs/manage-catalog.html).

*   The Databricks schema name for the volume. Get the schema name for [AWS](https://docs.databricks.com/schemas/manage-schema.html), [Azure](https://learn.microsoft.com/azure/databricks/schemas/manage-schema), or [GCP](https://docs.gcp.databricks.com/schemas/manage-schema.html).

*   The Databricks volume name, and optionally any path in that volume that you want to access directly. Get the volume information for [AWS](https://docs.databricks.com/files/volumes.html), [Azure](https://learn.microsoft.com/azure/databricks/files/volumes), or [GCP](https://docs.gcp.databricks.com/files/volumes.html).

*   Make sure that the target user or service principal has access to the target volume. To learn more, see the documentation for
    [AWS](https://docs.databricks.com/volumes/utility-commands.html#change-permissions-on-a-volume),
    [Azure](https://learn.microsoft.com/azure/databricks/volumes/utility-commands#change-permissions-on-a-volume),
    or [GCP](https://docs.gcp.databricks.com/volumes/utility-commands.html#change-permissions-on-a-volume).

To create the destination connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Destinations**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Databricks Volumes**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Host** (*required*): The Databricks workspace host URL.
*   **Catalog** (*required*): The name of the catalog to use.
*   **Schema** : The name of the associated schema. If not specified, **default** is used.
*   **Volume** (*required*): The name of the associated volume.
*   **Volume Path** : Any optional path to access within the volume.
*   **Client ID** (*required*): The application ID value for the Databricks-managed service principal that has access to the volume.
*   **Client Secret** (*required*): The associated OAuth secret value for the Databricks-managed service principal that has access to the volume.

To learn how to create a Databricks-managed service principal, get its application ID, and generate an associated OAuth secret,
see the documentation for
[AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
[Azure](https://learn.microsoft.com/databricks/dev-tools/auth/oauth-m2m),
or [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html).

For Azure, only Databricks-managed service principals are supported. Microsoft Entra ID-managed service principals are not supported.

To learn how to grant a Databricks-managed service principal access to a volume, see the documentation for
[AWS](https://docs.databricks.com/volumes/utility-commands.html#change-permissions-on-a-volume),
[Azure](https://learn.microsoft.com/azure/databricks/volumes/utility-commands#change-permissions-on-a-volume),
or [GCP](https://docs.gcp.databricks.com/volumes/utility-commands.html#change-permissions-on-a-volume).


# Delta Table



Send processed data from Unstructured to a Delta Table, stored in Amazon S3.

You'll need:

The Delta Table prerequisites for Amazon S3:

The following video shows how to fulfill the minimum set of S3 prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/_W4565dcUGI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The preceding video does not show how to create an AWS account or an S3 bucket.

For more information about prerequisites, see the following:

*   An AWS account. [Create an AWS account](https://aws.amazon.com/free).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/lIdh92JmWtg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An S3 bucket. [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
    Additional approaches are in the following video and in the how-to sections at the end of this page.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/e6w9LwZJFIA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/y4SfQoJpipo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at
    minimum the permission of `s3:PutObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

*   For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account.
    [Create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/MoFTaGJE65Q" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   If the target files are in the root of the bucket, the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
    If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

*   If the target files are in a folder, make sure the authenticated AWS IAM user has
    authenticated access to the folder as well. [Enable authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2.  Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3.  Browse to the existing bucket and open it.

4.  Click the **Permissions** tab.

5.  In the **Bucket policy** area, click **Edit**.

6.  In the **Policy** text area, copy the following JSON-formatted policy.
    To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this policy, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-bucket-name>` in two places with the name of your bucket.

    ```json
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::<my-account-id>:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::<my-bucket-name>",
                    "arn:aws:s3:::<my-bucket-name>/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    ```

7.  Click **Save changes**.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
    the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    ```yaml
    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

    Parameters:
      BucketName:
        Type: String
        Description: 'Name of the S3 bucket to create'

    Resources:
      MyS3Bucket:
        Type: 'AWS::S3::Bucket'
        Properties:
          BucketName: !Ref BucketName
          PublicAccessBlockConfiguration:
            BlockPublicAcls: true
            BlockPublicPolicy: false
            IgnorePublicAcls: true
            RestrictPublicBuckets: true

      BucketPolicy:
        Type: 'AWS::S3::BucketPolicy'
        Properties:
          Bucket: !Ref MyS3Bucket
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Sid: AllowAllAuthenticatedUsersInAccount
                Effect: Allow
                Principal:
                  AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
                Action:
                  - 's3:GetObject'
                  - 's3:PutObject'
                  - 's3:ListBucket'
                  - 's3:DeleteObject'
                Resource:
                  - !Sub 'arn:aws:s3:::${BucketName}'
                  - !Sub 'arn:aws:s3:::${BucketName}/*'

    Outputs:
      BucketName:
        Description: 'Name of the created S3 bucket'
        Value: !Ref MyS3Bucket
    ```

2.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3.  Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4.  Click **Create stack > With new resources (standard)**.

5.  On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6.  Click **Choose file**, and browse to and select the YAML file from your local machine.

7.  Click **Next**.

8.  Enter a unique **Stack name** and **BucketName**.

9.  Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2.  [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3.  Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
    To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this script, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-unique-bucket-name>` with the name of your bucket.
    *   Replace `<us-east-1>` with your AWS Region.

    ```bash
    #!/bin/bash

    # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
    ACCOUNT_ID="<my-account-id>"
    BUCKET_NAME="<my-unique-bucket-name>"
    REGION="<us-east-1>"

    # Temporary filename for the bucket policy.
    # Do not change this variable.
    POLICY_FILE="bucket_policy.json"

    # Create the bucket.
    aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

    # Wait for the bucket to exist.
    echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
    aws s3api wait bucket-exists --bucket $BUCKET_NAME

    # Check if the wait command was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket '$BUCKET_NAME' has been fully created."
    else
        echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
        exit 1
    fi

    # Remove the "block public policy" bucket access setting.
    aws s3api put-public-access-block \
        --bucket $BUCKET_NAME \
        --public-access-block-configuration \
        '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

    # Check if the operation was successful.
    if [ $? -eq 0 ]; then
        echo "The block public policy access setting was removed from '$BUCKET_NAME'."
    else
        echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
        exit 1
    fi

    # Create the bucket policy.
    cat << EOF > $POLICY_FILE
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::$BUCKET_NAME",
                    "arn:aws:s3:::$BUCKET_NAME/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    EOF

    # Apply the bucket policy.
    aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

    # Check if the policy application was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket policy was applied to '$BUCKET_NAME'."
    else
        echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
        exit 1
    fi

    # Verify the applied policy.
    echo "Verifying the applied policy:"
    aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

    # Remove the temporary bucket policy file.
    rm $POLICY_FILE
    ```

4.  Run the script, for example:

    ```bash
    sh create-s3-bucket.sh
    ```

5.  After the bucket is created, you can delete the script file, if you want.

To create the destination connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Destinations**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Delta Table**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **AWS Region** (*required*): The AWS Region identifier (for example, `us-east-1`) for the Amazon S3 bucket you want to store the Delta Table in.
*   **Bucket URI** (*required*): The URI of the Amazon S3 bucket you want to store the Delta Table in. This typically takes the format `s3://my-bucket/my-folder`.
*   **AWS Access Key ID** (*required*): The AWS access key ID for the AWS IAM principal (such as an IAM user) that has the appropriate access to the S3 bucket.
*   **AWS Secret Access Key** (*required*): The AWS secret access key for the corresponding AWS access key ID.


# Elasticsearch



Send processed data from Unstructured to Elasticsearch.

You'll need:

The Elasticsearch prerequisites.

*   For the [Unstructured Platform](/platform/overview), only Elastic Cloud instances are supported.

*   For [Unstructured Ingest](/ingestion/overview), Elastic Cloud instances and self-manged Elasticsearch instances are supported.

*   For Elastic Cloud, you will need an [Elastic Cloud service instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#hosted-elasticsearch-service).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/HEYIFC7uOus" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For self-managed Elasticsearch, you will need a [self-managed Elasticsearch instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#elasticsearch-deployment-options).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/9znbAkNG1Ok" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   You will need the name of the index on the instance. See [Create index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html) and [Get index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html).

    The Elasticsearch index that you use must have a schema that is compatible with the schema of the documents
    that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all
    circumstances. This is because these schemas will vary based on your source files' types; how you
    want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

    You can adapt the following index schema example for your own needs:

    ```json
    {
        "settings": {
            "index": {
                "knn": true,
                "knn.algo_param.ef_search": 100
            }
        },
        "mappings": {
            "properties": {
                "element_id": {
                    "type": "keyword"
                },
                "text": {
                    "type": "text"
                },
                "embeddings": {
                    "type": "dense_vector",
                    "dims": 384,
                    "index": true,
                    "similarity": "cosine"
                },
                "metadata": {
                    "type": "object",
                    "properties": {
                        "parent_id": {
                            "type": "text"
                        },
                        "page_number": {
                            "type": "integer"
                        },
                        "is_continuation": {
                            "type": "boolean"
                        },
                        "orig_elements": {
                            "type": "text"
                        }
                    }
                }
            }
        }
    }
    ```

    See also:

    *   [An Introduction to Elasticsearch Mapping](https://www.elastic.co/blog/found-elasticsearch-mapping-introduction)
    *   [Explicit mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html)
    *   [Dynamic field mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html)
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

    <iframe width="560" height="315" src="https://www.youtube.com/embed/0f0tEcrvP8g" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For Elastic Cloud, you will need the Elastic Cloud service instance's API key. If you are using Unstructured Ingest, you will also need the instance's Cloud ID. To get these, see your Elasticsearch Service web console.

*   For self-managed Elasticsearch, you will need:

    *   The self-managed instance's hostname and port number. See [Networking](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html).
    *   If you're using basic authentication to the self-managed instance, the user's name and password.
    *   If you're using token-based authentication to the self-managed instance, the bearer token or API key for the instance. See [Token-based authentication services](https://www.elastic.co/guide/en/elasticsearch/reference/current/token-authentication-services.html) and [Create API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html).
    *   If you're using certificate authentication to the self-managed instance, the path to the Certificate Authority (CA) file on the instance, and the certificate fingerprint. See [SSL certificate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-ssl.html) and [Where can I see my Certificate Fingerprint?](https://discuss.elastic.co/t/where-can-i-see-my-certificate-fingerprint/319335/3).

To create the destination connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Destinations**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Elasticsearch**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Host** (*required*): The endpoint URL for the target cluster.
*   **Index Name** (*required*): The name of the target index in the cluster.
*   **API Key** (*required*): The Elastic Cloud API key for the target cluster.


# Google Cloud Storage



Send processed data from Unstructured to Google Cloud Storage.

You'll need:

The Google Cloud Storage prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/HYaALQ0F-L4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A Google Cloud service account. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#console).

*   A service account key for the service account. See [Create a service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating) in
    [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete).

    To ensure maximum compatibility across Unstructured service offerings, you should give the service account key information to Unstructured as
    a single-line string that contains the contents of the downloaded service account key file (and not the service account key file itself).
    To print this single-line string without line breaks, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
    In this command, replace `<path-to-downloaded-key-file>` with the path to the service account key file that you downloaded by following the preceding instructions.

    *   For macOS or Linux:
        ```text
        tr -d '\n' < <path-to-downloaded-key-file>
        ```
    *   For Windows:
        ```text
        (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
        ```

*   The URI for a Google Cloud Storage bucket. This URI consists of the target bucket name, plus any target folder within the bucket, expressed as `gs://<bucket-name>[/folder-name]`. [Create a bucket](https://cloud.google.com/storage/docs/creating-buckets#console).

    This bucket must have, at minimum, one of the following roles applied to the target Google Cloud service account:

    *   `Storage Object Viewer` for bucket read access.
    *   `Storage Object Creator` for bucket write access.
    *   The `Storage Object Admin` role provides read and write access, plus access to additional bucket operations.

    To apply one of these roles to a service account for a bucket, see [Add a principal to a bucket-level policy](https://cloud.google.com/storage/docs/access-control/using-iam-permissions#bucket-add)
    in [Set and manage IAM policies on buckets](https://cloud.google.com/storage/docs/access-control/using-iam-permissions).

To create the destination connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Destinations**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Google GCS**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Bucket URI** (*required*): The URI for the Google Cloud Storage bucket and any target folder path within the bucket. This URI takes the format `gs://<bucket-name>[/folder-name]`.
*   **Service Account Key** (*required*): The contents of a service account key file, expressed as a single string without line breaks, for a Google Cloud service account that has the required access permissions to the bucket.
*   **Recursive**: Check this box to ingest data recursively from any subfolders, starting from the path specified by **Bucket URI**.


# Milvus



Send processed data from Unstructured to Milvus.

The following video shows how to fulfill the minimum set of Milvus prerequisites, demonstrating Milvus on IBM watsonx.data.

<iframe width="560" height="315" src="https://www.youtube.com/embed/tl9eV-_nAHI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The Milvus prerequisites:

*   A [Milvus instance](https://milvus.io/docs/install-overview.md).
*   The [URI](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md) of the instance.
*   The [username and password, or token](https://milvus.io/docs/authenticate.md) to access the instance.
*   The name of the [database](https://milvus.io/docs/manage_databases.md) in the instance.
*   The name of the [collection](https://milvus.io/docs/manage-collections.md) in the database.

<iframe width="560" height="315" src="https://www.youtube.com/embed/tl9eV-_nAHI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

To create the destination connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Destinations**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Milvus**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Username** (*required*): The username to access the Milvus instance.
*   **Password** (*required*): The password corresponding to the username to access the instance.
*   **Uri** (*required*): The URI of the instance, for example: `https://12345.serverless.gcp-us-west1.cloud.zilliz.com`.
*   **Database Name** (*required*): The name of the database in the instance.
*   **Collection Name** (*required*): The name of the collection in the database.


# MongoDB



Send processed data from Unstructured to MongoDB.

You'll need:

The MongoDB prerequisites for a MongoDB Atlas deployment:

<iframe width="560" height="315" src="https://www.youtube.com/embed/g6qDfbg808M" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A MongoDB Atlas account. [Create an account](https://www.mongodb.com/cloud/atlas/register).
*   A MongoDB Atlas cluster. [Create a cluster](https://www.mongodb.com/docs/atlas/tutorial/deploy-free-tier-cluster).
*   The cluster must be reachable from your application environment. [Learn how](https://www.mongodb.com/docs/atlas/setup-cluster-security/#network-and-firewall-requirements).
*   The cluster must be configured to use your IP address. [Learn how](https://www.mongodb.com/docs/atlas/setup-cluster-security/#ip-access-list).
*   The cluster must have at least one database. [Create a database](https://www.mongodb.com/docs/compass/current/databases/#create-a-database).
*   The database must have at least one user, and that user must have sufficient access to the database. [Create a database user](https://www.mongodb.com/docs/atlas/security-add-mongodb-users/#add-database-users). [Give the user database access](https://www.mongodb.com/docs/manual/core/authorization/).
*   The database must have at least one collection. [Create a collection](https://www.mongodb.com/docs/compass/current/collections/#create-a-collection).
*   The connection string for the cluster. For MongoDB Atlas, this connection string must include the protocol, username, password, host, and cluster name. For example:

    ```text
    mongodb+srv://<db_user>:<db_password>@<host>/?retryWrites=true&w=majority&appName=<cluster>
    ```

    To get the connection string in MongoDB Atlas, do the following:

    1.  Log in to your MongoDB Atlas console.
    2.  In the sidebar, under **Databases**, click **Clusters**.
    3.  Click on the cluster you want to connect to.
    4.  Click **Connect**, or click the **Cmd Line Tools** tab and then click **Connect Instructions**.
    5.  Click **Drivers**.
    6.  Under **Add your connection string into your application code**, copy the connection string.
        You can then close the **Connect** dialog in MongoDB Atlas.

        Before you use this connection string, be sure to fill in any placeholders in the string, such as your MongoDB Atlas database user's password value.

    [Learn more](https://www.mongodb.com/resources/products/fundamentals/mongodb-connection-string).

To create the destination connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Destinations**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **MongoDB**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Database** (*required*): The name of the database on the instance.
*   **Collection** (*required*): The name of the collection within the database.
*   **Connection String** (*required*): The MongoDB instance connection string.


# OneDrive



Send processed data from Unstructured to OneDrive.

You'll need:

The OneDrive prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/7uSu0LsbcEQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A OneDrive account.
*   The User Principal Name (UPN) for the OneDrive account. This is typically the OneDrive account user's email address.
*   The path to the target OneDrive folder, starting from the OneDrive account's root folder, for example `my-folder/my-subfolder`.
*   The client ID, tenant ID, and client secret for the Microsoft Entra ID app registration that has access to the target OneDrive account and
    also has the correct set of Microsoft Graph authentication scopes. These scopes include:

    *   `Files.ReadWrite.All` (if both reading and writing are needed)
    *   `Sites.ReadWrite.All` (if both reading and writing are needed)
    *   `User.Read.All`
    *   `Directory.Read.All`

See also:

*   [Registering your app for Microsoft Graph](https://learn.microsoft.com/onedrive/developer/rest-api/getting-started/app-registration)
*   [OneDrive authentication and sign-in](https://learn.microsoft.com/onedrive/developer/rest-api/getting-started/msa-oauth)
*   [OneDrive API documentation](https://docs.microsoft.com/onedrive/developer/rest-api/)

To create the destination connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Destinations**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **OneDrive**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Client ID** (*required*): The application (client) ID of the Microsoft Entra ID app registration that has access to the OneDrive account.
*   **Tenant ID** (*required*): The directory (tenant) ID of the Entra ID app registration.
*   **Authority URL** (*required*): The authentication token provider URL for the Entra ID app registration. The default is `https://login.microsoftonline.com`.
*   **Principal Name** (*required*): The User Principal Name (UPN) for the OneDrive user account in Entra ID. This is typically the user's email address.
*   **Path**: The path to the target folder in the OneDrive account, starting with the account's root folder, for example `my-folder/my-subfolder`.
*   **Recursive**: Check this box to recursively access files from subfolders within the specified OneDrive path.
*   **Client Credential** (*required*): The client secret for the Entra ID app registration.


# Overview

Destination connectors in the Unstructured Platform are designed to specify the endpoint for data processed within the platform. These connectors ensure that the transformed and analyzed data is securely and efficiently transferred to a storage system for future use, often to a vector database for tasks that involve high-speed retrieval and advanced data analytics operations.

![Destinations in the sidebar](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Destinations-Sidebar.png)

To see your existing destination connectors, on the sidebar, click **Connectors**, and then click **Destinations**.

To create a destination connector:

1.  In the sidebar, click **Connectors**.
2.  Click **Destinations**.
3.  Click **Add new**.
4.  For **Name**, enter some unique name for this connector.
5.  In the **Provider** area, click the destination location type that matches yours.
6.  Click **Continue**.
7.  Fill in the fields according to your connector type. To learn how, click your connector type in the following list:

    *   [Astra DB](/platform/destinations/astradb)
    *   [Azure AI Search](/platform/destinations/azure-ai-search)
    *   [Couchbase](/platform/destinations/couchbase)
    *   [Delta Table](/platform/destinations/delta-table)
    *   [Elasticsearch](/platform/destinations/elasticsearch)
    *   [Google Cloud Storage](/platform/destinations/google-cloud)
    *   [Milvus](/platform/destinations/milvus)
    *   [MongoDB](/platform/destinations/mongodb)
    *   [OneDrive](/platform/destinations/onedrive)
    *   [Pinecone](/platform/destinations/pinecone)
    *   [S3](/platform/destinations/s3)
    *   [Weaviate](/platform/destinations/weaviate)
8.  If a **Continue** button appears, click it, and fill in any additional settings fields.
9.  Click **Save and Test**.


# Pinecone



Send processed data from Unstructured to Pinecone.

The following video shows how to fulfill the minimum set of Pinecone prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/Li0yhaeguYQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

Here are some more details about the prerequisites:

The Pinecone prerequisites:

*   A Pinecone account. [Get an account](https://app.pinecone.io/).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/uPoa7PLcuZ4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A Pinecone API key. [Get an API key](https://docs.pinecone.io/guides/get-started/authentication#find-your-pinecone-api-key).

*   A Pinecone serverless index. [Create a serverless index](https://docs.pinecone.io/guides/indexes/create-an-index).

To create the destination connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Destinations**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Pinecone**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Index Name** (*required*): The name of the index in the Pinecone database.
*   **Environment** (*required*): The name of the Pinecone environment for the index instance.
*   **Batch Size** (*required*): The number of records to use in a single batch.
*   **API Key** (*required*): The Pinecone API key.


# S3



Send processed data from Unstructured to Amazon S3.

You'll need:

The Amazon S3 prerequisites:

The following video shows how to fulfill the minimum set of S3 prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/hyDHfhVVAhs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but
not recommended); or generate an AWS STS session token for temporary access, if required by your organization's security
requirements. For more information about prerequisites, see the following:

*   An AWS account. [Create an AWS account](https://aws.amazon.com/free).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/lIdh92JmWtg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An S3 bucket. [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
    Additional approaches are in the following video and in the how-to sections at the end of this page.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/e6w9LwZJFIA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   Anonymous (supported but not recommended) or authenticated access to the bucket.

    *   [Enable anonymous bucket access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user).

        ![Enable anonymous bucket access](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/connectors/s3-anon-access.gif)

    *   [Enable authenticated bucket access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/walkthrough1.html).

        ![Enable authenticated bucket access](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/connectors/s3-auth-access.gif)

*   For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/y4SfQoJpipo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at
    minimum the permission of `s3:PutObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

*   For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account.
    [Create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/MoFTaGJE65Q" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For authenticated access in untrusted environments or enhanced security scenarios, an AWS STS session token for temporary access, in addition to an AWS access key and secret access key. [Create a session token](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_getsessiontoken).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/4_csSXc_GNU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   If the target files are in the root of the bucket, the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
    If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

*   If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has
    authenticated access to the folder as well. [Enable authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

To create the destination connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Destinations**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Amazon S3**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Bucket Name** (*required*): The path to the bucket or folder, formatted as `s3://my-bucket/` (if the files are in the bucket's root) or `s3://my-bucket/my-folder/`.
*   **AWS Access Key**: The AWS access key ID for the authenticated AWS IAM user, if the bucket has authenticated access enabled.
*   **AWS Secret Key**: The AWS secret access key corresponding to the preceding AWS access key ID, if the bucket has authenticated access enabled
*   **Token**: If required, the AWS STS session token for temporary access.
*   **Endpoint URL**: A custom URL, if connecting to a non-AWS S3 bucket.
*   **Recursive**: Check this box to access subfolders within the bucket.
*   **Connect Anonymously** (*source connectors only*): Check this box if reading from a bucket with anonymous access enabled.

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2.  Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3.  Browse to the existing bucket and open it.

4.  Click the **Permissions** tab.

5.  In the **Bucket policy** area, click **Edit**.

6.  In the **Policy** text area, copy the following JSON-formatted policy.
    To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this policy, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-bucket-name>` in two places with the name of your bucket.

    ```json
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::<my-account-id>:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::<my-bucket-name>",
                    "arn:aws:s3:::<my-bucket-name>/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    ```

7.  Click **Save changes**.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
    the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    ```yaml
    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

    Parameters:
      BucketName:
        Type: String
        Description: 'Name of the S3 bucket to create'

    Resources:
      MyS3Bucket:
        Type: 'AWS::S3::Bucket'
        Properties:
          BucketName: !Ref BucketName
          PublicAccessBlockConfiguration:
            BlockPublicAcls: true
            BlockPublicPolicy: false
            IgnorePublicAcls: true
            RestrictPublicBuckets: true

      BucketPolicy:
        Type: 'AWS::S3::BucketPolicy'
        Properties:
          Bucket: !Ref MyS3Bucket
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Sid: AllowAllAuthenticatedUsersInAccount
                Effect: Allow
                Principal:
                  AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
                Action:
                  - 's3:GetObject'
                  - 's3:PutObject'
                  - 's3:ListBucket'
                  - 's3:DeleteObject'
                Resource:
                  - !Sub 'arn:aws:s3:::${BucketName}'
                  - !Sub 'arn:aws:s3:::${BucketName}/*'

    Outputs:
      BucketName:
        Description: 'Name of the created S3 bucket'
        Value: !Ref MyS3Bucket
    ```

2.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3.  Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4.  Click **Create stack > With new resources (standard)**.

5.  On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6.  Click **Choose file**, and browse to and select the YAML file from your local machine.

7.  Click **Next**.

8.  Enter a unique **Stack name** and **BucketName**.

9.  Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2.  [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3.  Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
    To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this script, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-unique-bucket-name>` with the name of your bucket.
    *   Replace `<us-east-1>` with your AWS Region.

    ```bash
    #!/bin/bash

    # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
    ACCOUNT_ID="<my-account-id>"
    BUCKET_NAME="<my-unique-bucket-name>"
    REGION="<us-east-1>"

    # Temporary filename for the bucket policy.
    # Do not change this variable.
    POLICY_FILE="bucket_policy.json"

    # Create the bucket.
    aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

    # Wait for the bucket to exist.
    echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
    aws s3api wait bucket-exists --bucket $BUCKET_NAME

    # Check if the wait command was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket '$BUCKET_NAME' has been fully created."
    else
        echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
        exit 1
    fi

    # Remove the "block public policy" bucket access setting.
    aws s3api put-public-access-block \
        --bucket $BUCKET_NAME \
        --public-access-block-configuration \
        '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

    # Check if the operation was successful.
    if [ $? -eq 0 ]; then
        echo "The block public policy access setting was removed from '$BUCKET_NAME'."
    else
        echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
        exit 1
    fi

    # Create the bucket policy.
    cat << EOF > $POLICY_FILE
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::$BUCKET_NAME",
                    "arn:aws:s3:::$BUCKET_NAME/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    EOF

    # Apply the bucket policy.
    aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

    # Check if the policy application was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket policy was applied to '$BUCKET_NAME'."
    else
        echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
        exit 1
    fi

    # Verify the applied policy.
    echo "Verifying the applied policy:"
    aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

    # Remove the temporary bucket policy file.
    rm $POLICY_FILE
    ```

4.  Run the script, for example:

    ```bash
    sh create-s3-bucket.sh
    ```

5.  After the bucket is created, you can delete the script file, if you want.


# Weaviate



Send processed data from Unstructured to Weaviate.

You'll need:

The Weaviate prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/Ldb7PZU-pR4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

1.  A Weaviate database instance. The following information assumes that you have a Weaviate Cloud (WCD) account with a Weaviate database cluster in that account.
    [Create a WCD account](https://weaviate.io/developers/wcs/quickstart#create-a-wcd-account). [Create a database cluster](https://weaviate.io/developers/wcs/quickstart#create-a-weaviate-cluster). For other database options, [learn more](https://weaviate.io/developers/weaviate/installation).

2.  The URL and API key for the database cluster. [Get the URL and API key](https://weaviate.io/developers/wcs/quickstart#explore-the-details-panel).

3.  A collection in the database cluster. Note the name of the collection, also known as the collection's *class name*. [Create a collection](https://weaviate.io/developers/wcs/tools/collections-tool).

    The schema of the collection that you use must match the data that Unstructured writes to it. Otherwise, you might get unexpected results or errors.
    Unstructured cannot provide a schema that is guaranteed to work for everyone in all circumstances. This is because these schemas will vary based on
    your source files' types; how you want Unstructured to partition, chunk, and generate embeddings;
    any custom post-processing code that you run; and other factors.

    You can adapt the following collection schema example for your own needs:

    ```json
    {
        "class": "Elements",
        "properties": [
            {
                "name": "element_id",
                "dataType": ["text"]
            },
            {
                "name": "text",
                "dataType": ["text"]
            },
            {
                "name": "embeddings",
                "dataType": ["number[]"]
            },
            {
                "name": "metadata",
                "dataType": ["object"],
                "nestedProperties": [
                    {
                        "name": "parent_id",
                        "dataType": ["text"]
                    },
                    {
                        "name": "page_number",
                        "dataType": ["text"]
                    },
                    {
                        "name": "is_continuation",
                        "dataType": ["boolean"]
                    },
                    {
                        "name": "orig_elements",
                        "dataType": ["text"]
                    }
                ]
            }
        ]
    }
    ```

    See also :

    *   [Collection schema](https://weaviate.io/developers/weaviate/config-refs/schema)
    *   [Unstructured document elements and metadata](/api-reference/api-services/document-elements)

To create the destination connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Destinations**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Weaviate**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for the connector.
*   **Host URL** (*required*): The URL of the Weaviate database cluster.
*   **Collection Name** (*required*): The name of the target collection within the cluster.
*   **Batch Size**: The maximum number of records to be transmitted in a single batch.
*   **API Key** (*required*): The API key provided by Weaviate to access the cluster.


# Document elements and metadata



When Unstructured [partitions](/platform/partitioning) a file, the result is a list of *document elements*, sometimes referred to simply as *elements*. These elements represent different components of the source file.

## Element example

Here's an example of what an element might look like:

```json
{
    "type": "NarrativeText",
    "element_id": "5ef1d1117721f0472c1ad825991d7d37",
    "text": "The Unstructured documentation covers the following services:",
    "metadata": {
        "last_modified": "2024-05-01T14:15:22",
        "page_number": 1,
        "languages": ["eng"],
        "parent_id": "56f24319ae258b735cac3ec2a271b1d9",
        "file_directory": "/content",
        "filename": "Unstructured documentation.html",
        "filetype": "text/html"
    }
}
```

Every element has a [type](#element-type); an [element\_id](#element-id); the extracted `text`; and some [metadata](#metadata) which might
vary depending on the element type, file structure, and some additional settings that are applied during
[partitioning](/platform/partitioning), chunking, summarizing, and embedding.

## Element type

Instead of treating all files as strings of plain text, Unstructured preserves the semantic structure of the files.
This gives you more control and flexibility over how you further use the processed files and allows you to take their
structure into consideration. At the same time, normalizing data from various file formats to the Unstructured element
type scheme lets you treat all files the same in your downstream processing, regardless of source format.
For example, if you plan to summarize a file, you might only be interested in the narrative
of the file and not care about its headers and footers. You can easily filter out the elements you don't need by specifying their type.

Here are some examples of the element types your file might contain:

| Element type        | Description                                                                                                                                          |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Address`           | A text element for capturing physical addresses.                                                                                                     |
| `EmailAddress`      | A text element for capturing email addresses.                                                                                                        |
| `FigureCaption`     | An element for capturing text associated with figure captions.                                                                                       |
| `Footer`            | An element for capturing document footers.                                                                                                           |
| `Formula`           | An element containing formulas in a file.                                                                                                            |
| `Header`            | An element for capturing document headers.                                                                                                           |
| `Image`             | A text element for capturing image metadata.                                                                                                         |
| `ListItem`          | `ListItem` is a `NarrativeText` element that is part of a list.                                                                                      |
| `NarrativeText`     | `NarrativeText` is an element consisting of multiple, well-formulated sentences. This excludes elements such titles, headers, footers, and captions. |
| `PageBreak`         | An element for capturing page breaks.                                                                                                                |
| `Table`             | An element for capturing tables.                                                                                                                     |
| `Title`             | A text element for capturing titles.                                                                                                                 |
| `UncategorizedText` | Base element for capturing free text from within files.                                                                                              |

If you apply chunking, you will also see the `CompositeElement` type.
`CompositeElement` is a chunk formed from text (non-`Table`) elements.
A composite element might be formed by combining one or more sequential elements produced by partitioning. For example,
several individual list items might be combined into a single chunk.

## Element ID

By default, the element ID is a SHA-256 hash of the element's text, its position on the page, the page number it's on,
and the name of the related file. This is to ensure that the ID is deterministic and unique at the file level.

## Metadata

Unstructured tracks a variety of metadata about the elements extracted from files. Metadata is tracked at the element level within `metadata`.

Element metadata enables you to do things such as:

*   Filter file elements based on an element's metadata value. For instance, you might want to limit your scope to elements from a certain page, or you might want to use only elements that have an email matching a regular expression in their metadata.
*   Map an element to the page where it occurred so that the original page can be retrieved when that element matches search criteria.

### Common metadata fields

All file types return the following `metadata` fields when the information is available from the source file:

| Metadata field name        | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `category_depth`           | The depth of the element relative to other elements of the same category. Category depth is the depth of an element relative to other elements of the same category. It is set by a file partitioner and enables the [document hierarchy](#document-hierarchy) after processing to compute more accurate hierarchies. Category depth might be set using native document hierarchies, for example reflecting `<H1>` or `<H2>` tags within an HTML file or the indentation level of a bulleted list item in a Word document. |
| `coordinates`              | Any X-Y bounding box [coordinates](#element-coordinates).                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `detection_class_prob`     | The detection model class probabilities. Applies only to Unstructured inference using the **High Res** strategy.                                                                                                                                                                                                                                                                                                                                                                                                           |
| `emphasized_text_contents` | The related emphasized text (bold or italic) in the original file.                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `emphasized_text_tags`     | Any tags on the text that are emphasized in the original file.                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `file_directory`           | The related file's directory.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `filename`                 | The related file's filename.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `filetype`                 | The related file's type.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `is_continuation`          | True if the element is a continuation of a previous element. Only relevant for chunking, if an element was divided into two due to **Max Characters**.                                                                                                                                                                                                                                                                                                                                                                     |
| `languages`                | Document languages at the file or element level. The list is ordered by probability of being the primary language of the text.                                                                                                                                                                                                                                                                                                                                                                                             |
| `last_modified`            | The related file's last modified date.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `parent_id`                | The ID of the element's parent element. `parent_id` might be used to infer where an element resides within the overall [document hierarchy](#document-hierarchy). For instance, a `NarrativeText` element might have a `Title` element as a parent (a “subtitle”), which in turn might have another `Title` element as its parent (a "title").                                                                                                                                                                             |
| `text_as_html`             | The HTML representation of the related extracted table. Only applicable to [table elements](#table-specific-metadata).                                                                                                                                                                                                                                                                                                                                                                                                     |

Notes on common metadata fields:

#### Document hierarchy

`parent_id` and `category_depth` enhance hierarchy detection to identify the document
structure in various file formats by measuring relative depth of an element within its category. This is especially
useful in files with native hierarchies like HTML or Word files, where elements like headings or list items inherently define structure.

#### Element coordinates

Some file types support location data for the elements, usually in the form of bounding boxes.

The `coordinates` metadata field contains:

*   `points` : These specify the corners of the bounding box starting from the top left corner and proceeding counter-clockwise. The points represent pixels, the origin is in the top left and the `y` coordinate increases in the downward direction.
*   `system`: The points have an associated coordinate system. A typical example of a coordinate system is `PixelSpace`, which is used for representing the coordinates of images. The coordinate system has a name, orientation, layout width, and layout height.

### Additional metadata fields by file type

| Field name             | Applicable file types | Description                                                                                                                         |
| ---------------------- | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `page_number`          | DOCX, PDF, PPT, XLSX  | The related file's page number.                                                                                                     |
| `page_name`            | XLSX                  | The related sheet's name in an [Excel file](#microsoft-excel-files).                                                                |
| `sent_from`            | EML                   | The related [email](#email) sender.                                                                                                 |
| `sent_to`              | EML                   | The related [email](#email) recipient.                                                                                              |
| `subject`              | EML                   | The related [email](#email) subject.                                                                                                |
| `attached_to_filename` | MSG                   | The name of the file that the attached file is attached to.                                                                         |
| `header_footer_type`   | Word Doc              | The pages that a header or footer applies to in a [Word document](#microsoft-word-files): `primary`, `even_only`, and `first_page`. |
| `link_urls`            | HTML                  | The URL that is associated with a link in a document.                                                                               |
| `link_texts`           | HTML                  | The text that is associated with a link in a document.                                                                              |
| `section`              | EPUB                  | The book section title corresponding to a table of contents.                                                                        |

Here are some notes on additional metadata fields by file type:

#### Email

Emails will include `sent_from`, `sent_to`, and `subject` metadata. `sent_from` is a list of strings because
the [RFC 822](https://www.rfc-editor.org/rfc/rfc822) spec for emails allows for multiple sent from email addresses.

#### Microsoft Excel files

For Excel files, metadata will contain a `page_name` element, which corresponds to the sheet name in the Excel
file.

#### Microsoft Word files

Headers and footers in Word files include a `header_footer_type` indicating which page a header or footer applies to.
Valid values are `"primary"`, `"even_only"`, and `"first_page"`.

### Table-specific metadata

For `Table` elements, the raw text of the table will be stored in the `text` attribute for the element, and HTML representation
of the table will be available in the element metadata under `text_as_html`.
Unstructured will automatically extract all tables for all doc types if you check the **Infer Table Structure** in the **ConnectorSettings** area of the **Transform** section of a workflow.

Here's an example of a table element. The `text` of the element will look like this:

```text
Dataset Base Model1 Large Model Notes PubLayNet [38] F / M M Layouts of modern scientific documents PRImA [3] M - Layouts of scanned modern magazines and scientific reports Newspaper [17] F - Layouts of scanned US newspapers from the 20th century TableBank [18] F F Table region on modern scientific and business document HJDataset [31] F / M - Layouts of history Japanese documents
```

And the `text_as_html` metadata for the same element will look like this:

```html
<table><thead><th>Dataset</th><th>| Base Model’</th><th>| Notes</th></thead><tr><td>PubLayNet</td><td>[38] F/M</td><td>Layouts of modern scientific documents</td></tr><tr><td>PRImA [3]</td><td>M</td><td>Layouts of scanned modern magazines and scientific reports</td></tr><tr><td>Newspaper</td><td>F</td><td>Layouts of scanned US newspapers from the 20th century</td></tr><tr><td>TableBank</td><td>F</td><td>Table region on modern scientific and business document</td></tr><tr><td>HJDataset [31]</td><td>F/M</td><td>Layouts of history Japanese documents</td></tr></table>
```

### Data connector metadata fields

Documents can include additional file metadata, based on the specified source connector.

#### Common data connector metadata fields

*   `date_created`
*   `date_modified`
*   `date_processed`
*   `record_locator`
*   `url`
*   `version`

#### Additional metadata fields by connector type (within record\_locator)

| Source connector | Additional metadata                  |
| ---------------- | ------------------------------------ |
| Azure            | `protocol`, `remote_file_path`       |
| Elasticsearch    | `document_id`, `index_name`, `url`   |
| Google Drive     | `drive_id`, `file_id`                |
| OneDrive         | `server_relative_path`, `user_pname` |
| S3               | `protocol`, `remote_file_path`       |
| SharePoint       | `server_path`, `site_url`            |


# Embedding



After partitioning, chunking, and summarizing, the *embedding* step creates arrays of numbers
known as *vectors*, representing the text that is extracted by Unstructured.
These vectors are stored or *embedded* next to the text itself. These vector embeddings are generated by an
[embedding model](https://python.langchain.com/v0.2/docs/concepts/#embedding-models) that is provided by
an *embedding provider*.

You typically save these embeddings in a *vector store*.
When a user queries a retrieval augmented generation (RAG) application, the application can use a vector database to perform
a [similarity search](https://www.pinecone.io/learn/what-is-similarity-search/) in that vector store
and then return the items whose embeddings are the closest to that user's query.

Here is an example of a document element generated by Unstructured, along with its vector embeddings generated by
the embedding model [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)
on Hugging Face:

```json
{
    "type": "Title",
    "element_id": "fdbf5369-4485-453b-9701-1bb42c83b00b",
    "text": "THE CONSTITUTION of the United States",
    "metadata": {
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 1,
        "filename": "constitution.pdf",
        "data_source": {
            "record_locator": {
                "path": "/input/constitution.pdf"
            },
            "date_created": "1723069423.0536132",
            "date_modified": "1723069423.055078",
            "date_processed": "1725666244.571788",
            "permissions_data": [
                { 
                    "mode": 33188
                }
            ]
        }
    },
    "embeddings": [
        -0.06138836592435837,
        0.08634615689516068,
        -0.019471267238259315,
        "<full-results-omitted-for-brevity>",
        0.0895417109131813,
        0.05604064092040062,
        0.01376157347112894
    ]
}
```

[Learn more](https://unstructured.io/blog/understanding-embedding-models-make-an-informed-choice-for-your-rag).

## Generate embeddings

To generate embeddings, choose one of the following embedding providers and models in the **Providers** section of an **Embedder** node in a workflow:

<Note>You can change a workflow's predefined provider only through [Custom](/platform/workflows#create-a-custom-workflow) workflow settings.</Note>

*   **OpenAI**: Use [OpenAI](https://openai.com) to generate embeddings. Also, choose the model to use:

    *   **text-embedding-3-small**, with 1536 dimensions.
    *   **text-embedding-3-large**, with 3072 dimensions.
    *   **Ada 002 (Text)**, with 1536 dimensions.

    [Learn more](https://platform.openai.com/docs/guides/embeddings).

*   **Vertex AI**: Use [Vertex AI](https://cloud.google.com/vertex-ai) to generate embeddings by using the [textembedding-gecko@001](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings) model, with 768 dimensions.


# Jobs



## Jobs dashboard

![Jobs in the sidebar](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Jobs-Sidebar.png)

To view the jobs dashboard, on the sidebar, click **Jobs**.

The jobs dashboard provides a centralized view for managing and monitoring the execution of data processing tasks within your workflows.

The jobs dashboard lists each job and its associated **Status**, **Job ID**, **Created** date and time, **Workflow** name, and **Runtime** duration.

Each job's status, shown in the **Status** column, can be:

*   **Scheduled**: The job will run in the future.

*   **In Progress**: The job's documents are being processed.

*   **Finished**: The job has completed successfully.

*   **Failed**: The job has errors.

## Run a job

<Warning>
  You must first have an existing workflow to run a job against.

  If you do not have an existing workflow, stop. [Create a workflow](/platform/workflows#create-a-workflow), and then return here.

  To see your existing workflows, on the sidebar, click **Workflows**.
</Warning>

To run a job, on the sidebar, click **Workflows**, and then click **Run** in the row for the workflow that you want to run.

## Monitor a job

![Completed job](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Job-Complete.png)

The job details pane is a comprehensive section for monitoring the specific details of jobs executed within a particular workflow. To access this pane, click the specific job on the jobs dashboard.

Clicking the **Details** button shows:

*   The job's ID.
*   The job's start date.
*   The source and destination connectors for the job.
*   Information about the files that were processed in each of the job's stages. This information shows how many files are:

    *   **Ready**: Waiting to be processed.
    *   **In Progress**: Being processed.
    *   **Success**: Successfully processed.
    *   **Failure**: Failed to process successfully.

To see the job's logs, click the **Logs** button.


# Overview



The Unstructured Platform is a no-code user interface, pay-as-you-go platform for transforming your unstructured data into data that is ready for Retrieval Augmented Generation (RAG).

<Tip>To start using the Unstructured Platform right away, skip ahead to the [quickstart](/platform/quickstart).</Tip>

![Partial view of the Unstructured Platform](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Start-Screen-Partial.png)

## How does it work?

To get your data RAG-ready, the Unstructured Platform moves it through the following process:

```mermaid
  flowchart LR
    Connect[1. Connect]-->Route[2. Route]-->Transform[3. Transform]-->Chunk[4. Chunk]-->Enrich[5. Enrich]-->Embed[6. Embed]-->Persist[7. Persist]
```

<Steps>
  <Step title="Connect">
    The Unstructured Platform offers multiple [source connectors](/platform/sources/overview) to connect to your data in its existing location.
  </Step>

  <Step title="Route">
    Routing determines which strategy Unstructured Platform uses to transforming your documents into Unstructured's canonical JSON schema. The Unstructured Platform provides these [partitioning](/platform/partitioning) strategies for document transformation:

    *   **Fast** is great for when there is extractable text available, like in HTML files or in the Microsoft Office Document format.
    *   **Hi Res** is best for PDFs and tables and where accurate classification of document elements is critical.
    *   If you're unsure which strategy to use, choose **Auto**, and the Unstructured Platform will handle the decision for you.
  </Step>

  <Step title="Transform">
    Your source document is transformed into Unstructured's canonical JSON schema. Regardless of the input document, this JSON schema we gives you a [standardized output](/platform/document-elements). It contains more than 20 elements, such as `Header`, `Footer`, `Title`, `NarrativeText`, `Table`, `Image`, and many more. Each document is wrapped in extensive metadata so you can understand languages, file types, sources, hierarchies, and much more.
  </Step>

  <Step title="Chunk">
    The Unstructured Platform provides these [chunking](/platform/chunking) strategies:

    *   **Basic** combines sequential elements up to specified size limits. Oversized elements are split, while tables are isolated and divided if necessary. Overlap between chunks is optional.
    *   **By Title** uses semantic chunking, understands the layout of the document, and makes intelligent splits.
    *   **By Page** attempts to preserve page boundaries when determining the chunks' contents.
    *   **By Similarity** uses an embedding model to identify topically similar sequential elements and combines them into chunks.
  </Step>

  <Step title="Enrich">
    Images and tables can be optionally summarized. This generates enriched content around the images or tables that were parsed during the transformation process.
  </Step>

  <Step title="Embed">
    The Unstructured Platform uses optional third-party [embedding](/platform/embedding) providers such as OpenAI.
  </Step>

  <Step title="Persist">
    The Unstructured Platform offers multiple [destination connectors](/platform/destinations/overview), including all major vector databases.
  </Step>
</Steps>

To simplify this process and provide it as a no-code solution, the Unstructured Platform brings together four key concepts:

```mermaid
  flowchart LR
    subgraph Workflow[3. Workflow]
        direction LR
        Source[1. Source Connector] --> Destination[2. Destination Connector]
    end
    Jobs
    Workflow[3. Workflow] --> Jobs[4. Jobs]
```

<Steps>
  <Step title="Source Connector">
    [Source connectors](/platform/sources/overview) to ingest your data into the Unstructured Platform for transformation.
  </Step>

  <Step title="Destination Connector">
    [Destination connectors](/platform/destinations/overview) tell the Unstructured Platform where to write your transformed data to.
  </Step>

  <Step title="Workflow">
    [Workflows](/platform/workflows) connect sources to destinations and provide chunking, embedding, and scheduling options.
  </Step>

  <Step title="Jobs">
    [Jobs](/platform/jobs) enable you to monitor data transformation progress.
  </Step>
</Steps>

## What support is there for compliance?

The platform is designed for global reach with SOC2 Type 1, SOC2 Type 2, and HIPAA compliance. It has support for over 50 languages.

## How do I get started?

Skip ahead to the [quickstart](/platform/quickstart).


# Partitioning



*Partitioning* extracts content from raw unstructured files and outputs that content as structured [document elements](/platform/document-elements).

For specific file types, such as image files and PDF files, the Unstructured Platform offers special strategies to partition them. Each of these
strategies has trade-offs for output speed, cost to output, and quality of output.

PDF files, for example, vary in quality and complexity. In simple cases, traditional natural language processing (NLP) extraction techniques might
be enough to extract all the text out of a document. In other cases, advanced image-to-text models are required
to process a PDF file. Some of these strategies implement rule-based workflows, which can be faster and cheaper, because they always
extract in the same way, but you might sometimes get lower-quality resolution. Other strategies implement
model-based workflows, which can be slower and costlier because they require a model that performs inference, but you can get higher-quality resolution.
When you choose a partitioning strategy for your files, you should be mindful of these speed, cost, and quality trade-offs.
For example, the **Fast** strategy can be about 100 times faster than leading image-to-text models.

To choose one of these strategies, select one of the **Partition Strategy** options in the **Partitioner** node of a workflow:

<Note>You can change a workflow's predefined strategy only through [Custom](/platform/workflows#create-a-custom-workflow) workflow settings.</Note>

*   **Fast**: This strategy is ideal for simple, text-based documents.
*   **Hi-Res**: This strategy is best for PDFs, images, and complex file types.
*   **VLM**: For your most challenging documents, including scanned and handwritten content, use this strategy, which leverages vision
    language models (VLMs). During processing, files that are not PDFs or images are processed by using the **Hi-Res** strategy and are charged
    at the **Hi-Res** rate instead.
*   **Auto**: This strategy examines each file before processing it. If the file is an image, or if the file is a PDF and at least one embedded table
    or image is found in it, **Hi-Res** is used to process that file and charged at the **Hi-Res** rate for that file. Otherwise, **Fast** is used and charged at the
    **Fast** rate for that file.


# Unstructured Platform quickstart



This quickstart uses a no-code, point-and-click user interface in your web browser to get all of your data RAG-ready. Data is processed on Unstructured-hosted compute resources.

You will need:

*   A compatible source (input) location in cloud storage that contains your documents for Unstructured to process. [See the list of supported source types](/platform/connectors#sources).
*   Compatible files in your source location. [See the list of supported file types](/platform/supported-file-types). If you do not have any files available, you can download some from the [example-docs](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/example-docs) folder in the Unstructured repo on GitHub.
*   A compatible destination (output) location in cloud storage for Unstructured to put the processed data. [See the list of supported destination types](/platform/connectors#destinations).

<iframe width="560" height="315" src="https://www.youtube.com/embed/25u3qJ_7WiA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

<Steps>
  <Step title="Sign in to your Unstructured account">
    ![Sign in to your Unstructured account](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Signin.png)

    1.  Sign in to your Unstructured account, at [https://app.unstructured.io](https://app.unstructured.io).
    2.  Click **Google** or **GitHub** to sign in with your Google or GitHub account. Or, enter your email address and then click **Sign In**.
    3.  If you entered your email address, check your email inbox for a message from Unstructured. In that email, click the **Sign In** link.
    4.  The first time you sign in, read the terms and conditions, and then click **Accept**.
  </Step>

  <Step title="Open your Unstructured Platform dashboard">
    ![Open your Unstructured Platform dashboard](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/GoToPlatform.png)

    *   From your Unstructured account dashboard, in the sidebar, click **API > Platform**.

    <Note>
      The **API > Platform** sidebar option is visible only to users who signed up for an Unstructured API services account before
      October 23, 2024. To get access to the Unstructured Platform on or after this date,
      [request access](https://unstructured.io/platform), and wait for an access enablement email from Unstructured.
    </Note>
  </Step>

  <Step title="Set the source (input) location">
    ![Sources in the sidebar](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Sources-Sidebar.png)

    1.  From your Unstructured Platform dashboard, in the sidebar, click **Connectors**.
    2.  Click **Sources**.
    3.  Click **Add new**.
    4.  For **Name**, enter some unique name for this connector.
    5.  In the **Provider** area, click the source location type that matches yours.
    6.  Click **Continue**.
    7.  Fill in the fields with the appropriate settings. [Learn more](/platform/sources/overview).
    8.  If a **Continue** button appears, click it, and fill in any additional settings fields.
    9.  Click **Save and Test**.
  </Step>

  <Step title="Set the destination (output) location">
    ![Destinations in the sidebar](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Destinations-Sidebar.png)

    1.  In the sidebar, click **Connectors**.
    2.  Click **Destinations**.
    3.  Click **Add new**.
    4.  For **Name**, enter some unique name for this connector.
    5.  In the **Provider** area, click the destination location type that matches yours.
    6.  Click **Continue**.
    7.  Fill in the fields with the appropriate settings. [Learn more](/platform/sources/overview).
    8.  If a **Continue** button appears, click it, and fill in any additional settings fields.
    9.  Click **Save and Test**.
  </Step>

  <Step title="Define the workflow">
    ![Workflows in the sidebar](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Workflows-Sidebar.png)

    1.  In the sidebar, click **Workflows**.

    2.  Click **New Workflow**.

    3.  Next to **Build it with me**, click **Create Workflow**.

        <Note>If a radio button appears instead of **Build it with me**, select it, and then click **Continue**.</Note>

    4.  For **Workflow Name**, enter some unique name for this workflow.

    5.  In the **Sources** dropdown list, select your source location from Step 3.

    6.  In the **Destinations** dropdown list, select your destination location from Step 4.

        <Note>You can select multiple source and destination locations. Files will be ingested from all of the selected source locations, and the processed data will be delivered to all of the selected destination locations.</Note>

    7.  Click **Continue**.

    8.  In the **Optimize for** section, select the option to choose one of these predefined workflow settings groups:

        *   **Basic**: Ideal for simple, text-only documents.
        *   **Advanced**: Best for PDFs, images, and complex file types.
        *   **Platinum**: For your most challenging documents, including scanned and handwritten content. It uses vision language models (VLMs).
            During processing, files that are not PDFs or images are processed by using the **Advanced** strategy and are charged at the **Advanced** rate instead.

    9.  The **Reprocess all** box applies only to the Amazon S3 and Azure Blob Storage source connectors:

        *   Checking this box reprocesses all documents in the source location on every workflow run.
        *   Unchecking this box causes only new documents that are added to the source location since the last workflow run to be processed on future runs. Previously processed documents are not processed again, even if those documents' contents change.

    10. If you want to retry processing any documents that failed to process, check the **Retry Failed Documents** box.

    11. Click **Continue**.

    12. If you want this workflow to run on a schedule, in the **Repeat Run** dropdown list, select one of the scheduling options, and fill in the scheduling settings. Otherwise, select **Don't repeat**.

    13. Click **Complete**.
  </Step>

  <Step title="Process the documents">
    ![Workflows in the sidebar](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Workflows-Sidebar.png)

    1.  If you did not choose to run this workflow on a schedule in Step 5, you can run the workflow now: on the sidebar, click **Workflows**.
    2.  Next to your workflow from Step 5, click **Run**.
  </Step>

  <Step title="Monitor the processing job">
    ![Select a job](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Select-Job.png)
    ![Completed job](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Job-Complete.png)

    1.  In the sidebar, click **Jobs**.
    2.  In the list of jobs, wait for the job's **Status** to change to **Finished**.
    3.  Click the row for the job.
    4.  If **Overview** displays **Success**, go to the next Step.
  </Step>

  <Step title="View the processed data">
    Go to your destination location to view the processed data.
  </Step>
</Steps>


# Azure



Ingest your files into Unstructured from Azure Blob Storage.

You'll need:

The Azure Storage account prerequisites.

The following video shows how to fulfill the minimum set of Azure Storage account prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/Vl3KCphlh9Y" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

<Note>
  If you are generating an SAS token as shown in the preceding video, be sure to set the following permissions:

  *   **Read** and **List** for reading from the container only.
  *   **Write** and **List** for writing to the container only.
  *   **Read**, **Write**, and **List** for both reading from and writing to the container.
</Note>

Here are some more details about these prerequisites:

*   An Azure account. To create one, [learn how](https://azure.microsoft.com/pricing/purchase-options/azure-account).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/2bQ6WiJ1ncA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An Azure Storage account, and a container within that account. [Create a storage account](https://learn.microsoft.com/azure/storage/common/storage-account-create). [Create a container](https://learn.microsoft.com/azure/storage/blobs/blob-containers-portal).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/AhuNgBafmUo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    <iframe width="560" height="315" src="https://www.youtube.com/embed/xmndjYnGvcs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Azure Storage remote URL, using the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

    For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
    Azure Storage remote URL would be `az://my-container/my-folder/`.

*   An SAS token (recommended), access key, or connection string for the Azure Storage account.  [Create an SAS token (recommended)](https://learn.microsoft.com/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens). [Get an access key](https://learn.microsoft.com/azure/storage/common/storage-account-keys-manage#view-account-access-keys). [Get a connection string](https://learn.microsoft.com/azure/storage/common/storage-configure-connection-string#configure-a-connection-string-for-an-azure-storage-account).

    Create an SAS token (recommended):

    <iframe width="560" height="315" src="https://www.youtube.com/embed/X6cmJ2IbVzo?start=240&end=370" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    Get an access key or connection string:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/muMmcwVfFqs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

To create the source connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Sources**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Azure Blob Storage**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.

*   **Remote URL** (*required*): The Azure Storage remote URL, with the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

    For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
    Azure Storage remote URL would be `az://my-container/my-folder/`.

*   **Azure Account Name**: The Azure Storage account name.

*   **Azure Account Key**: The key for the Azure Storage account, or...

*   **Azure Connection String**: The connection string for the Azure Storage account, or...

*   **SAS Token**: The SAS token for the Azure Storage account.

*   **Recursive**: Check this box to recursively access files from subfolders within the container.


# Confluence



Ingest your files into Unstructured from Confluence.

You'll need:

The Confluence prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/tXu6S6Pd7Dc" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A [Confluence account](https://www.atlassian.com/software/confluence).
*   The URL for the target Confluence instance.
*   A [Confluence user](https://confluence.atlassian.com/doc/add-and-invite-users-138313.html) with
    the correct [permissions](https://support.atlassian.com/confluence-cloud/docs/what-are-confluence-cloud-permissions-and-restrictions/) to
    access the target spaces and pages in the Confluence instance.
*   A [Confluence API token](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/) for the target Confluence user.
*   Optionally, the names of the specific [spaces](https://support.atlassian.com/confluence-cloud/docs/navigate-spaces/) in the Confluence instance to access.

To create the source connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Sources**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Confluence**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **URL** (*required*): The URL to the target Confluence Cloud instance.
*   **User Email** (*required*): The email address of the user who has access to the instance.
*   **API Token** (*required*): The Confluence API token that provides access to the instance.
*   **Max Number of Spaces**: The maximum number of Confluence spaces to access within the Confluence Cloud instance.
    The default is 500 unless otherwise specified.
*   **Max Number of Docs Per Space**: The maximum number of documents to access within each space.
    The default is 150 unless otherwise specified.
*   **List of Spaces**: A comma-separated string that lists the names of all of the spaces to access, for example: `luke,paul`.
    By default, if no space names are specified, and the **Max Number of Spaces** is reached for the instance, be aware that you might get
    unexpected results.


# Databricks Volumes



Ingest your files into Unstructured from Databricks Volumes.

You'll need:

The Databricks Volumes prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/rNZpwa1-g7M" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The preceding video shows how to use Databricks personal access tokens (PATs), which are supported only for [Unstructured Ingest](/ingestion/overview).

To learn how to use Databricks-managed service principals, which are supported by both the [Unstructured Platform](/platform/overview) and Unstructured Ingest,
see the additional videos later on this page.

*   The Databricks workspace URL. Get the workspace URL for
    [AWS](https://docs.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids),
    [Azure](https://learn.microsoft.com/azure/databricks/workspace/workspace-details#workspace-instance-names-urls-and-ids),
    or [GCP](https://docs.gcp.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids).

    Examples:

    *   AWS: `https://<workspace-id>.cloud.databricks.com`
    *   Azure: `https://adb-<workspace-id>.<random-number>.azuredatabricks.net`
    *   GCP: `https://<workspace-id>.<random-number>.gcp.databricks.com`

*   The Databricks authentication details. For more information, see the documentation for
    [AWS](https://docs.databricks.com/dev-tools/auth/index.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/),
    or [GCP](https://docs.gcp.databricks.com/dev-tools/auth/index.html).

    The following videos show how to create a Databricks-managed service principal and then grant it access to a Databricks volume:

    <iframe width="560" height="315" src="https://www.youtube.com/embed/wBmqv5DaA1E" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    <iframe width="560" height="315" src="https://www.youtube.com/embed/DykQRxgh2aQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

    For the [Unstructured Platform](/platform/overview), only the following Databricks authentication type is supported:

    *   For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.
        Note that for Azure, only Databricks-managed service principals are supported. Microsoft Entra ID-managed service principals are not supported.

    For [Unstructured Ingest](/ingestion/overview), the following Databricks authentication types are supported:

    *   For Databricks personal access token authentication (AWS, Azure, and GCP): The personal access token's value.
    *   For username and password (basic) authentication (AWS only): The user's name and password values.
    *   For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.
    *   For OAuth user-to-machine (U2M) authentication (AWS, Azure, and GCP): No additional values.
    *   For Azure managed identities (MSI) authentication (Azure only): The client ID value for the corresponding managed identity.
    *   For Microsoft Entra ID service principal authentication (Azure only): The tenant ID, client ID, and client secret values for the corresponding service principal.
    *   For Azure CLI authentication (Azure only): No additional values.
    *   For Microsoft Entra ID user authentication (Azure only): The Entra ID token for the corresponding Entra ID user.
    *   For Google Cloud Platform credentials authentication (GCP only): The local path to the corresponding Google Cloud service account's credentials file.
    *   For Google Cloud Platform ID authentication (GCP only): The Google Cloud service account's email address.

*   The Databricks catalog name for the volume. Get the catalog name for [AWS](https://docs.databricks.com/catalogs/manage-catalog.html), [Azure](https://learn.microsoft.com/azure/databricks/catalogs/manage-catalog), or [GCP](https://docs.gcp.databricks.com/catalogs/manage-catalog.html).

*   The Databricks schema name for the volume. Get the schema name for [AWS](https://docs.databricks.com/schemas/manage-schema.html), [Azure](https://learn.microsoft.com/azure/databricks/schemas/manage-schema), or [GCP](https://docs.gcp.databricks.com/schemas/manage-schema.html).

*   The Databricks volume name, and optionally any path in that volume that you want to access directly. Get the volume information for [AWS](https://docs.databricks.com/files/volumes.html), [Azure](https://learn.microsoft.com/azure/databricks/files/volumes), or [GCP](https://docs.gcp.databricks.com/files/volumes.html).

*   Make sure that the target user or service principal has access to the target volume. To learn more, see the documentation for
    [AWS](https://docs.databricks.com/volumes/utility-commands.html#change-permissions-on-a-volume),
    [Azure](https://learn.microsoft.com/azure/databricks/volumes/utility-commands#change-permissions-on-a-volume),
    or [GCP](https://docs.gcp.databricks.com/volumes/utility-commands.html#change-permissions-on-a-volume).

To create the source connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Sources**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Databricks Volumes**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Host** (*required*): The Databricks workspace host URL.
*   **Catalog** (*required*): The name of the catalog to use.
*   **Schema** : The name of the associated schema. If not specified, **default** is used.
*   **Volume** (*required*): The name of the associated volume.
*   **Volume Path** : Any optional path to access within the volume.
*   **Client ID** (*required*): The application ID value for the Databricks-managed service principal that has access to the volume.
*   **Client Secret** (*required*): The associated OAuth secret value for the Databricks-managed service principal that has access to the volume.

To learn how to create a Databricks-managed service principal, get its application ID, and generate an associated OAuth secret,
see the documentation for
[AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
[Azure](https://learn.microsoft.com/databricks/dev-tools/auth/oauth-m2m),
or [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html).

For Azure, only Databricks-managed service principals are supported. Microsoft Entra ID-managed service principals are not supported.

To learn how to grant a Databricks-managed service principal access to a volume, see the documentation for
[AWS](https://docs.databricks.com/volumes/utility-commands.html#change-permissions-on-a-volume),
[Azure](https://learn.microsoft.com/azure/databricks/volumes/utility-commands#change-permissions-on-a-volume),
or [GCP](https://docs.gcp.databricks.com/volumes/utility-commands.html#change-permissions-on-a-volume).


# Google Cloud Storage



Ingest your files into Unstructured from Google Cloud Storage.

You'll need:

The Google Cloud Storage prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/HYaALQ0F-L4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A Google Cloud service account. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#console).

*   A service account key for the service account. See [Create a service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating) in
    [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete).

    To ensure maximum compatibility across Unstructured service offerings, you should give the service account key information to Unstructured as
    a single-line string that contains the contents of the downloaded service account key file (and not the service account key file itself).
    To print this single-line string without line breaks, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
    In this command, replace `<path-to-downloaded-key-file>` with the path to the service account key file that you downloaded by following the preceding instructions.

    *   For macOS or Linux:
        ```text
        tr -d '\n' < <path-to-downloaded-key-file>
        ```
    *   For Windows:
        ```text
        (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
        ```

*   The URI for a Google Cloud Storage bucket. This URI consists of the target bucket name, plus any target folder within the bucket, expressed as `gs://<bucket-name>[/folder-name]`. [Create a bucket](https://cloud.google.com/storage/docs/creating-buckets#console).

    This bucket must have, at minimum, one of the following roles applied to the target Google Cloud service account:

    *   `Storage Object Viewer` for bucket read access.
    *   `Storage Object Creator` for bucket write access.
    *   The `Storage Object Admin` role provides read and write access, plus access to additional bucket operations.

    To apply one of these roles to a service account for a bucket, see [Add a principal to a bucket-level policy](https://cloud.google.com/storage/docs/access-control/using-iam-permissions#bucket-add)
    in [Set and manage IAM policies on buckets](https://cloud.google.com/storage/docs/access-control/using-iam-permissions).

To create the source connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Sources**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Google GCS**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Bucket URI** (*required*): The URI for the Google Cloud Storage bucket and any target folder path within the bucket. This URI takes the format `gs://<bucket-name>[/folder-name]`.
*   **Service Account Key** (*required*): The contents of a service account key file, expressed as a single string without line breaks, for a Google Cloud service account that has the required access permissions to the bucket.
*   **Recursive**: Check this box to ingest data recursively from any subfolders, starting from the path specified by **Bucket URI**.


# MongoDB



Ingest your files into Unstructured from MongoDB.

You'll need:

The MongoDB prerequisites for a MongoDB Atlas deployment:

<iframe width="560" height="315" src="https://www.youtube.com/embed/g6qDfbg808M" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A MongoDB Atlas account. [Create an account](https://www.mongodb.com/cloud/atlas/register).
*   A MongoDB Atlas cluster. [Create a cluster](https://www.mongodb.com/docs/atlas/tutorial/deploy-free-tier-cluster).
*   The cluster must be reachable from your application environment. [Learn how](https://www.mongodb.com/docs/atlas/setup-cluster-security/#network-and-firewall-requirements).
*   The cluster must be configured to use your IP address. [Learn how](https://www.mongodb.com/docs/atlas/setup-cluster-security/#ip-access-list).
*   The cluster must have at least one database. [Create a database](https://www.mongodb.com/docs/compass/current/databases/#create-a-database).
*   The database must have at least one user, and that user must have sufficient access to the database. [Create a database user](https://www.mongodb.com/docs/atlas/security-add-mongodb-users/#add-database-users). [Give the user database access](https://www.mongodb.com/docs/manual/core/authorization/).
*   The database must have at least one collection. [Create a collection](https://www.mongodb.com/docs/compass/current/collections/#create-a-collection).
*   The connection string for the cluster. For MongoDB Atlas, this connection string must include the protocol, username, password, host, and cluster name. For example:

    ```text
    mongodb+srv://<db_user>:<db_password>@<host>/?retryWrites=true&w=majority&appName=<cluster>
    ```

    To get the connection string in MongoDB Atlas, do the following:

    1.  Log in to your MongoDB Atlas console.
    2.  In the sidebar, under **Databases**, click **Clusters**.
    3.  Click on the cluster you want to connect to.
    4.  Click **Connect**, or click the **Cmd Line Tools** tab and then click **Connect Instructions**.
    5.  Click **Drivers**.
    6.  Under **Add your connection string into your application code**, copy the connection string.
        You can then close the **Connect** dialog in MongoDB Atlas.

        Before you use this connection string, be sure to fill in any placeholders in the string, such as your MongoDB Atlas database user's password value.

    [Learn more](https://www.mongodb.com/resources/products/fundamentals/mongodb-connection-string).

To create the source connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Sources**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **MongoDB**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Database** (*required*): The name of the database on the instance.
*   **Collection** (*required*): The name of the collection within the database.
*   **Connection String** (*required*): The MongoDB instance connection string.


# OneDrive



Ingest your files into Unstructured from OneDrive.

You'll need:

The OneDrive prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/7uSu0LsbcEQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   A OneDrive account.
*   The User Principal Name (UPN) for the OneDrive account. This is typically the OneDrive account user's email address.
*   The path to the target OneDrive folder, starting from the OneDrive account's root folder, for example `my-folder/my-subfolder`.
*   The client ID, tenant ID, and client secret for the Microsoft Entra ID app registration that has access to the target OneDrive account and
    also has the correct set of Microsoft Graph authentication scopes. These scopes include:

    *   `Files.ReadWrite.All` (if both reading and writing are needed)
    *   `Sites.ReadWrite.All` (if both reading and writing are needed)
    *   `User.Read.All`
    *   `Directory.Read.All`

See also:

*   [Registering your app for Microsoft Graph](https://learn.microsoft.com/onedrive/developer/rest-api/getting-started/app-registration)
*   [OneDrive authentication and sign-in](https://learn.microsoft.com/onedrive/developer/rest-api/getting-started/msa-oauth)
*   [OneDrive API documentation](https://docs.microsoft.com/onedrive/developer/rest-api/)

To create the source connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Sources**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **OneDrive**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Client ID** (*required*): The application (client) ID of the Microsoft Entra ID app registration that has access to the OneDrive account.
*   **Tenant ID** (*required*): The directory (tenant) ID of the Entra ID app registration.
*   **Authority URL** (*required*): The authentication token provider URL for the Entra ID app registration. The default is `https://login.microsoftonline.com`.
*   **Principal Name** (*required*): The User Principal Name (UPN) for the OneDrive user account in Entra ID. This is typically the user's email address.
*   **Path**: The path to the target folder in the OneDrive account, starting with the account's root folder, for example `my-folder/my-subfolder`.
*   **Recursive**: Check this box to recursively access files from subfolders within the specified OneDrive path.
*   **Client Credential** (*required*): The client secret for the Entra ID app registration.


# Outlook



Ingest your files into Unstructured from Outlook.

You'll need:

The Outlook prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/9yESRp9pzv0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The Outlook user's email address.
*   A Microsoft Entra ID app registration in the same Azure account as the Outlook account. You will need
    this app registration's application (client) ID, client secret, and directory (tenant) ID. [Learn how](https://learn.microsoft.com/entra/identity-platform/quickstart-register-app).
*   The Entra ID app registration must have the following Graph API permission levels of the application (not delegated) type:

    *   `Mail.Read`
    *   `Mail.ReadBasic`
    *   `User.Read.All`

    [Learn how](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions).

To create the source connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Sources**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Outlook**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **User Email** (*required*): The user's email address for the target Outlook account.
*   **Authentication Provider URL** (*required*): The authentication token provider URL for the Entra ID app registration. The default is `https://login.microsoftonline.com`.
*   **Tenant** (*required*): The directory (tenant) ID of the Microsoft Entra ID app registration that has access to the Outlook account.
*   **Client ID** (*required*): The application (client) ID of the Entra ID app registration.
*   **Client Credential** (*required*): The client secret for the Entra ID app registration.
*   **Folders** (*required*): A list of top-level folder names to access in the Outlook account. Separate multiple folder names by commas. Do not specify any subfolder names.
*   **Recursive**: Check this box to access any and all subfolders within each of the specified top-level folders.


# Overview

Source connectors are essential components in data integration systems that establish a link between your files and the data ingestion process. They facilitate the batch processing of files, allowing for the systematic retrieval and ingestion of data stored in various file formats.

![Sources in the sidebar](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Sources-Sidebar.png)

To see your existing source connectors, on the sidebar, click **Connectors**, and then click **Sources**.

To create a source connector:

1.  In the sidebar, click **Connectors**.
2.  Click **Sources**.
3.  Click **Add new**.
4.  For **Name**, enter some unique name for this connector.
5.  In the **Provider** area, click the source location type that matches yours.
6.  Click **Continue**.
7.  Fill in the fields according to your connector type. To learn how, click your connector type in the following list:

    *   [Azure](/platform/sources/azure-blob-storage)
    *   [Confluence](/platform/sources/confluence)
    *   [Databricks Volumes](/platform/sources/databricks-volumes)
    *   [Google Cloud Storage](/platform/sources/google-cloud)
    *   [MongoDB](/platform/sources/mongodb)
    *   [OneDrive](/platform/sources/onedrive)
    *   [Outlook](/platform/sources/outlook)
    *   [S3](/platform/sources/s3)
    *   [SharePoint](/platform/source/sharepoint)
8.  If a **Continue** button appears, click it, and fill in any additional settings fields.
9.  Click **Save and Test**.


# S3



Ingest your files into Unstructured from Amazon S3.

You'll need:

The Amazon S3 prerequisites:

The following video shows how to fulfill the minimum set of S3 prerequisites:

<iframe width="560" height="315" src="https://www.youtube.com/embed/hyDHfhVVAhs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but
not recommended); or generate an AWS STS session token for temporary access, if required by your organization's security
requirements. For more information about prerequisites, see the following:

*   An AWS account. [Create an AWS account](https://aws.amazon.com/free).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/lIdh92JmWtg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   An S3 bucket. [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
    Additional approaches are in the following video and in the how-to sections at the end of this page.

    <iframe width="560" height="315" src="https://www.youtube.com/embed/e6w9LwZJFIA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   Anonymous (supported but not recommended) or authenticated access to the bucket.

    *   [Enable anonymous bucket access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user).

        ![Enable anonymous bucket access](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/connectors/s3-anon-access.gif)

    *   [Enable authenticated bucket access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/walkthrough1.html).

        ![Enable authenticated bucket access](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/connectors/s3-auth-access.gif)

*   For authenticated bucket read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/y4SfQoJpipo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For bucket write access, authenticated access to the bucket must be enabled (anonymous access must not be enabled), and the authenticated AWS IAM user must have at
    minimum the permission of `s3:PutObject` for that bucket. [Learn how](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-policies-s3.html).

*   For authenticated access, an AWS access key and secret access key for the authenticated AWS IAM user in the account.
    [Create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/MoFTaGJE65Q" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   For authenticated access in untrusted environments or enhanced security scenarios, an AWS STS session token for temporary access, in addition to an AWS access key and secret access key. [Create a session token](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_getsessiontoken).

    <iframe width="560" height="315" src="https://www.youtube.com/embed/4_csSXc_GNU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   If the target files are in the root of the bucket, the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
    If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

*   If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has
    authenticated access to the folder as well. [Enable authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

To create the source connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Sources**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **Amazon S3**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Bucket Name** (*required*): The path to the bucket or folder, formatted as `s3://my-bucket/` (if the files are in the bucket's root) or `s3://my-bucket/my-folder/`.
*   **AWS Access Key**: The AWS access key ID for the authenticated AWS IAM user, if the bucket has authenticated access enabled.
*   **AWS Secret Key**: The AWS secret access key corresponding to the preceding AWS access key ID, if the bucket has authenticated access enabled
*   **Token**: If required, the AWS STS session token for temporary access.
*   **Endpoint URL**: A custom URL, if connecting to a non-AWS S3 bucket.
*   **Recursive**: Check this box to access subfolders within the bucket.
*   **Connect Anonymously** (*source connectors only*): Check this box if reading from a bucket with anonymous access enabled.

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2.  Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3.  Browse to the existing bucket and open it.

4.  Click the **Permissions** tab.

5.  In the **Bucket policy** area, click **Edit**.

6.  In the **Policy** text area, copy the following JSON-formatted policy.
    To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this policy, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-bucket-name>` in two places with the name of your bucket.

    ```json
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::<my-account-id>:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::<my-bucket-name>",
                    "arn:aws:s3:::<my-bucket-name>/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    ```

7.  Click **Save changes**.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
    the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    ```yaml
    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

    Parameters:
      BucketName:
        Type: String
        Description: 'Name of the S3 bucket to create'

    Resources:
      MyS3Bucket:
        Type: 'AWS::S3::Bucket'
        Properties:
          BucketName: !Ref BucketName
          PublicAccessBlockConfiguration:
            BlockPublicAcls: true
            BlockPublicPolicy: false
            IgnorePublicAcls: true
            RestrictPublicBuckets: true

      BucketPolicy:
        Type: 'AWS::S3::BucketPolicy'
        Properties:
          Bucket: !Ref MyS3Bucket
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Sid: AllowAllAuthenticatedUsersInAccount
                Effect: Allow
                Principal:
                  AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
                Action:
                  - 's3:GetObject'
                  - 's3:PutObject'
                  - 's3:ListBucket'
                  - 's3:DeleteObject'
                Resource:
                  - !Sub 'arn:aws:s3:::${BucketName}'
                  - !Sub 'arn:aws:s3:::${BucketName}/*'

    Outputs:
      BucketName:
        Description: 'Name of the created S3 bucket'
        Value: !Ref MyS3Bucket
    ```

2.  Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3.  Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4.  Click **Create stack > With new resources (standard)**.

5.  On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6.  Click **Choose file**, and browse to and select the YAML file from your local machine.

7.  Click **Next**.

8.  Enter a unique **Stack name** and **BucketName**.

9.  Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1.  [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2.  [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3.  Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
    To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
    specific username.

    In this script, replace the following:

    *   Replace `<my-account-id>` with your AWS account ID.
    *   Replace `<my-unique-bucket-name>` with the name of your bucket.
    *   Replace `<us-east-1>` with your AWS Region.

    ```bash
    #!/bin/bash

    # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
    ACCOUNT_ID="<my-account-id>"
    BUCKET_NAME="<my-unique-bucket-name>"
    REGION="<us-east-1>"

    # Temporary filename for the bucket policy.
    # Do not change this variable.
    POLICY_FILE="bucket_policy.json"

    # Create the bucket.
    aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

    # Wait for the bucket to exist.
    echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
    aws s3api wait bucket-exists --bucket $BUCKET_NAME

    # Check if the wait command was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket '$BUCKET_NAME' has been fully created."
    else
        echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
        exit 1
    fi

    # Remove the "block public policy" bucket access setting.
    aws s3api put-public-access-block \
        --bucket $BUCKET_NAME \
        --public-access-block-configuration \
        '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

    # Check if the operation was successful.
    if [ $? -eq 0 ]; then
        echo "The block public policy access setting was removed from '$BUCKET_NAME'."
    else
        echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
        exit 1
    fi

    # Create the bucket policy.
    cat << EOF > $POLICY_FILE
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                ],
                "Resource": [
                    "arn:aws:s3:::$BUCKET_NAME",
                    "arn:aws:s3:::$BUCKET_NAME/*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:PrincipalType": "IAMUser"
                    }
                }
            }
        ]
    }
    EOF

    # Apply the bucket policy.
    aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

    # Check if the policy application was successful.
    if [ $? -eq 0 ]; then
        echo "The bucket policy was applied to '$BUCKET_NAME'."
    else
        echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
        exit 1
    fi

    # Verify the applied policy.
    echo "Verifying the applied policy:"
    aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

    # Remove the temporary bucket policy file.
    rm $POLICY_FILE
    ```

4.  Run the script, for example:

    ```bash
    sh create-s3-bucket.sh
    ```

5.  After the bucket is created, you can delete the script file, if you want.


# SharePoint



Ingest your files into Unstructured from SharePoint.

You'll need:

The SharePoint prerequisites, which include the following settings.

<iframe width="560" height="315" src="https://www.youtube.com/embed/r4BLhGgpAWQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

*   The SharePoint site URL.

    *   Site URLs typically have the format `https://<tenant>.sharepoint.com`.
    *   Relative site URLs typically have the format `https://<tenant>.sharepoint.com/sites/<site_name>`.
    *   To process all sites within a tenant, use a site URL of `https://<tenant>-admin.sharepoint.com`. This requires the app to be registered at a tenant level.

    [Learn more](https://learn.microsoft.com/microsoft-365/community/query-string-url-tricks-sharepoint-m365).

*   The path in the SharePoint site from which to start parsing files, for example `"Shared Documents"`. If the connector is to process all sites within the tenant, this filter will be applied to all site document libraries.

*   A SharePoint app principal with its application (client) ID, client secret, and access permissions to the SharePoint instance. [Get a client ID and client secret, and set access permissions](https://github.com/vgrem/Office365-REST-Python-Client/wiki/How-to-connect-to-SharePoint-Online-and-and-SharePoint-2013-2016-2019-on-premises--with-app-principal).

    You can create SharePoint app principals through `https://<tenant>.sharepoint.com/_layouts/15/appregnew.aspx`.

    For a SharePoint app principal with site-scoped permissions, use app permission request XML such as the following to grant
    permissions through `https://<tenant>.sharepoint.com/_layouts/15/appinv.aspx`:

    ```xml
    <AppPermissionRequests AllowAppOnlyPolicy="true">
      <AppPermissionRequest Scope="http://sharepoint/content/sitecollection" Right="FullControl" />
    </AppPermissionRequests>
    ```

    For a SharePoint app principal with tenant-scoped permissions, use app permission request XML such as the following to grant
    permissions through `https://<tenant>-admin.sharepoint.com/_layouts/15/appinv.aspx` instead:

    ```xml
    <AppPermissionRequests AllowAppOnlyPolicy="true">
      <AppPermissionRequest Scope="http://sharepoint/content/tenant" Right="FullControl" />
    </AppPermissionRequests>
    ```

    Available `Right` settings include `Read`, `Write`, `Manage`, and `FullControl`. To learn more, see
    [Add-in permissions in SharePoint](https://learn.microsoft.com/sharepoint/dev/sp-add-ins/add-in-permissions-in-sharepoint).

To create the source connector:

1.  On the sidebar, click **Connectors**.
2.  Click **Sources**.
3.  Click **Add new**.
4.  Give the connector some unique **Name**.
5.  In the **Provider** area, click **SharePoint**.
6.  Click **Continue**.
7.  Follow the on-screen instructions to fill in the fields as described later on this page.
8.  Click **Save and Test**.

Fill in the following fields:

*   **Name** (*required*): A unique name for this connector.
*   **Client ID** (*required*): The client ID provided by SharePoint for the app registration.
*   **Client Credential** (*required*): The client secret associated with the client ID.
*   **Site URL** (*required*): The base URL of the SharePoint site to connect to.
*   **Path** (*required*): The path from which to start parsing files, for example `Shared Documents`.
*   **Recursive**: Check this box to recursively process data from subfolders within the specified path.
*   **Files Only**: Check this box to disregard the folder structure.


# Summarizing



After partitioning and chunking, *summarizing* generates text-based summaries of images and tables.
This summarization is done by using the [GPT-4o](https://openai.com/index/hello-gpt-4o/) or
[Claude 3.5 Sonnet](https://www.anthropic.com/news/claude-3-5-sonnet) models.

Here is an example of the output of image summarization using GPT-4o. Note specifically the `text` field that is added.
Line breaks have been inserted here for readability. The output will not contain these line breaks.

```json
{
    "type": "Image",
    "element_id": "3303aa13098f5a26b9845bd18ee8c881",
    "text": "{\n  \"type\": \"graph\",\n  \"description\": \"The graph shows 
        the relationship between Potential (V) and Current Density (A/cm2). 
        The x-axis is labeled 'Current Density (A/cm2)' and ranges from 
        0.0000001 to 0.1. The y-axis is labeled 'Potential (V)' and ranges 
        from -2.5 to 1.5. There are six different data series represented 
        by different colors: blue (10g), red (4g), green (6g), purple (2g), 
        orange (Control), and light blue (8g). The data points for each series 
        show how the potential changes with varying current density.\"\n}",
    "metadata": {
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 1,
        "image_base64": "/9j...<full results omitted for brevity>...Q==",
        "image_mime_type": "image/jpeg",
        "filename": "7f239e1d4ef3556cc867a4bd321bbc41.pdf",
        "data_source": {}
    }
}
```

Here is an example of the output of table summarization using GPT-4o. Note specifically the `text` field that is added.
Line breaks have been inserted here for readability. The output will not contain these line breaks.

```json
{
    "type": "Table",
    "element_id": "5713c0e90194ac7f0f2c60dd614bd24d",
    "text": "The table consists of 6 rows and 7 columns. The columns represent 
        inhibitor concentration (g), bc (V/dec), ba (V/dec), Ecorr (V), icorr 
        (A/cm\u00b2), polarization resistance (\u03a9), and corrosion rate 
        (mm/year). As the inhibitor concentration increases, the corrosion 
        rate generally decreases, indicating the effectiveness of the 
        inhibitor. Notably, the polarization resistance increases with higher 
        inhibitor concentrations, peaking at 6 grams before slightly 
        decreasing. This suggests that the inhibitor is most effective at 
        6 grams, significantly reducing the corrosion rate and increasing 
        polarization resistance. The data provides valuable insights into the 
        optimal concentration of the inhibitor for corrosion prevention.",
    "metadata": {
        "text_as_html": "<table>...<full results omitted for brevity>...</table>",
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 1,
        "image_base64": "/9j...<full results omitted for brevity>...//Z",
        "image_mime_type": "image/jpeg",
        "filename": "7f239e1d4ef3556cc867a4bd321bbc41.pdf",
        "data_source": {}
    }
}
```

## Summarize images and tables

To summarize images and tables, in the **Enrichment model** section of an **Enrichment** node in a workflow, specify the following:

<Note>You can change a workflow's summarization settings only through [Custom](/platform/workflows#create-a-custom-workflow) workflow settings.</Note>

For image summarization, choose one of the following:

*   **OpenAI Image Description**: Use GPT-4o to summarize images. [Learn more](https://openai.com/index/hello-gpt-4o/).
*   **Anthropic Image Description**: Use Claude 3.5 Sonnet to summarize images. [Learn more](https://www.anthropic.com/news/claude-3-5-sonnet).

For table summarization, choose one of the following:

*   **OpenAI Table Description**: Use GPT-4o to summarize tables. [Learn more](https://openai.com/index/hello-gpt-4o/).
*   **Anthropic Table Description**: Use Claude 3.5 Sonnet to summarize tables. [Learn more](https://www.anthropic.com/news/claude-3-5-sonnet).


# Supported file types



Unstructured supports processing of the following file types:

By file extension:

| File extension |
| -------------- |
| `.bmp`         |
| `.csv`         |
| `.doc`         |
| `.docx`        |
| `.eml`         |
| `.epub`        |
| `.heic`        |
| `.html`        |
| `.jpeg`        |
| `.png`         |
| `.md`          |
| `.msg`         |
| `.odt`         |
| `.org`         |
| `.p7s`         |
| `.pdf`         |
| `.png`         |
| `.ppt`         |
| `.pptx`        |
| `.rst`         |
| `.rtf`         |
| `.tiff`        |
| `.txt`         |
| `.tsv`         |
| `.xls`         |
| `.xlsx`        |
| `.xml`         |

By file type:

| Category          | File types                                |
| ----------------- | ----------------------------------------- |
| CSV               | `.csv`                                    |
| E-mail            | `.eml`, `.msg`, `.p7s`                    |
| EPUB              | `.epub`                                   |
| Excel             | `.xls`, `.xlsx`                           |
| HTML              | `.html`                                   |
| Image             | `.bmp`, `.heic`, `.jpeg`, `.png`, `.tiff` |
| Markdown          | `.md`                                     |
| Org Mode          | `.org`                                    |
| Open Office       | `.odt`                                    |
| PDF               | `.pdf`                                    |
| Plain text        | `.txt`                                    |
| PowerPoint        | `.ppt`, `.pptx`                           |
| reStructured Text | `.rst`                                    |
| Rich Text         | `.rtf`                                    |
| TSV               | `.tsv`                                    |
| Word              | `.doc`, `.docx`                           |
| XML               | `.xml`                                    |


# Workflows



## Workflows dashboard

![Workflows in the sidebar](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Workflows-Sidebar.png)

To view the workflows dashboard, on the sidebar, click **Workflows**.

A workflow in the Unstructured Platform is a defined sequence of processes that automate the data handling from source to destination. It allows users to configure how and when data should be ingested, processed, and stored.

Workflows are crucial for establishing a systematic approach to managing data flows within the platform, ensuring consistency, efficiency, and adherence to specific data processing requirements.

## Create a workflow

![Choose a workflow type](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Choose-Workflow-Type.png)

The Unstructured Platform provides two types of workflow builders:

*   [Automatic](#create-an-automatic-workflow) workflows, which use sensible default workflow settings to enable you to get good-quality results faster.
*   [Custom](#create-a-custom-worklow) workflows, which enable you to fine-tune the workflow settings behind the scenes to get very specific results.

All Unstructured accounts can create automatic worklows.

To create custom workflows, you must request Unstructured to enable your account first. [Learn how](#create-a-custom-worklow).

### Create an automatic workflow

<Warning>
  You must first have an existing source connector and destination connector to add to the workflow.

  If you do not have an existing connector for either your target source (input) or destination (output) location, [create the source connector](/platform/sources/overview), [create the destination connector](/platform/destinations/overview), and then return here.

  To see your existing connectors, on the sidebar, click **Connectors**, and then click **Sources** or **Destinations**.
</Warning>

To create an automatic workflow:

1.  On the sidebar, click **Workflows**.

2.  Click **New Workflow**.

3.  Next to **Build it with me**, click **Create Workflow**.

    <Note>If a radio button appears instead of **Build it with me**, select it, and then click **Continue**.</Note>

4.  For **Workflow Name**, enter some unique name for this workflow.

5.  In the **Sources** dropdown list, select your source location.

6.  In the **Destinations** dropdown list, select your destination location.

    <Note>You can select multiple source and destination locations. Files will be ingested from all of the selected source locations, and the processed data will be delivered to all of the selected destination locations.</Note>

7.  Click **Continue**.

8.  In the **Optimize for** section, select the option to choose one of these predefined workflow settings groups:

    *   **Basic** Ideal for simple, text-only documents.
    *   **Advanced** Best for PDFs, images, and complex file types.
    *   **Platinum** For your most challenging documents, including scanned and handwritten content. It uses vision language models (VLMs).
        During processing, files that are not PDFs or images are processed by using the **Advanced** strategy and are charged at the **Advanced** rate instead.

9.  The **Reprocess all** box applies only to the Amazon S3 and Azure Blob Storage source connectors:

    *   Checking this box reprocesses all documents in the source location on every workflow run.
    *   Unchecking this box causes only new documents that are added to the source location since the last workflow run to be processed on future runs. Previously processed documents are not processed again, even if those documents' contents change.

10. If you want to retry processing any documents that failed to process, check the **Retry Failed Documents** box.

11. Click **Continue**.

12. If you want this workflow to run on a schedule, in the **Repeat Run** dropdown list, select one of the scheduling options, and fill in the scheduling settings. Otherwise, select **Don't repeat**.

13. Click **Complete**.

14. If you did not set the workflow to run on a schedule, you can [run the worklow](#edit-delete-or-run-a-workflow) now.

### Create a custom workflow

<Warning>
  You must first have an existing source connector and destination connector to add to the workflow.

  If you do not have an existing connector for either your target source (input) or destination (output) location, [create the source connector](/platform/sources/overview), [create the destination connector](/platform/destinations/overview), and then return here.

  To see your existing connectors, on the sidebar, click **Connectors**, and then click **Sources** or **Destinations**.
</Warning>

There are two ways to create a custom workflow:

*   Through [Build it with me > Custom](#build-it-with-me-custom). This option enables you to fine-tune the kinds of settings that are in **Basic**, **Advanced**, and **Platinum**.
*   Through [Build it myself](#build-it-myself). This option offers a visual workflow designer with even more fine-tuning than the **Custom** option.

#### Build it with me - Custom

1.  On the sidebar, click **Workflows**.

2.  Click **New Workflow**.

3.  Next to **Build it with me**, click **Create Workflow**.

    <Note>If a radio button appears instead of **Build it with me**, select it, and then click **Continue**.</Note>

4.  For **Workflow Name**, enter some unique name for this workflow.

5.  In the **Sources** dropdown list, select your source location.

6.  In the **Destinations** dropdown list, select your destination location.

    <Note>You can select multiple source and destination locations. Files will be ingested from all of the selected source locations, and the processed data will be delivered to all of the selected destination locations.</Note>

7.  Click **Continue**.

8.  In the **Optimize for** section, click the **Custom** option, and then click **Continue**.

    <Note>
      If the **Custom** option is disabled, inside the **Custom** option click **Notify me**, and follow the on-screen directions to complete the request.
      Unstructured will notify you when your account has been enabled with the **Custom** option. After you receive this notification, click the
      **Custom** option, and then click **Continue**.
    </Note>

9.  In the **Strategy** area, choose one of the following:

    *   **Fast**: Ideal for simple, text-only documents.
    *   **Hi-Res**: Best for PDFs, images, and complex file types.
    *   **VLM**: For your most challenging documents, including scanned and handwritten content. It uses vision language models (VLMs).
        During processing, files that are not PDFs or images are processed by using the **Hi-Res** strategy and are charged at the **Hi-Res** rate instead.
    *   **Auto**: This strategy examines each file before processing it. If the file is an image, or if the file is a PDF and at least one embedded table
        or image is found in it, **Hi-Res** is used to process that file and charged at the **Hi-Res** rate for that file. Otherwise, **Fast** is used and charged at the
        **Fast** rate for that file.

    [Learn more](/platform/partitioning).

10. In the **Image Summzarizer** drop-down list, choose one of the following:

    *   **None**: Do not provide summaries for any detected images in any of the files.
    *   **GPT-4o**: Use GPT-4o to provide summaries for any detected images in any of the files. [Learn more](https://openai.com/index/hello-gpt-4o/).
    *   **Claude 3.5 Sonnet**: Use Claude 3.5 Sonnet to provide summaries for any detected images in any of the files. [Learn more](https://www.anthropic.com/news/claude-3-5-sonnet).

    [Learn more](/platform/summarizing).

11. In the **Table Summzarizer** drop-down list, choose one of the following:

    *   **None**: Do not provide summaries for any detected tables in any of the files.
    *   **GPT-4o**: Use GPT-4o to provide summaries for any detected tables in any of the files. [Learn more](https://openai.com/index/hello-gpt-4o/).
    *   **Claude 3.5 Sonnet**: Use Claude 3.5 Sonnet to provide summaries for any detected tables in any of the files. [Learn more](https://www.anthropic.com/news/claude-3-5-sonnet).

    [Learn more](/platform/summarizing).

12. Check the **Include Page Breaks** box to include page breaks in the output, if the file type support it.

13. Check the **Infer Table Structure** box to extract any detected table elements in PDF files as HTML format into a `metadata` output field named `text_as_html`.

14. In the **Elements to Exclude** drop-down list, select any element types to exclude from the output.

15. In the **Chunk** area, for **Chunker Type**, select one of the following:

    *   **None**: Do not apply special chunking rules to the output.

    *   **Chunk by Character** (also known as *basic* chunking): Combine sequential elements to maximally fill each chunk. Also, specify the following:

        *   **Include Original Elements**: Check this box to output the elements that were used to form a chunk, to appear in the `metadata` field's `orig_elements` field for that chunk. By default, this box is unchecked.
        *   **Max Characters**: Cut off new sections after reaching a length of this many characters. The default is **2048**.
        *   **New After n chars**: Cut off new sections after reaching a length of this many characters. This is an approximate limit. The default is **1500**.
        *   **Overlap**: Apply a prefix of this many trailing characters from the prior text-split chunk to second and later chunks formed from oversized elements by text-splitting. The default is **160**.
        *   **Overlap all**: Check this box to apply overlap to "normal" chunks formed by combining whole elements. Use with caution as this can introduce noise into otherwise clean semantic units. By default, this box is unchecked.

    *   **Chunk by Page**: Preserve page boundaries. When a new page is detected, the existing chunk is closed and a new one is started, even if the next element would fit in the prior chunk. Also, specify the following:

        *   **Include Original Elements**: Check this box to output the elements that were used to form a chunk, to appear in the `metadata` field's `orig_elements` field for that chunk. By default, this box is unchecked.
        *   **Max Characters**: Cut off new sections after reaching a length of this many characters. This is a strict limit. The default is **500**.
        *   **New After n Characters**: Cut off new sections after reaching a length of this many characters. This is an approximate limit. The default is **50**.
        *   **Overlap**: Apply a prefix of this many trailing characters from the prior text-split chunk to second and later chunks formed from oversized elements by text-splitting. The default is **30**.
        *   **Overlap all**: Check this box to apply overlap to "normal" chunks formed by combining whole elements. Use with caution as this can introduce noise into otherwise clean semantic units. By default, this box is unchecked.

    *   **Chunk by Title**: Preserve section boundaries and optionally page boundaries as well. A single chunk will never contain text that occurred in two different sections. When a new section starts, the existing chunk is closed and a new one is started, even if the next element would fit in the prior chunk. Also, specify the following:

        *   **Combine Text Under n Chars**: Combine elements until a section reaches a length of this many characters. The default is **0**.
        *   **Include Original Elements**: Check this box to output the elements that were used to form a chunk, to appear in the `metadata` field's `orig_elements` field for that chunk. By default, this box is unchecked.
        *   **Max Characters**: Cut off new sections after reaching a length of this many characters. This is a strict limit. The default is **2048**.
        *   **Multipage Sections**: Check this box to allow sections to span multiple pages. By default, this box is unchecked.
        *   **New After n Characters**: Cut off new sections after reaching a length of this many characters. This is an approximate limit. The default is \**1500*.
        *   **Overlap**: Apply a prefix of this many trailing characters from the prior text-split chunk to second and later chunks formed from oversized elements by text-splitting. The default is **160**.
        *   **Overlap all**: Check this box to apply overlap to "normal" chunks formed by combining whole elements. Use with caution as this can introduce noise into otherwise clean semantic units. By default, this box is unchecked.

    *   **Chunk by Similarity**: Use the [sentence-transformers/multi-qa-mpnet-base-dot-v1](https://huggingface.co/sentence-transformers/multi-qa-mpnet-base-dot-v1) embedding model to identify topically similar sequential elements and combine them into chunks. Also, specify the following:

        *   **Include Original Elements**: Check this box to output the elements that were used to form a chunk, to appear in the `metadata` field's `orig_elements` field for that chunk. By default, this box is unchecked.
        *   **Max Characters**: Cut off new sections after reaching a length of this many characters. This is a strict limit. The default is **500**.
        *   **Similarity Threshold**: Specify a threshold between 0 and 1 exclusive (0.01 to 0.99 inclusive), where 0 indicates completely dissimilar vectors and 1 indicates identical vectors, taking into consider the trade-offs between precision (a higher threshold) and recall (a lower threshold). The default is **0.5**. [Learn more](https://towardsdatascience.com/introduction-to-embedding-clustering-and-similarity-11dd80b00061).

    Learn more:

    *   [Chunking overview](/platform/chunking)
    *   [Chunking for RAG: best practices](https://unstructured.io/blog/chunking-for-rag-best-practices)

16. In the **Embed** area, for **Provider**, choose one of the following:

    *   **None**: Do not generate embeddings.

    *   **OpenAI**: Use OpenAI to generate embeddings. Also, choose the model to use:

        *   **text-embedding-3-small**, with 1536 dimensions.
        *   **text-embedding-3-large**, with 3072 dimensions.
        *   **Ada 002 (Text)**, with 1536 dimensions.

        [Learn more](https://platform.openai.com/docs/guides/embeddings).

    *   **Vertex AI**: Use Vertex AI to generate embeddings by using the `textembedding-gecko@001` model, with 768 dimensions. [Learn more](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings).

    Learn more:

    *   [Embedding overview](/platform/embedding)
    *   [Understanding embedding models: make an informed choice for your RAG](https://unstructured.io/blog/understanding-embedding-models-make-an-informed-choice-for-your-rag).

17. The **Reprocess all** box applies only to the Amazon S3 and Azure Blob Storage source connectors:

    *   Checking this box reprocesses all documents in the source location on every workflow run.
    *   Unchecking this box causes only new documents that are added to the source location since the last workflow run to be processed on future runs. Previously processed documents are not processed again, even if those documents' contents change.

18. Check the **Retry Failed Documents** box if you want to retry processing any documents that failed to process.

19. Click **Continue**.

20. If you want this workflow to run on a schedule, in the **Repeat Run** dropdown list, select one of the scheduling options, and fill in the scheduling settings. Otherwise, select **Don't repeat**.

21. Click **Complete**.

22. If you did not set the workflow to run on a schedule, you can [run the worklow](#edit-delete-or-run-a-workflow) now.

#### Build it myself

1.  On the sidebar, click **Workflows**.

2.  Click **New Workflow**.

3.  Click the **Build it myself** option, and then click **Continue**.

    <Note>
      If the **Build it myself** option is disabled, inside the **Build it myself** option click **Notify me**, and follow the on-screen directions to complete the request.
      Unstructured will notify you when your account has been enabled with the **Build it myself** option. After you receive this notification, click the
      **Build it myself** option, and then click **Continue**.
    </Note>

4.  In the **This workflow** pane, click the **Details** button.

    ![Workflow details](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Workflow-Details.png)

5.  Next to **Name**, click the pencil icon, enter some unique name for this workflow, and then click the check mark icon.

6.  If you want this workflow to run on a schedule, click the **Schedule** button. In the **Repeat Run** dropdown list, select one of the scheduling options, and fill in the scheduling settings.

7.  To overwrite any previously processed files, or to retry any documents that fail to process, click the **Settings** button, and check either or both of the boxes.

    The **Reprocess all** box applies only to the Amazon S3 and Azure Blob Storage source connectors:

    *   Checking this box reprocesses all documents in the source location on every workflow run.
    *   Unchecking this box causes only new documents that are added to the source location since the last workflow run to be processed on future runs. Previously processed documents are not processed again, even if those documents' contents change.

8.  In the pipeline designer, click the **Source** node. In the **Source** pane, select the source location. Then click **Save**.

    ![Workflow designer](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Workflow-Designer.png)

9.  Click the **Destination** node. In the **Destination** pane, select the destination location. Then click **Save**.

10. As needed, add more nodes by clicking the plus icon (recommended) or **Add Node** button:

    ![Add node to workflow](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Workflow-Add-Node.png)

    *   Click **Connect** to add another **Source** or **Destination** node. You can add multiple source and destination locations. Files will be ingested from all of the source locations, and the processed data will be delivered to all of the destination locations. [Learn more](#custom-workflow-node-types).
    *   Click **Enrich** to add a **Chunker** or **Enrichment** node. [Learn more](#custom-workflow-node-types).
    *   Click **Transform** to add a **Partitioner** or **Embedder** node. [Learn more](#custom-workflow-node-types).

    <Tip>
      Make sure to add nodes in the correct order. If you are unsure, see the usage hints in the blue note that appears
      in the node's settings pane.

      ![Node usage hints note](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Node-Usage-Hints.png)
    </Tip>

    To edit a node, click that node, and then change its settings.

    To delete a node, click that node, and then click the trash can icon above it.

11. Click **Save**.

12. If you did not set the workflow to run on a schedule, you can [run the worklow](#edit-delete-or-run-a-workflow) now.

#### Custom workflow node types

<AccordionGroup>
  <Accordion title="Partitioner node">
    For **Partition Strategy**, choose one of the following:

    *   **Fast**: Ideal for simple, text-only documents.
    *   **Hi-Res**: Best for PDFs, images, and complex file types.
    *   **VLM**: For your most challenging documents, including scanned and handwritten content. It uses vision language models (VLMs).
        During processing, files that are not PDFs or images are processed by using the **Hi-Res** strategy and are charged at the **Hi-Res** rate instead.
    *   **Auto**: This strategy examines each file before processing it. If the file is an image, or if the file is a PDF and at least one embedded table
        or image is found in it, **Hi-Res** is used to process that file and charged at the **Hi-Res** rate for that file. Otherwise, **Fast** is used and charged at the
        **Fast** rate for that file.

    [Learn more](/platform/partitioning).
  </Accordion>

  <Accordion title="Chunker node">
    For **Chunkers**, select one of the following:

    *   **Chunk by title**: Preserve section boundaries and optionally page boundaries as well. A single chunk will never contain text that occurred in two different sections. When a new section starts, the existing chunk is closed and a new one is started, even if the next element would fit in the prior chunk. Also, specify the following:

        *   **Combine text under n chars**: Combine elements until a section reaches a length of this many characters. The default is **0**.
        *   **Include original elements**: Check this box to output the elements that were used to form a chunk, to appear in the `metadata` field's `orig_elements` field for that chunk. By default, this box is unchecked.
        *   **Max characters**: Cut off new sections after reaching a length of this many characters. This is a strict limit. The default is **2048**.
        *   **Multipage sections**: Check this box to allow sections to span multiple pages. By default, this box is unchecked.
        *   **New after n chars**: Cut off new sections after reaching a length of this many characters. This is an approximate limit. The default is \**1500*.
        *   **Overlap**: Apply a prefix of this many trailing characters from the prior text-split chunk to second and later chunks formed from oversized elements by text-splitting. The default is **160**.
        *   **Overlap all**: Check this box to apply overlap to "normal" chunks formed by combining whole elements. Use with caution as this can introduce noise into otherwise clean semantic units. By default, this box is unchecked.

    *   **Chunk by character** (also known as *basic* chunking): Combine sequential elements to maximally fill each chunk. Also, specify the following:

        *   **Include original elements**: Check this box to output the elements that were used to form a chunk, to appear in the `metadata` field's `orig_elements` field for that chunk. By default, this box is unchecked.
        *   **Max characters**: Cut off new sections after reaching a length of this many characters. The default is **2048**.
        *   **New after n chars**: Cut off new sections after reaching a length of this many characters. This is an approximate limit. The default is \**1500*.
        *   **Overlap**: Apply a prefix of this many trailing characters from the prior text-split chunk to second and later chunks formed from oversized elements by text-splitting. The default is **160**.
        *   **Overlap All**: Check this box to apply overlap to "normal" chunks formed by combining whole elements. Use with caution as this can introduce noise into otherwise clean semantic units. By default, this box is unchecked.

    *   **Chunk by page**: Preserve page boundaries. When a new page is detected, the existing chunk is closed and a new one is started, even if the next element would fit in the prior chunk. Also, specify the following:

        *   **Include original elements**: Check this box to output the elements that were used to form a chunk, to appear in the `metadata` field's `orig_elements` field for that chunk. By default, this box is unchecked.
        *   **Max characters**: Cut off new sections after reaching a length of this many characters. This is a strict limit. The default is **500**.
        *   **New after n chars**: Cut off new sections after reaching a length of this many characters. This is an approximate limit. The default is **50**.
        *   **Overlap**: Apply a prefix of this many trailing characters from the prior text-split chunk to second and later chunks formed from oversized elements by text-splitting. The default is **30**.
        *   **Overlap all**: Check this box to apply overlap to "normal" chunks formed by combining whole elements. Use with caution as this can introduce noise into otherwise clean semantic units. By default, this box is unchecked.

    *   **Chunk by similarity**: Use the [sentence-transformers/multi-qa-mpnet-base-dot-v1](https://huggingface.co/sentence-transformers/multi-qa-mpnet-base-dot-v1) embedding model to identify topically similar sequential elements and combine them into chunks. Also, specify the following:

        *   **Include original elements**: Check this box to output the elements that were used to form a chunk, to appear in the `metadata` field's `orig_elements` field for that chunk. By default, this box is unchecked.
        *   **Max characters**: Cut off new sections after reaching a length of this many characters. This is a strict limit. The default is **500**.
        *   **Similarity threshold**: Specify a threshold between 0 and 1 exclusive (0.01 to 0.99 inclusive), where 0 indicates completely dissimilar vectors and 1 indicates identical vectors, taking into consider the trade-offs between precision (a higher threshold) and recall (a lower threshold). The default is **0.5**. [Learn more](https://towardsdatascience.com/introduction-to-embedding-clustering-and-similarity-11dd80b00061).

    Learn more:

    *   [Chunking overview](/platform/chunking)
    *   [Chunking for RAG: best practices](https://unstructured.io/blog/chunking-for-rag-best-practices)
  </Accordion>

  <Accordion title="Enrichment node">
    For **Enrichment model**, choose one of the following:

    *   **OpenAI Image Description** to use GPT-4o to summarize images. [Learn more](https://openai.com/index/hello-gpt-4o/).
    *   **OpenAI Table Description** to use se GPT-4o to summarize tables. [Learn more](https://openai.com/index/hello-gpt-4o/).
    *   **OpenAI Table to HTML** to use GPT-4o to convert tables to HTML. [Learn more](https://openai.com/index/hello-gpt-4o/).
    *   **Anthropic Image Description** to use Claude 3.5 Sonnet to summarize images. [Learn more](https://www.anthropic.com/news/claude-3-5-sonnet).
    *   **Anthropic Table Description** to use Claude 3.5 Sonnet to summarize tables. [Learn more](https://www.anthropic.com/news/claude-3-5-sonnet).

    [Learn more](/platform/summarizing).
  </Accordion>

  <Accordion title="Embedder node">
    For **Providers**, select one of the following:

    *   **OpenAI**: Use OpenAI to generate embeddings. Also, choose the model to use:

        *   **text-embedding-3-small**, with 1536 dimensions.
        *   **text-embedding-3-large**, with 3072 dimensions.
        *   **Ada 002 (Text)**, with 1536 dimensions.

        [Learn more](https://platform.openai.com/docs/guides/embeddings).

    *   **Vertex AI**: Use Vertex AI to generate embeddings by using the `textembedding-gecko@001` model, with 768 dimensions. [Learn more](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings).

    Learn more:

    *   [Embedding overview](/platform/embedding)
    *   [Understanding embedding models: make an informed choice for your RAG](https://unstructured.io/blog/understanding-embedding-models-make-an-informed-choice-for-your-rag).
  </Accordion>
</AccordionGroup>

## Edit, delete, or run a workflow

To run a workflow:

1.  On the sidebar, click **Workflows**.
2.  In the list of workflows, click **Run** in the row for the workflow that you want to run.

For each of the workflows on the **Workflows** list page, the following actions are available by clicking the ellipses (the three dots) in the row for the respective workflow:

*   **Edit via Form**: Changes the existing configuration of your workflow.
*   **Delete**: Removes the workflow from the platform. Use this action cautiously, as it will permanently delete the workflow and its configurations.
*   **Open**: Opens the workflow's settings page.


# Unstructured



<iframe width="560" height="315" src="https://www.youtube.com/embed/b2AcxJDXOLs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

Unstructured provides a platform and tools to ingest and process unstructured documents for Retrieval Augmented Generation (RAG) and model fine-tuning.

This 40-second video demonstrates a simple use case that Unstructured helps solve:

<iframe width="560" height="315" src="https://www.youtube.com/embed/E-tupjji22U" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

### Product offerings

<Icon icon="dash" iconType="duotone" />  <Icon icon="computer" />  [Unstructured Platform](/platform/overview) - No-code UI. Production-ready. Pay as you go.<br /><br />

<Tip>To start using the Unstructured Platform right away, skip ahead to the [quickstart](#quickstart-unstructured-platform).</Tip>

![Partial view of the Unstructured Platform](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Start-Screen-Partial.png)

<Icon icon="dash" iconType="duotone" />  <Icon icon="square-terminal" />  [Unstructured Serverless API services](/api-reference/api-services/overview) - Use scripts or code. Production-ready. Pay as you go. (There is also a non-production, free edition with [limits](/api-reference/api-services/free-api#free-unstructured-api-limitations).)<br /><br />

<Tip>To start using Unstructured Serverless API services right away, skip ahead to the [quickstart](#quickstart-unstructured-serverless-api).</Tip>

![Unstructured Ingest Python library example code](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/api/Ingest-IDE.png)

Learn more about these products:

<Card title="Unstructured Platform" icon="computer" href="/platform/overview">
  <br />No-code user interface, pay-as-you-go platform to get all of your data RAG-ready.<br /><br />
  Data is processed on Unstructured-hosted compute resources.<br /><br />
  [Try the quickstart](#quickstart-unstructured-platform).<br /><br />
  [Learn more](/platform/overview).<br /><br />
  <Icon icon="blog" />  [Read the announcement](https://unstructured.io/blog/introducing-unstructured-platform).
</Card>

<Card title="Unstructured Serverless API services" icon="square-terminal" href="/api-reference/api-services/overview">
  <br />Use scripts or code to call the Unstructured Ingest CLI or Ingest Python library, to get all of your data RAG-ready.<br /><br />
  Unstructured Serverless API services have a [Serverless](api-reference/api-services/saas-api-development-guide) pay-as-you-go edition and a [Free](/api-reference/api-services/free-api_) [limited](/api-reference/api-services/free-api#free-unstructured-api-limitations) edition that process data on Unstructured-hosted compute resources.<br /><br />
  If you need to use compute resources that you host instead, there are also [Azure](/api-reference/api-services/azure) pay-as-you-go and [AWS](/api-reference/api-services/aws) pay-as-you-go editions; these editions process data by using the Unstructured API installed on compute resources hosted in your own Azure or AWS account.<br /><br />
  [Try the quickstart](#quickstart-unstructured-serverless-api).<br /><br />
  [Learn more](/api-reference/api-services/overview).<br /><br />
  <Icon icon="blog" />  [Read the launch announcement](https://unstructured.io/blog/introducing-unstructured-serverless-api).
</Card>

***

### <Icon icon="files" />   Supported file types

Unstructured supports processing of the following file types:

By file extension:

| File extension |
| -------------- |
| `.bmp`         |
| `.csv`         |
| `.doc`         |
| `.docx`        |
| `.eml`         |
| `.epub`        |
| `.heic`        |
| `.html`        |
| `.jpeg`        |
| `.png`         |
| `.md`          |
| `.msg`         |
| `.odt`         |
| `.org`         |
| `.p7s`         |
| `.pdf`         |
| `.png`         |
| `.ppt`         |
| `.pptx`        |
| `.rst`         |
| `.rtf`         |
| `.tiff`        |
| `.txt`         |
| `.tsv`         |
| `.xls`         |
| `.xlsx`        |
| `.xml`         |

By file type:

| Category          | File types                                |
| ----------------- | ----------------------------------------- |
| CSV               | `.csv`                                    |
| E-mail            | `.eml`, `.msg`, `.p7s`                    |
| EPUB              | `.epub`                                   |
| Excel             | `.xls`, `.xlsx`                           |
| HTML              | `.html`                                   |
| Image             | `.bmp`, `.heic`, `.jpeg`, `.png`, `.tiff` |
| Markdown          | `.md`                                     |
| Org Mode          | `.org`                                    |
| Open Office       | `.odt`                                    |
| PDF               | `.pdf`                                    |
| Plain text        | `.txt`                                    |
| PowerPoint        | `.ppt`, `.pptx`                           |
| reStructured Text | `.rst`                                    |
| Rich Text         | `.rtf`                                    |
| TSV               | `.tsv`                                    |
| Word              | `.doc`, `.docx`                           |
| XML               | `.xml`                                    |

***

### <Icon icon="computer" />   Quickstart: Unstructured Platform

<Info>If you want to use your local machine for either your source (input) files, or the destination (output) location for Unstructured to deliver the processed data, you cannot use this quickstart. You must run code on your local machine instead: skip to the [Quickstart: Unstructured Serverless API](#quickstart-unstructured-serverless-api), later in this article.</Info>

This quickstart uses a no-code, point-and-click user interface in your web browser to get all of your data RAG-ready. Data is processed on Unstructured-hosted compute resources.

You will need:

*   A compatible source (input) location in cloud storage that contains your documents for Unstructured to process. [See the list of supported source types](/platform/connectors#sources).
*   Compatible files in your source location. [See the list of supported file types](/platform/supported-file-types). If you do not have any files available, you can download some from the [example-docs](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/example-docs) folder in the Unstructured repo on GitHub.
*   A compatible destination (output) location in cloud storage for Unstructured to put the processed data. [See the list of supported destination types](/platform/connectors#destinations).

<iframe width="560" height="315" src="https://www.youtube.com/embed/25u3qJ_7WiA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen />

<Steps>
  <Step title="Sign in to your Unstructured account">
    ![Sign in to your Unstructured account](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Signin.png)

    1.  Sign in to your Unstructured account, at [https://app.unstructured.io](https://app.unstructured.io).
    2.  Click **Google** or **GitHub** to sign in with your Google or GitHub account. Or, enter your email address and then click **Sign In**.
    3.  If you entered your email address, check your email inbox for a message from Unstructured. In that email, click the **Sign In** link.
    4.  The first time you sign in, read the terms and conditions, and then click **Accept**.
  </Step>

  <Step title="Open your Unstructured Platform dashboard">
    ![Open your Unstructured Platform dashboard](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/GoToPlatform.png)

    *   From your Unstructured account dashboard, in the sidebar, click **API > Platform**.

    <Note>
      The **API > Platform** sidebar option is visible only to users who signed up for an Unstructured API services account before
      October 23, 2024. To get access to the Unstructured Platform on or after this date,
      [request access](https://unstructured.io/platform), and wait for an access enablement email from Unstructured.
    </Note>
  </Step>

  <Step title="Set the source (input) location">
    ![Sources in the sidebar](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Sources-Sidebar.png)

    1.  From your Unstructured Platform dashboard, in the sidebar, click **Connectors**.
    2.  Click **Sources**.
    3.  Click **Add new**.
    4.  For **Name**, enter some unique name for this connector.
    5.  In the **Provider** area, click the source location type that matches yours.
    6.  Click **Continue**.
    7.  Fill in the fields with the appropriate settings. [Learn more](/platform/sources/overview).
    8.  If a **Continue** button appears, click it, and fill in any additional settings fields.
    9.  Click **Save and Test**.
  </Step>

  <Step title="Set the destination (output) location">
    ![Destinations in the sidebar](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Destinations-Sidebar.png)

    1.  In the sidebar, click **Connectors**.
    2.  Click **Destinations**.
    3.  Click **Add new**.
    4.  For **Name**, enter some unique name for this connector.
    5.  In the **Provider** area, click the destination location type that matches yours.
    6.  Click **Continue**.
    7.  Fill in the fields with the appropriate settings. [Learn more](/platform/sources/overview).
    8.  If a **Continue** button appears, click it, and fill in any additional settings fields.
    9.  Click **Save and Test**.
  </Step>

  <Step title="Define the workflow">
    ![Workflows in the sidebar](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Workflows-Sidebar.png)

    1.  In the sidebar, click **Workflows**.

    2.  Click **New Workflow**.

    3.  Next to **Build it with me**, click **Create Workflow**.

        <Note>If a radio button appears instead of **Build it with me**, select it, and then click **Continue**.</Note>

    4.  For **Workflow Name**, enter some unique name for this workflow.

    5.  In the **Sources** dropdown list, select your source location from Step 3.

    6.  In the **Destinations** dropdown list, select your destination location from Step 4.

        <Note>You can select multiple source and destination locations. Files will be ingested from all of the selected source locations, and the processed data will be delivered to all of the selected destination locations.</Note>

    7.  Click **Continue**.

    8.  In the **Optimize for** section, select the option to choose one of these predefined workflow settings groups:

        *   **Basic**: Ideal for simple, text-only documents.
        *   **Advanced**: Best for PDFs, images, and complex file types.
        *   **Platinum**: For your most challenging documents, including scanned and handwritten content. It uses vision language models (VLMs).
            During processing, files that are not PDFs or images are processed by using the **Advanced** strategy and are charged at the **Advanced** rate instead.

    9.  The **Reprocess all** box applies only to the Amazon S3 and Azure Blob Storage source connectors:

        *   Checking this box reprocesses all documents in the source location on every workflow run.
        *   Unchecking this box causes only new documents that are added to the source location since the last workflow run to be processed on future runs. Previously processed documents are not processed again, even if those documents' contents change.

    10. If you want to retry processing any documents that failed to process, check the **Retry Failed Documents** box.

    11. Click **Continue**.

    12. If you want this workflow to run on a schedule, in the **Repeat Run** dropdown list, select one of the scheduling options, and fill in the scheduling settings. Otherwise, select **Don't repeat**.

    13. Click **Complete**.
  </Step>

  <Step title="Process the documents">
    ![Workflows in the sidebar](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Workflows-Sidebar.png)

    1.  If you did not choose to run this workflow on a schedule in Step 5, you can run the workflow now: on the sidebar, click **Workflows**.
    2.  Next to your workflow from Step 5, click **Run**.
  </Step>

  <Step title="Monitor the processing job">
    ![Select a job](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Select-Job.png)
    ![Completed job](https://mintlify.s3-us-west-1.amazonaws.com/unstructured-53/img/platform/Job-Complete.png)

    1.  In the sidebar, click **Jobs**.
    2.  In the list of jobs, wait for the job's **Status** to change to **Finished**.
    3.  Click the row for the job.
    4.  If **Overview** displays **Success**, go to the next Step.
  </Step>

  <Step title="View the processed data">
    Go to your destination location to view the processed data.
  </Step>
</Steps>

[Learn more about the Unstructured Platform](/platform/overview).

***

### <Icon icon="square-terminal" />   Quickstart: Unstructured Serverless API

This quickstart uses your local machine, with the [Unstructured Ingest Python library](/ingestion/overview#unstructured-ingest-python-library) installed. It preprocesses source (input) files on your local machine, and it uses the Unstructured Serverless API to deliver the processed data to a destination (output) location, also on your local machine. Data is processed on Unstructured-hosted compute resources.

You will need:

*   Python installed on your local machine.
*   Compatible files on your local machine to be processed. [See the list of supported file types](#supported-file-types). If you do not have any files available, you can download some from the [example-docs](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/example-docs) folder in the Unstructured repo on GitHub.

<Steps>
  <Step title="Sign up">
    Sign up for a 14-day trial of the Unstructured Serverless API at [https://app.unstructured.io](https://app.unstructured.io).
  </Step>

  <Step title="Sign in">
    1.  Enter your email address and then click **Sign In** to receive a magic link to sign in to your personalized dashboard, or authenticate yourself with your Google or GitHub account.
    2.  Once you authenticate with the magic link or with your Google or GitHub account, your dashboard appears.
  </Step>

  <Step title="Get your API key and API URL">
    1.  On the sidebar, click **API Keys**, if it is not already selected.
    2.  To get your API key, click the copy icon in the **Actions** column for your API key, and then click **Key Only**. Store your copied API key in a secure location. Do not share it with others.
    3.  To get your API URL, click the copy icon next to the URL next to **API URL**. Store your copied API URL in a secure location. Do not share it with others.
  </Step>

  <Step title="Set environment variables">
    1.  Set an environment variable named `UNSTRUCTURED_API_KEY` to the value of your Unstructured API key.
    2.  Set another environment variable named `UNSTRUCTURED_API_URL` to the value of your Unstructured API URL.
  </Step>

  <Step title="Install the Ingest Python library">
    Run the following command:

    ```bash
    pip install unstructured-ingest
    ```

    You might also need to install additional dependencies, depending on your needs. [Learn more](/api-reference/ingest/ingest-dependencies).
  </Step>

  <Step title="Run the code">
    Run the following code, replacing:

    *   `<path/to/input>` with the source (input) path to the directory on your local machine that contains the compatible files for Unstructured to process on its hosted compute resources.
    *   `<path/to/output>` with the destination (output) path to the directory on your local machine that will contain the processed data that Unstructured returns from its hosted compute resources.

    ```python Python Ingest v2
    import os

    from unstructured_ingest.v2.pipeline.pipeline import Pipeline
    from unstructured_ingest.v2.interfaces import ProcessorConfig
    from unstructured_ingest.v2.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.v2.processes.partitioner import PartitionerConfig

    if __name__ == "__main__":
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                partition_by_api=True,
                api_key=os.getenv("UNSTRUCTURED_API_KEY"),
                partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
                strategy="hi_res",
                additional_partition_args={
                    "split_pdf_page": True,
                    "split_pdf_allow_failed": True,
                    "split_pdf_concurrency_level": 15
                }
            ),
            uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
        ).run()
    ```
  </Step>

  <Step title="View the processed data">
    Go to your destination location to view the processed data.
  </Step>
</Steps>

[Learn more about the Unstructured Serverless API](/api-reference/api-services/overview).

***

## Get in touch

If you can't find the information you're looking for in the documentation, or if you need help,
get in touch with our Support team at [support@unstructured.io](mailto:support%40unstructured.io),
or [join our Slack](https://short.unstructured.io/pzw05l7) where our team and community can help you.