Package: unstructured
Open-Source Pre-Processing Tools for Unstructured DataTheunstructuredlibrary provides open-source components for ingesting and pre-processing images and text documents, such as PDFs, HTML, Word docs, andmany more. The use cases ofunstructuredrevolve around streamlining and optimizing the data processing workflow for LLMs.unstructuredmodular functions and connectors form a cohesive system that simplifies data ingestion and pre-processing, making it adaptable to different platforms and efficient in transforming unstructured data into structured outputs.API Announcement!We are thrilled to announce our newly launchedUnstructured API, providing the Unstructured capabilities fromunstructuredas an API. Check out theunstructured-apiGitHub repositoryto start making API calls. You’ll also find instructions about how to host your own API version.While access to the hosted Unstructured API will remain free, API Keys are required to make requests. To prevent disruption, get yourshereand start using it today! Check out theunstructured-apiREADMEto start making API calls.:rocket: Beta Feature: Chipper ModelWe are releasing the beta version of our Chipper model to deliver superior performance when processing high-resolution, complex documents. To start using the Chipper model in your API request, you can utilize thehi_res_model_name=chipperparameter. Please refer to the documentationhere.As the Chipper model is in beta version, we welcome feedback and suggestions. For those interested in testing the Chipper model, we encourage you to connect with us onSlack community.:eight_pointed_black_star: Quick StartThere are several ways to use theunstructuredlibrary:Run the library in a containerorInstall the libraryInstall from PyPIInstall for local developmentFor installation withcondaon Windows system, please refer to thedocumentationRun the library in a containerThe following instructions are intended to help you get up and running using Docker to interact withunstructured.
Seehereif you don't already have docker installed on your machine.NOTE: we build multi-platform images to support both x86_64 and Apple silicon hardware.docker pullshould download the corresponding image for your architecture, but you can specify with--platform(e.g.--platform linux/amd64) if needed.We build Docker images for all pushes tomain. We tag each image with the corresponding short commit hash (e.g.fbc7a69) and the application version (e.g.0.5.5-dev1). We also tag the most recent image withlatest. To leverage this,docker pullfrom our image repository.dockerpulldownloads.unstructured.io/unstructured-io/unstructured:latestOnce pulled, you can create a container from this image and shell to it.# create the containerdockerrun-dt--nameunstructureddownloads.unstructured.io/unstructured-io/unstructured:latest# this will drop you into a bash shell where the Docker image is runningdockerexec-itunstructuredbashYou can also build your own Docker image.If you only plan on parsing one type of data you can speed up building the image by commenting out some
of the packages/requirements necessary for other data types. See Dockerfile to know which lines are necessary
for your use case.makedocker-build# this will drop you into a bash shell where the Docker image is runningmakedocker-start-bashOnce in the running container, you can try things directly in Python interpreter's interactive mode.# this will drop you into a python console so you can run the below partition functionspython3

>>>fromunstructured.partition.pdfimportpartition_pdf
>>>elements=partition_pdf(filename="example-docs/layout-parser-paper-fast.pdf")>>>fromunstructured.partition.textimportpartition_text
>>>elements=partition_text(filename="example-docs/fake-text.txt")Installing the libraryUse the following instructions to get up and running withunstructuredand test your
installation.Install the Python SDK to support all document types withpip install "unstructured[all-docs]"For plain text files, HTML, XML, JSON and Emails that do not require any extra dependencies, you can runpip install unstructuredTo process other doc types, you can install the extras required for those documents, such aspip install "unstructured[docx,pptx]"Install the following system dependencies if they are not already available on your system.
Depending on what document types you're parsing, you may not need all of these.libmagic-dev(filetype detection)poppler-utils(images and PDFs)tesseract-ocr(images and PDFs, installtesseract-langfor additional language support)libreoffice(MS Office docs)pandoc(EPUBs, RTFs and Open Office docs)For suggestions on how to install on the Windows and to learn about dependencies for other features, see the
installation documentationhere.At this point, you should be able to run the following code:fromunstructured.partition.autoimportpartitionelements=partition(filename="example-docs/eml/fake-email.eml")print("\n\n".join([str(el)forelinelements]))Installation Instructions for Local DevelopmentThe following instructions are intended to help you get up and running withunstructuredlocally if you are planning to contribute to the project.Usingpyenvto manage virtualenv's is recommended but not necessaryMac install instructions. Seeherefor more detailed instructions.brew install pyenv-virtualenvpyenv install 3.10Linux instructions are availablehere.Create a virtualenv to work in and activate it, e.g. for one namedunstructured:pyenv  virtualenv 3.10 unstructuredpyenv activate unstructuredRunmake installOptional:To install models and dependencies for processing images and PDFs locally, runmake install-local-inference.For processing image files,tesseractis required. Seeherefor installation instructions.For processing PDF files,tesseractandpopplerare required. Thepdf2image docshave instructions on installingpoppleracross various platforms.Additionally, if you're planning to contribute tounstructured, we provide you an optionalpre-commitconfiguration
file to ensure your code matches the formatting and linting standards used inunstructured.
If you'd prefer not to have code changes auto-tidied before every commit, you can usemake checkto see
whether any linting or formatting changes should be applied, andmake tidyto apply them.If using the optionalpre-commit, you'll just need to install the hooks withpre-commit installsince thepre-commitpackage is installed as part ofmake installmentioned above. Finally, if you decided to usepre-commityou can also uninstall the hooks withpre-commit uninstall.In addition to develop in your local OS we also provide a helper to use docker providing a development environment:makedocker-start-devThis starts a docker container with your local repo mounted to/mnt/local_unstructured. This docker image allows you to develop without worrying about your OS's compatibility with the repo and its dependencies.:clap: Quick TourDocumentationThis README overviews how to install, use and develop the library. For more comprehensive documentation, visithttps://unstructured-io.github.io/unstructured/.Concepts GuideTheunstructuredlibrary includes core functionality for partitioning, chunking, cleaning, and
staging raw documents for NLP tasks.
You can see a complete list of available functions and how to use them from theCore Functionality documentation.In general, these functions fall into several categories:Partitioningfunctions break raw documents into standard, structured elements.Cleaningfunctions remove unwanted text from documents, such as boilerplate and sentence fragments.Stagingfunctions format data for downstream tasks, such as ML inference and data labeling.Chunkingfunctions split documents into smaller sections for use in RAG apps and similarity
search.Embeddingencoder classes provide an interfaces for easily converting preprocessed text to
vectors.TheConnectors🔗 inunstructuredserve as vital links between the pre-processing pipeline and various data storage platforms. They allow for the batch processing of documents across various sources, including cloud services, repositories, and local directories. Each connector is tailored to a specific platform, such as Azure, Google Drive, or Github, and comes with unique commands and dependencies. To see the list of Connectors available inunstructuredlibrary, please check out theConnectors GitHub folderanddocumentationPDF Document Parsing ExampleThe following examples show how to get started with theunstructuredlibrary. You can parse over a dozen document types with one line of code! Use thisColab notebookto run the example below.The easiest way to parse a document in unstructured is to use thepartitionfunction. If you usepartitionfunction,unstructuredwill detect the file type and route it to the appropriate file-specific partitioning function. If you are using thepartitionfunction, you may need to install additional parameters viapip install unstructured[local-inference]. Ensure you first installlibmagicusing the instructions outlinedherepartitionwill always apply the default arguments. If you need advanced features, use a document-specific partitioning function.fromunstructured.partition.autoimportpartitionelements=partition("example-docs/layout-parser-paper.pdf")Runprint("\n\n".join([str(el) for el in elements]))to get a string representation of the
output, which looks like:LayoutParser : A Uniﬁed Toolkit for Deep Learning Based Document Image Analysis

Zejiang Shen 1 ( (cid:0) ), Ruochen Zhang 2 , Melissa Dell 3 , Benjamin Charles Germain Lee 4 , Jacob Carlson 3 , and
Weining Li 5

Abstract. Recent advances in document image analysis (DIA) have been primarily driven by the application of neural
networks. Ideally, research outcomes could be easily deployed in production and extended for further investigation.
However, various factors like loosely organized codebases and sophisticated model conﬁgurations complicate the easy
reuse of important innovations by a wide audience. Though there have been ongoing eﬀorts to improve reusability and
simplify deep learning (DL) model development in disciplines like natural language processing and computer vision, none
of them are optimized for challenges in the domain of DIA. This represents a major gap in the existing toolkit, as DIA
is central to academic research across a wide range of disciplines in the social sciences and humanities. This paper
introduces LayoutParser, an open-source library for streamlining the usage of DL in DIA research and applications.
The core LayoutParser library comes with a set of simple and intuitive interfaces for applying and customizing DL models
for layout detection, character recognition, and many other document processing tasks. To promote extensibility,
LayoutParser also incorporates a community platform for sharing both pre-trained models and full document digitization
pipelines. We demonstrate that LayoutParser is helpful for both lightweight and large-scale digitization pipelines in
real-word use cases. The library is publicly available at https://layout-parser.github.io

Keywords: Document Image Analysis · Deep Learning · Layout Analysis · Character Recognition · Open Source library ·
Toolkit.

Introduction

Deep Learning(DL)-based approaches are the state-of-the-art for a wide range of document image analysis (DIA) tasks
including document image classiﬁcation [11,See thepartitioningsection in our documentation for a full list of options and instructions on how to use
file-specific partitioning functions.:guardsman: Security PolicySee oursecurity policyfor
information on how to report security vulnerabilities.:bug: Reporting BugsEncountered a bug? Please create a newGitHub issueand use our bug report template to describe the problem. To help us diagnose the issue, use thepython scripts/collect_env.pycommand to gather your system's environment information and include it in your report. Your assistance helps us continuously improve our software - thank you!:books: Learn moreSectionDescriptionCompany WebsiteUnstructured.io product and company infoDocumentationFull API documentationBatch ProcessingIngesting batches of documents through Unstructured:chart_with_upwards_trend: AnalyticsWe’ve partnered with Scarf (https://scarf.sh) to collect anonymized user statistics to understand which features our community is using and how to prioritize product decision-making in the future. To learn more about how we collect and use this data, please read ourPrivacy Policy.
To opt out of this data collection, you can set the environment variableSCARF_NO_ANALYTICS=truebefore running anyunstructuredcommands.