Package: llama-index-readers-preprocess
Preprocess LoaderPreprocessis an API service that splits any kind of document into optimal chunks of text for use in language model tasks.
Given documents in inputPreprocesssplits them into chunks of text that respect the layout and semantics of the original document.
We split the content by taking into account sections, paragraphs, lists, images, data tables, text tables, and slides, and following the content semantics for long texts.
We support PDFs, Microsoft Office documents (Word, PowerPoint, Excel), OpenOffice documents (ods, odt, odp), HTML content (web pages, articles, emails), and plain text.This loader integrates with thePreprocessAPI library to provide document conversion and chunking or to load already chunked files inside LlamaIndex.RequirementsInstall the PythonPreprocesslibrary if it is not already present:pip install pypreprocessUsageTo use this loader, you need to pass thePreprocess API Key.
When initializingPreprocessReader, you should pass yourAPI Key, if you don't have it yet, please ask for one atsupport@preprocess.co. Without anAPI Key, the loader will raise an error.To chunk a file pass a valid filepath and the reader will start converting and chunking it.Preprocesswill chunk your files by applying an internalSplitter. For this reason, you should not parse the document into nodes using aSplitteror applying aSplitterwhile transforming documents in yourIngestionPipeline.If you want to handle the nodes directly:fromllama_indeximportVectorStoreIndexfromllama_indeximportdownload_loaderPreprocessReader=download_loader("PreprocessReader")# pass a filepath and get the chunks as nodesloader=PreprocessReader(api_key="your-api-key",filepath="valid/path/to/file")nodes=loader.get_nodes()# import the nodes in a Vector Store with your configurationindex=VectorStoreIndex(nodes)query_engine=index.as_query_engine()By default load_data() returns a document for each chunk, remember to not apply any splitting to these documentsfromllama_indeximportVectorStoreIndexfromllama_indeximportdownload_loaderPreprocessReader=download_loader("PreprocessReader")# pass a filepath and get the chunks as nodesloader=PreprocessReader(api_key="your-api-key",filepath="valid/path/to/file")documents=loader.load_data()# don't apply any Splitter parser to documents# if you have an ingestion pipeline you should not apply a Splitter in the transformations# import the documents in a Vector Store, if you set the service_context parameter remember to avoid including a splitterindex=VectorStoreIndex.from_documents(documents)query_engine=index.as_query_engine()If you want to return only the extracted text and handle it with custom pipelines setreturn_whole_document = True# pass a filepath and get the chunks as nodesloader=PreprocessReader(api_key="your-api-key",filepath="valid/path/to/file")document=loader.load_data(return_whole_document=True)If you want to load already chunked files you can do it viaprocess_idpassing it to the reader.# pass a process_id obtained from a previous instance and get the chunks as one string inside a Documentloader=PreprocessReader(api_key="your-api-key",process_id="your-process-id")This loader is designed to be used as a way to load data intoLlamaIndexand/or subsequently used as a Tool in aLangChainAgent. Seeherefor examples.Other infoPreprocessReaderis based onpypreprocessfromPreprocesslibrary.
For more information or other integration needs please check thedocumentation.