Package: pypreprocess
Preprocess SDKPreprocessis an API service that splits any kind of document into optimal chunks of text for use in language model tasks.
Given documents in inputPreprocesssplits them into chunks of text that respect the layout and semantics of the original document.
We split the content by taking into account sections, paragraphs, lists, images, data tables, text tables, and slides, and following the content semantics for long texts.
We support:PDFsMicrosoft Office documents (Word, PowerPoint, Excel)OpenOffice documents (ods, odt, odp)HTML content (web pages, articles, emails)Plain textInstallationInstall the PythonPreprocesslibrary if it is not already present:pipinstallpreprocessAlternatively, if you want to add it as a dependency with poetry:poetryaddpypreprocess
poetryinstallYou need aPreprocess API Keyto use the SDK, to get one please reach out tosupport@preprocess.coasking for an API key.Getting startedGet the chunks from a file for use in your language model tasks.frompypreprocessimportPreprocess#init the SDK with a filepreprocess=Preprocess(api_key=YOUR_API_KEY,filepath="path/for/file")#chunk the filepreprocess.chunk()preprocess.wait()#get the resultresult=preprocess.result()foriinresult.data['chunks']:#use chunksInitialise a connectionYou can initialize the SDK in 3 ways.1- Passing a localfilepathwhen you want to init the SDK to chunk a local filefrompypreprocessimportPreprocesspreprocess=Preprocess(api_key=YOUR_API_KEY,filepath="path/for/file")2- Passing aprocess_idwhen the chunking process startsPreprocessgenerate a process_id, you can use it to instance the SDKfrompypreprocessimportPreprocesspreprocess=Preprocess(api_key=YOUR_API_KEY,process_id="id_of_the_process")3- Passing aPreprocessResponseObjectWhen you need to store the result of a chunking process permanently, you can later load them in the SDK via a PreprocessResponse objectimportjsonfrompypreprocessimportPreprocess,ProcessResponseresponse=PreprocessResponse(**json.loads("The JSON result from calling chunk before."))preprocess=Preprocess(api_key=YOUR_API_KEY,process=PreprocessResponse)Chunking optionsWe support a few options you can configure to get the best result for your ingestion pipeline.Preprocess tries to output chunks of less than 512 tokens. Sometimes longer chunks can be produced to preserve the content integrity. We are currently working to allow you to set an arbitrary chunk token length, stay tuned.ParameterTypeDefaultDescriptionmergeboolFalseIfTruesmall paragraphs will be merged to maximize chunk lengthrepeat_titleboolFalseIfTrueeach chunk will start with the title of the section in which is containedrepeat_table_headerboolFalseIfTrueeach chunk will start with the header of the table in which is containedtable_output_formatenum ['text', 'markdown', 'html']'text'Return tables in the format you need for your ingestion pipelinesYou can pass each parameter from those during the SDK initializationpreprocess=Preprocess(api_key=YOUR_API_KEY,filepath="path/for/file",merge=True,repeat_title=True,...)Or set them later with theset_optionsmethod using adictpreprocess.set_options({"merge":True,"repeat_title":True,...})ChunkAfter initing the SDK with afilepathyou should call thechunk()method.frompypreprocessimportPreprocesspreprocess=Preprocess(api_key=YOUR_API_KEY,filepath="path/for/file")response=preprocess.chunk()the response contains theprocess_idand details about the success of the API call.Getting the resultsThe conversion and chunking process may take a while.
You can use the built-inwait()method to wait and get the result as the process finishes.result=preprocess.wait()print(result.data['chunks'])In a more complex scenario, you can store theprocess_idafter initiating the chunking process and then use it in a different flow.#initing the chunking processfrompypreprocessimportPreprocesspreprocess=Preprocess(api_key=YOUR_API_KEY,filepath="path/for/file")preprocess.chunk()preocess_id=preprocess.get_process_id()#in a different flowfrompypreprocessimportPreprocesspreprocess=Preprocess(api_key=YOUR_API_KEY,process_id=process_id)result=preprocess.wait()print(result.data['chunks'])If you want to implement a different logic for getting the result you can use the methodresult()and check if the status if it'sFINISHED.result=preprocess.result()ifresult.data['process']['status']=="FINISHED":print(result.data['chunks'])Other methodsHere is a list of some methods inside the SDK that may help youset_filepath(path)setting the file path after initializing the object.set_process_id(id)setting the process_id param by id.set_process(PreprocessResponse)setting the process_id param byPreprocessResponseobject.set_options(dict)set the parameters for configuring chunking options.to_json()returning a JSON str representing the current object.get_process_id()returning the current process_id.get_filepath()returning the filepath.get_options()returning the current chunking options.