Package: unstructured-api-tools
Open-Source Pre-Processing Tools for Unstructured DataTheunstructured_api_toolslibrary includes utilities for converting pipeline notebooks into
REST API applications.unstructured_api_toolsis intended for use in conjunction with
pipeline repos. Seepipeline-sec-filingsfor an example of a repo that usesunstructured_api_tools.InstallationTo install the library, runpip install unstructured_api_tools.Developer Quick StartUsingpyenvto manage virtualenv's is recommendedMac install instructions. Seeherefor more detailed instructions.brew install pyenv-virtualenvpyenv install 3.8.15Linux instructions are availablehere.Create a virtualenv to work in and activate it, e.g. for one namedunstructured_api_tools:pyenv  virtualenv 3.8.15 unstructured_api_toolspyenv activate unstructured_api_toolsRunmake install-project-localUsageUse the CLI command to convert pipeline notebooks to scripts, for example:unstructured_api_toolsconvert-pipeline-notebooks\--input-directorypipeline-family-sec-filings/pipeline-notebooks\--output-directorypipeline-family-sec-filings/prepline_sec_filings/api\--pipeline-familysec-filings\--semver0.2.1If you do not provide thepipeline-familyandsemverarguments, those values are parsed frompreprocessing-pipeline-family.yaml. You can provide thepreprocessing-pipeline-family.yamlfile
explicitly with--config-filenameor thePIPELINE_FAMILY_CONFIGenvironment variable. If neither
of those is specified, the fallback is to use thepreprocessing-pipeline-family.yamlfile in the
current working directory.The API file undergoesblack,flake8andmypychecks after being generated. If you wantflake8to ignore specific errors, you can specify them through the CLI with--flake8-ignore F401, E402.
See theflake8docsfor a full list of error codes.Conversion frompipeline_apito FastAPIThe command described inUsagegenerates a FastAPI API route for eachpipeline_apifunction defined in the notebook. The signature of thepipeline_apimethod determines what
parameters the generated FastAPI accepts.Currently, only plain text file uploads are supported and as such the first argument must always betext, but support for multiple files and binary files is coming soon!In addition, any number of string array parameters may be specified. Any kwarg beginning withm_indicates a multi-value string parameter that is accepted by the FastAPI API.For example, in a notebook containing:def pipeline_api(text, m_subject=[], m_name=[]):textrepresents the content of a file posted to the FastAPI API, and them_subjectandm_namekeyword args represent optional parameters that may be posted to the API as well, both allowing
multiple string parameters. Acurlrequest against such an API could look like this:curl -X 'POST' \
  'https://<hostname>/<pipeline-family-name>/<pipeline-family-version>/<api-name>' \
  -H 'accept: application/json'  \
  -H 'Content-Type: multipart/form-data' \
  -F 'file=@file-to-process.txt' \
  -F 'subject=art' \
  -F 'subject=history'
  -F 'subject=math' \
  -F 'name=feynman'In addition, you can specify the response type ifpipeline_apican support both "application/json"
and "text/csv" as return types.For example, in a notebook containing a kwargresponse_type:def pipeline_api(text, response_type="text/csv", m_subject=[], m_name=[]):The consumer of the API may then specify "text/csv" as the requested response content type with the usual
HTTP Accept header, e.g.Accept: application/jsonorAccept: text/csv.Security PolicySee oursecurity policyfor
information on how to report security vulnerabilities.Learn moreSectionDescriptionCompany WebsiteUnstructured.io product and company info