Package: llmsherpa
LLM SherpaLLM Sherpa provides strategic APIs to accelerate large language model (LLM) use cases.What's New[!IMPORTANT]
llmsherpa back end service is now fully open sourced under Apache 2.0 Licence. Seehttps://github.com/nlmatics/nlm-ingestorYou can now run your own servers using a docker image!Support for different file formats: DOCX, PPTX, HTML, TXT, XMLOCR Support is built inBlocks now have co-ordinates - use bbox propery of blocks such as sectionsA new indent parser to better align all headings in a document to their corresponding levelThe free server and paid server are not updated with latest code and users are requested to spawn their own servers using instructions in nlm-ingestorLayoutPDFReaderMost PDF to text parsers do not provide layout information. Often times, even the sentences are split with arbritrary CR/LFs making it very difficult to find paragraph boundaries. This poses various challenges in chunking and adding long running contextual information such as section header to the passages while indexing/vectorizing PDFs for LLM applications such as retrieval augmented generation (RAG).LayoutPDFReader solves this problem by parsing PDFs along with hierarchical layout information such as:Sections and subsections along with their levels.Paragraphs - combines lines.Links between sections and paragraphs.Tables along with the section the tables are found in.Lists and nested lists.Join content spread across pages.Removal of repeating headers and footers.Watermark removal.With LayoutPDFReader, developers can find optimal chunks of text to vectorize, and a solution for limited context window sizes of LLMs.You can experiment with the library directly inGoogle ColabhereHere's awriteupexplaining the problem and our approach.Here'a LlamaIndexblogexplaining the need for smart chunking.API Reference:https://llmsherpa.readthedocs.io/How to use with Google Gemini ProHow to use with Cohere Embed3Important NotesThe LayoutPDFReader is tested on a wide variety of PDFs. That being said, it is still challenging to get every PDF parsed correctly.OCR is currently not supported. Only PDFs with a text layer are supported.[!NOTE]
LLMSherpa uses a free and open api server. The server does not store your PDFs except for temporary storage during parsing. This server will be decommissioned soon.
Self-host your own private server using instructions athttps://github.com/nlmatics/nlm-ingestor[!IMPORTANT]
Private available atMicrosoft Azure Marketplacewill be decommissioned soon. Please move to your self-hosted instance using instructions athttps://github.com/nlmatics/nlm-ingestor.InstallationpipinstallllmsherpaRead a PDF fileThe first step in using the LayoutPDFReader is to provide a url or file path to it and get back a document object.fromllmsherpa.readersimportLayoutPDFReaderllmsherpa_api_url="https://readers.llmsherpa.com/api/document/developer/parseDocument?renderFormat=all"pdf_url="https://arxiv.org/pdf/1910.13461.pdf"# also allowed is a file path e.g. /home/downloads/xyz.pdfpdf_reader=LayoutPDFReader(llmsherpa_api_url)doc=pdf_reader.read_pdf(pdf_url)Install LlamaIndexIn the following examples, we will useLlamaIndexfor simplicity. Install the library if you haven't already.pipinstallllama-indexSetup OpenAIimportopenaiopenai.api_key=#<Insert API Key>Vector search and Retrieval Augmented Generation with Smart ChunkingLayoutPDFReader does smart chunking keeping related text due to document structure together:All list items are together including the paragraph that precedes the list.Items in a table are chuncked togetherContextual information from section headers and nested section headers is includedThe following code creates a LlamaIndex query engine from LayoutPDFReader document chunksfromllama_index.readers.schema.baseimportDocumentfromllama_indeximportVectorStoreIndexindex=VectorStoreIndex([])forchunkindoc.chunks():index.insert(Document(text=chunk.to_context_text(),extra_info={}))query_engine=index.as_query_engine()Let's run one query:response=query_engine.query("list all the tasks that work with bart")print(response)We get the following response:BART works well for text generation, comprehension tasks, abstractive dialogue, question answering, and summarization tasks.Let's try another query that needs answer from a table:response=query_engine.query("what is the bart performance score on squad")print(response)Here's the response we get:The BART performance score on SQuAD is 88.8 for EM and 94.6 for F1.Summarize a Section using promptsLayoutPDFReader offers powerful ways to pick sections and subsections from a large document and use LLMs to extract insights from a section.The following code looks for the Fine-tuning section of the document:fromIPython.core.displayimportdisplay,HTMLselected_section=None# find a section in the document by titleforsectionindoc.sections():ifsection.title=='3 Fine-tuning BART':selected_section=sectionbreak# use include_children=True and recurse=True to fully expand the section.# include_children only returns at one sublevel of children whereas recurse goes through all the descendantsHTML(section.to_html(include_children=True,recurse=True))Running the above code yields the following HTML output:3 Fine-tuning BARTThe representations produced by BART can be used in several ways for downstream applications.3.1 Sequence Classiﬁcation TasksFor sequence classiﬁcation tasks, the same input is fed into the encoder and decoder, and the ﬁnal hidden state of the ﬁnal decoder token is fed into new multi-class linear classiﬁer.\nThis approach is related to the CLS token in BERT; however we add the additional token to the end so that representation for the token in the decoder can attend to decoder states from the complete input (Figure 3a).3.2 Token Classiﬁcation TasksFor token classiﬁcation tasks, such as answer endpoint classiﬁcation for SQuAD, we feed the complete document into the encoder and decoder, and use the top hidden state of the decoder as a representation for each word.\nThis representation is used to classify the token.3.3 Sequence Generation TasksBecause BART has an autoregressive decoder, it can be directly ﬁne tuned for sequence generation tasks such as abstractive question answering and summarization.\nIn both of these tasks, information is copied from the input but manipulated, which is closely related to the denoising pre-training objective.\nHere, the encoder input is the input sequence, and the decoder generates outputs autoregressively.3.4 Machine TranslationWe also explore using BART to improve machine translation decoders for translating into English.\nPrevious work Edunov et al.\n(2019) has shown that models can be improved by incorporating pre-trained encoders, but gains from using pre-trained language models in decoders have been limited.\nWe show that it is possible to use the entire BART model (both encoder and decoder) as a single pretrained decoder for machine translation, by adding a new set of encoder parameters that are learned from bitext (see Figure 3b).More precisely, we replace BART’s encoder embedding layer with a new randomly initialized encoder.\nThe model is trained end-to-end, which trains the new encoder to map foreign words into an input that BART can de-noise to English.\nThe new encoder can use a separate vocabulary from the original BART model.We train the source encoder in two steps, in both cases backpropagating the cross-entropy loss from the output of the BART model.\nIn the ﬁrst step, we freeze most of BART parameters and only update the randomly initialized source encoder, the BART positional embeddings, and the self-attention input projection matrix of BART’s encoder ﬁrst layer.\nIn the second step, we train all model parameters for a small number of iterations.Now, let's create a custom summary of this text using a prompt:fromllama_index.llmsimportOpenAIcontext=selected_section.to_html(include_children=True,recurse=True)question="list all the tasks discussed and one line about each task"resp=OpenAI().complete(f"read this text and answer question:{question}:\n{context}")print(resp.text)The above code results in following output:Tasks discussed in the text:

1. Sequence Classification Tasks: The same input is fed into the encoder and decoder, and the final hidden state of the final decoder token is used for multi-class linear classification.
2. Token Classification Tasks: The complete document is fed into the encoder and decoder, and the top hidden state of the decoder is used as a representation for each word for token classification.
3. Sequence Generation Tasks: BART can be fine-tuned for tasks like abstractive question answering and summarization, where the encoder input is the input sequence and the decoder generates outputs autoregressively.
4. Machine Translation: BART can be used to improve machine translation decoders by incorporating pre-trained encoders and using the entire BART model as a single pretrained decoder. The new encoder parameters are learned from bitext.Analyze a Table using promptsWith LayoutPDFReader, you can iterate through all the tables in a document and use the power of LLMs to analyze a Table
Let's look at the 6th table in this document. If you are using a notebook, you can display the table as follows:fromIPython.core.displayimportdisplay,HTMLHTML(doc.tables()[5].to_html())The output table structure looks like this:SQuAD 1.1 EM/F1SQuAD 2.0 EM/F1MNLI m/mmSST AccQQP AccQNLI AccSTS-B AccRTE AccMRPC AccCoLA MccBERT84.1/90.979.0/81.886.6/-93.291.392.390.070.488.060.6UniLM-/-80.5/83.487.0/85.994.5-92.7-70.9-61.1XLNet89.0/94.586.1/88.889.8/-95.691.893.991.883.889.263.6RoBERTa88.9/94.686.5/89.490.2/90.296.492.294.792.486.690.968.0BART88.8/94.686.1/89.289.9/90.196.692.594.991.287.090.462.8Now let's ask a question to analyze this table:fromllama_index.llmsimportOpenAIcontext=doc.tables()[5].to_html()resp=OpenAI().complete(f"read this table and answer question: which model has the best performance on squad 2.0:\n{context}")print(resp.text)The above question will result in the following output:The model with the best performance on SQuAD 2.0 is RoBERTa, with an EM/F1 score of 86.5/89.4.That's it! LayoutPDFReader also supports tables with nested headers and header rows.Here's an example with nested headers:from IPython.core.display import display, HTML
HTML(doc.tables()[6].to_html())CNN/DailyMailXSum-R1R2RLR1R2RL---------------------Lead-340.4217.6236.6716.301.6011.95PTGEN (See et al., 2017)36.4415.6633.4229.709.2123.24PTGEN+COV (See et al., 2017)39.5317.2836.3828.108.0221.72UniLM43.3320.2140.51---BERTSUMABS (Liu & Lapata, 2019)41.7219.3938.7638.7616.3331.15BERTSUMEXTABS (Liu & Lapata, 2019)42.1319.6039.1838.8116.5031.27BART44.1621.2840.9045.1422.2737.25Now let's ask an interesting question:fromllama_index.llmsimportOpenAIcontext=doc.tables()[6].to_html()question="tell me about R1 of bart for different datasets"resp=OpenAI().complete(f"read this table and answer question:{question}:\n{context}")print(resp.text)And we get the following answer:R1 of BART for different datasets:

- For the CNN/DailyMail dataset, the R1 score of BART is 44.16.
- For the XSum dataset, the R1 score of BART is 45.14.Get the Raw JSONTo get the complete json returned by llmsherpa service and process it differently, simply get the json attributedoc.json