style="color: #ff00ff; text-decoration-color: #ff00ff">example-non-utf8.txt
The file example-non-utf8.txt uses a different encoding the load() function fails with a helpful message indicating which file failed decoding. With the default behavior of TextLoader any failure to load any of the documents will fail the whole loading process and no documents are loaded. B. Silent failâ€‹We can pass the parameter silent_errors to the DirectoryLoader to skip the files which could not be loaded and continue the load process.loader = DirectoryLoader(path, glob="**/*.txt", loader_cls=TextLoader, silent_errors=True)docs = loader.load()    Error loading ../../../../../tests/integration_tests/examples/example-non-utf8.txtdoc_sources = [doc.metadata['source']  for doc in docs]doc_sources    ['../../../../../tests/integration_tests/examples/whatsapp_chat.txt',     '../../../../../tests/integration_tests/examples/example-utf8.txt']C. Auto detect encodingsâ€‹We can also ask TextLoader to auto detect the file encoding before failing, by passing the autodetect_encoding to the loader class.text_loader_kwargs={'autodetect_encoding': True}loader = DirectoryLoader(path, glob="**/*.txt", loader_cls=TextLoader, loader_kwargs=text_loader_kwargs)docs = loader.load()doc_sources = [doc.metadata['source']  for doc in docs]doc_sources    ['../../../../../tests/integration_tests/examples/example-non-utf8.txt',     '../../../../../tests/integration_tests/examples/whatsapp_chat.txt',     '../../../../../tests/integration_tests/examples/example-utf8.txt']PreviousCSVNextHTMLCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


PDF | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionDocument loadersCSVFile DirectoryHTMLJSONMarkdownPDFDocument transformersText embedding modelsVector storesRetrieversChainsMemoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesData connectionDocument loadersPDFPDFPortable Document Format (PDF), standardized as ISO 32000, is a file format developed by Adobe in 1992 to present documents, including text formatting and images, in a manner independent of application software, hardware, and operating systems.This covers how to load PDF documents into the Document format that we use downstream.Using PyPDFâ€‹Load PDF using pypdf into array of documents, where each document contains the page content and metadata with page number.pip install pypdffrom langchain.document_loaders import PyPDFLoaderloader = PyPDFLoader("example_data/layout-parser-paper.pdf")pages = loader.load_and_split()pages[0]    Document(page_content='LayoutParser : A Uni\x0ced Toolkit for Deep\nLearning Based Document Image Analysis\nZejiang Shen1( \x00), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain\nLee4, Jacob Carlson3, and Weining Li5\n1Allen Institute for AI\nshannons@allenai.org\n2Brown University\nruochen zhang@brown.edu\n3Harvard University\nfmelissadell,jacob carlson g@fas.harvard.edu\n4University of Washington\nbcgl@cs.washington.edu\n5University of Waterloo\nw422li@uwaterloo.ca\nAbstract. Recent advances in document image analysis (DIA) have been\nprimarily driven by the application of neural networks. Ideally, research\noutcomes could be easily deployed in production and extended for


application of neural networks. Ideally, research\noutcomes could be easily deployed in production and extended for further\ninvestigation. However, various factors like loosely organized codebases\nand sophisticated model con\x0cgurations complicate the easy reuse of im-\nportant innovations by a wide audience. Though there have been on-going\ne\x0borts to improve reusability and simplify deep learning (DL) model\ndevelopment in disciplines like natural language processing and computer\nvision, none of them are optimized for challenges in the domain of DIA.\nThis represents a major gap in the existing toolkit, as DIA is central to\nacademic research across a wide range of disciplines in the social sciences\nand humanities. This paper introduces LayoutParser , an open-source\nlibrary for streamlining the usage of DL in DIA research and applica-\ntions. The core LayoutParser library comes with a set of simple and\nintuitive interfaces for applying and customizing DL models for layout de-\ntection, character recognition, and many other document processing tasks.\nTo promote extensibility, LayoutParser also incorporates a community\nplatform for sharing both pre-trained models and full document digiti-\nzation pipelines. We demonstrate that LayoutParser is helpful for both\nlightweight and large-scale digitization pipelines in real-word use cases.\nThe library is publicly available at https://layout-parser.github.io .\nKeywords: Document Image Analysis Â·Deep Learning Â·Layout Analysis\nÂ·Character Recognition Â·Open Source library Â·Toolkit.\n1 Introduction\nDeep Learning(DL)-based approaches are the state-of-the-art for a wide range of\ndocument image analysis (DIA) tasks including document image classi\x0ccation [ 11,arXiv:2103.15348v2  [cs.CV]  21 Jun 2021', metadata={'source': 'example_data/layout-parser-paper.pdf', 'page':


Jun 2021', metadata={'source': 'example_data/layout-parser-paper.pdf', 'page': 0})An advantage of this approach is that documents can be retrieved with page numbers.We want to use OpenAIEmbeddings so we have to get the OpenAI API Key.import osimport getpassos.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key:')    OpenAI API Key: Â·Â·Â·Â·Â·Â·Â·Â·from langchain.vectorstores import FAISSfrom langchain.embeddings.openai import OpenAIEmbeddingsfaiss_index = FAISS.from_documents(pages, OpenAIEmbeddings())docs = faiss_index.similarity_search("How will the community be engaged?", k=2)for doc in docs:    print(str(doc.metadata["page"]) + ":", doc.page_content[:300])    9: 10 Z. Shen et al.    Fig. 4: Illustration of (a) the original historical Japanese document with layout    detection results and (b) a recreated version of the document image that achieves    much better character recognition recall. The reorganization algorithm rearranges    the tokens based on the their detect    3: 4 Z. Shen et al.    Efficient Data AnnotationC u s t o m i z e d  M o d e l  T r a i n i n gModel Cust omizationDI A Model HubDI A Pipeline SharingCommunity PlatformLa y out Detection ModelsDocument Images     T h e  C o r e  L a y o u t P a r s e r  L i b r a r yOCR ModuleSt or age & VisualizationLa y ouUsing MathPixâ€‹Inspired by Daniel Gross's


or age & VisualizationLa y ouUsing MathPixâ€‹Inspired by Daniel Gross's https://gist.github.com/danielgross/3ab4104e14faccc12b49200843adab21from langchain.document_loaders import MathpixPDFLoaderloader = MathpixPDFLoader("example_data/layout-parser-paper.pdf")data = loader.load()Using Unstructuredâ€‹from langchain.document_loaders import UnstructuredPDFLoaderloader = UnstructuredPDFLoader("example_data/layout-parser-paper.pdf")data = loader.load()Retain Elementsâ€‹Under the hood, Unstructured creates different "elements" for different chunks of text. By default we combine those together, but you can easily keep that separation by specifying mode="elements".loader = UnstructuredPDFLoader("example_data/layout-parser-paper.pdf", mode="elements")data = loader.load()data[0]    Document(page_content='LayoutParser: A Uniï¬�ed Toolkit for Deep\nLearning Based Document Image Analysis\nZejiang Shen1 (ï¿½), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain\nLee4, Jacob Carlson3, and Weining Li5\n1 Allen Institute for AI\nshannons@allenai.org\n2 Brown University\nruochen zhang@brown.edu\n3 Harvard University\n{melissadell,jacob carlson}@fas.harvard.edu\n4 University of Washington\nbcgl@cs.washington.edu\n5 University of Waterloo\nw422li@uwaterloo.ca\nAbstract. Recent advances in document image analysis (DIA) have been\nprimarily driven by the application of neural networks. Ideally, research\noutcomes could be easily deployed in production and extended for further\ninvestigation. However, various factors like loosely organized codebases\nand sophisticated model conï¬�gurations complicate the easy reuse of im-\nportant innovations by a wide audience. Though


complicate the easy reuse of im-\nportant innovations by a wide audience. Though there have been on-going\neï¬€orts to improve reusability and simplify deep learning (DL) model\ndevelopment in disciplines like natural language processing and computer\nvision, none of them are optimized for challenges in the domain of DIA.\nThis represents a major gap in the existing toolkit, as DIA is central to\nacademic research across a wide range of disciplines in the social sciences\nand humanities. This paper introduces LayoutParser, an open-source\nlibrary for streamlining the usage of DL in DIA research and applica-\ntions. The core LayoutParser library comes with a set of simple and\nintuitive interfaces for applying and customizing DL models for layout de-\ntection, character recognition, and many other document processing tasks.\nTo promote extensibility, LayoutParser also incorporates a community\nplatform for sharing both pre-trained models and full document digiti-\nzation pipelines. We demonstrate that LayoutParser is helpful for both\nlightweight and large-scale digitization pipelines in real-word use cases.\nThe library is publicly available at https://layout-parser.github.io.\nKeywords: Document Image Analysis Â· Deep Learning Â· Layout Analysis\nÂ· Character Recognition Â· Open Source library Â· Toolkit.\n1\nIntroduction\nDeep Learning(DL)-based approaches are the state-of-the-art for a wide range of\ndocument image analysis (DIA) tasks including document image classiï¬�cation [11,\narXiv:2103.15348v2  [cs.CV]  21 Jun 2021\n', lookup_str='', metadata={'file_path': 'example_data/layout-parser-paper.pdf', 'page_number': 1, 'total_pages': 16, 'format': 'PDF 1.5', 'title': '', 'author': '', 'subject': '', 'keywords': '', 'creator':


'title': '', 'author': '', 'subject': '', 'keywords': '', 'creator': 'LaTeX with hyperref', 'producer': 'pdfTeX-1.40.21', 'creationDate': 'D:20210622012710Z', 'modDate': 'D:20210622012710Z', 'trapped': '', 'encryption': None}, lookup_index=0)Fetching remote PDFs using Unstructuredâ€‹This covers how to load online pdfs into a document format that we can use downstream. This can be used for various online pdf sites such as https://open.umn.edu/opentextbooks/textbooks/ and https://arxiv.org/archive/Note: all other pdf loaders can also be used to fetch remote PDFs, but OnlinePDFLoader is a legacy function, and works specifically with UnstructuredPDFLoader.from langchain.document_loaders import OnlinePDFLoaderloader = OnlinePDFLoader("https://arxiv.org/pdf/2302.03803.pdf")data = loader.load()print(data)    [Document(page_content='A WEAK ( k, k ) -LEFSCHETZ THEOREM FOR PROJECTIVE TORIC ORBIFOLDS\n\nWilliam D. Montoya\n\nInstituto de MatemÂ´atica, EstatÂ´Ä±stica e ComputaÂ¸cËœao CientÂ´Ä±ï¬�ca,\n\nIn [3] we proved that, under suitable conditions, on a very general codimension s quasi- smooth intersection subvariety X in a projective toric orbifold P d Î£ with d + s = 2 ( k + 1 ) the Hodge conjecture holds, that is, every ( p, p ) -cohomology class, under the PoincarÂ´e duality is a rational linear combination of fundamental classes of algebraic subvarieties of X


duality is a rational linear combination of fundamental classes of algebraic subvarieties of X . The proof of the above-mentioned result relies, for p â‰  d + 1 âˆ’ s , on a Lefschetz\n\nKeywords: (1,1)- Lefschetz theorem, Hodge conjecture, toric varieties, complete intersection Email: wmontoya@ime.unicamp.br\n\ntheorem ([7]) and the Hard Lefschetz theorem for projective orbifolds ([11]). When p = d + 1 âˆ’ s the proof relies on the Cayley trick, a trick which associates to X a quasi-smooth hypersurface Y in a projective vector bundle, and the Cayley Proposition (4.3) which gives an isomorphism of some primitive cohomologies (4.2) of X and Y . The Cayley trick, following the philosophy of Mavlyutov in [7], reduces results known for quasi-smooth hypersurfaces to quasi-smooth intersection subvarieties. The idea in this paper goes the other way around, we translate some results for quasi-smooth intersection subvarieties to\n\nAcknowledgement. I thank Prof. Ugo Bruzzo and Tiago Fonseca for useful discus- sions. I also acknowledge support from FAPESP postdoctoral grant No. 2019/23499-7.\n\nLet M be a free abelian group of rank d , let N = Hom ( M, Z ) , and N R = N âŠ— Z R .\n\nif there exist k linearly independent primitive elements e\n\n, . . . , e k âˆˆ N such that Ïƒ = { Âµ\n\ne\n\n+ â‹¯ + Âµ k e k } . â€¢ The generators e i are integral if for every i and


k e k } . â€¢ The generators e i are integral if for every i and any nonnegative rational number Âµ the product Âµe i is in N only if Âµ is an integer. â€¢ Given two rational simplicial cones Ïƒ , Ïƒ â€² one says that Ïƒ â€² is a face of Ïƒ ( Ïƒ â€² < Ïƒ ) if the set of integral generators of Ïƒ â€² is a subset of the set of integral generators of Ïƒ . â€¢ A ï¬�nite set Î£ = { Ïƒ\n\n, . . . , Ïƒ t } of rational simplicial cones is called a rational simplicial complete d -dimensional fan if:\n\nall faces of cones in Î£ are in Î£ ;\n\nif Ïƒ, Ïƒ â€² âˆˆ Î£ then Ïƒ âˆ© Ïƒ â€² < Ïƒ and Ïƒ âˆ© Ïƒ â€² < Ïƒ â€² ;\n\nN R = Ïƒ\n\nâˆª â‹… â‹… â‹… âˆª Ïƒ t .\n\nA rational simplicial complete d -dimensional fan Î£ deï¬�nes a d -dimensional toric variety P d Î£ having only orbifold singularities which we assume to be projective. Moreover, T âˆ¶ = N âŠ— Z C âˆ— â‰ƒ ( C âˆ— ) d is the torus action on P d Î£ . We denote by Î£ ( i ) the i


the torus action on P d Î£ . We denote by Î£ ( i ) the i -dimensional cones\n\nFor a cone Ïƒ âˆˆ Î£, Ë† Ïƒ is the set of 1-dimensional cone in Î£ that are not contained in Ïƒ\n\nand x Ë† Ïƒ âˆ¶ = âˆ� Ï� âˆˆ Ë† Ïƒ x Ï� is the associated monomial in S .\n\nDeï¬�nition 2.2. The irrelevant ideal of P d Î£ is the monomial ideal B Î£ âˆ¶ =< x Ë† Ïƒ âˆ£ Ïƒ âˆˆ Î£ > and the zero locus Z ( Î£ ) âˆ¶ = V ( B Î£ ) in the aï¬ƒne space A d âˆ¶ = Spec ( S ) is the irrelevant locus.\n\nProposition 2.3 (Theorem 5.1.11 [5]) . The toric variety P d Î£ is a categorical quotient A d âˆ– Z ( Î£ ) by the group Hom ( Cl ( Î£ ) , C âˆ— ) and the group action is induced by the Cl ( Î£ ) - grading of S .\n\nNow we give a brief introduction to complex orbifolds and we mention the needed theorems for the next section. Namely: de Rham theorem and Dolbeault theorem for complex orbifolds.\n\nDeï¬�nition 2.4. A complex orbifold of complex dimension d is a singular complex space whose singularities are locally isomorphic to quotient singularities C d / G , for ï¬�nite sub- groups G âŠ‚


C d / G , for ï¬�nite sub- groups G âŠ‚ Gl ( d, C ) .\n\nDeï¬�nition 2.5. A diï¬€erential form on a complex orbifold Z is deï¬�ned locally at z âˆˆ Z as a G -invariant diï¬€erential form on C d where G âŠ‚ Gl ( d, C ) and Z is locally isomorphic to d\n\nRoughly speaking the local geometry of orbifolds reduces to local G -invariant geometry.\n\nWe have a complex of diï¬€erential forms ( A â—� ( Z ) , d ) and a double complex ( A â—� , â—� ( Z ) , âˆ‚, Â¯ âˆ‚ ) of bigraded diï¬€erential forms which deï¬�ne the de Rham and the Dolbeault cohomology groups (for a ï¬�xed p âˆˆ N ) respectively:\n\n(1,1)-Lefschetz theorem for projective toric orbifolds\n\nDeï¬�nition 3.1. A subvariety X âŠ‚ P d Î£ is quasi-smooth if V ( I X ) âŠ‚ A #Î£ ( 1 ) is smooth outside\n\nExample 3.2 . Quasi-smooth hypersurfaces or more generally quasi-smooth intersection sub-\n\nExample 3.2 . Quasi-smooth hypersurfaces or more generally quasi-smooth intersection sub- varieties are quasi-smooth subvarieties (see [2] or [7] for more details).\n\nRemark 3.3 . Quasi-smooth subvarieties are suborbifolds of P d Î£ in the sense of


subvarieties are suborbifolds of P d Î£ in the sense of Satake in [8]. Intuitively speaking they are subvarieties whose only singularities come from the ambient\n\nProof. From the exponential short exact sequence\n\nwe have a long exact sequence in cohomology\n\nH 1 (O âˆ— X ) â†’ H 2 ( X, Z ) â†’ H 2 (O X ) â‰ƒ H 0 , 2 ( X )\n\nwhere the last isomorphisms is due to Steenbrink in [9]. Now, it is enough to prove the commutativity of the next diagram\n\nwhere the last isomorphisms is due to Steenbrink in [9]. Now,\n\nH 2 ( X, Z ) / / H 2 ( X, O X ) â‰ƒ Dolbeault H 2 ( X, C ) deRham â‰ƒ H 2 dR ( X, C ) / / H 0 , 2 Â¯ âˆ‚ ( X )\n\nof the proof follows as the ( 1 , 1 ) -Lefschetz theorem in [6].\n\nRemark 3.5 . For k = 1 and P d Î£ as the projective space, we recover the classical ( 1 , 1 ) - Lefschetz theorem.\n\nBy the Hard Lefschetz Theorem for projective orbifolds (see [11] for details) we\n\nBy the Hard Lefschetz Theorem for projective orbifolds (see [11] for details) we get an isomorphism of cohomologies :\n\ngiven by the Lefschetz morphism and since it is a morphism of Hodge structures, we have:\n\nH


morphism and since it is a morphism of Hodge structures, we have:\n\nH 1 , 1 ( X, Q ) â‰ƒ H dim X âˆ’ 1 , dim X âˆ’ 1 ( X, Q )\n\nCorollary 3.6. If the dimension of X is 1 , 2 or 3 . The Hodge conjecture holds on X\n\nProof. If the dim C X = 1 the result is clear by the Hard Lefschetz theorem for projective orbifolds. The dimension 2 and 3 cases are covered by Theorem 3.5 and the Hard Lefschetz.\n\nCayley trick and Cayley proposition\n\nThe Cayley trick is a way to associate to a quasi-smooth intersection subvariety a quasi- smooth hypersurface. Let L 1 , . . . , L s be line bundles on P d Î£ and let Ï€ âˆ¶ P ( E ) â†’ P d Î£ be the projective space bundle associated to the vector bundle E = L 1 âŠ• â‹¯ âŠ• L s . It is known that P ( E ) is a ( d + s âˆ’ 1 ) -dimensional simplicial toric variety whose fan depends on the degrees of the line bundles and the fan Î£. Furthermore, if the Cox ring, without considering the grading, of P d Î£ is C [ x 1 , . . . , x m ] then the Cox ring of P ( E ) is\n\nMoreover for X a quasi-smooth intersection subvariety cut oï¬€ by f 1 , . . . , f s with deg ( f i ) = [ L i ] we relate the hypersurface Y cut oï¬€ by F = y 1 f


i ] we relate the hypersurface Y cut oï¬€ by F = y 1 f 1 + â‹… â‹… â‹… + y s f s which turns out to be quasi-smooth. For more details see Section 2 in [7].\n\nWe will denote P ( E ) as P d + s âˆ’ 1 Î£ ,X to keep track of its relation with X and P d Î£ .\n\nThe following is a key remark.\n\nRemark 4.1 . There is a morphism Î¹ âˆ¶ X â†’ Y âŠ‚ P d + s âˆ’ 1 Î£ ,X . Moreover every point z âˆ¶ = ( x, y ) âˆˆ Y with y â‰  0 has a preimage. Hence for any subvariety W = V ( I W ) âŠ‚ X âŠ‚ P d Î£ there exists W â€² âŠ‚ Y âŠ‚ P d + s âˆ’ 1 Î£ ,X such that Ï€ ( W â€² ) = W , i.e., W â€² = { z = ( x, y ) âˆ£ x âˆˆ W } .\n\nFor X âŠ‚ P d Î£ a quasi-smooth intersection variety the morphism in cohomology induced by the inclusion i âˆ— âˆ¶ H d âˆ’ s ( P d Î£ , C ) â†’ H d âˆ’ s ( X, C ) is injective by Proposition 1.4 in [7].\n\nDeï¬�nition 4.2. The primitive cohomology of H d âˆ’ s prim ( X ) is the quotient H d


cohomology of H d âˆ’ s prim ( X ) is the quotient H d âˆ’ s ( X, C )/ i âˆ— ( H d âˆ’ s ( P d Î£ , C )) and H d âˆ’ s prim ( X, Q ) with rational coeï¬ƒcients.\n\nH d âˆ’ s ( P d Î£ , C ) and H d âˆ’ s ( X, C ) have pure Hodge structures, and the morphism i âˆ— is com- patible with them, so that H d âˆ’ s prim ( X ) gets a pure Hodge structure.\n\nThe next Proposition is the Cayley proposition.\n\nProposition 4.3. [Proposition 2.3 in [3] ] Let X = X 1 âˆ©â‹… â‹… â‹…âˆ© X s be a quasi-smooth intersec- tion subvariety in P d Î£ cut oï¬€ by homogeneous polynomials f 1 . . . f s . Then for p â‰  d + s âˆ’ 1 2 , d + s âˆ’ 3 2\n\nRemark 4.5 . The above isomorphisms are also true with rational coeï¬ƒcients since H â—� ( X, C ) = H â—� ( X, Q ) âŠ— Q C . See the beginning of Section 7.1 in [10] for more details.\n\nTheorem 5.1. Let Y = { F = y 1 f 1 + â‹¯ + y k f k = 0 } âŠ‚ P 2 k + 1 Î£ ,X be the quasi-smooth


âŠ‚ P 2 k + 1 Î£ ,X be the quasi-smooth hypersurface associated to the quasi-smooth intersection surface X = X f 1 âˆ© â‹… â‹… â‹… âˆ© X f k âŠ‚ P k + 2 Î£ . Then on Y the Hodge conjecture holds.\n\nthe Hodge conjecture holds.\n\nProof. If H k,k prim ( X, Q ) = 0 we are done. So let us assume H k,k prim ( X, Q ) â‰  0. By the Cayley proposition H k,k prim ( Y, Q ) â‰ƒ H 1 , 1 prim ( X, Q ) and by the ( 1 , 1 ) -Lefschetz theorem for projective\n\ntoric orbifolds there is a non-zero algebraic basis Î» C 1 , . . . , Î» C n with rational coeï¬ƒcients of H 1 , 1 prim ( X, Q ) , that is, there are n âˆ¶ = h 1 , 1 prim ( X, Q ) algebraic curves C 1 , . . . , C n in X such that under the PoincarÂ´e duality the class in homology [ C i ] goes to Î» C i , [ C i ] â†¦ Î» C i . Recall that the Cox ring of P k + 2 is contained in the Cox ring of P 2 k + 1 Î£ ,X without considering the grading. Considering the grading we have that if Î± âˆˆ Cl ( P k + 2 Î£ ) then ( Î±, 0 ) âˆˆ Cl ( P 2 k + 1 Î£ ,X ) . So the polynomials


Cl ( P 2 k + 1 Î£ ,X ) . So the polynomials deï¬�ning C i âŠ‚ P k + 2 Î£ can be interpreted in P 2 k + 1 X, Î£ but with diï¬€erent degree. Moreover, by Remark 4.1 each C i is contained in Y = { F = y 1 f 1 + â‹¯ + y k f k = 0 } and\n\nfurthermore it has codimension k .\n\nClaim: { C i } ni = 1 is a basis of prim ( ) . It is enough to prove that Î» C i is diï¬€erent from zero in H k,k prim ( Y, Q ) or equivalently that the cohomology classes { Î» C i } ni = 1 do not come from the ambient space. By contradiction, let us assume that there exists a j and C âŠ‚ P 2 k + 1 Î£ ,X such that Î» C âˆˆ H k,k ( P 2 k + 1 Î£ ,X , Q ) with i âˆ— ( Î» C ) = Î» C j or in terms of homology there exists a ( k + 2 ) -dimensional algebraic subvariety V âŠ‚ P 2 k + 1 Î£ ,X such that V âˆ© Y = C j so they are equal as a homology class of P 2 k + 1 Î£ ,X ,i.e., [ V âˆ© Y ] = [ C j ] . It is easy to check that Ï€ ( V ) âˆ© X = C j as a subvariety of P k + 2 Î£ where Ï€ âˆ¶ ( x, y ) â†¦


2 Î£ where Ï€ âˆ¶ ( x, y ) â†¦ x . Hence [ Ï€ ( V ) âˆ© X ] = [ C j ] which is equivalent to say that Î» C j comes from P k + 2 Î£ which contradicts the choice of [ C j ] .\n\nRemark 5.2 . Into the proof of the previous theorem, the key fact was that on X the Hodge conjecture holds and we translate it to Y by contradiction. So, using an analogous argument we have:\n\nargument we have:\n\nProposition 5.3. Let Y = { F = y 1 f s +â‹¯+ y s f s = 0 } âŠ‚ P 2 k + 1 Î£ ,X be the quasi-smooth hypersurface associated to a quasi-smooth intersection subvariety X = X f 1 âˆ© â‹… â‹… â‹… âˆ© X f s âŠ‚ P d Î£ such that d + s = 2 ( k + 1 ) . If the Hodge conjecture holds on X then it holds as well on Y .\n\nCorollary 5.4. If the dimension of Y is 2 s âˆ’ 1 , 2 s or 2 s + 1 then the Hodge conjecture holds on Y .\n\nProof. By Proposition 5.3 and Corollary 3.6.\n\n[\n\n] Angella, D. Cohomologies of certain orbifolds. Journal of Geometry and Physics\n\n(\n\n),\n\nâ€“\n\n[\n\n] Batyrev, V. V., and Cox, D. A. On the Hodge structure of projective hypersur- faces in toric


D. A. On the Hodge structure of projective hypersur- faces in toric varieties. Duke Mathematical Journal\n\n,\n\n(Aug\n\n). [\n\n] Bruzzo, U., and Montoya, W. On the Hodge conjecture for quasi-smooth in- tersections in toric varieties. SËœao Paulo J. Math. Sci. Special Section: Geometry in Algebra and Algebra in Geometry (\n\n). [\n\n] Caramello Jr, F. C. Introduction to orbifolds. a\n\niv:\n\nv\n\n(\n\n). [\n\n] Cox, D., Little, J., and Schenck, H. Toric varieties, vol.\n\nAmerican Math- ematical Soc.,\n\n[\n\n] Griffiths, P., and Harris, J. Principles of Algebraic Geometry. John Wiley & Sons, Ltd,\n\n[\n\n] Mavlyutov, A. R. Cohomology of complete intersections in toric varieties. Pub- lished in Paciï¬�c J. of Math.\n\nNo.\n\n(\n\n),\n\nâ€“\n\n[\n\n] Satake, I. On a Generalization of the Notion of Manifold. Proceedings of the National Academy of Sciences of the United States of America\n\n,\n\n(\n\n),\n\nâ€“\n\n[\n\n] Steenbrink, J. H. M. Intersection form for quasi-homogeneous singularities. Com- positio Mathematica\n\n,\n\n(\n\n),\n\nâ€“\n\n[\n\n] Voisin, C. Hodge Theory and Complex Algebraic Geometry I, vol.\n\nof Cambridge Studies in Advanced Mathematics . Cambridge University Press,\n\n[\n\n] Wang, Z. Z., and


Advanced Mathematics . Cambridge University Press,\n\n[\n\n] Wang, Z. Z., and Zaffran, D. A remark on the Hard Lefschetz theorem for KÂ¨ahler orbifolds. Proceedings of the American Mathematical Society\n\n,\n\n(Aug\n\n).\n\n[2] Batyrev, V. V., and Cox, D. A. On the Hodge structure of projective hypersur- faces in toric varieties. Duke Mathematical Journal 75, 2 (Aug 1994).\n\n[\n\n] Bruzzo, U., and Montoya, W. On the Hodge conjecture for quasi-smooth in- tersections in toric varieties. SËœao Paulo J. Math. Sci. Special Section: Geometry in Algebra and Algebra in Geometry (\n\n).\n\n[3] Bruzzo, U., and Montoya, W. On the Hodge conjecture for quasi-smooth in- tersections in toric varieties. SËœao Paulo J. Math. Sci. Special Section: Geometry in Algebra and Algebra in Geometry (2021).\n\nA. R. Cohomology of complete intersections in toric varieties. Pub-', lookup_str='', metadata={'source': '/var/folders/ph/hhm7_zyx4l13k3v8z02dwp1w0000gn/T/tmpgq0ckaja/online_file.pdf'}, lookup_index=0)]Using PyPDFium2â€‹from langchain.document_loaders import PyPDFium2Loaderloader = PyPDFium2Loader("example_data/layout-parser-paper.pdf")data = loader.load()Using PDFMinerâ€‹from langchain.document_loaders import PDFMinerLoaderloader = PDFMinerLoader("example_data/layout-parser-paper.pdf")data = loader.load()Using PDFMiner to generate HTML textâ€‹This can be


= loader.load()Using PDFMiner to generate HTML textâ€‹This can be helpful for chunking texts semantically into sections as the output html content can be parsed via BeautifulSoup to get more structured and rich information about font size, page numbers, pdf headers/footers, etc.from langchain.document_loaders import PDFMinerPDFasHTMLLoaderloader = PDFMinerPDFasHTMLLoader("example_data/layout-parser-paper.pdf")data = loader.load()[0]   # entire pdf is loaded as a single Documentfrom bs4 import BeautifulSoupsoup = BeautifulSoup(data.page_content,'html.parser')content = soup.find_all('div')import recur_fs = Nonecur_text = ''snippets = []   # first collect all snippets that have the same font sizefor c in content:    sp = c.find('span')    if not sp:        continue    st = sp.get('style')    if not st:        continue    fs = re.findall('font-size:(\d+)px',st)    if not fs:        continue    fs = int(fs[0])    if not cur_fs:        cur_fs = fs    if fs == cur_fs:        cur_text += c.text    else:        snippets.append((cur_text,cur_fs))        cur_fs = fs        cur_text = c.textsnippets.append((cur_text,cur_fs))# Note: The above logic is very straightforward. One can also add more strategies such as removing duplicate snippets (as# headers/footers in a PDF appear on multiple pages so if we find duplicatess safe to assume that it is redundant


a PDF appear on multiple pages so if we find duplicatess safe to assume that it is redundant info)from langchain.docstore.document import Documentcur_idx = -1semantic_snippets = []# Assumption: headings have higher font size than their respective contentfor s in snippets:    # if current snippet's font size > previous section's heading => it is a new heading    if not semantic_snippets or s[1] > semantic_snippets[cur_idx].metadata['heading_font']:        metadata={'heading':s[0], 'content_font': 0, 'heading_font': s[1]}        metadata.update(data.metadata)        semantic_snippets.append(Document(page_content='',metadata=metadata))        cur_idx += 1        continue        # if current snippet's font size <= previous section's content => content belongs to the same section (one can also create    # a tree like structure for sub sections if needed but that may require some more thinking and may be data specific)    if not semantic_snippets[cur_idx].metadata['content_font'] or s[1] <= semantic_snippets[cur_idx].metadata['content_font']:        semantic_snippets[cur_idx].page_content += s[0]        semantic_snippets[cur_idx].metadata['content_font'] = max(s[1], semantic_snippets[cur_idx].metadata['content_font'])        continue        # if current snippet's font size > previous section's content but less than previous section's heading than also make a new     # section (e.g. title of a pdf will have the highest font size but we don't want it to subsume all


title of a pdf will have the highest font size but we don't want it to subsume all sections)    metadata={'heading':s[0], 'content_font': 0, 'heading_font': s[1]}    metadata.update(data.metadata)    semantic_snippets.append(Document(page_content='',metadata=metadata))    cur_idx += 1semantic_snippets[4]    Document(page_content='Recently, various DL models and datasets have been developed for layout analysis\ntasks. The dhSegment [22] utilizes fully convolutional networks [20] for segmen-\ntation tasks on historical documents. Object detection-based methods like Faster\nR-CNN [28] and Mask R-CNN [12] are used for identifying document elements [38]\nand detecting tables [30, 26]. Most recently, Graph Neural Networks [29] have also\nbeen used in table detection [27]. However, these models are usually implemented\nindividually and there is no uniï¬�ed framework to load and use such models.\nThere has been a surge of interest in creating open-source tools for document\nimage processing: a search of document image analysis in Github leads to 5M\nrelevant code pieces 6; yet most of them rely on traditional rule-based methods\nor provide limited functionalities. The closest prior research to our work is the\nOCR-D project7, which also tries to build a complete toolkit for DIA. However,\nsimilar to the platform developed by Neudecker et al. [21], it is designed for\nanalyzing historical documents, and provides no supports for recent DL models.\nThe DocumentLayoutAnalysis project8 focuses on processing born-digital PDF\ndocuments via analyzing the stored PDF data. Repositories like DeepLayout9\nand Detectron2-PubLayNet10 are individual deep learning models trained on\nlayout analysis datasets without support for the full DIA


are individual deep learning models trained on\nlayout analysis datasets without support for the full DIA pipeline. The Document\nAnalysis and Exploitation (DAE) platform [15] and the DeepDIVA project [2]\naim to improve the reproducibility of DIA methods (or DL models), yet they\nare not actively maintained. OCR engines like Tesseract [14], easyOCR11 and\npaddleOCR12 usually do not come with comprehensive functionalities for other\nDIA tasks like layout analysis.\nRecent years have also seen numerous eï¬€orts to create libraries for promoting\nreproducibility and reusability in the ï¬�eld of DL. Libraries like Dectectron2 [35],\n6 The number shown is obtained by specifying the search type as â€˜codeâ€™.\n7 https://ocr-d.de/en/about\n8 https://github.com/BobLd/DocumentLayoutAnalysis\n9 https://github.com/leonlulu/DeepLayout\n10 https://github.com/hpanwar08/detectron2\n11 https://github.com/JaidedAI/EasyOCR\n12 https://github.com/PaddlePaddle/PaddleOCR\n4\nZ. Shen et al.\nFig. 1: The overall architecture of LayoutParser. For an input document image,\nthe core LayoutParser library provides a set of oï¬€-the-shelf tools for layout\ndetection, OCR, visualization, and storage, backed by a carefully designed layout\ndata structure. LayoutParser also supports high level customization via eï¬ƒcient\nlayout annotation and model training functions. These improve model accuracy\non the target samples. The community platform enables the easy sharing of DIA\nmodels and whole digitization pipelines to promote reusability and reproducibility.\nA collection of detailed documentation, tutorials and exemplar projects make\nLayoutParser easy to learn and


collection of detailed documentation, tutorials and exemplar projects make\nLayoutParser easy to learn and use.\nAllenNLP [8] and transformers [34] have provided the community with complete\nDL-based support for developing and deploying models for general computer\nvision and natural language processing problems. LayoutParser, on the other\nhand, specializes speciï¬�cally in DIA tasks. LayoutParser is also equipped with a\ncommunity platform inspired by established model hubs such as Torch Hub [23]\nand TensorFlow Hub [1]. It enables the sharing of pretrained models as well as\nfull document processing pipelines that are unique to DIA tasks.\nThere have been a variety of document data collections to facilitate the\ndevelopment of DL models. Some examples include PRImA [3](magazine layouts),\nPubLayNet [38](academic paper layouts), Table Bank [18](tables in academic\npapers), Newspaper Navigator Dataset [16, 17](newspaper ï¬�gure layouts) and\nHJDataset [31](historical Japanese document layouts). A spectrum of models\ntrained on these datasets are currently available in the LayoutParser model zoo\nto support diï¬€erent use cases.\n', metadata={'heading': '2 Related Work\n', 'content_font': 9, 'heading_font': 11, 'source': 'example_data/layout-parser-paper.pdf'})Using PyMuPDFâ€‹This is the fastest of the PDF parsing options, and contains detailed metadata about the PDF and its pages, as well as returns one document per page.from langchain.document_loaders import PyMuPDFLoaderloader = PyMuPDFLoader("example_data/layout-parser-paper.pdf")data = loader.load()data[0]    Document(page_content='LayoutParser: A Uniï¬�ed Toolkit for Deep\nLearning Based Document Image Analysis\nZejiang Shen1 (ï¿½), Ruochen


for Deep\nLearning Based Document Image Analysis\nZejiang Shen1 (ï¿½), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain\nLee4, Jacob Carlson3, and Weining Li5\n1 Allen Institute for AI\nshannons@allenai.org\n2 Brown University\nruochen zhang@brown.edu\n3 Harvard University\n{melissadell,jacob carlson}@fas.harvard.edu\n4 University of Washington\nbcgl@cs.washington.edu\n5 University of Waterloo\nw422li@uwaterloo.ca\nAbstract. Recent advances in document image analysis (DIA) have been\nprimarily driven by the application of neural networks. Ideally, research\noutcomes could be easily deployed in production and extended for further\ninvestigation. However, various factors like loosely organized codebases\nand sophisticated model conï¬�gurations complicate the easy reuse of im-\nportant innovations by a wide audience. Though there have been on-going\neï¬€orts to improve reusability and simplify deep learning (DL) model\ndevelopment in disciplines like natural language processing and computer\nvision, none of them are optimized for challenges in the domain of DIA.\nThis represents a major gap in the existing toolkit, as DIA is central to\nacademic research across a wide range of disciplines in the social sciences\nand humanities. This paper introduces LayoutParser, an open-source\nlibrary for streamlining the usage of DL in DIA research and applica-\ntions. The core LayoutParser library comes with a set of simple and\nintuitive interfaces for applying and customizing DL models for layout de-\ntection, character recognition, and many other document processing tasks.\nTo promote extensibility, LayoutParser also incorporates a community\nplatform for sharing both pre-trained models and full document digiti-\nzation pipelines. We demonstrate that LayoutParser is helpful for both\nlightweight and


digiti-\nzation pipelines. We demonstrate that LayoutParser is helpful for both\nlightweight and large-scale digitization pipelines in real-word use cases.\nThe library is publicly available at https://layout-parser.github.io.\nKeywords: Document Image Analysis Â· Deep Learning Â· Layout Analysis\nÂ· Character Recognition Â· Open Source library Â· Toolkit.\n1\nIntroduction\nDeep Learning(DL)-based approaches are the state-of-the-art for a wide range of\ndocument image analysis (DIA) tasks including document image classiï¬�cation [11,\narXiv:2103.15348v2  [cs.CV]  21 Jun 2021\n', lookup_str='', metadata={'file_path': 'example_data/layout-parser-paper.pdf', 'page_number': 1, 'total_pages': 16, 'format': 'PDF 1.5', 'title': '', 'author': '', 'subject': '', 'keywords': '', 'creator': 'LaTeX with hyperref', 'producer': 'pdfTeX-1.40.21', 'creationDate': 'D:20210622012710Z', 'modDate': 'D:20210622012710Z', 'trapped': '', 'encryption': None}, lookup_index=0)Additionally, you can pass along any of the options from the PyMuPDF documentation as keyword arguments in the load call, and it will be pass along to the get_text() call.PyPDF Directoryâ€‹Load PDFs from directoryfrom langchain.document_loaders import PyPDFDirectoryLoaderloader = PyPDFDirectoryLoader("example_data/")docs = loader.load()Using pdfplumberâ€‹Like PyMuPDF, the output Documents contain detailed metadata about the PDF and its pages, and returns one document per page.from langchain.document_loaders import PDFPlumberLoaderloader = PDFPlumberLoader("example_data/layout-parser-paper.pdf")data =


PDFPlumberLoaderloader = PDFPlumberLoader("example_data/layout-parser-paper.pdf")data = loader.load()data[0]    Document(page_content='LayoutParser: A Unified Toolkit for Deep\nLearning Based Document Image Analysis\nZejiang Shen1 ((cid:0)), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain\nLee4, Jacob Carlson3, and Weining Li5\n1 Allen Institute for AI\n1202 shannons@allenai.org\n2 Brown University\nruochen zhang@brown.edu\n3 Harvard University\nnuJ {melissadell,jacob carlson}@fas.harvard.edu\n4 University of Washington\nbcgl@cs.washington.edu\n12 5 University of Waterloo\nw422li@uwaterloo.ca\n]VC.sc[\nAbstract. Recentadvancesindocumentimageanalysis(DIA)havebeen\nprimarily driven by the application of neural networks. Ideally, research\noutcomescouldbeeasilydeployedinproductionandextendedforfurther\ninvestigation. However, various factors like loosely organized codebases\nand sophisticated model configurations complicate the easy reuse of im-\n2v84351.3012:viXra portantinnovationsbyawideaudience.Thoughtherehavebeenon-going\nefforts to improve reusability and simplify deep learning (DL) model\ndevelopmentindisciplineslikenaturallanguageprocessingandcomputer\nvision, none of them are optimized for challenges in the domain of DIA.\nThis represents a major gap in the existing toolkit, as DIA is central to\nacademicresearchacross awiderangeof disciplinesinthesocialsciences\nand humanities. This paper introduces LayoutParser, an open-source\nlibrary for streamlining the usage of DL in DIA research and applica-\ntions. The core LayoutParser library comes with a set of simple


research and applica-\ntions. The core LayoutParser library comes with a set of simple and\nintuitiveinterfacesforapplyingandcustomizingDLmodelsforlayoutde-\ntection,characterrecognition,andmanyotherdocumentprocessingtasks.\nTo promote extensibility, LayoutParser also incorporates a community\nplatform for sharing both pre-trained models and full document digiti-\nzation pipelines. We demonstrate that LayoutParser is helpful for both\nlightweight and large-scale digitization pipelines in real-word use cases.\nThe library is publicly available at https://layout-parser.github.io.\nKeywords: DocumentImageAnalysisÂ·DeepLearningÂ·LayoutAnalysis\nÂ· Character Recognition Â· Open Source library Â· Toolkit.\n1 Introduction\nDeep Learning(DL)-based approaches are the state-of-the-art for a wide range of\ndocumentimageanalysis(DIA)tasksincludingdocumentimageclassification[11,', metadata={'source': 'example_data/layout-parser-paper.pdf', 'file_path': 'example_data/layout-parser-paper.pdf', 'page': 1, 'total_pages': 16, 'Author': '', 'CreationDate': 'D:20210622012710Z', 'Creator': 'LaTeX with hyperref', 'Keywords': '', 'ModDate': 'D:20210622012710Z', 'PTEX.Fullbanner': 'This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020) kpathsea version 6.3.2', 'Producer': 'pdfTeX-1.40.21', 'Subject': '', 'Title': '', 'Trapped': 'False'})PreviousMarkdownNextDocument transformersCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


CSV | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionDocument loadersCSVFile DirectoryHTMLJSONMarkdownPDFDocument transformersText embedding modelsVector storesRetrieversChainsMemoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesData connectionDocument loadersCSVCSVA comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas.Load CSV data with a single row per document.from langchain.document_loaders.csv_loader import CSVLoaderloader = CSVLoader(file_path='./example_data/mlb_teams_2012.csv')data = loader.load()print(data)    [Document(page_content='Team: Nationals\n"Payroll (millions)": 81.34\n"Wins": 98', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 0}, lookup_index=0), Document(page_content='Team: Reds\n"Payroll (millions)": 82.20\n"Wins": 97', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 1}, lookup_index=0), Document(page_content='Team: Yankees\n"Payroll (millions)": 197.96\n"Wins": 95', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 2}, lookup_index=0), Document(page_content='Team: Giants\n"Payroll (millions)": 117.62\n"Wins": 94', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 3},


'./example_data/mlb_teams_2012.csv', 'row': 3}, lookup_index=0), Document(page_content='Team: Braves\n"Payroll (millions)": 83.31\n"Wins": 94', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 4}, lookup_index=0), Document(page_content='Team: Athletics\n"Payroll (millions)": 55.37\n"Wins": 94', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 5}, lookup_index=0), Document(page_content='Team: Rangers\n"Payroll (millions)": 120.51\n"Wins": 93', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 6}, lookup_index=0), Document(page_content='Team: Orioles\n"Payroll (millions)": 81.43\n"Wins": 93', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 7}, lookup_index=0), Document(page_content='Team: Rays\n"Payroll (millions)": 64.17\n"Wins": 90', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 8}, lookup_index=0), Document(page_content='Team: Angels\n"Payroll (millions)": 154.49\n"Wins": 89', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 9}, lookup_index=0), Document(page_content='Team: Tigers\n"Payroll (millions)": 132.30\n"Wins": 88', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row':


metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 10}, lookup_index=0), Document(page_content='Team: Cardinals\n"Payroll (millions)": 110.30\n"Wins": 88', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 11}, lookup_index=0), Document(page_content='Team: Dodgers\n"Payroll (millions)": 95.14\n"Wins": 86', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 12}, lookup_index=0), Document(page_content='Team: White Sox\n"Payroll (millions)": 96.92\n"Wins": 85', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 13}, lookup_index=0), Document(page_content='Team: Brewers\n"Payroll (millions)": 97.65\n"Wins": 83', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 14}, lookup_index=0), Document(page_content='Team: Phillies\n"Payroll (millions)": 174.54\n"Wins": 81', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 15}, lookup_index=0), Document(page_content='Team: Diamondbacks\n"Payroll (millions)": 74.28\n"Wins": 81', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 16}, lookup_index=0), Document(page_content='Team: Pirates\n"Payroll (millions)": 63.43\n"Wins": 79', lookup_str='', metadata={'source':


63.43\n"Wins": 79', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 17}, lookup_index=0), Document(page_content='Team: Padres\n"Payroll (millions)": 55.24\n"Wins": 76', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 18}, lookup_index=0), Document(page_content='Team: Mariners\n"Payroll (millions)": 81.97\n"Wins": 75', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 19}, lookup_index=0), Document(page_content='Team: Mets\n"Payroll (millions)": 93.35\n"Wins": 74', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 20}, lookup_index=0), Document(page_content='Team: Blue Jays\n"Payroll (millions)": 75.48\n"Wins": 73', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 21}, lookup_index=0), Document(page_content='Team: Royals\n"Payroll (millions)": 60.91\n"Wins": 72', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 22}, lookup_index=0), Document(page_content='Team: Marlins\n"Payroll (millions)": 118.07\n"Wins": 69', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 23}, lookup_index=0), Document(page_content='Team: Red Sox\n"Payroll (millions)": 173.18\n"Wins": 69',


Sox\n"Payroll (millions)": 173.18\n"Wins": 69', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 24}, lookup_index=0), Document(page_content='Team: Indians\n"Payroll (millions)": 78.43\n"Wins": 68', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 25}, lookup_index=0), Document(page_content='Team: Twins\n"Payroll (millions)": 94.08\n"Wins": 66', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 26}, lookup_index=0), Document(page_content='Team: Rockies\n"Payroll (millions)": 78.06\n"Wins": 64', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 27}, lookup_index=0), Document(page_content='Team: Cubs\n"Payroll (millions)": 88.19\n"Wins": 61', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 28}, lookup_index=0), Document(page_content='Team: Astros\n"Payroll (millions)": 60.65\n"Wins": 55', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 29}, lookup_index=0)]Customizing the csv parsing and loadingâ€‹See the csv module documentation for more information of what csv args are supported.loader = CSVLoader(file_path='./example_data/mlb_teams_2012.csv', csv_args={    'delimiter': ',',    'quotechar': '"',    'fieldnames': ['MLB Team', 'Payroll


'"',    'fieldnames': ['MLB Team', 'Payroll in millions', 'Wins']})data = loader.load()print(data)    [Document(page_content='MLB Team: Team\nPayroll in millions: "Payroll (millions)"\nWins: "Wins"', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 0}, lookup_index=0), Document(page_content='MLB Team: Nationals\nPayroll in millions: 81.34\nWins: 98', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 1}, lookup_index=0), Document(page_content='MLB Team: Reds\nPayroll in millions: 82.20\nWins: 97', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 2}, lookup_index=0), Document(page_content='MLB Team: Yankees\nPayroll in millions: 197.96\nWins: 95', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 3}, lookup_index=0), Document(page_content='MLB Team: Giants\nPayroll in millions: 117.62\nWins: 94', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 4}, lookup_index=0), Document(page_content='MLB Team: Braves\nPayroll in millions: 83.31\nWins: 94', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 5}, lookup_index=0), Document(page_content='MLB Team: Athletics\nPayroll in millions: 55.37\nWins: 94', lookup_str='', metadata={'source':


in millions: 55.37\nWins: 94', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 6}, lookup_index=0), Document(page_content='MLB Team: Rangers\nPayroll in millions: 120.51\nWins: 93', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 7}, lookup_index=0), Document(page_content='MLB Team: Orioles\nPayroll in millions: 81.43\nWins: 93', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 8}, lookup_index=0), Document(page_content='MLB Team: Rays\nPayroll in millions: 64.17\nWins: 90', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 9}, lookup_index=0), Document(page_content='MLB Team: Angels\nPayroll in millions: 154.49\nWins: 89', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 10}, lookup_index=0), Document(page_content='MLB Team: Tigers\nPayroll in millions: 132.30\nWins: 88', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 11}, lookup_index=0), Document(page_content='MLB Team: Cardinals\nPayroll in millions: 110.30\nWins: 88', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 12}, lookup_index=0), Document(page_content='MLB Team: Dodgers\nPayroll in millions: 95.14\nWins: 86', lookup_str='', metadata={'source':


in millions: 95.14\nWins: 86', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 13}, lookup_index=0), Document(page_content='MLB Team: White Sox\nPayroll in millions: 96.92\nWins: 85', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 14}, lookup_index=0), Document(page_content='MLB Team: Brewers\nPayroll in millions: 97.65\nWins: 83', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 15}, lookup_index=0), Document(page_content='MLB Team: Phillies\nPayroll in millions: 174.54\nWins: 81', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 16}, lookup_index=0), Document(page_content='MLB Team: Diamondbacks\nPayroll in millions: 74.28\nWins: 81', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 17}, lookup_index=0), Document(page_content='MLB Team: Pirates\nPayroll in millions: 63.43\nWins: 79', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 18}, lookup_index=0), Document(page_content='MLB Team: Padres\nPayroll in millions: 55.24\nWins: 76', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 19}, lookup_index=0), Document(page_content='MLB Team: Mariners\nPayroll in millions: 81.97\nWins: 75', lookup_str='', metadata={'source':


in millions: 81.97\nWins: 75', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 20}, lookup_index=0), Document(page_content='MLB Team: Mets\nPayroll in millions: 93.35\nWins: 74', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 21}, lookup_index=0), Document(page_content='MLB Team: Blue Jays\nPayroll in millions: 75.48\nWins: 73', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 22}, lookup_index=0), Document(page_content='MLB Team: Royals\nPayroll in millions: 60.91\nWins: 72', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 23}, lookup_index=0), Document(page_content='MLB Team: Marlins\nPayroll in millions: 118.07\nWins: 69', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 24}, lookup_index=0), Document(page_content='MLB Team: Red Sox\nPayroll in millions: 173.18\nWins: 69', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 25}, lookup_index=0), Document(page_content='MLB Team: Indians\nPayroll in millions: 78.43\nWins: 68', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 26}, lookup_index=0), Document(page_content='MLB Team: Twins\nPayroll in millions: 94.08\nWins: 66', lookup_str='', metadata={'source':


in millions: 94.08\nWins: 66', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 27}, lookup_index=0), Document(page_content='MLB Team: Rockies\nPayroll in millions: 78.06\nWins: 64', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 28}, lookup_index=0), Document(page_content='MLB Team: Cubs\nPayroll in millions: 88.19\nWins: 61', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 29}, lookup_index=0), Document(page_content='MLB Team: Astros\nPayroll in millions: 60.65\nWins: 55', lookup_str='', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 30}, lookup_index=0)]Specify a column to identify the document sourceâ€‹Use the source_column argument to specify a source for the document created from each row. Otherwise file_path will be used as the source for all documents created from the CSV file.This is useful when using documents loaded from CSV files for chains that answer questions using sources.loader = CSVLoader(file_path='./example_data/mlb_teams_2012.csv', source_column="Team")data = loader.load()print(data)    [Document(page_content='Team: Nationals\n"Payroll (millions)": 81.34\n"Wins": 98', lookup_str='', metadata={'source': 'Nationals', 'row': 0}, lookup_index=0), Document(page_content='Team: Reds\n"Payroll (millions)": 82.20\n"Wins": 97', lookup_str='', metadata={'source': 'Reds', 'row': 1}, lookup_index=0),


metadata={'source': 'Reds', 'row': 1}, lookup_index=0), Document(page_content='Team: Yankees\n"Payroll (millions)": 197.96\n"Wins": 95', lookup_str='', metadata={'source': 'Yankees', 'row': 2}, lookup_index=0), Document(page_content='Team: Giants\n"Payroll (millions)": 117.62\n"Wins": 94', lookup_str='', metadata={'source': 'Giants', 'row': 3}, lookup_index=0), Document(page_content='Team: Braves\n"Payroll (millions)": 83.31\n"Wins": 94', lookup_str='', metadata={'source': 'Braves', 'row': 4}, lookup_index=0), Document(page_content='Team: Athletics\n"Payroll (millions)": 55.37\n"Wins": 94', lookup_str='', metadata={'source': 'Athletics', 'row': 5}, lookup_index=0), Document(page_content='Team: Rangers\n"Payroll (millions)": 120.51\n"Wins": 93', lookup_str='', metadata={'source': 'Rangers', 'row': 6}, lookup_index=0), Document(page_content='Team: Orioles\n"Payroll (millions)": 81.43\n"Wins": 93', lookup_str='', metadata={'source': 'Orioles', 'row': 7}, lookup_index=0), Document(page_content='Team: Rays\n"Payroll (millions)": 64.17\n"Wins": 90', lookup_str='', metadata={'source': 'Rays', 'row': 8}, lookup_index=0), Document(page_content='Team: Angels\n"Payroll (millions)": 154.49\n"Wins": 89', lookup_str='', metadata={'source': 'Angels', 'row': 9},


89', lookup_str='', metadata={'source': 'Angels', 'row': 9}, lookup_index=0), Document(page_content='Team: Tigers\n"Payroll (millions)": 132.30\n"Wins": 88', lookup_str='', metadata={'source': 'Tigers', 'row': 10}, lookup_index=0), Document(page_content='Team: Cardinals\n"Payroll (millions)": 110.30\n"Wins": 88', lookup_str='', metadata={'source': 'Cardinals', 'row': 11}, lookup_index=0), Document(page_content='Team: Dodgers\n"Payroll (millions)": 95.14\n"Wins": 86', lookup_str='', metadata={'source': 'Dodgers', 'row': 12}, lookup_index=0), Document(page_content='Team: White Sox\n"Payroll (millions)": 96.92\n"Wins": 85', lookup_str='', metadata={'source': 'White Sox', 'row': 13}, lookup_index=0), Document(page_content='Team: Brewers\n"Payroll (millions)": 97.65\n"Wins": 83', lookup_str='', metadata={'source': 'Brewers', 'row': 14}, lookup_index=0), Document(page_content='Team: Phillies\n"Payroll (millions)": 174.54\n"Wins": 81', lookup_str='', metadata={'source': 'Phillies', 'row': 15}, lookup_index=0), Document(page_content='Team: Diamondbacks\n"Payroll (millions)": 74.28\n"Wins": 81', lookup_str='', metadata={'source': 'Diamondbacks', 'row': 16}, lookup_index=0), Document(page_content='Team: Pirates\n"Payroll (millions)": 63.43\n"Wins": 79', lookup_str='', metadata={'source':


63.43\n"Wins": 79', lookup_str='', metadata={'source': 'Pirates', 'row': 17}, lookup_index=0), Document(page_content='Team: Padres\n"Payroll (millions)": 55.24\n"Wins": 76', lookup_str='', metadata={'source': 'Padres', 'row': 18}, lookup_index=0), Document(page_content='Team: Mariners\n"Payroll (millions)": 81.97\n"Wins": 75', lookup_str='', metadata={'source': 'Mariners', 'row': 19}, lookup_index=0), Document(page_content='Team: Mets\n"Payroll (millions)": 93.35\n"Wins": 74', lookup_str='', metadata={'source': 'Mets', 'row': 20}, lookup_index=0), Document(page_content='Team: Blue Jays\n"Payroll (millions)": 75.48\n"Wins": 73', lookup_str='', metadata={'source': 'Blue Jays', 'row': 21}, lookup_index=0), Document(page_content='Team: Royals\n"Payroll (millions)": 60.91\n"Wins": 72', lookup_str='', metadata={'source': 'Royals', 'row': 22}, lookup_index=0), Document(page_content='Team: Marlins\n"Payroll (millions)": 118.07\n"Wins": 69', lookup_str='', metadata={'source': 'Marlins', 'row': 23}, lookup_index=0), Document(page_content='Team: Red Sox\n"Payroll (millions)": 173.18\n"Wins": 69', lookup_str='', metadata={'source': 'Red Sox', 'row': 24}, lookup_index=0), Document(page_content='Team: Indians\n"Payroll (millions)": 78.43\n"Wins": 68',


Indians\n"Payroll (millions)": 78.43\n"Wins": 68', lookup_str='', metadata={'source': 'Indians', 'row': 25}, lookup_index=0), Document(page_content='Team: Twins\n"Payroll (millions)": 94.08\n"Wins": 66', lookup_str='', metadata={'source': 'Twins', 'row': 26}, lookup_index=0), Document(page_content='Team: Rockies\n"Payroll (millions)": 78.06\n"Wins": 64', lookup_str='', metadata={'source': 'Rockies', 'row': 27}, lookup_index=0), Document(page_content='Team: Cubs\n"Payroll (millions)": 88.19\n"Wins": 61', lookup_str='', metadata={'source': 'Cubs', 'row': 28}, lookup_index=0), Document(page_content='Team: Astros\n"Payroll (millions)": 60.65\n"Wins": 55', lookup_str='', metadata={'source': 'Astros', 'row': 29}, lookup_index=0)]PreviousDocument loadersNextFile DirectoryCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Vector stores | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionDocument loadersDocument transformersText embedding modelsVector storesRetrieversChainsMemoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesData connectionVector storesOn this pageVector storesinfoHead to Integrations for documentation on built-in integrations with 3rd-party vector stores.One of the most common ways to store and search over unstructured data is to embed it and store the resulting embedding
vectors, and then at query time to embed the unstructured query and retrieve the embedding vectors that are
'most similar' to the embedded query. A vector store takes care of storing embedded data and performing vector search


for you.Get startedâ€‹This walkthrough showcases basic functionality related to VectorStores. A key part of working with vector stores is creating the vector to put in them, which is usually created via embeddings. Therefore, it is recommended that you familiarize yourself with the text embedding model interfaces before diving into this.There are many great vector store options, here are a few that are free, open-source, and run entirely on your local machine. Review all integrations for many great hosted offerings.ChromaFAISSLanceThis walkthrough uses the chroma vector database, which runs on your local machine as a library.pip install chromadbWe want to use OpenAIEmbeddings so we have to get the OpenAI API Key.import osimport getpassos.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key:')from langchain.document_loaders import TextLoaderfrom langchain.embeddings.openai import OpenAIEmbeddingsfrom langchain.text_splitter import CharacterTextSplitterfrom langchain.vectorstores import Chroma# Load the document, split it into chunks, embed each chunk and load it into the vector store.raw_documents = TextLoader('../../../state_of_the_union.txt').load()text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)documents = text_splitter.split_documents(raw_documents)db = Chroma.from_documents(documents, OpenAIEmbeddings())This walkthrough uses the FAISS vector database, which makes use of the Facebook AI Similarity Search (FAISS) library.pip install faiss-cpuWe want to use OpenAIEmbeddings so we have to get the OpenAI API Key.import osimport getpassos.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key:')from langchain.document_loaders import TextLoaderfrom langchain.embeddings.openai import OpenAIEmbeddingsfrom langchain.text_splitter import CharacterTextSplitterfrom langchain.vectorstores import


langchain.text_splitter import CharacterTextSplitterfrom langchain.vectorstores import FAISS# Load the document, split it into chunks, embed each chunk and load it into the vector store.raw_documents = TextLoader('../../../state_of_the_union.txt').load()text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)documents = text_splitter.split_documents(raw_documents)db = FAISS.from_documents(documents, OpenAIEmbeddings())This notebook shows how to use functionality related to the LanceDB vector database based on the Lance data format.pip install lancedbWe want to use OpenAIEmbeddings so we have to get the OpenAI API Key.import osimport getpassos.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key:')from langchain.document_loaders import TextLoaderfrom langchain.embeddings.openai import OpenAIEmbeddingsfrom langchain.text_splitter import CharacterTextSplitterfrom langchain.vectorstores import LanceDBimport lancedbdb = lancedb.connect("/tmp/lancedb")table = db.create_table(    "my_table",    data=[        {            "vector": embeddings.embed_query("Hello World"),            "text": "Hello World",            "id": "1",        }    ],    mode="overwrite",)# Load the document, split it into chunks, embed each chunk and load it into the vector store.raw_documents = TextLoader('../../../state_of_the_union.txt').load()text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)documents = text_splitter.split_documents(raw_documents)db = LanceDB.from_documents(documents, OpenAIEmbeddings(),


= LanceDB.from_documents(documents, OpenAIEmbeddings(), connection=table)Similarity searchâ€‹query = "What did the president say about Ketanji Brown Jackson"docs = db.similarity_search(query)print(docs[0].page_content)    Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while youâ€™re at it, pass the Disclose Act so Americans can know who is funding our elections.    Tonight, Iâ€™d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyerâ€”an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.    One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.    And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nationâ€™s top legal minds, who will continue Justice Breyerâ€™s legacy of excellence.Similarity search by vectorâ€‹It is also possible to do a search for documents similar to a given embedding vector using similarity_search_by_vector which accepts an embedding vector as a parameter instead of a string.embedding_vector = OpenAIEmbeddings().embed_query(query)docs = db.similarity_search_by_vector(embedding_vector)print(docs[0].page_content)The query is the same, and so the result is also the same.    Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while youâ€™re at it, pass the Disclose Act so Americans can know who is funding our elections.    Tonight, Iâ€™d like to honor someone who has dedicated his life


elections.    Tonight, Iâ€™d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyerâ€”an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.    One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.    And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nationâ€™s top legal minds, who will continue Justice Breyerâ€™s legacy of excellence.Asynchronous operationsâ€‹Vector stores are usually run as a separate service that requires some IO operations, and therefore they might be called asynchronously. That gives performance benefits as you don't waste time waiting for responses from external services. That might also be important if you work with an asynchronous framework, such as FastAPI.Langchain supports async operation on vector stores. All the methods might be called using their async counterparts, with the prefix a, meaning async.Qdrant is a vector store, which supports all the async operations, thus it will be used in this walkthrough.pip install qdrant-clientfrom langchain.vectorstores import QdrantCreate a vector store asynchronouslyâ€‹db = await Qdrant.afrom_documents(documents, embeddings, "http://localhost:6333")Similarity searchâ€‹query = "What did the president say about Ketanji Brown Jackson"docs = await db.asimilarity_search(query)print(docs[0].page_content)    Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while youâ€™re at it, pass the Disclose Act so Americans can know who is funding our elections.    Tonight, Iâ€™d like to


Americans can know who is funding our elections.    Tonight, Iâ€™d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyerâ€”an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.    One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.    And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nationâ€™s top legal minds, who will continue Justice Breyerâ€™s legacy of excellence.Similarity search by vectorâ€‹embedding_vector = embeddings.embed_query(query)docs = await db.asimilarity_search_by_vector(embedding_vector)Maximum marginal relevance search (MMR)â€‹Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. It is also supported in async API.query = "What did the president say about Ketanji Brown Jackson"found_docs = await qdrant.amax_marginal_relevance_search(query, k=2, fetch_k=10)for i, doc in enumerate(found_docs):    print(f"{i + 1}.", doc.page_content, "\n")1. Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while youâ€™re at it, pass the Disclose Act so Americans can know who is funding our elections.Tonight, Iâ€™d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyerâ€”an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.One of the most serious constitutional responsibilities a President has is nominating someone


thank you for your service.One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nationâ€™s top legal minds, who will continue Justice Breyerâ€™s legacy of excellence.2. We canâ€™t change how divided weâ€™ve been. But we can change how we move forwardâ€”on COVID-19 and other issues we must face together.I recently visited the New York City Police Department days after the funerals of Officer Wilbert Mora and his partner, Officer Jason Rivera.They were responding to a 9-1-1 call when a man shot and killed them with a stolen gun.Officer Mora was 27 years old.Officer Rivera was 22.Both Dominican Americans whoâ€™d grown up on the same streets they later chose to patrol as police officers.I spoke with their families and told them that we are forever in debt for their sacrifice, and we will carry on their mission to restore the trust and safety every community deserves.Iâ€™ve worked on these issues a long time.I know what works: Investing in crime preventionand community police officers whoâ€™ll walk the beat, whoâ€™ll know the neighborhood, and who can restore trust and safety.PreviousText embedding modelsNextRetrieversGet startedAsynchronous operationsCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Retrievers | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionDocument loadersDocument transformersText embedding modelsVector storesRetrieversMultiQueryRetrieverContextual compressionEnsemble RetrieverSelf-queryingTime-weighted vector store retrieverVector store-backed retrieverChainsMemoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesData connectionRetrieversOn this pageRetrieversinfoHead to Integrations for documentation on built-in retriever integrations with 3rd-party tools.A retriever is an interface that returns documents given an unstructured query. It is more general than a vector store.
A retriever does not need to be able to store documents, only to return (or retrieve) it. Vector stores can be used


as the backbone of a retriever, but there are other types of retrievers as well.Get startedâ€‹The public API of the BaseRetriever class in LangChain is as follows:from abc import ABC, abstractmethodfrom typing import Any, Listfrom langchain.schema import Documentfrom langchain.callbacks.manager import Callbacksclass BaseRetriever(ABC):    ...    def get_relevant_documents(        self, query: str, *, callbacks: Callbacks = None, **kwargs: Any    ) -> List[Document]:        """Retrieve documents relevant to a query.        Args:            query: string to find relevant documents for            callbacks: Callback manager or list of callbacks        Returns:            List of relevant documents        """        ...    async def aget_relevant_documents(        self, query: str, *, callbacks: Callbacks = None, **kwargs: Any    ) -> List[Document]:        """Asynchronously get documents relevant to a query.        Args:            query: string to find relevant documents for            callbacks: Callback manager or list of callbacks        Returns:            List of relevant documents        """        ...It's that simple! You can call get_relevant_documents or the async get_relevant_documents methods to retrieve documents relevant to a query, where "relevance" is defined by
the specific retriever object you are calling.Of course, we also help construct what we think useful Retrievers are. The main type of Retriever that we focus on is a Vectorstore retriever. We will focus on that for the rest of this guide.In order to understand what a vectorstore retriever is, it's important to understand what a Vectorstore is. So let's look at that.By default, LangChain uses Chroma as the vectorstore to index and search embeddings. To walk through this tutorial, we'll first need to install chromadb.pip install chromadbThis example showcases question answering over documents.


We have chosen this as the example for getting started because it nicely combines a lot of different elements (Text splitters, embeddings, vectorstores) and then also shows how to use them in a chain.Question answering over documents consists of four steps:Create an indexCreate a Retriever from that indexCreate a question answering chainAsk questions!Each of the steps has multiple sub steps and potential configurations. In this notebook we will primarily focus on (1). We will start by showing the one-liner for doing so, but then break down what is actually going on.First, let's import some common classes we'll use no matter what.from langchain.chains import RetrievalQAfrom langchain.llms import OpenAINext in the generic setup, let's specify the document loader we want to use. You can download the state_of_the_union.txt file herefrom langchain.document_loaders import TextLoaderloader = TextLoader('../state_of_the_union.txt', encoding='utf8')One Line Index Creationâ€‹To get started as quickly as possible, we can use the VectorstoreIndexCreator.from langchain.indexes import VectorstoreIndexCreatorindex = VectorstoreIndexCreator().from_loaders([loader])    Running Chroma using direct local API.    Using DuckDB in-memory for database. Data will be transient.Now that the index is created, we can use it to ask questions of the data! Note that under the hood this is actually doing a few steps as well, which we will cover later in this guide.query = "What did the president say about Ketanji Brown Jackson"index.query(query)    " The president said that Ketanji Brown Jackson is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and has received a broad range of support from the


He also said that she is a consensus builder and has received a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans."query = "What did the president say about Ketanji Brown Jackson"index.query_with_sources(query)    {'question': 'What did the president say about Ketanji Brown Jackson',     'answer': " The president said that he nominated Circuit Court of Appeals Judge Ketanji Brown Jackson, one of the nation's top legal minds, to continue Justice Breyer's legacy of excellence, and that she has received a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans.\n",     'sources': '../state_of_the_union.txt'}What is returned from the VectorstoreIndexCreator is VectorStoreIndexWrapper, which provides these nice query and query_with_sources functionality. If we just wanted to access the vectorstore directly, we can also do that.index.vectorstore    <langchain.vectorstores.chroma.Chroma at 0x119aa5940>If we then want to access the VectorstoreRetriever, we can do that with:index.vectorstore.as_retriever()    VectorStoreRetriever(vectorstore=<langchain.vectorstores.chroma.Chroma object at 0x119aa5940>, search_kwargs={})Walkthroughâ€‹Okay, so what's actually going on? How is this index getting created?A lot of the magic is being hid in this VectorstoreIndexCreator. What is this doing?There are three main steps going on after the documents are loaded:Splitting documents into chunksCreating embeddings for each documentStoring documents and embeddings in a vectorstoreLet's walk through this in codedocuments = loader.load()Next, we will split the documents into chunks.from langchain.text_splitter import CharacterTextSplittertext_splitter =


split the documents into chunks.from langchain.text_splitter import CharacterTextSplittertext_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)texts = text_splitter.split_documents(documents)We will then select which embeddings we want to use.from langchain.embeddings import OpenAIEmbeddingsembeddings = OpenAIEmbeddings()We now create the vectorstore to use as the index.from langchain.vectorstores import Chromadb = Chroma.from_documents(texts, embeddings)    Running Chroma using direct local API.    Using DuckDB in-memory for database. Data will be transient.So that's creating the index. Then, we expose this index in a retriever interface.retriever = db.as_retriever()Then, as before, we create a chain and use it to answer questions!qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=retriever)query = "What did the president say about Ketanji Brown Jackson"qa.run(query)    " The President said that Judge Ketanji Brown Jackson is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He said she is a consensus builder and has received a broad range of support from organizations such as the Fraternal Order of Police and former judges appointed by Democrats and Republicans."VectorstoreIndexCreator is just a wrapper around all this logic. It is configurable in the text splitter it uses, the embeddings it uses, and the vectorstore it uses. For example, you can configure it as below:index_creator = VectorstoreIndexCreator(    vectorstore_cls=Chroma,    embedding=OpenAIEmbeddings(),    text_splitter=CharacterTextSplitter(chunk_size=1000, chunk_overlap=0))Hopefully this highlights


chunk_overlap=0))Hopefully this highlights what is going on under the hood of VectorstoreIndexCreator. While we think it's important to have a simple way to create indexes, we also think it's important to understand what's going on under the hood.PreviousVector storesNextMultiQueryRetrieverGet startedCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Contextual compression | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionDocument loadersDocument transformersText embedding modelsVector storesRetrieversMultiQueryRetrieverContextual compressionEnsemble RetrieverSelf-queryingTime-weighted vector store retrieverVector store-backed retrieverChainsMemoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesData connectionRetrieversContextual compressionOn this pageContextual compressionOne challenge with retrieval is that usually you don't know the specific queries your document storage system will face when you ingest data into the system. This means that the information most relevant to a query may be buried in a document with a lot of irrelevant text. Passing that full document through your application can lead to more expensive LLM calls and poorer responses.Contextual compression is meant to fix this. The idea is simple: instead of immediately returning retrieved documents as-is, you can compress them using the context of the given query, so that only the relevant information is returned. â€œCompressingâ€� here refers to both compressing the contents of an individual document and filtering out documents wholesale.To use the Contextual Compression Retriever, you'll need:a base Retrievera Document CompressorThe Contextual Compression Retriever passes queries to the base Retriever, takes the initial documents and passes them through the Document Compressor. The Document Compressor takes a list of Documents and shortens it by reducing the contents of Documents or dropping Documents altogether.Get startedâ€‹# Helper function for printing docsdef pretty_print_docs(docs):    print(f"\n{'-' * 100}\n".join([f"Document {i+1}:\n\n" + d.page_content for i, d in enumerate(docs)]))Using a vanilla vector store


+ d.page_content for i, d in enumerate(docs)]))Using a vanilla vector store retrieverâ€‹Let's start by initializing a simple vector store retriever and storing the 2023 State of the Union speech (in chunks). We can see that given an example question our retriever returns one or two relevant docs and a few irrelevant docs. And even the relevant docs have a lot of irrelevant information in them.from langchain.text_splitter import CharacterTextSplitterfrom langchain.embeddings import OpenAIEmbeddingsfrom langchain.document_loaders import TextLoaderfrom langchain.vectorstores import FAISSdocuments = TextLoader('../../../state_of_the_union.txt').load()text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)texts = text_splitter.split_documents(documents)retriever = FAISS.from_documents(texts, OpenAIEmbeddings()).as_retriever()docs = retriever.get_relevant_documents("What did the president say about Ketanji Brown Jackson")pretty_print_docs(docs)    Document 1:        Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while youâ€™re at it, pass the Disclose Act so Americans can know who is funding our elections.         Tonight, Iâ€™d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyerâ€”an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.         One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.         And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown


I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nationâ€™s top legal minds, who will continue Justice Breyerâ€™s legacy of excellence.    ----------------------------------------------------------------------------------------------------    Document 2:        A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder. Since sheâ€™s been nominated, sheâ€™s received a broad range of supportâ€”from the Fraternal Order of Police to former judges appointed by Democrats and Republicans.         And if we are to advance liberty and justice, we need to secure the Border and fix the immigration system.         We can do both. At our border, weâ€™ve installed new technology like cutting-edge scanners to better detect drug smuggling.          Weâ€™ve set up joint patrols with Mexico and Guatemala to catch more human traffickers.          Weâ€™re putting in place dedicated immigration judges so families fleeing persecution and violence can have their cases heard faster.         Weâ€™re securing commitments and supporting partners in South and Central America to host more refugees and secure their own borders.    ----------------------------------------------------------------------------------------------------    Document 3:        And for our LGBTQ+ Americans, letâ€™s finally get the bipartisan Equality Act to my desk. The onslaught of state laws targeting transgender Americans and their families is wrong.         As I said last year, especially to our younger transgender Americans, I will always have your back as your President, so you can be yourself and reach your God-given potential.         While it often


yourself and reach your God-given potential.         While it often appears that we never agree, that isnâ€™t true. I signed 80 bipartisan bills into law last year. From preventing government shutdowns to protecting Asian-Americans from still-too-common hate crimes to reforming military justice.         And soon, weâ€™ll strengthen the Violence Against Women Act that I first wrote three decades ago. It is important for us to show the nation that we can come together and do big things.         So tonight Iâ€™m offering a Unity Agenda for the Nation. Four big things we can do together.          First, beat the opioid epidemic.    ----------------------------------------------------------------------------------------------------    Document 4:        Tonight, Iâ€™m announcing a crackdown on these companies overcharging American businesses and consumers.         And as Wall Street firms take over more nursing homes, quality in those homes has gone down and costs have gone up.          That ends on my watch.         Medicare is going to set higher standards for nursing homes and make sure your loved ones get the care they deserve and expect.         Weâ€™ll also cut costs and keep the economy going strong by giving workers a fair shot, provide more training and apprenticeships, hire them based on their skills not degrees.         Letâ€™s pass the Paycheck Fairness Act and paid leave.          Raise the minimum wage to $15 an hour and extend the Child Tax Credit, so no one has to raise a family in poverty.         Letâ€™s increase Pell Grants and increase our historic support of


Letâ€™s increase Pell Grants and increase our historic support of HBCUs, and invest in what Jillâ€”our First Lady who teaches full-timeâ€”calls Americaâ€™s best-kept secret: community colleges.Adding contextual compression with an LLMChainExtractorâ€‹Now let's wrap our base retriever with a ContextualCompressionRetriever. We'll add an LLMChainExtractor, which will iterate over the initially returned documents and extract from each only the content that is relevant to the query.from langchain.llms import OpenAIfrom langchain.retrievers import ContextualCompressionRetrieverfrom langchain.retrievers.document_compressors import LLMChainExtractorllm = OpenAI(temperature=0)compressor = LLMChainExtractor.from_llm(llm)compression_retriever = ContextualCompressionRetriever(base_compressor=compressor, base_retriever=retriever)compressed_docs = compression_retriever.get_relevant_documents("What did the president say about Ketanji Jackson Brown")pretty_print_docs(compressed_docs)    Document 1:        "One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.         And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nationâ€™s top legal minds, who will continue Justice Breyerâ€™s legacy of excellence."    ----------------------------------------------------------------------------------------------------    Document 2:        "A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder. Since sheâ€™s been nominated, sheâ€™s received a broad range of supportâ€”from the


been nominated, sheâ€™s received a broad range of supportâ€”from the Fraternal Order of Police to former judges appointed by Democrats and Republicans."More built-in compressors: filtersâ€‹LLMChainFilterâ€‹The LLMChainFilter is slightly simpler but more robust compressor that uses an LLM chain to decide which of the initially retrieved documents to filter out and which ones to return, without manipulating the document contents.from langchain.retrievers.document_compressors import LLMChainFilter_filter = LLMChainFilter.from_llm(llm)compression_retriever = ContextualCompressionRetriever(base_compressor=_filter, base_retriever=retriever)compressed_docs = compression_retriever.get_relevant_documents("What did the president say about Ketanji Jackson Brown")pretty_print_docs(compressed_docs)    Document 1:        Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while youâ€™re at it, pass the Disclose Act so Americans can know who is funding our elections.         Tonight, Iâ€™d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyerâ€”an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.         One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.         And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nationâ€™s top legal minds, who will continue Justice Breyerâ€™s legacy of excellence.EmbeddingsFilterâ€‹Making an extra LLM call over


legacy of excellence.EmbeddingsFilterâ€‹Making an extra LLM call over each retrieved document is expensive and slow. The EmbeddingsFilter provides a cheaper and faster option by embedding the documents and query and only returning those documents which have sufficiently similar embeddings to the query.from langchain.embeddings import OpenAIEmbeddingsfrom langchain.retrievers.document_compressors import EmbeddingsFilterembeddings = OpenAIEmbeddings()embeddings_filter = EmbeddingsFilter(embeddings=embeddings, similarity_threshold=0.76)compression_retriever = ContextualCompressionRetriever(base_compressor=embeddings_filter, base_retriever=retriever)compressed_docs = compression_retriever.get_relevant_documents("What did the president say about Ketanji Jackson Brown")pretty_print_docs(compressed_docs)    Document 1:        Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while youâ€™re at it, pass the Disclose Act so Americans can know who is funding our elections.         Tonight, Iâ€™d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyerâ€”an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.         One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.         And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nationâ€™s top legal minds, who will continue Justice Breyerâ€™s legacy of excellence.    ----------------------------------------------------------------------------------------------------    Document 2:        A former top litigator


Document 2:        A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder. Since sheâ€™s been nominated, sheâ€™s received a broad range of supportâ€”from the Fraternal Order of Police to former judges appointed by Democrats and Republicans.         And if we are to advance liberty and justice, we need to secure the Border and fix the immigration system.         We can do both. At our border, weâ€™ve installed new technology like cutting-edge scanners to better detect drug smuggling.          Weâ€™ve set up joint patrols with Mexico and Guatemala to catch more human traffickers.          Weâ€™re putting in place dedicated immigration judges so families fleeing persecution and violence can have their cases heard faster.         Weâ€™re securing commitments and supporting partners in South and Central America to host more refugees and secure their own borders.    ----------------------------------------------------------------------------------------------------    Document 3:        And for our LGBTQ+ Americans, letâ€™s finally get the bipartisan Equality Act to my desk. The onslaught of state laws targeting transgender Americans and their families is wrong.         As I said last year, especially to our younger transgender Americans, I will always have your back as your President, so you can be yourself and reach your God-given potential.         While it often appears that we never agree, that isnâ€™t true. I signed 80 bipartisan bills into law last year. From preventing government shutdowns to protecting Asian-Americans from still-too-common hate crimes to reforming military justice.


from still-too-common hate crimes to reforming military justice.         And soon, weâ€™ll strengthen the Violence Against Women Act that I first wrote three decades ago. It is important for us to show the nation that we can come together and do big things.         So tonight Iâ€™m offering a Unity Agenda for the Nation. Four big things we can do together.          First, beat the opioid epidemic.Stringing compressors and document transformers togetherUsing the DocumentCompressorPipeline we can also easily combine multiple compressors in sequence. Along with compressors we can add BaseDocumentTransformers to our pipeline, which don't perform any contextual compression but simply perform some transformation on a set of documents. For example TextSplitters can be used as document transformers to split documents into smaller pieces, and the EmbeddingsRedundantFilter can be used to filter out redundant documents based on embedding similarity between documents.Below we create a compressor pipeline by first splitting our docs into smaller chunks, then removing redundant documents, and then filtering based on relevance to the query.from langchain.document_transformers import EmbeddingsRedundantFilterfrom langchain.retrievers.document_compressors import DocumentCompressorPipelinefrom langchain.text_splitter import CharacterTextSplittersplitter = CharacterTextSplitter(chunk_size=300, chunk_overlap=0, separator=". ")redundant_filter = EmbeddingsRedundantFilter(embeddings=embeddings)relevant_filter = EmbeddingsFilter(embeddings=embeddings, similarity_threshold=0.76)pipeline_compressor = DocumentCompressorPipeline(    transformers=[splitter, redundant_filter, relevant_filter])compression_retriever = ContextualCompressionRetriever(base_compressor=pipeline_compressor, base_retriever=retriever)compressed_docs = compression_retriever.get_relevant_documents("What did the president say about Ketanji Jackson


= compression_retriever.get_relevant_documents("What did the president say about Ketanji Jackson Brown")pretty_print_docs(compressed_docs)    Document 1:        One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.         And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson    ----------------------------------------------------------------------------------------------------    Document 2:        As I said last year, especially to our younger transgender Americans, I will always have your back as your President, so you can be yourself and reach your God-given potential.         While it often appears that we never agree, that isnâ€™t true. I signed 80 bipartisan bills into law last year    ----------------------------------------------------------------------------------------------------    Document 3:        A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builderPreviousMultiQueryRetrieverNextEnsemble RetrieverGet startedCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


MultiQueryRetriever | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionDocument loadersDocument transformersText embedding modelsVector storesRetrieversMultiQueryRetrieverContextual compressionEnsemble RetrieverSelf-queryingTime-weighted vector store retrieverVector store-backed retrieverChainsMemoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesData connectionRetrieversMultiQueryRetrieverMultiQueryRetrieverDistance-based vector database retrieval embeds (represents) queries in high-dimensional space and finds similar embedded documents based on "distance". But, retrieval may produce difference results with subtle changes in query wording or if the embeddings do not capture the semantics of the data well. Prompt engineering / tuning is sometimes done to manually address these problems, but can be tedious.The MultiQueryRetriever automates the process of prompt tuning by using an LLM to generate multiple queries from different perspectives for a given user input query. For each query, it retrieves a set of relevant documents and takes the unique union across all queries to get a larger set of potentially relevant documents. By generating multiple perspectives on the same question, the MultiQueryRetriever might be able to overcome some of the limitations of the distance-based retrieval and get a richer set of results.# Build a sample vectorDBfrom langchain.vectorstores import Chromafrom langchain.document_loaders import WebBaseLoaderfrom langchain.embeddings.openai import OpenAIEmbeddingsfrom langchain.text_splitter import RecursiveCharacterTextSplitter# Load blog postloader = WebBaseLoader("https://lilianweng.github.io/posts/2023-06-23-agent/")data = loader.load()# Splittext_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)splits = text_splitter.split_documents(data)#


chunk_overlap=0)splits = text_splitter.split_documents(data)# VectorDBembedding = OpenAIEmbeddings()vectordb = Chroma.from_documents(documents=splits, embedding=embedding)Simple usageSpecify the LLM to use for query generation, and the retriver will do the rest.from langchain.chat_models import ChatOpenAIfrom langchain.retrievers.multi_query import MultiQueryRetrieverquestion = "What are the approaches to Task Decomposition?"llm = ChatOpenAI(temperature=0)retriever_from_llm = MultiQueryRetriever.from_llm(    retriever=vectordb.as_retriever(), llm=llm)# Set logging for the queriesimport logginglogging.basicConfig()logging.getLogger("langchain.retrievers.multi_query").setLevel(logging.INFO)unique_docs = retriever_from_llm.get_relevant_documents(query=question)len(unique_docs)    INFO:langchain.retrievers.multi_query:Generated queries: ['1. How can Task Decomposition be approached?', '2. What are the different methods for Task Decomposition?', '3. What are the various approaches to decomposing tasks?']    5Supplying your own promptYou can also supply a prompt along with an output parser to split the results into a list of queries.from typing import Listfrom langchain import LLMChainfrom pydantic import BaseModel, Fieldfrom langchain.prompts import PromptTemplatefrom langchain.output_parsers import PydanticOutputParser# Output parser will split the LLM result into a list of queriesclass LineList(BaseModel):    # "lines" is the key (attribute name) of the parsed output    lines: List[str] = Field(description="Lines of text")class LineListOutputParser(PydanticOutputParser):    def __init__(self) -> None: