does not know the answer to a question, it truthfully says it does not know. The AI ONLY uses information contained in the "Relevant Information" section and does not hallucinate.        Relevant Information:                Conversation:    Human: Hi, what's up?    AI:        > Finished chain.    " Hi there! I'm doing great. I'm currently in the process of learning about the world around me. I'm learning about different cultures, languages, and customs. It's really fascinating! How about you?"conversation_with_kg.predict(    input="My name is James and I'm helping Will. He's an engineer.")            > Entering new ConversationChain chain...    Prompt after formatting:    The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context.     If the AI does not know the answer to a question, it truthfully says it does not know. The AI ONLY uses information contained in the "Relevant Information" section and does not hallucinate.        Relevant Information:                Conversation:    Human: My name is James and I'm helping Will. He's an engineer.    AI:        > Finished chain.    " Hi James, it's nice to meet you. I'm an AI and I understand you're helping Will, the engineer. What kind of engineering does he do?"conversation_with_kg.predict(input="What do you know about Will?")            > Entering new ConversationChain


Will?")            > Entering new ConversationChain chain...    Prompt after formatting:    The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context.     If the AI does not know the answer to a question, it truthfully says it does not know. The AI ONLY uses information contained in the "Relevant Information" section and does not hallucinate.        Relevant Information:        On Will: Will is an engineer.        Conversation:    Human: What do you know about Will?    AI:        > Finished chain.    ' Will is an engineer.'PreviousEntity memoryNextHow to use multiple memory classes in the same chainUsing in a chainCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


How to customize conversational memory | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryHow to add Memory to an LLMChainHow to add memory to a Multi-Input ChainHow to add Memory to an AgentAdding Message Memory backed by a database to an AgentConversation buffer memoryConversation buffer window memoryHow to customize conversational memoryHow to create a custom Memory classEntity memoryConversation Knowledge Graph MemoryHow to use multiple memory classes in the same chainConversation summary memoryConversationSummaryBufferMemoryConversationTokenBufferMemoryVector store-backed memoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesMemoryHow to customize conversational memoryOn this pageHow to customize conversational memoryThis notebook walks through a few ways to customize conversational memory.from langchain.llms import OpenAIfrom langchain.chains import ConversationChainfrom langchain.memory import ConversationBufferMemoryllm = OpenAI(temperature=0)AI Prefixâ€‹The first way to do so is by changing the AI prefix in the conversation summary. By default, this is set to "AI", but you can set this to be anything you want. Note that if you change this, you should also change the prompt used in the chain to reflect this naming change. Let's walk through an example of that in the example below.# Here it is by default set to "AI"conversation = ConversationChain(    llm=llm, verbose=True, memory=ConversationBufferMemory())conversation.predict(input="Hi there!")            > Entering new ConversationChain chain...    Prompt after formatting:    The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the


is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.        Current conversation:        Human: Hi there!    AI:        > Finished ConversationChain chain.    " Hi there! It's nice to meet you. How can I help you today?"conversation.predict(input="What's the weather?")            > Entering new ConversationChain chain...    Prompt after formatting:    The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.        Current conversation:        Human: Hi there!    AI:  Hi there! It's nice to meet you. How can I help you today?    Human: What's the weather?    AI:        > Finished ConversationChain chain.    ' The current weather is sunny and warm with a temperature of 75 degrees Fahrenheit. The forecast for the next few days is sunny with temperatures in the mid-70s.'# Now we can override it and set it to "AI Assistant"from langchain.prompts.prompt import PromptTemplatetemplate = """The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.Current conversation:{history}Human: {input}AI Assistant:"""PROMPT = PromptTemplate(input_variables=["history", "input"],


Assistant:"""PROMPT = PromptTemplate(input_variables=["history", "input"], template=template)conversation = ConversationChain(    prompt=PROMPT,    llm=llm,    verbose=True,    memory=ConversationBufferMemory(ai_prefix="AI Assistant"),)conversation.predict(input="Hi there!")            > Entering new ConversationChain chain...    Prompt after formatting:    The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.        Current conversation:        Human: Hi there!    AI Assistant:        > Finished ConversationChain chain.    " Hi there! It's nice to meet you. How can I help you today?"conversation.predict(input="What's the weather?")            > Entering new ConversationChain chain...    Prompt after formatting:    The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.        Current conversation:        Human: Hi there!    AI Assistant:  Hi there! It's nice to meet you. How can I help you today?    Human: What's the weather?    AI Assistant:        > Finished ConversationChain chain.    ' The current weather is sunny and warm with a temperature of 75 degrees Fahrenheit. The forecast for the rest of


weather is sunny and warm with a temperature of 75 degrees Fahrenheit. The forecast for the rest of the day is sunny with a high of 78 degrees and a low of 65 degrees.'Human Prefixâ€‹The next way to do so is by changing the Human prefix in the conversation summary. By default, this is set to "Human", but you can set this to be anything you want. Note that if you change this, you should also change the prompt used in the chain to reflect this naming change. Let's walk through an example of that in the example below.# Now we can override it and set it to "Friend"from langchain.prompts.prompt import PromptTemplatetemplate = """The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.Current conversation:{history}Friend: {input}AI:"""PROMPT = PromptTemplate(input_variables=["history", "input"], template=template)conversation = ConversationChain(    prompt=PROMPT,    llm=llm,    verbose=True,    memory=ConversationBufferMemory(human_prefix="Friend"),)conversation.predict(input="Hi there!")            > Entering new ConversationChain chain...    Prompt after formatting:    The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.        Current conversation:        Friend: Hi there!    AI:        > Finished ConversationChain chain.    " Hi there!


> Finished ConversationChain chain.    " Hi there! It's nice to meet you. How can I help you today?"conversation.predict(input="What's the weather?")            > Entering new ConversationChain chain...    Prompt after formatting:    The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.        Current conversation:        Friend: Hi there!    AI:  Hi there! It's nice to meet you. How can I help you today?    Friend: What's the weather?    AI:        > Finished ConversationChain chain.    ' The weather right now is sunny and warm with a temperature of 75 degrees Fahrenheit. The forecast for the rest of the day is mostly sunny with a high of 82 degrees.'PreviousConversation buffer window memoryNextHow to create a custom Memory classAI PrefixHuman PrefixCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Entity memory | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryHow to add Memory to an LLMChainHow to add memory to a Multi-Input ChainHow to add Memory to an AgentAdding Message Memory backed by a database to an AgentConversation buffer memoryConversation buffer window memoryHow to customize conversational memoryHow to create a custom Memory classEntity memoryConversation Knowledge Graph MemoryHow to use multiple memory classes in the same chainConversation summary memoryConversationSummaryBufferMemoryConversationTokenBufferMemoryVector store-backed memoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesMemoryEntity memoryEntity memoryEntity Memory remembers given facts about specific entities in a conversation. It extracts information on entities (using an LLM) and builds up its knowledge about that entity over time (also using an LLM).Let's first walk through using this functionality.from langchain.llms import OpenAIfrom langchain.memory import ConversationEntityMemoryllm = OpenAI(temperature=0)memory = ConversationEntityMemory(llm=llm)_input = {"input": "Deven & Sam are working on a hackathon project"}memory.load_memory_variables(_input)memory.save_context(    _input,    {"output": " That sounds like a great project! What kind of project are they working on?"})memory.load_memory_variables({"input": 'who is Sam'})    {'history': 'Human: Deven & Sam are working on a hackathon project\nAI:  That sounds like a great project! What kind of project are they working on?',     'entities': {'Sam': 'Sam is working on a hackathon project with Deven.'}}memory = ConversationEntityMemory(llm=llm, return_messages=True)_input = {"input":


= ConversationEntityMemory(llm=llm, return_messages=True)_input = {"input": "Deven & Sam are working on a hackathon project"}memory.load_memory_variables(_input)memory.save_context(    _input,    {"output": " That sounds like a great project! What kind of project are they working on?"})memory.load_memory_variables({"input": 'who is Sam'})    {'history': [HumanMessage(content='Deven & Sam are working on a hackathon project', additional_kwargs={}),      AIMessage(content=' That sounds like a great project! What kind of project are they working on?', additional_kwargs={})],     'entities': {'Sam': 'Sam is working on a hackathon project with Deven.'}}Using in a chainâ€‹Let's now use it in a chain!from langchain.chains import ConversationChainfrom langchain.memory import ConversationEntityMemoryfrom langchain.memory.prompt import ENTITY_MEMORY_CONVERSATION_TEMPLATEfrom pydantic import BaseModelfrom typing import List, Dict, Anyconversation = ConversationChain(    llm=llm,     verbose=True,    prompt=ENTITY_MEMORY_CONVERSATION_TEMPLATE,    memory=ConversationEntityMemory(llm=llm))conversation.predict(input="Deven & Sam are working on a hackathon project")            > Entering new ConversationChain chain...    Prompt after formatting:    You are an assistant to a human, powered by a large language model trained by OpenAI.        You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you


language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.        You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics.        Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.        Context:    {'Deven': 'Deven is working on a hackathon project with Sam.', 'Sam': 'Sam is working on a hackathon project with Deven.'}        Current conversation:        Last line:    Human: Deven & Sam are working on a hackathon project    You:        > Finished chain.    ' That sounds like a great project! What kind of project are they working on?'conversation.memory.entity_store.store    {'Deven': 'Deven is working on a hackathon project with Sam, which they are entering into a hackathon.',     'Sam': 'Sam is working on a hackathon project with Deven.'}conversation.predict(input="They are trying to add more complex memory structures to Langchain")


are trying to add more complex memory structures to Langchain")            > Entering new ConversationChain chain...    Prompt after formatting:    You are an assistant to a human, powered by a large language model trained by OpenAI.        You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.        You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics.        Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.        Context:    {'Deven': 'Deven is working on a hackathon project with Sam, which they are entering into a hackathon.', 'Sam': 'Sam is working on a hackathon project with Deven.', 'Langchain': ''}        Current conversation:    Human: Deven & Sam are working on a hackathon project    AI:


Human: Deven & Sam are working on a hackathon project    AI:  That sounds like a great project! What kind of project are they working on?    Last line:    Human: They are trying to add more complex memory structures to Langchain    You:        > Finished chain.    ' That sounds like an interesting project! What kind of memory structures are they trying to add?'conversation.predict(input="They are adding in a key-value store for entities mentioned so far in the conversation.")            > Entering new ConversationChain chain...    Prompt after formatting:    You are an assistant to a human, powered by a large language model trained by OpenAI.        You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.        You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics.        Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just


and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.        Context:    {'Deven': 'Deven is working on a hackathon project with Sam, which they are entering into a hackathon. They are trying to add more complex memory structures to Langchain.', 'Sam': 'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain.', 'Langchain': 'Langchain is a project that is trying to add more complex memory structures.', 'Key-Value Store': ''}        Current conversation:    Human: Deven & Sam are working on a hackathon project    AI:  That sounds like a great project! What kind of project are they working on?    Human: They are trying to add more complex memory structures to Langchain    AI:  That sounds like an interesting project! What kind of memory structures are they trying to add?    Last line:    Human: They are adding in a key-value store for entities mentioned so far in the conversation.    You:        > Finished chain.    ' That sounds like a great idea! How will the key-value store help with the project?'conversation.predict(input="What do you know about Deven & Sam?")            > Entering new ConversationChain chain...    Prompt after formatting:    You are an assistant to a human, powered by a large language model trained by OpenAI.        You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics.


tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.        You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics.        Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.        Context:    {'Deven': 'Deven is working on a hackathon project with Sam, which they are entering into a hackathon. They are trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.', 'Sam': 'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.'}        Current conversation:    Human: Deven & Sam are working on a hackathon project    AI:  That sounds like a great project! What kind of project are they working on?    Human: They are trying to add more complex memory structures to


are they working on?    Human: They are trying to add more complex memory structures to Langchain    AI:  That sounds like an interesting project! What kind of memory structures are they trying to add?    Human: They are adding in a key-value store for entities mentioned so far in the conversation.    AI:  That sounds like a great idea! How will the key-value store help with the project?    Last line:    Human: What do you know about Deven & Sam?    You:        > Finished chain.    ' Deven and Sam are working on a hackathon project together, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be working hard on this project and have a great idea for how the key-value store can help.'Inspecting the memory storeâ€‹We can also inspect the memory store directly. In the following examples, we look at it directly, and then go through some examples of adding information and watch how it changes.from pprint import pprintpprint(conversation.memory.entity_store.store)    {'Daimon': 'Daimon is a company founded by Sam, a successful entrepreneur.',     'Deven': 'Deven is working on a hackathon project with Sam, which they are '              'entering into a hackathon. They are trying to add more complex '              'memory structures to Langchain, including a key-value store for '              'entities mentioned so far in the conversation, and seem to be '              'working hard on this project


'              'working hard on this project with a great idea for how the '              'key-value store can help.',     'Key-Value Store': 'A key-value store is being added to the project to store '                        'entities mentioned in the conversation.',     'Langchain': 'Langchain is a project that is trying to add more complex '                  'memory structures, including a key-value store for entities '                  'mentioned so far in the conversation.',     'Sam': 'Sam is working on a hackathon project with Deven, trying to add more '            'complex memory structures to Langchain, including a key-value store '            'for entities mentioned so far in the conversation. They seem to have '            'a great idea for how the key-value store can help, and Sam is also '            'the founder of a company called Daimon.'}conversation.predict(input="Sam is the founder of a company called Daimon.")            > Entering new ConversationChain chain...    Prompt after formatting:    You are an assistant to a human, powered by a large language model trained by OpenAI.        You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth


to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.        You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics.        Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.        Context:    {'Daimon': 'Daimon is a company founded by Sam, a successful entrepreneur.', 'Sam': 'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to have a great idea for how the key-value store can help, and Sam is also the founder of a company called Daimon.'}        Current conversation:    Human: They are adding in a key-value store for entities mentioned so far in the conversation.    AI:  That sounds like a great idea! How will the key-value store help with the project?


That sounds like a great idea! How will the key-value store help with the project?    Human: What do you know about Deven & Sam?    AI:  Deven and Sam are working on a hackathon project together, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be working hard on this project and have a great idea for how the key-value store can help.    Human: Sam is the founder of a company called Daimon.    AI:     That's impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon?    Last line:    Human: Sam is the founder of a company called Daimon.    You:        > Finished chain.    " That's impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon?"from pprint import pprintpprint(conversation.memory.entity_store.store)    {'Daimon': 'Daimon is a company founded by Sam, a successful entrepreneur, who '               'is working on a hackathon project with Deven to add more complex '               'memory structures to Langchain.',     'Deven': 'Deven is working on a hackathon project with Sam, which they are '              'entering into a hackathon. They are trying to add more complex '              'memory structures to Langchain, including a key-value store for '


including a key-value store for '              'entities mentioned so far in the conversation, and seem to be '              'working hard on this project with a great idea for how the '              'key-value store can help.',     'Key-Value Store': 'A key-value store is being added to the project to store '                        'entities mentioned in the conversation.',     'Langchain': 'Langchain is a project that is trying to add more complex '                  'memory structures, including a key-value store for entities '                  'mentioned so far in the conversation.',     'Sam': 'Sam is working on a hackathon project with Deven, trying to add more '            'complex memory structures to Langchain, including a key-value store '            'for entities mentioned so far in the conversation. They seem to have '            'a great idea for how the key-value store can help, and Sam is also '            'the founder of a successful company called Daimon.'}conversation.predict(input="What do you know about Sam?")            > Entering new ConversationChain chain...    Prompt after formatting:    You are an assistant to a human, powered by a large language model trained by OpenAI.


You are an assistant to a human, powered by a large language model trained by OpenAI.        You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.        You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics.        Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.        Context:    {'Deven': 'Deven is working on a hackathon project with Sam, which they are entering into a hackathon. They are trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation, and seem to be working hard on this project with a great idea for how the key-value store can help.', 'Sam': 'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to


Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to have a great idea for how the key-value store can help, and Sam is also the founder of a successful company called Daimon.', 'Langchain': 'Langchain is a project that is trying to add more complex memory structures, including a key-value store for entities mentioned so far in the conversation.', 'Daimon': 'Daimon is a company founded by Sam, a successful entrepreneur, who is working on a hackathon project with Deven to add more complex memory structures to Langchain.'}        Current conversation:    Human: What do you know about Deven & Sam?    AI:  Deven and Sam are working on a hackathon project together, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be working hard on this project and have a great idea for how the key-value store can help.    Human: Sam is the founder of a company called Daimon.    AI:     That's impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon?    Human: Sam is the founder of a company called Daimon.    AI:  That's impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon?    Last line:    Human: What do you know about Sam?    You:        > Finished chain.    ' Sam is the founder of a successful company called Daimon. He is also working on a hackathon project with Deven to add more complex memory structures to Langchain. They seem to have a great idea for how the key-value store can


memory structures to Langchain. They seem to have a great idea for how the key-value store can help.'PreviousHow to create a custom Memory classNextConversation Knowledge Graph MemoryCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


How to add Memory to an Agent | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryHow to add Memory to an LLMChainHow to add memory to a Multi-Input ChainHow to add Memory to an AgentAdding Message Memory backed by a database to an AgentConversation buffer memoryConversation buffer window memoryHow to customize conversational memoryHow to create a custom Memory classEntity memoryConversation Knowledge Graph MemoryHow to use multiple memory classes in the same chainConversation summary memoryConversationSummaryBufferMemoryConversationTokenBufferMemoryVector store-backed memoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesMemoryHow to add Memory to an AgentHow to add Memory to an AgentThis notebook goes over adding memory to an Agent. Before going through this notebook, please walkthrough the following notebooks, as this will build on top of both of them:Adding memory to an LLM ChainCustom AgentsIn order to add a memory to an agent we are going to the the following steps:We are going to create an LLMChain with memory.We are going to use that LLMChain to create a custom Agent.For the purposes of this exercise, we are going to create a simple custom Agent that has access to a search tool and utilizes the ConversationBufferMemory class.from langchain.agents import ZeroShotAgent, Tool, AgentExecutorfrom langchain.memory import ConversationBufferMemoryfrom langchain import OpenAI, LLMChainfrom langchain.utilities import GoogleSearchAPIWrappersearch = GoogleSearchAPIWrapper()tools = [    Tool(        name="Search",        func=search.run,        description="useful for when you need to answer questions about current events",    )]Notice the usage of the chat_history variable in the


questions about current events",    )]Notice the usage of the chat_history variable in the PromptTemplate, which matches up with the dynamic key name in the ConversationBufferMemory.prefix = """Have a conversation with a human, answering the following questions as best you can. You have access to the following tools:"""suffix = """Begin!"{chat_history}Question: {input}{agent_scratchpad}"""prompt = ZeroShotAgent.create_prompt(    tools,    prefix=prefix,    suffix=suffix,    input_variables=["input", "chat_history", "agent_scratchpad"],)memory = ConversationBufferMemory(memory_key="chat_history")We can now construct the LLMChain, with the Memory object, and then create the agent.llm_chain = LLMChain(llm=OpenAI(temperature=0), prompt=prompt)agent = ZeroShotAgent(llm_chain=llm_chain, tools=tools, verbose=True)agent_chain = AgentExecutor.from_agent_and_tools(    agent=agent, tools=tools, verbose=True, memory=memory)agent_chain.run(input="How many people live in canada?")            > Entering new AgentExecutor chain...    Thought: I need to find out the population of Canada    Action: Search    Action Input: Population of Canada    Observation: The current population of Canada is 38,566,192 as of Saturday, December 31, 2022, based on Worldometer elaboration of the latest United Nations data. Â· CanadaÂ ... Additional information related to Canadian population trends can be found on Statistics Canada's Population and Demography Portal. Population of Canada (real-Â ... Index to the latest information from the Census of Population. This survey conducted by Statistics Canada provides a statistical portrait of Canada and itsÂ ... 14 records ... Estimated


conducted by Statistics Canada provides a statistical portrait of Canada and itsÂ ... 14 records ... Estimated number of persons by quarter of a year and by year, Canada, provinces and territories. The 2021 Canadian census counted a total population of 36,991,981, an increase of around 5.2 percent over the 2016 figure. ... Between 1990 and 2008, theÂ ... ( 2 ) Census reports and other statistical publications from national statistical offices, ( 3 ) Eurostat: Demographic Statistics, ( 4 ) United NationsÂ ... Canada is a country in North America. Its ten provinces and three territories extend from ... Population. â€¢ Q4 2022 estimate. 39,292,355 (37th). Information is available for the total Indigenous population and each of the three ... The term 'Aboriginal' or 'Indigenous' used on the Statistics CanadaÂ ... Jun 14, 2022 ... Determinants of health are the broad range of personal, social, economic and environmental factors that determine individual and populationÂ ... COVID-19 vaccination coverage across Canada by demographics and key populations. Updated every Friday at 12:00 PM Eastern Time.    Thought: I now know the final answer    Final Answer: The current population of Canada is 38,566,192 as of Saturday, December 31, 2022, based on Worldometer elaboration of the latest United Nations data.    > Finished AgentExecutor chain.    'The current population of Canada is 38,566,192 as of Saturday, December 31, 2022, based on Worldometer elaboration of the latest United Nations data.'To test the memory of this agent, we can ask a followup question that relies on information in the previous exchange to be answered correctly.agent_chain.run(input="what is their national anthem called?")


answered correctly.agent_chain.run(input="what is their national anthem called?")            > Entering new AgentExecutor chain...    Thought: I need to find out what the national anthem of Canada is called.    Action: Search    Action Input: National Anthem of Canada    Observation: Jun 7, 2010 ... https://twitter.com/CanadaImmigrantCanadian National Anthem O Canada in HQ - complete with lyrics, captions, vocals & music.LYRICS:O Canada! Nov 23, 2022 ... After 100 years of tradition, O Canada was proclaimed Canada's national anthem in 1980. The music for O Canada was composed in 1880 by CalixaÂ ... O Canada, national anthem of Canada. It was proclaimed the official national anthem on July 1, 1980. â€œGod Save the Queenâ€� remains the royal anthem of CanadaÂ ... O Canada! Our home and native land! True patriot love in all of us command. Car ton bras sait porter l'Ã©pÃ©e,. Il sait porter la croix! "O Canada" (French: Ã” Canada) is the national anthem of Canada. The song was originally commissioned by Lieutenant Governor of Quebec ThÃ©odore RobitailleÂ ... Feb 1, 2018 ... It was a simple tweak â€” just two words. But with that, Canada just voted to make its national anthem, â€œO Canada,â€� gender neutral,Â ... "O Canada" was proclaimed Canada's national anthem on July 1,. 1980, 100 years after it was first sung on June 24, 1880. The music. Patriotic music in Canada dates back over 200 years as a distinct category from British or French patriotism, preceding the first legal steps toÂ ... Feb


as a distinct category from British or French patriotism, preceding the first legal steps toÂ ... Feb 4, 2022 ... English version: O Canada! Our home and native land! True patriot love in all of us command. With glowing hearts weÂ ... Feb 1, 2018 ... Canada's Senate has passed a bill making the country's national anthem gender-neutral. If you're not familiar with the words to â€œO Canada,â€�Â ...    Thought: I now know the final answer.    Final Answer: The national anthem of Canada is called "O Canada".    > Finished AgentExecutor chain.    'The national anthem of Canada is called "O Canada".'We can see that the agent remembered that the previous question was about Canada, and properly asked Google Search what the name of Canada's national anthem was.For fun, let's compare this to an agent that does NOT have memory.prefix = """Have a conversation with a human, answering the following questions as best you can. You have access to the following tools:"""suffix = """Begin!"Question: {input}{agent_scratchpad}"""prompt = ZeroShotAgent.create_prompt(    tools, prefix=prefix, suffix=suffix, input_variables=["input", "agent_scratchpad"])llm_chain = LLMChain(llm=OpenAI(temperature=0), prompt=prompt)agent = ZeroShotAgent(llm_chain=llm_chain, tools=tools, verbose=True)agent_without_memory = AgentExecutor.from_agent_and_tools(    agent=agent, tools=tools, verbose=True)agent_without_memory.run("How many people live in canada?")            > Entering new AgentExecutor chain...    Thought: I need to find out the population of Canada    Action: Search    Action Input: Population of


find out the population of Canada    Action: Search    Action Input: Population of Canada    Observation: The current population of Canada is 38,566,192 as of Saturday, December 31, 2022, based on Worldometer elaboration of the latest United Nations data. Â· CanadaÂ ... Additional information related to Canadian population trends can be found on Statistics Canada's Population and Demography Portal. Population of Canada (real-Â ... Index to the latest information from the Census of Population. This survey conducted by Statistics Canada provides a statistical portrait of Canada and itsÂ ... 14 records ... Estimated number of persons by quarter of a year and by year, Canada, provinces and territories. The 2021 Canadian census counted a total population of 36,991,981, an increase of around 5.2 percent over the 2016 figure. ... Between 1990 and 2008, theÂ ... ( 2 ) Census reports and other statistical publications from national statistical offices, ( 3 ) Eurostat: Demographic Statistics, ( 4 ) United NationsÂ ... Canada is a country in North America. Its ten provinces and three territories extend from ... Population. â€¢ Q4 2022 estimate. 39,292,355 (37th). Information is available for the total Indigenous population and each of the three ... The term 'Aboriginal' or 'Indigenous' used on the Statistics CanadaÂ ... Jun 14, 2022 ... Determinants of health are the broad range of personal, social, economic and environmental factors that determine individual and populationÂ ... COVID-19 vaccination coverage across Canada by demographics and key populations. Updated every Friday at 12:00 PM Eastern Time.    Thought: I now know the final answer    Final Answer: The current population of Canada is 38,566,192 as of Saturday, December 31,


The current population of Canada is 38,566,192 as of Saturday, December 31, 2022, based on Worldometer elaboration of the latest United Nations data.    > Finished AgentExecutor chain.    'The current population of Canada is 38,566,192 as of Saturday, December 31, 2022, based on Worldometer elaboration of the latest United Nations data.'agent_without_memory.run("what is their national anthem called?")            > Entering new AgentExecutor chain...    Thought: I should look up the answer    Action: Search    Action Input: national anthem of [country]    Observation: Most nation states have an anthem, defined as "a song, as of praise, devotion, or patriotism"; most anthems are either marches or hymns in style. List of all countries around the world with its national anthem. ... Title and lyrics in the language of the country and translated into English, Aug 1, 2021 ... 1. Afghanistan, "Milli Surood" (National Anthem) Â· 2. Armenia, "Mer Hayrenik" (Our Fatherland) Â· 3. Azerbaijan (a transcontinental country withÂ ... A national anthem is a patriotic musical composition symbolizing and evoking eulogies of the history and traditions of a country or nation. National Anthem of Every Country ; Fiji, â€œMeda Dau Dokaâ€� (â€œGod Bless Fijiâ€�) ; Finland, â€œMaammeâ€�. (â€œOur Landâ€�) ; France, â€œLa Marseillaiseâ€� (â€œThe Marseillaiseâ€�). You can find an anthem in the menu at the top alphabetically or you can use the search


You can find an anthem in the menu at the top alphabetically or you can use the search feature. This site is focussed on the scholarly study of national anthemsÂ ... Feb 13, 2022 ... The 38-year-old country music artist had the honor of singing the National Anthem during this year's big game, and she did not disappoint. Oldest of the World's National Anthems ; France, La Marseillaise (â€œThe Marseillaiseâ€�), 1795 ; Argentina, Himno Nacional Argentino (â€œArgentine National Anthemâ€�)Â ... Mar 3, 2022 ... Country music star Jessie James Decker gained the respect of music and hockey fans alike after a jaw-dropping rendition of "The Star-SpangledÂ ... This list shows the country on the left, the national anthem in the ... There are many countries over the world who have a national anthem of their own.    Thought: I now know the final answer    Final Answer: The national anthem of [country] is [name of anthem].    > Finished AgentExecutor chain.    'The national anthem of [country] is [name of anthem].'PreviousHow to add memory to a Multi-Input ChainNextAdding Message Memory backed by a database to an AgentCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Data connection | ğŸ¦œï¸�ğŸ”— Langchain

Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionDocument loadersDocument transformersText embedding modelsVector storesRetrieversChainsMemoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesData connectionData connectionMany LLM applications require user-specific data that is not part of the model's training set. LangChain gives you the
building blocks to load, transform, store and query your data via:Document loaders: Load documents from many different sourcesDocument transformers: Split documents, convert documents into Q&A format, drop redundant documents, and moreText embedding models: Take unstructured text and turn it into a list of floating point numbersVector stores: Store and search over embedded dataRetrievers: Query your dataPreviousStructured output parserNextDocument loadersCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Document transformers | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionDocument loadersDocument transformersText splittersPost retrievalText embedding modelsVector storesRetrieversChainsMemoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesData connectionDocument transformersOn this pageDocument transformersinfoHead to Integrations for documentation on built-in document transformer integrations with 3rd-party tools.Once you've loaded documents, you'll often want to transform them to better suit your application. The simplest example
is you may want to split a long document into smaller chunks that can fit into your model's context window. LangChain
has a number of built-in document transformers that make it easy to split, combine, filter, and otherwise manipulate documents.Text splittersâ€‹When you want to deal with long pieces of text, it is necessary to split up that text into chunks.
As simple as this sounds, there is a lot of potential complexity here. Ideally, you want to keep the semantically related pieces of text together. What "semantically related" means could depend on the type of text.


This notebook showcases several ways to do that.At a high level, text splitters work as following:Split the text up into small, semantically meaningful chunks (often sentences).Start combining these small chunks into a larger chunk until you reach a certain size (as measured by some function).Once you reach that size, make that chunk its own piece of text and then start creating a new chunk of text with some overlap (to keep context between chunks).That means there are two different axes along which you can customize your text splitter:How the text is splitHow the chunk size is measuredGet started with text splittersâ€‹The default recommended text splitter is the RecursiveCharacterTextSplitter. This text splitter takes a list of characters. It tries to create chunks based on splitting on the first character, but if any chunks are too large it then moves onto the next character, and so forth. By default the characters it tries to split on are ["\n\n", "\n", " ", ""]In addition to controlling which characters you can split on, you can also control a few other things:length_function: how the length of chunks is calculated. Defaults to just counting number of characters, but it's pretty common to pass a token counter here.chunk_size: the maximum size of your chunks (as measured by the length function).chunk_overlap: the maximum overlap between chunks. It can be nice to have some overlap to maintain some continuity between chunks (eg do a sliding window).add_start_index: whether to include the starting position of each chunk within the original document in the metadata.# This is a long document we can split up.with open('../../state_of_the_union.txt') as f:    state_of_the_union = f.read()from langchain.text_splitter import RecursiveCharacterTextSplittertext_splitter = RecursiveCharacterTextSplitter(    # Set a really small chunk size, just to show.    chunk_size = 100,


a really small chunk size, just to show.    chunk_size = 100,    chunk_overlap  = 20,    length_function = len,    add_start_index = True,)texts = text_splitter.create_documents([state_of_the_union])print(texts[0])print(texts[1])    page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and' metadata={'start_index': 0}    page_content='of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.' metadata={'start_index': 82}Other transformations:â€‹Filter redundant docs, translate docs, extract metadata, and moreâ€‹We can do perform a number of transformations on docs which are not simply splitting the text. With the


EmbeddingsRedundantFilter we can identify similar documents and filter out redundancies. With integrations like
doctran we can do things like translate documents from one language
to another, extract desired properties and add them to metadata, and convert conversational dialogue into a Q/A format
set of documents.PreviousPDFNextSplit by characterText splittersGet started with text splittersCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Lost in the middle: The problem with long contexts | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionDocument loadersDocument transformersText splittersPost retrievalLost in the middle: The problem with long contextsText embedding modelsVector storesRetrieversChainsMemoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesData connectionDocument transformersPost retrievalLost in the middle: The problem with long contextsLost in the middle: The problem with long contextsNo matter the architecture of your model, there is a substantial performance degradation when you include 10+ retrieved documents.
In brief: When models must access relevant information in the middle of long contexts, then tend to ignore the provided documents.


See: https://arxiv.org/abs/2307.03172To avoid this issue you can re-order documents after retrieval to avoid performance degradation.import osimport chromadbfrom langchain.vectorstores import Chromafrom langchain.embeddings import HuggingFaceEmbeddingsfrom langchain.document_transformers import (    LongContextReorder,)from langchain.chains import StuffDocumentsChain, LLMChainfrom langchain.prompts import PromptTemplatefrom langchain.llms import OpenAI# Get embeddings.embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")texts = [    "Basquetball is a great sport.",    "Fly me to the moon is one of my favourite songs.",    "The Celtics are my favourite team.",    "This is a document about the Boston Celtics",    "I simply love going to the movies",    "The Boston Celtics won the game by 20 points",    "This is just a random text.",    "Elden Ring is one of the best games in the last 15 years.",    "L. Kornet is one of the best Celtics players.",    "Larry Bird was an iconic NBA player.",]# Create a retrieverretriever = Chroma.from_texts(texts, embedding=embeddings).as_retriever(    search_kwargs={"k": 10})query = "What can you tell me about the Celtics?"# Get relevant documents ordered by relevance scoredocs = retriever.get_relevant_documents(query)docs    [Document(page_content='This is a document about the Boston Celtics', metadata={}),     Document(page_content='The Celtics are my favourite team.', metadata={}),     Document(page_content='L. Kornet is one of the best Celtics players.', metadata={}),


Kornet is one of the best Celtics players.', metadata={}),     Document(page_content='The Boston Celtics won the game by 20 points', metadata={}),     Document(page_content='Larry Bird was an iconic NBA player.', metadata={}),     Document(page_content='Elden Ring is one of the best games in the last 15 years.', metadata={}),     Document(page_content='Basquetball is a great sport.', metadata={}),     Document(page_content='I simply love going to the movies', metadata={}),     Document(page_content='Fly me to the moon is one of my favourite songs.', metadata={}),     Document(page_content='This is just a random text.', metadata={})]# Reorder the documents:# Less relevant document will be at the middle of the list and more# relevant elements at begining / end.reordering = LongContextReorder()reordered_docs = reordering.transform_documents(docs)# Confirm that the 4 relevant documents are at begining and end.reordered_docs    [Document(page_content='The Celtics are my favourite team.', metadata={}),     Document(page_content='The Boston Celtics won the game by 20 points', metadata={}),     Document(page_content='Elden Ring is one of the best games in the last 15 years.', metadata={}),     Document(page_content='I simply love going to the movies', metadata={}),     Document(page_content='This is just a random text.', metadata={}),     Document(page_content='Fly me to the moon is one of my favourite songs.', metadata={}),     Document(page_content='Basquetball is a great sport.', metadata={}),     Document(page_content='Larry Bird was an iconic NBA player.', metadata={}),     Document(page_content='L.


Bird was an iconic NBA player.', metadata={}),     Document(page_content='L. Kornet is one of the best Celtics players.', metadata={}),     Document(page_content='This is a document about the Boston Celtics', metadata={})]# We prepare and run a custom Stuff chain with reordered docs as context.# Override promptsdocument_prompt = PromptTemplate(    input_variables=["page_content"], template="{page_content}")document_variable_name = "context"llm = OpenAI()stuff_prompt_override = """Given this text extracts:-----{context}-----Please answer the following question:{query}"""prompt = PromptTemplate(    template=stuff_prompt_override, input_variables=["context", "query"])# Instantiate the chainllm_chain = LLMChain(llm=llm, prompt=prompt)chain = StuffDocumentsChain(    llm_chain=llm_chain,    document_prompt=document_prompt,    document_variable_name=document_variable_name,)chain.run(input_documents=reordered_docs, query=query)PreviousSplit by tokensNextText embedding modelsCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Split by character | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionDocument loadersDocument transformersText splittersSplit by characterSplit codeMarkdownHeaderTextSplitterRecursively split by characterSplit by tokensPost retrievalText embedding modelsVector storesRetrieversChainsMemoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesData connectionDocument transformersText splittersSplit by characterSplit by characterThis is the simplest method. This splits based on characters (by default "\n\n") and measure chunk length by number of characters.How the text is split: by single characterHow the chunk size is measured: by number of characters# This is a long document we can split up.with open('../../../state_of_the_union.txt') as f:    state_of_the_union = f.read()from langchain.text_splitter import CharacterTextSplittertext_splitter = CharacterTextSplitter(            separator = "\n\n",    chunk_size = 1000,    chunk_overlap  = 200,    length_function = len,)texts = text_splitter.create_documents([state_of_the_union])print(texts[0])    page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \n\nLast year COVID-19 kept us apart. This year we are finally together again. \n\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n\nWith a duty to one another to the American people to the Constitution. \n\nAnd with an unwavering resolve that freedom will always triumph over tyranny. \n\nSix days ago,


an unwavering resolve that freedom will always triumph over tyranny. \n\nSix days ago, Russiaâ€™s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \n\nHe thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \n\nHe met the Ukrainian people. \n\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.' lookup_str='' metadata={} lookup_index=0Here's an example of passing metadata along with the documents, notice that it is split along with the documents.metadatas = [{"document": 1}, {"document": 2}]documents = text_splitter.create_documents([state_of_the_union, state_of_the_union], metadatas=metadatas)print(documents[0])    page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \n\nLast year COVID-19 kept us apart. This year we are finally together again. \n\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n\nWith a duty to one another to the American people to the Constitution. \n\nAnd with an unwavering resolve that freedom will always triumph over tyranny. \n\nSix days ago, Russiaâ€™s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \n\nHe thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \n\nHe met the Ukrainian people. \n\nFrom President Zelenskyy


\n\nHe met the Ukrainian people. \n\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.' lookup_str='' metadata={'document': 1} lookup_index=0text_splitter.split_text(state_of_the_union)[0]    'Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \n\nLast year COVID-19 kept us apart. This year we are finally together again. \n\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n\nWith a duty to one another to the American people to the Constitution. \n\nAnd with an unwavering resolve that freedom will always triumph over tyranny. \n\nSix days ago, Russiaâ€™s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \n\nHe thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \n\nHe met the Ukrainian people. \n\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.'PreviousDocument transformersNextSplit codeCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Document loaders | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionDocument loadersCSVFile DirectoryHTMLJSONMarkdownPDFDocument transformersText embedding modelsVector storesRetrieversChainsMemoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesData connectionDocument loadersOn this pageDocument loadersinfoHead to Integrations for documentation on built-in document loader integrations with 3rd-party tools.Use document loaders to load data from a source as Document's. A Document is a piece of text
and associated metadata. For example, there are document loaders for loading a simple .txt file, for loading the text
contents of any web page, or even for loading a transcript of a YouTube video.Document loaders expose a "load" method for loading data as documents from a configured source. They optionally


implement a "lazy load" as well for lazily loading data into memory.Get startedâ€‹The simplest loader reads in a file as text and places it all into one Document.from langchain.document_loaders import TextLoaderloader = TextLoader("./index.md")loader.load()[    Document(page_content='---\nsidebar_position: 0\n---\n# Document loaders\n\nUse document loaders to load data from a source as `Document`\'s. A `Document` is a piece of text\nand associated metadata. For example, there are document loaders for loading a simple `.txt` file, for loading the text\ncontents of any web page, or even for loading a transcript of a YouTube video.\n\nEvery document loader exposes two methods:\n1. "Load": load documents from the configured source\n2. "Load and split": load documents from the configured source and split them using the passed in text splitter\n\nThey optionally implement:\n\n3. "Lazy load": load documents into memory lazily\n', metadata={'source': '../docs/docs_skeleton/docs/modules/data_connection/document_loaders/index.md'})]PreviousData connectionNextCSVGet startedCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


JSON | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionDocument loadersCSVFile DirectoryHTMLJSONMarkdownPDFDocument transformersText embedding modelsVector storesRetrieversChainsMemoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesData connectionDocument loadersJSONJSONJSON (JavaScript Object Notation) is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attributeâ€“value pairs and arrays (or other serializable values).JSON Lines is a file format where each line is a valid JSON value.The JSONLoader uses a specified jq schema to parse the JSON files. It uses the jq python package.


Check this manual for a detailed documentation of the jq syntax.#!pip install jqfrom langchain.document_loaders import JSONLoaderimport jsonfrom pathlib import Pathfrom pprint import pprintfile_path='./example_data/facebook_chat.json'data = json.loads(Path(file_path).read_text())pprint(data)    {'image': {'creation_timestamp': 1675549016, 'uri': 'image_of_the_chat.jpg'},     'is_still_participant': True,     'joinable_mode': {'link': '', 'mode': 1},     'magic_words': [],     'messages': [{'content': 'Bye!',                   'sender_name': 'User 2',                   'timestamp_ms': 1675597571851},                  {'content': 'Oh no worries! Bye',                   'sender_name': 'User 1',                   'timestamp_ms': 1675597435669},                  {'content': 'No Im sorry it was my mistake, the blue one is not '                              'for sale',                   'sender_name': 'User 2',                   'timestamp_ms':


'timestamp_ms': 1675596277579},                  {'content': 'I thought you were selling the blue one!',                   'sender_name': 'User 1',                   'timestamp_ms': 1675595140251},                  {'content': 'Im not interested in this bag. Im interested in the '                              'blue one!',                   'sender_name': 'User 1',                   'timestamp_ms': 1675595109305},                  {'content': 'Here is $129',                   'sender_name': 'User 2',                   'timestamp_ms': 1675595068468},                  {'photos': [{'creation_timestamp': 1675595059,                               'uri': 'url_of_some_picture.jpg'}],


'url_of_some_picture.jpg'}],                   'sender_name': 'User 2',                   'timestamp_ms': 1675595060730},                  {'content': 'Online is at least $100',                   'sender_name': 'User 2',                   'timestamp_ms': 1675595045152},                  {'content': 'How much do you want?',                   'sender_name': 'User 1',                   'timestamp_ms': 1675594799696},                  {'content': 'Goodmorning! $50 is too low.',                   'sender_name': 'User 2',                   'timestamp_ms': 1675577876645},                  {'content': 'Hi! Im interested in your bag. Im offering $50. Let '                              'me know if you are interested. Thanks!',


'me know if you are interested. Thanks!',                   'sender_name': 'User 1',                   'timestamp_ms': 1675549022673}],     'participants': [{'name': 'User 1'}, {'name': 'User 2'}],     'thread_path': 'inbox/User 1 and User 2 chat',     'title': 'User 1 and User 2 chat'}Using JSONLoaderâ€‹Suppose we are interested in extracting the values under the content field within the messages key of the JSON data. This can easily be done through the JSONLoader as shown below.JSON fileâ€‹loader = JSONLoader(    file_path='./example_data/facebook_chat.json',    jq_schema='.messages[].content')data = loader.load()pprint(data)    [Document(page_content='Bye!', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 1}),     Document(page_content='Oh no worries! Bye', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 2}),     Document(page_content='No Im sorry it was my mistake, the blue one is not for sale', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 3}),     Document(page_content='I thought you were selling the blue one!', metadata={'source':


Document(page_content='I thought you were selling the blue one!', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 4}),     Document(page_content='Im not interested in this bag. Im interested in the blue one!', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 5}),     Document(page_content='Here is $129', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 6}),     Document(page_content='', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 7}),     Document(page_content='Online is at least $100', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 8}),     Document(page_content='How much do you want?', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 9}),     Document(page_content='Goodmorning! $50 is too low.', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 10}),     Document(page_content='Hi! Im interested in your bag. Im offering $50. Let me know if you are interested. Thanks!', metadata={'source':


Im offering $50. Let me know if you are interested. Thanks!', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 11})]JSON Lines fileâ€‹If you want to load documents from a JSON Lines file, you pass json_lines=True


and specify jq_schema to extract page_content from a single JSON object.file_path = './example_data/facebook_chat_messages.jsonl'pprint(Path(file_path).read_text())    ('{"sender_name": "User 2", "timestamp_ms": 1675597571851, "content": "Bye!"}\n'     '{"sender_name": "User 1", "timestamp_ms": 1675597435669, "content": "Oh no '     'worries! Bye"}\n'     '{"sender_name": "User 2", "timestamp_ms": 1675596277579, "content": "No Im '     'sorry it was my mistake, the blue one is not for sale"}\n')loader = JSONLoader(    file_path='./example_data/facebook_chat_messages.jsonl',    jq_schema='.content',    json_lines=True)data = loader.load()pprint(data)    [Document(page_content='Bye!', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat_messages.jsonl', 'seq_num': 1}),     Document(page_content='Oh no worries! Bye', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat_messages.jsonl', 'seq_num': 2}),     Document(page_content='No Im sorry it was my mistake, the blue one is not for sale', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat_messages.jsonl', 'seq_num': 3})]Another option is set jq_schema='.' and provide content_key:loader = JSONLoader(    file_path='./example_data/facebook_chat_messages.jsonl',    jq_schema='.',    content_key='sender_name',


jq_schema='.',    content_key='sender_name',    json_lines=True)data = loader.load()pprint(data)    [Document(page_content='User 2', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat_messages.jsonl', 'seq_num': 1}),     Document(page_content='User 1', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat_messages.jsonl', 'seq_num': 2}),     Document(page_content='User 2', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat_messages.jsonl', 'seq_num': 3})]Extracting metadataâ€‹Generally, we want to include metadata available in the JSON file into the documents that we create from the content.The following demonstrates how metadata can be extracted using the JSONLoader.There are some key changes to be noted. In the previous example where we didn't collect the metadata, we managed to directly specify in the schema where the value for the page_content can be extracted from..messages[].contentIn the current example, we have to tell the loader to iterate over the records in the messages field. The jq_schema then has to be:.messages[]This allows us to pass the records (dict) into the metadata_func that has to be implemented. The metadata_func is responsible for identifying which pieces of information in the record should be included in the metadata stored in the final Document object.Additionally, we now have to explicitly specify in the loader, via the content_key argument, the key from the record where the value for the page_content needs to be extracted from.# Define the metadata extraction function.def metadata_func(record: dict, metadata: dict) -> dict:    metadata["sender_name"] = record.get("sender_name")    metadata["timestamp_ms"] =


= record.get("sender_name")    metadata["timestamp_ms"] = record.get("timestamp_ms")    return metadataloader = JSONLoader(    file_path='./example_data/facebook_chat.json',    jq_schema='.messages[]',    content_key="content",    metadata_func=metadata_func)data = loader.load()pprint(data)    [Document(page_content='Bye!', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 1, 'sender_name': 'User 2', 'timestamp_ms': 1675597571851}),     Document(page_content='Oh no worries! Bye', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 2, 'sender_name': 'User 1', 'timestamp_ms': 1675597435669}),     Document(page_content='No Im sorry it was my mistake, the blue one is not for sale', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 3, 'sender_name': 'User 2', 'timestamp_ms': 1675596277579}),     Document(page_content='I thought you were selling the blue one!', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 4, 'sender_name': 'User 1', 'timestamp_ms': 1675595140251}),     Document(page_content='Im not interested in this bag. Im interested in the blue one!', metadata={'source':


not interested in this bag. Im interested in the blue one!', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 5, 'sender_name': 'User 1', 'timestamp_ms': 1675595109305}),     Document(page_content='Here is $129', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 6, 'sender_name': 'User 2', 'timestamp_ms': 1675595068468}),     Document(page_content='', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 7, 'sender_name': 'User 2', 'timestamp_ms': 1675595060730}),     Document(page_content='Online is at least $100', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 8, 'sender_name': 'User 2', 'timestamp_ms': 1675595045152}),     Document(page_content='How much do you want?', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 9, 'sender_name': 'User 1', 'timestamp_ms': 1675594799696}),     Document(page_content='Goodmorning! $50 is too low.', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 10, 'sender_name': 'User 2',


'seq_num': 10, 'sender_name': 'User 2', 'timestamp_ms': 1675577876645}),     Document(page_content='Hi! Im interested in your bag. Im offering $50. Let me know if you are interested. Thanks!', metadata={'source': '/Users/avsolatorio/WBG/langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 11, 'sender_name': 'User 1', 'timestamp_ms': 1675549022673})]Now, you will see that the documents contain the metadata associated with the content we extracted.The metadata_funcâ€‹As shown above, the metadata_func accepts the default metadata generated by the JSONLoader. This allows full control to the user with respect to how the metadata is formatted.For example, the default metadata contains the source and the seq_num keys. However, it is possible that the JSON data contain these keys as well. The user can then exploit the metadata_func to rename the default keys and use the ones from the JSON data.The example below shows how we can modify the source to only contain information of the file source relative to the langchain directory.# Define the metadata extraction function.def metadata_func(record: dict, metadata: dict) -> dict:    metadata["sender_name"] = record.get("sender_name")    metadata["timestamp_ms"] = record.get("timestamp_ms")    if "source" in metadata:        source = metadata["source"].split("/")        source = source[source.index("langchain"):]        metadata["source"] = "/".join(source)    return metadataloader = JSONLoader(    file_path='./example_data/facebook_chat.json',    jq_schema='.messages[]',    content_key="content",


jq_schema='.messages[]',    content_key="content",    metadata_func=metadata_func)data = loader.load()pprint(data)    [Document(page_content='Bye!', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 1, 'sender_name': 'User 2', 'timestamp_ms': 1675597571851}),     Document(page_content='Oh no worries! Bye', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 2, 'sender_name': 'User 1', 'timestamp_ms': 1675597435669}),     Document(page_content='No Im sorry it was my mistake, the blue one is not for sale', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 3, 'sender_name': 'User 2', 'timestamp_ms': 1675596277579}),     Document(page_content='I thought you were selling the blue one!', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 4, 'sender_name': 'User 1', 'timestamp_ms': 1675595140251}),     Document(page_content='Im not interested in this bag. Im interested in the blue one!', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 5, 'sender_name': 'User 1', 'timestamp_ms': 1675595109305}),     Document(page_content='Here is $129', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num':


'seq_num': 6, 'sender_name': 'User 2', 'timestamp_ms': 1675595068468}),     Document(page_content='', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 7, 'sender_name': 'User 2', 'timestamp_ms': 1675595060730}),     Document(page_content='Online is at least $100', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 8, 'sender_name': 'User 2', 'timestamp_ms': 1675595045152}),     Document(page_content='How much do you want?', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 9, 'sender_name': 'User 1', 'timestamp_ms': 1675594799696}),     Document(page_content='Goodmorning! $50 is too low.', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 10, 'sender_name': 'User 2', 'timestamp_ms': 1675577876645}),     Document(page_content='Hi! Im interested in your bag. Im offering $50. Let me know if you are interested. Thanks!', metadata={'source': 'langchain/docs/modules/indexes/document_loaders/examples/example_data/facebook_chat.json', 'seq_num': 11, 'sender_name': 'User 1', 'timestamp_ms': 1675549022673})]Common JSON structures with jq schemaâ€‹The list below provides a reference to the possible jq_schema the user can use to extract content from the JSON data depending on the structure.JSON        ->


to extract content from the JSON data depending on the structure.JSON        -> [{"text": ...}, {"text": ...}, {"text": ...}]jq_schema   -> ".[].text"JSON        -> {"key": [{"text": ...}, {"text": ...}, {"text": ...}]}jq_schema   -> ".key[].text"JSON        -> ["...", "...", "..."]jq_schema   -> ".[]"PreviousHTMLNextMarkdownCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


HTML | ğŸ¦œï¸�ğŸ”— Langchain

Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionDocument loadersCSVFile DirectoryHTMLJSONMarkdownPDFDocument transformersText embedding modelsVector storesRetrieversChainsMemoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesData connectionDocument loadersHTMLHTMLThe HyperText Markup Language or HTML is the standard markup language for documents designed to be displayed in a web browser.This covers how to load HTML documents into a document format that we can use downstream.from langchain.document_loaders import UnstructuredHTMLLoaderloader = UnstructuredHTMLLoader("example_data/fake-content.html")data = loader.load()data    [Document(page_content='My First Heading\n\nMy first paragraph.', lookup_str='', metadata={'source': 'example_data/fake-content.html'}, lookup_index=0)]Loading HTML with BeautifulSoup4â€‹We can also use BeautifulSoup4 to load HTML documents using the BSHTMLLoader.  This will extract the text from the HTML into page_content, and the page title as title into metadata.from langchain.document_loaders import BSHTMLLoaderloader = BSHTMLLoader("example_data/fake-content.html")data = loader.load()data    [Document(page_content='\n\nTest Title\n\n\nMy First Heading\nMy first paragraph.\n\n\n', metadata={'source': 'example_data/fake-content.html', 'title': 'Test Title'})]PreviousFile DirectoryNextJSONCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Markdown | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionDocument loadersCSVFile DirectoryHTMLJSONMarkdownPDFDocument transformersText embedding modelsVector storesRetrieversChainsMemoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesData connectionDocument loadersMarkdownMarkdownMarkdown is a lightweight markup language for creating formatted text using a plain-text editor.This covers how to load Markdown documents into a document format that we can use downstream.# !pip install unstructured > /dev/nullfrom langchain.document_loaders import UnstructuredMarkdownLoadermarkdown_path = "../../../../../README.md"loader = UnstructuredMarkdownLoader(markdown_path)data = loader.load()data    [Document(page_content="Ã°\x9fÂ¦\x9cÃ¯Â¸\x8fÃ°\x9fâ€�\x97 LangChain\n\nÃ¢\x9aÂ¡ Building applications with LLMs through composability Ã¢\x9aÂ¡\n\nLooking for the JS/TS version? Check out LangChain.js.\n\nProduction Support: As you move your LangChains into production, we'd love to offer more comprehensive support.\nPlease fill out this form and we'll set up a dedicated support Slack channel.\n\nQuick Install\n\npip install langchain\nor\nconda install langchain -c conda-forge\n\nÃ°\x9fÂ¤â€� What is this?\n\nLarge language models (LLMs) are emerging as a transformative technology, enabling developers to build applications that they previously could not. However, using these LLMs in isolation is often insufficient for creating a truly powerful app - the real power comes when you can combine them with other sources of computation or knowledge.\n\nThis


the real power comes when you can combine them with other sources of computation or knowledge.\n\nThis library aims to assist in the development of those types of applications. Common examples of these applications include:\n\nÃ¢\x9dâ€œ Question Answering over specific documents\n\nDocumentation\n\nEnd-to-end Example: Question Answering over Notion Database\n\nÃ°\x9fâ€™Â¬ Chatbots\n\nDocumentation\n\nEnd-to-end Example: Chat-LangChain\n\nÃ°\x9fÂ¤\x96 Agents\n\nDocumentation\n\nEnd-to-end Example: GPT+WolframAlpha\n\nÃ°\x9fâ€œ\x96 Documentation\n\nPlease see here for full documentation on:\n\nGetting started (installation, setting up the environment, simple examples)\n\nHow-To examples (demos, integrations, helper functions)\n\nReference (full API docs)\n\nResources (high-level explanation of core concepts)\n\nÃ°\x9f\x9a\x80 What can this help with?\n\nThere are six main areas that LangChain is designed to help with.\nThese are, in increasing order of complexity:\n\nÃ°\x9fâ€œ\x83 LLMs and Prompts:\n\nThis includes prompt management, prompt optimization, a generic interface for all LLMs, and common utilities for working with LLMs.\n\nÃ°\x9fâ€�\x97 Chains:\n\nChains go beyond a single LLM call and involve sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications.\n\nÃ°\x9fâ€œ\x9a Data Augmented Generation:\n\nData Augmented Generation involves specific types of chains that first


Data Augmented Generation:\n\nData Augmented Generation involves specific types of chains that first interact with an external data source to fetch data for use in the generation step. Examples include summarization of long pieces of text and question/answering over specific data sources.\n\nÃ°\x9fÂ¤\x96 Agents:\n\nAgents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end-to-end agents.\n\nÃ°\x9fÂ§\xa0 Memory:\n\nMemory refers to persisting state between calls of a chain/agent. LangChain provides a standard interface for memory, a collection of memory implementations, and examples of chains/agents that use memory.\n\nÃ°\x9fÂ§\x90 Evaluation:\n\n[BETA] Generative models are notoriously hard to evaluate with traditional metrics. One new way of evaluating them is using language models themselves to do the evaluation. LangChain provides some prompts/chains for assisting in this.\n\nFor more information on these concepts, please see our full documentation.\n\nÃ°\x9fâ€™\x81 Contributing\n\nAs an open-source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation.\n\nFor detailed information on how to contribute, see here.", metadata={'source': '../../../../../README.md'})]Retain Elementsâ€‹Under the hood, Unstructured creates different "elements" for different chunks of text. By default we combine those together, but you can easily keep that separation by specifying mode="elements".loader = UnstructuredMarkdownLoader(markdown_path, mode="elements")data = loader.load()data[0]


mode="elements")data = loader.load()data[0]    Document(page_content='Ã°\x9fÂ¦\x9cÃ¯Â¸\x8fÃ°\x9fâ€�\x97 LangChain', metadata={'source': '../../../../../README.md', 'page_number': 1, 'category': 'Title'})PreviousJSONNextPDFCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


File Directory | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionDocument loadersCSVFile DirectoryHTMLJSONMarkdownPDFDocument transformersText embedding modelsVector storesRetrieversChainsMemoryAgentsCallbacksModulesGuidesEcosystemAdditional resourcesModulesData connectionDocument loadersFile DirectoryFile DirectoryThis covers how to load all documents in a directory.Under the hood, by default this uses the UnstructuredLoaderfrom langchain.document_loaders import DirectoryLoaderWe can use the glob parameter to control which files to load. Note that here it doesn't load the .rst file or the .html files.loader = DirectoryLoader('../', glob="**/*.md")docs = loader.load()len(docs)    1Show a progress barâ€‹By default a progress bar will not be shown. To show a progress bar, install the tqdm library (e.g. pip install tqdm), and set the show_progress parameter to True.loader = DirectoryLoader('../', glob="**/*.md", show_progress=True)docs = loader.load()    Requirement already satisfied: tqdm in /Users/jon/.pyenv/versions/3.9.16/envs/microbiome-app/lib/python3.9/site-packages (4.65.0)    0it [00:00, ?it/s]Use multithreadingâ€‹By default the loading happens in one thread. In order to utilize several threads set the use_multithreading flag to true.loader = DirectoryLoader('../', glob="**/*.md", use_multithreading=True)docs = loader.load()Change loader classâ€‹By default this uses the UnstructuredLoader class. However, you can change up the type of loader pretty easily.from langchain.document_loaders import TextLoaderloader =


can change up the type of loader pretty easily.from langchain.document_loaders import TextLoaderloader = DirectoryLoader('../', glob="**/*.md", loader_cls=TextLoader)docs = loader.load()len(docs)    1If you need to load Python source code files, use the PythonLoader.from langchain.document_loaders import PythonLoaderloader = DirectoryLoader('../../../../../', glob="**/*.py", loader_cls=PythonLoader)docs = loader.load()len(docs)    691Auto detect file encodings with TextLoaderâ€‹In this example we will see some strategies that can be useful when loading a big list of arbitrary files from a directory using the TextLoader class.First to illustrate the problem, let's try to load multiple text with arbitrary encodings.path = '../../../../../tests/integration_tests/examples'loader = DirectoryLoader(path, glob="**/*.txt", loader_cls=TextLoader)A. Default Behaviorâ€‹loader.load()
â•â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€ Traceback <span style="color: #bf7f7f; text-decoration-color: #bf7f7f;


#bf7f7f; text-decoration-color: #bf7f7f; font-weight: bold">(most recent call last) â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â•®â”‚ /data/source/langchain/langchain/document_loaders/text.py:29 in load                             â”‚â”‚


â”‚â”‚   26 â”‚   â”‚   text = ""                                                                           â”‚â”‚   27 â”‚


text-decoration-color: #7f7f7f">â”‚   â”‚   with open(self.file_path, encoding=self.encoding) as f:                             â”‚â”‚   28 â”‚   â”‚   â”‚   try:                                                                            <span style="color:


â”‚â”‚ â�± 29 â”‚   â”‚   â”‚   â”‚   text = f.read()                                                             â”‚â”‚   30 â”‚   â”‚   â”‚   except UnicodeDecodeError as e:


text-decoration-color: #0000ff">as e:                                                 â”‚â”‚   31 â”‚   â”‚   â”‚   â”‚   if self.autodetect_encoding:                                                â”‚â”‚   32 â”‚   â”‚


#7f7f7f">â”‚   â”‚   â”‚   â”‚   â”‚   detected_encodings = self.detect_file_encodings()                       â”‚â”‚                                                                                â”‚â”‚ /home/spike/.pyenv/versions/3.9.11/lib/python3.9/codecs.py:322 in <span style="color: #00ff00; text-decoration-color:


in decode                         â”‚â”‚                                                                                â”‚â”‚    319 â”‚   def decode(self, input, final=<span


#00ffff; text-decoration-color: #00ffff">input, final=False):                                                 â”‚â”‚    320 â”‚   â”‚   # decode input (taking the buffer into account)                                   â”‚â”‚    321 â”‚   â”‚   data = <span


â”‚   data = self.buffer + input                                                        â”‚â”‚ â�±  322 â”‚   â”‚   (result, consumed) = self._buffer_decode(data, self.errors, final)                â”‚â”‚    323 <span style="color: #7f7f7f; text-decoration-color:


â”‚   â”‚   # keep undecoded input until the next call                                        â”‚â”‚    324 â”‚   â”‚   self.buffer = data[consumed:]                                                     â”‚â”‚    325 <span style="color:


#7f7f7f"> 325 â”‚   â”‚   return result                                                                     â”‚<span style="color: #800000; text-decoration-color:


style="color: #800000; text-decoration-color: #800000">â•°â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â•¯UnicodeDecodeError: 'utf-8' codec can't decode byte <span style="color: #008080; text-decoration-color: #008080; font-weight:


style="color: #008080; text-decoration-color: #008080; font-weight: bold">0xca in position 0: invalid continuation byteThe above exception was the direct cause of the following exception:â•â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€ Traceback (most recent call last) â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â•®â”‚


#800000; text-decoration-color: #800000">â”‚ in <module>:1                                                                                â”‚â”‚                                                                                â”‚â”‚ â�± 1 loader.load()


â”‚â”‚   2                                                                                â”‚â”‚                                                                                <span style="color: #800000; text-decoration-color:


â”‚â”‚ /data/source/langchain/langchain/document_loaders/directory.py:84 in load                        â”‚â”‚                                                                                â”‚â”‚   <span style="color: #7f7f7f;


81 â”‚   â”‚   â”‚   â”‚   â”‚   â”‚   if self.silent_errors:                                              â”‚â”‚   82 â”‚   â”‚   â”‚   â”‚   â”‚   â”‚   â”‚   logger.warning(e)                                               â”‚<span


#800000; text-decoration-color: #800000">â”‚â”‚   83 â”‚   â”‚   â”‚   â”‚   â”‚   â”‚   else:                                                               â”‚â”‚ â�± 84 â”‚   â”‚   â”‚   â”‚   â”‚   â”‚   â”‚   raise e


â”‚â”‚   85 â”‚   â”‚   â”‚   â”‚   â”‚   finally:                                                                â”‚â”‚   86 â”‚   â”‚   â”‚   â”‚   â”‚


â”‚   â”‚   â”‚   â”‚   â”‚   if pbar:                                                            â”‚â”‚   87 â”‚   â”‚   â”‚   â”‚   â”‚   â”‚   â”‚   pbar.update(1)                                                  â”‚â”‚


â”‚â”‚ /data/source/langchain/langchain/document_loaders/directory.py:78 in load                        â”‚â”‚


â”‚â”‚   75 â”‚   â”‚   â”‚   if i.is_file():                                                                 â”‚â”‚   76 â”‚   â”‚   â”‚   â”‚   <span style="color: #0000ff; text-decoration-color:


if _is_visible(i.relative_to(p)) or self.load_hidden:                       â”‚â”‚   77 â”‚   â”‚   â”‚   â”‚   â”‚   try:                                                                    â”‚â”‚ â�±


style="color: #800000; text-decoration-color: #800000">â�± 78 â”‚   â”‚   â”‚   â”‚   â”‚   â”‚   sub_docs = self.loader_cls(str(i), **self.loader_kwargs).load()     â”‚â”‚   79 â”‚   â”‚   â”‚   â”‚   â”‚   â”‚   docs.extend(sub_docs)                                               â”‚â”‚


#800000; text-decoration-color: #800000">â”‚   80 â”‚   â”‚   â”‚   â”‚   â”‚   except Exception as e:                                                  â”‚â”‚   81 â”‚   â”‚   â”‚   â”‚   â”‚   â”‚   if <span style="color: #00ffff; text-decoration-color:


self.silent_errors:                                              â”‚â”‚                                                                                â”‚â”‚ /data/source/langchain/langchain/document_loaders/text.py:44 in load


â”‚â”‚                                                                                â”‚â”‚   41 â”‚   â”‚   â”‚   â”‚   â”‚   â”‚   except UnicodeDecodeError:                                          <span style="color:


â”‚â”‚   42 â”‚   â”‚   â”‚   â”‚   â”‚   â”‚   â”‚   continue                                                        â”‚â”‚   43 â”‚   â”‚   â”‚   â”‚   else:


â”‚â”‚ â�± 44 â”‚   â”‚   â”‚   â”‚   â”‚   raise RuntimeError(f"Error loading {self.file_path}") from e            â”‚<span


#800000; text-decoration-color: #800000">â”‚â”‚   45 â”‚   â”‚   â”‚   except Exception as e:                                                          â”‚â”‚   46 â”‚   â”‚   â”‚   â”‚   raise <span style="color:


text-decoration-color: #0000ff">raise RuntimeError(f"Error loading {self.file_path}") from e                â”‚â”‚   47                                                                                â”‚<span style="color: #800000; text-decoration-color:


style="color: #800000; text-decoration-color: #800000">â•°â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â•¯RuntimeError: Error loading ../../../../../tests/integration_tests/examples/<span style="color: #ff00ff; text-decoration-color: