delayedkarma/langchain-issues huggingface dataset:
core: Update messages/__init__.py to account for AIMessageChunk which breaks message history runnable.
Labels: name: auto:bug, name: area: langserve, name: size:S

  - **Description:** fix parse issue for AIMessageChunk when using 
  - **Issue:** https://github.com/langchain-ai/langchain/issues/14511
  - **Dependencies:** none
  - **Twitter handle:** none

Taken from this fix: https://github.com/gpt-engineer-org/gpt-engineer/issues/804#issuecomment-1769853850

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
DOC: how to use embeddings in langchain with fireworks?
Labels: name: area: embeddings, name: auto:question

### Issue with current documentation:

how to use embeddings in langchain with fireworks?(I need it for RAG) It's just that the documentation only talks about OpenAIEmbeddings
https://python.langchain.com/docs/modules/data_connection/text_embedding/

### Idea or request for content:

RAG with fireworks API
langchain: Exclude non-utf8 file from loader since it causes an error in the code_understanding example
Labels: name: area: doc loader, name: auto:bug, name: size:XS

  - **Description:** in the code_understanding.ipynb example, the loader errors out on the langchain/libs/community/tests/examples/non-utf8-encoding.py file, so I updated the loader to exclude that file. Excluding that file allows the example to run.
  - **Issue:** not applicable
  - **Dependencies:** none
langchain: minor changes to StuffDocumentsChain._get_inputs
Labels: name: lgtm, name: area: doc loader, name: auto:nit, name: size:XS

Correcting a small typo ('the' instead of 'then') and changing another 'the' (instead of 'then' too, it was a hard day for the 'n' key :D) to 'also' to match better with what is done in the code

<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Issue: Error Handling in Tools used in custom agents
Labels: name: area: agent, name: auto:bug

### Issue you'd like to raise.

I have built a custom LLM Agent by following the Documentation provided. The custom agent contains multiple tools, one of them is the "LLMMathChain" which is giving me ValueError, cause my agent is passing "None" as an Action Input. I want to handle that error. So that my chatbot doesn't break in the middle of a conversation.

## My Custom Agent
![image](https://github.com/langchain-ai/langchain/assets/51261048/0750c101-871d-4146-9dfb-5a2a52e7ff3b)


## Calculator Tool
![image](https://github.com/langchain-ai/langchain/assets/51261048/123708d8-afc0-4ed3-82ad-84c28881d95e)

## Prompt Template
![image](https://github.com/langchain-ai/langchain/assets/51261048/90ee198d-075f-40bc-b248-ea0ba1292f20)

## Output Parser
![image](https://github.com/langchain-ai/langchain/assets/51261048/1ba3af29-c650-4fcd-ad64-f11b371f0977)


### Suggestion:

_No response_
Add common mulit model support 
Labels: name: auto:enhancement, name: area: models

### Feature request

Presently, JSON can be utilized to enable the multimodal capability of GPT-4 series models within ChatOpenAI and OpenAI. However, this functionality lacks portability.

### Motivation

Using multimodal approaches lacks portability, and GPT-4 isn't the sole model employing multimodal capabilities. Therefore, it becomes imperative to establish a standardized method for accessing various multimodal models.

### Your contribution

I may submit a PR about this if I have spare time
Issue: How can I implement Conversation Chain along with ConversationSummaryMemory
Labels: name: area: memory, name: auto:question, name: integration: chroma

### Issue you'd like to raise.

below is my code, How can I implement Conversation Chain along with ConversationSummaryMemory in my code 
`def retreival_qa_chain(chroma_db_path):
    embedding = OpenAIEmbeddings()
    vectordb = Chroma(persist_directory=chroma_db_path, embedding_function=embedding)
    llm = ChatOpenAI(temperature=0.1)
    retriever = vectordb.as_retriever(search_kwargs={"k": 2})
    qa = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=retriever,return_source_documents=True)
    return qa


def create_global_qa_chain():
    chroma_db_path = "chroma-databases"
    folders = os.listdir(chroma_db_path)
    qa_chains = {}
    for index, folder in enumerate(folders):
        folder_path = f"{chroma_db_path}/{folder}"
        project = retreival_qa_chain(folder_path)
        qa_chains[folder] = project
    return qa_chains`

### Suggestion:

_No response_
Issue: document_variable_name context was not found in llm_chain input_variables
Labels: name: auto:bug, name: area: models

### Issue you'd like to raise.

I am trying to add a specific prompt template to my ConversationalRetrievalChain. This is my current code:

> PROMPT_TEMPLATE = """
Act as the policies interactive Bot that gives advice on the Company policies, Travel policies, and Information security policies for the company.
Do not try to make up an answer. Use only the given pieces of context; do not use your own knowledge.
Chat History:
{chat_history}
Follow Up Input: {question}
"""
qa_prompt = PromptTemplate(input_variables=["chat_history", "question",], template=PROMPT_TEMPLATE)

>    chat = ChatOpenAI(
        verbose=True,
        model_name=MODEl_NAME,
        temperature=TEMPERATURE,
        max_retries=MAX_RETRIES,
        streaming=True,
    )
  qa_chain =ConversationalRetrievalChain.from_llm(
        llm=chat,
        retriever=MyVectorStoreRetriever(
            vectorstore=vectordb,
            search_type="similarity_score_threshold",
            search_kwargs={"score_threshold": SIMILARITY_THRESHOLD, "k": 1},
        ),
        return_source_documents=True,
        combine_docs_chain_kwargs={'prompt': qa_prompt}, )
      response = qa_chain(
        {
            "question": query,
            "chat_history": chat_history,
        },
        callbacks=[stream_handler],
    )
    

This is the error I'm  currently getting,

>   qa_chain =ConversationalRetrievalChain.from_llm(
  File "/home/sfm/anaconda3/envs/chat_v2/lib/python3.10/site-packages/langchain/chains/conversational_retrieval/base.py", line 360, in from_llm
    doc_chain = load_qa_chain(
  File "/home/sfm/anaconda3/envs/chat_v2/lib/python3.10/site-packages/langchain/chains/question_answering/__init__.py", line 249, in load_qa_chain
    return loader_mapping[chain_type](
  File "/home/sfm/anaconda3/envs/chat_v2/lib/python3.10/site-packages/langchain/chains/question_answering/__init__.py", line 81, in _load_stuff_chain
    return StuffDocumentsChain(
  File "/home/sfm/anaconda3/envs/chat_v2/lib/python3.10/site-packages/langchain/load/serializable.py", line 97, in __init__
    super().__init__(**kwargs)
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for StuffDocumentsChain
__root__
  document_variable_name context was not found in llm_chain input_variables: ['chat_history', 'question'] (type=value_error)



can you  help me to figure out the error and correct it?




### Suggestion:

_No response_
Feat: support RunnableBranch constructed with `|`
Labels: name: auto:improvement, name: area: langserve, name: size:S

```python
from langchain_core.runnables import RunnableBranch

branch = RunnableBranch(
    (lambda x: isinstance(x, str), lambda x: x.upper()),
    (lambda x: isinstance(x, int), lambda x: x + 1),
    (lambda x: isinstance(x, float), lambda x: x * 2),
    lambda x: "goodbye",
  )

branch.invoke("hello") # "HELLO"
branch.invoke(None) # "goodbye"

# A RunnableBranch constructed using the `|` operator
branch = branch | (lambda x: isinstance(x, bool), lambda x: f'{x}')
branch.invoke(true) # "true"
```
Slow aadd_documents using Qdrant and HuggingfaceEmbeddings on CPU
Labels: name: area: embeddings, name: auto:bug

### System Info

lc: 0.0.352, os: ubuntu 22, python 3.10

### Who can help?

### Description

I am encountering a significant performance issue when using Qdrant with HuggingfaceEmbeddings in a CPU-only environment, specifically within a FastAPI endpoint. The process is notably slow, particularly at the `aadd_documents(...)` stage.


### Additional Information

- As a comparison, I tried embedding a document directly using `sentence_transformers`. This approach utilized all CPU cores, resulting in a much faster process.
- I also experimented with a custom implementation, using only necessary functions from [this Qdrant file](https://github.com/langchain-ai/langchain/blob/master/libs/community/langchain_community/vectorstores/qdrant.py) to perform `aadd_documents`. This approach also showed improved performance and full CPU utilization.

### Question

Does anyone have an idea or suggestion on what might be causing this performance bottleneck when using Qdrant with HuggingfaceEmbeddings in a CPU-only environment?



### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

### Steps to Reproduce

1. Set up Qdrant with HuggingfaceEmbeddings in a CPU-only machine (no GPU).
2. Integrate it within a FastAPI endpoint.
3. Execute `aadd_documents(...)` for documents (for example, documents with around 45K characters).

### Expected behavior

### Expected Behavior

I expected the embedding and addition of documents to Qdrant to be efficient and utilize multiple CPU cores effectively.

### Observed Behavior

- The embedding process for a document of approximately 45K characters took over one minute.
- Resource utilization monitoring showed that only one out of 70 CPU cores was being utilized during the embedding process.
The retrieval cannot be given the document correctly
Labels: name: area: vector store, name: auto:bug

### System Info

langchain 0.0.340

### Who can help?

@eyurtsev

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

My vector library preserves tens of thousands of documents, but as the document increases, the accuracy rate of the retriever is becoming low, and the correct document cannot be retrieved.
The retrieval cannot be given the document correctly

### Expected behavior

    db = FAISS.load_local(VS['comixfaiss'], embeddings)
    retriever = db.as_retriever(search_type="similarity_score_threshold", search_kwargs={"score_threshold": 0.5,"k":5})
    logger.info(retriever.get_relevant_documents('3736085'))


LOG:
2023-12-29 15:44:51,650 - loader.py[line:56] - INFO: Successfully loaded faiss with AVX2 support.
2023-12-29 15:44:52,843 - local_doc_qa.py[line:204] - INFO: [Document(page_content='\ufeffSelection: 非校验\n商品编号: 3730559\n商品名称: 飞捷 FJ21325 39-45码（QXGZ）中筒 防水鞋户外雨靴套鞋胶鞋 黑色(单位：双）\n物料编码: 3730559\n大类: 生活用品\n中类: 办公日杂\n小类: 雨伞雨具\n品牌: 梦奇\n颜色: 黑色\n型号: FJ21325\n建议零售价: 77.45\n卖点: 品牌：飞捷 颜色：黑色 型号：FJ21325 包装清单：雨靴*1\n上架状态: 上架\n状态: \n状态信息:', metadata={'source': '/mnt/data/pdf/comixgpt/pd/齐心商城商品数据2万条2023-12-27.csv', 'row': 8975}), Document(page_content='\ufeffSelection: 非校验\n商品编号: 3051396\n商品名称: 得力 9387 三联送(销)货单据 129*188mm 20份/本 黄色 单位：本\n物料编码: 3051396\n大类: 办公文具\n中类: 财务行政用品\n小类: 财务单据\n品牌: 得力\n颜色: 黄色\n型号: 9387\n建议零售价: 4.35\n卖点: 0\n上架状态: 上架\n状态: \n状态信息:', metadata={'source': '/mnt/data/pdf/comixgpt/pd/齐心商城商品数据2万条2023-12-27.csv', 'row': 20709}), Document(page_content='\ufeffSelection: 非校验\n商品编号: 3278812\n商品名称: 惠普\xa0W9055MC\xa0成像鼓 彩色 （单位：个）\n物料编码: 3278812\n大类: 办公耗材\n中类: 打印机耗材\n小类: 硒鼓\n品牌: 惠普\n颜色: 彩色\n型号: W9055MC\n建议零售价: 3645.88\n卖点: 打印机耗材\n上架状态: 上架\n状态: \n状态信息:', metadata={'source': '/mnt/data/pdf/comixgpt/pd/齐心商城商品数据2万条2023-12-27.csv', 'row': 12167}), Document(page_content='\ufeffSelection: 非校验\n商品编号: 3197277\n商品名称: 得力 9307 报刊架 480*360*1450 银色 单位：个\n物料编码: 3197277\n大类: 办公文具\n中类: 会议展示用品\n小类: 报刊/杂志架\n品牌: 得力\n颜色: \n型号: 9307\n建议零售价: 330.33\n卖点: 0\n上架状态: 上架\n状态: \n状态信息:', metadata={'source': '/mnt/data/pdf/comixgpt/pd/齐心商城商品数据2万条2023-12-27.csv', 'row': 16685}), Document(page_content='\ufeffSelection: 非校验\n商品编号: 3278811\n商品名称: 惠普 W9054MC 成像鼓 黑色 （单位：个）\n物料编码: 3278811\n大类: 办公耗材\n中类: 打印机耗材\n小类: 硒鼓\n品牌: 惠普\n颜色: 黑色\n型号: W9054MC\n建议零售价: 2471.19\n卖点: 打印机耗材\n上架状态: 上架\n状态: \n状态信息:', metadata={'source': '/mnt/data/pdf/comixgpt/pd/齐心商城商品数据2万条2023-12-27.csv', 'row': 12168})]
new feature: add github file loader to load any github file content b…
Labels: name: area: doc loader, name: auto:enhancement, name: size:L

### Description
support load any github file content based on file extension.  

Why not use [git loader](https://python.langchain.com/docs/integrations/document_loaders/git#load-existing-repository-from-disk) ? 
git loader clones the whole repo even only interested part of files, that's too heavy.  This GithubFileLoader only downloads that you are interested files. 

### Twitter handle
my twitter: @shufanhaotop
How do i use similarity caching in my code?
Labels: name: area: embeddings, name: auto:question

Hi @dosu-bot,
This is my code
```
import langchain
from langchain.cache import SQLAlchemyCache, Emb
from sqlalchemy import create_engine
from sqlalchemy.orm import declarative_base
from sqlalchemy import Column, Integer, Text
from urllib.parse import quote_plus

from langchain.llms import OpenAI

Base = declarative_base()

class FulltextLLMCache(Base):
    __tablename__ = "llm_cache_full_text"
    id = Column(Integer, primary_key=True)
    prompt = Column(Text, nullable=False)
    llm = Column(Text, nullable=False)
    idx = Column(Integer)
    response = Column(Text)
    
db_uri = f"mssql+pyodbc://JUPYTER\SQLEXPRESS/my_database?driver=ODBC+Driver+17+for+SQL Server"
cache_engine = create_engine(db_uri, pool_recycle=240, pool_size=20, max_overflow=30)
# Assigning to llm_cache
langchain.llm_cache = SQLAlchemyCache(cache_engine, FulltextLLMCache)
```

The above code is for exact cache which is very low hit rate, how can i do similarity caching?
Cache not getting saved in ConversationalRetrievalChain
Labels: name: area: memory, name: auto:question

Hi @dosu-bot. 
Below is my code,

```
from langchain.cache import SQLAlchemyCache
from sqlalchemy import create_engine

engine = create_engine("mssql+pyodbc://JUPYTER\SQLEXPRESS/my_database?driver=ODBC+Driver+17+for+SQL Server")
set_llm_cache(SQLAlchemyCache(engine))

memory = ConversationBufferWindowMemory(k=2, memory_key="chat_history", chat_memory=chat_message_history ,return_messages=True, output_key="answer", input_key="question")
retriever = load_emdeddings(cfg.faiss_persist_directory, cfg.embeddings).as_retriever(search_type="similarity_score_threshold", 
                                 search_kwargs={"score_threshold": .65, 
                                                "k": 2})
custom_prompt_template = """
You are a friendly chatbot named "XYZ", designed to provide assistance and answer queries.


{context}
Chat History: {chat_history}

Question: {question}

"""

# Create a PromptTemplate instance with your custom template
custom_prompt = PromptTemplate(
    template=custom_prompt_template,
    input_variables=["context", "question", "chat_history", "User_Name", "User_Location"],
)
# Use your custom prompt when creating the ConversationalRetrievalChain
qa = ConversationalRetrievalChain.from_llm(
    llm,
    verbose=False,
    retriever=retriever,
    memory=memory,
    combine_docs_chain_kwargs={"prompt": custom_prompt},
    return_source_documents = True
)
```
if i use llm.predict("Tell me a joke"). I can see the cache is getting stored in db.
but for qa if i ask question its not saving , why?
Using chipper model with hi_res strategy gives an error
Labels: name: area: doc loader, name: auto:bug

### System Info

Hi,
I'm new to this, so I apologize if my lack of in-depth understanding to how this library works caused to me raise a false alarm. Im trying to an ocr on pdf image using the  UnstructuredPDFLoader, Im passing the following args:
`
loader = UnstructuredPDFLoader(file_path="myfile.pdf", mode="elements",include_page_break=True,infer_table_structure=False,languages=["Eng"],strategy="hi_res",include_metadata=True,model_name="chipper")`

However I keep getting the following error:

```
OSError: unstructuredio/chipper-v3 is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`
```

Not sure what Im missing here ?
Thanks



### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction


```
from langchain.document_loaders import UnstructuredPDFLoader
loader = UnstructuredPDFLoader(file_path="myfile.pdf", mode="elements",include_page_break=True,infer_table_structure=False,languages=["Eng"],strategy="hi_res",include_metadata=True,model_name="chipper")
                           

documents = loader.load()

print(documents)
```

### Expected behavior

I should be getting the metadata similar to when I use other models like "yolox" which works fine. I heard chipper model is much better so I wanted to try it.
Cannot specify asyn clienct for OpenAIAssistantRunnable
Labels: name: area: agent, name: auto:bug

### System Info

langchain = "^0.0.352"

@agola11 

### Who can help?

@agola11 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Specify async open ai client upon intialization

        client = openai.AsyncOpenAI()
        assistant = OpenAIAssistantRunnable(assistant_id=self.assistant_id,as_agent=as_agent,client=client)
        
        
   Produces error

pydantic.v1.errors.ConfigError: field "client" not yet prepared so type is still a ForwardRef, you might need to call OpenAIAssistantRunnable.update_forward_refs().

### Expected behavior

expect intialization to be successful
Improve markdown list parser
Labels: name: auto:improvement, name: area: langserve, name: size:S

- do not match text after - in the middle of a sentence

<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Strip code block fences and extra test from xml when doing streaming …
Labels: name: auto:improvement, name: area: langserve, name: size:M

…parse

<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->