delayedkarma/langchain-issues huggingface dataset:
Langchain agent not executing properly in Celery worker running as Docker container
Labels: name: area: agent, name: auto:bug

### System Info

Langchain version: 0.0.341
OpenAI version: 1.3.5
Model: gpt-4-1106-preview
Python version:3.10.13

Platform: Celery worker in Docker Container

### Who can help?

@eyurtsev @hwchase17 @ag

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I am working on implementing LangChain Agents in my Python project I am running this project using docker compose. In my project I am using Celery worker and have multiple worker services which executes from the queue. The entire setup is working fine and all celery tasks are executed as expected.

One of these workers in agent worker where I have configured LangChain Agent. I have created a function where I am loading tools, initializing agent and passing agent input. Here's the full code of my **agent module**:
Add `rag-conversation-cohere` template
Labels: name: area: embeddings, name: auto:enhancement, name: template, name: size:L

- **Description:** Following a private discussion with @rlancemartin and @SandraKublik, this PR adds a new template for RAG Conversation using Cohere Embed v3 and Cohere Chat.

This template is very similar to the [Rag Conversation template](https://github.com/langchain-ai/langchain/tree/master/templates/rag-conversation) which uses OpenAI's API instead for the embeddings and chat endpoints.

I have tested it locally and it works as expected.

If this PR gets merged, we may want to rename the Rag Conversation template to specify that OpenAI is used for the embeddings and chat (for instance to something like: `rag-conversation-openai`).

- **Issue:** N/A
- **Dependencies:** No dependencies added for this change.
- **Twitter handle:** [@smlpth](https://twitter.com/smlpth)


<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Add `rag-pinecone-cohere-embed-rerank` template
Labels: name: area: embeddings, name: auto:enhancement, name: integration: pinecone, name: template, name: size:L

- **Description:** Following a private discussion with @rlancemartin and @SandraKublik, this PR adds a new template using:
  - Cohere Embed v3
  - Cohere Chat
  - Cohere Re-Rank

This template is very similar to the [Rag Pinecone Rerank template](https://github.com/langchain-ai/langchain/blob/master/templates/rag-pinecone-rerank) which uses OpenAI's API instead for the embeddings and chat endpoints.

I have tested it locally and it works as expected.

If this PR gets merged, we may want to rename the Rag Pinecone Rerank template to specify that OpenAI is used for the embeddings and chat (for instance to something like: `rag-pinecone-openai-embed-rerank`).

- **Issue:** N/A
- **Dependencies:** No dependencies added for this change.
- **Twitter handle:** [@smlpth](https://twitter.com/smlpth)


<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
How to query with new variables in LCEL mode?
Labels: name: auto:question, name: area: lcel

### System Info

![image](https://github.com/langchain-ai/langchain/assets/128944251/ba7425de-9f08-44b1-93a5-aad3d6115add)

I am plannign to add new param like "affeciton"

How could I set the query databody to fill up the params here?( Langserve setup !)

![image](https://github.com/langchain-ai/langchain/assets/128944251/57c56f5d-407e-40d8-826d-1f6400409e71)



### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

How to handle the input variables in LCEL mode?

### Expected behavior

get the right place I put in when I  new a variables in the prompt.
community: avoid KeyError when language not in LANGUAGE_SEGMENTERS
Labels: name: area: doc loader, name: auto:bug, name: size:XS

**Description:**

Handle unsupported languages in same way as when none is provided 
 
**Issue:**

The following line will throw a KeyError if the language is not supported.
```python
self.Segmenter = LANGUAGE_SEGMENTERS[language]
```
E.g. when using `Language.CPP` we would get `KeyError: <Language.CPP: 'cpp'>`
Error using ConversationalRetrievalChain.from_llm: "document_variable_name context was not found in llm_chain input_variables: [] (type=value_error)"
Labels: name: auto:bug, name: area: models

### System Info

MacOS, M1 Pro

### Who can help?

@hwchase17 @agola11 

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [X] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Run the following code:

```
import os
from dotenv import load_dotenv
from langchain.chains import ConversationalRetrievalChain
from langchain.prompts.chat import (
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
    SystemMessagePromptTemplate,
)
from langchain.chat_models import ChatOllama
from langchain.vectorstores import FAISS
from langchain.document_loaders import PyPDFLoader
from langchain.embeddings.openai import OpenAIEmbeddings

load_dotenv()

messages = [
    SystemMessagePromptTemplate.from_template(
        "You are a truthful, accurate AI agent that responds to the user's questions, given an AI paper by Apple."
    ),
    HumanMessagePromptTemplate.from_template("What is the paper about, in summary?"),
]
qa_prompt = ChatPromptTemplate.from_messages(messages)
chat_model = ChatOllama(
    model="mistral",
)

loader = PyPDFLoader("./llm_in_a_flash_apple.pdf")
pages = loader.load_and_split()
embeddings = OpenAIEmbeddings(api_key=os.getenv("OPENAI_API_KEY"))

print(pages[0])


db = None
if not os.path.exists("./faiss_index"):
    db = FAISS.from_documents(pages, embeddings)
    db.save_local("./faiss_index")
else:
    db = FAISS.load_local("faiss_index", embeddings)

query = "What is the paper about?"
docs = db.similarity_search_with_score(query)
print(docs[0])

ConversationalRetrievalChain.from_llm(
    llm=chat_model,
    retriever=db.as_retriever(search_type="similarity", search_kwargs={"k": 0.8}),
    verbose=True,
    combine_docs_chain_kwargs={"prompt": qa_prompt},
    return_source_documents=True,
)

```

additional files here: https://github.com/polooner/chatpdf/blob/main/main.py

### Expected behavior

An answer from the Chat Model
community: Fix Baichuan Chat.
Labels: name: auto:bug, name: area: models, name: size:L

  - **Description:** Baichuan Chat (with both Baichuan-Turbo and Baichuan-Turbo-192K models) has updated their APIs. There are breaking changes. For example, BAICHUAN_SECRET_KEY is removed in the latest API but is still required in Langchain. Baichuan's Langchain integration needs to be updated to the latest version. 
  - **Issue:** #15206
  - **Dependencies:** None,
  - **Twitter handle:** None

@hwchase17.
Fix Baichuan's integration and introduce Baichuan-Turbo-192K API.
Labels: name: auto:improvement, name: area: models

### System Info

Baichuan Chat (with both Baichuan-Turbo and Baichuan-Turbo-192K models) has updated their APIs. There are breaking changes. For example, BAICHUAN_SECRET_KEY is removed in the latest API but is still required in Langchain. Baichuan's Langchain integration needs to be updated to the latest version. 

Also we have released out new Baichuan-Turbo-192K API. We are adding support for this.

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

https://python.langchain.com/docs/integrations/chat/baichuan

SECRET_KEY has been deprecated.

### Expected behavior

Baichuan Chat works normally.
Issue: Streaming Response contains the rephrased question in ConversationalRetrievalChain
Labels: name: auto:bug, name: area: models

### Issue you'd like to raise.

I am using ConversationalRetrievalChain
with a callback handler for streaming responses back.
  

>    qa_chain =ConversationalRetrievalChain.from_llm(
        llm=chat,
        retriever=MyVectorStoreRetriever(
            vectorstore=vectordb,
            search_type="similarity_score_threshold",
            search_kwargs={"score_threshold": SIMILARITY_THRESHOLD, "k": 1},
        ),
        return_source_documents=True,
        rephrase_question=False,
        return_generated_question=False,
    )  







> response = qa_chain(
        {
            "question": user_input,
            "chat_history": chat_history,
        },
        callbacks=[stream_handler],
    )


```
class StreamHandler(BaseCallbackHandler):
    def __init__(self):
        self.text = ""

      def on_llm_new_token(self, token: str, **kwargs: Any):
        # Initialize old_text
        old_text = self.text
        print("old text ", old_text)

        # Check if the token is not part of the prompts before adding it to the queue
        print("token is", token)
        if token is not None and token != "":
            self.text += token

        # Calculate the new content since the last emission
        new_content = self.text[len(old_text) :]

        socketio.emit("update_response", {"response": new_content})

```

I have provided the value of rephrase_question and return_generated_question False.
Even after that the streaming response contains the rephrased question.
But the final response from the LLM does not contain this rephrased question.
what could be the reason, please provide an appropriate solution. 



### Suggestion:

_No response_
Pydantic forward ref issue when creating using LlamaCpp with grammar
Labels: name: auto:bug, name: area: models

### System Info

OS: MacOS Sonoma
Python: 3.11.6
LangChain: 0.0.352
llama-cpp-python = 0.2.25
pydantic: 1.10.13 (I know that it is not the latest version, but version 1 is still officially supported)

### Who can help?

@hwchase17
@agola11

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

When trying to use LlamaCpp in conjunction with grammar, I get an error from pydantic. The following code snipped was adapted from the [docs](https://python.langchain.com/docs/integrations/llms/llamacpp#grammars): so that a `LlamaGrammar` is passed, instead of the path to the grammar file.

```python
from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain_community.llms.llamacpp import LlamaCpp
from llama_cpp.llama_grammar import LlamaGrammar
from pydantic import BaseModel


class SomeSchema(BaseModel):
    some_field: str


LlamaCpp(
    model_path="some model",
    callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]),
    verbose=True,
    grammar=LlamaGrammar.from_json_schema(SomeSchema.schema_json()),
)
# Fails with:
# pydantic.errors.ConfigError: field "grammar" not yet prepared so type is still a ForwardRef, you might need to call LlamaCpp.update_forward_refs().
```

The following works though (and the grammar object is used properly:

```python
from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain_community.llms.llamacpp import LlamaCpp
from llama_cpp.llama_grammar import LlamaGrammar
from pydantic import BaseModel


class SomeSchema(BaseModel):
    some_field: str

model = LlamaCpp(
    model_path="some model",
    callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]),
    verbose=True,
)
model.grammar = LlamaGrammar.from_json_schema(SomeSchema.schema_json())
```

### Expected behavior

It should be possible to pass a `LlamaGrammar` object in the `__init__` of `LlamaCpp`, as per its [definition](https://github.com/langchain-ai/langchain/blob/f36ef0739dbb548cabdb4453e6819fc3d826414f/libs/community/langchain_community/llms/llamacpp.py#L129)

I had a quick look at the pydantic [documentation regarding this problem](https://docs.pydantic.dev/1.10/usage/postponed_annotations/), but I couldn't find the postponed annotation in question.
Issue: Identical Content in Two Files
Labels: name: area: doc loader, name: auto:refactor

### Issue you'd like to raise.

https://github.com/langchain-ai/langchain/blob/master/libs/community/langchain_community/tools/render.py
https://github.com/langchain-ai/langchain/blob/master/libs/community/langchain_community/tools/convert_to_openai.py

For backward compatibility purposes, should we proceed with a direct import?

### Suggestion:

_No response_
"Could not parse LLM output" when using QianfanChatEndpoint in agent.
Labels: name: auto:bug, name: area: models

### System Info

langchian=0.0.352
qianfan=0.2.4

When I tried the usage of agent in this [video](https://learn.deeplearning.ai/langchain/lesson/7/agents), I changed the model in it from ChatGpt-3.5-turbo to ERNIE-Bot, and the output of agent showed the following error:

```bash
> Entering new AgentExecutor chain...
Could not parse LLM output: xxxxxxxxx
Observation: Invalid or incomplete response
Thought: Could not parse LLM output: xxxxx
Observation: Invalid or incomplete response
...
```

And, ERNIE-Bot can't call (llm-math) tool correctly.

I wonder if the problem is a lack of capability in the qianfan model itself, or if there is a problem in the qianfan code.
Or is there something wrong with my usage or other issues?

### Who can help?

@danielhjz 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

**my code**

```python
llm = QianfanChatEndpoint(
    temperature=0.000001,
    model='ERNIE-Bot'
)
tools = load_tools(
    ["llm-math", "wikipedia"],
    llm=llm
)
agent = initialize_agent(
    tools,
    llm,
    agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION,
    handle_parsing_errors=True,
    verbose=True
)
agent("300的1/4是多少？")
```

**code in the video**

```python
# code in the video
llm = ChatOpenAI(
    temperature=0
)
tools = load_tools(
    ["llm-math", "wikipedia"],
    llm=llm
)
agent = initialize_agent(
    tools,
    llm,
    agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION,
    handle_parsing_errors=True,
    verbose=True
)
agent("What is the 25% of 300?")
```

### Expected behavior

**Run by ChatOpenAI(temperature=0)**

````bash
> Entering new AgentExecutor chain...
Thought: We need to calculate 25% of 300, which means we need to multiply 300 by 0.25.

Action:
```
{
  "action": "Calculator",
  "action_input": "300*0.25"
}
```

Observation: Answer: 75.0
Thought:The calculator tool returned the answer 75.0, which is correct.

Final Answer: 25% of 300 is 75.0.

> Finished chain.
{'input': 'What is the 25% of 300?', 'output': '25% of 300 is 75.0.'}
````
community: Fix inconsistent initialization for ChatTongyi
Labels: name: auto:bug, name: area: models, name: size:S

## Description: 
When using ChatTongyi, specifying the "model_name" parameter does not take effect (only the "model" parameter works). This inconsistency with other chat models can lead to unexpected model calls.

###  Demonstration:

```Python
In[1]: DASHSCOPE_API_KEY = "sk-xxx"
In[2]: from langchain.chat_models.tongyi import ChatTongyi
In[3]: tongyi_model = ChatTongyi(model_name="qwen-plus",dashscope_api_key=DASHSCOPE_API_KEY)
In[4]: tongyi_model.model_name
Out[5]: 'qwen-turbo'  # expected 'qwen-plus'
```

However, other chat models like ChatOpenAI can be assigned using the "model_name" parameter, which makes the behavior of the ChatTongyi model inconsistent.

```python
In[6]: from langchain.chat_models import ChatOpenAI
In[7]: open_ai_model = ChatOpenAI(openai_api_key="sk-xxx", model_name="gpt-4")
In[8]: open_ai_model.model_name
Out[9]: 'gpt-4'
```

## How to fix:

Set pydantic configuration `allow_population_by_field_name=True`

@baskaryan
Patch: improve openai functions call parser compatibility
Labels: name: area: agent, name: auto:improvement, name: size:XS

```shell
Python 3.11.6 (main, Nov  2 2023, 04:39:43) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> s = {'name': 'gc', 'arguments': '{"prompt":"hi\nbob."}'}
>>> import json
>>> json.loads(s['arguments'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/homebrew/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
               ^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Invalid control character at: line 1 column 14 (char 13)
>>> json.loads(s['arguments'].replace('\n', '\\n'))
{'prompt': 'hi\nbob.'}
>>>
```
Fixed minor type in  self_query.ipynb
Labels: name: lgtm, name: area: doc loader, name: auto:documentation, name: size:XS

<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Issue: lcel langserve with_fallbacks streaming
Labels: name: auto:question, name: area: langserve

### Issue you'd like to raise.

```python
conversational_qa_chain = (
        _inputs | _context | ConfigurableTokenLimitProcessor(model="gpt_35_turbo").configurable_fields(
    model=ConfigurableFieldSingleOption(
        id="model",
        name="model",
        options={
            "gpt_35_turbo": "gpt_35_turbo",
            "gpt_35_turbo_1106": "gpt_35_turbo_1106",
            "gpt_4_1106_preview": "gpt_4_1106_preview",
            "gpt_4_32k": "gpt_4_32k"
        },
        default="gpt_35_turbo",
    )
) | ANSWER_PROMPT | llm | StrOutputParser()
)
```
```python
chain = conversational_qa_chain.with_types(input_type=ChatHistory).with_fallbacks([RunnableLambda(when_all_is_lost)])
```
```python
add_routes(app,
           chain,
           enable_feedback_endpoint=True,
           path="/test",
           config_keys=["llm", "collection_name", "model", "configurable"]
           )
```

It's a code developed with langserve, but if i send a request to `/test/stream` using playground, unlike before adding the `with_fallbacks` function, the response is not exposed on the screen by token, but all the responses are shown on the screen at once, what's the reason?

### Suggestion:

Even if i add `with_fallbacks`, it should be streamed on the screen for each token.