delayedkarma/langchain-issues huggingface dataset:
Issue: <Please tell me how to combine Routing and RAG in a chain.>
Labels: name: auto:question, name: area: models

### Issue you'd like to raise.

I apologize for the naive question.it's not about an error or a bug.
I'm trying to implement routing by following the guide here: https://python.langchain.com/docs/modules/chains/foundational/router

However, I can't figure out how to use RAG.
I tried changing the last code in the guide like this.

```python
final_chain = (
    RunnablePassthrough.assign(topic=itemgetter("input") | classifier_chain)
    | prompt_branch
    | ChatOpenAI()
    | StrOutputParser()
)
```

```python
final_chain = (
    {
        "context": retriever,
        "topic": itemgetter("input") | classifier_chain,
    }
    | prompt_branch
    | llm
    | StrOutputParser()
)
```
But I get the following error:

```shell
  File "/Users/user/Library/Python/3.9/lib/python/site-packages/tiktoken/core.py", line 116, in encode
    if match := _special_token_regex(disallowed_special).search(text):
TypeError: expected string or buffer
```

### Suggestion:

_No response_
Issue: lcel `with_fallbacks` streaming
Labels: name: auto:question, name: area: lcel

### Issue you'd like to raise.

I'm going to make a chain through lcel and try to process an error through `with_fallbacks` at the end, but unlike before I put `with_fallbacks`, streaming is not possible and all responses go down at once. Can i process streaming using `with_fallbacks`?

### Suggestion:

lcel `with_fallbacks` streaming
Issue: openai api key rate limit error handing
Labels: name: auto:improvement, name: area: lcel

### Issue you'd like to raise.

If the rate limit of the api key is exceeded when developing a chain through lcel, I want to dynamically change another api key and retry to give the client a normal response, is there a way?

### Suggestion:

Dynamically catch an error in lcel, change the api key, and try again
Use tz-aware utc datetimes in tracer
Labels: name: lgtm, name: auto:improvement, name: area: langserve, name: size:L

<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Better input and output schemas for chains that start or end with a R…
Labels: name: auto:improvement, name: area: models, name: size:L

…unnableAssign or RunnablePick

<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Implement RunnablePassthrough.pick()
Labels: name: auto:improvement, name: area: models, name: size:L

<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
community: image_url download + image_url.url
Labels: name: auto:improvement, name: area: models, name: size:M

**Description:**
- adding actual image_url support by downloading the image using requests.get
- support for nested urls

**Issue:**
```python
HumanMessage(
    content=[
        {"type": "text", "text": "Describe the image:"},
        {
            "type": "image_url",
            "image_url": {
                "url": "https://images.unsplash.com/photo-1682687220777-2c60708d6889"
            },
        },
    ],
)
```
This was not possible because actual urls were not supportet and url objects inside the image_url was not supported.
Fix runnable vistitor for funcs without pos args
Labels: name: auto:improvement, name: area: langserve, name: size:XS

<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
DOC: How to write my own short scripts within a fork to test some code?
Labels: name: area: doc loader, name: auto:question, name: integration: chroma

### Issue with current documentation:

I want to contribute to one of the libs and started a fork. Here are the steps I took:

I am trying to add a new feature but first need to experiment with it. I am unsure on how to get started writing some short scripts to use the libs. 

1. I went into ```libs/experimental```, ```libs/core```, ```libs/community``` ```libs/langchain``` and ran ```poetry install``` in all of them.

2. I start an environment from ```libs/langchain``` with ```poetry shell```

3. I created a file inside of it, made some short code:

```from langchain.document_loaders import PyPDFLoader
from langchain.embeddings.sentence_transformer import SentenceTransformerEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import Chroma

# load the document and split it into chunks
loader = PyPDFLoader("./llm_in_a_flash_apple.pdf")
documents = loader.load_and_split()

# create the open-source embedding function
embedding_function = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")

# load it into Chroma
db = Chroma.from_documents(documents, embedding_function)

# query it
query = "What did the president say about Ketanji Brown Jackson"
docs = db.similarity_search(query)

# print results
print(docs[0].page_content)
```

and got the following error:

```
Traceback (most recent call last):
  File "/Users/polo/langchain/libs/langchain/test.py", line 9, in <module>
    loader = PyPDFLoader("./llm_in_a_flash_apple.pdf")
  File "/Users/polo/langchain/libs/community/langchain_community/document_loaders/pdf.py", line 154, in __init__
    raise ImportError(
ImportError: pypdf package not found, please install it with `pip install pypdf`
```

This is my first time in a Python project like this. I am unsure how to get started using all the different packages while in a fork of the repository. If anyone can guide me I would love to make a PR on this, it is quite daunting for beginners to get around and start contributing!

### Idea or request for content:

_No response_
pgvecto.rs: retriever filter not working
Labels: name: area: vector store, name: auto:bug

### System Info

OS: Windows
Python: 3.9.10
Langchain version: 0.0.352
openai version: 1.6.1

### Who can help?

@BeautyyuYanli
@baskaryan 

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
from langchain.vectorstores.pgvecto_rs import PGVecto_rs
from langchain.embeddings import AzureOpenAIEmbeddings
from dotenv import dotenv_values
import os
```

```
config = dotenv_values(".env")
# os.environ["TIKTOKEN_CACHE_DIR"] = "./cache/tiktoken/"

embeddings = AzureOpenAIEmbeddings(
    max_retries=3,
    timeout=60,
    api_key=config["api_key"],
    model="text-embedding-ada-002",
    openai_api_type=config["api_type"],
    azure_endpoint=config["api_base"]
)

URL = "postgresql+psycopg://{username}:{password}@{host}:{port}/{db_name}".format(
    port=config["db_port"],
    host=config["db_host"],
    username=config["db_user"],
    password=config["db_pass"],
    db_name=config["db_name"],
)

db = PGVecto_rs(
    embedding=embeddings,
    db_url=URL,
    dimension=1536, # text-embedding-ada-002
    collection_name="test",
)
```

```
docs = ["a text about mathematics", "a text about physics"]
meta = [{"id": "1"}, {"id": "2"}]

db.add_texts(
    texts=docs,
    metadatas=meta
)

retr = db.as_retriever(
    search_kwargs = {
        "k": 1,
        "filter": {"id": "1"}
    }
)
```

```
retr.invoke("physics")
```
>[Document(page_content='a text about physics', metadata={'id': '2'})]

### Expected behavior

The search should only be performed on documents where the `metadata` field contains `{"id": "1"}`.

In this case, adding a filter makes no difference to the retrieval.
community: correct spelling mistakes of "Suffle" and "reporoducibility"
Labels: name: area: doc loader, name: auto:nit, name: size:XS

  - **Description:** Correct spelling mistakes of "Suffle" and "reporoducibility" in `DirectoryLoader` class
  - **Issue:** N/A
  - **Dependencies:** N/A
  - **Twitter handle:** N/A
core, community: propagate context between threads
Labels: name: auto:improvement, name: area: models, name: size:L

While using `chain.batch`, the default implementation uses a `ThreadPoolExecutor` and run the chains in separate threads. An issue with this approach is that that [the token counting callback](https://python.langchain.com/docs/modules/callbacks/token_counting) fails to work as a consequence of the context not being propagated between threads. This PR adds context propagation to the new threads and adds some thread synchronization in the OpenAI callback. With this change, the token counting callback works as intended.

Having the context propagation change would be highly beneficial for those implementing custom callbacks for similar functionalities as well.
community: Async Ollama + ChatOllama
Labels: name: auto:improvement, name: area: models, name: size:L

**Description:**
Adding async methods to booth OllamaLLM and ChatOllama to enable async streaming and async .on_llm_new_token callbacks.

**Issue:**
ChatOllama is not working in combination with an AsyncCallbackManager because the .on_llm_new_token method is not awaited.
Issue: Custom Mistral based LLM from API for RetrievalQA chain
Labels: name: auto:bug, name: area: models

### Issue you'd like to raise.

I would like to build RAG based on Mistral 7B model
The model is already hosted, and I provide llm_url in the custom LLM setup

I am able to make a request and get a response from the URL using the `llm._call` method, however something is wrong with the callbacks in `RetrievalQA.from_chain_type` method

It gives me below error
`'Mistral7B_LLM' object has no attribute 'callbacks'`

Am I missing anything in the below code

```
from pydantic import Extra
import requests
from typing import Any, List,Dict, Callable, Type, Mapping, Optional

from langchain.callbacks.manager import CallbackManagerForLLMRun
from langchain.llms.base import LLM, BaseLLM


class Mistral7B_LLM(LLM):

    def __init__(self):
        self.__post_init__()

    def __post_init__(self) -> None:

        def _import_mistral7B_llm() -> Any:
            from svcs.vector.src.controllers.llm.mistral7B_serving import Mistral7B_LLM

            return Mistral7B_LLM
        
        def __getattr__() -> Any:
            return Mistral7B_LLM()
        
        def get_type_to_cls_dict() -> Dict[str, Callable[[], Type[BaseLLM]]]:
            return {
                "Mistral7B_LLM": _import_mistral7B_llm,
            }
        
        __all__ = ["Mistral7B_LLM"]

    class Config:
        extra = Extra.forbid

    @property
    def _llm_type(self) -> str:
        return "Mistral7B_LLM"

    def _call(
        self,
        prompt: str,
        stop: Optional[List[str]] = None,
        run_manager: Optional[CallbackManagerForLLMRun] = None,
        max_new_tokens: Optional[int] = 156,
        temperature: Optional[float] = 0.7,
        **kwargs: Any,
    ) -> str:
        if stop is not None:
            raise ValueError("stop kwargs are not permitted.")

        payload = {
            "inputs": [prompt],
            "max_new_tokens": max_new_tokens,
            "temperature": temperature,
        }

        headers = {"Content-Type": "application/json"}


        llm_url = 'my url'

        response = requests.post(llm_url, json=payload, headers=headers, verify=False)
        response.raise_for_status()

        # print("API Response:", response.json())
        answer = response.json()["outputs"].split("
")[-1]
        return answer

    @property
    def _identifying_params(self) -> Mapping[str, Any]:
        """Get the identifying parameters."""
        return {"llmUrl": self.llm_url}
```

### Suggestion:

_No response_
Oxford comma, consistent with format elsewhere
Labels: name: area: doc loader, name: auto:documentation, name: size:XS

This document uses Oxford comma (A, B, and C), in this list the comma was missing before "and".

This PR corrects that.

<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Issue: Pass additional data through AgentExecutor
Labels: name: area: agent, name: auto:question

### Issue you'd like to raise.

Given a tool that generates a dataframe, how can I pass it through the chain?
```
llm_with_tools = llm.bind(functions=[format_tool_to_openai_function(t) for t in tools])
prompt = ChatPromptTemplate.from_messages(
        [
            ("system", """
             You are a helpful assistant for marketing department.
            """),
            MessagesPlaceholder(variable_name="history"),
            ("user", """
                Provide the answer to the question with 3 sentences long.
                If the response is related to video-on-demand. Please make sure you return the content id to the answers
                Question: {input}
            """),
            MessagesPlaceholder(variable_name="agent_scratchpad"),
        ]
    )

agent = (
        {
            "input": lambda x: x["input"],
            "dataframe": <<my_dataframe>>,
            "agent_scratchpad": lambda x: format_to_openai_function_messages(
                x["intermediate_steps"]
            ),
            "history": lambda x: x['history']
        }
        | prompt
        | llm_with_tools
        | OpenAIFunctionsAgentOutputParser()

AgentExecutor(agent=agent, tools=tools, verbose=True)
```


Is it possible?

### Suggestion:

_No response_
Issue: How it can be splitted ?
Labels: name: area: doc loader, name: auto:question

is it correct using CharacterTextSplitter in Confluence
### Issue you'd like to raise.

        confluence_url = config.get("confluence_url", None)
        username = config.get("username", None)
        api_key = config.get("api_key", None)
        space_key = config.get("space_key", None)
        documents = []

        embedding = OpenAIEmbeddings()
        loader = ConfluenceLoader(
            url=confluence_url,
            username=username,
            api_key=api_key
        )
        for space_key in space_key:
            try:
                documents.extend(loader.load(space_key=space_key,include_attachments=True,limit=100))
            except:
                documents=[]
        text_splitter = CharacterTextSplitter(chunk_size=6000, chunk_overlap=0)
        texts = text_splitter.split_documents(documents)
        text_splitter = TokenTextSplitter(chunk_size=1000, chunk_overlap=10, encoding_name="cl100k_base")
        texts = text_splitter.split_documents(texts)

### Suggestion:

_No response_
Issue: i want to use langchain callbacks to pass a tool parameter to it?  what should i do?
Labels: name: auto:question, name: area: models

### Issue you'd like to raise.

i'm using openai function call agent , gpt llm offen gives bad tool parameters, i want  to achieve this:  pass certain params to all tools through through some path,  before every tool  get executed, i can check whether the llm produced params is right or directly use the certain params already get

### Suggestion:

_No response_
Issue: When using Qdrant as retriever, how to retrieve the relevant documents with the similarity score?
Labels: name: area: vector store, name: auto:question

### Issue you'd like to raise.

When using Qdrant as retriever, how to retrieve the relevant documents with the similarity score? For now, I do not see any methods that I can use to retrieve the documents and also return me the similarity score. However, if use the vector store to run similarity search, I have the option to get the documents and corresponding scores. Isn't there a way to achieve the same thing via retriever?

### Suggestion:

_No response_
DOC: Need some clarification on Neo4j DB QA chain documentation
Labels: name: area: doc loader, name: auto:documentation

### Issue with current documentation:

I've wondering that in this part of the code in order to define `cypher generation template` of langchain with neo4j graph database from Neo4j DB QA chain Documentation
```python
from langchain.prompts.prompt import PromptTemplate

CYPHER_GENERATION_TEMPLATE = """Task:Generate Cypher statement to query a graph database.
Instructions:
Use only the provided relationship types and properties in the schema.
Do not use any other relationship types or properties that are not provided.
Schema:
{schema}
Note: Do not include any explanations or apologies in your responses.
Do not respond to any questions that might ask anything else than for you to construct a Cypher statement.
Do not include any text except the generated Cypher statement.
Examples: Here are a few examples of generated Cypher statements for particular questions:
# How many people played in Top Gun?
MATCH (m:Movie {{title:"Top Gun"}})<-[:ACTED_IN]-()
RETURN count(*) AS numberOfActors

The question is:
{question}"""

CYPHER_GENERATION_PROMPT = PromptTemplate(
    input_variables=["schema", "question"], template=CYPHER_GENERATION_TEMPLATE
)

chain = GraphCypherQAChain.from_llm(
    ChatOpenAI(temperature=0),
    graph=graph,
    verbose=True,
    cypher_prompt=CYPHER_GENERATION_PROMPT,
)

```
Just want to ask that what variables that `schema` and `question` in **input_variables** parameters in `PromptTemplate` refers to ?

### Idea or request for content:

Please explain what schema and question refers to, did schema from our connected neo4j database and question is a text we pass into `chain.run("text input")`. Since i'm a little bit confused with documentation itself and need some explanation. Maybe use an example from it to explain will be much understanable