delayedkarma/langchain-issues huggingface dataset:
Issue: Filling `ConversationSummaryMemory` with existing conversation from an SQLite database 
Labels: name: area: memory, name: auto:improvement

### Issue you'd like to raise.

I'd like to make `ConversationSummaryMemory` is filled with the previous questions and answers for a specific conversation from an SQLite database so I can have my agent already aware of previous conversation with the user.

Here's my current code:
```py
import os
import sys
from langchain.chains import ConversationalRetrievalChain
from langchain.chat_models import ChatOpenAI
from langchain.embeddings import OpenAIEmbeddings
from langchain.indexes import VectorstoreIndexCreator
from langchain.indexes.vectorstore import VectorStoreIndexWrapper
from langchain.vectorstores.chroma import Chroma
from langchain.memory import ConversationSummaryMemory
from langchain.tools import Tool
from langchain.agents.types import AgentType
from langchain.agents import initialize_agent
from dotenv import load_dotenv

load_dotenv()
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

query = " ".join(sys.argv[1:]) if len(sys.argv) > 1 else None

retriever = # retriever stuff here for the `local-docs` tool

llm = ChatOpenAI(temperature=0.7, model="gpt-3.5-turbo-1106")

memory = ConversationSummaryMemory(
    llm=llm,
    memory_key="chat_history",
    return_messages=True,
)


chain = ConversationalRetrievalChain.from_llm(
    llm=llm,
    memory=memory,
    chain_type="stuff",
    retriever=index.vectorstore.as_retriever(search_kwargs={"k": 4}),
    get_chat_history=lambda h: h,
    verbose=False,
)

system_message = (
    "Be helpful to your users".
)
tools = [
    Tool(
        name="local-docs",
        func=chain,
        description="Useful when you need to answer docs-related questions",
    )
]


def ask(input: str) -> str:
    result = ""
    try:
        result = executor({"input": input})
    except Exception as e:
        response = str(e)
        if response.startswith("Could not parse LLM output: `"):
            response = response.removeprefix(
                "Could not parse LLM output: `"
            ).removesuffix("`")
            return response
        else:
            raise Exception(str(e))
    return result


chat_history = []
executor = initialize_agent(
    agent=AgentType.CHAT_CONVERSATIONAL_REACT_DESCRIPTION,
    tools=tools,
    llm=llm,
    memory=memory,
    agent_kwargs={"system_message": system_message},
    verbose=True,
    max_execution_time=30,
    max_iterations=6,
    handle_parsing_errors=True,
    early_stopping_method="generate",
    stop=["\nObservation:"],
)

result = ask(query)
print(result["output"])
```
add multimodal support
Labels: name: auto:enhancement, name: area: models

### Feature request

add support for other multimodal models like Llava, Fuyu, Bakllava... This would help with RAG, where documents have non text data.

### Motivation

I have a lot of tables and images to proccess in PDFs when doing RAG, and right now this is not ideal.

### Your contribution

no time :(
Issue: How to add randomness when using max_marginal_relevance_search with Qdrant
Labels: name: area: vector store, name: auto:question

### Issue you'd like to raise.

I am using Qdrant as my vector store, and now, every time I use 'max_marginal_relevance_search' with a fix k parameter. It will always return the same documents. How to add some randomness? So it will return something different(Still within the score_threshold) each time. Here is my sample code of using 'max_marginal_relevance_search':

related_docs = vectorstore.max_marginal_relevance_search(target_place, k=fetch_amount, score_threshold=0.5, filter=rest.Filter(must=[rest.FieldCondition(
                    key='metadata.category',
                    match=rest.MatchValue(value=category),
                ),rest.FieldCondition(
                    key='metadata.related_words',
                    match=rest.MatchAny(any=related_words),
                )]))

### Suggestion:

_No response_
Update mongodb_atlas.py to fix ImportError: cannot import name 'DriverInfo' from 'pymongo'
Labels: name: area: vector store, name: auto:bug



<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** fix ImportError: cannot import name 'DriverInfo' from 'pymongo'
  - **Issue:** As above
  - **Dependencies:** NA
  - **Tag maintainer:** @efriis @jarib 

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Adrijanik patch 1
Labels: name: area: agent, name: auto:improvement

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Issue: can i generate a nested dic output
Labels: name: auto:question

### Issue you'd like to raise.

I know I can generate a python dictionary output using StructuredOutputParser like { "a":1, "b":2, "c":3}. However, I would like to generate a nested dic like { "a":1, "b":2, "c":{"d":4, "e":5}}
How can I do it?

### Suggestion:

_No response_
Fix typo in MongoDB memory docs
Labels: name: area: memory, name: auto:documentation

  - **Description:** Fix typo in MongoDB memory docs
  - **Tag maintainer:** @eyurtsev

<!-- Thank you for contributing to LangChain!

  - **Description:** Fix typo in MongoDB memory docs
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** @baskaryan
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Issue: can i access memory buffer through chain?
Labels: name: area: memory, name: auto:question, name: area: models

### Issue you'd like to raise.

Hi, I'm using a conversationchain that contains memory. It is defined as:
llm = ChatOpenAI(temperature=0.0, model=llm_model)
memory = ConversationBufferMemory()
conversation = ConversationChain(
    llm=llm, 
    memory = memory,
    verbose=True
)
I know I can access the current memory by using "memory.buffer". However, I was wondering if there is a way to access memory only through ConversationChain instance "conversation"? 

### Suggestion:

_No response_
fix: Make YoutubeLoader support on demand language translation
Labels: name: lgtm, name: area: doc loader, name: auto:bug

**Description:**
Enhance the functionality of YoutubeLoader to enable the translation of available transcripts by refining the existing logic.

**Issue:**
Encountering a problem with YoutubeLoader (#13523) where the translation feature is not functioning as expected.

Tag maintainers/contributors who might be interested:
@eyurtsev
fix: call _on_llm_error()
Labels: name: lgtm, name: auto:bug, name: area: models, name: size:M

Description: There's a copy-paste typo where on_llm_error() calls _on_chain_error() instead of _on_llm_error().
Issue: #13580 
Dependencies: None
Tag maintainer: @hwchase17 
Twitter handle: @jwatte

"Run `make format`, `make lint` and `make test` to check this locally."
The test scripts don't work in a plain Ubuntu LTS 20.04 system.
It looks like the dev container pulling is stuck. Or maybe the internet is just ornery today.
The tracing on_llm_error() implementation calls _on_chain_error(), not _on_llm_error()
Labels: name: auto:bug, name: area: models

### System Info

Linux 20.04 LTS
Python 3.6


### Who can help?

@hwchase17 seems like this got introduced on 2023-11-16


### Information

- [X] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [X] Callbacks/Tracing
- [ ] Async

### Reproduction

1. Attempt to use a tracer to trace an LLM error
2. Note that the tracer hook for _on_chain_error is called instead


### Expected behavior

_on_llm_error hook should be called.
Confluence loader fails to retrieve specific pages when 'pages_ids' is given
Labels: name: area: doc loader, name: auto:bug

### System Info

Mac M1

### Who can help?

@eyurtsev 

Here:
https://github.com/langchain-ai/langchain/blob/78a1f4b264fbdca263a4f8873b980eaadb8912a7/libs/langchain/langchain/document_loaders/confluence.py#L284 

We start adding the "max_pages" first pages to the "docs" list that will be the output of loader.load.
So we are sure that I cannot retrieve only one specific `page_id`.

`loader.load(..., page_ids=['1234'], max_pages=N)`
will output X pages where X in [min(N, # pages in my confluence),  N + 1] 

In other words, if I want only a specific page, I will always have at least 2 pages (in case max_pages = 1)
So page_ids does not work at all because space_key is mandatory. 

adding ìf space_key and not page_ids` fix my problem but may lead to other problems (I did not check)

Dirty hack would be to collect the F last elements of the return list if pages where F is the number of found pages asked in page_ids


### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

No time to do but easy when reading code

### Expected behavior

I can retrieve only the page_ids specified
ConversationChain failure after changing template text
Labels: name: area: memory, name: auto:bug, name: area: models

I am having a wonderful time with my code, but after changing my template  it now fails  before I even get to give my input. Baffling!

all the required imports are not shown here nor is all the prompt text (containing no special characters)
    template = '''Your task is to extract the relationships between terms in the input text,
                        Format your output as a json list. '''
    prompt = ChatPromptTemplate.from_messages([
                        SystemMessagePromptTemplate.from_template(template),
                        HumanMessagePromptTemplate.from_template("{input}"),
                        MessagesPlaceholder(variable_name="history "),
                        ])
    llm = ChatOpenAI(temperature=0.8, model_name='gpt-4-1106-preview') 
    memory = ConversationBufferMemory(return_messages=True)
    conversation = ConversationChain(memory=memory, prompt=prompt, llm=llm)](url)

  Traceback  .........
 conversation = ConversationChain(memory=memory, prompt=prompt, llm=llm)
  File "pydantic\main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for ConversationChain
__root__    Got unexpected prompt input variables. The prompt expects ['input', 'history '], but got ['history'] as inputs from memory, and input as the normal input key. (type=value_error)
openapi/planner.py: Deal with json in markdown output cases
Labels: name: lgtm, name: auto:improvement, name: area: models, name: size:S

  - **Description:** In openapi/planner deal with json in markdown output cases
  - **Issue:**  In some cases LLMs could return json in markdown which can't be loaded.  
  - **Dependencies:**
  - **Tag maintainer:** @eyurtsev
  - **Twitter handle:**
Implementing Hybrid Search (RRF) in LangChain Using OpenSearch Vector Store
Labels: name: area: vector store, name: auto:question, name: auto:enhancement

### Feature request

The feature request I am proposing involves the implementation of hybrid search, specifically using the Reciprocal Rank Fusion (RRF) method, in LangChain through the integration of OpenSearch's vector store.
This would enable the combination of keyword and similarity search. Currently, LangChain doesn't appear to support this functionality, even though OpenSearch has had this capability since its 2.10 release. The goal is to allow LangChain to call search pipelines using OpenSearch's vector implementation, enabling OpenSearch to handle the complexities of hybrid search.

**Relevant Links**:
https://opensearch.org/docs/latest/query-dsl/compound/hybrid



### Motivation

The motivation behind this request stems from the current limitation in LangChain regarding hybrid search capabilities. As someone working on a search project currently, I find it frustrating that despite OpenSearch supporting hybrid search since version 2.10, LangChain has not yet integrated this feature.

### Your contribution

I would gladly help as long as I get guidance..
Update geodataframe.py
Labels: name: lgtm, name: area: doc loader, name: auto:improvement

here it is validating shapely.geometry.point.Point:       if not isinstance(data_frame[page_content_column].iloc[0], gpd.GeoSeries): raise ValueError(
f"Expected data_frame[{page_content_column}] to be a GeoSeries"         you need it to validate the geoSeries and not the shapely.geometry.point.Point

if not isinstance(data_frame[page_content_column], gpd.GeoSeries):
            raise ValueError(
                f"Expected data_frame[{page_content_column}] to be a GeoSeries"

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Upgrading to OpenAI Python 1.0+ = ImportError: Could not import openai python package.
Labels: name: auto:bug, name: area: models

### System Info

Using langchain 0.0.337 python, FastAPI.

When I use openai up through 0.28.1 it works fine. Upgrading to 1.0.0 or above results in the following error (when I try to use ChatOpenAI from langchain.chat_models):

"ImportError: Could not import openai python package. Please install it with `pip install openai`."

Trying to follow this notebook to integrate vision preview model:
https://github.com/langchain-ai/langchain/blob/master/cookbook/openai_v1_cookbook.ipynb

Any thoughts on what I might try? Thanks!

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Steps to reproduce:

1. install openai (1.0.0), langchain (0.0.337) & langchain-experimental (0.0.39)
2. in FastAPI route, import ChatOpenAI from langchain.chat_models
3. Use ChatOpenAI as usual (working fine w/ openai <= 0.28.1

`    llm = ChatOpenAI(
      temperature=temperature,
      streaming=True,
      verbose=True,
      model_name=nameCode,
      max_tokens=tokens,
      callbacks=[callback],
      openai_api_key=relevantAiKey,
    )`

### Expected behavior

I would expect to not get a "failed import" error when the package is clearly installed.
from_llm_and_api_docs fails on gpt-4-1106-preview
Labels: name: auto:bug, name: auto:improvement, name: area: models

### System Info

```
poetry show langchain
 name         : langchain                                             
 version      : 0.0.259                                               
 description  : Building applications with LLMs through composability 

dependencies
 - aiohttp >=3.8.3,<4.0.0
 - async-timeout >=4.0.0,<5.0.0
 - dataclasses-json >=0.5.7,<0.6.0
 - langsmith >=0.0.11,<0.1.0
 - numexpr >=2.8.4,<3.0.0
 - numpy >=1,<2
 - openapi-schema-pydantic >=1.2,<2.0
 - pydantic >=1,<2
 - PyYAML >=5.3
 - requests >=2,<3
 - SQLAlchemy >=1.4,<3
 - tenacity >=8.1.0,<9.0.0
 ```

Python: v3.10.12

### Who can help?

@hwchase17 @agola11 
With the current GPT-4 model, the invocation of `from_llm_and_api_docs` works as expected. However, when switching the model to the upcoming `gpt-4-1106-preview`, the function fails as the LLM, instead of returning the URL for the API call, returns a verbose response: 

```
LLM response on_text:  To generate the API URL for the user's question "basketball tip of the day", we need to include the `sport` parameter with the value "Basketball" since the user is asking about basketball. We also need to include the `event_start` parameter with today's date to get the tip of the day. Since the user is asking for a singular "tip", we should set the `limit` parameter to 1. The `order` parameter should be set to "popularity" if not specified, as per the documentation.

Given that today is 2023-11-18, the API URL would be:

http://<domain_name_hidden>/search/ai?date=2023-11-18&limit=1&order=popularity
```

The prompt should be refined or extra logic should be added to retrieve just the URL with the upcoming GPT-4 model.

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Try to get the upcoming GPT-4 model to return just the URL of the API call. 

```
ERROR:root:No connection adapters were found for 'To generate the API URL for the user\'s question "<question edited>", we need to include the `sport` parameter with the value "Basketball" since the user is asking about basketball. We also need to include the `event_start` parameter with today\'s date to get the tip of the day. The `order` parameter should be set to "popularity" if not specified, as per the documentation.\n\nGiven that today is 2023-11-18, the API URL would be:\n\nhttp://<domain_removed>/search/ai?date=2023-11-18&limit=1&order=popularity'
```

### Expected behavior

The LLM to return just the URL and for Langchain to not error out.
pydantic.error_wrappers.ValidationError: 1 validation error for RetrievalQA separators   extra fields not permitted (type=value_error.extra)
Labels: name: auto:bug, name: area: models

### System Info

Facing this error while executing the langchain code.
```
pydantic.error_wrappers.ValidationError: 1 validation error for RetrievalQA
separators
  extra fields not permitted (type=value_error.extra)
```

Code for retrivalQA
def retrieval_qa_chain(llm, prompt, retriever):
    qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever = retriever,
    verbose=True,
    callbacks=[handler],
    chain_type_kwargs={"prompt": prompt},
    return_source_documents=True
    )
    return qa_chain

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
def retrieval_qa_chain(llm, prompt, retriever):
    qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever = retriever,
    verbose=True,
    callbacks=[handler],
    chain_type_kwargs={"prompt": prompt},
    return_source_documents=True
    )
    return qa_chain
```
```
    
def retrieval_qa_chain(llm, prompt, retriever):
    qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever = retriever,
    verbose=True,
    callbacks=[handler],
    chain_type_kwargs={"prompt": prompt},
    return_source_documents=True
    )
    return qa_chain
```

### Expected behavior

Need a fix for the above error
BedrockRerank using newly available Cohere embedding model
Labels: name: area: embeddings, name: auto:question, name: auto:enhancement, name: integration: aws

### Feature request

Hi, the new Cohere embedding models are now available on Amazon Bedrock. How can we use them for their reranking capability (instead of just embedding via BedrockEmbedding class)

### Motivation

These models perform well for reranking