delayedkarma/langchain-issues huggingface dataset:
feat: add Google BigQueryVectorSearch in vectorstore
Labels: name: lgtm, name: area: vector store, name: auto:enhancement, name: size:XXL

BigQuery vector search lets you use GoogleSQL to do semantic search, using vector indexes for fast but approximate results, or using brute force for exact results.

This PR integrates LangChain vectorstore with BigQuery Vector Search.

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
How can I grab only the AI answer of my langchain agent?
Labels: name: area: agent, name: auto:question

### Issue you'd like to raise.

My code is :
```
llm = ChatOpenAI(temperature=0, verbose=True, model="gpt-3.5-turbo-16k")
toolkit = SQLDatabaseToolkit(db=db, llm=llm)
agent_type = AgentType.ZERO_SHOT_REACT_DESCRIPTION

agent_executor_1 = initialize_agent()

agent_executor = create_sql_agent(llm=llm, toolkit=toolkit, verbose=True, agent_type=agent_type)
agent_executor.run(input="How many orders in 2023-12-15")
```
I got the result with the following format:
```
> Entering new AgentExecutor chain...

Thought: ...
Action: ...
Action Input: ...
Observation: ...
Thought: ...
AI: ...
```
I want to save AI result to a variable. I try print(agent_executor.run) but it didn't work. Is there any way that suit my need.

### Suggestion:

_No response_
Fix link typo to `/docs/integrations/text_embedding/nvidia_ai_endpoints`
Labels: name: lgtm, name: area: embeddings, name: auto:documentation, name: size:XS

This page doesn't exist:
- https://python.langchain.com/docs/integrations/text_embeddings/nvidia_ai_endpoints

but this one does:
- https://python.langchain.com/docs/integrations/text_embedding/nvidia_ai_endpoints
added history and support for system_message as param
Labels: name: lgtm, name: auto:improvement, name: area: models, name: size:L

  - **Description:** added support for chat_history for Google GenerativeAI (to actually use the `chat` API) plus since Gemini currently doesn't have a support for SystemMessage, added support for it only if a user provides additional `convert_system_message_to_human` flag during model initialization (in this case, SystemMessage would be prepanded to the first HumanMessage)
  - **Issue:** #14710 
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** lkuligin
Small update in the documentation to ensure consistency in declaring LANGCHAIN_API_KEY...
Labels: name: lgtm, name: area: doc loader, name: auto:documentation, name: size:XS

... variable, accompanied by a quote

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
PostgresChatHistory or any other database for storing history does not work with ConversationSummaryBufferMemory(or other memory taht produces 'system' massage type after summarisation
Labels: name: area: memory, name: auto:bug

### System Info

mac os latest, latest LangChain

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [x] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [x] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

ConversationBufferWindowMemory(
            chat_memory=PostgresChatMessageHistory(
        session_id=session_id, connection_string=os.getenv("DB_URL")
    ),
            return_messages=True,
            memory_key=memory_key,
        )
  try to run a chat with this

### Expected behavior

I think it should save the massages as they are to SQL, but use the properties of the ConversationSummaryBufferMemory for the chat itself, but it reality it just uses SQL and history is jus tall the massages
support for Sybase SQL anywhere added.
Labels: name: lgtm, name: auto:improvement, name: size:XS


Please find changes details as below 

  - **Description:** support for Sybase SQL anywhere added in sql_database.py file at path langchain\libs\community\langchain_community\utilities 
  - **Issue:**  It will resolve default schema setting for Sybase SQL anywhere 
  - **Dependencies:** No,
  - **Tag maintainer:** @baskaryan, @eyurtsev, @hwchase17,
  - **Twitter handle:** NA
Gemini with ConversationalRetrievalChain
Labels: name: auto:question, name: area: models

### Issue you'd like to raise.

Is there a similar method for using `gemini-pro` with `ConversationalRetrievalChain.from_llm`, as there is for utilizing models from VertexAI with `ChatVertexAI` or `VertexAI`, where you specify the `model_name`?

### Suggestion:

_No response_
Restore self message sent before OSX 12 Monterey
Labels: name: area: doc loader, name: auto:improvement, name: size:M

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Fix failing serpapi response processing for Google Maps API
Labels: name: lgtm, name: auto:bug, name: size:XS

**Description:** Fix for processing for serpapi response for Google Maps API 
**Issue:** Due to the fact corresponding [api](https://serpapi.com/google-maps-api) returns 'local_results' as list, and old version requested `res["local_results"].keys()` of the list. As the result we got exception: ```AttributeError: 'list' object has no attribute 'keys'```.

Way to reproduce wrong behaviour:
```
    params = {
        "engine": "google_maps",
        "type": "search",
        "google_domain": "google.de",
        "ll": "@51.1917,10.525,14z",
        "hl": "de",
        "gl": "de",
    }
    search = SerpAPIWrapper(params=params)
    results = search.run("cafe")
```
When using the OpenAIFunctionsAgentOutputParser() to parse agent output meet an error.
Labels: name: area: agent, name: auto:bug

### Issue you'd like to raise.

The following issue occurred when parsing the output of the proxy.
> Expected an AI message got <class \'langchain_core.messages.chat.ChatMessage\'>

Code:
```python
self.agent_output_parser = OpenAIFunctionsAgentOutputParser()
self.agent = {
                 "paths": lambda x: x["paths"],
                 "in_name": lambda x: x["in_name"],
                 "out_name": lambda x: x["out_name"],
                 "agent_scratchpad": lambda x: format_to_openai_functions(x['intermediate_steps'])
             } | self.gen_chatprompt | llm_with_tools | self.agent_output_parser
```
There were no errors in this code before the update.
current version of langchain is 0.0.350.

### Suggestion:

_No response_
fix agenerate return value
Labels: name: lgtm, name: auto:bug, name: area: models, name: size:S

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Fixed:
  -  `_agenerate` return value in the YandexGPT Chat Model
  - duplicate line in the documentation
feat: Qdrant sparse vector retriever
Labels: name: lgtm, name: area: vector store, name: auto:enhancement, name: size:XL

## Description

This PR intends to add support for Qdrant's new [sparse vector retrieval](https://qdrant.tech/articles/sparse-vectors/) by introducing a new retriever class, `QdrantSparseVectorRetriever`. 

Necessary usage docs and integration tests have been added for the retriever.
Missing "Prompt" in Together ai
Labels: name: auto:bug, name: area: models

### System Info

In this file: https://github.com/langchain-ai/langchain/blob/master/libs/community/langchain_community/llms/together.py

There is no variable Prompt, however it is available in the documentation: https://docs.together.ai/reference/inference

```python
if config.model == "together":
            return Together(
                    model="togethercomputer/StripedHyena-Nous-7B",
                    temperature=0.7,
                    max_tokens=128,
                    top_k=1,
                    together_api_key=config.together_api_key,
                    prompt="The capital of France is"
                )
```

error I get is:
```bash
pydantic.v1.error_wrappers.ValidationError: 1 validation error for Together
prompt
  extra fields not permitted (type=value_error.extra)
```

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Try to run together Ai model and pass Prompt on model it will fail. My goal is to set a system Prompt.

### Expected behavior

Accept Prompt as the system Prompt.
Add Ollama multimodal llm support for image with prompt
Labels: name: auto:enhancement, name: area: models, name: size:M

  - **Description:** Adds a class OllamaMultiModal to the Ollama llms file: lanchain/llms/ollama.py. This class supports sending an image to the ollama endpoint for models that support the image with a prompt message. 
  - **Twitter handle:** Daniel_OHeron

Passes make format, make lint, make test.

If this code change is useful. Will add unit test, notebook and documentation. As well as build out additional features. Please let me know.

Example usage: 
`
from OllamaMultiModalMain import OllamaMultiModal
import base64

def encode_image_to_base64(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode('utf-8')

llm = OllamaMultiModal(model="bakllava")

while True:
    user_input = input("User (text): ")

    image_input = input("User (image path, press enter to skip): ")
    image_data = encode_image_to_base64(image_input) if image_input else None

    if image_data:
        response = llm.invoke(
            input=user_input, images=[image_data])
    else:
        response = llm.invoke(input=user_input)
        # response = conversational_chain.predict(input=user_input)

    print("Chat:", response)
`
how to add LCEL memory | OpenAIFunctionsAgentOutputParser？
Labels: name: area: agent, name: auto:question

### Issue you'd like to raise.

            # 使用LCEL创建代理
            prompt = ChatPromptTemplate.from_messages(
                [
                    ("system", "You are a helpful assistant"),
                    ("user", "{input}"),
                    MessagesPlaceholder(variable_name="agent_scratchpad"),
                ]
            )
            llm_with_tools = self.llm.bind(functions=[format_tool_to_openai_function(t) for t in self.tools])
            agent = (
                    {
                        "input": lambda x: x["input"],
                        "agent_scratchpad": lambda x: format_to_openai_function_messages(
                            x["intermediate_steps"]
                        ),
                    }
                    | prompt
                    | llm_with_tools
                    | OpenAIFunctionsAgentOutputParser()
            )
            # 创建AgentExecutor并调用
            agent_executor = AgentExecutor(agent=agent,
                                           tools=self.tools,
                                           verbose=True,
                                           return_intermediate_steps=True,
                                           handle_parsing_errors=True,
                                           )
            try:
                response = agent_executor.invoke(
                    {
                        "input": message
                    }
                )

### Suggestion:

_No response_
core(minor): Implement stream and astream for RunnableBranch
Labels: name: lgtm, name: auto:improvement, name: size:L

**Description:** Implement stream and astream methods for RunnableBranch
  - **Issue:** https://github.com/langchain-ai/langchain/issues/13723
  - **Dependencies:** No new dependencies
  - **Twitter handle:** https://twitter.com/qtangs
Enhance iMessage chat loader with timestamp parsing and message ownership
Labels: name: lgtm, name: area: doc loader, name: auto:improvement, name: size:M

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
ModuleNotFound error in using UnstructuredRSTLoader
Labels: name: area: doc loader, name: auto:bug

### System Info

#### Virtualenv
Python:         3.11.6
Implementation: CPython
Path:           /Users/max/Library/Caches/pypoetry/virtualenvs/qa-oj4cEcx_-py3.11
Executable:     /Users/max/Library/Caches/pypoetry/virtualenvs/qa-oj4cEcx_-py3.11/bin/python
Valid:          True

#### System
Platform:   darwin
OS:         posix
Python:     3.11.6
Path:       /usr/local/opt/python@3.11/Frameworks/Python.framework/Versions/3.11
Executable: /usr/local/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/bin/python3.11


### Who can help?

@eyurtsev @baskaryan (https://github.com/langchain-ai/langchain/pull/14463)

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
from langchain_community.document_loaders import UnstructuredRSTLoader
loader = UnstructuredRSTLoader("example.rst", mode="elements", strategy="fast")
docs = loader.load()
```
In this case, "example.rst" is the downloaded rst from the lanchain source itself.

### Expected behavior

I would expect the document loader to result in a list of documents. Instead there is an error in referencing a module:

```
  File ".../virtualenvs/qa-oj4cEcx_-py3.11/lib/python3.11/site-packages/langchain_community/document_loaders/unstructured.py", line 14, in satisfies_min_unstructured_version
    from unstructured.__version__ import __version__ as __unstructured_version__
ModuleNotFoundError: No module named 'unstructured'
```
why the memory is not work? who can help me!
Labels: name: area: memory, name: auto:question

### Issue you'd like to raise.
`
```
class SimpleChat:
    def __init__(self) -> None:
        self.llm = ChatOpenAI(
            temperature=0,
            model="gpt-4-0613",
            openai_api_key="sk-",
            openai_api_base = "https://---",
        )
        self.memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
    
    def get_tools(self):
        return [
            Tool(
                name="Search",
                func = Tooluse().get_google().run,
                description="useful for when you want to search for something on the internet",
            )
        ]
    
    def get_agent(self):
        conversational_agent = initialize_agent(
            agent=AgentType.CHAT_CONVERSATIONAL_REACT_DESCRIPTION, 
            tools=self.get_tools(), 
            llm=self.llm,
            verbose=False,
            memory=self.memory,
        )
        sys_prompt = """You are a chatbot for a Serverless company AntStack and strictly answer the question based on the context below, and if the question can't be answered based on the context, say \"I'm sorry I cannot answer the question, contact connect@antstack.com\""""
        prompt = conversational_agent.agent.create_prompt(
            system_message=sys_prompt,
            tools=self.get_tools(),
        )
        conversational_agent.agent.llm_chain.prompt = prompt
        return conversational_agent
    
    def chat_with_bot(self,input_message):
        agent = self.get_agent()
        print("agent",agent)
        response = agent.run(input_message)
        return response
````

this is my code,and why the memory is not work?? anyone can help me, thanks!

### Suggestion:

the doc of langchain is so sucked!
Fixed typos - LLaMA2_sql_chat.ipynb
Labels: name: lgtm, name: auto:nit, name: area: models, name: size:XS

"language" (right) vs "langugae" (wrong)

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
--> Fixed spelling errors

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
How do i make my below code async of SQLAlchemy?
Labels: name: auto:question

Hi @dosu-bot,
Check out my below code,
```
class CustomMessage(Base):
    tablename = "custom_message_store"

    id = Column(Integer, primary_key=True, autoincrement=True)
    session_id = Column(Text)
    type = Column(Text)
    content = Column(Text)
    created_at = Column(DateTime)
    author_email = Column(Text)


class CustomMessageConverter(BaseMessageConverter):
    def init(self, author_email: str):
        self.author_email = author_email

    def from_sql_model(self, sql_message: Any) -> BaseMessage:
        if sql_message.type == "human":
            return HumanMessage(
                content=sql_message.content,
            )
        elif sql_message.type == "ai":
            return AIMessage(
                content=sql_message.content,
            )
        else:
            raise ValueError(f"Unknown message type: {sql_message.type}")

    def to_sql_model(self, message: BaseMessage, session_id: str) -> Any:
        now = datetime.now()
        return CustomMessage(
            session_id=session_id,
            type=message.type,
            content=message.content,
            created_at=now,
            author_email=self.author_email,
        )

    def get_sql_model_class(self) -> Any:
        return CustomMessage


chat_message_history = SQLChatMessageHistory(
    session_id="user1@example.com",
    connection_string="mssql+pyodbc://User\SQLEXPRESS/db_name?driver=ODBC+Driver+17+for+SQL+Server",
    custom_message_converter=CustomMessageConverter(author_email="user1@example.com"),
)
```


how do i make the code async?