delayedkarma/langchain-issues huggingface dataset:
OpenAIAssistantRunnable input validation error
Labels: name: area: agent, name: auto:bug, name: area: models

### System Info

Python 3.11 running locally in PyCharm.


### Who can help?

@hwchase17 @agola11 

### Information

- [x] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
def research_tool(input, thread_id=None):

    # Initialize Tavily Search
    search = TavilySearchAPIWrapper()
    tavily_tool = TavilySearchResults(api_wrapper=search)
    salesforce_history_tool = create_pinecone_tool()

    # Initialize PlayWright Web Browser
    sync_browser = create_sync_playwright_browser()
    toolkit = PlayWrightBrowserToolkit.from_browser(sync_browser=sync_browser)

    # Initialize the Toolkit
    tools = toolkit.get_tools()
    tools.append(tavily_tool) # Add Tavily Search to the Toolkit
    tools.append(salesforce_history_tool) # Add Salesforce History to the Toolkit
    tools.extend(sf_tools) # Add Salesforce Tools to the Toolkit
    
    agent = OpenAIAssistantRunnable.create_assistant(
        name="Research Assistant",
        instructions="You are a personal research assistant on company information",
        tools=tools,
        model="gpt-4-1106-preview",
        as_agent=True,
    )
    
    agent_executor = AgentExecutor(agent=agent, tools=tools)
    
    if thread_id: 
        result = agent_executor.invoke({"content": input, "thread_id": thread_id})
    else:
        result = agent_executor.invoke({"content": input})

    output = result['output']
    thread_id = result['thread_id']
    
    return output, thread_id
```

### Expected behavior

I am looking to have my agent run using the Assistants API. Instead, I receive the following error:

```
[chain/start] [1:chain:AgentExecutor > 4:chain:OpenAIAssistantRunnable] Entering Chain run with input:
[inputs]
[chain/error] [1:chain:AgentExecutor > 4:chain:OpenAIAssistantRunnable] [315ms] Chain run errored with error:
"BadRequestError(\"Error code: 400 - {'error': {'message': '1 validation error for Request\\\\nbody -> tool_outputs -> 0 -> output\\\\n  str type expected (type=type_error.str)', 'type': 'invalid_request_error', 'param': None, 'code': None}}\")"
```
[core/minor] Runnables: Implement a context api
Labels: name: lgtm, name: auto:enhancement, name: size:XL

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
DOC: Expand documentation on how to authenticate and connect to Amazon OpenSearch Serverless
Labels: name: area: doc loader, name: area: vector store, name: auto:documentation, name: integration: aws

### Issue with current documentation:

The current [section on Amazon OpenSearch Serverless][1] (AOSS) vector store uses the `AWS4Auth` class to authenticate to AOSS, yet the official [OpenSearch documentation][2] suggests using the `AWS4SignerAuth` class instead. Using the latter does currently **not** work with LangChain due to how AOSS is detected when the class is instantiated (c.f. #14129 ) 

Further, the notebook lacks information on where to import the `AWS4Auth` class from and how to configure it with different AWS credentials (static access key/secret key, temporary credentials, etc.). It also lacks references on how to configure access policies (IAM, AOSS data access policies, etc.)

[1]: https://github.com/langchain-ai/langchain/blob/master/docs/docs/integrations/vectorstores/opensearch.ipynb
[2]: https://opensearch.org/docs/latest/clients/python-low-level/#connecting-to-amazon-opensearch-serverless

### Idea or request for content:

Add installation instructions for the [requests_aws4auth][1] package and links to its [Github repo][2] in order to showcase configuration with different AWS credentials. Additionally reference [AWS documentation for AOSS][3] in order to get started [setting up permissions][4]

[1]: https://pypi.org/project/requests-aws4auth/
[2]: https://github.com/tedder/requests-aws4auth#basic-usage
[3]: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-getting-started.html
[4]: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-data-access.html
Missing metadata in some callback handler methods
Labels: name: auto:improvement, name: area: models

### Issue you'd like to raise.

I only see `metadata` as parameter for few callback handler methods, but I would like to have access to metadata in other methods such as `on_chain_error()` and `on_llm_error()`.

Currently, I can only see the `tags` in these methods.

I have error handling in my callback handler using `on_chain_error()` and I'd like to add information in my metadata dictionary to my exceptions (such as LLM name, model, things related to my chain, etc...).

I can put a list of tags, but I'd much prefer to use a dictionary to get certain keys and have my exceptions instantiated properly.

### Suggestion:

Please make both tags and metadata available for all callback handler methods.
Add Hugging Face chat wrapper
Labels: name: auto:enhancement, name: area: models, name: size:XL

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->

## Issue
There previously has been no easy way to make use of models hosted on Hugging Face (via Inference API or Inference Endpoints) in combination with LangChains ChatModel abstraction. 

## Description
This PR introduces a new chat_model integration that creates a wrapper around the `BaseChatModel` class that interfaces between LangChain's and the hosted LLM by leveraging [Hugging Face's Chat Templates](https://huggingface.co/docs/transformers/chat_templating).


## To do
- [x] Add wrapper class around`BaseChatModel` to interface with HF LLM integrations
- [x] Create a notebook  `docs/integrations/chat` that demonstrates its use
- [x] Add unit test

## Tag maintainer
@hwchase17

## Twitter handle
@andrewrreed
Add Optional Parameter for Unique IDs in FAISS.add_documents Function
Labels: name: area: vector store, name: auto:enhancement

### Feature request

I notice the other functions, such as add_texts and add_embeddings allow you to pass a unique list of IDs that can get paired with your embedding. There is no such parameter for the add_documents function. This means when you delete a document and add an updated version of it using add_documents, its unique ID won't be added to the vector store.

### Motivation

Inability to add unique ID to document after calling the add_documents function from FAISS

### Your contribution

Not sure as far as I know. If I can, let me know what you need from me.
Issue: support for Azure Open AI latest GPT models like, GPT 4 turpo in the get_openai_callback()
Labels: name: auto:enhancement, name: area: models

### Issue you'd like to raise.

Would be great to add the support for Azure GPT latest models in the get_openai_callback() --> https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/callbacks/openai_info.py

https://github.com/langchain-ai/langchain/issues/12994

### Suggestion:

Please add the Azure GPT models (latest) https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/callbacks/openai_info.py
Optimal Integration of the ConversationalRetrievalChain (Open source llama-2) into gradio.
Labels: name: area: memory, name: auto:question, name: auto:improvement, name: area: models

### Issue you'd like to raise.

I want to integrate ConversationalRetrievalChain with history into Gradio app.

For now I have the following approach:

```
# Memory buffer
memory = ConversationBufferWindowMemory(k=2,memory_key="chat_history", return_messages=True)

# LLM chain
chain = ConversationalRetrievalChain.from_llm(llm=llm, chain_type='stuff',
                                              retriever=vector_store.as_retriever(
                                                  search_kwargs={"k": 2}),
                                                  memory=memory)

with gr.Blocks() as demo:
    gr.Markdown("# SageMaker Docs Chat 🤗")
    gr.Markdown("### Ask me question about Amazon SageMaker!")

    chatbot = gr.Chatbot(label="Chat history")
    message = gr.Textbox(label="Ask me a question!")
    clear = gr.Button("Clear")

    def user(user_message, chat_history):
        return gr.update(value="", interactive=False), chat_history + [[user_message, None]]

    def bot(chat_history):
        user_message = chat_history[-1][0]
        llm_response = qa({"question": user_message})
        bot_message = llm_response["answer"]
        chat_history[-1][1] = ""
        for character in bot_message:
            chat_history[-1][1] += character
            time.sleep(0.005)
            yield chat_history

    response = message.submit(user, [message, chatbot], [message, chatbot], queue=False).then(
        bot, chatbot, chatbot
    )
    response.then(lambda: gr.update(interactive=True), None, [message], queue=False)

demo.queue()
demo.launch()
```

which works fine for the simple question answer without history. I tried to implement something similar to this guide (https://github.com/facebookresearch/llama-recipes/blob/main/demo_apps/Llama2_Gradio.ipynb), but failed to do so. Do you have any solution for this use-case or any specific guide?

### Suggestion:

_No response_
feat: Add a feature to serialize and deserialize memory types to and from JSON
Labels: name: area: memory, name: auto:enhancement, name: size:XXL

Description: Add a feature to serialize and deserialize the memory types into JSON format,
Issue: #11275,
Dependencies: No new dependencies,
Tag maintainer: @baskaryan, @eyurtsev, @hwchase17
Co-Authors: @D3nam, @avaove, @malharpandya
Finished chain without an answer but full context have results
Labels: name: area: agent, name: auto:bug, name: area: models

### System Info

langchain==0.0.330, python 3.9

### Who can help?

@hwchase17 @agola11 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I created an agent as follows:

  ```
def batch_embedding(node_label:str, text_node_properties:list):
      vector_index = Neo4jVector.from_existing_graph(
          OpenAIEmbeddings(),
          url=URL_DB_GRAPH,
          username=USERNAME,
          password=PASSWORD,
          index_name= node_label,
          node_label= node_label,
          text_node_properties=text_node_properties,
          embedding_node_property='embedding',
      )
      return vector_index
```


    model_name= "gpt-4"

    llm= ChatOpenAI(temperature=0, model_name=model_name)

    vector_index = batch_embedding("Alarms", ["solution", "description", "type"])

    vector_qa = RetrievalQA.from_chain_type(
        llm=llm, chain_type="stuff", retriever=vector_index.as_retriever())

```
cypher_chain = GraphCypherQAChain.from_llm(
        cypher_llm = llm,
        qa_llm = ChatOpenAI(temperature=0), graph=graph, verbose=True, cypher_prompt=CYPHER_GENERATION_PROMPT
    )

```
    tools = [
        Tool(
            name="Alarms",
            func=vector_qa.run,
            description=prompt_per_alarms,
        ),
        Tool(
            name="Graph",
            func=cypher_chain.run,
            description=prompt_per_Graph,
        ),
    ]

    mrkl = initialize_agent(
        tools, ChatOpenAI(temperature=0, model_name=model_name), agent=AgentType.OPENAI_FUNCTIONS, verbose=True, memory=memory
    )


    message = request.form['message']
    response = mrkl.run(message)


When  I receive an answer from cypher_chain.run I can see that I've a full context with an output but the finished chain says "I'm sorry, but I don't have the information.." (see the image attached)
![example](https://github.com/langchain-ai/langchain/assets/119319987/5d2e7308-e944-4c32-b2ad-9dd8643ef905)
. 
I noticed that this issue come back when I have a full context with an array of data.  

### Expected behavior

Finished chain contains the full context and write the answer.
Fix .env file path in integration_test README.md
Labels: name: lgtm, name: auto:documentation, name: size:XS

<!-- Thank you for contributing to LangChain!



Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->

### Description
Hello, 

The [integration_test README](https://github.com/langchain-ai/langchain/tree/master/libs/langchain/tests) was indicating incorrect paths for the `.env.example` and `.env` files. 

`tests/.env.example` ->`tests/integration_tests/.env.example`

While it’s a minor error, it could **potentially lead to confusion** for the document’s readers, so I’ve made the necessary corrections.

Thank you! ☺️

### Related Issue
- https://github.com/langchain-ai/langchain/pull/2806
Update text_splitter.py
Labels: name: auto:bug, name: size:M

Description: Fixes a minor bug to make HTMLHeaderTextSplitter work.
Issue: https://github.com/langchain-ai/langchain/issues/14024
Dependencies: none,

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** Fixes a minor bug to make HTMLHeaderTextSplitter work., 
  - **Issue:** [the issue # it fixes (if applicable),](https://github.com/langchain-ai/langchain/issues/14024)
  - **Dependencies:** None
  - **Tag maintainer:** @hwchase17 
  - **Twitter handle:** @GovindaTotla

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
HTMLHeaderTextSplitter throws TypeError: cannot parse from 'PosixPath'
Labels: name: area: vector store, name: auto:bug

### System Info

Python 3.10
Langchain 0.0.311


### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```python
from langchain.text_splitter import RecursiveCharacterTextSplitter

url = "https://plato.stanford.edu/entries/goedel/"

headers_to_split_on = [
    ("h1", "Header 1"),
    ("h2", "Header 2"),
    ("h3", "Header 3"),
    ("h4", "Header 4"),
]

html_splitter = HTMLHeaderTextSplitter(headers_to_split_on=headers_to_split_on)

# for local file use html_splitter.split_text_from_file(<path_to_file>)
html_header_splits = html_splitter.split_text_from_url(url)

chunk_size = 500
chunk_overlap = 30
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size, chunk_overlap=chunk_overlap
)

# Split
splits = text_splitter.split_documents(html_header_splits)
splits[80:85]
```

[Reference](https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/HTML_header_metadata#2-pipelined-to-another-splitter-with-html-loaded-from-a-web-url)

The bug seems to be in [etree](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/text_splitter.py#L586). A simple fix is perhaps like below:

```python
from lxml import etree
from pathlib import Path

path = Path(".../langchain/document_transformers/xsl/html_chunks_with_headers.xslt")

# etree.parse(path) Throws

etree.parse(str(path))
```

### Expected behavior

The code in the reproducer should work.
Unable to Integrate ConversationChain with Tools like duckduckgo or serp-api
Labels: name: auto:question, name: auto:enhancement, name: area: models

### Issue you'd like to raise.

Hi, I'm looking for some implementation that's utilizing a ConversationChain and giving it access to Internet. I want to integrate duckduckgo or bing-api or serpapi into my ConversationChain. 

Would appreciate any help! Thanks

### Suggestion:

_No response_
Doctran translate documents
Labels: name: area: doc loader, name: auto:bug, name: auto:documentation

### System Info

IPython          : 8.15.0
ipykernel        : 6.25.0
ipywidgets       : 8.0.4
jupyter_client   : 7.4.9
jupyter_core     : 5.5.0
jupyter_server   : 1.23.4
jupyterlab       : 3.5.3
nbclient         : 0.8.0
nbconvert        : 7.10.0
nbformat         : 5.9.2
notebook         : 6.5.4
qtconsole        : 5.4.2
traitlets        : 5.7.1

Python 3.11.6
Langchain '0.0.340'


### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [X] Async

### Reproduction

TypeError: object Document can't be used in 'await' expression

translated_document = await qa_translator.atransform_documents(documents)
from official documentation notebook :
https://github.com/langchain-ai/langchain/blob/1cd9d5f3328e144cbe5d6ef52a22029d4fdf0cce/docs/docs/integrations/document_transformers/doctran_translate_document.ipynb

### Expected behavior

The problem might be arise due to specific python version or asyncio .
Ability to use custom tools and openai build in functions on OpenAIAssistantRunnable
Labels: name: area: agent, name: auto:enhancement, name: area: models

### Feature request

It would interesting to have the option tu run a OpenAIAssistantRunnable that has access to custom tools and OpenAI build in tools like code_intrepreter, vision and retrieval.

### Motivation

It would increase the developers capabilty of creating even more powerful agents.

### Your contribution

n/a
Issue: openai.NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}
Labels: name: area: embeddings, name: area: vector store, name: auto:bug, name: integration: chroma

### Issue you'd like to raise.

I use the below code to load data and splitted it and embedded it and finally pushing it into vector store. during that process,
I'm getting **openai.NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}**. After this method failed, I also tried with AzurecosmosDBVectorsearch vector store it also failed and returned the same error. Kindly help on this.

```
from langchain.embeddings.azure_openai import AzureOpenAIEmbeddings
from langchain.vectorstores.azure_cosmos_db import AzureCosmosDBVectorSearch
from langchain.vectorstores.chroma import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.document_loaders import PyPDFLoader

# Load PDF
loaders = [
    PyPDFLoader("ai.pdf")
]
docs = []
for loader in loaders:
    docs.extend(loader.load())
    # Define the Text Splitter
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1500,
    chunk_overlap=150
)

# Create a split of the document using the text splitter
res_splits = text_splitter.split_documents(docs)

embedding = AzureOpenAIEmbeddings(
    openai_api_version="1699-02-30",
    openai_api_key="xxxxxxxxxxxxxxxxxxxxxxxxx",
    #   model_name="gpt-35-turbo",
    azure_endpoint="https://ggggggggggggggggggggggg.openai.azure.com/")
persist_directory = 'docs/chroma/'



#  Create the vector store
vectordb = Chroma.from_documents(
    documents=res_splits,
    embedding=embedding,
    persist_directory=persist_directory
)

print(vectordb._collection.count())

```

### Suggestion:

_No response_