delayedkarma/langchain-issues huggingface dataset:
Fix "additional_kwargs["arguments"] already exists in this message, but with a different type." for Azure openai API. 
Labels: name: auto:bug


  - **Description:** This PR addresses an issue with the OpenAI API streaming response, where initially the key (arguments) is provided but the value is None. Subsequently, it updates with {"arguments": "{\n"}, leading to a type inconsistency that causes an exception. The specific error encountered is ValueError: additional_kwargs["arguments"] already exists in this message, but with a different type. This change aims to resolve this inconsistency and ensure smooth API interactions.
  - **Issue:** None.
  - **Dependencies:** None.
  - **Tag maintainer:** @eyurtsev,
Image template
Labels: name: auto:enhancement, name: area: models

example usage
```python
ffrom langchain.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([
    ("system", "foo bar"),
    ("human", [
        "{question}",
        {"text": "{second_question}"},
        # image with var name 'first_image' and no preset values
        {"image_url": "{first_image}"}, 
        # image with var name 'second_image' and preset "detail" level
        {"image_url": {"variable_name": "second_image", "detail": "low"}}, 
        # constant image, not a var
        {"type": "image_url", "image_url": {"url": "https://python.langchain.com/assets/images/langchain_stack-f195a2c417953cb5828f76e5f9039c9f.png"}}]), 
])
prompt.format_messages(
    question="what's the difference between these images", 
    second_question="what's upppppp",
    # specify an image path and other params
    first_image={
        "path": "/Users/bagatur/Downloads/LangChain Stack - V8.png", 
        "detail":"auto"
    }, 
    # specify just a url. preset value "detail": "low" will be attached
    second_image="data:blah"
)
```
fleet_context docs update
Labels: name: auto:documentation


  - **Description:** Changed the fleet_context documentation to use `context.download_embeddings()` from the latest release from our package. More details here: https://github.com/fleet-ai/context/tree/main#api
  - **Issue:** n/a
  - **Dependencies:** n/a
  - **Tag maintainer:** @baskaryan 
  - **Twitter handle:** @andrewthezhou
Adding support for amazon personalize Integration
Labels: name: auto:enhancement, name: area: models, name: integration: aws

## Amazon Personalize support on Langchain

This PR introduces an integration with [Amazon Personalize](https://aws.amazon.com/personalize/) to help you to retrieve recommendations and use them in your natural language applications. This integration provides two new components:


1. An `AmazonPersonalize` client, that provides a wrapper around the Amazon Personalize API.
2. An `AmazonPersonalizeChain`, that provides a chain to pull in recommendations using the client, and then generating the response in natural language. 

Here is some sample code to explain the usage.

```python
from langchain.utilities import AmazonPersonalize
from langchain.chains import AmazonPersonalizeChain
from langchain.llms.bedrock import Bedrock

recommender_arn = "<insert_arn>"

client=AmazonPersonalize(
    credentials_profile_name="default",
    region_name="us-west-2",
    recommender_arn=recommender_arn
)
bedrock_llm = Bedrock(
    model_id="anthropic.claude-v2", 
    region_name="us-west-2"
)

chain = AmazonPersonalizeChain.from_llm(
    llm=bedrock_llm, 
    client=client
)
response = chain({'user_id': '1'})
```


Reviewer: @3coins
Add token usage tracker module
Labels: name: auto:enhancement, name: area: models, name: integration: aws

  - **Description:** A new, flexible, extensible token usage tracker module 
  - **Dependencies:** only optional dependencies base on the LLM used,
  - **Tag maintainer:** @agola11 @baskaryan @hwchase17 
  - **Twitter handle:** jtolgyesi

The highly customizable `token_usage` module  offers to track the number of consumed tokens, measure the latency of the LLM responses, and even estimate the usage cost of your application. You can analyze the token usage of your application directly from the code, or send the usage data to a metrics repository such as Amazon CloudWatch.

The `token_usage` module splits the task of collecting the usage metrics from the LLMs via standard Callbacks in the `handlers` submodule, and the processing of the collected metrics in the `reporters` submodule.

Some LLM APIs (for example, OpenAI) sends you back the number of consumed tokens after each call. Other APIs do not provide such usage information. In this case you can configure a `LocalTokenUsageCallbackHandler` instance to track the token usage directly on client side (your application's code).
fix: Reduce post-processing time for `DocAIParser`
Labels: name: area: doc loader, name: auto:improvement

- Remove `WrappedDocument` introduced in https://github.com/langchain-ai/langchain/pull/11413
  - https://github.com/googleapis/python-documentai-toolbox/issues/198 in Document AI Toolbox to improve initialization time for `WrappedDocument` object.

@lkuligin

@baskaryan

@hwchase17
Mask API key for SerpApi.com tools
Labels: name: auto:improvement

* **Description:** Mask API key for SerpApi.com tools.
  * Mask SerpApi API key in `SerpAPIWrapper` and `GoogleScholarAPIWrapper`
  * Use [`SerpApiClient#pagination`](https://github.com/serpapi/google-search-results-python#pagination-using-iterator) in the Google Scholar Search Tool (related to https://github.com/langchain-ai/langchain/issues/11513)
* **Issue:** https://github.com/langchain-ai/langchain/issues/12165
* **Tag maintainer:** @eyurtsev

@keenborder786, please review the changes in `GoogleScholarAPIWrapper`.
Update README.md - Added notebook for extraction_openai_tools 
Labels: name: lgtm, name: auto:documentation

added Parallel Function Calling for Structured Data Extraction notebook

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
NameError: name '_httpx_client' is not defined
Labels: name: auto:bug

### System Info

Hello All,

I have just installed the helm chart with some small additions to the basic values.yaml and the "langchain-langsmith-backend" contain keeps breaking with the following error, Has anyone had this before?
INFO: Started server process [1]
--
Fri, Nov 10 2023 2:41:24 pm | INFO: Waiting for application startup.
Fri, Nov 10 2023 2:41:24 pm | INFO: Application startup complete.
Fri, Nov 10 2023 2:41:24 pm | INFO: Uvicorn running on http://0.0.0.0:1984 (Press CTRL+C to quit)
Fri, Nov 10 2023 2:42:18 pm | INFO: Shutting down
Fri, Nov 10 2023 2:42:18 pm | INFO: Waiting for application shutdown.
Fri, Nov 10 2023 2:42:19 pm | ERROR:root:Error closing httpx client for clickhouse name '_httpx_client' is not defined
Fri, Nov 10 2023 2:42:19 pm | Traceback (most recent call last):
Fri, Nov 10 2023 2:42:19 pm | File "/code/smith-backend/app/main.py", line 140, in shutdown_event
Fri, Nov 10 2023 2:42:19 pm | File "/code/lc_database/lc_database/clickhouse.py", line 36, in close_clickhouse_client
Fri, Nov 10 2023 2:42:19 pm | await _httpx_client.aclose()
Fri, Nov 10 2023 2:42:19 pm | ^^^^^^^^^^^^^
Fri, Nov 10 2023 2:42:19 pm | NameError: name '_httpx_client' is not defined
Fri, Nov 10 2023 2:42:19 pm | INFO: Application shutdown complete.
Fri, Nov 10 2023 2:42:19 pm | INFO: Finished server process [1]




### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

helm repo add langchain https://langchain-ai.github.io/helm/
helm install langchain/langsmith . --values values.yaml --namespace it-dev-langchain

Running this on an on prem Rancher cluster.

### Expected behavior

The contain runs normally.
Fix #11737 issue (extra_tools option of create_pandas_dataframe_agent is not working)
Labels: name: area: agent, name: lgtm, name: auto:bug, name: size:M

  - **Description:** Fix #11737 issue (extra_tools option of create_pandas_dataframe_agent is not working), 
  - **Issue:** #11737 ,
  - **Dependencies:** no,
  - **Tag maintainer:** @baskaryan, @eyurtsev, @hwchase17 I needed this method at work, so I modified it myself and used it. There is a similar issue(#11737) and PR(#13018) of @PyroGenesis, so I combined my code at the original PR. 
  You may be busy, but it would be great help for me if you checked. Thank you.
  - **Twitter handle:** @lunara_x 

If you need an .ipynb example about this, please tag me. 
I will share what I am working on after removing any work-related content.
Inconsistent output format of ChatVertexAI compared to ChatOpenAI
Labels: name: auto:bug, name: area: models

### System Info

I tried to use `ChatVertexAI` as a replacement for `ChatOpenAI` as the latter is quite slow these days. 

I have this code for Chat OpenAI

```
template_string = """ # some explanation
give me suggestions in JSON format where the suggestions are a list of dictionaries with the following keys:        
      - before:
      - after:
      - reason:
"""

chat = ChatOpenAI(
    temperature=0.0,
    openai_api_key=self.session_info.openai_api_key,
    model=llm_model,
)

prompt_template = ChatPromptTemplate.from_template(template_string)

service_messages = prompt_template.format_messages(
    text=self.text
)
response = chat(service_messages)
info = json.loads(response.content)
```

Then I print `response.content`, here is the format as expected:

```
[{"before": ..., "after": ...}, ...]
```

Then I use `ChatVertexAI` with the same `template_string`

```
chat = ChatVertexAI(
    temperature=0.0,
    google_api_key=google_api_key,
    model="codechat-bison",
    max_output_tokens=2048,
)

prompt_template = ChatPromptTemplate.from_template(template_string)

service_messages = prompt_template.format_messages(
    text=self.text
)
response = chat(service_messages)
```

I print the `response.content` coming from Vertex AI and the output format is encapsulated by 
```(3 backticks)JSON [{...}] (3 backticks)```. To make `response.content` the same, I used the following code

```
json_match = re.search(r'```JSON(.*?)\n```', response.content, re.DOTALL)

if json_match:
    json_data = json_match.group(1).strip()
    info = json.loads(json_data)
else:
    info = {}
    print("No JSON data found in the input string.")
```

I am not sure of the potential failures of this solution, but unless this is intentional, it might be better to make it consistent. 

I will try to use the other LLMs and see how others work.

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I shared my codes above, but the steps are:

1. setting up openai_api_key
2. setting up gcp environment
3. running ChatOpenAI
4. running ChatVertexAI
5. comparing the responses (i.e. `response.content`)

### Expected behavior

My expectation is to obtain the outputs of different models the same way. i.e., `json.loads` should output a list of dictionary in each case. Users should not have to deal with `regex` to obtain the data in the expected format.
Adds an in-memory implementation of RecordManager
Labels: name: area: memory, name: auto:enhancement, name: size:L

**Description:**
langchain offers three technologies to save data:
- [vectorstore](https://python.langchain.com/docs/modules/data_connection/vectorstores/)
- [docstore](https://js.langchain.com/docs/api/schema/classes/Docstore)
- [record manager](https://python.langchain.com/docs/modules/data_connection/indexing)

If you want to combine these technologies in a sample persistence stategy you need a common implementation for each. `DocStore` propose `InMemoryDocstore`. 

We propose the class `MemoryRecordManager` to complete the system.

This is the prelude to another full-request, which needs a consistent combination of persistence components.

**Tag maintainer:**
@baskaryan

**Twitter handle:**
@pprados
More than 800 errors detected with the latest version of mypy
Labels: name: area: memory, name: area: agent, name: area: embeddings, name: area: doc loader, name: area: vector store, name: auto:bug, name: area: models

### System Info

langchain 0.0.333


### Who can help?

@hwchase17

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [X] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [X] Document Loaders
- [X] Vector Stores / Retrievers
- [X] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [X] Chains
- [X] Callbacks/Tracing
- [X] Async

### Reproduction

The current mypy crash with  `make lint` and propose to install the last master version.
So, after a `git clone` and `poetry install --with dev,test,lint`, run

```bash
pip install git+https://github.com/python/mypy.git
make lint
```
**Mypy found 834 errors in 238 files**

### Expected behavior

0 errors
MlflowAIGatewayEmbeddings : Default Batch size incompatible with Azure OpenAI text-embedding-ada-002
Labels: name: area: embeddings, name: auto:bug, name: auto:improvement

### System Info

The text-embedding-ada-002 OpenAI embedding model on Azure OpenAI has a maximum batch size of 16. MlflowAIGatewayEmbeddings has a hard-coded batch size of 20 which results in it being unusable with Azure OpenAI's text-embedding-ada-002.

The best fix would be to allow a configurable batchsize as an argument to MlflowAIGatewayEmbeddings,

### Who can help?

@hwchase17 @agola11 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. Create a gateway route to the text-embedding-ada-002 on azure openai
```
ROUTE_NAME = "azure-ada-002"

# Try to delete the route if it already exists
try:
    delete_route(ROUTE_NAME)
    print("Route deleted")
except:
    print("Route does not exist, creating..")

create_route(
    name=ROUTE_NAME,
    route_type="llm/v1/embeddings",
    model = {
        "name" : "text-embedding-ada-002",
        "provider" : "openai",
        "openai_config" : {
            "openai_api_type" : azure_openai_type,
            "openai_api_key": azure_openai_key,
            "openai_deployment_name": "ada-embed-v1",
            "openai_api_base": azure_openai_base,
            "openai_api_version": "2023-05-15"
        }
    }
)
```
2.  Initialize the `MlflowAIGatewayEmbeddings` and try to embed more than 16 documents
```
from langchain.embeddings import MlflowAIGatewayEmbeddings
import pandas as pd

azure_ada = MlflowAIGatewayEmbeddings(route="azure-ada-002")

test_strings = [
    "aaaa" for i in range(20)
]
azure_ada.embed_documents(test_strings)
```
3. Observe the error
```
HTTPError: 400 Client Error: Bad Request for url: https://ohio.cloud.databricks.com/gateway/azure-ada-002/invocations. Response text: {
  "error": {
    "message": "Too many inputs. The max number of inputs is 16.  We hope to increase the number of inputs per request soon. Please contact us through an Azure support request at: https://go.microsoft.com/fwlink/?linkid=2213926 for further questions.",
    "type": "invalid_request_error",
    "param": null,
    "code": null
  }
}
```

### Expected behavior

I should be able to use the `MlflowAIGatewayEmbeddings` class with Azure OpenAI.
langchain Retrieval
Labels: name: area: embeddings, name: area: doc loader, name: area: vector store, name: auto:question, name: auto:improvement, name: integration: chroma

### Issue you'd like to raise.

i have the following code:

```
# import
from langchain.embeddings.sentence_transformer import SentenceTransformerEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import Chroma
from langchain.document_loaders import TextLoader
from silly import no_ssl_verification
from langchain.embeddings.huggingface import HuggingFaceEmbeddings

with no_ssl_verification():
    # load the document and split it into chunks
    loader = TextLoader("state_of_the_union.txt")
    documents = loader.load()

    # split it into chunks
    text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
    docs = text_splitter.split_documents(documents)

    # create the open-source embedding function
    embedding_function = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
    # hfemb = HuggingFaceEmbeddings()
    # load it into Chroma
    db = Chroma.from_documents(docs, embedding_function)

    # query it
    query = "What did the president say about Ketanji Brown Jackson"
    docs = db.similarity_search(query)

    # print results
    print(docs[0].page_content)
```
i must use retrieval mechanism. and i am using Turkish language data. please change my code 



### Suggestion:

_No response_
Example for chat chain with documents retrieval and history capability 
Labels: name: area: embeddings, name: area: vector store, name: auto:question, name: area: models, name: integration: chroma

### System Info

Latest langchain, Mac

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Hi Community,

I'm trying to build a chain with Chroma database as context, AzureOpenAI embeddings and AzureOpenAI GPT model.

The following libs work fine for me and doing their work
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.chat_models import AzureChatOpenAI

But I can't figure out how to build the chain itself and what kind of chain I should use.
Could you kindly provide any suggestions?

Thanks, Artem.

### Expected behavior

Desired behaviour: 
- user asks question
- code searches it in Chroma using AzureOpenAI embeddings to transform text
- In prompt for AzureOpenAI GPT there is context with info retrieved from Chroma and current chat history with Human/AI questions and answers
- Also there should be a custom prompt template used
- Also I want to wrap it in Flask server, so users chat history could be stored in session with username as key until it's forcibly cleared
The Agent is not using Custom Tools.
Labels: name: area: agent, name: auto:bug

### System Info

I just updated langchain to newest version and my Agent is not working anymore. 

Tool structure : 

 ```
class Data_Retriever(BaseModel):
    db : Any
   
    class Config:
        extra = Extra.forbid

    def run(self,request) -> str:
        data = self.db.get(request)
        return data
```

When Agent uses the above tool, it will always stop at "Action Input" step : 

```
> Entering new AgentExecutor chain...

AI: Thought: Do I need to use a tool? Yes
Action: Data_Retriever
Action Input: DATA 

> Finished chain.
``` 

Does anyone know how to fix this ? I'm using langchain==0.0.332

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```from langchain.tools import BaseTool, StructuredTool, Tool, tool
from langchain.agents import AgentType, initialize_agent,load_tools

tools = [
        Tool(
        name="Music Search",
        func=lambda x: "'All I Want For Christmas Is You' by Mariah Carey.", # Mock Function
        description="A Music search engine. Use this more than the normal search if the question is about Music, like 'who is the singer of yesterday?' or 'what is the most popular song in 2022?'",
        ),
        ]

agent = initialize_agent(
            agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION, 
            tools=tools, 
            llm=llm, 
            verbose=True)

agent.run(input = "Random song")
```
### Expected behavior

The output should be All I Want For Christmas Is You' by Mariah Carey.


My agent stopped once it hit the action input step.

> Entering new AgentExecutor chain...
AI: Thought: Do I need to use a tool? Yes
Action: Music Search
Action Input: "Random song"


> Finished chain.