delayedkarma/langchain-issues huggingface dataset:
Could not parse tool input: {'name': 'AI_tool', 'arguments': 'What is a pre-trained chatbot?'} because the arguments is not valid JSON.
Labels: name: area: agent, name: area: embeddings, name: auto:bug, name: area: models

### System Info

      I am using OpenAIFunctionsAgent  with langchain-0.0.285, parse tool input occurs frequently when provided an input

Could not parse tool input: {'name': 'AI_tool', 'arguments': 'What is a pre-trained chatbot?'} because the arguments is not valid JSON.


### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [X] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction


     retriever = db.as_retriever()  # Milvus 

        tool = create_retriever_tool(
            retriever,
            "document_search_tool",
            "useful for answering questions related to XXXXXXXX."
        )

        tool_sales = create_retriever_tool(
            retriever,
            "sales_tool",
            "useful for answering questions related to buying or subscribing XXXXXXXX."
        )

        tool_support = create_retriever_tool(
            retriever,
            "support_tool",
            "useful for when you need to answer questions related to support humans on XXXXXXXX."
        )
        tools = [tool, tool_sales]



        llm = ChatOpenAI(model_name="gpt-3.5-turbo-0613", temperature=0.3)
        system_message = SystemMessage(
            content=(
                "You are a digital team member of XXXXXXXX Organization, specialising in XXXXXXXX."
                "Always respond and act as an office manager of XXXXXXXX, never referring to the XXXXXXXX "
                "as an external or separate entity. "
           
                "* Please answer questions directly from the context, and strive for brevity, keeping answers under 30 words."
                "* Convey information in a manner that's both professional and empathetic, embodying the values of XXXXXXXX."

            )
        )
        prompt = OpenAIFunctionsAgent.create_prompt(
            system_message=system_message,
            extra_prompt_messages=[MessagesPlaceholder(variable_name="chat_history")]
        )
        agent = OpenAIFunctionsAgent(llm=llm, tools=tools, prompt=prompt, verbose=True)

        memory = ConversationBufferWindowMemory(memory_key="chat_history", return_messages=True, k=6)
        agent_executor = AgentExecutor(
            agent=agent,
            tools=tools,
            memory=memory,
            verbose=True,
        )



      result = agent({"input": question, "chat_history": chat_history})
      answer = str(result["output"])

     print(answer)

### Expected behavior

i need to remove the error
add  vearch vectorstore
Labels: name: area: vector store, name: auto:enhancement

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
Azure Doc Intelligence 0.2 - support paragraphs and tables for multiple models
Labels: name: area: doc loader, name: auto:improvement

This PR introduces enhancements to the Azure Document Intelligence document loader.

- Uses paragraphs for creation of full page text introducing efficiency with lesser iterations. Paragraphs are supported by [all models](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/concept-model-overview?view=doc-intel-3.1.0#model-data-extraction).
- Supports paragraphs via `split_mode` during initialization of `DocumentIntelligenceLoader`. This defaults to `page` in which case the full text of the page will be returned. If `paragraph` is used in `split_mode` then Documents will be returned in chunks by paragraphs. Paragraphs may be useful in generating embeddings in smaller chunks instead of having to split the full page text yet again.
- Provides table data extraction if the model specified is either `prebuilt-document`, `prebuilt-layout`, or `prebuilt-invoice`. This is useful for developers who intend to use tables with [Self-query](https://python.langchain.com/docs/modules/data_connection/retrievers/self_query/).
- Introduces a `type` key in `Document` metadata to help distinguish just page text vs paragraph vs tables with `PAGE`, `PARAGRAPH`, `TABLE_HEADER` and `TABLE_ROW`.
- For tables, provides the headers and rows in CSV format along with the table index while retaining the page number, which can be used to load vector db for self query. Note: metadata formatting with `Document` schema for self-query will still be needed which can be done with the help of `type` key (`TABLE_HEADER` and `TABLE_ROW`), `table_index`, and `page`.

Sample usage

```python
from azure.core.credentials import AzureKeyCredential
from azure.ai.formrecognizer import DocumentAnalysisClient
from langchain.document_loaders.pdf import DocumentIntelligenceLoader

document_analysis_client = DocumentAnalysisClient(endpoint="<endpoint>", credential=AzureKeyCredential("<key>"))

loader = DocumentIntelligenceLoader("./document.pdf",
    client=document_analysis_client,
    model="prebuilt-document",
    split_mode="paragraph"          # optional, defaults to `page`
) 
documents = loader.load()

tables = list()
for doc in documents:
  if doc.metadata['type'] in ['PAGE', 'PARAGRAPH']:
    # page text
    print(f"====Page {doc.metadata['page']} {doc.metadata['type']}-text====\n\n")
    print(doc)
    print("\n\n")
  elif doc.metadata['type'] in ['TABLE_HEADER', 'TABLE_ROW']:
    tables.append(doc)

# first table in the document
table1 = [d for d in tables if d.metadata['table_index'] == 0]
# second table in the document
table2 = [d for d in tables if d.metadata['table_index'] == 1]
# third table in the document
table3 = [d for d in tables if d.metadata['table_index'] == 2]
```
BaseChatModel implementation using CTransformers
Labels: name: auto:enhancement, name: area: models

### Feature request

Could you add an implementation of BaseChatModel using CTransformers?

### Motivation

I prefer to use a local model instead of an API. the LLM works, but I need the wrapper for it

### Your contribution

My failed attempt

```
from pydantic import BaseModel, Field
from typing import Any, List, Optional

from ctransformers import AutoModelForCausalLM, LLM
from langchain.callbacks.manager import CallbackManagerForLLMRun
from langchain.chat_models.base import SimpleChatModel
from langchain.schema import BaseMessage, HumanMessage


class CTransformersChatModel(SimpleChatModel, BaseModel):
    ctransformers_model: LLM = Field(default_factory=AutoModelForCausalLM)

    def __init__(self, model_path: str, model_type: Optional[str] = "llama", **kwargs: Any):
        super().__init__(**kwargs)
        self.ctransformers_model = AutoModelForCausalLM.from_pretrained(model_path)

    def _call(
            self,
            messages: List[BaseMessage],
            stop: Optional[List[str]] = None,
            run_manager: Optional[CallbackManagerForLLMRun] = None,
            **kwargs: Any,
    ) -> str:
        # Convert messages to string prompt
        prompt = " ".join([message.content for message in messages if isinstance(message, HumanMessage)])
        return self.ctransformers_model(prompt, stop=stop, run_manager=run_manager, **kwargs)

    @property
    def _llm_type(self) -> str:
        """Return type of chat model."""
        return "ctransformers_chat_model"
```
Remove broken documentation links
Labels: name: auto:documentation

Description: Removed some broken links for popular chains and additional/advanced chains.
Issue: None
Dependencies: None
Tag maintainer: none yet
Twitter handle: ferrants 

Alternatively, these pages could be created, there are snippets for the popular pages, but no popular page itself.

![Screenshot from 2023-09-10 14-53-01](https://github.com/langchain-ai/langchain/assets/882183/7b79ab37-f6ca-4c41-a9dd-251b9bd8c214)

![Screenshot from 2023-09-10 15-00-26](https://github.com/langchain-ai/langchain/assets/882183/86c5ec64-aee1-4e44-b409-108b92a75da9)
Conversational Retrieval Agent returning partial output
Labels: name: area: embeddings, name: area: vector store, name: auto:bug, name: area: models

### System Info

As far as I tried, this reproduced in many versions, including the latest `langchain==0.0.285`



### Who can help?

@agola11  @hwchase17 

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Using the following code
```
    llm = ChatOpenAI(model_name="gpt-4", temperature=0, verbose=True) # sometimes with streaming=True

    # example of one tool thats being used 
    loader = PyPDFLoader(insurance_file)
    pages = loader.load_and_split()
    faiss_index = FAISS.from_documents(pages, OpenAIEmbeddings())
    health_insurance_retriever = faiss_index.as_retriever()
    tool = create_retriever_tool(health_insurance_retriever, "health_insurance_plan",
                                 "XXX Description")

    agent_executor = create_conversational_retrieval_agent(
        llm, [tool1, tool2], verbose=True, system_message="...")
    agent_executor("Some question that requires usage of retrieval tools")


```

The results often (statistically, but reproduces pretty frequently) is returned with some references such as the following 


```I'm sorry to hear that you're experiencing back pain. Let's look into your health insurance plan to see what coverage you have for this issue.

[Assistant to=functions.health_insurance_plan]
{
"__arg1": "back pain"
}
...
[Assistant to=functions.point_solutions]
{
"__arg1": "back pain"
}
````

### Expected behavior

Chain using the retrieval tools to actually query the vector store, instead of returning the placeholders

Thank you for your help!
Is there a support for extracting images from pdf?
Labels: name: area: doc loader, name: auto:enhancement

### Feature request

Provide a parameter to determine whether to extract images from the pdf and give the support for it.

### Motivation

There may exist several images in pdf that contain abundant information but it seems that there is no support for extracting images from pdf when I read the code. 

### Your contribution

I'd like to add the feature if it is really lacking.
Gmail Toolkit sends message and rootId of message
Labels: name: area: agent, name: auto:bug

### System Info

Issue: Agent + GmailToolkit sends message AND rootId to the recipient address

Current Behaviour: 
1. Instruct the Agent to send a message to the recipient
2. Agent emails recipient with a message and then sends a new message of just the rootId (e.g., `r25406384....`)
Example:
<img width="1181" alt="Screenshot 2023-09-10 at 10 48 29 AM" src="https://github.com/langchain-ai/langchain/assets/94654154/68258bee-985e-4844-9ae8-00b81248d166">



Desired Behaviour:
1. Instruct the Agent to send a message to the recipient
2. Agent emails recipient with only the message and NOT the rootId



My initial suspicion is that this has to do with the prompting of the agent and the multistep process of writing, drafting, and sending all in one go. Currently looking into this and will add any updates/findings here. All help and suggestions welcome!

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Run the agent notebook from the docs: https://python.langchain.com/docs/integrations/toolkits/gmail

### Expected behavior

Desired Behaviour:
1. Instruct the Agent to send a message to the recipient
2. Agent emails recipient with only the message and NOT the rootId
Fix of issue:
Labels: name: area: memory, name: lgtm, name: auto:documentation

DOC: Inversion of 'True' and 'False' in ConversationTokenBufferMemory Property Comments #10420

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: Swap the True and False word in ConverstaionTokenBufferMemory buffer_as_str and buffer_as_messages methods, 
  - Issue: the issue #10420,
  - Dependencies: None
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
DOC: Inversion of 'True' and 'False' in ConversationTokenBufferMemory Property Comments
Labels: name: area: memory, name: auto:documentation

### Issue with current documentation:

### Problem:
In the _**libs/langchain/langchain/memory/token_buffer.py**_ file:

 ```
 @property
    def buffer(self) -> Any:
        """String buffer of memory."""
        return self.buffer_as_messages if self.return_messages else self.buffer_as_str

    @property
    def buffer_as_str(self) -> str:
        """Exposes the buffer as a string in case return_messages is True."""
        return get_buffer_string(
            self.chat_memory.messages,
            human_prefix=self.human_prefix,
            ai_prefix=self.ai_prefix,
        )

    @property
    def buffer_as_messages(self) -> List[BaseMessage]:
        """Exposes the buffer as a list of messages in case return_messages is False."""
        return self.chat_memory.messages
```

The **True** and **False** words should be inverted in the **buffer_as_str** and **buffer_as_messages** methods' documentation.
See the logic in the **buffer** method's return:

> return self.buffer_as_messages if self.return_messages else self.buffer_as_str

### Correction:

Swap both words: 
**True**  :arrow_backward: :arrow_forward: **False** 

To get that result:
 ```
 @property
    def buffer(self) -> Any:
        """String buffer of memory."""
        return self.buffer_as_messages if self.return_messages else self.buffer_as_str

    @property
    def buffer_as_str(self) -> str:
        """Exposes the buffer as a string in case return_messages is False."""
        return get_buffer_string(
            self.chat_memory.messages,
            human_prefix=self.human_prefix,
            ai_prefix=self.ai_prefix,
        )

    @property
    def buffer_as_messages(self) -> List[BaseMessage]:
        """Exposes the buffer as a list of messages in case return_messages is True."""
        return self.chat_memory.messages
```

### Idea or request for content:

_No response_
DOC: Passing parameters to tools from Agent
Labels: name: area: agent, name: auto:documentation

### Issue with current documentation:
#  Combine the LLM with the tools with  make a ReAct agent

Currently, the document says , agent can take only user question . Is there any example to pass multiple parameters? 

inputdata ={"input": COMPLEX_QUERY, "channel":"mychannel", "product":"myproduct"}


react_agent = initialize_agent(tools,
                               llm,
                               agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
                               verbose=True)


react_agent.run(inputdata)

channel and product custom parameters that needs to be passed to Tool 


### Idea or request for content:

_No response_
:bulb: SmartLLMChain  > Rrandomized temperatures for ideation_llm for better crowd diversity simulation
Labels: name: auto:enhancement, name: area: models

### Feature request

In the [SmartLLMChain](https://python.langchain.com/docs/use_cases/more/self_check/smart_llm), I would like to randomize the temperature of the `ideation_llm` . It could have a positive impact on its creativity then evaluation.

We could ask for specific pairs of nb of llm/temperature or automate temperature distribution with classical ones

- Gaussian
- Poisson
- Uniform
- Exponential
- Geometric
- Log-Normal
- ...

### Motivation

Potentally enhance overall chain performances.

### Your contribution

I could write a blog post about the benchmark.
Save options or examples instead of history with memory.save_context() in VectorStoreRetrieverMemory
Labels: name: area: memory, name: area: vector store, name: auto:question, name: auto:enhancement

### Feature request

Is there a way to save options or examples instead of history with `memory.save_context()` in `VectorStoreRetrieverMemory`? 

For example, if A, B, and C are expected as the answer to a certain question, A, B, and C are returned as the predict results for that question. The current memory function takes into account the flow of history, so it is not possible to select options or define behavior like the switch statement in programming languages.

If I just don't know about this feature, I'd appreciate it if you could let me know. In a past issue, if someone wanted to have a rule-based conversation, the answerer just only said to use `VectorStoreRetrieverMemory`, but no examples were introduced. If you have any simple exmaples, please let me know.

### Motivation

This is to stably control the chatbot's behavior.

### Your contribution

I checked and found out that this feature does not currently exist.
Supported custom ernie_api_base for Ernie
Labels: name: area: embeddings, name: auto:improvement

Description: Supported custom ernie_api_base for Ernie
 - ernie_api_base：Support Ernie custom endpoints
 - Rectifying omitted code modifications. #10398

Issue: None
Dependencies: None
Tag maintainer: @baskaryan 
Twitter handle: @JohnMai95
Support for llama-cpp-python server
Labels: name: auto:question, name: auto:enhancement, name: area: models

### Feature request

Currently [llama-cpp-python](https://github.com/abetlen/llama-cpp-python#web-server) provides server package which acts like a drop-in replacement for the OpenAI API.

Is there any specific langchain LLM class which supports the above server or do we need to alter the existing `OpenAI` class with a different `openai_api_base` ?

### Motivation

I would like to have a dedicated machine or host which runs only the llama-cpp-python server wheres as the client which uses langchain should interact with just like we are doing with OpenAI.

### Your contribution

I would like to contribute but before that I need to check if there's any solution already available.
CharacterTextSplitter doesn't break down text into specified chunk sizes
Labels: name: area: doc loader, name: auto:bug, name: auto:documentation

### System Info

langchain version 0.0.285

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

CharacterTextSplitter has options for `chunk_size` and `chunk_overlap` but doesn't make use of them in the splitting. Though this is not technically erroneous, it is misleading given that a lot of the documentation shows CharacterTextSplitter with these arguments specified, implying that the class is creating desirably sized chunks when in reality it is not. Here is an [example](https://python.langchain.com/docs/integrations/vectorstores/activeloop_deeplake) of such documentation implying this.

Below is a code sample reproducing the problem. RecursiveCharacterTextSplitter works to reorganize the texts into chunks of the specified `chunk_size`, with chunk overlap where appropriate. Meanwhile, CharacterTextSplitter doesn't do this. You can observe the difference in the overlap behavior by printing out `texts_c` and `texts_rc`.

```
from langchain.schema.document import Document
from langchain.text_splitter import CharacterTextSplitter, RecursiveCharacterTextSplitter

doc1 = Document(page_content="Just a test document to assess splitting/chunking")
doc2 = Document(page_content="Short doc")
docs = [doc1, doc2]

text_splitter_c = CharacterTextSplitter(chunk_size=30, chunk_overlap=10)
text_splitter_rc = RecursiveCharacterTextSplitter(chunk_size=30, chunk_overlap=10)

texts_c = text_splitter_c.split_documents(docs)
texts_rc = text_splitter_rc.split_documents(docs)

max_chunk_c = max([ len(x.to_json()['kwargs']['page_content']) for x in texts_c])
max_chunk_rc = max([ len(x.to_json()['kwargs']['page_content']) for x in texts_rc])

print(f"Max chunk in CharacterTextSplitter output is of length {max_chunk_c}")
print(f"Max chunk in RecursiveCharacterTextSplitter output is of length {max_chunk_rc}")
```

### Expected behavior

Either remove the arguments from CharacterTextSplitter to avoid ambiguity, use RecursiveCharacterTextSplitter which performs the expected behavior of resizing into appropriately sized chunks, or add to CharacterTextSplitter a split_text function to perform the aforesaid expected behavior
Adding File-Like object support in CSV Agent Toolkit
Labels: name: area: agent, name: lgtm, name: auto:improvement

If loading a CSV from a direct or temporary source, loading the file-like object (subclass of IOBase) directly allows the agent creation process to succeed, instead of throwing a ValueError. 

Added an additional elif and tweaked value error message.
Added test to validate this functionality.

Pandas from_csv supports this natively but this current implementation only accepts strings or paths to files. https://pandas.pydata.org/docs/user_guide/io.html#io-read-csv-table