delayedkarma/langchain-issues huggingface dataset:
Custom Tool Return Direct BUT only if it doesnt raise a Tool Exception
Labels: name: area: agent, name: auto:bug, name: area: models

### System Info

0.0.287 langchain

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Currently I have a Custom Tool like this:
```
def _handle_error(error: ToolException) -> str:
    return (
        "The following errors occurred during tool execution:"
        + error.args[0]
        + "please try another tool."
    )
    tools = [
      Tool.from_function(
             name="GetCurrentWeather",
             func=get_current_weather,
             description="useful for getting the current weather for a location",
             return_direct=True,
             handle_tool_error=_handle_error)]
```
It works unless I add the return_direct=True.

When I do return_direct= True instead of trying other tools it just returns this:

```
> Entering new AgentExecutor chain...
16 AgentAction --> Thought: The question is asking about the current weather in Uludağ.
17 Action: GetCurrentWeather
18 Action Input: Uludağ
19 Thought: The question is asking about the current weather in Uludağ.
20 Action: GetCurrentWeather
21 Action Input: Uludağ
22 Observation:The following errors occurred during tool execution:the location is not found. please try another tool.
23 > Finished chain.
24 The following errors occurred during tool execution:location is not found.please try another tool.
```

### Expected behavior

Let's see the same thing without the `return_direct=True`:
```
> Entering new AgentExecutor chain...
Thought: The question is asking about the current weather in Uludağ.
Action: GetCurrentWeather
Action Input: Uludağ

Observation:The following errors occurred during tool execution:location is not found. please try another tool.
I couldn't find the current weather for Uludağ. I should try another tool.
Action: DuckDuckSearch
Action Input: "Uludağ current weather"

Observation:Current Weather. 7:35 PM. 78° F. RealFeel® 74°. RealFeel Shade™ 74°. Air Quality Fair. Wind W 14 mph. Wind Gusts 24 mph. Sunny More Details. 7:14. —. This table gives the weather forecast for Uludag at the specific elevation of 2543 m. Our advanced weather models allow us to provide distinct weather forecasts for several elevations of Uludag. ..
```
So it goes for the other tools. It should be like this for return_direct.

It should return directly only if it doesn't get an error.
DOC: Custom Tool Error Handling is not Clear
Labels: name: area: agent, name: auto:documentation

### Issue with current documentation:

I cannot handle my custom_tool's raised errors.

I have this weatherTool:

```
class GetCurrentWeatherTool(BaseTool):
    name = "get_current_weather"
    description = "Get the current weather for a specified location"
    args_schema: Type[WeatherInput] = WeatherInput

    def _run(
        self, 
        location: str,
        run_manager: Optional[CallbackManagerForToolRun] = None
    ) -> str:
        """Use the tool."""
        return get_current_weather(location)

    async def _arun(
        self, 
        location: str,
        run_manager: Optional[AsyncCallbackManagerForToolRun] = None
    ) -> str:
        """Use the tool asynchronously."""
        # You can implement the async version of the method here or raise a NotImplementedError if async is not supported
        raise NotImplementedError("get_current_weather does not support async")

    def _handle_error(error: ToolException) -> str:
        return (
            "The following errors occurred during tool execution:"
            + error.args[0]
            + "Location couldn't be found, please try another tool."
        )
```
And I raise ToolException when the API call gets an error:
`raise ToolException("location has been found.")` inside of `get_current_weather`


Now I am adding this tool to my tools when I'm creating my agent:

```
self.weather = GetCurrentWeatherTool()
            Tool(
                 name="Weather",
                 func=self.weather.run,
                 description="useful for weather related questions",
                 return_direct=True,
             )
```
In the documentation [Custom Tools](https://python.langchain.com/docs/modules/agents/tools/custom_tools#handling-tool-errors) it explains how to add it and gives an example:

```
    Tool.from_function(
        func=search_tool1,
        name="Search_tool1",
        description=description,
        handle_tool_error=True,
    ),
```
             
However I couldnt find the equal of that for Subclassing the BaseTool.

### Idea or request for content:

It would be great to give an example for Subclassed CustomTool(BaseTool) when defining the tools.
Bring MRKL prompts in line with the other Agent prompts
Labels: name: area: agent, name: auto:improvement

### Feature request

The current MRKL `FORMAT_INSTRUCTIONS` prompt ([here](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/agents/mrkl/prompt.py)) looks like this: 

``` python
FORMAT_INSTRUCTIONS = """Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question"""
```

Which is not very good- the similar tool selection prompts in other agents are MUCH better, and contain detailed formatting instructions.  

This prompt should be brought in line with the likes of the [Structured Chat Prompt](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/agents/structured_chat/prompt.py) which is much better, in a number of ways (clearer instructions, better formatting, etc.)

As always, I appreciate your hard work!

### Motivation

The motivation for this is to bring the prompt quality up throughout the project.  Overall, there is probably some refactoring that could be done in this area to make it easier to centralize how prompts are used... I know that there's the hub now, but I am not sure if that fits this bill.

### Your contribution

I will try to contribute to this, but my workload is intense, and I don't have time currently... however, I have a number of things that I have modified in my own instance that I really should contribute back. 🥲
fix(embeddings): number of texts in Azure OpenAIEmbeddings batch
Labels: name: area: embeddings, name: auto:improvement

This PR addresses the limitation of Azure OpenAI embeddings, which can handle at maximum 16 texts in a batch. This can be solved setting `chunk_size=16`. However, I'd love to have this automated, not to force the user to figure where the issue comes from and how to solve it. 

Closes #4575. 

@baskaryan
fix(embeddings): number of texts in Azure OpenAIEmbeddings batch
Labels: name: area: embeddings, name: auto:improvement

This PR addresses the limitation of Azure OpenAI embeddings, which can handle at maximum 16 texts in a batch. This can be solved setting `chunk_size=16`. However, I'd love to have this automated, not to force the user to figure where the issue comes from and how to solve it.

@baskaryan
DOC: Link to Langchain Expression Language link results in Page not Found
Labels: name: auto:documentation

### Issue with current documentation:

In the **Multiple Retrieval Sources** page (https://python.langchain.com/docs/use_cases/question_answering/how_to/multiple_retrieval)
There is a link to **Langchain Expression Language** (https://python.langchain.com/docs/use_cases/)
Selecting the link results in the following message:

**Page Not Found**
We could not find what you were looking for.

Please contact the owner of the site that linked you to the original URL and let them know their link is broken.

### Idea or request for content:

The link is missing. Broken Link needs to be fixed.
Change kwargs to the proper line
Labels: name: area: vector store, name: auto:bug


   - **Description:**  modified the method a `from_texts` as the` **kwargs` was wrongly set to the self initialization. This produced errors when additional optional parameters were added. Also, they weren't passed to the subsequent add_text method which causes not being able to add optional parameters to reach `pymilvus`. Now you can add any additional parameters like `partition_name` which would make effective inserting in partitions. 
  - **Issue:**  [#10671](https://github.com/langchain-ai/langchain/issues/10671)
  - **Dependencies:** N/A
  - **Tag maintainer:** @hwchase17
Issue:Sitemap Loader filter updated pages
Labels: name: area: doc loader, name: auto:question

### Issue you'd like to raise.

Hi , I want to know how it would be possible to filter with the Sitemap Loader updated pages of 2023 from a website. I do not want to get all the website but only 2023 updated pages. 

Here is my code:
`# fixes a bug with asyncio and jupyter
import nest_asyncio
nest_asyncio.apply()

from langchain.document_loaders.sitemap import SitemapLoader

loader = SitemapLoader(
    "",
    filter_urls=[""]
)
docs = loader.load()`

How can it be possible in it please?
Thank you,

### Suggestion:

_No response_
Aleph Alpha Embeddings: Adding missing parameters check.
Labels: name: area: embeddings, name: auto:bug

Hi,

i stumbled over this error because i didn't read the docs after i updated my version so i thought maybe for the next person that doesn't read the docs.

Description: Check if the parameters `compress to size` and `normalize` are set because if you do not define these in the AlephAlphaAsymmetricSemanticEmbedding explicit it fails with this error:  `ValueError: (400, '{"error":"Json deserialize error: invalid type: null, expected a boolean at line 1 column 2378","code":"PAYLOAD_ERROR"}') `This seems hard to figure out therefore the friendly error message :)

Dependencies: None
Tag maintainer: @hwchase17.

It may be a good idea to write a test for aleph alpha embeddings in the future because i did not find one yet, the only question remains is if you have a token for aleph alpha in your ci that one can use? Then i would offer to write one if that helps.
Aleph ALpha Embeddings:Changing from None to the defaults because otherwise the Request to A…
Labels: name: area: embeddings, name: auto:bug

Changing from None to the defaults because otherwise the Request to Aleph Alpha will fail due to None instead of boolean.


  - **Description:** Change of the parameters `compress to size` and `normalize` because if you do not define these in the AlephAlphaAsymmetricSemanticEmbedding explicit it fails with this error: ` ValueError: (400, '{"error":"Json deserialize error: invalid type: null, expected a boolean at line 1 column 2378","code":"PAYLOAD_ERROR"}')` This seems hard to figure out therefore i set the settings to the default ones.


  - **Dependencies:** None
  - **Tag maintainer:** @hwchase17.
  - **Twitter handle:** @mfmezger 

It may be a good idea to write a test for aleph alpha embeddings because i did not find one yet, the only question remains is if you have a token for aleph alpha in your ci that one can use?
DOC: <Please write a comprehensive title after the 'DOC: Please update the instructions on the langchain site for istallation llama-cpp-python on Windows with GPU NVidia Support' prefix>
Labels: name: auto:documentation, name: area: models

### Issue with current documentation:

There is now no setup.py file in the llama-cpp-python library

I tried installing using these instructions:

Llama.cpp
https://python.langchain.com/docs/integrations/llms/llamacpp

The installation was done as described in the paragraph:
Windows installation

### Idea or request for content:

_No response_
Make some functions work with Milvus
Labels: name: area: vector store, name: auto:improvement

**Description**
Make some functions work with Milvus:
1. get_ids: Get primary keys by field in the metadata
2. delete: Delete one or more entities by ids
3. upsert: Update/Insert one or more entities

**Issue**
None
**Dependencies**
None
**Tag maintainer:**
@hwchase17 
**Twitter handle:**
None
Custom Agent Doesn't Accept Custom Tool
Labels: name: area: agent, name: auto:bug

### System Info

0.0.287

### Who can help?

People who knows to create custom tools/ toolkits for custom agents.



### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Here is the GetCurrentWeatherTool and I'm trying to give this in the tools:
```
class WeatherInput(BaseModel):
    location: str = Field(..., description="The location for which to get the current weather")


class GetCurrentWeatherTool(BaseTool):
    name = "get_current_weather"
    description = "Get the current weather for a specified location"
    args_schema: Type[WeatherInput] = WeatherInput

    def _run(
        self, 
        location: str,
        run_manager: Optional[CallbackManagerForToolRun] = None
    ) -> str:
        """Use the tool."""
        return get_current_weather(location)

    async def _arun(
        self, 
        location: str,
        run_manager: Optional[AsyncCallbackManagerForToolRun] = None
    ) -> str:
        """Use the tool asynchronously."""
        # You can implement the async version of the method here or raise a NotImplementedError if async is not supported
        raise NotImplementedError("get_current_weather does not support async")

```
Tools:

```
self.tools = [        
            # Tool(
            #     name="DuckDuckSearch",
            #     func=DuckDuckGoSearchRun().run, 
            #     description="useful for when you need to search the web, cheap"
            # ),
            # Tool(
            #     name="Search",
            #     func=self.search.run,
            #     description="useful for when you need to answer questions about current events, use it if DuckDucSearch doesn't give you the answer you need",
            # ), 
            GetCurrentWeatherTool
        ]
self.tool_names = [tool.name for tool in self.tools]
```
But I'm getting this error:
```
    self.tool_names = [tool.name for tool in self.tools]
AttributeError: type object 'GetCurrentWeatherTool' has no attribute 'name'
```

### Expected behavior

I was expecting to use the tool with an example query like _How is the weather in London right now?_ and I thought It would call the actual get_current_weather function.
DOC: parameters listed in the DallEAPIWrapper documentation are incorrect
Labels: name: area: doc loader, name: auto:documentation

### Issue with current documentation:

The parameters listed in the [DallEAPIWrapper documentation](https://api.python.langchain.com/en/latest/utilities/langchain.utilities.dalle_image_generator.DallEAPIWrapper.html) are incorrect for:
- n
- open_api_key
- size

### Idea or request for content:

See [OpenAI API reference](https://platform.openai.com/docs/api-reference/images/create) for the correct parameter definitions
DallEAPIWrapper currently only returns the first image URL, even if `n` parameter is set to >1
Labels: name: auto:improvement, name: area: models

### Feature request

The [`openai.Image.create()` endpoint](https://platform.openai.com/docs/api-reference/images/create) has an `n` parameter that allows you to generate 1-10 images. However, I used the looked at the langchain `DallEAPIWrapper()` and only got one image URL. After looking at its [source code](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/utilities/dalle_image_generator.py), I see that the langchain `DallEAPIWrapper()` only returns the first image URL. This should be to easy to fix.

### Motivation

It would be great to be able to generate multiple images from a single input and API call.

### Your contribution

I can submit a pull request to modify `def _dalle_image_url` method to return all image URLs.
Issue: How to get Pandas agent to return and write full steps to a streamlit chat
Labels: name: area: agent, name: auto:question, name: area: models

### Issue you'd like to raise.

Hey, I am trying to follow various approaches online to create a simple chat interface in streamlit to chat with your pandas dataframe. I always face the issue, that I cant extract the full description of the steps. One example is the following code:

```
from langchain.agents import AgentType
from langchain.agents import create_pandas_dataframe_agent
from langchain.callbacks import StreamlitCallbackHandler
from langchain.chat_models import AzureChatOpenAI

if prompt := st.chat_input(placeholder = " "):
    st.session_state.messages.append({"role": "user", "content": prompt})
    st.chat_message("user").write(prompt)
    llm = AzureChatOpenAI()
    pandas_df_agent = 
create_pandas_dataframe_agent(llm,df,verbose=True,agent_type=AgentType.OPENAI_FUNCTIONS,handle_parsing_errors=True)
    with st.chat_message("assistant"):
            st_cb = StreamlitCallbackHandler(st.container(), expand_new_thoughts=False)
            response = pandas_df_agent.run(st.session_state.messages, callbacks=[st_cb])
            st.session_state.messages.append({"role": "assistant", "content": response})
            st.write(response)
```


However if the steps executed are too long it truncates the message as seen below?
![image](https://github.com/langchain-ai/langchain/assets/141514724/066dd5d2-1c32-4d30-b015-b39705706dba)

Anyone who knows how to fix this, or have seen a solution where its no the case?

### Suggestion:

I assume its something to do with the StreamlitCallbackHandler, outputting to the st.container. Maybe I need to change the properties of that container or the settings in StreamlitCallbackHandler?
Support kwargs in RunnableWithFallbacks
Labels: name: auto:improvement

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Is there a way to pass Dictionary as an Action input when you use Zero Shot Agent?
Labels: name: area: agent, name: auto:bug

### System Info

python==3.8
langchain==217

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I'm testing one of the tools that is being used in the zero shot agent, which is making an external api call and get json response. I'm trying to use LLMRequestsChain, and this chain needs an input as dictionary with query and url keys to make a call.

`inputs = {
                "query": user_message,
                "url": f"{endpoint}/user/{mxUser}/accounts?page=1&records_per_page=10"
  }  
  result = agent_chain.zero_shot_agent_chain().run({"question": inputs})
`
Once the dict has been passed to the tool which is the LLMRequestsChain, It throws InvalidSchema exception saying `No connection adapters were found for '{\n    "query": "what\'s my balance?",\n    "url": "targeted url"\n}'`



### Expected behavior

looks like the dict has been passed to the tool as a string. Is there any workaround or any idea to work with?
iMessageChatLoader doesn't load new text conent , due to ChatHistory schema change 
Labels: name: area: doc loader, name: auto:bug

*I may take a stab at fixing this, but don't have the time now, sharing my research notes*

In my iMessage DB, many of my messages have null ```text```, because the content is now encoded in ```attributedBody```. Here is the research I did in case it helps others

Discussion of the issue: 

https://www.reddit.com/r/osx/comments/uevy32/comment/j5ifbku/?utm_source=share&utm_medium=web2x&context=3

Encoding described in Stack Overflow:

https://stackoverflow.com/questions/75330393/how-can-i-read-the-attributedbody-column-in-macos-imessage-database/75330394#75330394


Code that needs to change: 

https://github.com/langchain-ai/langchain/blob/116cc7998cac563894bdf0037259db767882b55c/libs/langchain/langchain/chat_loaders/imessage.py#L63C41-L63C53