delayedkarma/langchain-issues huggingface dataset:
GoogleSerperRun._arun() got an unexpected keyword argument 'arg1'
Labels: name: area: agent, name: auto:bug

### Issue you'd like to raise.

llm = OpenAI(temperature=0, model_name = args.ModelName)
system_message = SystemMessage(content=template)
agent_kwargs = {
    "extra_prompt_messages": [MessagesPlaceholder(variable_name="memory")],
    "system_message": system_message,
}

tools = load_tools(["google-serper"], llm=llm)
agent_executor = initialize_agent(
        tools,
        llm,
        agent=AgentType.OPENAI_FUNCTIONS,
        agent_kwargs= agent_kwargs,
        verbose = True,
        memory = memory,
        handle_parsing_errors=True,
    )


I am getting this error GoogleSerperRun._arun() got an unexpected keyword argument 'arg1'.
Can anyone explain why is it so


### Suggestion:

_No response_
Fix for SVM retriever discarding document metadata
Labels: name: lgtm, name: area: doc loader, name: auto:bug

As stated in the title the SVM retriever discarded the metadata of passed in docs. This code fixes that. I also added one unit test that should test that.

Ultimately it's a simple fix, but unfortunately poetry was not correctly installing dependencies in my environment so I was unable to run the unit test and linting (took way longer to try to set up environment than make my change). Would appreciate if someone with the environment set up could quickly try it out. I did test by directly modifying the langchain source code in my miniconda environment where I initially found the issue and it fixed the bug there, so it's not completely untested.
docstrings: document_loaders consistency
Labels: name: area: doc loader, name: auto:documentation

Formatted docstrings from different formats to consistent format, lile:
>Loads processed docs from Docugami.
"Load from `Docugami`."

>Loader that uses Unstructured to load HTML files.
"Load `HTML` files using `Unstructured`."

>Load documents from a directory.
"Load from a directory."
 
- `Load` - no `Loads`
- DocumentLoader always loads Documents, so no more "documents/docs/texts/ etc"
- integrated systems and APIs enclosed in backticks,
feat: Docusaurus Loader
Labels: name: lgtm, name: area: doc loader, name: auto:enhancement

Added a Docusaurus Loader

Issue: #6353

I had to implement this for working with the Ionic documentation, and wanted to open this up as a draft to get some guidance on building this out further. I wasn't sure if having it be a light extension of the SitemapLoader was in the spirit of a proper feature for the library -- but I'm grateful for the opportunities Langchain has given me and I'd love to build this out properly for the sake of the community. 

Any feedback welcome!
DOC: `API Reference` `langchain.agents Functions ` table unreadable
Labels: name: area: agent, name: auto:documentation

### Issue with current documentation:

 `API Reference` `langchain.agents Functions` table unreadable:
![image](https://github.com/langchain-ai/langchain/assets/2256422/ba09a924-51d2-4f69-bfe8-3d8de1a3939a)
The `name` column is soo long, which makes the `description` column unreadable.

It is because of this value:
`agents.agent_toolkits.conversational_retrieval.openai_functions.create_conversational_retrieval_agent`



### Idea or request for content:

Make the namespace [and name] of this function shorter.
Make intent identification in SPARQL QA chain more resilient
Labels: name: auto:improvement, name: area: models

### Feature request

**Make the intent identification (SELECT vs UPDATE) within the SPARQL QA chain more resilient**
As originally described in #7758 and further discussed in #8521, some models struggle with providing an unambiguous response to the intent identification prompt, i.e., they return a sentence that contains both keywords instead of returning exactly one keyword. \
This should be resolvable with a loop that reprompts the model if its response is ambiguous.
The loop should probably be limited to one or two retries.

### Motivation

Make the intent identification within the SPARQL QA chain more resilient, so that it can more easily be used with models other than the OpenAI ones, e.g., StarCoder and NeoGPT3 mentioned in #7758, which are still able to generate sensible SPARQL


### Your contribution

I should be able to create a PR soonish, but if somebody else wants to give this a go I would also be happy to review
zep/zep-python 1.0 compat
Labels: name: area: vector store, name: auto:improvement

Updating Zep's ZepRetriever and ZepChatHistory classes. The former to be compatible with VectorStoreRetrieverMemory. Both for compatibility with zep-python 1.0.

- update zep-python to 1.0.1
- ZepChatHistory, ZepRetriever to zep-python 1.0 compat
- Update ZepRetriever + ZepChatHistory unit tests
LangChain tools as gpt func, always fails at the JSON decode
Labels: name: area: agent, name: auto:bug, name: area: models

### System Info

Python 3.10
LangChain: 0.0.245

### Who can help?

@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I write a 'Python interpreter' tool, as below sample code:

```
class PythonInterpreterTool(BaseTool):
    name = "python_interpreter"
    name_display = "Python Interpreter"
    description = """Give me the Python to execute，it returns the execution result"""

    def _run(self, code: str) -> str:
        with PythonInterpreter() as executor: # type: ignore
            output = executor.run_python_code(code)

            return output

    async def _arun(self, code: str) -> str:
        raise NotImplementedError
```
But, for many times, I always got "Could not parse tool input... because the `arguments` is not valid JSON." error:

Could not parse tool input: {'name': 'python_interpreter', 'arguments': '{\n  "__arg1": "\nimport moviepy.editor as mp\n\nvideo_path = \'https://video-clip.oss.com/upload/40ae1ae0-3799-11ee-9c80-476f00f64986.mp4\'\noutput_path = \'output.mp3\'\n\nvideo = mp.VideoFileClip(video_path)\nvideo.audio.write_audiofile(output_path)\n"\n}'} because the `arguments` is not valid JSON.


### Expected behavior

I write a 'Python interpreter' tool, as below sample code:

```
class PythonInterpreterTool(BaseTool):
    name = "python_interpreter"
    name_display = "Python Interpreter"
    description = """Give me the Python to execute，it returns the execution result"""

    def _run(self, code: str) -> str:
        with PythonInterpreter() as executor: # type: ignore
            output = executor.run_python_code(code)

            return output

    async def _arun(self, code: str) -> str:
        raise NotImplementedError
```
But, for many times, I always got "Could not parse tool input... because the `arguments` is not valid JSON." error:

Could not parse tool input: {'name': 'python_interpreter', 'arguments': '{\n  "__arg1": "\nimport moviepy.editor as mp\n\nvideo_path = \'https://video-clip.oss.com/upload/40ae1ae0-3799-11ee-9c80-476f00f64986.mp4\'\noutput_path = \'output.mp3\'\n\nvideo = mp.VideoFileClip(video_path)\nvideo.audio.write_audiofile(output_path)\n"\n}'} because the `arguments` is not valid JSON.
add basic support for redis cluster server
Labels: name: lgtm, name: area: vector store, name: auto:improvement

This change updates the central utility class to recognize a Redis cluster server after connection and returns an new cluster aware Redis client. The "normal" Redis client would not be able to talk to a cluster node because keys might be stored on other shards of the Redis cluster and therefor not readable or writable.

With this patch clients do not need to know what Redis server it is, they just connect though the same API calls for standalone and cluster server.

There are no dependencies added due to this MR.

Remark - with current redis-py client library (4.6.0) a cluster cannot be used as VectorStore. It can be used for other use-cases. There is a bug / missing feature(?) in the Redis client breaking the VectorStore implementation. I opened an issue at the client library too (redis/redis-py#2888) to fix this. As soon as this is fixed in `redis-py` library it should be usable there too.
fix MRKLOutputParser
Labels: name: area: agent, name: auto:bug

 # Description:

This PR aims to fix the output parser when the agent hallucinates.
Indeed, sometimes the agent outputs an hallucinated **Observation** resulting to errors.
Also, when the agent have found the **Final Answer**, it happens that the agent loops to a new hallucinated question and start the MRKL process again.

Then, this PR fix the `MRKLOutputParser`:
- when agent has hallucinated an observation
- when agent has hallucinated a next question after the final answer

 # Issue:

Multiple issues are about errors returned by the output parser, in my case it fixed many parsing errors.


@baskaryan
Create pydantic v1 namespace, add partial compatibility for pydantic v2
Labels: name: area: agent, name: auto:refactor

First of a few PRs to add full compatibility to both pydantic v1 and v2.

This PR creates pydantic v1 namespace and adds it to sys.modules.

Upcoming changes: 
1. Handle `openapi-schema-pydantic = "^1.2"` and dependent chains/tools
2. bump dependencies to versions that are cross compatible for pydantic or remove them (see below)
3. Add tests to github workflows to test with pydantic v1 and v2

**Dependencies**

From a quick look (could be wrong since was done manually)

**dependencies pinning pydantic below 2** (some of these can be bumped to newer versions are provide cross-compatible code)
anthropic
bentoml
confection
fastapi
langsmith
octoai-sdk
openapi-schema-pydantic
qdrant-client
spacy
steamship
thinc
zep-python

Unpinned

marqo (*)
nomic (*)
xinference(*)
Intermediate answer from STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION received as a final answer
Labels: name: area: memory, name: area: agent, name: auto:bug, name: stale, name: area: models

### System Info

langchain==0.0.260
model = "gpt-3.5-turbo-16k"
temperature = 0.0

### Who can help?

@hwchase17 and @agola11 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [X] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
search_agent = initialize_agent(
        tools=tools,
        llm=llm,
        agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,
        memory = memory,

        agent_kwargs = {
        #"suffix": NEW_SUFFIX,
        "memory_prompts": [chat_history],
        "input_variables": ["input", "agent_scratchpad", "chat_history"]
        },
        #prompt=cf_template,
        #handle_parsing_errors=True,
        #max_iterations=2,
        #max_execution_time=10,
        verbose = True,
        handle_parsing_errors=True,
    )
from pprint import pprint
#planner = load_chat_planner(llm)
#executor = load_agent_executor(llm, tools, verbose=True)
#search_agent = PlanAndExecute(planner=planner, executor=executor, verbose=True)
#print(search_agent.agent.llm_chain.prompt)

search_agent.agent.llm_chain.verbose=True
question = "What are the stock price for Apple"
response =  search_agent.run(input = question)
print(response)
#pprint(search_agent.agent.llm_chain.prompt)
question = "What about Google."
response =  search_agent.run(input = question)
print(response)
#pprint(search_agent.agent.llm_chain.prompt)
question = "Show me news for fires in Greece in the last month."
response =  search_agent.run(input = question)
print(response)
question = "what about Macedonia in the previous week."
response =  search_agent.run(input = question)
print(response)
```

### Expected behavior

For the:
```
question = "What about Google."
response =  search_agent.run(input = question)
```
The Agent response is:
```
Human: What about Google.
AI: To provide the current stock price for Google, I will use the "get_current_stock_price" tool. Please wait a moment.
```
Instead it should execute the tool and get the results similar as Apple which is the first query...
Add `do_sample` to HuggingFaceTextGenInference
Labels: name: auto:question, name: auto:improvement, name: area: models

### Feature request

Many parameters described on the TGI [Swagger](https://huggingface.github.io/text-generation-inference/#/Text%20Generation%20Inference/generate) find their direct equivalent in the corresponding LangChain [API](https://api.python.langchain.com/en/latest/llms/langchain.llms.huggingface_text_gen_inference.HuggingFaceTextGenInference.html?highlight=textgeninference#langchain.llms.huggingface_text_gen_inference.HuggingFaceTextGenInference). 

### Motivation

At least one of those parameters is missing: the `do_sample` parameter which I needed.

### Your contribution

Why is this parameter missing? How hard would it be to add this and can this be circumvented?
Chain for planning using PDDL
Labels: name: auto:enhancement, name: area: models

### Feature request

__A chain for planning using the [Planning Domain Definition Language](https://en.wikipedia.org/wiki/Planning_Domain_Definition_Language) and a dedicated solver__
As described in this [paper by Liu et al.](https://arxiv.org/abs/2304.11477), the LLM can generate the PDDL inputs required by a solver (they used [FastDownward](https://github.com/aibasel/downward), which has PDDL support, see [this info page](https://www.fast-downward.org/PddlSupport)). \
There is an implementation available from the authors, see [llm-pddl](https://github.com/Cranial-XIX/llm-pddl). However, I did not see any license information. \
It would be great having this readily available as a part of langchain. \
I guess implementing this as a chain would make the most sense, since iterations for reformulating the problem might be necessary.

### Motivation

LLMs are limited regarding planning in an optimal way and dedicated solvers require inputs that are tedious to create - putting the pieces together just makes sense

### Your contribution

I could create a PR, but this might take some time
StructuredOutputParser.get_format_instructions with only_json=True doesn't return the closing backticks
Labels: name: auto:bug, name: area: models

### System Info

- langchain version 0.0.262
- Python version 3.10

### Who can help?

Users I've found through blame ([exact commit](https://github.com/langchain-ai/langchain/commit/1d649b127eb10c426f9b9a67cbd1fe6ec8e6befa)):
- @MassimilianoBiancucci 
- @baskaryan 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Just use the function or look into the source code ([this seems to be the exact commit](https://github.com/langchain-ai/langchain/commit/1d649b127eb10c426f9b9a67cbd1fe6ec8e6befa)).

### Expected behavior

Either return the sole JSON or the whole markdown block (including the closing three backticks).
BedrockEmbeddings can't load aws credential profile
Labels: name: area: embeddings, name: auto:bug

### System Info

langchain==0.0.261
python 3.10

------------------

ValidationError                           Traceback (most recent call last)
Cell In[17], line 1
----> 1 embeddings = BedrockEmbeddings(
      2     credentials_profile_name="monisdas-ibm", region_name="us-east-1"
      3 )
      4 vectorstore = Chroma.from_documents(docs, embeddings)

File ~/Documents/moni/knowlege/unstructured/examples/chroma-news-of-the-day/.py310_unstrcutured/lib/python3.10/site-packages/pydantic/main.py:341, in pydantic.main.BaseModel.__init__()

ValidationError: 1 validation error for BedrockEmbeddings
__root__
  Could not load credentials to authenticate with AWS client. Please check that credentials in the specified profile name are valid. (type=value_error)

-------------------

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

try to use BedrockEmbeddings

### Expected behavior

it should work.
feat(llms): support ernie chat
Labels: name: lgtm, name: auto:enhancement, name: area: models

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
Description: support ernie (文心一言) chat model
Related issue: #7990
Dependencies: None
Tag maintainer: @baskaryan
`ArgillaCallbackHandler` to properly use default values for `api_url` and `api_key`
Labels: name: auto:bug, name: area: models

Hi to @agola11 or whoever is reading this! 🤗 

## What's in this PR?

As of the recent PR at #9043, after some testing we've realised that the default values were not being used for `api_key` and `api_url`. Besides that, the default for `api_key` was set to `argilla.apikey`, but since the default values are intended for people using the Argilla Quickstart (easy to run and setup), the defaults should be instead `owner.apikey` if using Argilla 1.11.0 or higher, or `admin.apikey` if using a lower version of Argilla.

Additionally, we've removed the f-string replacements from the docstrings.

P.S. Regarding the Twitter/X mention feel free to do so at either https://twitter.com/argilla_io or https://twitter.com/alvarobartt and https://twitter.com/gabrielmbmb_, or both if applicable, otherwise, just the first Twitter/X handle.
Issue: when using the map_rerank & refine, occur the " list index out of range" (already modify the llm.py file, but only map_reduce can work well)
Labels: name: auto:bug, name: area: models

### Issue you'd like to raise.

I just tried the simple example with the latest version of langchain, but I meet the problem with " list index out of the range", when I modify the chain_type in "RetrievalQA.from_chain_type" function, the error messages are as follows :

map_rerank :
-----------------------------------------------------------------------------------------------------------------------------
File ~\anaconda3\envs\pytorch\Lib\site-packages\langchain\chains\combine_documents\map_rerank.py:194, in MapRerankDocumentsChain._process_results(self, docs, results)
    190 typed_results = cast(List[dict], results)
    191 sorted_res = sorted(
    192     zip(typed_results, docs), key=lambda x: -int(x[0][self.rank_key])
    193 )
--> 194 output, document = sorted_res[0]
    195 extra_info = {}
    196 if self.metadata_keys is not None:
IndexError: list index out of range
-----------------------------------------------------------------------------------------------------------------------------

refine:
-----------------------------------------------------------------------------------------------------------------------------
File ~\anaconda3\envs\pytorch\Lib\site-packages\langchain\chains\combine_documents\refine.py:203, in RefineDocumentsChain._construct_initial_inputs(self, docs, **kwargs)
    200 def _construct_initial_inputs(
    201     self, docs: List[Document], **kwargs: Any
    202 ) -> Dict[str, Any]:
--> 203     base_info = {"page_content": docs[0].page_content}
    204     base_info.update(docs[0].metadata)
    205     document_info = {k: base_info[k] for k in self.document_prompt.input_variables}
IndexError: list index out of range
-----------------------------------------------------------------------------------------------------------------------------



Thanks for your help!

### Suggestion:

_No response_
LLM with empty list of document
Labels: name: auto:bug, name: area: models

### System Info

If, for some reason, the list of document is empty, an exception is sent from llm.py.

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

llm.prep_prompts(input_list=[])

raise exception

### Expected behavior

Continue

See the pull [request #9109](https://github.com/langchain-ai/langchain/pull/9109)
fix: chunk_size passing
Labels: name: area: embeddings, name: auto:bug

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->