delayedkarma/langchain-issues huggingface dataset:
DocArray as a Retriever
Labels: name: lgtm

## DocArray as a Retriever

[DocArray](https://github.com/docarray/docarray) is an open-source tool for managing your multi-modal data. It offers flexibility to store and search through your data using various document index backends. This PR introduces `DocArrayRetriever` - which works with any available backend and serves as a retriever for Langchain apps. 

Also, I added 2 notebooks:
DocArray Backends - intro to all 5 currently supported backends, how to initialize, index, and use them as a retriever
DocArray Usage - showcasing what additional search parameters you can pass to create versatile retrievers

Example:
```python
from docarray.index import InMemoryExactNNIndex
from docarray import BaseDoc, DocList
from docarray.typing import NdArray
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.retrievers import DocArrayRetriever


# define document schema
class MyDoc(BaseDoc):
    description: str
    description_embedding: NdArray[1536]


embeddings = OpenAIEmbeddings()
# create documents
descriptions = ["description 1", "description 2"]
desc_embeddings = embeddings.embed_documents(texts=descriptions)
docs = DocList[MyDoc](
    [
        MyDoc(description=desc, description_embedding=embedding)
        for desc, embedding in zip(descriptions, desc_embeddings)
    ]
)

# initialize document index with data
db = InMemoryExactNNIndex[MyDoc](docs)

# create a retriever
retriever = DocArrayRetriever(
    index=db,
    embeddings=embeddings,
    search_field="description_embedding",
    content_field="description",
)

# find the relevant document
doc = retriever.get_relevant_documents("action movies")
print(doc)
```

#### Who can review?

@dev2049
Add OctoAI LLM endpoint wrapper and embedding endpoint wrapper
Labels: name: enhancement, name: llms, name: lgtm

This pull request introduces support for [OctoAI's Compute Service](https://octoai.cloud/) embedding endpoints and large language model endpoints into the LangChain project. The changes encompass additions to various files to accommodate OctoAI's API integration.

**Details of the changes:**

**langchain/__init__.py** and **langchain/llms/__init__.py:** Updates to these initialization files allow OctoAI's services to be utilized in the LangChain's workflow. The changes add 'OctoAIEndpoint' as an additional language model option.

**langchain/embeddings/octoai_embeddings.py:** This new file creates a wrapper for OctoAI embedding endpoints. This addition allows users to compute embeddings of documents and queries using OctoAI's service.

**langchain/llms/octoai_endpoint.py**: This file introduces a new class, 'OctoAIEndpoint', which serves as a wrapper for OctoAI's large language model endpoints. It includes methods for interacting with OctoAI's API and getting responses from it.

**tests/integration_tests/embeddings/test_octoai_embeddings.py**: This file introduces tests to ensure the correct functioning of the OctoAI embedding endpoint using the Instructor-Large model endpoint. It includes tests for document and query embedding functionalities.

**tests/integration_tests/llms/test_octoai_endpoint.py**: This file includes tests for OctoAI's LLM wrapper for MPT-7B model endpoint. It covers various scenarios including valid API calls, calls that result in errors, and the saving/loading of language model endpoints.

**docs/modules/models/llms/integrations/octoai.ipynb**: This notebook shows how to use LangChain to interact with `OctoAI` [LLM endpoints](https://octoai.cloud/templates)

Please note that to use the newly added OctoAI's services, users should have the octoai python package installed `pip install octoai`, and the environment variables OCTOAI_API_TOKEN set with the API token and ENDPOINT_URL set with endpoint_url. The API token and endpoint URL can be obtained by cloning a template and creating an API token at https://octoai.cloud.

Thanks in advance for your review!

@hwchase17 @agola11
ValueError: not enough values to unpack (expected 2, got 1)

### Issue you'd like to raise.

I keep getting this error when generating Chroma vectors. Here's my code:
`from langchain.chat_models import ChatOpenAI
from langchain.chains import RetrievalQA
from langchain.document_loaders import GoogleDriveLoader
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter

folder_id = ''
loader = GoogleDriveLoader(folder_id = folder_id, 
                           recursive=False)

docs = loader.load(encoding='utf8')

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=4000, chunk_overlap=0, separators=["", "\n", ","]
)

texts = text_splitter.split_documents(docs)
embeddings = OpenAIEmbeddings(openai_api_key="")
db = Chroma.from_documents(texts, embeddings)
retriever = db.as_retriever()

llm = ChatOpenAI(temperature=0, model_name="davinci")
qa = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=retriever)

while True:
    question = input("> ")
    answer = qa.run(question)
    print(answer)`

### Suggestion:

_No response_
ArxivAPIWrapper 

The documentation says:

>     It limits the Document content by doc_content_chars_max.
>     Set doc_content_chars_max=None if you don't want to limit the content size.

But the claim type of int prevents this to be set as None:
https://github.com/hwchase17/langchain/blob/289e9aeb9d122d689d68b2e77236ce3dfcd606a7/langchain/utilities/arxiv.py#LL41C5-L41C38

> ValidationError: 1 validation error for ArxivAPIWrapper
> doc_content_chars_max
>   none is not an allowed value (type=type_error.none.not_allowed)

Can you change that?

In addition, can you also expose this parameter to the `ArxivLoader`?

Thank you!
Issue : Agent Executor stops due to iteration limit or time limit.

### System Info

python 3.9.13
langchain (0.0.163)

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

from langchain.chat_models import AzureChatOpenAI
from langchain.agents import create_pandas_dataframe_agent
from langchain.agents import create_csv_agent
import os
import openai

os.environ[“OPENAI_API_BASE”] = os.environ[“AZURE_OPENAI_ENDPOINT”] = AZURE_OPENAI_ENDPOINT
os.environ[“OPENAI_API_KEY”] = os.environ[“AZURE_OPENAI_API_KEY”] = AZURE_OPENAI_API_KEY
os.environ[“OPENAI_API_VERSION”] = os.environ[“AZURE_OPENAI_API_VERSION”] = AZURE_OPENAI_API_VERSION
os.environ[“OPENAI_API_TYPE”] = “azure”

df=pd.read_csv(‘data.csv’).fillna(value = 0)

llm = AzureChatOpenAI(deployment_name=“gpt-35-turbo”, model_name=“gpt-35-turbo”,temperature=0)
agent_executor = create_pandas_dataframe_agent(llm=llm,df=df,verbose=True)

response = agent_executor.run(prompt + QUESTION)
print(response)

### Expected behavior

The current execution of agent results in errors like "Agent stopped due to iteration limit or time limit" and "Couldn't parse LLM output".
Also noticed that agent runs gets into multiple loops in identifying the valid tool but fails to identify tool , says as not valid tool and tries another one and ultimately stops with iteration limit or time limit.

Kindly suggest me on advice/suggestions to resolve the same.
Is there any particular tabular data structure that langchain works best with?
Do I need to add any particular kind of tool (As observed it tries multiple tools and says not valid tool)
Any modifications I need to make with the code?
LLM技术和训练问题的微信交流群 

### Issue you'd like to raise.

扫码进群，交流Chinese LLM 模型训练的问题和LLM的技术


![1acaa4cf83de26d556929f1f6ef94ca](https://github.com/hwchase17/langchain/assets/90118245/a10bdb88-aa9c-4438-b77e-9f6fbc7b71e1)
Last chunk output by RecursiveCharacterTextSplitter is often too small to be useful

https://github.com/hwchase17/langchain/blob/18af149e91e62b3ac7728ddea420688d41043734/langchain/text_splitter.py#L420

Because it goes from top to bottom the last chunk has the potential to be of any size and is frequently too small to be useful.

I wrote a new class that is more or less a copy of the original function, but adjusts the output if if the last chunk is too small (less than 75% of chunk_size). It does this by adjusting the chunk size upwards to chunk_size = chunk_sizer + (small sized chunk / (chunks - 1)). This allows the last chunks token count to be distributed across all chunks and the end result is that there are no longer bad (small) chunks.

I'm hesitant to create a PR from this because it's such a large change. I believe the correct course would be to integrate it into the main function, but because it rewrites the merge_splits function it would impact all splitters. It's also quite a bit slower than the original class because it can take a few tries to get to the write size. There are optimizations to be had.
@hwchase17 
https://github.com/ShelbyJenkins/langchain/blob/master/langchain/text_splitter.py#L793
Langchain decorators

Added description of LangChain Decorators ✨ into the integration section

<!--
Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.

Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.

After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
-->

<!-- Remove if not applicable -->


#### Before submitting

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->

#### Who can review?

Tag maintainers/contributors who might be interested:

@hwchase17 

<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @vowelparrot

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
extra semicolon with SQLDatabaseChain when used with Oracle

### System Info

SQLDatabaseChain is appending a semicolon to created SQL. which is not liked by cx_oracle (or Oracle database by implication) - since it appends another semicolon in the end. making two semicolons in effect and causing error 

sqlalchemy.exc.DatabaseError: (cx_Oracle.DatabaseError) ORA-00933: SQL command not properly ended

https://cx-oracle.readthedocs.io/en/latest/user_guide/sql_execution.html

Need a way to control this behavior.

https://python.langchain.com/en/latest/modules/chains/examples/sqlite.html#

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Steps:

from langchain import OpenAI, SQLDatabase, SQLDatabaseChain
import cx_Oracle

db = SQLDatabase.from_uri("oracle://ora-user-name:ora-user-password@ora-host-name:1521/service")
llm = OpenAI(temperature=0, verbose=True)

db_chain = SQLDatabaseChain.from_llm(llm, db, verbose=True)
db_chain.run("How many employees are there?")


sqlalchemy.exc.DatabaseError: (cx_Oracle.DatabaseError) ORA-00933: SQL command not properly ended
[SQL: SELECT COUNT(*) FROM YOUR_TABLE_NAME;]
(Background on this error at: https://sqlalche.me/e/20/4xp6)


### Expected behavior

Need a way to optionally suppress a semicolon being appended. or if it could be avoided in favor of cx_oracle - since it doesnt like a semicolon anyways in SQL
PALChain In-context-learning won't scale to multiple problem types.

### Feature request

Currently the PALChain does in-context learning by passing an in-context learning prompt with 8 maths problem solutions in a single large prompt.

If we want the power of PALChain to increase and solve different problem types, we can't keep on adding more and more problem solutions in a single prompt.

As an alternative, we should store more problem solutions in a list, and only include the most relevant to the query in the prompt.

### Motivation

I would like to enhance PAL to solve many more types of problems that can be solved in Python code, e.g. those described in leetcode.com, maths problems in the UK 11+, GCSE and even "A" level example, etc. etc.

### Your contribution

I am happy to work on a PR for this.

How I see it working:
- long list of maths problems solutions in the existing format (natural language question; python code to print the answer).
- the first time the PALChain is invoked, every one of these is embedded using an embedding model.
- every query that a user submits is also embedded, and the most similar questions are passed into the prompt.

There is a risk that "Most similar" gets distracted by proper nouns.  E.g. if there is a problem solution saying "Jonathan has 5 gold balls, ..." and the user submits a question on a completely different maths topic, e.g. "Jonathan wants to express 3/5 as a percentage."    Jonathan is irrelevant but the embedding similiarity algorithm may flag them as similar.  For that reason I propose that as a first pass, both the problem solution and the user query is first passed through a named entity recognition pass to a LLM, which removed names entity related to people, geographical locations etc and replaces them with generic tokens.
[Bug fix] Fix the chat memory to save the result when output text is nested in json
Labels: name: area: memory, name: auto:bug

<!--
Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.

Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.

After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
-->

I used the`RetrievalQA` chain with the argument `return_source_documents=True` as part of the tools in the agent framework with conversation buffer memory. It turns out at the last step, the output text is still nested in json like `{...,'output':{'result': ..., 'source_documents':...}}` and then it caused the error in the `self.chat_memory.add_ai_message(output_str)` which only expects the output to be only string type. 

So I added another check to parse out the final output text if the `output`(original output key) is still the `Dict` type and apply some heuristics to retrieve the text with the new assumed output key based on the common output keys in common chains.

<!-- Remove if not applicable -->

Fixes # (issue)

#### Before submitting

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->

#### Who can review?
@vowelparrot @hwchase17 

Tag maintainers/contributors who might be interested:

<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @vowelparrot

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
[APIChain] enhance the robustness or url

<!--
Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.

Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.

After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
-->

I used the APIChain sometimes it failed during the intermediate step when generating the api url and calling the `request` function. After some digging, I found the url sometimes includes the space at the beginning, like `%20https://...api.com` which causes the ` self.requests_wrapper.get` internal function to fail.

Including a little string preprocessing `.strip` to remove the space seems to improve the robustness of the APIchain to make sure it can send the request and retrieve the API result more reliably. 

<!-- Remove if not applicable -->

Fixes # (issue)

#### Before submitting

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->

#### Who can review?
@vowelparrot
Tag maintainers/contributors who might be interested:

<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @vowelparrot

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
Update databricks.md

HuggingFace -> Hugging Face


#### Before submitting

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->

#### Who can review?
Raise an exception in MRKL if no tool is provided.
Labels: name: lgtm

<!--
Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.

Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.

After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
-->

Implemented validation for cases where no tools are provided. 
This change allows for the handling of scenarios where certain MRKL tools, such as Zapier Natural Language Actions, may not support any tools. With this validation, we prevent unnecessary chaining of API usage. 

This pull request is partially related to issue #5977.


#### Who can review?

@vowelparrot
Issue with retrieving the chunk number from similarity search in lang chain

### Issue you'd like to raise.

Description:
I am currently using the lang chain repository for my project, specifically the functionality related to similarity search. I have encountered an issue when attempting to retrieve the chunk number from the search results.

To provide some context, I have already performed the necessary steps to generate embeddings for each chunk using the provided functions in lang chain. Here is the relevant code snippet:
```
embeddings = OpenAIEmbeddings()
knowledge_base = FAISS.from_texts(chunks, embeddings)
```

After creating the knowledge base, I utilize the similarity_search function to find the most similar chunk to a given query:


`docs = knowledge_base.similarity_search(query)`


The docs object returned contains information about the search results, but I am struggling to access the specific chunk number associated with the most similar result.

My question is: Is there a method or property available in lang chain that allows me to retrieve the chunk number from the docs object?

I would greatly appreciate any assistance or guidance in resolving this issue. Thank you for your support!


### Suggestion:

I could not able to find any documentation, Once I know how to do it I can add it to the documentation
Mitigate issue #5872 (Prompt injection -> RCE in PAL chain)
Labels: name: auto:bug

Adds some selective security controls to the PAL chain:
1. Prevent imports
2. Prevent arbitrary execution commands
3. Enforce execution time limit (prevents DOS and long sessions where the flow is hijacked like remote shell)
4. Enforce the existence of the solution expression in the code

This is done mostly by static analysis of the code using the ast library.

Also added tests to the pal chain.

Fixes #5872 

@vowelparrot