delayedkarma/langchain-issues huggingface dataset:
Feature faiss delete
Labels: name: lgtm, name: area: vector store, name: auto:improvement

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: docstore had two main method: add and search, however, dealing with docstore sometimes requires deleting an entry from docstore. So I have added a simple delete method that deletes items from docstore. Additionally, I have added the delete method to faiss vectorstore for the very same reason. 
  - Issue: NA
  - Dependencies: NA
  - Tag maintainer:  @rlancemartin, @eyurtsev
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Add embeddings for LocalAI
Labels: name: area: embeddings, name: auto:enhancement

Description:

This PR adds embeddings for LocalAI ( https://github.com/go-skynet/LocalAI ), a self-hosted OpenAI drop-in replacement. As LocalAI can re-use OpenAI clients it is mostly following the lines of the OpenAI embeddings, however when embedding documents, it just uses string instead of sending tokens as sending tokens is best-effort depending on the model being used in LocalAI. Sending tokens is also tricky as token id's can mismatch with the model - so it's safer to just send strings in this case.

Partly related to: https://github.com/hwchase17/langchain/issues/5256

Dependencies: No new dependencies

Twitter: @mudler_it

Maintainers: @rlancemartin, @eyurtsev, @hwchase17

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
create_sql_aqent not working with `AgentType.OPENAI_FUNCTIONS`
Labels: name: area: agent, name: auto:bug

### System Info

Langchain version: ```langchain = "^0.0.236"````
Python 3.10.10
MacOSx 13.4.1 (c) (22F770820d)

### Who can help?

@hwchase17 @agola11

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Mysql 5.7 db. 

```
toolkit = SQLDatabaseToolkit(db=db, llm=OpenAI(temperature=0))

agent_executor = create_sql_agent(
    llm=ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613"),
    toolkit=toolkit,
    verbose=False,
    agent_type=AgentType.OPENAI_FUNCTIONS,
)

    response = agent_executor.run(question)
```

Error message:

```none is not an allowed value (type=type_error.none.not_allowed)```

Note: It does work with `AgentType.ZERO_SHOT_REACT_DESCRIPTION`



### Expected behavior

I would expect a response from the agent

Observed:

```none is not an allowed value (type=type_error.none.not_allowed)```
Create langflow
Labels: name: auto:enhancement

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
edit prompt format method
Labels: name: auto:improvement

  - Description:  Modify the `PromptTemplate`'s `format` method behavior by temporarily altering the `__repr__` and `__format__` methods of the `Document` class. This will result in only the `page_content` being added to the template when conduct formatting. Refer to the unit test case below for further clarification.
  - Issue: the issue #7967 it fixes
  - Dependencies: No
  - Tag maintainer: @baskaryan @rlancemartin
  - Twitter handle: dayuanjian21687

Below is the unit test cases.

```python
    prompt = PromptTemplate.from_template("This is a {var} test.")
    # if the variable is string
    output = prompt.format(var="good")
    assert output == "This is a good test."

    # if the variable is Document without meta_data
    doc = Document(page_content="good")
    output = prompt.format(var=doc)
    print(output)
    assert output == "This is a 'good' test."

    # if the variable is Document with meta_data
    doc = Document(page_content="good", meta_data={"bar": "baz"})
    output = prompt.format(var=doc)
    assert output == "This is a 'good' test."
    
    # if the variable is Document and it is inside a list
    doc1 = Document(page_content="good1", meta_data={"bar": "baz"})
    doc2 = Document(page_content="good2", meta_data={"bar": "baz"})
    output = prompt.format(var=[doc1, doc2])
    assert output == "This is a ['good1', 'good2'] test."
```
Issue: Using qdrant.as_retriver() to use the filtered data of a collection (filtered on the basis of payload data)
Labels: name: area: vector store, name: auto:improvement

### Issue you'd like to raise.

Unable to use metadata filtering of a collection qdrant and then use it in as_retriever function.

### Suggestion:

Qdrant.from_documents should include a filtering method as well to filer on the collection with payload (metadata)
call directly the prompt of GPT4All Model
Labels: name: area: memory, name: auto:bug

### System Info

Langchain 0.0.238
System: WSL - Ubuntu

### Who can help?

@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [X] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Use the ConversationBufferMemory and Prompt Templates.

if you have the template
```python
"""
Human: Ask question
Assistant:
"""
```

The prompt passed to GPT4AllModel is again templated by gpt4all.py to avoid which results in the following prompt which is passed to the model

```python
"""
### Human Human: Ask question
Assistant:
### Assistant:
"""
```
This leads to incorrect answers where models are trying to talk to them self as human and assistant

if we directly call GPT4AllModel.model.prompt_model_streaming we are giving the prompt the user indented.

### Expected behavior

That the prompt Template is used that the user/developer gives the Chain
Add batch processing and wait time to FAISS from_texts method
Labels: name: area: vector store, name: auto:improvement

Description: This pull request introduces two new optional parameters to the from_texts method in the FAISS class: batch_size and wait_time. When both parameters are provided, the method will process the texts in batches, waiting for a specified amount of time between each batch. This can be useful when dealing with rate-limited APIs such as the OpenAI API. If tqdm is not installed, an ImportError will be raised.

Issue: This change addresses the issue of handling large volumes of text processing when using rate-limited APIs (solution to issue #634).

Dependencies: This change requires the tqdm module.

Tag maintainer: @rlancemartin, @eyurtsev (as this change is related to VectorStores)

Twitter handle: @syusuke7777
Issue: Rewrite SelfQueryRetriever llm prompt for better results
Labels: name: auto:bug, name: auto:question

### Issue you'd like to raise.

How I can change the value of "retriever.llm_chain.prompt" accordingly for specific task to achieve better and accurate results as self Query retrieval isn't working with Open source LLMs it raises following error and there is an issue related to this error that's not yet resolved as well.

OutputParserException: Parsing text

{
    "query": "simon josu",
    "filter": "and(eq(\"from\", \"Simon assistant app Josu\"), geq(\"date_received\", \"2020-12-14T00:00:00Z\"))"
}
raised following error:
Received unrecognized function geq. Valid functions are [<Operator.AND: 'and'>, <Operator.OR: 'or'>, <Operator.NOT: 'not'>, <Comparator.EQ: 'eq'>, <Comparator.GT: 'gt'>, <Comparator.GTE: 'gte'>, <Comparator.LT: 'lt'>, <Comparator.LTE: 'lte'>]

### Suggestion:

_No response_
missing parameter in similarity_search function in azuresearch.py
Labels: name: area: vector store, name: auto:bug

### System Info

at line 250, self.semantic_hybrid_search function only pass two parameters, query and k. it missing "**kwargs "


241     def similarity_search(
242         self, query: str, k: int = 4, **kwargs: Any
243     ) -> List[Document]:
244         search_type = kwargs.get("search_type", self.search_type)
245         if search_type == "similarity":
246             docs = self.vector_search(query, k=k)
247         elif search_type == "hybrid":
248             docs = self.hybrid_search(query, k=k)
249         elif search_type == "semantic_hybrid":
250         #    docs = self.semantic_hybrid_search(query, k=k)
251             docs = self.semantic_hybrid_search(query, k=k ,**kwargs )
252         else:
253             raise ValueError(f"search_type of {search_type} not allowed.")
254         return docs



### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

without passing "**kwargs ", filter won't work 

### Expected behavior

 docs = vector_store.similarity_search(
                query=query,
                k=3,
                search_type="semantic_hybrid",
                searchFields='content_vector',
                filters="mrn eq '04c3a6bcfcc821d7b7ac722713f4bf7c'",
            )

return doc with filtered by filter condition
add HuggingGPT Agent
Labels: name: area: agent, name: auto:enhancement

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->

- Description: add HuggingGPT Agent, see the [paper](https://arxiv.org/abs/2303.17580) and the [original project](https://github.com/microsoft/JARVIS).
- Notebook: add a [notebook](https://github.com/tricktreat/langchain/blob/bc840fe4307690505b3c0b051e0e2d9726777437/docs/extras/use_cases/autonomous_agents/hugginggpt.ipynb) showing its use.
adding full path to .toml files for the COPY in the dev dockerfile
Labels: name: auto:nit

  - Description: in the .devcontainer, docker-compose build is currently failing due to the src paths in the COPY command. This change adds the full path to the pyproject.toml and poetry.toml to allow the build to run.
  - Issue: na
  - Dependencies: none
  - Twitter handle: byronsalty
add tests to redis vectorstore
Labels: name: area: vector store, name: auto:improvement

# What
- Add function to get similarity with score with threshold in Redis vector store.
- Add tests to Redis vector store.

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: Add tests to Redis vector store.
  - Issue: None
  - Dependencies: None
  - Tag maintainer: @rlancemartin, @eyurtsev
  - Twitter handle: @MlopsJ

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
changed to mlflow-ai-gateway in llms/__init__.py
Labels: name: auto:bug


  - Description: In the llms/__init__.py, the key name is wrong for mlflowaigateway. It should be mlflow-ai-gateway 
  - Issue: NA
  - Dependencies: NA
  - Tag maintainer: @hwchase17, @baskaryan
  - Twitter handle: na

Without this fix, when we run the code for mlflowaigateway, we will get error as below

ValueError: Loading mlflow-ai-gateway LLM not supported
add filter to sklearn vector store functions
Labels: name: lgtm, name: area: vector store, name: auto:improvement

# What
- This is to add filter option to sklearn vectore store functions

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: Add filter to sklearn vectore store functions.
  - Issue: None
  - Dependencies: None
  - Tag maintainer: @rlancemartin, @eyurtsev
  - Twitter handle: @MlopsJ

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
add save and load tfidf vectorizer and docs for TFIDFRetriever
Labels: name: lgtm, name: area: vector store, name: auto:improvement

This is to add save_local and load_local to tfidf_vectorizer and docs in tfidf_retriever to make the vectorizer reusable.

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: add save_local and load_local to tfidf_vectorizer and docs in tfidf_retriever
  - Issue: None
  - Dependencies: None
  - Tag maintainer: @rlancemartin, @eyurtsev
  - Twitter handle: @MlopsJ

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Save tfidf vectorizer and docs in TFIDFRetriever
Labels: name: area: vector store, name: auto:improvement

This is to add save_local and load_local to tfidf_vectorizer and docs in tfidf_retriever to make the vectorizer reusable.

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: add save_local and load_local to tfidf_vectorizer and docs in tfidf_retriever
  - Issue: None
  - Dependencies: None
  - Tag maintainer: @baskaryan
  - Twitter handle: @MlopsJ

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Save TFIDF Vectorizer in tfidf_retriever.py
Labels: name: area: vector store, name: auto:improvement

This is to add save_local and load_local to tfidf_vectorizer and docs in tfidf_retriever to make the vectorizer reusable.

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: add save_local and load_local to tfidf_vectorizer and docs in tfidf_retriever
  - Issue: None
  - Dependencies: None
  - Tag maintainer: @baskaryan
  - Twitter handle: @MlopsJ

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
add Hybrid retriever that not require any external service
Labels: name: lgtm, name: auto:enhancement

  - Until now, hybrid search was limited to modules requiring external services, such as Weaviate/Pinecone Hybrid Search. However, I have developed a hybrid retriever that can merge a list of retrievers using the [Reciprocal Rank Fusion](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf) algorithm. This new approach, similar to Weaviate hybrid search, does not require the initialization of any external service.
  - Dependencies: No
  - Tag maintainer: @rlancemartin, @eyurtsev
  - Twitter handle: dayuanjian21687