delayedkarma/langchain-issues huggingface dataset:
OperationalError with docsearch
Labels: name: area: embeddings, name: area: doc loader, name: area: vector store, name: auto:bug

### System Info

Running on Colab.

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

from langchain.document_loaders import GoogleDriveLoader
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.text_splitter import CharacterTextSplitter
from langchain import OpenAI
from langchain.document_loaders import PyPDFDirectoryLoader
from langchain.document_loaders import PyPDFLoader
from langchain.chains import RetrievalQA

os.environ["OPENAI_API_KEY"] = 'Your API Key'

loader = GoogleDriveLoader(
    folder_id="Your folder ID",
    recursive=True,
)

docs = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=100, chunk_overlap=0)
split_docs = text_splitter.split_documents(docs)

embeddings = OpenAIEmbeddings()
docsearch = Chroma.from_documents(split_docs, embeddings)


### Expected behavior

Expect to load embedding into Chroma, then create a QA object upon that. The code was able to run without a problem yesterday but encountered an error today
Chroma vectorstore add_texts() method does not return ids when there is a metadatas argument
Labels: name: area: vector store, name: auto:bug

### System Info

latest

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

collection.add_texts(["coucou"], metadatas = [{ 'source' : "here"}])
returns an empty list

### Expected behavior

original add_texts() method : 
https://github.com/hwchase17/langchain/blob/master/langchain/vectorstores/chroma.py#L144

I fixed it by adding the ids_copy at the beginning : 
    
```
...
        if ids is None:
            ids = [str(uuid.uuid1()) for _ in texts]

        ids_copy = ids
...
```
And by returning it :
 ```
        return ids_copy
```
add HuggingGPT Agent
Labels: name: area: agent, name: lgtm, name: auto:enhancement

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->

- Description: add HuggingGPT Agent, see the [paper](https://arxiv.org/abs/2303.17580) and the [original project](https://github.com/microsoft/JARVIS).
- Notebook: add a [notebook](https://github.com/tricktreat/langchain/blob/bc840fe4307690505b3c0b051e0e2d9726777437/docs/extras/use_cases/autonomous_agents/hugginggpt.ipynb) showing its use.
Add metadata and page_content filters of documents in AwaDB
Labels: name: lgtm, name: area: vector store, name: auto:improvement

1. Add the metadata filter of documents.
2. Add the text page_content filter of documents
3. fix the bug of similarity_search_with_score

Improvement and fix bug of AwaDB
Fix the conflict https://github.com/hwchase17/langchain/pull/7840
@rlancemartin @eyurtsev  Thanks!
WeaviateHybridSearchRetriever option to enable scores.
Labels: name: auto:improvement

Description: This PR adds the option to retrieve scores and explanations in the WeaviateHybridSearchRetriever. This feature improves the usability of the retriever by allowing users to understand the scoring logic behind the search results and further refine their search queries.

Issue: This PR is a solution to the issue #7855 
Dependencies: This PR does not introduce any new dependencies.

Tag maintainer: @rlancemartin, @eyurtsev

I have included a unit test for the added feature, ensuring that it retrieves scores and explanations correctly. I have also included an example notebook demonstrating its use.
WeaviateHybridSearchRetriever has no option to retrieve scores and explanations
Labels: name: auto:enhancement

### Feature request

The WeaviateHybridSearchRetriever does not currently have an option to retrieve scores and explanations. The lack of this feature limits the usability of the retriever, as users cannot gain insights into the scoring logic behind the search results. The feature to retrieve scores and explanations, as provided in the Weaviate library, needs to be integrated into the WeaviateHybridSearchRetriever.

Relevant Weaviate library documentation: [Hybrid search](https://weaviate.io/developers/weaviate/search/hybrid)

### Motivation

Having access to scores and explanations is crucial for users who need to understand how each result has been scored by the hybrid search algorithm. Such understanding can help in refining search queries and filtering results that do not meet certain score thresholds. In the absence of this feature, it becomes difficult to optimize search queries and the quality of the search results can be compromised.



### Your contribution

I am ready to contribute to implementing this feature. I propose to enhance the WeaviateHybridSearchRetriever to include the retrieval of score and explainScore properties from the Weaviate library. This will involve adding relevant parameters and methods in the WeaviateHybridSearchRetriever class. I will also ensure the addition of appropriate tests to validate the functionality and correctness of the implementation.
Strange chunks coming out of CharacterTextSplitter starting in version 0.0.226
Labels: name: auto:bug

### System Info

Langchain version 0.0.226
M1 Mac
Python 3.11.3

### Who can help?

@hwchase17 @mmz-001 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
from langchain.text_splitter import CharacterTextSplitter

def main():
    sample_text = "This is a series of short sentences. I want them to be separated at the periods.  Three sentences should be enough."
    text_splitter = CharacterTextSplitter(separator=". ", chunk_size=30, chunk_overlap=0)
    chunks = text_splitter.split_text(sample_text)
    for chunk in chunks:
        print("CHUNK:", chunk)

if __name__ == "__main__":
    main()
```

Output in version 0.0.225:
```
CHUNK: This is a series of short sentences
CHUNK: I want them to be separated at the periods
CHUNK: Three sentences should be enough.
```

Output in version 0.0.226:
```
CHUNK: Thi. i. serie. o. shor
CHUNK: sentences. wan. the. t. b
CHUNK: separate. a. th. periods
CHUNK: Thre. sentence. shoul. b
CHUNK: enough.
```

### Expected behavior

The output seen in version 0.0.225 should be the same in version 0.0.226.

I suspect that the bug is related to the fix for this issue https://github.com/hwchase17/langchain/pull/7263.  We have also noticed that in recent versions, the metadata start_index is always -1 when using create_documents().  Please let me know if I should file a separate issue for this.
propagate RetrievalQA chain callbacks through its own LLMChain and StuffDocumentsChain
Labels: name: auto:improvement

This is another case, similar to #5572 and #7565 where the callbacks are getting dropped during construction of the chains.

tagging @hwchase17 and @agola11 for callbacks propagation

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Error while creating WeaviateHybridSearchRetriever instance
Labels: name: area: vector store, name: auto:bug

### System Info

LangChain version: 0.0.232
Python version: 3.10.8
Platform: Windows 11, VS Code


For following usage of WeaviateHybridSearchRetriever - 
`w_url = os.environ["WEAVIATE_URL"]`
    `api_key_w= weaviate.AuthApiKey(api_key=os.environ["WEAVIATE_API_KEY"])`
    `wclient = weaviate.Client(url=w_url,
                              auth_client_secret=api_key_w,
                              additional_headers={
                                    "X-Openai-Api-Key": os.getenv("OPENAI_API_KEY"),
                               },)`
   `retriever = WeaviateHybridSearchRetriever(wclient, index_name="testindex", text_key="text")`

Getting following error for the line of retriever instance creation - 
`
TypeError: Serializable.__init__() takes 1 positional argument but 2 were given
`
### Who can help?

@hwchase17
@agola11 

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Follow the steps given in the official documentation - https://python.langchain.com/docs/modules/data_connection/retrievers/integrations/weaviate-hybrid

### Expected behavior

Retriever should be able to initiate correctly and should able to return queried documents.
feat: optional post-processing for Unstructured loaders
Labels: name: area: doc loader, name: auto:improvement

### Summary

Adds a post-processing method for Unstructured loaders that allows users to optionally modify or clean extracted elements.

### Testing

```python
from langchain.document_loaders import UnstructuredFileLoader
from unstructured.cleaners.core import clean_extra_whitespace

loader = UnstructuredFileLoader(
    "./example_data/layout-parser-paper.pdf",
    mode="elements",
    post_processors=[clean_extra_whitespace],
)

docs = loader.load()
docs[:5]
```


### Reviewrs
  - @rlancemartin
  - @eyurtsev
  - @hwchase17
fix: SQLDatabaseChain not return sql query even if return_direct=True
Labels: name: auto:bug

**Description:** SQLDatabaseChain not return sql query even if return_direct=True

**Issue:**
for SQLDatabaseChain, although its parameter set to return_direct=True, it still returns results (e.g., USB wire 98, Laptop 90, Desktop 86, ....) instead of sql query (e.g., SELECT name AS "Product Category" FROM category ORDER BY category_id LIMIT 10;)

I found that it's because final_result variable is wrongly set to result variable instead of sql_cmd if self.return_direct = True, which is different from the documentation: python.langchain.com/docs/modules/chains/popular/sqlite

**How to replicate the issue**:
```import os
from langchain import OpenAI, SQLDatabase, SQLDatabaseChain
from langchain.chat_models import ChatOpenAI

os.environ["OPENAI_API_KEY"] = "<your openai api key>"

db = SQLDatabase.from_uri("<path to your database>")
llm = ChatOpenAI(temperature=0, verbose=True, model_name="gpt-3.5-turbo")

question = "What are the top 10 product categories"
raw_sql = db_chain(question)
raw_sql # you will see the output will not be sql query before this bug fix

````

**Tag mantainer:** @hinthornw

**Dependencies:** No change, as it's just a simple fix by replacing a variable

**Supporting Evidence:** 

![image](https://github.com/hwchase17/langchain/assets/64836390/d26098eb-7879-4cce-b4b5-eb2ea90017a4)
DOC: What is the difference between the various QA chains?
Labels: name: area: doc loader, name: auto:improvement, name: auto:documentation

### Issue with current documentation:

I think there is lacking documentation on the multitude of chains regarding QA and retrieval.

There is:

- [retrieval_qa](https://github.com/hwchase17/langchain/tree/master/langchain/chains/retrieval_qa)
- [question_answering](https://github.com/hwchase17/langchain/tree/master/langchain/chains/question_answering)
- [qa_with_sources](https://github.com/hwchase17/langchain/tree/master/langchain/chains/qa_with_sources)
- [conversational_retrieval](https://github.com/hwchase17/langchain/tree/master/langchain/chains/conversational_retrieval)
- [chat_vector_db](https://github.com/hwchase17/langchain/tree/master/langchain/chains/chat_vector_db)
- [question answering](https://github.com/hwchase17/langchain/tree/master/langchain/chains/question_answering)

Which one should be used when? Which are the base chains used by the others etc?

### Idea or request for content:

A structured documentation of the different chains are needed. Maybe some refactoring of the directory structure to group chains.
InvalidVectorQuery error when using AzureSearch with vector db
Labels: name: area: vector store, name: auto:bug

### System Info

langchain==0.0.234, windows 10, azure-identity==1.13.0, Python 3.11.4

### Who can help?

I manage to create an index in Azure Cognitive Search with _id_, _content_, _vector_content_ and _metadata_ fields.
I check that docs and chunks are not nulls.
I'm getting and error when querying the vector store.

docs: 
[azuresearch-langchain-example](https://python.langchain.com/docs/modules/data_connection/vectorstores/integrations/azuresearch)

Any fix for this?
@hwchase17
@agola11
Regards

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction
Embedding is working as I test:

```
# Check that embbding is working
input_text = "This is for demonstration."
outcome = embeddings.embed_query(input_text)

```
When I'm trying to query with:
```
# Perform a similarity search
docs = vector_store.similarity_search(
    query="What did the president say about Ketanji Brown Jackson",
    k=3,
    search_type="similarity",
)
print(docs[0].page_content)
```
Error:

```
HttpResponseError: (InvalidRequestParameter) The 'value' property of the vector query can't be null or an empty array. Make sure to enclose the vector within a "value" property: '{"vector": { "value": [  ] } }'
Parameter name: vector
Code: InvalidRequestParameter
Message: The 'value' property of the vector query can't be null or an empty array. Make sure to enclose the vector within a "value" property: '{"vector": { "value": [  ] } }'
Parameter name: vector
Exception Details:	(InvalidVectorQuery) The 'value' property of the vector query can't be null or an empty array. Make sure to enclose the vector within a "value" property: '{"vector": { "value": [  ] } }'
	Code: InvalidVectorQuery
	Message: The 'value' property of the vector query can't be null or an empty array. Make sure to enclose the vector within a "value" property: '{"vector": { "value": [  ] } }'
```


### Expected behavior

I can't define is is the azure cognitive configuration index that i manually add or a bug in the code. 
Splitting and adding chunks to the vector store (Azure Cognitive Search) all where dont without any warning.
Add CLI command to create a new project
Labels: name: auto:enhancement

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
BaseRetriever: Latest langchain update is breaking the implementation of extended classes
Labels: name: area: vector store, name: auto:bug

### System Info

MacOS: Ventura 13.4
langchain: 0.0.234

### Who can help?

@eyurtsev

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. Extend the class BaseRetriever like the following:

´´´
class DocumentRetrieverExtended(BaseRetriever):
    def __init__(self, retriever, vector_field, text_field, k=3, return_source_documents=False, score_threshold=None, **kwargs):
        self.k = k
        self.vector_field = vector_field
        self.text_field = text_field
        self.return_source_documents = return_source_documents
        self.retriever = retriever
        self.filter = filter
        self.score_threshold = score_threshold
        self.kwargs = kwargs
´´´
2. Provide DocumentRetrieverExtended as retriever object to ConversationalRetrievalChainExtended.from_llm()
3. You will receive the following exception:

```
  File "pydantic/main.py", line 357, in pydantic.main.BaseModel.__setattr__
ValueError: "DocumentRetrieverExtended" object has no field "k"
```

### Expected behavior

Until version 0.0.189 it was working properly. Please consider to fix the bug or update the langchain documentation for providing the new way of implementing it (Honestly I was expecting that new releases should not break what was working previously)
ConversationalRetrievalChain: Add parameter for not invoking self.question_generator.run()
Labels: name: area: agent, name: auto:improvement

### Feature request

ConversationalRetrievalChain is implementing in the **_call()** method the following behavior:

```
if chat_history_str:
    callbacks = _run_manager.get_child()
    new_question = self.question_generator.run(
        question=question, chat_history=chat_history_str, callbacks=callbacks
    )
else:
    new_question = question 
```
Please add the possibility to avoid this step, like the following:

```
qa = ConversationalRetrievalChainExtended.from_llm(
            llm=self.llm,
            retriever=self.retriever,
            combine_docs_chain_kwargs={"prompt": PROMPT},
            return_source_documents=True,
            verbose=True,
            memory=self.memory,
			generate_question=False
        )
```

In order to avoid the generation of a new question

### Motivation

The current behavior is not generic and applicable to all the LLMs. Some Foundation Models require a specific format for the template (e.g.) and this is causing exceptions on generating the question (e.g. multiple times Anthropic is generating empty results). Moreover, most of the people is already formatting the template for having the desired behavior. Please include this feature ASAP

### Your contribution

I've already extended the class. Please let me know if you need the code (is a small change you should apply, strange it is not the standard behavior)
raise AttributeError(name) from None AttributeError: OPENAI_FUNCTIONS
Labels: name: area: agent, name: auto:bug

### System Info

![Screenshot 2023-07-17 184319](https://github.com/hwchase17/langchain/assets/90586668/e454e70a-66da-40bd-b3c7-92d26caeccea)



### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

The above code snippet  is not working when i'm trying to give LLM an input via streamlit or flask. Error occurs when we are trying to use:``` agent_type=AgentType.OPENAI_FUNCTIONS,```. It's Raising the error like as Below:
```
    raise AttributeError(name) from None
AttributeError: OPENAI_FUNCTIONS
```

### Expected behavior

I expect that when we give input from either post request via FLASK or via streamlit input box, it should accept the input and should pass on this input to agent to run it and give the result accordingly,.