delayedkarma/langchain-issues huggingface dataset:
Implementing search threshold on PGVector

### Issue you'd like to raise.

Implementing _similarity_search_with_relevance_scores on PGVector so users can set "search_type" to "similarity_score_threshold" without raising NotImplementedError.
```
retriever = pgvector.as_retriever(search_type="similarity_score_threshold", search_kwargs={"score_threshold": 0.7}) 
results = retriever.get_relevant_documents(query)
```

Using the search threshold on PGVector to avoid unrelated documents in the results.


### Suggestion:

_No response_
Async calls for HuggingFaceHub
Labels: name: auto:enhancement

### Feature request

It would be nice if HuggingFaceHub models could be called in async mode (concurrently), as currently supported by Anthropic and OpenAI models.

### Motivation

I wanted to compare performance of some models in a bunch of tasks. I was comparing Anthropic and OpenAI models, and when I tried using a HuggingFaceHub model, I noticed this functionality is not implemented.

### Your contribution

I would like to know if someone is already taking on this. I could try to replicate code from other APIs to do it, but I am not sure I would be able to get to a level that is high enough to submit to LangChain.
Adding code to allow pandas to show all columns instead of truncating…
Labels: name: area: agent, name: auto:improvement

  - Description: Adding code to set pandas dataframe to display all the columns.  Otherwise, some data get truncated (it puts a "..." in the middle and just shows the first 4 and last 4 columns) and the LLM doesn't realize it isn't getting the full data.  Default value is 8, so this helps Dataframes larger than that.
  - Issue: none
  - Dependencies: none
  - Tag maintainer: @hinthornw 
  - Twitter handle: none
docs(custom_chain) fix typo
Labels: name: lgtm, name: auto:documentation

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!
 

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Fix typo in the document of custom_chain
CSV Agent JSON Parsing Errors
Labels: name: area: agent, name: auto:bug

### System Info

Occasional error out of CSV agent with JSON parsing error.  Typically occurs when prompting a multi step task, but some multi step tasks are handled fine.  Even in the same multi step task, depending on the wording of the prompt it can be run successfully but with different wording will error out.

Heres an example of the message returned.


File "C:\Users\\env\Lib\site-packages\langchain\agents\openai_functions_agent\base.py", line 212, in plan
    agent_decision = _parse_ai_message(predicted_message)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\env\Lib\site-packages\langchain\agents\openai_functions_agent\base.py", line 114, in _parse_ai_message
    raise OutputParserException(
langchain.schema.OutputParserException: Could not parse tool input: {'name': 'python', 'arguments': "df_filtered = df[df['Version1Text'].str.contains('using your budget')]\nlabel_counts = df_filtered['Label'].value_counts()\nlabel_counts"} because the `arguments` is not valid JSON.

### Who can help?

@hwchase17 @agol

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Steps to reproduce behavior:

1. Start up CSV agent
2. One example prompt that errors: "of the rows where 'Version1Text' includes 'using your budget' what are the counts of each of the unique 'Label' values"

### Expected behavior

Expected behavior is to subset the csv based on the provided conditions and then return counts
Issue: OpenAI Function Agent with custom prompt from memory
Labels: name: area: memory, name: area: agent, name: auto:question, name: auto:improvement

### Issue you'd like to raise.

Hi,

I'm updating my code to use the new OpenAI function calling structure.

Requirements:

- New messages saved in DynamoDB together with past messages for a user
- Custom prompt : 10 last messages from DynamoDB memory of the user
- Function calling

### Past code
The `create_prompt_from_messages(n)` function create a custom prompt based on n last messages.

```python
chain = LLMChain(llm=llm,
                 prompt=create_prompt_from_messages(10),
                 verbose=False,
                 memory=memory
                 )
```

### New code without custom prompt
This code below works but send **all** past messages to the LLM. I want to limit to **n** last messages. I didn't find a way to pass custom prompt to an agent using `AgentType.OPENAI_FUNCTIONS`

Note that I don't want to delete past messages from the database.

```python
message_history = DynamoDBChatMessageHistory(table_name=conversation_table_name, session_id='0')
memory = ConversationBufferMemory(memory_key="chat_history", chat_memory=message_history, return_messages=True)

agent_kwargs = {
    "extra_prompt_messages": [MessagesPlaceholder(variable_name="chat_history")],
}

agent = initialize_agent(
    tools,
    llm,
    agent=AgentType.OPENAI_FUNCTIONS, # or OPENAI_MULTI_FUNCTIONS ?
    verbose=True,
    agent_kwargs=agent_kwargs,
    memory=memory
)
```

How can I only sent n last message from the agent memory? Or create the a custom prompt and pass it to the OpenAI function agent?

Thank you in advance!

### Suggestion:

Create a Notebook with this use case for OpenAI Function agent.
Pydantic exception when creating GPT4All model
Labels: name: area: embeddings, name: auto:bug

### System Info

Mac OS

Versions:
Python 3.8.15

Package               |  Version
-----------------------|--------
aiohttp                | 3.8.4
aiosignal              | 1.3.1
async-timeout      |     4.0.2
attrs                   |23.1.0
certifi                 |2023.5.7
charset-normalizer |     3.2.0
dataclasses-json       | 0.5.9
frozenlist          |    1.4.0
**gpt4all**           |      **1.0.5**
greenlet              |  2.0.2
idna                   | 3.4
**langchain**      |         **0.0.234**
langsmith          |     0.0.5
marshmallow      |       3.19.0
marshmallow-enum |       1.5.1
multidict             |  6.0.4
mypy-extensions   |      1.0.0
numexpr             |    2.8.4
numpy                  | 1.24.4
openapi-schema-pydantic | 1.2.4
packaging           |    23.1
pip                  |   23.2
**pydantic**     |           **1.10.11**
PyYAML           |       6.0
requests            |    2.31.0
setuptools           |   56.0.0
SQLAlchemy          |    2.0.19
tenacity           |     8.2.2
tqdm                  |  4.65.0
typing_extensions  |     4.7.1
typing-inspect      |    0.9.0
urllib3             |    2.0.3
yarl                 |   1.9.2

### Who can help?

@agola11 @hwchase17 

### Information

- [X] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

This is a minimum code example to reproduce the error:

```python
from langchain.llms.gpt4all import GPT4All

llm = GPT4All(model="./models/gpt4all-lora-quantized-ggml.bin")
```

I get the following error:
```
Traceback (most recent call last):
  File "gpt4all_me.py", line 3, in <module>
    llm = GPT4All(model="./models/gpt4all-lora-quantized-ggml.bin")
  File "/home/cserpell/git/activelooplangchain/a/lib/python3.8/site-packages/langchain/load/serializable.py", line 74, in __init__
    super().__init__(**kwargs)
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for GPT4All
__root__ -> __root__
  __init__() takes 1 positional argument but 2 were given (type=type_error)
```

I tried giving the directory without the `./`, without the `./model/`, putting the file in the current directory, and some other options, with no success.

### Expected behavior

The exception should not be raised, and GPT4All model should be available to use.
simplified namespace
Labels: name: area: agent, name: auto:improvement

label: documentation

Simplified namespace to reduce a column length in the API Reference table for the `Agents`.
Right now the `agents.agent_toolkits.azure_cognitive_services.toolkit.AzureCognitiveServicesToolkit` name makes the first column [unbalanced with the second column](https://api.python.langchain.com/en/latest/api_reference.html#module-langchain.agents). The first column is too wide and the second column is too narrow, which wraps the class descriptions in several lines.
This update reduces the ^ namespace to `agents.agent_toolkits.azure_cognitive_services.AzureCognitiveServicesToolkit` and balances the columns.

@baskaryan
@hwchase17
Gpapp/chromadb
Labels: name: area: vector store, name: auto:improvement

  - Description: version check to make sure chromadb >=0.4.0 does not throw an error, and uses the default sqlite persistence engine when the directory is set,
  - Issue: the issue #7887 

For attention of
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
Amadeus Flight and Travel Search Tool
Labels: name: area: agent, name: lgtm, name: auto:enhancement

## Background
With the addition on email and calendar tools, LangChain is continuing to complete its functionality to automate business processes.

## Challenge
One of the pieces of business functionality that LangChain currently doesn't have is the ability to search for flights and travel in order to book business travel.

## Changes
This PR implements an integration with the [Amadeus](https://developers.amadeus.com/) travel search API for LangChain, enabling seamless search for flights with a single authentication process.

## Who can review?
@hinthornw

## Appendix
@tsolakoua and @minjikarin, I utilized your [amadeus-python](https://github.com/amadeus4dev/amadeus-python) library extensively. Given the rising popularity of LangChain and similar AI frameworks, the convergence of libraries like amadeus-python and tools like this one is likely. So, I wanted to keep you updated on our progress.


<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
SelfQuery support for deeplake
Labels: name: lgtm, name: area: vector store, name: auto:improvement

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->

Added support SelfQuery for Deeplake
ChromaDB 0.4+ is no longer compatible with client config
Labels: name: area: vector store, name: auto:bug

### System Info

- Python 3.9.13

- langchain-0.0.235-py3-none-any.whl
- chromadb-0.4.0-py3-none-any.whl

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [x] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Steps to reproduce:

1. Create a Chroma store which is locally persisted
```
store = Chroma.from_texts(
    texts=docs, embedding=embeddings, metadatas=metadatas, persist_directory=environ["DB_DIR"]
)
```
2. Get the error `You are using a deprecated configuration of Chroma. Please pip install chroma-migrate and run `chroma-migrate` to upgrade your configuration. See https://docs.trychroma.com/migration for more information or join our discord at https://discord.gg/8g5FESbj for help!`
3. Suffer

### Expected behavior

1. Create locally persisted Chroma store
2. Use Chroma store


The issue:

Starting chromadb 0.40 the chroma_db_impl is no longer a supported parameter, it uses sqlite instead.

Removing the line
`
chroma_db_impl="duckdb+parquet",
`
from langchain.vectorstores/chroma.py solves the issue, but the earlier DB cannot be used or migrated.
Difference between ConversationChain and ConversationalRetrievalChain
Labels: name: area: agent, name: auto:question

### Issue you'd like to raise.

what is the difference between a conversationChain and a conversationalRetrieval chain. I had originially assumed that the conversational retrieval chain would be able to take in documents, input, and memory (which I have gotten to successfully work) and was under the assumption that the conversationChain could not take in our own documents. However, I found a demo online that suggests otherwise. So what exactly is the difference? It was natural to me to use conversationalretrieval chain becuase I had personal documents I wanted to use and I knew that the retrieval chains were made for that. 

### Suggestion:

_No response_
Inconsistent behaviour with the 'python_repl' tool and the ZeroShotAgent
Labels: name: area: agent, name: auto:bug, name: auto:question

### Issue you'd like to raise.

When using the python_repl tool with the ZeroShotAgent, I keep getting the following error: `Observation: SyntaxError('invalid syntax', ('<string>', 2, 1, '%matplotlib inline\n'))
Thought:`
The agent keeps looping over and over since it does not understand the issue with the magic command. 
Is this a known issue? Do we have a fix or ideally a way to enforce some behaviours within the python tool, via a custom prompt? 

### Suggestion:

_No response_
ArangoDB/AQL support for Graph QA Chain
Labels: name: lgtm, name: auto:enhancement

**Description**: Serves as an introduction to LangChain's support for [ArangoDB](https://github.com/arangodb/arangodb), similar to https://github.com/hwchase17/langchain/pull/7165 and https://github.com/hwchase17/langchain/pull/4881

**Issue**: No issue has been created for this feature

**Dependencies**: `python-arango` has been added as an optional dependency via the `CONTRIBUTING.md` guidelines 
 
**Twitter handle**: [at]arangodb

- Integration test has been added
- Notebook has been added: [graph_arangodb_qa.ipynb](https://github.com/amahanna/langchain/blob/master/docs/extras/modules/chains/additional/graph_arangodb_qa.ipynb)

[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/amahanna/langchain/blob/master/docs/extras/modules/chains/additional/graph_arangodb_qa.ipynb)

```
docker run -p 8529:8529 -e ARANGO_ROOT_PASSWORD= arangodb/arangodb
```

```
pip install git+https://github.com/amahanna/langchain.git
```

```python
from arango import ArangoClient

from langchain.chat_models import ChatOpenAI
from langchain.graphs import ArangoGraph
from langchain.chains import ArangoGraphQAChain

db = ArangoClient(hosts="localhost:8529").db(name="_system", username="root", password="", verify=True)

graph = ArangoGraph(db)

chain = ArangoGraphQAChain.from_llm(ChatOpenAI(temperature=0), graph=graph)

chain.run("Is Ned Stark alive?")
```
showing progress or message under process
Labels: name: auto:improvement

### System Info

I am using the following codes to get api call:

            user_query = prompt.format_prompt(user_prompt=input_text)

            user_query_output = chat_model(user_query.to_messages())

I am using Django and since it takes some time to get response the entire page freezes. Is there anyway to show progress bar or at least a message it is under progress?

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

include the below code under views.py:

            user_query = prompt.format_prompt(user_prompt=input_text)


            user_query_output = chat_model(user_query.to_messages())

upon calling api the page freezes which is normal

### Expected behavior

A message or progress bar
openai.error.InvalidRequestError: The completion operation does not work with the specified model, gpt-4. Please choose different model and try again. You can learn more about which models can be used with each operation here: https://go.microsoft.com/fwlink/?linkid=2197993.
Labels: name: area: embeddings, name: auto:bug

### System Info

langchain==0.0.219
python 3.9

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [X] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I am using the gpt-4 model with azureOpenAI using the below code.

```
from langchain.llms import AzureOpenAI
from langchain.chains.question_answering import load_qa_chain
from langchain.embeddings.huggingface import HuggingFaceEmbeddings
from langchain.document_loaders import DirectoryLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import Chroma
import os
import openai
openai.api_type = "azure"
openai.api_base = os.getenv("OPENAI_API_BASE")
openai.api_version = "2023-03-15-preview"
openai.api_key = os.getenv("OPENAI_API_KEY")
DEPLOYMENT_NAME = 'gpt-4 model'

llm = AzureOpenAI(
    openai_api_base=os.getenv("OPENAI_API_BASE"),
    openai_api_version="2023-03-15-preview",
    deployment_name=DEPLOYMENT_NAME,
    openai_api_key=os.getenv("OPENAI_API_KEY"),
    openai_api_type="azure",
)
embeddings = HuggingFaceEmbeddings(model_name='sentence-transformers/all-MiniLM-L6-v2')
query = "sample query"
my_loader = DirectoryLoader('/Data', glob='**/*.pdf')
docs = my_loader.load()
text_split = RecursiveCharacterTextSplitter(chunk_size = 3000, chunk_overlap = 2)
texts = text_split.split_documents(docs)
docsearch = Chroma.from_documents(texts, embeddings, metadatas=[{"source": str(i)} for i in range(len(texts))],persist_directory="./official_db").as_retriever(search_type="similarity")
docs = docsearch.get_relevant_documents(query)
chain = load_qa_chain(llm, chain_type="stuff")
result = chain.run(input_documents=docs, question=query)
```
But it returns the errror 
```
openai.error.InvalidRequestError: The completion operation does not work with the specified model, gpt-4. Please choose different model and try again. You can learn more about which models can be used with each operation here: https://go.microsoft.com/fwlink/?linkid=2197993.
```

**Note: The same code is working well with gpt-3.5 in azureOpenAI llm**

### Expected behavior

It should able to integrate gpt-4 without any issue.
Add Portkey LLMOps integration
Labels: name: lgtm, name: area: embeddings, name: auto:enhancement

Integrating Portkey, which adds production features like caching, tracing, tagging, retries, etc. to langchain apps.

  - Dependencies: None
  - Twitter handle: https://twitter.com/portkeyai
  - test_portkey.py added for tests
  - example notebook added in new utilities folder in modules
  
 Also fixed a bug with OpenAIEmbeddings where headers weren't passing.

cc @baskaryan
Difference between chain() and chain.run()
Labels: name: auto:question

### Issue you'd like to raise.

I came to know that we can do the document question answering in langchain in different ways. One way is - using **load_qa_chain** 

Also it seems there are two ways to use **load_qa_chain**

1) with **run()**
```
from langchain.chains.question_answering import load_qa_chain
chain = load_qa_chain(llm, chain_type="stuff")
result = chain.run(input_documents=docs, question=query)
```

2)without **run**
```
from langchain.chains.question_answering import load_qa_chain
chain = load_qa_chain(llm, chain_type="stuff")
result = chain({"input_documents": docs, "question": query}, return_only_outputs=True)
```

what is the exact difference between these two methods?

### Suggestion:

_No response_