delayedkarma/langchain-issues huggingface dataset:
Redis Cache to support ChatModel
Labels: name: auto:enhancement

### Feature request

I'm writing to propose the integration of Redis Cache support for ChatModels within Langchain. Chat functionality is an essential and perhaps the most common scenario in many applications. Enhancing Langchain with Redis Cache support for ChatModels can bring about the following benefits. 

### Motivation

1. Performance Improvement:
Utilizing Redis Cache can significantly reduce the latency, providing users with a faster and more responsive chat experience.
Caching chat messages and related data can lower the database load and enhance overall system performance.

2. Scalability:
Redis Cache can enable Langchain to efficiently manage large-scale real-time chat systems.
The support for various data structures in Redis enables sophisticated caching strategies, accommodating different chat scenarios and requirements.

3. Reliability:
Redis's replication and persistence features can offer a reliable caching layer that ensures data consistency and availability.
It supports various eviction policies that allow for a flexible and efficient utilization of memory.

4. Integration Ease:
Redis is widely used, well-documented, and has client libraries in most programming languages, making integration with Langchain straightforward.
The community around Redis is vibrant and active, which can be beneficial for ongoing support and enhancement.


### Your contribution
I am considering...
ClickHouse VectorStore score_threshold not working.
Labels: name: area: vector store, name: auto:bug, name: auto:improvement

### System Info
* python3.9
* langchain 0.0242

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [x] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```python
m3e: HuggingFaceEmbeddings = SentenceTransformerEmbeddings(model_name='moka-ai/m3e-base')

vectorStore: VectorStore = Clickhouse(embedding=m3e, config=settings)
retriever = vectorStore.as_retriever(search_type='similarity_score_threshold', search_kwargs={'score_threshold': 20.0})
documents = retriever.get_relevant_documents('音乐')
```

retriever invoke `_get_relevant_documents in vectorstores/base.py`, and `docs_and_similarities` has doc and score.

I found `score_threshold` is not used in `similarity_search_with_relevance_scores`.

Such as pgvector, it is implement `similarity_search_with_score`, so retriever invoke 
 chain is `similarity_search_with_relevance_scores(base.py)` -> `_similarity_search_with_relevance_scores(base.py)` -> `similarity_search_with_score(pgvector)`. So, score filter will be working in `similarity_search_with_relevance_scores(base.py)`.

But Clickhouse vectorstore return from `similarity_search_with_relevance_scores` and without score filter because clickhouse implement `similarity_search_with_relevance_scores` from `base.py`.

---

I have two suggestions for this issue.

1. use `score_threshold` in `similarity_search_with_relevance_scores`
```sql

SELECT document, 
    metadata, dist
FROM default.langchain
WHERE dist > {score_threshold}
ORDER BY L2Distance(embedding, [....]) 
    AS dist ASC
LIMIT 4 
```
2. rename function `similarity_search_with_relevance_scores` to `_similarity_search_with_relevance_scores`.
make invoke chain to `similarity_search_with_relevance_scores(base.py)` -> `_similarity_search_with_relevance_scores(clickhouse)`

got data that score less than score_threshold

### Expected behavior

score filtering normally in clickhouse

BTW, I can submit pr if necessary
Use FAISS  {"distance_strategy":"MAX_INNER_PRODUCT"}  Not work!
Labels: name: area: embeddings, name: area: vector store, name: auto:bug

### System Info

python 3.10
langchain 0.0.249

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction



loader = DirectoryLoader("content/zh/knowledge/", show_progress=True,loader_cls=TextLoader)

    raw_documents = loader.load()
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=500,
        chunk_overlap=0,
        separators=['\n\n','\n','。','？','！','|']
    )
    # embeddings = OpenAIEmbeddings(model="text-embedding-ada-002")
    embeddings = HuggingFaceEmbeddings(model_name="moka-ai/m3e-base")
    count = int(len(raw_documents) / 100) +1
    for index in range(count):
        print('当前：%s' % index)
        documents = text_splitter.split_documents(raw_documents[index*100:min((index+1)*100,len(raw_documents))-1])
        vectorstore = FAISS.from_documents(documents=documents, embedding=embeddings,kwargs={"distance_strategy":"MAX_INNER_PRODUCT"})
    # Save vectorstore
    with open("vectorstore.pkl", "wb") as f:
        pickle.dump(vectorstore, f)

### Expected behavior

Use FAISS  kwargs={"distance_strategy":"MAX_INNER_PRODUCT"}  Error! Not work!
<img width="1512" alt="image" src="https://github.com/langchain-ai/langchain/assets/4583537/fee0f9b9-49bc-4713-adb9-de8c9be76606">
LlamaCppEmbeddings Error  `type == GGML_TYPE_F16`
Labels: name: area: embeddings, name: auto:bug

### System Info

```
llama_print_timings:        load time =   127.16 ms
llama_print_timings:      sample time =     0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings: prompt eval time =  1753.93 ms /   237 tokens (    7.40 ms per token,   135.13 tokens per second)
llama_print_timings:        eval time =     0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =  1779.79 ms
GGML_ASSERT: /tmp/pip-install-7c_dsjad/llama-cpp-python_04ef08a182034167b5fee4a62dd2cdc2/vendor/llama.cpp/ggml-cuda.cu:3572: src0->type == GGML_TYPE_F16
中止 (核心已傾印)
```

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [x] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
    vectorstore = Chroma.from_documents(documents=all_splits, embedding=LlamaCppEmbeddings(model_path="/home/mefae1/llm/chinese-alpaca-2-7b/ggml-model-q4_0.bin",
            n_gpu_layers=40,
            n_batch=16,
            n_ctx=2048
        ))
```

### Expected behavior

return vectorstore and not error
Fix Async Retry Event Handling
Labels: name: auto:bug

It fails currently because the event loop is already running.

The `retry` decorator alraedy infers an `AsyncRetrying` handler for coroutines (see [tenacity line](https://github.com/jd/tenacity/blob/aa6f8f0a2428de696b237d1a86bc131c1cdb707a/tenacity/__init__.py#L535)) However before_sleep always gets called synchronously (see [tenacity line](https://github.com/jd/tenacity/blob/aa6f8f0a2428de696b237d1a86bc131c1cdb707a/tenacity/__init__.py#L338)).


Instead, check for a running loop and use that it exists. Of course, it's running an async method synchronously which is not _nice_. Given how important LLMs are, it may make sense to have a task list or something but I'd want to chat with @nfcampos on where that would live.

This PR also fixes the unit tests to check the handler is called and to make sure the async test is run (it looks like it's just been being skipped). It would have failed prior to the proposed fixes but passes now.
Update spec.py
Labels: name: area: agent, name: auto:bug

  - Description: The API doc passed to LLM only included the content of responses but did not include the content of requestBody, causing the agent to be unable to construct the correct request parameters based on the requestBody information. Add two lines of code fixed the bug, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: @hinthornw ,
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!
DOC: <Running Agent as an Iterator>' prefix>
Labels: name: area: doc loader, name: auto:question, name: auto:documentation

### Issue with current documentation:


```
# need to use GPT-4 here as GPT-3.5 does not understand, however hard you insist, that
# it should use the calculator to perform the final calculation
llm = ChatOpenAI(temperature=0, model="gpt-4")
```
This docs say that it need gpt -4，are there other models that can be used？For example, some open source large models LLM。H

### Idea or request for content:

_No response_
📖 docs: compact api reference
Labels: name: auto:documentation

Updated design of the "API Reference" text
Here is an example of the current format:
![image](https://github.com/langchain-ai/langchain/assets/2256422/8727f2ba-1b69-497f-aa07-07f939b6da3b)

It changed to
`langchain.retrievers.ElasticSearchBM25Retriever` format. The same format as it is in the API Reference Toc.

It also resembles code: 
`from langchain.retrievers import ElasticSearchBM25Retriever` (namespace THEN class_name)

Current format is
`ElasticSearchBM25Retriever from langchain.retrievers` (class_name THEN namespace) 

This change is in line with other formats and improves readability.

 @baskaryan
DOC: Agent Usecase Links are broken
Labels: name: area: doc loader, name: auto:documentation

### Issue with current documentation:

The documentation links seems to be broken, the Agents Use-cases links are no longer working and giving "Page Not Found".
Example Links - https://python.langchain.com/docs/use_cases/autonomous_agents/baby_agi_with_agent.html

### Idea or request for content:

_No response_
Deterministic fake embedding model
Labels: name: area: embeddings, name: auto:improvement

### Feature request

An implementation of a `FakeEmbeddingModel` that generates identical vectors given identical input texts.

### Motivation

Currently langchain has a `FakeEmbedding` model that generates a vector of random numbers, that is irrelevant to the content that needs to be embedded. 

This model is pretty useful in e.g., unit tests, because it doesn't need to load any actual models, or connect to the internet. 

However, there is one realistic feature it misses, which makes it "fake" compared to a realistic embedding model -- given input texts are the same, the generated embedding vectors should be the same as well.

This will make unit tests that involve using a `FakeEmbeddingModel` more realistic. .E.g., ou can test if the similarity search makes sense -- searching by the exact identical text from your vector store should give me an exact match, because the embeddings are identical, which gives a distance of 0.

### Your contribution

I can submit a PR for it.

We can use the hash of the text as a seed when generating random numbers.
In this case, if input texts are identical, we should get exactly the same embedding vectors.
GraphQL Execution Tool should support URL/headers as input at runtime
Labels: name: area: agent, name: auto:improvement

### Feature request

Current GraphQL Tool is very static https://python.langchain.com/docs/modules/agents/tools/integrations/graphql, one has to set up graphql url, headers etc at the time of configuring the tool.

This tool should follow [Request*Tool](https://github.com/hwchase17/langchain/blob/2b3da522eb866cb79dfdc9116a7c466b866cb3d0/langchain/tools/requests/tool.py#L53), where individual request information is sent as input to this tool. This way, GraphQL tool will become a generic and could be used to query different endpoints, or with different headers

This issue was wrongly created in the l[angchainjs repo](https://github.com/hwchase17/langchainjs/issues/1713) instead of the python langchain repo (this one).

### Motivation

We have a use case where the URL and headers would change between invocation of this tool.  And these could be very dynamic, every user might have a different "Authorization" header.  So we want to register the tool once to the agent, but override these values at the time of invocation.

### Your contribution

Yes, my team is willing to contribute.

Here is the pull request https://github.com/langchain-ai/langchain/pull/8616
Catch and Filter OpenAI "As an AI model" responses
Labels: name: auto:improvement

### Feature request

When connecting to ChatGPT API, it is possible to get responses by the chatbot as "As an AI model...", which ruins the chatbot experience on your application.

A way to filter this and generate a default response might be better.

### Motivation

For applications using OpenAI API, this would enable filtering those requests.

### Your contribution

Not really
Support parallel upserts in Pinecone
Labels: name: area: vector store, name: auto:improvement

### Feature request

It would be great if there was an explicit way to support parallel upserts in the `Pinecone` vectorstore wrapper. This could either be done by altering the existing `add_texts` method or adding a new "async" equivalent.

### Motivation

The native Pinecone client supports [parallel upsert](https://docs.pinecone.io/docs/insert-data#sending-upserts-in-parallel) operations which can improve throughput. This is technically supported with the current `langchain` vectorstore wrapper using the following combination of arguments `index.add_texts(texts, async_req=True, batch_size=None)`.

However, because `add_texts` [returns](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/vectorstores/pinecone.py#L102-L105) the list of `ids` instead of the response to `self._index.upsert`, we cannot call `get` to suspend the program until all of the responses are available (see code snippet from the Pinecone docs below).

```
# Upsert data with 100 vectors per upsert request asynchronously
# - Create pinecone.Index with pool_threads=30 (limits to 30 simultaneous requests)
# - Pass async_req=True to index.upsert()
with pinecone.Index('example-index', pool_threads=30) as index:
    # Send requests in parallel
    async_results = [
        index.upsert(vectors=ids_vectors_chunk, async_req=True)
        for ids_vectors_chunk in chunks(example_data_generator, batch_size=100)
    ]
    # Wait for and retrieve responses (this raises in case of error)
    [async_result.get() for async_result in async_results]
```

### Your contribution

I can try to put a PR up for this. Although I wonder if there was a deliberate decision to not return the raw response for [`upsert`](https://github.com/pinecone-io/pinecone-python-client/blob/main/pinecone/index.py#L73-L78) in order to have a cleaner abstraction.
Eden AI (LLM + embeddings) integration
Labels: name: area: embeddings, name: auto:enhancement

# Eden AI - Integration

This PR adds [EdenAI](https://edenai.co) as an LLM and Embeddings provider. 
Eden AI is a Meta API regrouping all the best AI APIs in the market (Amazon, Open AI, Cohere, Google, Anthropic, DeepL, AssemblyAI  ...etc) into one. Users can access all the AI tasks you need with a single account giving them total freedom switch between providers and combine them.

This allows langchain users to :
 💪 Increase Accuracy & Reduce Cost by choosing the right AI API for each AI task
 ☮️ Gain peace of mind with centralized usage restrictions & cost monitoring 
 🌐 Continually explore emerging AI capabilities in the market
 
Here is an example of how it's called:

```python
from langchain.llms import EdenAI

llm = EdenAI( edenai_api_key="...",
            provider="openai", 
            params={"temperature" : 0.2,"max_tokens" : 250})
```

It can also be used for text2image : 

```python
imagemodel = EdenAI( feature="image" ,   provider= "stabilityai",  
                                       params={  "resolution" : "512x512"  })

```

 A detailed notebook was added.
 
 No dependencies are added as we call a rest API.
 
**Upcoming** : This PR starts by adding the LLM and Embeddings capabilities and will be followed by a utilities PR to give agents new capabilities like : explicit content detection (image, texts and videos), speech to text and text to speech, pdf parsing (invoice, ID, resume...etc ) and a lot of other tasks (+50) from top AI APIs providers.
 
 
Ping: @hwchase17, @baskaryan
Pubmed needs to be called with "pupo
Labels: name: auto:bug, name: auto:documentation

### System Info

OS:  Windows
Name: langchain
Version: 0.0.249
Python 3.11.2

### Who can help?

@hw

### Information

- [X] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

When loading the pubmed Tool it has to be called with "pupmed".  Though this was possibly a typo since pupmed isn't mentioned in the documents.

### Expected behavior

Expect to call the pubmed tool with "pubmed" and not "pupmed" or documentation should be more clear.
add missing RemoteLangChainRetriever _get_relevant_documents test
Labels: name: lgtm, name: auto:improvement

# What
- Add missing RemoteLangChainRetriever _get_relevant_documents test

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: Add missing RemoteLangChainRetriever _get_relevant_documents test
  - Issue: None
  - Dependencies: None
  - Tag maintainer: @rlancemartin, @eyurtsev
  - Twitter handle: @MLOpsj

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Swap in Sympy instead of numexpr for more expressive LLMMath tool
Labels: name: auto:improvement

  - Description: Swapped in `sympy` instead of `numexpr` in the LLMMath Chain
  - Issue: Relevant to #8363 
  - Dependencies: None
  - Tag maintainer: @hinthornw 

Note that we already use `sympy` in the symbolic math tool, so this actually removes a dependency!

`numexpr` is actually quite fragile to input, whereas `sympify` is really quite hard to break. Take the iconic Langchain tool example mixing information retrieval and numerical calculation: _"Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?"_ (from this [tutorial page](https://python.langchain.com/docs/modules/agents/how_to/intermediate_steps)).

If you make the maths more exotic ("What is the first prime number larger than her age?", "What is the factorial of her age?", etc.) `numexpr` will raise an exception. `sympy` handles these with flying colours.


I used the chain from that example, with the prompt:
```
"In the sky, I am the number of sisters who dance, Bound by myth, in a celestial trance. On Earth, I am the continents that span, Holding diverse life, since time began.
What number am I?

Now multiply that number by the age of Zoe Saldana's partner, and raise it to the 2.43rd power.
Take the fourth digit of the answer, then take the factorial of that, and call it X.
What's the smallest prime number greater than X?
"""
```
and with `sympy`, my chain passed with flying colours :)
Add upsert support for pgvector
Labels: name: area: vector store, name: auto:improvement

## Description
Currently, there is no support for updates or upserts to PGVector through langchain.
This PR adds that support by leveraging `on_conflict_do_upsert`

This fixes https://github.com/langchain-ai/langchain/issues/6866

Maintainers: @rlancemartin, @eyurtsev

Twitter handle: [@lokeshdevnani](https://twitter.com/lokeshdevnani)
How to create a custom retriever
Labels: name: area: doc loader, name: auto:enhancement, name: auto:documentation

### Feature request

When I use RetrievalQA, I need to add and reorder the content retrieved by retriever
```
qa = RetrievalQA.from_chain_type(llm=chat, 
                                 chain_type="stuff",
                                 retriever=docsearch.as_retriever(search_kwargs={"k": 2}), 
                                 chain_type_kwargs={"prompt": PROMPT,"memory":memory })
```
I want the retrieved content from retriever to be reordered by "source".
The following code is used by me to manipulate the retrieved content to achieve my needs.
```
retriever = docsearch.as_retriever(search_kwargs={"k": 6})
search_result = retriever.get_relevant_documents(quey)
def sort_paragraphs(paragraphs):
    sorted_paragraphs = sorted(paragraphs, key=lambda x: x.metadata["source"])
    return sorted_paragraphs
paragraphs = sort_paragraphs(search_result)
sorted_paragraphs = ""
for i in paragraphs:
    sorted_paragraphs = sorted_paragraphs + i.page_content + "\n"
```
How can I define an retriever that I can use in my chain to organize the search content as I wish


### Motivation

Some custom retrieval logic is required to seek optimal performance.

### Your contribution

I would like to have tutorials related to custom retriever documentation
@hwchase17
https://twitter.com/hwchase17/status/1646272240202432512
I think langchain needs to be a custom retriever
Fix Issue #6650: Enable Azure Active Directory token-based auth access for AzureChatOpenAI
Labels: name: lgtm, name: auto:improvement

When using AzureChatOpenAI the openai_api_type defaults to "azure". The utils' get_from_dict_or_env() function triggered by the root validator does not look for user provided values from environment variables OPENAI_API_TYPE, so other values like "azure_ad" are replaced with "azure". This does not allow the use of token-based auth.

By removing the "default" value, this allows environment variables to be pulled at runtime for the openai_api_type and thus enables the other api_types which are expected to work.

This fixes #6650