delayedkarma/langchain-issues huggingface dataset:
Incompatibility with latest Anthropic Client
Labels: name: bug, name: breaking

### System Info

Massive client changes as of this pr:
https://github.com/anthropics/anthropic-sdk-python/commit/8d1d6af6527a3ef80b74dc3a466166ab7df057df

Installing the latest anthropic client will cause langchain Anthropic LLM utiltiies to fail.
Looks like api_url no longer exists (they've moved to base_url).

```
   ???
E   pydantic.error_wrappers.ValidationError: 1 validation error for ChatAnthropic
E   __root__
E     __init__() got an unexpected keyword argument 'api_url' (type=type_error)
```

### Who can help?

@hwchase17

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [x] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. Install latest version of anthropic.
2. Attempt to run Anthropic LLM and it will produce a runtime failure about api_url which no longer exists in the Anthropic client initializer.

### Expected behavior

Should work with latest release of Anthropic client.
PowerBI Toolkit additional logs

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Added some additional logs to better be able to troubleshoot and understand the performance of the call to PBI vs the rest of the work.
Time To First Token
Labels: name: needs work, name: needs documentation

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Allow skipping "condense_question_prompt" when using ConversationalRetrievalChain
Labels: name: enhancement, name: question

### Feature request

Add a parameter to ConversationalRetrievalChain to skip the condense question prompt procedure.


### Motivation

Currently, when using ConversationalRetrievalChain (with the from_llm() function), we have to run the input through a LLMChain with a default "condense_question_prompt" which condenses the chat history and the input to make a standalone question out of it.

This could be a good idea, but in my test cases (in French) it performs very poorly when switching between topics. It mixes things up and hallucinates a new question which is nowhere near the original question.

The only solution I can think of is to just allow the actual chat history to be sent in the prompt but we can't skip the condense question prompt part. For now, a workaround could be to provide a custom condense prompt asking the LLM to do nothing but that would call the LLM twice for no reason...

### Your contribution

Since I am not sure why this isn't already an option, I'd assume it's for a good reason so I'm asking here before thinking of implementing anything.
delete api_key requirements
Labels: name: area: vector store, name: auto:nit

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
In memory vector store as python object based on purely numpy 
Labels: name: enhancement

### Feature request

Can someone please create and integrate an in memory vector store based on purely numpy? 
I don't like that ChromaDB downloads to much deps when I just want something really simple.


### Motivation

I saw this https://github.com/jdagdelen/hyperDB and it works so this should be in langchain

### Your contribution

I dont have the time rn but I want to publish this idea
Added filter and delete all option to delete function in Pinecone integration, updated base VectorStore's delete function

### Description:
Updated the delete function in the Pinecone integration to allow for deletion of vectors by specifying a filter condition, and to delete all vectors in a namespace.

Made the ids parameter optional in the delete function in the base VectorStore class and allowed for additional keyword arguments.

Updated the delete function in several classes (Redis, Chroma, Supabase, Deeplake, Elastic, Weaviate, and Cassandra) to match the changes made in the base VectorStore class. This involved making the ids parameter optional and allowing for additional keyword arguments.
I think the SitemapLoader should be as robust with default options as the UnstructuredURLLoader, which has an option continue_on_failure set to true.
Labels: name: enhancement

### System Info

0.0.218
OSX

### Who can help?

@eyurtsev

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. Create or find a sitemap with url(s) that run into an infinite redirect loop.
2. Run `SitemapLoader` with that.
3. Observe it crashing

### Expected behavior

The loader to continue on such errors, or have a config flag to say so, like `UnstructuredURLLoader`'s `continue_on_failure`.
Issue: Unable to store user specific chat history in redis. Using ConversationalRetrievalChain along with ConversationBufferMemory
Labels: name: bug, name: question

### Issue you'd like to raise.

embeddings = OpenAIEmbeddings(model='text-embedding-ada-002',deployment='XXXXX',chunk_size=1)

db = Chroma.from_documents(texts, embeddings)

memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True,chat_memory=ChatMessageHistory(messages=[]))

model = ConversationalRetrievalChain.from_llm(llm=AzureOpenAI(model_name='gpt-35-turbo',deployment_name='XXXXX',temperature=0.7,openai_api_base=os.environ['OPENAI_API_BASE'],openai_api_key=os.environ['OPENAI_API_KEY']), retriever=db.as_retriever(), memory=memory)

I'm using ConversationalRetrievalChain to query from embeddings and also using memory to store chat history.
I've written a Flask api to fetch result from model.
I'm trying to store memory of each user in redis store, so that there wont be any mix and match in chat history.
For that I'm trying with below code, but I'm unable to store memory in redis.
        `r = redis.Redis(host='redis-1XXX.c3XXX.ap-south-1-1.ec2.cloud.redislabs.com', port=17506, db=0, password='XXXX')`
        `memory = {}`
        `memory_key = f"memory_{employee_code}"`
        `messages = r.get(memory_key)`
        `messages = json.loads(messages)`
        `memory[memory_key] = ConversationBufferMemory(memory_key="chat_history", return_messages=True, chat_memory=ChatMessageHistory(messages=messages))`
        `model = ConversationalRetrievalChain.from_llm(llm=AzureOpenAI(model_name='gpt-35-turbo',deployment_name='XXXXXX',temperature=0.7,openai_api_base=os.environ['OPENAI_API_BASE'],openai_api_key=os.environ['OPENAI_API_KEY']), retriever=db.as_retriever(), memory=memory[memory_key])`
        `res=model.run(query)`
        `messages.append({query:res})`
        `r.set(memory_key,json.dumps(messages))`

This is not working. Can anyone help me on this?


Thanks in advance.



### Suggestion:

_No response_
Support named vectors in Qdrant

# Description

This PR makes it possible to use named vectors from Qdrant in Langchain. That was requested multiple times, as people want to reuse externally created collections in Langchain. It doesn't change anything for the existing applications. The changes were covered with some integration tests and included in the docs.

## Example

```python
Qdrant.from_documents(
    docs,
    embeddings,
    location=":memory:",
    collection_name="my_documents",
    vector_name="custom_vector",
)
```

### Issue: #2594 

Tagging @rlancemartin & @eyurtsev. I'd appreciate your review.
Word2vec/Glove/FastText embedding support
Labels: name: enhancement, name: question, name: stale

### Feature request

Is there any way to store the word2vec/glove/fasttext based embeddings in the vector database using langchain
```
pages= "page content"
embeddings = OpenAIEmbeddings()
persist_directory = 'db'
vectordb = Chroma.from_documents(documents=pages, embedding=embeddings, persist_directory=persist_directory)
```
shall use **word2vec/glove/fasttext**  embeddings instead of **OpenAIEmbeddings()** in the above code?
if possible then what is the syntax for that?

### Motivation

For using native embedding formats

### Your contribution

For using native embedding formats like word2vec/glove/fasttext in langchain
add with score option for max marginal relevance
Labels: name: lgtm

### Adding the functionality to return the scores with retrieved documents when using the max marginal relevance
  - Description: Add the method `max_marginal_relevance_search_with_score_by_vector` to the FAISS wrapper. Functionality operates the same as `similarity_search_with_score_by_vector` except for using the max marginal relevance retrieval framework like is used in the `max_marginal_relevance_search_by_vector` method.
  - Dependencies: None
  - Tag maintainer: @rlancemartin @eyurtsev 
  - Twitter handle: @RianDolphin
added example notebook - use custom functions with openai agent

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Increase flexibility of ElasticVectorSearch 
Labels: name: lgtm, name: area: vector store, name: auto:improvement

Hey @rlancemartin, @eyurtsev ,

I did some minimal changes to the `ElasticVectorSearch` client so that it plays better with existing ES indices. 

Main changes are as follows:

1. You can pass the dense vector field name into `_default_script_query`
2. You can pass a custom script query implementation and the respective parameters to `similarity_search_with_score`
3. You can pass functions for building page content and metadata for the resulting `Document`

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  4. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
sqldatabasechain result incomplete
Labels: name: bug, name: question

### Issue you'd like to raise.

db_chain = SQLDatabaseChain(llm=llm, database=db, prompt=PROMPT, verbose=True, return_intermediate_steps=True, top_k=3)
result = db_chain("Make a list of those taking the exam")
result['result']  is incomplete，why ？token ?


### Suggestion:

_No response_
feat: ctransformers support async chain

  - Description: Adding async method for CTransformers 
  - Issue: I've found impossible without this code to run Websockets inside a FastAPI micro service and a CTransformers model.
  - Tag maintainer: Not necessary yet, I don't like to mention directly 
  - Twitter handle: @_semoal
  - 
<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: I've found impossible without this code to run Websockets inside a FastAPI micro service and a CTransformers model.
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Different results when loading Chroma() vs Chroma.from_documents
Labels: name: bug, name: question

### System Info

langchain==0.0.217
python=3.10

### Who can help?

@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Hi! I am working on a 'question answering' use case. I am loading PDF docs which I am storing in the Chroma vectorDB along with the instructor embeddings. I use the following command to do that:
```
vectordb = Chroma.from_documents(documents=texts, 
                                 embedding=embedding,
                                 persist_directory='db')
```
Here I am storing the vectordb on my local machine in the 'db' folder.
When I use this vectordb as retriever and then use RetrievalQA to ask questions I get 'X' answers.

After storing the vectordb on my local, the next time I directly load the db from the directory:
`vectordb = Chroma(persist_directory='db', embedding_function=embedding)`

When I use this vectordb as retriever and then use RetrievalQA to ask the same questions, I get different answers.

I hope I was able to explain the issue properly.

### Expected behavior

My understanding is that I should get the same answer after loading the vectordb from my local. Why am I getting different answers?  Is this an issue with langchain or am I doing this incorrectly? Can you please help me understand?
chat_models: Add support for calling Poe
Labels: name: enhancement, name: llms, name: needs documentation

- Add a ChatModel to call Poe bots
- Dependencies: fastapi-poe (optional project-wide)

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
max_length support for HuggingFaceTextGenInference
Labels: name: bug, name: enhancement

### System Info

TL;DR
The error is reported in the error reproduction section.

Here's a guess at the solution:

HuggingFaceTextGenInference [docs](https://python.langchain.com/docs/modules/model_io/models/llms/integrations/huggingface_textgen_inference) and [code](https://github.com/hwchase17/langchain/blob/master/langchain/llms/huggingface_text_gen_inference.py#L77-L90) don't yet support  [huggingface's native max_length generation kwarg](https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.GenerationConfig.max_length)

I'm guessing  adding max_length below max_new_tokens in places like [here](https://github.com/hwchase17/langchain/blob/master/langchain/llms/huggingface_text_gen_inference.py#L142)
would provide the desired behavior? Ctrl-F for max_length shows other places the addition may be required

### Who can help?

@hwchase17

### Information

- [ ] The official example notebooks/scripts
- [x] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

The  code snippet below works for local models

```
pipe = pipeline("text-generation", model=hf_llm, tokenizer=tokenizer, max_new_tokens=200)
llm = HuggingFacePipeline(pipeline=pipe)

chain = RetrievalQAWithSourcesChain.from_chain_type(
    llm, chain_type="stuff", retriever=db.as_retriever()
)
chain(
    {"question": "What did the president say about Justice Breyer"},
    return_only_outputs=True,
)
```

However, when replacing the llm definition with this snippet

```
  llm = HuggingFaceTextGenInference(
      inference_server_url="http://hf-inference-server:80/",
      max_new_tokens=256,    
      top_k=10,
      top_p=0.95,
      typical_p=0.95,
      temperature=0.01,
      repetition_penalty=1.03,
  )
```
Yields this error 
```
ValidationError: Input validation error: `inputs` tokens + `max_new_tokens` must be <= 1512. Given: 2342 `inputs` 
tokens and 512 `max_new_tokens`
```
The code snippet that fails here works on it's own when used like this

`generated_text = llm("
What is the capital of Hungary?

")`

### Expected behavior

Expecting a text based answer with no error.