delayedkarma/langchain-issues huggingface dataset:
Issue: Making sure `ConversationalRetrievalChain` only answer based on the retriever information
Labels: name: area: memory, name: area: embeddings, name: auto:question, name: area: models

### Issue you'd like to raise.

Hi,

I have been learning LangChain for the last month and I have been struggling in the last week to "_guarantee_" `ConversationalRetrievalChain` only answers based on the knowledge added on embeddings. I don't know if I am missing some LangChain configuration or if it is just a matter of tuning my prompt. I will add my code here (simplified, not the actual one, but I will try to preserve everything important).

```
chat = AzureChatOpenAI(        
        deployment_name="chat",
        model_name="gpt-3.5-turbo",
        openai_api_version=os.getenv('OPENAI_API_VERSION'),
        openai_api_key=os.getenv('OPENAI_API_KEY'),
        openai_api_base=os.getenv('OPENAI_API_BASE'),
        openai_api_type="azure",
        temperature=0
      )
      
embeddings = OpenAIEmbeddings(deployment_id="text-embedding-ada-002", chunk_size=1)

acs = AzureSearch(azure_search_endpoint=os.getenv('AZURE_COGNITIVE_SEARCH_SERVICE_NAME'),
              azure_search_key=os.getenv('AZURE_COGNITIVE_SEARCH_API_KEY'),
              index_name=os.getenv('AZURE_COGNITIVE_SEARCH_INDEX_NAME'),
              embedding_function=embeddings.embed_query)

custom_template = """You work for CompanyX which sells things located in United States.
        If you don't know the answer, just say that you don't. Don't try to make up an answer.
        Base your questions only on the knowledge provided here. Do not use any outside knowledge.
        Given the following chat history and a follow up question, rephrase the follow up question to be a standalone question, in its original language.
        Chat History:
        {chat_history}
        Follow Up Input: {question}
        Standalone question:
      """
      
CUSTOM_QUESTION_PROMPT = PromptTemplate.from_template(custom_template)

memory = ConversationBufferMemory(memory_key="chat_history", input_key="question", return_messages=True)
    
qa = ConversationalRetrievalChain.from_llm(
  llm=chat,
  retriever=acs.as_retriever(),
  condense_question_prompt=CUSTOM_QUESTION_PROMPT,
  memory=memory
)
```

When I ask it something like `qa({"question": "What is an elephant?"})` it still answers it, although it is totally unrelated to the knowledge base added to the AzureSearch via embeddings.

I tried different `condense_question_prompt`, with different results, but nothing near _good_. I've been reading the documentation and API for the last 3 weeks, but nothing else seems to help in this case. I'd appreciate any suggestions.

### Suggestion:

_No response_
FIX: Infer runnable agent single or multi action
Labels: name: area: agent, name: auto:improvement

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
IDE Support - Python Package - langchain.llms
Labels: name: auto:improvement, name: area: models

### Feature request

Add IDE auto-complete to `langchain.llm` module

Currently, IntelliJ-based IDEs (PyCharm) interpret an LLM model to be `typing.Any`. Below is an example for the `GPT4All` package

![image](https://github.com/langchain-ai/langchain/assets/49741340/27d699a7-d8d3-44f0-bb32-96d798a63a37)


### Motivation

Not having auto-complete on a `LLM` class can be a bit frustrating as a Python developer who works on an IntelliJ product. I've been performing a more direct import of the LLM models to handle this instead:

```python
from langchain.llms.gpt4all import GPT4All
```

Adding support for the `langchain.llms` API would improve the developer experience with the top level `langchain.llms` API

### Your contribution

The existing implementation uses a lazy-loading technique implemented in #11237 to speed up imports. Maintaining this performance is important for whatever solution is implemented. I believe this can be achieved with some imports behind an `if TYPE_CHECKING` block. If the below `Proposed Implementation` is acceptable I'd be happy to open a PR to add this functionality. 

<details><summary>Current Implementation</summary>
<p>

 `langchain.llms.__init__.py` (abbreviated)

```python
from typing import Any, Callable, Dict, Type

from langchain.llms.base import BaseLLM


def _import_anthropic() -> Any:
    from langchain.llms.anthropic import Anthropic

    return Anthropic

def _import_gpt4all() -> Any:
    from langchain.llms.gpt4all import GPT4All

    return GPT4All
    
def __getattr__(name: str) -> Any:
    if name == "Anthropic":
        return _import_anthropic()
    elif name == "GPT4All":
        return _import_gpt4all()
    else:
        raise AttributeError(f"Could not find: {name}")
        
__all__ = [
    "Anthropic",
    "GPT4All",
]
```

</p>
</details>

<details><summary>Proposed Implementation</summary>
<p>

 `langchain.llms.__init__.py` (abbreviated)

```python
from typing import Any, Callable, Dict, Type, TYPE_CHECKING

from langchain.llms.base import BaseLLM


if TYPE_CHECKING:
    from langchain.llms.anthropic import Anthropic
    from langchain.llms.gpt4all import GPT4All


def _import_anthropic() -> "Anthropic":
    from langchain.llms.anthropic import Anthropic

    return Anthropic

def _import_gpt4all() -> "GPT4All":
    from langchain.llms.gpt4all import GPT4All

    return GPT4All
    
def __getattr__(name: str) -> "BaseLLM":
    if name == "Anthropic":
        return _import_anthropic()
    elif name == "GPT4All":
        return _import_gpt4all()
    else:
        raise AttributeError(f"Could not find: {name}")
        
__all__ = [
    "Anthropic",
    "GPT4All",
]
```

</p>
</details>

<details><summary>IntelliJ Screenshot</summary>
<p>

Here's a screenshot after implementing the above `Proposed Implementation`

<img width="588" alt="image" src="https://github.com/langchain-ai/langchain/assets/49741340/1647f3b5-238e-4137-8fa4-f30a2234fdb0">

</p>
</details>
Missing required arguments; Expected either ('model' and 'prompt') or ('model', 'prompt' and 'stream')
Labels: name: auto:bug, name: area: models

### System Info

langchain = 0.0.335
openai = 1.2.4
python = 3.11



### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [x] My own modified scripts

### Related Components

- [x] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Modified example code (https://python.langchain.com/docs/integrations/llms/azure_openai) from langchain to access AzureOpenAI inferencing endpoint

```
import os

os.environ["OPENAI_API_TYPE"] = "azure"
os.environ["OPENAI_API_VERSION"] = "2023-05-15"
os.environ["OPENAI_API_BASE"] = "..."
os.environ["OPENAI_API_KEY"] = "..."

# Import Azure OpenAI
from langchain.llms import AzureOpenAI

# Create an instance of Azure OpenAI
# Replace the deployment name with your own
llm = AzureOpenAI(
    deployment_name="td2",
    model_name="text-davinci-002",
)

# Run the LLM
llm("Tell me a joke")
```

I get the following error:
TypeError: Missing required arguments; Expected either ('model' and 'prompt') or ('model', 'prompt' and 'stream') arguments to be given

If I modify the last line as follows:
`llm("Tell me a joke", model="text-davinci-002") `
i get a different error: 
Completions.create() got an unexpected keyword argument 'engine'

It appears to be passing all keywords to the create method, the first of which is 'engine', and it appears that and other kws are being added by the code.

### Expected behavior

I expect the model to return a response, such as is shown in the example.
Persistent client open file handles
Labels: name: area: vector store, name: auto:bug, name: integration: chroma

### System Info

Name: langchain
Version: 0.0.327

Name: chromadb
Version: 0.4.8
Summary: Chroma.

### Who can help?

@hwchase17 I believe the chromadb don't close the file handle during persistence making it difficult to use it on cloud services like Modal Labs. What about adding a close method or similar to make sure this doesn't happen?

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I am using ModalLabs with a simple example:

```
@stub.function(volumes={CHROMA_DIR: stub.volume})
def test_chroma():
    import chromadb
    from importlib.metadata import version
    print("ChromaDB: %s" % version('chromadb'))

    # Initialize ChromaDB client
    client = chromadb.PersistentClient(path=SENTENCE_DIR.as_posix())

    # Create the collection
    neo_collection = client.create_collection(name="neo")
    # Adding raw documents
    neo_collection.add(
         documents=["I know kung fu.", "There is no spoon."], ids=["quote_1", "quote_2"]
     )

    # Counting items in a collection
    item_count = neo_collection.count()
    print(f"Count of items in collection: {item_count}")
    stub.volume.commit()
```


Error:

`grpclib.exceptions.GRPCError: (<Status.FAILED_PRECONDITION: 9>, 'there are open files preventing the operation', None)
`


### Expected behavior

There shouldn't be any open file handles.
RunnableLambda: create afunc instance from func when not provided
Labels: name: lgtm, name: auto:bug, name: size:M

Fixes #13407.

This workaround consists in letting the RunnableLambda create its self.afunc from its self.func when self.afunc is not provided; the change has no dependency.

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
RunnableLambda: returned runnable called synchronously when using ainvoke
Labels: name: auto:bug, name: area: models

### System Info

langchain on master branch

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [X] Async

### Reproduction

```
from langchain.schema.runnable import RunnableLambda

import asyncio


def idchain_sync(__input):
    print(f'sync chain call: {__input}')
    return __input

async def idchain_async(__input):
    print(f'async chain call: {__input}')
    return __input

idchain = RunnableLambda(func=idchain_sync,afunc=idchain_async)




def func(__input):
    return idchain


asyncio.run(RunnableLambda(func).ainvoke('toto'))
#printss 'sync chain call: toto' instead of 'async chain call: toto'                                                                                                                                                  

```


### Expected behavior

LCEL's route can cause chains to be silently run synchronously, while the user uses ainvoke...

When calling a RunnableLambda A returning a chain B with ainvoke, we would expect the new chain B to be called with ainvoke;
However, if the function provided to RunnableLambda A is not async, then the chain B will be called with invoke, silently causing all the rest of the chain to be called synchronously.
Passthrough kwargs in runnable lambda
Labels: name: auto:improvement

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Add llama2-13b-chat-v1 support to `chat_models.BedrockChat`
Labels: name: lgtm, name: auto:improvement, name: area: models

Hi 👋 We are working with Llama2 on Bedrock, and would like to add it to Langchain. We saw a [pull request](https://github.com/langchain-ai/langchain/pull/13322) to add it to the `llm.Bedrock` class, but since it concerns a chat model, we would like to add it to `BedrockChat` as well.

  - **Description:** Add support for Llama2 to `BedrockChat` in `chat_models`  
  - **Issue:** the issue # it fixes (if applicable) [#13316](https://github.com/langchain-ai/langchain/issues/13316)
  - **Dependencies:** any dependencies required for this change `None`
  - **Tag maintainer:** /
  - **Twitter handle:** `@SimonBockaert @WouterDurnez`
WebResearchRetriever error handling in urls with connection error
Labels: name: auto:improvement

  - **Description:** Added a method `fetch_valid_documents` to `WebResearchRetriever` class that will test the connection for every url in `new_urls` and remove those that raise a `ConnectionError`.
  - **Issue:** [Previous PR](https://github.com/langchain-ai/langchain/pull/13353),
  - **Dependencies:** None,
  - **Tag maintainer:** @efriis 

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
Add support for multimodal embeddings from Google Vertex AI
Labels: name: area: embeddings, name: area: vector store, name: auto:enhancement

### Feature request

Currently no support for multi-modal embeddings from VertexAI exists. However, I did stumble upon this experimental implementation of [GoogleVertexAIMultimodalEmbeddings](https://js.langchain.com/docs/modules/data_connection/experimental/multimodal_embeddings/google_vertex_ai) in LangChain for Javascript. Hence, I think this would also be a very nice feature to implement in the Python version of LangChain.


### Motivation

Using multi-modal embeddings could positively affect applications that rely on information of different modalities. One example could be product search in a web catalogue. Since more cloud providers are making [endpoints for multi-modal embeddings](https://cloud.google.com/vertex-ai/docs/generative-ai/embeddings/get-multimodal-embeddings) available, it makes sense to incorporate these into LangChain as well. The embeddings of these endpoints could be stored in vector stores and hence be used in downstream applications that are built using LangChain.

### Your contribution

I can contribute to this feature.
ElasticSearch allow for multiple vector_query_fields than default text & make it a kwarg in search functions 
Labels: name: area: vector store, name: auto:improvement, name: integration: elasticsearch

### Feature request

Elastic supports natively to have multiple DenseVectors in a document, which can be selected during query time / search. The langchain vector search  interface enables to pass additional keyword args. But at the moment, the implementation of _search in the ElasticSearch implementation does not consider the `vector_query_field` variable, which could be passed through the kwargs. Furthermore, there should be a solution, to allow a document to have multiple text fields that get passed as a queryable vector, not just the standard text field.

### Motivation

If you have multiple vector fields in one index, this feature could simplify the query of the right one, like it's natively allowed in Elastic. In the current implementation one would need to add additional vector fields in the metadata and change the `vector_query_field` of ElasticSearchStore the whole class every time before you call the search. This is not a clean solution and I would vote for a more generic and clean solution.

### Your contribution

I could help by implementing this issue, although I need to state that I am not an expert in Elastic. I saw this issue when we tried to use an existing index in Elastic to add and retrieve Documents within the Langchain Framework.
Missing escape in `llms.utils.enforce_stop_tokens()`
Labels: name: auto:bug, name: area: models

### System Info

Not relevant.

### Who can help?

@hwchase17
@agola11


### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Use any LLM relying on stop words with special regex characters.

For example instanciate a `HuggingFacePipeline` LLM instance with [openchat](https://huggingface.co/openchat/openchat_3.5) model. This model uses `
` stop words.

Since the `llms.utils.enforce_stop_tokens()` function doesn't escape the provided stop words strings the `|` chars are interpreted as part of the regex instead of the stop word. So in this case any single `<` chars in the output would trigger the
split.

### Expected behavior

Stop words should be escaped with `re.escape()` so the split only happens on the complete words.
fix: OpenAPISpec.parse_obj unlimited recursion
Labels: name: lgtm, name: auto:bug, name: size:XS

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** fix OpenAPISpec.parse_obj unlimited recursion, when parse openapi spec with field not exist, the  recursion can not stop
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** none,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Make pirate-speak-configurable template not require env vars for alte…
Labels: name: auto:improvement, name: area: models

…rnative LLMs until used

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->