delayedkarma/langchain-issues huggingface dataset:
core dependencies update
Labels: name: auto:nit, name: size:XS

Updated `pyproject.toml` with missed dependencies.
Several dependencies are missed in the `pyproject.toml`. They are explicitly imported in code.
I listed the files and code lines below.
Those packages are still presented in the `poetry.lock`. So, those missed dependencies do not break anything.
This update is for consistency. Every explicitly imported package should be presented in the dependency configuration.
Note: `poetry.lock` is updated because `anyio` version here (=>4.0 - from poetry.lock) conflicts with `langchain` version (<4.0)

> langchain_core/language_models/llms.py:
> langchain_core/prompts/base.py:
> langchain_core/prompts/loading.py:
> import yaml
> 
> langchain_core/output_parsers/base.py:
> from typing_extensions import get_args
> 
> langchain_core/runnables/base.py:
> from typing_extensions import Literal, get_args
> 
> langchain_core/runnables/config.py:
> langchain_core/runnables/router.py:
> from typing_extensions import TypedDict
> 
> langchain_core/utils/iter.py:
> from typing_extensions import Literal
> 
> langchain_core/tracers/langchain_v1.py:
> langchain_core/utils/loading.py:
> import requests
> 
> langchain_core/utils/utils.py:
> from requests import HTTPError, Response
> 
> langchain_core/tracers/log_stream.py:
> from anyio import create_memory_object_stream
> 
> langchain_core/utils/utils.py:
> from packaging.version import parse
feat(llms): support more tasks in HuggingFaceHub LLM and remove deprecated dep
Labels: name: auto:improvement, name: area: models, name: size:M

- **Description:** this PR upgrades the `HuggingFaceHub` LLM:
   * support more tasks (`translation` and `conversational`)
   * replaced the deprecated `InferenceApi` with `InferenceClient`
   * adjusted the overall logic to use the "recommended" model for each task when no model is provided, and vice-versa.
- **Tag mainter(s)**: @baskaryan @hwchase17
update LanguageModelInput from List to Sequence
Labels: name: lgtm, name: auto:nit, name: area: models, name: size:XS

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->

- **Description:** Update (one of the branches of) the type alias `LanguageModelInput` from `List[BaseMessage]` to `Sequence[BaseMessage]` to disable the error from Pylance's type checker (see below).
- **Issue:** N/A.
- **Dependencies:** No changes.
- **Tag maintainer:** @baskaryan @eyurtsev
- **Twitter handle:** @alan_0127

---

- **The error:**

  ![image](https://github.com/langchain-ai/langchain/assets/70696274/dfbe53ac-fc6c-4051-8e93-f27287909fc5)
db_chain_sql_out.return_sql
Labels: name: auto:bug, name: area: models

### System Info

from langchain.llms import GooglePalm
from sqlalchemy import create_engine
from langchain.utilities import SQLDatabase
from langchain.llms import GooglePalm
from langchain_experimental.sql import SQLDatabaseChain

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

st.write(db_chain_sql_out.return_sql) is returning bool value "True" Instead of the model generated actual SQL Statement - 
I am using Google Palm - is this the normal output?

### Expected behavior

Expecting model generated SQL
Issue: ElasticsearchStore with ssl_verify = {'verify_certs':False} does not work
Labels: name: area: vector store, name: auto:bug, name: integration: elasticsearch

### Issue you'd like to raise.

I am using langchain 0.0.316 an trying to create a ElasticsearchStore to do some similarity_search

However, whenever I try to create it (using from_documents or not) I get the following error : 

`raise ValueError("check_hostname requires server_hostname")`

This is a SSL error, and I suspect it to be the problem as I cannot use SSL. Somewhere else in the project, I cannot with Python to Elastic using verify_certs = False and everything works perfectly.

Thus I tried to create ElasticsearchStore with the following arguments :

```
db = ElasticsearchStore(texts,
                                      es_url='...',
                                      index_name = '...',
                                      embedding,
                                      ssl_verify = {'verify_certs'=False'})
```

But I still have the error and nothing has changed. 
How can I make langchain to initialize Elastic without checking for ssl certificates ? 

### Suggestion:

_No response_
AzureOpenAIEmbeddings cannot authenticate with azure_ad_token_provider
Labels: name: area: embeddings, name: auto:bug

### System Info

Langchain: 0.0.346
OpenAI:  1.3.7

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Simple script to authenticate to Azure with RBAC

```
from langchain.embeddings import AzureOpenAIEmbeddings
from azure.identity import DefaultAzureCredential, get_bearer_token_provider

token_provider = get_bearer_token_provider(DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default")
embeddings = AzureOpenAIEmbeddings(azure_endpoint='xxxxxxx', azure_ad_token_provider=token_provider)
```


### Expected behavior

Should authenticate, but is seems like the `azure_ad_token_provider` is not added to the values dict

langchain/embeddings/azure_openai.py line 80-86
```
values["azure_endpoint"] = values["azure_endpoint"] or os.getenv(
            "AZURE_OPENAI_ENDPOINT"
        )
values["azure_ad_token"] = values["azure_ad_token"] or os.getenv(
            "AZURE_OPENAI_AD_TOKEN"
        )
```
Other parameters are added to values, but not `azure_ad_token_provider`
Issue: <Trouble with langchain Library: Error in LLMChain Validation>
Labels: name: auto:bug, name: auto:question, name: area: models

### Issue you'd like to raise.

Hello community,

I'm currently working on a project that involves using the langchain library for natural language processing. I'm encountering an issue with the LLMChain class, and I'm hoping someone can help me troubleshoot.

I've initialized a Hugging Face pipeline and constructed a prompt using PromptTemplate. However, when I attempt to load a QA chain using the load_qa_chain function, I get a ValidationError related to the Runnable type. The error suggests that an instance of Runnable is expected, but it seems there's a mismatch.

Here's a simplified version of my code:

```
from langchain.prompts import PromptTemplate
from langchain import load_qa_chain
from transformers import HuggingFacePipeline

hf = HuggingFacePipeline.from_model_id(
    model_id="ai-forever/rugpt3large_based_on_gpt2",
    task="text-generation",
    pipeline_kwargs={"max_new_tokens": 10},
)

prompt = """Question: {question}

Answer: {text}"""

# The next line is where the error occurs
chain = load_qa_chain(hf(prompt=prompt), chain_type="stuff")
```

```
ValidationError: 2 validation errors for LLMChain
llm
  instance of Runnable expected (type=type_error.arbitrary_type; expected_arbitrary_type=Runnable)
llm
  instance of Runnable expected (type=type_error.arbitrary_type; expected_arbitrary_type=Runnable)
```

I have checked the documentation and versions of the libraries, but I'm still having trouble understanding and resolving the issue. Could someone please provide guidance on what might be causing this ValidationError and how I can address it?

Thank you in advance for your help!

### Suggestion:

_No response_
value error when using huggingfacehub API
Labels: name: area: doc loader, name: auto:bug, name: area: models

### System Info

Hi Team,

Am trying to connect to SQL/CSV using the HuggingFaceHUb and I get value error. This value error occurs even when I use the same example as given in https://python.langchain.com/docs/use_cases/qa_structured/sql except that instead of openAI am using huggingfaceHub

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

st.title("SQL DB with Langchain")

#entering input through streamlit into the app for querying
input_text = st.text_input("enter the text for search")
#input_period = st.text_input("enter the period for which you need summarization")

#connecting to hugging face API
os.environ["HUGGINGFACEHUB_API_TOKEN"] = huggingface_write_key

#SQL
db = SQLDatabase.from_uri("sqlite:///Chinook.db")
repo_id = "google/flan-t5-xxl"
llm = HuggingFaceHub(
    repo_id=repo_id, model_kwargs={"temperature": 0.2}
)
db_chain = SQLDatabaseChain.from_llm(llm, db, verbose=True)

if input_text:
	st.write(db_chain.run(input_text))

### Expected behavior

Expected to give output for the query thats run
Issue:python3.10/site-packages/langchain/llms/bedrock.py:315: RuntimeWarning: coroutine 'AsyncCallbackManagerForLLMRun.on_llm_new_token' was never awaited
Labels: name: auto:bug, name: area: models

### Issue you'd like to raise.

python3.10/site-packages/langchain/llms/bedrock.py:315: RuntimeWarning: coroutine 'AsyncCallbackManagerForLLMRun.on_llm_new_token' was never awaited


### Suggestion:

_No response_
Fix Bedrock broad error catching
Labels: name: lgtm, name: auto:bug, name: area: models, name: size:XS

Fixes #14347 

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** Added the traceback of the previous error to keep the initial error type, 
  - **Issue:** #14347 ,
  - **Dependencies:** None,
  - **Tag maintainer:** 

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Issue: How to delete particular space embeddings for a confluence projects
Labels: name: area: embeddings, name: area: doc loader, name: auto:question, name: integration: chroma

### Issue you'd like to raise.


def delete_confluence_embeddings(file_path, persist_directory,not_uuid):
    chroma_db = chromadb.PersistentClient(path=persist_directory)
    collection = chroma_db.get_or_create_collection(name="langchain")
    project_instance = ProjectName.objects.get(not_uuid=not_uuid)
    confluence_data = json.loads(project_instance.media).get('confluence', [])
    confluence_url = project_instance.url
    username = project_instance.confluence_username
    api_key = base64.b64decode(project_instance.api_key).decode('utf-8')
    space_keys = [space_data['space_key'] for space_data in project_instance.space_key]

    documents = []
    loader = ConfluenceLoader(
        url=confluence_url,
        username=username,
        api_key=api_key
    )

    for space_key in space_keys:
        documents.extend(loader.load(space_key=space_key, limit=100))
    page_info = []
    for document in documents:
        page_id = document.metadata.get('id')
        page_title = document.metadata.get('title')

        formatted_title = page_title.replace(' ', '+')
        page_info.append({"id": page_id, "title": formatted_title})
        # print(f"Page ID: {page_id}, Page Title: {formatted_title}")
    for entry in page_info:
        entry_file_path = f"{file_path}/pages/{entry['id']}"
        ids = collection.get(where={"source": entry_file_path})['ids']
        collection.delete(where={"source": entry_file_path}, ids=ids)

    for entry in page_info:
        entry_file_path = f"{file_path}/pages/{entry['id']}/{entry['title']}"
        ids = collection.get(where={"source": entry_file_path})['ids']
        collection.delete(where={"source": entry_file_path}, ids=ids)
    chroma_db.delete_collection(name="langchain")
    print("Delete successfully")
    how can i delete for particular space.
    file path is url/spaces/space_key/

### Suggestion:

_No response_
Retrieval QA chain does not work 
Labels: name: auto:bug, name: area: models

### System Info

when i am using Retrieval QA with custom prompt on official llama2 model it gives back an empty result even though retriever has worked but LLM failed to give back the response but if i directly pass the query to chain without any prompt it works as expected 

## Versions 
Python - 3.10
Langchain - 0.0.306

@hwchase17 and @agola11 please take a loot at this issue 

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

`qa_chain = RetrievalQA.from_chain_type(
                llm=llm,
                chain_type="stuff",
                retriever=final_retriever,
                chain_type_kwargs={"prompt": prompt_template},
                return_source_documents=True
            )`

if i initialize the chain like this it is failing without 

### Expected behavior

`qa_chain = RetrievalQA.from_chain_type(
                llm=llm,
                chain_type="stuff",
                retriever=final_retriever,
                return_source_documents=True
            )`

if i initialize the chain like this it is working as expected
AttributeError: module 'openai' has no attribute 'error'
Labels: name: auto:bug, name: area: models

### System Info

Google colab 

### Who can help?

@agola

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
!pip install lmql==0.0.6.6 langchain==0.0.316 openai==0.28.1 -q

import lmql
import aiohttp
import os

os.environ["OPENAI_API_KEY"] = ""

from langchain import LLMChain, PromptTemplate
from langchain.chat_models import ChatOpenAI
from langchain.prompts.chat import (
    ChatPromptTemplate,
    SystemMessagePromptTemplate,
    AIMessagePromptTemplate,
    HumanMessagePromptTemplate,
)
from langchain.llms import OpenAI

# Setup the LM to be used by langchain
llm = OpenAI(temperature=0.9)

human_message_prompt = HumanMessagePromptTemplate(
    prompt=PromptTemplate(
        template="What is a good name for a company that makes {product}?",
        input_variables=["product"],
    )
)
chat_prompt_template = ChatPromptTemplate.from_messages([human_message_prompt])
chat = ChatOpenAI(temperature=0.9)
chain = LLMChain(llm=chat, prompt=chat_prompt_template)

# Run the chain
chain.run("colorful socks")
```
gives error:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-17-c7d901a7e281>](https://localhost:8080/#) in <cell line: 25>()
     23 
     24 # Run the chain
---> 25 chain.run("colorful socks")

11 frames
[/usr/local/lib/python3.10/dist-packages/langchain/chat_models/openai.py](https://localhost:8080/#) in _create_retry_decorator(self)
    300 
    301     def _combine_llm_outputs(self, llm_outputs: List[Optional[dict]]) -> dict:
--> 302         overall_token_usage: dict = {}
    303         for output in llm_outputs:
    304             if output is None:

AttributeError: module 'openai' has no attribute 'error'
```

### Expected behavior

show results
add api_base to _client_params
Labels: name: auto:improvement, name: area: models, name: size:XS

  - **Description:** This PR adds `api_base` to `_client_params` in the `chat_model` of LiteLLM to ensure it's included in API calls. Previously, `api_base` was set on the client but was not included in the parameters passed to the completion function. This change ensures that `api_base` is correctly passed to all API calls.
  - **Issue:** #14338
  - **Tag maintainer:** @hwchase17 @agola11
  - **Twitter handle:** @LMS_David_RS
Issue: <how does langchain's routerchain work with conversationbuffermemory>
Labels: name: area: memory, name: auto:question, name: area: models

### Issue you'd like to raise.

How does conversationbuffermemory works with routerchain for suppose if I wanted to create a chat application I need memory to store the conversations how does that thing work with routerchain?
I'm currently using the same implementation that has shown in the documentation 
please respond as soon as possible
thank you:))

### Suggestion:

_No response_
Azure DocumentIntelligenceLoader/Parser support update with latest SDK
Labels: name: lgtm, name: area: doc loader, name: auto:enhancement, name: size:L

  - **Description:**
 Add DocumentIntelligenceLoader & DocumentIntelligenceParser implementation using the latest Azure Document Intelligence SDK with markdown support.
The core logic resides in DocumentIntelligenceParser and DocumentIntelligenceLoader is a mere wrapper of the parser.
The parser will takes api_endpoint and api_key and creates DocumentIntelligenceClient for the user. 4 parsing modes are supported:
1. Markdown (default)
2. Single
3. Page 
4. Object

UT and notebook are also updated accordingly.

  - **Dependencies:** Azure Document Intelligence SDK: azure-ai-documentintelligence [azure-sdk-for-python/sdk/documentintelligence/azure-ai-documentintelligence at 7c42462ac662522a6fd21b17d2a20f4cd40d0356 · Azure/azure-sdk-for-python (github.com)](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-sdk-for-python%2Ftree%2F7c42462ac662522a6fd21b17d2a20f4cd40d0356%2Fsdk%2Fdocumentintelligence%2Fazure-ai-documentintelligence&data=05%7C01%7CZifei.Qian%40microsoft.com%7C298225aa3e31468a863108dbf07374ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638368150928704292%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=oE0Sl4HERnMKdbkV9KgBV46Z2xytcQAShdTWf7ZNl%2Bs%3D&reserved=0).
infra: Fix test filesystem paths incompatible with windows
Labels: name: lgtm, name: auto:bug, name: size:S

- **Description:** This PR fixes test failures on Windows caused by path handling differences and unescaped special characters in regex. The failing tests are:
```
FAILED tests/unit_tests/storage/test_filesystem.py::test_yield_keys - AssertionError: assert ['key1', 'subdir\\key2'] == ['key1', 'subdir/key2']
FAILED tests/unit_tests/test_imports.py::test_importable_all - ModuleNotFoundError: No module named 'langchain_community.langchain_community\\adapters'
FAILED tests/unit_tests/tools/file_management/test_utils.py::test_get_validated_relative_path_errs_on_absolute - re.error: incomplete escape \U at position 53
FAILED tests/unit_tests/tools/file_management/test_utils.py::test_get_validated_relative_path_errs_on_parent_dir - re.error: incomplete escape \U at position 69
FAILED tests/unit_tests/tools/file_management/test_utils.py::test_get_validated_relative_path_errs_for_symlink_outside_root - re.error: incomplete escape \U at position 64
```

- **Issue:** fixes https://github.com/langchain-ai/langchain/issues/11775 (partially)
- **Dependencies:** none
OutputFixingParser does not work with PydanticOutputParser
Labels: name: auto:bug, name: area: models

### System Info

langchain v 0.0.344
pydantic v 2.5.2
pydantic_code v 2.14.5

python v 3.10.13

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction


I noticed that the **OutputFixingParser** class does not work when applied to a **PydanticOutputParser** class.
Something has probably changed in **Pydantic**.

Doing step-by-step debugging, I saw that in **PydanticOutputParser**, at line 32 (see below)...

![code](https://github.com/langchain-ai/langchain/assets/16083297/b59a8d15-2e21-41e3-baf4-180904d1f3ec)

...the exception caught is indeed a **ValidationError**, but _it is not the same_ **ValidationError**...

The **ValidationError** expected from that `try..except` block is of this type.

![valerr01](https://github.com/langchain-ai/langchain/assets/16083297/17ea8ec2-4b5f-4820-8f95-3481d687cfba)

While the **ValidationError** raised is of this other type.

![valerr02](https://github.com/langchain-ai/langchain/assets/16083297/d2ef0fa4-0fba-4234-a6a8-b65c1fd4ac76)


### Expected behavior

I therefore imagine that the **LangChain** code needs to be updated to also handle the new exception (since the old one belongs to a "**v1**" package).