delayedkarma/langchain-issues huggingface dataset:
fixed import error for BashOutputParser
Labels: name: lgtm, name: auto:bug, name: size:XS

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
[OBSOLETE] added CrossEncoderReranker with HuggingFaceCrossEncoder and SagemakerEndpointCrossEncoder

  - **Description:** Support reranking based on cross encoder models available from HuggingFace.
      - Added `CrossEncoder` schema
      - Implemented `HuggingFaceCrossEncoder` and `SagemakerEndpointCrossEncoder`
      - Implemented `CrossEncoderReranker` that performs similar functionality to `CohereRerank`
      - Added `cross-encoder-reranker.ipynb` to demonstrate how to use it. Please let me know if anything else needs to be done to make it visible on the table-of-contents navigation bar on the left, or on the card list on [retrievers documentation page](https://python.langchain.com/docs/integrations/retrievers).
  - **Issue:** N/A
  - **Dependencies:** None other than the existing ones.
Issue: Queries on conversational retreival chain prompt
Labels: name: area: vector store, name: auto:question, name: area: models

### Issue you'd like to raise.

I am using conversational retreival chain .from_llm().  If i try to print the final prompt that is been sent to LLM there i see LLM calls total 2 times. I will paste both the prompt in both calls.

Below is the code i am using
`qa_chain = ConversationalRetrievalChain.from_llm(llm=llm, chain_type=chain_type,
                                                     retriever=vector_database.as_retriever(),
                                                     return_source_documents=True)
qa_chain({'question': 'summarize this document', "chat_history": [('what is this', 'this is something'),('who you are', 'i am nothing')]})`


Below i can see the prompt sending to LLM 2 times

**1st call to  LLM:**


Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question, in its original language.

Chat History:
human: what is this
assistant: this is something
human: who you are
assistant: i am nothing


Follow Up Input: summarize this document
Standalone question:"""

**2nd call to LLM:**

Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

Title: Details
Title:Details
Tile:Details

Question: what is the summary of the document?
Helpful Answers:


In the above mentioned 2 calls i can see clearly whatever i have passed in **chat_history** as a question and answer tuple pair that is used to frame my question in the **1st call prompt**. and i can see clearly my original question **summarize this document** is been modified to **what is the summary of the document?**

My question or i am trying to understand is, In place of context i see 3 times Title:Details. what is this?? Can you please explain? .  is this the output of vectordb.asretreiver? if yes why i am seeing Title:Details 3 times?. please help me understand what is **Title:Details.**? in the **2nd prompt call**

Let me know if u need additional context to understand my questions.



### Suggestion:

_No response_
Bugfix duckduckgo_search news search
Labels: name: lgtm, name: auto:bug, name: size:L

  - **Description:** 
Bugfix duckduckgo_search news search
  - **Issue:** 
https://github.com/langchain-ai/langchain/issues/13648
  - **Dependencies:** 
None
  - **Tag maintainer:** 
@baskaryan
Fix SearchApi integration
Labels: name: area: agent, name: auto:bug, name: size:M

When integrating into the [Langflow](https://github.com/logspace-ai/langflow) repo, we noticed some issues with the integration, mainly with imports.

This PR fixes the imports and updates the documentation.

@baskaryan since you reviewed our last PR - https://github.com/langchain-ai/langchain/pull/11023.
Can not load chain with ChatPromptTemplate
Labels: name: auto:bug, name: area: models

### System Info

LangChain version: `0.0.339`
Python version: `3.10`

### Who can help?

@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

When serializing the `ChatPromptTemplate`, it saves it in a JSON/YAML format like this:
```
{'input_variables': ['question', 'context'],
 'output_parser': None,
 'partial_variables': {},
 'messages': [{'prompt': {'input_variables': ['context', 'question'],
    'output_parser': None,
    'partial_variables': {},
    'template': "...",
    'template_format': 'f-string',
    'validate_template': True,
    '_type': 'prompt'},
   'additional_kwargs': {}}],
 '_type': 'chat'}
```
Note that the `_type` is "chat".

However, LangChain's `load_prompt_from_config` [does not recognize "chat" as the supported prompt type](https://github.com/langchain-ai/langchain/blob/master/libs/core/langchain_core/prompts/loading.py#L19).

Here is a minimal example to reproduce the issue:
```python
from langchain.prompts import ChatPromptTemplate
from langchain.chains import RetrievalQA
from langchain.llms import OpenAI
from langchain.vectorstores import FAISS
from langchain.embeddings import OpenAIEmbeddings
from langchain.chains.loading import load_chain

TEMPLATE = """Answer the question based on the context:
{context}
Question: {question}
Answer:
"""
chat_prompt = ChatPromptTemplate.from_template(TEMPLATE)

llm = OpenAI()
def get_retriever(persist_dir = None):
  vectorstore = FAISS.from_texts(
      ["harrison worked at kensho"], embedding=OpenAIEmbeddings()
  )
  return vectorstore.as_retriever()

chain_with_chat_prompt = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=get_retriever(),
    chain_type_kwargs={"prompt": chat_prompt},
)

chain_with_prompt_saved_path = "./chain_with_prompt.yaml"
chain_with_prompt.save(chain_with_prompt_saved_path)
loaded_chain = load_chain(chain_with_prompt_saved_path, retriever=get_retriever())
```
The above script failed with the error:
`ValueError: Loading chat prompt not supported`

### Expected behavior

Load a chain that contains `ChatPromptTemplate` should work.
Issue: How to validate Tool input arguments without raising ValidationError
Labels: name: auto:question, name: area: models

### Issue you'd like to raise.

I am creating a tool that takes a multiple input arguments (say input1, input2). I would like to validate their types and also make sure that the tool only receives input1 and input2. How do I validate this without breaking the llm chain. I would instead like to return a warning to the llm agent. something like, "The input passed were incorrect, please try again"

### Suggestion:

_No response_
Add CometTracer
Labels: name: lgtm, name: auto:enhancement, name: size:L

Hi! I'm Alex, Python SDK Team Lead from [Comet](https://www.comet.com/site/).

This PR contains our new integration between langchain and Comet - `CometTracer` class which uses new `comet_llm` python package for submitting data to Comet.

No additional dependencies for the langchain package are required directly, but if the user wants to use `CometTracer`, `comet-llm>=2.0.0` should be installed. Otherwise an exception will be raised from `CometTracer.__init__`.

A test for the feature is included.

There is also an already existing callback (and .ipynb file with example) which ideally should be deprecated in favor of a new tracer. I wasn't sure how exactly you'd prefer to do it. For example we could open a separate PR for that.

I'm open to your ideas :)
Fix Sagemaker Endpoint documentation
Labels: name: lgtm, name: area: doc loader, name: auto:documentation, name: integration: aws, name: size:XS

  - **Description:** fixed the transform_input method in the example., 
  - **Issue:** example didn't work,
  - **Dependencies:** None,
  - **Tag maintainer:** @baskaryan,
  - **Twitter handle:** @Ravidhu87
Tests: Fix regression in `ConsistentFakeEmbeddings.embed_query`
Labels: name: area: embeddings, name: auto:bug, name: size:XS

**Description**: Fixes a regression in `ConsistentFakeEmbeddings.embed_query`.
**Issue**: The function did not return the correct values.
**Dependencies**: -,
**Tag maintainer**: @baskaryan

----

Hi there,

first things first: Thanks a stack for conceiving and maintaining LangChain.

At https://github.com/crate-workbench/langchain/pull/1, we are working on a database adapter for [CrateDB](https://github.com/crate/crate) [^1], where we may have discovered a minor flaw recently introduced to the test utility `ConsistentFakeEmbeddings`.

The function `ConsistentFakeEmbeddings.embed_query` was changed to only return the embedding of the query expression, without considering the known texts, as advertised. It was the recent commit a28eea57674a9 which may have changed it inadvertently this way.

The fix works well on behalf of the test cases we are running (`pytest -vvv tests/ -k cratedb`) [^2], but we didn't run it on any other vector store adapters. It may well have some impact, depending on where `ConsistentFakeEmbeddings` has been used, and how its outcomes are being evaluated on behalf of other test cases.

With kind regards,
Andreas.

[^1]: After sorting out some remaining issues, we hope to be able to contribute this adapter soon, and hope it will be well received.
[^2]: Actually, it was needed by a test case we added with https://github.com/crate-workbench/langchain/pull/15, where it also has been discovered.
Issue: Unable to Serialize AgentExecutor Object for Centralized Storage
Labels: name: area: memory, name: area: agent, name: auto:bug, name: auto:question, name: integration: redis

### Issue you'd like to raise.

To develop a scalable solution, I hoped to store the AgentExecutor object in centralized memory, such as Redis. However, due to its non-serializable nature, this has not been possible. Is there a workaround to achieve this?

### Suggestion:

_No response_
fix the problem that redis vector store will change metadata with val…
Labels: name: area: vector store, name: auto:bug, name: integration: redis, name: size:XS

  - **Description:** This commit fixed the problem that Redis vector store will change the value of a metadata from 0 to empty when saving the document, which should be an un-intended behavior. 
  - **Issue:** N/A
  - **Dependencies:** N/A
Issue: getting error in deletion of embeddings for a file
Labels: name: area: embeddings, name: auto:bug, name: integration: chroma

### Issue you'd like to raise.

  def delete_embeddings(file_path, persist_directory):
    chroma_db = chromadb.PersistentClient(path=persist_directory)
    collection = chroma_db.get_or_create_collection(name="langchain")
    ids = collection.get(where={"source": file_path})['ids']
    collection.delete(where={"source": file_path},ids=ids)
    print("delete successfully")
  below is the error i am getting
  File "/home/aaditya/DBChat/CustomBot/user_projects/views.py", line 1028, in project_pages
    delete_embeddings(names, persist_directory)
  File "/home/aaditya/DBChat/CustomBot/accounts/common_langcain_qa.py", line 134, in delete_embeddings
    chroma_db = chromadb.PersistentClient(path=persist_directory)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aaditya/DBChat/chat/lib/python3.11/site-packages/chromadb/__init__.py", line 106, in PersistentClient
    return Client(settings)
           ^^^^^^^^^^^^^^^^
  File "/home/aaditya/DBChat/chat/lib/python3.11/site-packages/chromadb/__init__.py", line 143, in Client
    api = system.instance(API)
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/aaditya/DBChat/chat/lib/python3.11/site-packages/chromadb/config.py", line 248, in instance
    impl = type(self)
           ^^^^^^^^^^
  File "/home/aaditya/DBChat/chat/lib/python3.11/site-packages/chromadb/api/segment.py", line 81, in __init__
    self._sysdb = self.require(SysDB)
                  ^^^^^^^^^^^^^^^^^^^
  File "/home/aaditya/DBChat/chat/lib/python3.11/site-packages/chromadb/config.py", line 189, in require
    inst = self._system.instance(type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aaditya/DBChat/chat/lib/python3.11/site-packages/chromadb/config.py", line 248, in instance
    impl = type(self)

### Suggestion:

_No response_
Enable support for async streaming on VertexAI
Labels: name: auto:improvement, name: area: models, name: size:L

  - **Description:** Enable support for async streaming on VertexAI based on release [1.36.0](https://github.com/googleapis/python-aiplatform/releases/tag/v1.36.0) from vertexai SDK 
  - **Issue:** N/A
  - **Dependencies:** Users will need google-cloud-aiplatform>=1.36.0
  - **Tag maintainer:** @baskaryan @leo-gan 

PS:
I couldn't find unit test for vertex ai that's why I did not add those.
issue langchain-ai#12165 mask API key and IAM token in YandexGPT
Labels: name: auto:improvement, name: area: models, name: size:M

  - **Description:** The PR masks API key and IAM token secrets for the YandexGPT model.
  - **Issue:** #12165 
  - **Dependencies:** yandexcloud,
  - **Tag maintainer:** @eyurtsev ,
  - **Twitter handle:** cheparukhin_s
Update bedrock.py to fix bug
Labels: name: auto:bug, name: area: models, name: size:XS

fix bug

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Added partial_variables to BaseStringMessagePromptTemplate.from_template(...)
Labels: name: lgtm, name: auto:bug, name: area: models, name: size:S

**Description:** BaseStringMessagePromptTemplate.from_template was passing the value of partial_variables into cls(...) via **kwargs, rather than passing it to PromptTemplate.from_template. Which resulted in those *partial_variables being* lost and becoming required *input_variables*.

**twitter handle**: @josep_pon
Support for streaming in the langchain chains (eg., load_summarize_chain)
Labels: name: area: doc loader, name: auto:enhancement, name: area: models

### Feature request

Hello, 

I'm not sure if this is already supported or not, I couldn't find anything in the documentation. 
Is there a way to make chains support streaming ? It would be nice if we can get it working with something like the `load_summarize_chain`. 

Or something like this: 
```


    doc_prompt = PromptTemplate.from_template("{page_content}")

    chain = (
            {
                "content": lambda docs: "\n\n".join(
                    format_document(doc, doc_prompt) for doc in docs
                )
            }
            | PromptTemplate.from_template("Summarize the following content:\n\n{content}")
            | OpenAI(
        temperature=1,
        model_name=llm_model,
        stream=True,
    )
            | StrOutputParser()
    )
    docs = [
        Document(
            page_content=split,
            metadata={"source": "https://en.wikipedia.org/wiki/Nuclear_power_in_space"},
        )
        for split in text.split()
    ]

    for partial_result in chain.invoke(docs):
        print(partial_result)

```


### Motivation

I have long documents to summarize, so I would like to show the partial results in streaming mode and not make the user wait so long to get the final result.

### Your contribution

No. If it's not possible, I'm willing to implement the summarization chain from scratch and use the OpenAI lib.
Use pytest asyncio auto mode
Labels: name: lgtm, name: auto:refactor, name: area: models, name: size:L

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Adjust Cobol Splitter. Take into consideration all divisions
Labels: name: lgtm, name: area: doc loader, name: auto:improvement, name: size:S

  - **Description:** Enhancements to the CobolSegmenter

1- Improved COBOL code validation by checking for the presence of essential COBOL division keywords (IDENTIFICATION, DATA, PROCEDURE, ENVIRONMENT) using a regular expression.
2- Refinements made to the _process_lines method, which iterates through all lines in the COBOL code. It now effectively identifies the start index when inside a relevant section.
3- The regular expression for identifying paragraphs has been updated to exclude certain keywords (e.g., "END_PERFORM," "END-IF," etc.), preventing them from being mistakenly categorized as paragraphs and split.
4- With the comprehensive division splitting and consideration of all code segments, the need for "simplify_code" is eliminated.
DOC: Template "rag-gpt-crawler" doc is incorrect 
Labels: name: area: doc loader, name: auto:documentation

### Issue with current documentation:

Regarding:
https://github.com/langchain-ai/langchain/blob/master/templates/rag-gpt-crawler/README.md
I found 2 issues:

1. no such file as "server.py" ( there is "rag_gpt_crawler.ipynb" instead )
2. in docs linked line
`from rag_chroma import chain as rag_gpt_crawler`
is not the same as shown in terminal 
`from rag_gpt_crawler import chain as rag_gpt_crawler_chain`
I think the first is incorrect.



### Idea or request for content:

_No response_
DOC: Text metadata
Labels: name: area: doc loader, name: auto:documentation

### Issue with current documentation:

The documentation on [creating documents](https://js.langchain.com/docs/modules/data_connection/document_loaders/how_to/creating_documents) covers optional document metadata but doesn't mention that it's possible to create text metadata in `page_content`. For example if only a filename is given to `CSVLoader` it will assume the header is metadata and [delimit each key-value pair with a newline](https://github.com/aws-samples/multi-tenant-chatbot-using-rag-with-amazon-bedrock/pull/14).

Chat LangChain will talk about this source of metadata but I can't find any additional information in the provided references.

> In the context of the RAG model, if the CSV data is used as a source for retrieval or generation, the fieldnames can be utilized in several ways:
> 
> • Retrieval: The fieldnames can be used as query terms to retrieve relevant documents from the CSV dataset. The RAG model can leverage the fieldnames to understand the user's query and retrieve documents that match the specified criteria.
> 
> • Generation: The fieldnames can provide context and constraints for generating responses. The RAG model can use the fieldnames as prompts or conditioning information to generate responses that are specific to the content of the corresponding columns in the CSV file.
> 
> By incorporating the CSV fieldnames into the retrieval and generation processes, the RAG model can produce more accurate and contextually relevant results based on the specific attributes and structure of the CSV dataset.



### Idea or request for content:

A description of the consequences of including text metadata for different scenarios, models, data stores.
Strategies for including text metadata.