delayedkarma/langchain-issues huggingface dataset:
Implemented MongoDB Atlas Self-Query Retriever
Labels: name: lgtm, name: auto:enhancement, name: size:XL

# Description 
This PR implements Self-Query Retriever for MongoDB Atlas vector store.

I've implemented the comparators and operators that are supported by MongoDB Atlas vector store according to the section titled "Atlas Vector Search Pre-Filter" from https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-stage/. 

Namely:
```
allowed_comparators = [
      Comparator.EQ,
      Comparator.NE,
      Comparator.GT,
      Comparator.GTE,
      Comparator.LT,
      Comparator.LTE,
      Comparator.IN,
      Comparator.NIN,
  ]

"""Subset of allowed logical operators."""
allowed_operators = [
    Operator.AND,
    Operator.OR
]
```
Translations from comparators/operators to MongoDB Atlas filter operators(you can find the syntax in the "Atlas Vector Search Pre-Filter" section from the previous link) are done using the following dictionary:
```
map_dict = {
            Operator.AND: "$and",
            Operator.OR: "$or",
            Comparator.EQ: "$eq",
            Comparator.NE: "$ne",
            Comparator.GTE: "$gte",
            Comparator.LTE: "$lte",
            Comparator.LT: "$lt",
            Comparator.GT: "$gt",
            Comparator.IN: "$in",
            Comparator.NIN: "$nin",
        }
```

In visit_structured_query() the filters are passed as "pre_filter" and not "filter" as in the MongoDB link above since langchain's implementation of MongoDB atlas vector store(libs\langchain\langchain\vectorstores\mongodb_atlas.py) in _similarity_search_with_score() sets the "filter" key to have the value of the "pre_filter" argument. 
```
params["filter"] = pre_filter
```
Test cases and documentation have also been added.

# Issue
#11616 

# Dependencies
No new dependencies have been added.

# Documentation
I have created the notebook mongodb_atlas_self_query.ipynb outlining the steps to get the self-query mechanism working.

I worked closely with [@Farhan-Faisal](https://github.com/Farhan-Faisal) on this PR.
Framework for supporting more languages in LanguageParser
Labels: name: area: doc loader, name: auto:enhancement, name: size:XXL

## Description

I am submitting this for a school project as part of a team of 5. Other team members are @LeilaChr, @maazh10, @Megabear137, @jelalalamy. This PR also has contributions from community members @Harrolee and @Mario928.

Initial context is in the issue we opened (#11229).

This pull request adds:

- Generic framework for expanding the languages that `LanguageParser` can handle, using the [tree-sitter](https://github.com/tree-sitter/py-tree-sitter#py-tree-sitter) parsing library and existing language-specific parsers written for it
- Support for the following additional languages in `LanguageParser`:
  - C
  - C++
  - C#
  - Go
  - Java (contributed by @Mario928 https://github.com/ThatsJustCheesy/langchain/pull/2)
  - Kotlin
  - Lua
  - Perl
  - Ruby
  - Rust
  - Scala
  - TypeScript (contributed by @Harrolee https://github.com/ThatsJustCheesy/langchain/pull/1)

Here is the [design document](https://docs.google.com/document/d/17dB14cKCWAaiTeSeBtxHpoVPGKrsPye8W0o_WClz2kk) if curious, but no need to read it.

## Issues

- Closes #11229
- Closes #10996
- Closes #8405

## Dependencies

`tree_sitter` and `tree_sitter_languages` on PyPI. We have tried to add these as optional dependencies.

## Documentation

We have updated the list of supported languages, and also added a section to `source_code.ipynb` detailing how to add support for additional languages using our framework.

## Maintainer

- @hwchase17 (previously reviewed https://github.com/langchain-ai/langchain/pull/6486)

Thanks!!

## Git commits

We will gladly squash any/all of our commits (esp merge commits) if necessary. Let us know if this is desirable, or if you will be squash-merging anyway.

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Add support for Bedrock Llama 2 13b model (meta.llama2-13b-chat-v1)
Labels: name: auto:bug, name: area: models, name: integration: aws

### System Info

I get following error just by adding model parameters to existing code that works with other models 

"Malformed input request: 2 schema violations found, please reformat your input and try again."

```
model_name = "meta.llama2-13b-chat-v1"
model_kwargs = {
"max_gen_len": 512,
"temperature": 0.2,
 "top_p": 0.9
}
bedrock_boto = boto3.client("bedrock-runtime", "us-east-1")
bedrock_llm = Bedrock(model_id=model_name, client=bedrock_boto,model_kwargs=model_kwargs)
bedrock_llm("Hello!")
```

### Who can help?

@hwchase17
@agola11

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. Attempt to call the new llama2 bedrock model like so:

```
model_name = "meta.llama2-13b-chat-v1"
model_kwargs = {
"max_gen_len": 512,
"temperature": 0.2,
 "top_p": 0.9
}
bedrock_boto = boto3.client("bedrock-runtime", "us-east-1")
bedrock_llm = Bedrock(model_id=model_name, client=bedrock_boto,model_kwargs=model_kwargs)
bedrock_llm("Hello!")
```

### Expected behavior

The Bedrock class would work successfully as it does for other BedRock models
IMPROVEMENT more research-assistant configurability
Labels: name: auto:improvement

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Add a `ListRerank` document compressor
Labels: name: lgtm, name: auto:enhancement, name: size:L

- **Description:** This PR adds a new document compressor called `ListRerank`. It's derived from `BaseDocumentCompressor`. It's a near exact implementation of introduced by this paper: [Zero-Shot Listwise Document Reranking with a Large Language Model](https://arxiv.org/pdf/2305.02156.pdf) which it finds to outperform pointwise reranking, which is somewhat implemented in LangChain as [LLMChainFilter](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/retrievers/document_compressors/chain_filter.py). 
- **Issue:** None
- **Dependencies:** None
- **Tag maintainer:** @hwchase17 @izzymsft
- **Twitter handle:** @HarrisEMitchell

Notes:
1. I didn't add anything to `docs`. I wasn't exactly sure which patterns to follow as [cohere reranker is under Retrievers](https://python.langchain.com/docs/integrations/retrievers/cohere-reranker) with other external document retrieval integrations, but other contextual compression is [here](https://python.langchain.com/docs/modules/data_connection/retrievers/contextual_compression/). Happy to contribute to either with some direction.
2. I followed syntax, docstrings, implementation patterns, etc. as well as I could looking at nearby modules. One thing I didn't do was put the default prompt in a separate `.py` file like [Chain Filter](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/retrievers/document_compressors/chain_filter_prompt.py) and [Chain Extract](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/retrievers/document_compressors/chain_extract_prompt.py). Happy to follow that pattern if it would be preferred.
Remove deprecated param and flexibility for prompt 
Labels: name: auto:improvement, name: area: models, name: size:S


  - **Description:** Updated to remove deprecated parameter penalty_alpha, and use string variation of prompt rather than json object for better flexibility.  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** N/A
  - **Tag maintainer:** @eyurtsev
  - **Twitter handle:** @symbldotai
Pinecone SelfQueryRetriever with datetime filter
Labels: name: area: vector store, name: auto:improvement, name: integration: pinecone

### Feature request

I wan´t to use Pinecone SelfQueryRetriever with a metadata in datetime format. When I try to ask anything with a date show the error Bad Request.

I already change my class PineconeTranslator with this code with the same pattern of Scale, change the translator class. Since Pinecone just accept integer values and they ask you to use epoch format.

langchain\retrievers\self_query\pinecone.py
Class PineconeTranslator

`    def visit_comparison(self, comparison: Comparison) -> Dict:
        value = comparison.value

        # convert timestamp to float as epoch format.
        if type(value) is date:
            value = time.mktime(value.timetuple())

        return {
            comparison.attribute: {
                self._format_func(comparison.comparator): value
            }
        }`

### Motivation

I want to use selfretriever with pinecone and langchain the error limit the use of langchain with pinecone.

### Your contribution

Yes, I already made the changes in my machine like the code in the description
fix litellm openai imports
Labels: name: auto:bug, name: area: models

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Ollama LLM: Implement async functionality
Labels: name: auto:enhancement, name: area: models

### Feature request

Implement Async methods in ollama llm and chat model classes.

### Motivation

Ollama implementation doesn't include the async methods _astream and _agenerate and i cannot create a async agent... 

### Your contribution

This is my first issue, i can try but i am working in 3 different projects right now...
SQL-PGVector Template
Labels: name: area: vector store, name: auto:enhancement

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
bugfix in using search_kwargs in Vectara retriever
Labels: name: lgtm, name: area: vector store, name: auto:bug

  - **Description:** fix a bug that prevented as_retriever() in Vectara to use the desired input arguments
  - **Issue:** as_retriever did not pass the arguments properly
  - **Tag maintainer:** @baskaryan
  - **Twitter handle:** @ofermend
Langchain Template for RAG using Opensearch 
Labels: name: area: vector store, name: auto:enhancement, name: area: models

### Feature request

We have several RAG templates in [langchain template](https://github.com/langchain-ai/langchain/tree/master/templates) using other vector DB except Opensearch. 

We need to add a similar template for Opensearch also as opensearch support the same feature. 

As a first step we will start with OpenAI and then we expand this to other LLMs including Bedrock.

### Motivation

Opensearch supports this feature and it will be super helpful for opensearch users. 

### Your contribution

I will take the initiative to raise PR on this issue.
Context retrieval 
Labels: name: area: memory, name: auto:question, name: area: models

### System Info

Linux and langchain 




### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [X] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I am currently implementing a RAG chatbot using `ConversationBufferMemory` and `ConversationalRetrievalChain`as follow: 

```
 memory = ConversationBufferMemory(
        memory_key="chat_history",
        output_key="answer",
        return_messages=True,
    )
```

    chain = ConversationalRetrievalChain.from_llm(
        llm,
        chain_type="stuff",
        retriever=db.as_retriever(search_kwargs={"k": 1}),
        memory=memory,
        combine_docs_chain_kwargs={"prompt": PROMPT},
        return_source_documents=True,
        verbose=True,
    )`
Where PROMPT is a prompt template that inputs chat_history, context and question. 

### Expected behavior

I would like the retrieval chain and memory chain to keep track of the context i.e. instead of updating the context with the retrieved documents every time a user input a new question, I can keep track of the context.
My end goal is to have a function that takes into the old context, validate if retrieval of new documents is necessary, and if those documents can replace the new context or they need to be append to the old context. Is there a way to do this?
Improve openai callback
Labels: name: auto:improvement

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->

As we discussed in #12994 the current callback only returns 0 when the cost fails to calculate. This way it will raise an error instead to prevent users to falsely relying on an incorrect cost. I also improved the copy functions so that the actually create a new instance of the object and does not just return the same object.
Issue: Create async streaming custom LLM
Labels: name: auto:question, name: auto:enhancement, name: area: models

### Issue you'd like to raise.

I currently have an OLLAMA model running locally. I aim to develop a CustomLLM model that supports asynchronous streaming to interface with this model. However, the OLLAMA adapter provided by LangChain lacks support for these specific features. How can I accomplish this integration?

### Suggestion:

_No response_
Support for Text Embedding Inference (TEI) from HuggingFace
Labels: name: area: embeddings, name: auto:enhancement

### Feature request

Similar to Text Generation Inference (TGI) for LLMs, HuggingFace created an inference server for text embeddings models called Text Embedding Inference (TEI). 
See: https://github.com/huggingface/text-embeddings-inference
Could you integrate TEI into the supported LangChain text embedding models or do you guys already plan to do this? 

### Motivation

We currently develop a rag based chat app and plan to deploy the components as microservices (LLM, DB, Embedding Model). Currently the only other suitable solution for us would be to use SagemakerEndpointEmbeddings. However being able to use TEI would be a great benefit.

### Your contribution

I work as an ML Engineer and could probably assist in some way if necessary.
Update index.mdx
Labels: name: lgtm, name: auto:documentation, name: size:XS

grammar correction

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} when using openai==1.2.3 and langchain==0.0.335
Labels: name: auto:bug, name: area: models

### System Info

from langchain.chat_models import AzureChatOpenAI

llm_chat = AzureChatOpenAI(deployment_name="gpt-4_32k", model_name = 'gpt-4-32k', openai_api_version=openai.api_version, temperature=0)

from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate

prompt = PromptTemplate(
    input_variables = ["text"],
    template="{text}"
)

llmchain = LLMChain(llm=llm_chat, prompt=prompt)
llmchain.run(text)

NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

openai==1.2.3 and langchain==0.0.335

from langchain.chat_models import AzureChatOpenAI
text = "Where is Germany located?"
llm_chat = AzureChatOpenAI(deployment_name="gpt-4_32k", model_name = 'gpt-4-32k', openai_api_version=openai.api_version, temperature=0)

from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate

prompt = PromptTemplate(
    input_variables = ["text"],
    template="{text}"
)

llmchain = LLMChain(llm=llm_chat, prompt=prompt)
llmchain.run(text)

### Expected behavior

In europe