delayedkarma/langchain-issues huggingface dataset:
Update stuff_prompt.py (Fix the sources field is blank)
Labels: name: auto:bug

**Description:** This change aims to resolve the issue by enhancing source extraction from the following link: [LangChain Issue #9913](https://github.com/langchain-ai/langchain/issues/9913). The problem is fixed by replacing "source" in the prompt with "reference."

**Dependencies:** There are no specific dependencies required for this change.

**Twitter handle:** [@davinci137](https://twitter.com/davinci137).

The "SOURCES" field is currently empty due to a split occurring at the word "Source:."

```python
# In the middle of the answer, a split is performed
def _split_sources(self, answer: str) -> Tuple[str, str]:
    """
    This function is responsible for splitting sources from the answer.
    Args:
        answer (str): The answer text.
    Returns:
        Tuple[str, str]: A tuple containing the modified answer and extracted sources.
    """
    if re.search(r"SOURCES?:", answer, re.IGNORECASE):
        # It utilizes regular expressions to split the answer and sources based on "SOURCES:" or "QUESTION:" (case-insensitive).
        answer, sources = re.split(r"SOURCES?:|QUESTION:\s", answer, flags=re.IGNORECASE)[:2]
        # The sources are split based on newline characters, and the first line is extracted, with leading and trailing whitespace removed.
        sources are then stripped.
    else:
        sources = ""
    return answer, sources
response parser for ArceeRetriever
Labels: name: area: doc loader, name: auto:improvement

  - **Description:** Response parser for arcee retriever, 
  - **Issue:** follow-up pr on #11578 and [discussion](https://github.com/arcee-ai/arcee-python/issues/15#issuecomment-1759874053),
  - **Dependencies:** NA

This pr implements a parser for the response from ArceeRetreiver to convert to langchain `Document`. This closes the loop of generation and retrieval for Arcee DALMs in langchain.

The reference for the response parser is [api-docs:retrieve](https://api.arcee.ai/docs#/v2/retrieve_model)

Attaching screenshot of working implementation:
<img width="1984" alt="Screenshot 2023-10-25 at 7 42 34 PM" src="https://github.com/langchain-ai/langchain/assets/65639964/026987b9-34b2-4e4b-b87d-69fcd0c6641a">
\*api key deleted

---
Successful tests, lints, etc.
```shell
Re-run pytest with --snapshot-update to delete unused snapshots.
==================================================================================================================== slowest 5 durations =====================================================================================================================
1.56s call     tests/unit_tests/schema/runnable/test_runnable.py::test_retrying
0.63s call     tests/unit_tests/schema/runnable/test_runnable.py::test_map_astream
0.33s call     tests/unit_tests/schema/runnable/test_runnable.py::test_map_stream_iterator_input
0.30s call     tests/unit_tests/schema/runnable/test_runnable.py::test_map_astream_iterator_input
0.20s call     tests/unit_tests/indexes/test_indexing.py::test_cleanup_with_different_batchsize
======================================================================================================= 1265 passed, 270 skipped, 32 warnings in 6.55s =======================================================================================================
[ "." = "" ] || poetry run black .
All done! ✨ 🍰 ✨
1871 files left unchanged.
[ "." = "" ] || poetry run ruff --select I --fix .
./scripts/check_pydantic.sh .
./scripts/check_imports.sh
poetry run ruff .
[ "." = "" ] || poetry run black . --check
All done! ✨ 🍰 ✨
1871 files would be left unchanged.
[ "." = "" ] || poetry run mypy .
Success: no issues found in 1868 source files
poetry run codespell --toml pyproject.toml
poetry run codespell --toml pyproject.toml -w
```
Finetuned openai azure models cost calculation
Labels: name: auto:improvement, name: area: models

**Description:**
Add cost calculation for fine tuned **Azure** with relevant unit tests.
see https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/fine-tuning?tabs=turbo&pivots=programming-language-studio for more information.
this PR is the result of this PR: https://github.com/langchain-ai/langchain/pull/12190

Twitter handle: @nirkopler
Token limitation due to model's maximum context length 
Labels: name: auto:bug, name: area: models

### System Info

I've been using the conversation chain to retrieve answers from gpt3.5, vertex ai chat-bison. For the user's memory I've been passing the session memory appended to the conversation chain. But once the token limit exceeds I'm getting this following error from the llm 

`[chain/error] [1:chain:LLMChain > 2:llm:ChatOpenAI] [1.69s] Llm run errored with error:
"InvalidRequestError(message=\"This model's maximum context length is 4097 tokens. However, your messages resulted in 9961 tokens. Please reduce the length of the messages.\", param='messages', code='context_length_exceeded', http_status=400, request_id=None)"
[chain/error] [1:chain:LLMChain] [1.72s] Chain run errored with error:
"InvalidRequestError(message=\"This model's maximum context length is 4097 tokens. However, your messages resulted in 9961 tokens. Please reduce the length of the messages.\", param='messages', code='context_length_exceeded', http_status=400, request_id=None)"
`

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Once you add history appended to the chain, with more than 4097 tokens this will appear.

### Expected behavior

The chat should work normally without any internal exception. Is there any langchain support for this issue?
ConversationalRetrievalChain returns sources to questions without context
Labels: name: auto:bug, name: area: models

### Issue you'd like to raise.

`ConversationalRetrievalChain` returns sources to questions without relevant documents. The returned sources are not related to the question or the answer.

Example with relevant document:

```
chain = ConversationalRetrievalChain(
    retriever=vectorstore.as_retriever(),
    question_generator=question_generator,
    combine_docs_chain=doc_chain,
)

chat_history = []
query = "What did the president say about Ketanji Brown Jackson"
result = chain({"question": query, "chat_history": chat_history})
result['answer']

"""
The president said that he nominated Circuit Court of Appeals Judge Ketanji Brown Jackson, who is one of the nation's top legal minds ...

SOURCES: /content/state_of_the_union.txt
``` 

Example without relevant document:
```
query = "How are you doing?"
result = chain({"question": query, "chat_history": chat_history})
result['answer']

"""
I'm doing well, thank you.

SOURCES: /content/state_of_the_union.txt
"""
```

Another example without relevant document:
```
query = "Who is Elon Musk?"
result = chain({"question": query, "chat_history": chat_history})
result['answer']

"""
I don't know.

SOURCES: /content/state_of_the_union.txt
"""
```


### Suggestion:

Sources should not be returned for questions without relevant documents.
Issue: No module named 'langchain.vectorstores.azure_cosmos_db_vector_search'
Labels: name: area: vector store, name: auto:bug

### Issue you'd like to raise.

I am getting below error while trying to use Azure Cosmos DB as vector store.

**_No module named 'langchain.vectorstores.azure_cosmos_db_vector_search'_**

### Suggestion:

_No response_
Returning detailed results of intermediate components
Labels: name: auto:improvement, name: auto:documentation, name: area: models, name: area: lcel

### Issue you'd like to raise.

I am currently using a chain with the following components:
FewshotPrompt ( with SemanticSimilarityExampleSelector) -> LLM -> Outputparser

By default, this will give me the parsed text output of my LLM. 
For my purposes, I would need the output of this chain to include all intermediate steps (preferably in a dict).

For this chain that would be: 
1. The samples selected by the Example Selector
2. The generated Fewshot Prompt
3. The raw LLM output
4. The parsed LLM output

From the documentation, it is not apparent to me how this could be achieved with the current state of the library.
(The documentation also does not outline details about the usage of Callbacks in the LCEL language)

### Suggestion:

If this is currently implemented then add a simple example to the documentation. If this is not the case then some type of callback that allows collecting this data from intermediate steps should be added.
User based memorization
Labels: name: area: memory, name: auto:question

### System Info

How i can implement user based memorization please can provide a link for refrence

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [X] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

How i can implement user based memorization please can provide a link for refrence

### Expected behavior

How i can implement user based memorization please can provide a link for refrence
refactor: DeterministicFakeEmbedding -> DeterministicFakeEmbeddings
Labels: name: area: embeddings, name: auto:refactor

 **Description:** I simply add an "s" to the current `DeterministicFakeEmbedding` in order to have a unique naming that is aligned to ALL `Embeddings` classes.

Note: right now, there are also `AlephAlphaAsymmetricSemanticEmbedding` and `AlephAlphaSymmetricSemanticEmbedding` classes that don't have this "s". If you want, I can also align them as well.
Update vectorstore.mdx[Make an improvement]
Labels: name: area: vector store, name: auto:documentation

correct some grammatical errors

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Allow disablling `safe_len_embeddings`(OpenAIEmbeddings)
Labels: name: area: embeddings, name: auto:improvement, name: size:M

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->

For servers compatible with OpenAI API may not support `safe_len_embedding`.

With this PR, we can add `disable_safe_len_embeddings=True` to `OpenAIEmbeddings(...)` to disable using `safe_len_embedding`.
Vertex AI Models Throws AttributeError: After Langchain 0.0295 Version
Labels: name: area: memory, name: auto:bug, name: area: models

### System Info

1

I have been working with LangChain for some time, and after updating my LangChain module to version 0.0.295 or higher, my Vertex AI models throw an error in my application. The error is

  File "E:\Office\2023\AI-Platform\kaya-aia-general-bot\venv\Lib\site-packages\langchain\llms\vertexai.py", line 301, in _generate
    generations.append([_response_to_generation(r) for r in res.candidates])
                                                            ^^^^^^^^^^^^^^
AttributeError: 'TextGenerationResponse' object has no attribute 'candidates'

I was expecting my previous implementation to work seamlessly. This is how I initialized the chain with the vertex model.

case 'public_general_palm':
llm = ChatVertexAI(model_name="chat-bison", temperature=0)
chain = self.__initialize_basic_chain(llm, self.chain_memory)
return chain(request.msg, return_only_outputs=True)
This is the __initialize_basic_chain method.

def __initialize_basic_chain(self, llm, memory):
   return ConversationChain(llm=llm, memory=memory)
However, I have been unable to find a solution for this error. Kindly assist in resolving this issue to ensure the smooth operation of the application.

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [X] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Update langchain module 0.315 and higher. 

1. Initialize llm with the ChatVertexAI(model_name="chat-bison", temperature=0). 
2. Pass into a basic chain or an agent with the input messages

### Expected behavior

Throws an internal module error called 

 File "E:\Office\2023\AI-Platform\kaya-aia-general-bot\venv\Lib\site-packages\langchain\llms\vertexai.py", line 301, in _generate
    generations.append([_response_to_generation(r) for r in res.candidates])
                                                            ^^^^^^^^^^^^^^
AttributeError: 'TextGenerationResponse' object has no attribute 'candidates'
Integration Doctran: DoctranPropertyExtractor class methods transform_documents() and atransform_documents() error
Labels: name: auto:bug, name: area: models

### System Info

Under doctran_text_extract.py, DoctranPropertyExtractor class. The methods for the class contains errors. 

The first method transform_documents() does not have any operations and simply returns a NotImplementedError. This method should be working and implemented, with the following code:
```
def transform_documents(
        self, documents: Sequence[Document], **kwargs: Any
    ) -> Sequence[Document]:
        """Extracts properties from text documents using doctran."""
        try:
            from doctran import Doctran, ExtractProperty

            doctran = Doctran(
                openai_api_key=self.openai_api_key, openai_model=self.openai_api_model
            )
        except ImportError:
            raise ImportError(
                "Install doctran to use this parser. (pip install doctran)"
            )
        properties = [ExtractProperty(**property) for property in self.properties]
        for d in documents:
            doctran_doc = (
                doctran.parse(content=d.page_content)
                .extract(properties=properties)
                .execute()
            )

            d.metadata["extracted_properties"] = doctran_doc.extracted_properties
        return documents
```

For the second method atransform_documents(), doctran currently does not support async so this should raise a NotImplementedError instead.

### Who can help?

@agola11 

### Information

- [X] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [X] Async

### Reproduction

https://python.langchain.com/docs/integrations/document_transformers/doctran_extract_properties

official code does not reproduce the displayed behavior

### Expected behavior

official code does not reproduce the displayed behavior. Modified source code as mentioned above and used the class method transform_documents works with the displayed outcome.
rfc: make ChatModel._generate a batch method
Labels: name: auto:refactor, name: area: models

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Update README.md - fixed two grammatical mistakes in README.
Labels: name: auto:documentation

I have noticed two small grammatical issues in **README.md** file inside section "**Breaking changes for select chains**".

"**but we are remaining backwards compatible...**" should be "**but we are remaining backward compatible..**"  and also

"**On that date, we will remove functionality from langchain..**"  should be "**On that date, we will remove the functionality from langchain..**"

Below I have attached a screenshot of it.

![Screenshot (89)](https://github.com/langchain-ai/langchain/assets/115995339/6d3cb198-a7f8-461a-93c0-b342aba20697)
Add URL's to the output of TelegramChatAPILoader 🙏
Labels: name: area: doc loader, name: auto:improvement

### Feature request

Currently the `TelegramChatAPILoader` returns the Date, Sender, and Text of each message. Adding the URL of each message would very helpful for a Natural Language Search Engine for a large Channel.

SOURCE: `langchain.document_loaders.telegram` 

### Motivation

We are building a Natural Language assistant for a large channel on Telegram. The users currently search by key words and hashtags. The Language model can only return a URL if it's contained in the text of the message. With the URL's we would be able to easily direct the user to the relevant information with a Natural Language query instead of the archaic methods of the days before Ai.

### Your contribution

I do not know Telethon, however we can see that the URL can be pulled in the `fetch_data_from_telegram` function in the [langchain.document_loaders.telegram ](https://api.python.langchain.com/en/latest/_modules/langchain/document_loaders/telegram.html#TelegramChatApiLoader.fetch_data_from_telegram)class. 

We thank you for LangChain and wish we could spare the time to contribute more. We may have a pull request in the future regarding Telegram Loaders for media messages. It's a feature that we will have to incorporate as we get closer to a public release of what we're working on.
retriever.get_relevant_documents
Labels: name: area: vector store, name: auto:bug

### System Info

langchain in /opt/conda/lib/python3.11/site-packages (0.0.322)

Python 3.11.6

installed on 
image: jupyter/datascience-notebook



### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

As per the documentation 

retriever = SelfQueryRetriever.from_llm(llm, vectorstore, document_content_description, metadata_field_info, verbose=True)
retriever.get_relevant_documents("What are some movies about dinosaurs")


### Expected behavior

We would see verbose output such as
`    query=' ' filter=Comparison(comparator=<Comparator.GT: 'gt'>, attribute='rating', value=8.5)
` 
however no verbose output is printed