delayedkarma/langchain-issues huggingface dataset:
Updates to Nomic Atlas and GPT4All documentation
Labels: name: lgtm, name: area: embeddings, name: auto:documentation

Description: Updates for Nomic AI Atlas and GPT4All integrations documentation.

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
Cache black formatting information across CI runs.
Labels: name: auto:improvement

Save and persist `black`'s formatted files cache across CI runs.

Around a ~20s win, 21s -> 2s. Most cases should be close to this best case scenario, since most PRs don't modify most files — and this PR makes sure we don't re-check files that haven't changed.

Before:
![image](https://github.com/langchain-ai/langchain/assets/2348618/6c5670c5-be70-4a18-aa2a-ece5e4425d1e)

After:
![image](https://github.com/langchain-ai/langchain/assets/2348618/37810d27-c611-4f76-b9bd-e827cefbaa0a)
while importing the llama_index occurred error that
Labels: name: area: vector store, name: auto:bug, name: pydantic


---------------------------------------------------------------------------
from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext

--------------------------------------------------------------------------
/usr/local/lib/python3.10/dist-packages/pydantic/_internal/_config.py:257: UserWarning: Valid config keys have changed in V2:
* 'allow_population_by_field_name' has been renamed to 'populate_by_name'
  warnings.warn(message, UserWarning)

PydanticUserError                         Traceback (most recent call last)
[<ipython-input-15-2df7a532c2da>](https://localhost:8080/#) in <cell line: 1>()
----> 1 from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
      2 from llama_index.llms import HuggingFaceLLM

6 frames
[/usr/local/lib/python3.10/dist-packages/pydantic/deprecated/class_validators.py](https://localhost:8080/#) in root_validator(pre, skip_on_failure, allow_reuse, *__args)
    226     mode: Literal['before', 'after'] = 'before' if pre is True else 'after'
    227     if pre is False and skip_on_failure is not True:
--> 228         raise PydanticUserError(
    229             'If you use `@root_validator` with pre=False (the default) you MUST specify `skip_on_failure=True`.'
    230             ' Note that `@root_validator` is deprecated and should be replaced with `@model_validator`.',

PydanticUserError: If you use `@root_validator` with pre=False (the default) you MUST specify `skip_on_failure=True`. Note that `@root_validator` is deprecated and should be replaced with `@model_validator`.

For further information visit https://errors.pydantic.dev/2.0/u/root-validator-pre-skip
Issue: Can't import Langchain
Labels: name: auto:bug, name: pydantic

While trying to import langchain in Jupyter Notebook getting this error.

> PydanticUserError: If you use `@root_validator` with pre=False (the default) you MUST specify `skip_on_failure=True`. Note that `@root_validator` is deprecated and should be replaced with `@model_validator`.
> 
> For further information visit https://errors.pydantic.dev/2.1.1/u/root-validator-pre-skip

### Suggestion:

using langchain 0.0.27 version
Do not bust `poetry install` cache when manually installing pydantic v2.
Labels: name: auto:improvement

Using `poetry add` to install `pydantic@2.1` was also causing poetry to change its lockfile. This prevented dependency caching from working:
- When attempting to restore a cache, it would hash the lockfile in git and use it as part of the cache key. Say this is a cache miss.
- Then, it would attempt to save the cache -- but the lockfile will have changed, so the cache key would be *different* than the key in the lookup. So the cache save would succeed, but to a key that cannot be looked up in the next run -- meaning we never get a cache hit.

In addition to busting the cache, the lockfile update itself is also non-trivially long, over 30s:
![image](https://github.com/langchain-ai/langchain/assets/2348618/d84d3b56-484d-45eb-818d-54126a094a40)

This PR fixes the problems by using `pip` to perform the installation, avoiding the lockfile change.
Cache .mypy_cache across lint runs.
Labels: name: auto:nit

Preserve the `.mypy_cache` directory across lint runs, to avoid having to re-parse all dependencies and their type information.

Approximately a 1min perf win for CI.

Before:
![image](https://github.com/langchain-ai/langchain/assets/2348618/6524f2a9-efc0-4588-a94c-69914b98b382)

After:
![image](https://github.com/langchain-ai/langchain/assets/2348618/dd0af954-4dc9-43d3-8544-25846616d41d)
SequentialChain runs the same callbacks over and over in async mode 
Labels: name: auto:bug, name: area: models

### System Info

In the Async mode, SequentialChain implementation seems to run the same callbacks over and over since it is re-using the same callbacks object. 

Langchain version: 0.0.264

The implementation of this aysnc route differs from the sync route and sync approach follows the right pattern of generating a new callbacks object instead of re-using the old one and thus avoiding the cascading run of callbacks at each step.

[Async code](https://github.com/langchain-ai/langchain/blob/v0.0.264/libs/langchain/langchain/chains/sequential.py#L194)
```
        _run_manager = run_manager or AsyncCallbackManagerForChainRun.get_noop_manager()
        callbacks = _run_manager.get_child()
        ...
        for i, chain in enumerate(self.chains):
            _input = await chain.arun(_input, callbacks=callbacks)
            ...
```
      

[Sync code](https://github.com/langchain-ai/langchain/blob/v0.0.264/libs/langchain/langchain/chains/sequential.py#L180)
```
        _run_manager = run_manager or CallbackManagerForChainRun.get_noop_manager()
        for i, chain in enumerate(self.chains):
            _input = chain.run(_input, callbacks=_run_manager.get_child(f"step_{i+1}"))
            ...
```

Notice how we are reusing the `callbacks` object in the Async code which will have a cascading effect as we run through the chain. It runs the same callbacks over and over resulting in issues.

CC @agola11 

### Who can help?

@agola11 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [x] Chains
- [X] Callbacks/Tracing
- [X] Async

### Reproduction

You can write a simple sequential chain with 3 tasks with callbacks and run the code in async and notice that the callbacks run over and over as we can see in the log.   

### Expected behavior

We should ideally see the callbacks get run once per task.
Bump the awadb version from v0.3.9 to v0.3.10
Labels: name: auto:improvement

Mainly support string array field type and support dynamically to add new fields.
Document(
        page_content="A bunch of scientists bring back dinosaurs and mayhem breaks loose",
        metadata={"year": 1993, "rating": 7.7, "genre": ["action", "science fiction"]},
    )
Such as the document above,  the metadata field 'genre' has multiple strings.
Now the version of 0.3.10 for AwaDB supports the multiple strings.
@baskaryan Please review,  thanks!
ConversationalRetrievalChain doesn't work along with memory and return_source_documents
Labels: name: area: memory, name: auto:bug

### Issue you'd like to raise.

I'm trying to use a `ConversationalRetrievalChain` along with a `ConversationBufferMemory` and `return_source_documents` set to `True`. The problem is that, under this setting, I get an error when I call the overall chain.

```
from langchain.chains import ConversationalRetrievalChain

chain = ConversationalRetrievalChain(
    retriever=retriever,
    question_generator=question_generator,
    combine_docs_chain=doc_chain,
    memory = memory,
    return_source_documents=True,
)

query = 'dummy query'
chain({"question": query})
```
The error message says:
```
File [~/.local/share/virtualenvs/qa-8mHXn5ez/lib/python3.10/site-packages/langchain/chains/base.py:354](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/my_user/~/.local/share/virtualenvs/env-8mHXn5ez/lib/python3.10/site-packages/langchain/chains/base.py:354), in Chain.prep_outputs(self, inputs, outputs, return_only_outputs)
    352 self._validate_outputs(outputs)
    353 if self.memory is not None:
--> 354     self.memory.save_context(inputs, outputs)
...
---> 28         raise ValueError(f"One output key expected, got {outputs.keys()}")
     29     output_key = list(outputs.keys())[0]
     30 else:

ValueError: One output key expected, got dict_keys(['answer', 'source_documents'])
```
It works if I remove either the memory or the `return_source_documents` parameter.

So far, the only workaround that I found out is querying the chain using an external chat history, like this:

`chain({"question": query, "chat_history":"dummy chat history"})`

Thank you in advance for your help.
Fallbacks with LLMChain
Labels: name: area: agent, name: auto:bug, name: area: models

### System Info

@hwchase17
@agola11 

Trying to implement https://python.langchain.com/docs/guides/fallbacks into our current environment that is using LLMChain, but when I pass in the fallback llm into the LLMChain, it throws the following error: 

`**ValidationError: 1 validation error for LLMChain**
llm
  Can't instantiate abstract class BaseLanguageModel with abstract methods agenerate_prompt, apredict, apredict_messages, generate_prompt, invoke, predict, predict_messages (type=type_error)`

Here is part of the code that I am trying to apply:

`
tools = [
        Tool(
            func=search.run, description=search_desc, name="Search_the_web"
        ),
]

conv0memory = ConversationBufferMemory(
        memory_key="chat_history_lines",
        return_messages=True,
        input_key="input",
        human_prefix="Human",
        ai_prefix="AI"
    )

openai_llm = ChatOpenAI(max_retries=0)
anthropic_llm = ChatAnthropic()
llm = openai_llm.with_fallbacks([anthropic_llm])

input_variables = ["input", "chat_history", "agent_scratchpad"]
prompt = ZeroShotAgent.create_prompt(
        tools,
        prefix=prefix,
        suffix=suffix,
        input_variables=input_variables
    )
    
    llm_chain = LLMChain(
        llm=llm, callbacks=[StreamingStdOutCallbackHandler()], prompt=prompt
    )
`

What must be done to pass the fallback LLM into LLMChain?

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [X] Callbacks/Tracing
- [ ] Async

### Reproduction

1. To reproduce the same output error, follow the steps indicated in https://python.langchain.com/docs/guides/fallbacks  to initiate the fallback
2. Pass the fallback LLM into LLMChain, where it usually passes the standard LLM methods i.e ChatModels, we add the fallback LLM to it instead. 
3. Use the code snippet provided in my ticket.

### Expected behavior

To have the fallback LLM work as per the documentation within LLMChain, whereas it fallback to the second LLM encase the API is busy, returns max token limit error, or to fall back for a better model to answer certain questions.
Nc/runnable lambda recurse
Labels: name: auto:improvement, name: area: models

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
Issue: Reduced query in a agent tool _run method
Labels: name: area: agent, name: auto:question, name: auto:improvement

### Issue you'd like to raise.

I am using Agent with OpenAI.Functions and have a Structured Tool:
```
class SearchSchema(BaseModel):
    """Inputs for get_current_stock_price"""
    country_filter: Optional[str] = Field(default="United Kingdom", description="The country for events") 

class EventsAPIWrapper(BaseModel):
    args_schema: Type[BaseModel] = SearchSchema
    
    def run(self, query: str, country_filter: str) -> str:
        """Run Events search and get page summaries."""
        print('QUERY: ', query, flush=True)
        print('COUNTRY: ', country_filter, flush=True)
```
For the user question: **Show me medititaion events in Mexico**
The output in cmd is:
```
QUERY: meditation
Country: Mexico
```


### Suggestion:

I need to have the following output:
```
QUERY: meditation in mexico
Country: Mexico
```
i.e. I do not want to remove the Country from the Agent Input for this tool.
How to achieve this?
Replaced instances of raising ValueError with raising ImportError.
Labels: name: lgtm, name: area: embeddings, name: auto:refactor

Refactored code to ensure consistent handling of ImportError. Replaced instances of raising ValueError with raising ImportError.

The choice of raising a ValueError here is somewhat unconventional and might lead to confusion for anyone reading the code. Typically, when dealing with import-related errors, the recommended approach is to raise an ImportError with a descriptive message explaining the issue. This provides a clearer indication that the problem is related to importing the required module.

@hwchase17 , @baskaryan , @eyurtsev 

Thanks
Aashish
ElasticsearchStore: Support max_marginal_relevance
Labels: name: area: vector store, name: auto:enhancement

### Feature request

Support Max marginal relevance on the clientside. Other vectorstores use the langchain lib to do re-ranking on clientside. add `fetch_k` to gather number of candidates to be retrieved. honour `k` to return only x number of documents. 

### Motivation

  Support a diverse set of results

### Your contribution

Will contribute feature on behalf of elastic
Update pydantic_compatibility.md
Labels: name: lgtm, name: auto:documentation

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
fixed spelling mistake and added missing bracket in parent_document_r…
Labels: name: lgtm, name: auto:documentation

…etriever.ipynb

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
DOC: Little error in the Chroma integration documentation
Labels: name: area: doc loader, name: area: vector store, name: auto:improvement, name: auto:documentation

### Issue with current documentation:

Hello! I'm currently developing using LangChain and Chroma and I've stumbled upon this line:
`View full [docs](https://docs.trychroma.com/reference/Collection) at docs. To access these methods directly, you can do ._collection_.method()`
Instead, you have to do `._collection.method()` . That is, without the last '_'.
At: [https://python.langchain.com/docs/integrations/vectorstores/chroma](https://python.langchain.com/docs/integrations/vectorstores/chroma)


### Idea or request for content:


For those of us that are just starting, an example would be pretty helpful. Something like this:
```
# Replace peek() with whatever method you'd like to use:
db = Chroma.from_documents(documents = docs, embedding = embedding_function)
db._collection.peek()
```

Also, I'd love to know how I could edit these kinds of things on my own, so they only have to be accepted after review. Any link to a tutorial would be really helpful.
Streaming with function calling feature
Labels: name: auto:question, name: auto:enhancement, name: area: models

### Issue you'd like to raise.

Hi everybody, I am using langchain and I want to use new feature "function_calling" of openai. Actually, my app worked when I add functions and function_call params into function apredict_message. But I want to use stream, currently when I add function_calling into my call (with stream = true), it does not work and worked when I remove function_calling. 
So I want to know that does langchain support function_calling + stream ?
Thanks all

### Suggestion:

_No response_
Use a submodule for pydantic v1 compat
Labels: name: auto:refactor, name: area: models

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
feat(llms): support ERNIE Embedding-V1 
Labels: name: area: embeddings, name: auto:enhancement

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
- Description: support [ERNIE Embedding-V1](https://cloud.baidu.com/doc/WENXINWORKSHOP/s/alj562vvu), which is part of ERNIE ecology
- Issue: None
- Dependencies: None
- Tag maintainer: @baskaryan
Add {language} in all template
Labels: name: auto:improvement, name: area: models

### Feature request

Actually, all prompt are in english. To generate a summary or answer a question in another language, all the templates need to be modified.
The modification can be :

```
prompt = PromptTemplate(
            template=re.sub(
                "CONCISE SUMMARY:",
                "CONCISE SUMMARY IN {language}:",
                summarize.stuff_prompt.prompt_template,
            ),
            input_variables=["text", "language"],
        )
```
The default "language" must be english.

### Motivation


It would be nice for all non-English speakers around the world to be able to simply appreciate the language. By default, English can be used.


### Your contribution

No contribution at this time.