delayedkarma/langchain-issues huggingface dataset:
Allow config propagation, Add default lambda name, Improve ergonomics of config passed in
Labels: name: auto:improvement

Makes it easier to do recursion using regular python compositional patterns 

```py
def lambda_decorator(func):
    """Decorate function as a RunnableLambda"""
    return runnable.RunnableLambda(func)

@lambda_decorator
def fibonacci(a, config: runnable.RunnableConfig) -> int:
    if a <= 1:
        return a
    else:
        return fibonacci.invoke(
            a - 1, config
        ) + fibonacci.invoke(a - 2, config)

fibonacci.invoke(10)
```
https://smith.langchain.com/public/cb98edb4-3a09-4798-9c22-a930037faf88/r

Also makes it more natural to do things like error handle and call other langchain objects in ways we probably don't want to support in `with_fallbacks()`

```py
@lambda_decorator
def handle_errors(a, config: runnable.RunnableConfig) -> int:
    try:
        return my_chain.invoke(a, config)
    except MyExceptionType as exc:
        return my_other_chain.invoke({"original": a, "error": exc}, config)
```

In this case, the next chain takes in the exception object. Maybe this could be something we toggle in `with_fallbacks` but I fear we'll get into uglier APIs + heavier cognitive load if we try to do too much there
DOC: One link in Chain->Diff cll methods is not working
Labels: name: area: doc loader, name: auto:documentation

### Issue with current documentation:

[This](https://python.langchain.com/docs/modules/chains/how_to/call_methods#:~:text=See%20an%20example%20here.) link is not working. 

<img width="1228" alt="image" src="https://github.com/langchain-ai/langchain/assets/84584929/19a6f523-8584-4338-80e0-ffe78c51714a">


### Idea or request for content:

_No response_
Query data base using natural language
Labels: name: auto:question, name: auto:documentation, name: area: models

### Issue you'd like to raise.

Where given a question in a natural language it has to convert to sql query and then operate on the sql data base then has to return the answer.
Earlier I did by using **SQLDatabaseChain**
But now I couldn't find this in your document. 
So what are all supported here for my use case?

### Suggestion:

_No response_
Our whole business is down. please help!! HuggingFaceBgeEmbeddings(model_name="BAAI/bge-large-en")
Labels: name: area: embeddings, name: auto:bug, name: stale, name: area: models

### System Info

Our startup using this line all across out scripts.

`HuggingFaceBgeEmbeddings(model_name="BAAI/bge-large-en")`

Today, someone updated langchain, everything has stopped, please help!!

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

We see if is in the docs?

https://api.python.langchain.com/en/latest/embeddings/langchain.embeddings.self_hosted_hugging_face.SelfHostedHuggingFaceEmbeddings.html



### Expected behavior

Should import and work.
add konko chat_model files
Labels: name: auto:enhancement, name: area: models

_Thank you to the LangChain team for the great project and in advance for your review. Let me know if I can provide any other additional information or do things differently in the future to make your lives easier 🙏 _

@hwchase17 please let me know if you're not the right person to review 😄 

This PR enables LangChain to access the Konko API via the chat_models API wrapper. 

Konko API is a fully managed API designed to help application developers:

1. Select the right LLM(s) for their application
2. Prototype with various open-source and proprietary LLMs
3. Move to production in-line with their security, privacy, throughput, latency SLAs without infrastructure set-up or administration using Konko AI's SOC 2 compliant infrastructure

_Note on integration tests:_ 
We added 14 integration tests. They will all fail unless you export the right API keys. 13 will pass with a KONKO_API_KEY provided and the other one will pass with a OPENAI_API_KEY provided. When both are provided, all 14 integration tests pass. If you would like to test this yourself, please let me know and I can provide some temporary keys.

### Installation and Setup

1. **First you'll need an API key**
2. **Install Konko AI's Python SDK**
    1. Enable a Python3.8+ environment
    
    `pip install konko`
    
3.  **Set API Keys**
    
          **Option 1:** Set Environment Variables
    
    You can set environment variables for
    
    1. KONKO_API_KEY (Required)
    2. OPENAI_API_KEY (Optional)
    
    In your current shell session, use the export command:
    
    `export KONKO_API_KEY={your_KONKO_API_KEY_here}`
    `export OPENAI_API_KEY={your_OPENAI_API_KEY_here} #Optional`
    
    Alternatively, you can add the above lines directly to your shell startup script (such as .bashrc or .bash_profile for Bash shell and .zshrc for Zsh shell) to have them set automatically every time a new shell session starts.
    
    **Option 2:** Set API Keys Programmatically
    
    If you prefer to set your API keys directly within your Python script or Jupyter notebook, you can use the following commands:
    
    ```python
    konko.set_api_key('your_KONKO_API_KEY_here')
    konko.set_openai_api_key('your_OPENAI_API_KEY_here') # Optional
    
    ```
    

### Calling a model

Find a model on the [[Konko Introduction page](https://docs.konko.ai/docs#available-models)](https://docs.konko.ai/docs#available-models)

For example, for this [[LLama 2 model](https://docs.konko.ai/docs/meta-llama-2-13b-chat)](https://docs.konko.ai/docs/meta-llama-2-13b-chat). The model id would be: `"meta-llama/Llama-2-13b-chat-hf"`

Another way to find the list of models running on the Konko instance is through this [[endpoint](https://docs.konko.ai/reference/listmodels)](https://docs.konko.ai/reference/listmodels).

From here, we can initialize our model:

```python
chat_instance = ChatKonko(max_tokens=10, model = 'meta-llama/Llama-2-13b-chat-hf')

```

And run it:

```python
msg = HumanMessage(content="Hi")
chat_response = chat_instance([msg])

```
ModuleNotFoundError: No module named 'langchain.schema.document'; 'langchain.schema' is not a package
Labels: name: area: embeddings, name: area: vector store, name: auto:bug, name: area: models

### System Info



Since we updated to the latest langchain we are also getting this issue. We are a startup, just launched, updated yesterday now all our apps don't work. First customer annoyed.

**Please, please help!**

Name: langchain
Version: 0.0.281

`    data_state_nsw_legisation_index_instance = FAISS.load_local("data_indexes/federal/federal_legislativeinstruments_inforce_index", embeddings)
  File "/opt/homebrew/lib/python3.10/site-packages/langchain/vectorstores/faiss.py", line 472, in load_local
    docstore, index_to_docstore_id = pickle.load(f)
ModuleNotFoundError: No module named 'langchain.schema.document'; 'langchain.schema' is not a package`

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

data_state_nsw_legisation_runner = RetrievalQA.from_chain_type(llm=ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0,
                                                openai_api_key=openai_api_key_value),
                                                chain_type="stuff",
                                                retriever=data_state_nsw_legisation_index_instance.as_retriever())


### Expected behavior

yesterday before we updated is was working perfectly, we updated to the latest version and it all stopped :(

**Please help!** the team is scrambling
HuggingFaceBgeEmbeddings Error, Please help!
Labels: name: area: embeddings, name: auto:bug

### System Info

Yesterday is works, someone accidentally update langchain now the whole platform is down.

We built the whole platform using his code all over the place. Now nothing works. 
We have around 50 models. All our models are build like this and we just went live as a startup. 

**We are scrabbling here guys. Please help us.**

`
from langchain.embeddings import HuggingFaceBgeEmbeddings

embeddings = HuggingFaceBgeEmbeddings(model_name="BAAI/bge-large-en")

news_instance = FAISS.load_local("federal_legislativeinstruments_inforce_index", embeddings)

data_state_nsw_legisation_index_instance = FAISS.load_local("data_indexes/federal/federal_legislativeinstruments_inforce_index", embeddings)

data_state_nsw_legisation_runner = RetrievalQA.from_chain_type(llm=ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0,
                                                openai_api_key=openai_api_key_value),
                                                chain_type="stuff",
                                                retriever=data_state_nsw_legisation_index_instance.as_retriever())
`

Please, please help. How to we refactor so it works. How team is going crazy to get it live again, Our very first customers are ringing us complaining. Please help.




### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

from langchain.embeddings import HuggingFaceBgeEmbeddings

embeddings = HuggingFaceBgeEmbeddings(model_name="BAAI/bge-large-en")

news_instance = FAISS.load_local("federal_legislativeinstruments_inforce_index", embeddings)

data_state_nsw_legisation_index_instance = FAISS.load_local("data_indexes/federal/federal_legislativeinstruments_inforce_index", embeddings)

data_state_nsw_legisation_runner = RetrievalQA.from_chain_type(llm=ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0,
                                                openai_api_key=openai_api_key_value),
                                                chain_type="stuff",
                                                retriever=data_state_nsw_legisation_index_instance.as_retriever())

### Expected behavior

How load the embedding like yesterday and all the time before.
Fix bug in MarkdownHeaderTextSplitter for codeblock
Labels: name: needs test, name: auto:bug

  - Description: The previous version of the MarkdownHeaderTextSplitter did not take into account the possibility of '#' appearing within code blocks, which caused segmentation anomalies in these situations. This PR has fixed this issue.
  - Issue: 
  - Dependencies: No
  - Tag maintainer: 
  - Twitter handle: 

cc @baskaryan @eyurtsev  @rlancemartin
docs: fix `tools/sqlite` page
Labels: name: auto:documentation

The `/docs/integrations/tools/sqlite` page is not about the  tool integrations. 
I've moved it into `/docs/use_cases/sql/sqlite`. 
`vercel.json` modified
As a result two pages now under the  `/docs/use_cases/sql/` folder. So the `sql` root page moved down together with `sqlite` page.
SystemMessage in create_pandas_dataframe_agent
Labels: name: area: agent, name: auto:bug

### System Info

For some reason SystemMessage does not work for me (agent ignores it). Here is my code:

```
system_message = SystemMessage(content="write response in uppercase")
agent_kwargs = {
    "system_message": system_message,
}

agent_func = create_pandas_dataframe_agent(
    llm,
    df,
    verbose=True,
    agent_type=AgentType.OPENAI_FUNCTIONS,
    agent_kwargs=agent_kwargs,
)

```


I tried to do with system_message directly but agent still ignores SystemMessage:

```
system_message = SystemMessage(content="write response in uppercase")
agent_func = create_pandas_dataframe_agent(
    llm,
    df,
    verbose=True,
    agent_type=AgentType.OPENAI_FUNCTIONS,
    system_message=system_message,
)
```

Also, I tried to use `system_message.context` instead of `system_message` but still no luck. 


Langchain version is 0.0.281


### Who can help?

@hwchase17 

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

let me know if additional info is needed

### Expected behavior

create_pandas_dataframe_agent should work with SystemMessage
docs: `integrations/memory` consistency
Labels: name: area: memory, name: auto:documentation

- updated titles and descriptions of the  `integrations/memory` notebooks into consistent and laconic format;
- removed `docs/extras/integrations/memory/motorhead_memory_managed.ipynb` file as a duplicate of the `docs/extras/integrations/memory/motorhead_memory.ipynb`;
- added `integrations/providers` Integration Cards for `dynamodb`, `motorhead`.
- updated `integrations/providers/redis.mdx` with links
- renamed several notebooks; updated `vercel.json` to reroute new names.
Add Support of Azure Cognitive Search for SelfQueryRetriever 
Labels: name: area: vector store, name: auto:enhancement

### Feature request

 Azure Cognitive Search already has [hybrid search functionality](https://python.langchain.com/docs/integrations/vectorstores/azuresearch#perform-a-hybrid-search) and it makes sense to add support of SelfQueryRetriever as well

### Motivation

 Azure Cognitive Search is production ready solution.

### Your contribution

I can help with testing of this feature
Disable loading jinja2 `PromptTemplate` from file.
Labels: name: auto:bug

jinja2 templates are not sandboxed and are at risk for arbitrary code execution. To mitigate this risk:
- We no longer support loading jinja2-formatted prompt template files.
- `PromptTemplate` with jinja2 may still be constructed manually, but the class carries a security warning reminding the user to not pass untrusted input into it.

Resolves #4394.

Resolves: CVE-2023-36281
Error when Instantiating GPT4AllEmbeddings
Labels: name: area: embeddings, name: auto:bug

### System Info

langchain  0.0.281, pydantic 1.10.9, gpt4all 1.0.9, Linux Gardua(Arch), Python 3.11.5

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [x] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

This is my code:

```python
from langchain.embeddings import GPT4AllEmbeddings

gpt4all_embd = GPT4AllEmbeddings()
```
I get this error:

Found model file at  /home/chigoma333/.cache/gpt4all/ggml-all-MiniLM-L6-v2-f16.bin
Invalid model file
Traceback (most recent call last):
  File "/home/chigoma333/Desktop/Program/test.py", line 3, in <module>
    gpt4all_embd = GPT4AllEmbeddings()
                   ^^^^^^^^^^^^^^^^^^^
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for GPT4AllEmbeddings
__root__
  Unable to instantiate model (type=value_error)

### Expected behavior

    I expected the GPT4AllEmbeddings instance to be created successfully without errors.
Install dev, lint, test, typing extra deps for linting steps.
Labels: name: area: doc loader, name: auto:nit

`mypy` cannot type-check code that relies on dependencies that aren't installed.

Eventually we'll probably want to install as many optional dependencies as possible. However, the full "extended deps" setup for langchain creates a 3GB cache file and takes a while to unpack and install. We'll probably want something a bit more targeted.

This is a first step toward something better.
Fix SQL search_path for Trino query engine
Labels: name: lgtm, name: auto:bug

This PR replaces the generic `SET search_path TO` statement by `USE` for the Trino dialect since Trino does not support `SET search_path`.  Official Trino documentation can be found [here](https://trino.io/docs/current/sql/use.html).

With this fix, the `SQLdatabase` will now be able to set the current schema and execute queries using the Trino engine.  It will use the catalog set as default by the connection uri.
Update amazon_comprehend_chain.ipynb
Labels: name: auto:documentation

Huggingface, HuggingFace -> Hugging Face

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
Extract abstracts from Pubmed articles, even if they have no extra label
Labels: name: lgtm, name: area: doc loader, name: auto:improvement

### Description
This pull request involves modifications to the extraction method for abstracts/summaries within the PubMed utility. A condition has been added to verify the presence of unlabeled abstracts. Now an abstract will be extracted even if it does not have a subtitle. In addition, the extraction of the abstract was extended to books.

### Issue
The PubMed utility occasionally returns an empty result when extracting abstracts from articles, despite the presence of an abstract for the paper on PubMed. This issue arises due to the varying structure of articles; some articles follow a "subtitle/label: text" format, while others do not include subtitles in their abstracts. An example of the latter case can be found at: [https://pubmed.ncbi.nlm.nih.gov/37666905/](url)

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
Issue: Lower requirements for PyYAML to 5.3.1 due to Cython bug
Labels: name: auto:improvement

### Issue you'd like to raise.

PyYAML has [known issues](https://github.com/yaml/pyyaml/issues/724) with version 5.4.1 and several other libraries are rolling back requirements to allow compatibility.

### Suggestion:

Lower requirement to allow PyYAML 5.3.1
RL Chain with VowpalWabbit
Labels: name: auto:enhancement, name: area: models

  - Description: This PR adds a new chain `rl_chain.PickBest` for learned prompt variable injection, detailed description and usage can be found in the example notebook added. It essentially adds a [VowpalWabbit](https://github.com/VowpalWabbit/vowpal_wabbit) layer before the llm call in order to learn or personalize prompt variable selections.

Most of the code is to make the API simple and provide lots of defaults and data wrangling that is needed to use Vowpal Wabbit, so that the user of the chain doesn't have to worry about it.

  - Dependencies: [vowpal-wabbit-next](https://pypi.org/project/vowpal-wabbit-next/),
     - sentence-transformers (already a dep)
     - numpy (already a dep)
  - tagging @ataymano who contributed to this chain
  - Tag maintainer: @baskaryan
  - Twitter handle: @olgavrou


Added example notebook and unit tests
Error message when using force_recreate on Qdrant
Labels: name: area: vector store, name: auto:bug

### System Info

```
langchain[qdrant]==0.0.281
qdrant-client==1.4.0
```

Qdrant server v1.4.1

### Who can help?

@agola11 @hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I started seeing this issue after using `force_recreate=True`, so for example this is the method I am using to populate my collection:
```python
qdrant.from_documents(
        docs,
        url=self.settings.QDRANT_API_URL,
        collection_name=collection,
        embedding=self.embeddings,
        force_recreate=True,
        replication_factor=3,
        timeout=60
)
```

And I keep getting the following error messages in my logs:
```
[2023-09-05T16:33:45.308Z WARN storage::content_manager::consensus_manager] Failed to apply collection meta operation entry with user error: Wrong input: Replica 6838680705292431 of shard 4 has state Some(Active), but expected Some(Initializing)
```

I found where this error is being raised on Qdrant: https://github.com/qdrant/qdrant/blob/383fecf64b6d97e4718deb2bf0f46422060e7e52/lib/collection/src/collection.rs#L339

I understand the issue might be in the Qdrant client or the Qdrant server itself.

### Expected behavior

This message doesn't make sense, and the data is being stored without any issues.
Implement NucliaDB vector store
Labels: name: area: vector store, name: auto:enhancement

# Description

This pull request allows to use the [NucliaDB](https://docs.nuclia.dev/docs/docs/nucliadb/intro) as a vector store in LangChain.

It works with both a [local NucliaDB instance](https://docs.nuclia.dev/docs/docs/nucliadb/deploy/basics) or with [Nuclia Cloud](https://nuclia.cloud).

# Dependencies

It requires an up-to-date version of the `nuclia` Python package.

@rlancemartin, @eyurtsev,  @hinthornw, please review it when you have a moment :)

Note: our Twitter handler is `@NucliaAI`