delayedkarma/langchain-issues huggingface dataset:
Fix `config` arg detection for wrapped lambdarunnable
Labels: name: lgtm, name: auto:bug, name: size:XS

**Description:**
When a RunnableLambda only receives a synchronous callback, this callback is wrapped into an async one since #13408. However, this wrapping with `(*args, **kwargs)` causes the `accepts_config` check at [/libs/core/langchain_core/runnables/config.py#L342](https://github.com/langchain-ai/langchain/blob/ee94ef55ee6ab064da08340817955f821dfa6261/libs/core/langchain_core/runnables/config.py#L342) to fail, as this checks for the presence of a "config" argument in the method signature. 

Adding a `functools.wraps` around it, resolves it.


**Issue:** 
None, writing this PR already took longer than the fix ;)

**Tag maintainer:** 
@nbondoux @efriis (participants of the original PR)
DOC: How to create a custom filtered retriever
Labels: name: area: doc loader, name: auto:question, name: auto:documentation

### Issue with current documentation:

I cant find documentattion about filtered retrievers, basically I want to fix this issue: https://github.com/langchain-ai/langchain/issues/14227

### Idea or request for content:

And according to other github issues, the way to fix it is with custom filtered retrievers.

So I tried the following:

```
class FilteredRetriever:
    def __init__(self, retriever, title):
        self.retriever = retriever
        self.title = title

    def retrieve(self, *args, **kwargs):
        results = self.retriever.retrieve(*args, **kwargs)
        return [doc for doc in results if doc['title'].startswith(self.title)]

filtered_retriever = FilteredRetriever(vector_store.as_retriever(), '25_1_0.pdf')

llm = AzureChatOpenAI(
    azure_deployment="chat",
    openai_api_version="2023-05-15",
)
    
retriever = vector_store.as_retriever(search_type="similarity",  kwargs={"k": 3})
chain = RetrievalQA.from_chain_type(llm=llm,
                                    chain_type="stuff",
                                    retriever=filtered_retriever,
                                    return_source_documents=True)

result = chain({"query": 'Can Colleagues contact their managers??'})

for res in result['source_documents']:
    print(res.metadata['title'])
```

However I get this error:
```
ValidationError: 1 validation error for RetrievalQA
retriever
  value is not a valid dict (type=type_error.dict)
```

but as I cant find documentation about it, I am not sure how to solve it
Filters dont work with Azure Search Vector Store retriever
Labels: name: area: vector store, name: auto:bug

### System Info

langchain                 0.0.342
langchain-core            0.0.7
azure-search-documents    11.4.0b8

Python: 3.10

### Who can help?

@hw

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

The following code works fine:

```
from langchain_core.vectorstores import VectorStore, VectorStoreRetriever

index_name: str = "langchain-vector-demo"
vector_store: AzureSearch = AzureSearch(
    azure_search_endpoint=vector_store_address,
    azure_search_key=vector_store_password,
    index_name="vector-1701341754619",
    embedding_function=embeddings.embed_query
)

res = vector_store.similarity_search(
    query="Can Colleagues contact their managers?", k=20, search_type="hybrid",  filters="title eq '25_1_0.pdf'")

The res object contains the chunkcs where title is 25_1_0.pdf' ONLY
```

However when using it with an LLM:

```
llm = AzureChatOpenAI(
    azure_deployment="chat",
    openai_api_version="2023-05-15",
)
    
retriever = vector_store.as_retriever(search_type="similarity", filters="title eq '25_1_0.pdf'", kwargs={"k": 3})
chain = RetrievalQA.from_chain_type(llm=llm,
                                    chain_type="stuff",
                                    retriever=retriever,
                                    return_source_documents=True)

result = chain({"query": 'Can Colleagues contact their managers??'})

for res in result['source_documents']:
    print(res.metadata['title'])
```

My output has chunks which dont respect the filter:

142_2_0.pdf
99_9_0.docx
99_9_0.docx
142_2_0.pdf

### Expected behavior

The answer generated with source_documents, should contain chunks which respects the given filters.
SelfQueryRetriever with OpenSearch generating invalid queries with Date type
Labels: name: area: vector store, name: auto:bug, name: integration: aws

### System Info

Python: 3.10
LangChain: 0.0.344
OpenSearch: Amazon OpenSearch Serverless Vector Engine

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I have loaded documents with metadata into OpenSearch which includes a date value. When generating a SelfQuerying Retriever the generated query specifies the date type as "date", and results in an invalid query being generated and sent to OpenSearch that then returns an error. I have set the AttributeInfo to "string", but even if it was "date" the query would be invalid.

```
document_content_description = "Feedback from users"
metadata_field_info = [
  AttributeInfo(
    name="timestamp",
    description="A string representing the date that the feedback was submitted in the 'yyyy-MM-dd' format",
    type="string",
  )
]
retriever = SelfQueryRetriever.from_llm(
  llm, vectorstore, document_content_description, metadata_field_info, verbose=True
)
retriever.get_relevant_documents("Summarize feedback submitted on 2023-06-01")
```

Output Logs:
```
"repr": "StructuredQuery(query=' ', filter=Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='timestamp', value={'date': '2023-06-01', 'type': 'date'}), limit=None)"
...
{
    "size": 4,
    "query": {
        "bool": {
            "filter": {
                "term": {
                    "metadata.timestamp": {
                        "date": "2023-06-01",
                        "type": "date"
                    }
                }
            },
        ...
}

{
    "error": {
        "root_cause": [
            {
                "type": "parsing_exception",
                "reason": "[term] query does not support [date]",
                "line": 1,
                "col": 76
            }
        ],
        "type": "x_content_parse_exception",
        "reason": "[1:76] [bool] failed to parse field [filter]",
        "caused_by": {
            "type": "parsing_exception",
            "reason": "[term] query does not support [date]",
            "line": 1,
            "col": 76
        }
    },
    "status": 400
}
```

### Expected behavior

LangChain should generate a valid OpenSearch query such as:


```
{
    "size": 4,
    "query": {
        "bool": {
            "filter": {
                "term": {
                    "metadata.timestamp.keyword": "2023-06-01"
                }
            },
        ...
}
```
Adds `llm_chain_kwargs` to `BaseRetrievalQA.from_llm`
Labels: name: lgtm, name: auto:improvement, name: area: models, name: size:XS

- **Description:** Adds `llm_chain_kwargs` to `BaseRetrievalQA.from_llm` so these can be passed to the LLM at runtime, 
- **Issue:** https://github.com/langchain-ai/langchain/issues/14216,
Feature: GitLab url from ENV
Labels: name: auto:improvement, name: size:S

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** add gitlab url from env, 
  - **Issue:** no issue,
  - **Dependencies:** no,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
add(feat): Text Embeddings by Cloudflare Workers AI
Labels: name: lgtm, name: area: embeddings, name: auto:enhancement, name: size:L

Add [Text Embeddings by Cloudflare Workers AI](https://developers.cloudflare.com/workers-ai/models/text-embeddings/). It's a new integration.
Trying to align it with its langchain-js version counterpart [here](https://api.js.langchain.com/classes/embeddings_cloudflare_workersai.CloudflareWorkersAIEmbeddings.html).
- Dependencies: N/A
- Done `make format` `make lint` `make spell_check` `make integration_tests` and all my changes was passed
add bedrock callback for token counting
Labels: name: auto:improvement, name: area: models, name: size:XL

  - **Description:** Bedrock API now returns [token usage](https://docs.aws.amazon.com/bedrock/latest/userguide/monitoring-cw.html) details across all hosted models which can be used to collect statistics on token usage. This can be used to implement a similar [token usage callback](https://python.langchain.com/docs/modules/model_io/llms/token_usage_tracking) as OpenAI.
  - **Issue:** #14120
  - **Dependencies:** None
  - **Tag maintainer:** @3coins

Token counting (as for OpenAI's callback) only works for the non-stream variant currently and hence the warning has been included.
Table data extraction from HTML files.
Labels: name: area: doc loader, name: auto:question, name: auto:improvement

### Issue you'd like to raise.

I am working on extracting data from HTML files. I need to extract table data to store in a data frame as a table. With the help of langchain document loader I can extract the data row wise but the headers of columns are not getting extracted. How to extract column headers along with the data.

### Suggestion:

_No response_
[Revised] Virtual column name for self-querying retrievers
Labels: name: area: vector store, name: auto:improvement, name: size:L

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->

Self query retrievers are powerful and it can be even stronger than before. Taking MyScale as an example, sometimes the user may add complex SQL function to a column name. So we add a `VirtualColumnName` for self-query retriever to reduce token usage with extra functionality.

1. a function that dynamically creates function mapped column names, which becomes handy when the user want to compare between current time / location or other attributes along with a column.
2. complex function call in plain string can now be replaced with a shorter nickname in prompt, which saves number of token for very long function names.
3. refined the logic in MyScale translator to meet this standard.

This change will not affect other self query retrievers as it preserves the original plain string interface under comparison. ~~The actual difference happens under `QueryTransformer` in `query_construct.parser` for `Lark` parser.~~

`Comparator.attribute` can now be either a string or a `VirtualColumnName`. ~~I have defined the default behaviour which stops the user to use `VirtualColumnName` under other vectorstores than MyScale.~~ Parser for `VirtualColumnName`s are chained with original structured query parser, using LCEL.

This functionality will boost the self-query retrievers as a bridge between text and simple SQL query. And we believe this will help users to expand their usage to this retriever. Other SQL vector database is theoretically compatible to this new feature as well.

Hope you find this interesting.
@hwchase17  @baskaryan @efriis 

Our contact on twitter: @MyScaleDB @mpsk_liu

=========

Updates:

* Separate from `StructuredQueryOutputParser` and created new `VirtualColumnParser`
* Rewritten with LCEL
* Won't hurt any inheritance / API signatures
Add `llm_kwargs` to `BaseRetrievalQA.from_llm`
Labels: name: auto:improvement, name: area: models

https://github.com/langchain-ai/langchain/blob/ee94ef55ee6ab064da08340817955f821dfa6261/libs/langchain/langchain/chains/llm.py#L71

In my humble opinion, I think `llm_kwargs` variable should be a parameter declared in the LLM class rather than the LLM Chain. One of the reasons for this, is that when you are declaring, for instance, a `RetrievalQA` chain `from_chain_type` or `from_llm` , you cannot specify these `llm_kwargs`.

You can workaround that by calling the inner `llm_chain` with (given that you're using a `combine_documents_chain`):

```python
chain.combine_documents_chain.llm_chain.llm_kwargs = {'test': 'test'}
```

But seems very odd that you have to do that. And the kwargs seems that it should be a responsibility of the LLM, not the LLM Chain.

Happy to be proven wrong! 🙂
BiliBiliLoader Credential No Sessdata error
Labels: name: area: doc loader, name: auto:bug

### System Info

Mac os
Python3.9

### Who can help?

@Jiaaming
I'm working on this issue.

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

when you run the BiliBiliLoader example from the official [docs](https://python.langchain.com/docs/integrations/document_loaders/bilibili)
```
loader = BiliBiliLoader(
        [
            "https://www.bilibili.com/video/BV1g84y1R7oE/",
        ]
    )
docs = loader.load()
print(docs)
```
will get
```
bilibili_api.exceptions.CredentialNoSessdataException.CredentialNoSessdataException: Credential 类未提供 sessdata 或者为空。

Process finished with exit code 1
```

The is because the original [bilibili_api](https://nemo2011.github.io/bilibili-api/#/get-credential) require a Credential to fetch the info from the video

### Expected behavior

Should return a Document object
```
[Document(page_content="Video Title:...,description:....)]
```
Support `openai_proxy` compatible with openai>1.0.0
Labels: name: area: embeddings, name: auto:improvement, name: size:L

 ### **Description:** 

`openai` v1 no longer supports the method of setting `openai.proxy`. If using `OPENAI_PRPXY`, the user needs to initialize an http_client.

 ### **Issue:**

https://github.com/langchain-ai/langchain/issues/13939
ConversationChain error with multiple inputs
Labels: name: auto:bug, name: area: models

### Issue you'd like to raise.

ValidationError: 2 validation errors for ConversationChain
advisor_summary
  extra fields not permitted (type=value_error.extra)
__root__
  Got unexpected prompt input variables. The prompt expects ['advisor_summary', 'history', 'input'], but got ['history'] as inputs from memory, and input as the normal input key. (type=value_error)
  
  How to resolve this error?

### Suggestion:

_No response_
Issue: How to avoid modifying user input and output in initialize_agent?
Labels: name: area: agent, name: auto:question, name: auto:improvement

### Issue you'd like to raise.

Hi, I am trying to use `initialize_agent` to leverage custom tools but when the agent is accepting it, it is modifying the user's input in its own way and causing error in the output. For example, in SQLTool, it is automatically converting it into SQL query which is wrong and ended up giving me wrong answer. I want to know how can I ensure raw input is getting passed into the `action_input` of `initialize_agent`! An example below:

User input: How many users have purchased PlayStation from New York in October?

Expected flow of execution will start like this:
```
> Entering new AgentExecutor chain...
    Action:
    ```
    {
      "action": "SQLTool",
      "action_input": {
        "raw_input": "How many users have purchased PlayStation from New York in October?"
      }
    }
    ```
```

Also, since I am using BaseTool in the custom tools, I had to ensure output format is in string - can we customize it like getting it in JSON format?

### Suggestion:

_No response_
Issue: I'd like to use Hugging Face's Chat UI frontend with LangChain.
Labels: name: auto:question, name: area: models

### Issue you'd like to raise.

I'd like to use Hugging Face's Chat UI frontend with LangChain.

https://github.com/huggingface/chat-ui

But it looks like the Chat UI is only available through Hugginf Face's Text Generation Inference endpoint. 

https://github.com/huggingface/chat-ui/issues/466

How can I serve the chain I have configured with LangChain in TGI format so I can use Chat UI?

Thank you in advance.

### Suggestion:

_No response_
Modify LLMs/Anyscale work with OpenAI API v1
Labels: name: auto:improvement, name: area: models, name: size:L

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
- **Description:** 
1. Modify LLMs/Anyscale to work with OAI v1
2. Get rid of openai_ prefixed variables in Chat_model/ChatAnyscale
3. Modify `anyscale_api_base` to `anyscale_base_url` to follow OAI name convention (reverted)
moved semadb.mdx file
Labels: name: lgtm, name: auto:documentation, name: size:XS

SemaDB.mdx file was placed with additional sub-folder: `https://python.langchain.com/docs/integrations/providers/providers/semadb`
- Moved file to the `https://python.langchain.com/docs/integrations/providers/semadb`
- Added a redirect for the file URL
ConversationalRetrievalChain returns sources to questions without context
Labels: name: auto:bug, name: area: models

### Issue you'd like to raise.

Hi,

I am wondering if anyone has a work around using ConversationalRetrievalChain to retrieve documents with their sources, and prevent the chain from returning sources for questions without sources.

query = "How are you doing?"
result = chain({"question": query, "chat_history": chat_history})
result['answer']

"""
I'm doing well, thank you.

SOURCES: /content/xxx.pdf
"""

### Suggestion:

SOURCES:
Allow base_store to be used directly with MultiVectorRetriever
Labels: name: lgtm, name: area: vector store, name: auto:improvement, name: size:L

Allow users to pass a generic `BaseStore[str, bytes]` to MultiVectorRetriever, removing the need to use the `create_kv_docstore` method. This encoding will now happen internally.

@rlancemartin @eyurtsev
Prefer byte store interface for Upstash BaseStore to match other Redis
Labels: name: lgtm, name: auto:improvement, name: integration: redis, name: size:M

If we are not going to make the existing Docstore class also implement `BaseStore[str, Document]`, IMO all base store implementations should always be `[str, bytes]` so that they are more interchangeable.

CC @rlancemartin @eyurtsev
Amazon Kendra: Full answer text not returned for FAQ type answers
Labels: name: area: vector store, name: auto:bug, name: integration: aws

### System Info

Langchain version: 0.0.342
Python version: 3.9
OS: Mac OS

### Who can help?

@3coins 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Use any Kendra index that contains FAQ style answers and try to query them as a retriever.

### Expected behavior

### Background

When using `AmazonKendraRetriever`, `get_relevant_documents` first calls the `retrieve` API and if nothing is returned, it calls the `query` API (see [here](https://github.com/langchain-ai/langchain/blob/0bdb4343838c4513d15cd9702868adf6f652421c/libs/langchain/langchain/retrievers/kendra.py#L391-L399)).

The `query` API has the capability to return answers from not just documents but also Kendra's FAQs (as described in [API docs for Kendra](https://docs.aws.amazon.com/kendra/latest/dg/query-responses-types.html#response-types)).

### Issue

While Kendra normally only returns snippets, it does return full text for FAQs. The problem is that it is ignored by langchain today hence return snippet for FAQs also. The problem lies in these line of code:

https://github.com/langchain-ai/langchain/blob/0bdb4343838c4513d15cd9702868adf6f652421c/libs/langchain/langchain/retrievers/kendra.py#L225-L244

The Key `AnswerText` is always assumed to be at the 0th index when it can often be in fact on the 1st index. No such assumption can made also based on the [documentation](https://docs.aws.amazon.com/kendra/latest/APIReference/API_QueryResultItem.html#API_QueryResultItem_Contents) of the `QueryResultItem` structure.

For the Kendra index I am testing with, this is indeed the case which is how I discovered the bug. 

This is easily fixable where we loop over all indices to search for the key `AnswerText`. I can make the PR if this is deemed as the desired behavior.

Side note: I am happy to add some tests for `kendra.py` which I realized has no testing impemented.