delayedkarma/langchain-issues huggingface dataset:
add support for weaviate vectorstore using weaviate's v4 beta client
Labels: name: area: vector store, name: auto:enhancement, name: size:XXL

Hi there, a few days ago I created #14531 and asked to add support for Weaviate's new v4 python client which is currently in beta. As I mentioned in that issue, I started working on it myself and I believe I have created a working weaviate vectorstore using their new client.

## Changelog

* Uses the new weaviate-client **v4** with updated syntax and functions. The new client is faster and safer to use because of Weaviate's new [gRPC API](https://weaviate.io/developers/weaviate/api/grpc).
* Added a new function to support [BM25 keyword search](https://weaviate.io/developers/weaviate/search/bm25).
* Changed the delete function to use batch for faster execution (there was a [TODO](https://github.com/langchain-ai/langchain/blob/b9ef92f2f4bb1e4322dff040c4474c0922fbf546/libs/community/langchain_community/vectorstores/weaviate.py#L526) for it)
* The function names and attributes have remained the same from the original weaviate vectorstore.

As this version of weaviate's client is still in beta, I think it is wise to add this as a new vectorstore, without replacing the existing one. For this reason, I did not change the original weaviate.py vectorstore file, but I added a new one called weaviate_v4.py. This way users using the original weaviate vectorstore will not be affected, but they'll be able to try the new vectorstore if they wish.

The are no dependencies other than having installed weaviate-client pre-release >= v4.
I also added a file in docs/docs, which is the same as the original one for weaviate but with the new client's structure.
I ran `make format`, `make lint` and `make test` before committing. :)

I did not create a unit test file because I saw there are almost none for other vectorstores. Let me know if it's essential and I may try to create one.

My Twitter (or X) handle is @mantzouranidis. Thanks for your work, I hope my contribution helps.
DALLEAPIWrapper to support Azure OpenAI
Labels: name: auto:enhancement, name: area: models

### Feature request

Azure OpenAI now previews the DALLE-3 model.  Today, DALLEAPIWrapper only supports the openai API.  

### Motivation

My customers are using Azure OpenAI and would like to use DALL-E-3 in their solutions.  

### Your contribution

PR may not be possible but I'd like to help anyway I can.
Add support for Vertex AI Endpoint
Labels: name: auto:enhancement, name: area: models

### Feature request

Add first-class support for Vertex AI Endpoints in Langchain. This would involve providing a similar interface to the existing SageMakerEndpoint class, allowing users to easily connect to and interact with Vertex AI Endpoints.

### Motivation

Although VertexAIModelGarden already exist, there may be instances where users require custom models with unique input and output formats. To address this need, a more versatile class could be developed, upon which VertexAIModelGarden could be built. This would allow for seamless integration of custom models without compromising the functionality of the existing Model Garden class.

### Your contribution

The implementation taking inspiration from SageMakerEndpoint if pertinent.
docs `API Reference` top menu update
Labels: name: auto:documentation, name: size:XS

An issue with the top menu. It is not in the right priority order: "API/Core/Community/Experimental/Python Docs".
The second issue is that the "API" menu item is actually not for the whole API docs, but only for the "langchain" package.
I've updated menu to: "Core/Langchain/Community/Experimental/Docs"
Now, "Core" is the top namespace, then "Langchain".

Notes:
- "Langchain", not "LangChain" (the capital "C" inside). Reason: namespaces are in the lower case. So, here the capitalization helps to visually separate one menu item from another. "LangChain" messes with this visual comprehension.
- The last item, "Python Docs" replaced with "Docs". Reason: A 2-word item makes an impression of two menu items. Now we have more menu items and all but this are 1-word items. This single 2-word item looks like two menu items. Since we don't have a link to the JS Docs, removing "Python" is OK.
- "Docs" menu item is bold. It visually separates Docs from other menu items. Other menu items are linked to API Refernce, but Docs is linked to Documentation.
changed default for VertexAIEmbeddings
Labels: name: area: embeddings, name: auto:improvement, name: size:S


Replace this entire comment with:
  - **Description:** @kurtisvg has raised a point that it's a good idea to have a fixed version for embeddings (since otherwise a user might run a query with one version vs a vectorstore where another version was used). 
  In order to avoid breaking changes, I'd suggest to give users a warning, and make a `model_name` a required argument in 1.5 months.
\Fix tool_calls message merge
Labels: name: lgtm, name: auto:bug, name: area: models, name: size:L

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
community[patch]: Release 0.0.2
Labels: name: auto:improvement, name: size:L

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
SlackToolkit() does not support proxy configuration
Labels: name: auto:bug, name: area: models

### System Info

langchain Version: 0.0.348
Python: 3.9.16

Docs suggests to use proxy entry as follows but it does not work:

from slacktoolkit import SlackToolkit

# Proxy settings
proxies = {
    'http': 'http://proxy.example.com:8080',
    'https': 'https://proxy.example.com:8080'
}

# Initialize SlackToolkit with proxy
slack_toolkit = SlackToolkit(proxies=proxies)



### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [x] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

# Proxy settings
proxies = {
    'http': 'your_proxy',
    'https': 'your_proxy'
}

# Initialize SlackToolkit with proxy
toolkit = SlackToolkit(proxies=proxies)
tools = toolkit.get_tools()
llm = OpenAI(temperature=0)

agent = initialize_agent(
    tools=toolkit.get_tools(),
    llm=llm,
    verbose=True,
    agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,
)
agent.run("Send a greeting to my coworkers in the #slack1 channel. Your name is chatbot. Set the sender name as chatbot.")


### Expected behavior

slack message sent in #slack1 channel
adding vectorstore_kwarg attribute to search_similarity function
Labels: name: lgtm, name: area: vector store, name: auto:improvement, name: size:XS

  - **Description:** the ability to add all extra parameter of vectorstore and using them SemanticSimilarityExampleSelector.
  - **Issue:** #14583
  - **Dependencies:** no dependensies
  - **Tag maintainer:** 
  - **Twitter handle:** @AmirMalekiz
docs: update multi_modal_RAG_chroma.ipynb
Labels: name: lgtm, name: area: doc loader, name: auto:documentation, name: integration: chroma, name: size:XS

seperate -> separate

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Add cohere librarian template
Labels: name: lgtm, name: auto:enhancement, name: area: models, name: size:L

Adding the example I build for the Cohere hackathon.

It can:

use a vector database to reccommend books

<img width="840" alt="image" src="https://github.com/langchain-ai/langchain/assets/144115527/96543a18-217b-4445-ab4b-950c7cced915">

Use a prompt template to provide information about the library

<img width="834" alt="image" src="https://github.com/langchain-ai/langchain/assets/144115527/996c8e0f-cab0-4213-bcc9-9baf84f1494b">

Use Cohere RAG to provide grounded results

<img width="822" alt="image" src="https://github.com/langchain-ai/langchain/assets/144115527/7bb4a883-5316-41a9-9d2e-19fd49a43dcb">
WatsonxLLM updates/enhancements
Labels: name: auto:enhancement, name: auto:documentation, name: area: models, name: size:L

  - **Description:** updates/enhancements to IBM [watsonx.ai](https://www.ibm.com/products/watsonx-ai) LLM provider (prompt tuned models and prompt templates deployments support)
  - **Dependencies:** [ibm-watsonx-ai](https://pypi.org/project/ibm-watsonx-ai/),
  - **Tag maintainer:** : @hwchase17 , @eyurtsev , @baskaryan 
  - **Twitter handle:** details in comment below.

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally. ✅
QaWithSources default prompt is massive
Labels: name: auto:improvement, name: area: models

### System Info

Not relevant

### Who can help?

@hwchase17
@agola11

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Use the QA With Sources chain with the default prompt. If the chain type is `stuff` or `map_reduce` the default prompts used are [this](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/chains/qa_with_sources/stuff_prompt.py) and [this](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/chains/qa_with_sources/map_reduce_prompt.py) respectively. These files are massive, and easily adds a 1000+ tokens to every request. In models like PALM2, there's barely any tokens left for other questions.

### Expected behavior

These prompts should be much more short, even if as a sample input.
Sources returned in QaWithSources can be optimised
Labels: name: area: vector store, name: auto:improvement, name: area: models

### Feature request

As of the current implementation of the QA with sources chain, if `return_source_documents` is set to `True`, [all the sources](https://github.com/langchain-ai/langchain/blob/76905aa043e3e604b5b34faf5e91d0aedb5ed6dd/libs/langchain/langchain/chains/qa_with_sources/base.py#L165) that are retrieved from the vector DB are returned. The `sources` field returns a list of file names that were used by the LLM. I propose we could do something like this
1. Assign each `Document` a unique UUID as the `source` before passing it to the LLM.
2. Once the LLM returns the relevant sources, we can backmap this to the actual `Document`s that were used by the LLM as opposed to getting just the filename.



### Motivation

This information seems vastly more useful than the entire response from the vector DB. For our current use cases, we've ended up overriding these functions to add this functionality.

### Your contribution

I can raise a PR adding this if this is something that you'd find useful as well.

An additional issue to report, the `_split_sources` [function](https://github.com/langchain-ai/langchain/blob/76905aa043e3e604b5b34faf5e91d0aedb5ed6dd/libs/langchain/langchain/chains/qa_with_sources/base.py#L124) splits at the first instance of `SOURCE` which seems to be a bit problematic. I can fix this to split at the last occurence.
Fix detection of Amazon OpenSearch Serverless
Labels: name: area: vector store, name: auto:bug, name: integration: aws, name: size:M

- **Description:** Fixes detection Amazon OpenSearch Serverless (AOSS) when using the `RequestsAWSV4SignerAuth` utility from [opensearch-py][1] for authentication. Keeps compatibility with using [requests-aws4auth][2].
- **Issue:** #14129 
- **Dependencies:** [opensearch-py][3], [requests-aws4auth][2] (for testing purposes only)
- **Tag maintainer:** @naveentatikonda @eyurtsev 

[1]: https://github.com/opensearch-project/opensearch-py/blob/c4e53d444aef698e08801bfccd55a2478721a97c/opensearchpy/helpers/signer.py#L73C21-L73C21
[2]: https://pypi.org/project/requests-aws4auth/
[3]: https://pypi.org/project/opensearch-py/
Caching with Bedrock does not distinguish models or params
Labels: name: auto:bug, name: area: models

### System Info

Python 3.9.12, LangChain 0.0.346

### Who can help?

@agola11 @3coins 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

When you use caching with LangChain, it does not distinguish different LLM models. For example, the response for LLama2 was used for a prompt for Claude 2. 

```
def ask_name(model_id):
    question = 'what is your name?'
    bedrock = Bedrock(model_id=model_id, model_kwargs={'temperature': 0.1})

    print('me: ' + question)
    t0 = datetime.datetime.now()
    print(f'{bedrock.model_id}: ' + bedrock(question).strip())
    print('({:.2f} sec)'.format((datetime.datetime.now() - t0).total_seconds()))
    print()

model_ids = ['meta.llama2-70b-chat-v1','anthropic.claude-v2',]
for model_id in model_ids:
    ask_name(model_id)
    ask_name(model_id)
```
==>
```
me: what is your name?
meta.llama2-70b-chat-v1: Answer: My name is LLaMA, I'm a large language model trained by a team of researcher at Meta AI.
(2.24 sec)

me: what is your name?
meta.llama2-70b-chat-v1: Answer: My name is LLaMA, I'm a large language model trained by a team of researcher at Meta AI.
(0.00 sec)

me: what is your name?
anthropic.claude-v2: Answer: My name is LLaMA, I'm a large language model trained by a team of researcher at Meta AI.
(0.00 sec)
```

This is because of https://github.com/langchain-ai/langchain/blob/db6bf8b022c17353b46f97ab3b9f44ff9e88a488/libs/langchain/langchain/llms/bedrock.py#L235

```
    @property
    def _identifying_params(self) -> Mapping[str, Any]:
        """Get the identifying parameters."""
        _model_kwargs = self.model_kwargs or {}
        return {
            **{"model_kwargs": _model_kwargs},
        }
```

My current workaround is subclassing the `Bedrock` class:

```
class MyBedrock(Bedrock):
    @property
    def _identifying_params(self) -> Mapping[str, Any]:
        """Get the identifying parameters."""
        return {
            'model_id': self.model_id,
            **BedrockBase._identifying_params.__get__(self)
        }
```
This seems working.

### Expected behavior

LLama 2 should replay with "My name is LLaMa..." while Claude 2 should reply with "My name is Claude."
Unable to dump/load custom classes since langchain_core 0.0.13
Labels: name: auto:bug

### System Info

python: 3.11
langchain: 0.0.347-0.0.349
langchain_core: 0.0.12, 0.0.13

### Who can help?

@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```python
import langchain_core.load

class Custom(langchain_core.load.Serializable):
    @classmethod
    def is_lc_serializable(cls) -> bool:
        return True

out = langchain_core.load.dumps(Custom())

langchain_core.load.loads(
    out,
    valid_namespaces=["langchain", "__main__"],
)
```



### Expected behavior

I'm expecting it's possible to make a custom class serializable, but since langchain_core 0.0.13 the `valid_namespaces` is effectively ignored as it relies on a whitelist of what can be serialized `SERIALIZABLE_MAPPING`.

So I get the error:

> ValueError: Trying to deserialize something that cannot be deserialized in current version of langchain-core: ('__main__', 'Custom')

Triggered in [load.py#L68](https://github.com/langchain-ai/langchain/blob/v0.0.349/libs/core/langchain_core/load/load.py#L68)

--- 

I'm not sure if serialization was ever intended to be part of the public API, but I've found it convenient to be able to make my custom parts of chains also abide by the serialization protocol and still be able to dump/load my chains
OutputFixingParser should use callbacks
Labels: name: auto:improvement

### System Info

Python 3.9.13, LangChain 0.0.347, Windows 10

### Who can help?

@hwchase17 @agola

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [X] Callbacks/Tracing
- [ ] Async

### Reproduction

Running this snippet:
```python
answers = (answer for answer in ["answer1", "answer2"])

class CustomLLM(LLM):
    def _llm_type(self) -> str:
        return "custom"

    def _call(self, prompt: str, **_) -> str:
        return next(answers)

class CustomOutputParser(BaseOutputParser[str]):
    reject_output: bool = True

    def parse(self, text: str) -> str:
        if self.reject_output:
            self.reject_output = False
            raise OutputParserException(f"Parsing failed")
        return text

    def get_format_instructions(self) -> str:
        return "format instructions"

class CustomCallbackHandler(BaseCallbackHandler):
    def on_llm_end(self, response: LLMResult, **_):
        print(f"received LLM response: {response}")

llm = CustomLLM()
chain = LLMChain(
    llm=llm,
    prompt=PromptTemplate.from_template("Testing prompt"),
    output_key="chain_output",
    verbose=True,
    output_parser=OutputFixingParser.from_llm(llm, CustomOutputParser()),
)

result = chain({}, callbacks=[CustomCallbackHandler()])
print(f"Chain result is {result}")

```
produces the following output:
```
> Entering new LLMChain chain...
Prompt after formatting:
Testing prompt
received LLM response: generations=[[Generation(text='answer1')]] llm_output=None run=None

> Finished chain.
Chain result is {'chain_output': 'answer2'}


```

### Expected behavior

The output line `received LLM response: generations=[[Generation(text='answer1')]] llm_output=None run=None` should also appear with `answer2`, because that one is also generated while running the chain that was configured to use the `CustomCallbackHandler`.

The callbacks should also be used in the chain that is created inside the OutputFixingParser. In my opinion, the chain doing the retry, should have the overall chain as a parent in the callback methods (`on_chain_start()` an so on).
fix a bug in RedisNum filter againt value 0
Labels: name: lgtm, name: area: vector store, name: auto:bug, name: integration: redis, name: size:XS


  - **Description:** There is a bug in RedisNum filter that filter towards value 0 will be parsed as "*". This is a fix to it.
  - **Issue:** NA
  - **Dependencies:** NA
  - **Tag maintainer:** NA
  - **Twitter handle:** NA
Issue: Getting error while integrating Confluence Spaces including attachments
Labels: name: area: doc loader, name: auto:bug

### Issue you'd like to raise.


below is my code 
`        confluence_url = config.get("confluence_url", None)
        username = config.get("username", None)
        api_key = config.get("api_key", None)
        space_key = config.get("space_key", None)
        documents = []

        embedding = OpenAIEmbeddings()
        loader = ConfluenceLoader(
            url=confluence_url,
            username=username,
            api_key=api_key
        )
        for space_key in space_key:
            documents.extend(loader.load(space_key=space_key,include_attachments=True,limit=100))`
The error I am getting:
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.media.atlassian.com/file/UNKNOWN_MEDIA_ID/binary?token=sometoken&name=Invalid%20file%20id%20-%20UNKNOWN_MEDIA_ID&max-age=2940

### Suggestion:

_No response_