delayedkarma/langchain-issues huggingface dataset:
Better synergy with `poetry` groups and extras
Labels: name: auto:improvement

### Feature request

Currently when getting started developing, there is:
- `extras`: `extended_testing`
- `poetry`: `dev`, `test`, `test_integration`

There is also an annoying `rapidfuzz` error one can hit: https://github.com/langchain-ai/langchain/issues/12237

Can we have the poetry groups reusing `extras`?

### Motivation

Less moving parts, and a more clear installation workflow

### Your contribution

I propose:

- Removing `dev` group
- Having the `test` group install the `extended_testing` extra
DOC: TextGen (Text Generation Web UI) - the code no longer works.
Labels: name: area: doc loader, name: auto:bug, name: auto:documentation

### Issue with current documentation:

I believe the Oobabooga Text Generation Web UI API was rewritten, causing the code on the TextGen page of the Langchain docs to stop working.

e.g.: the way the code handles talking to a ws: causes a 403.  I can execute API calls that work well, e.g.: curl http://127.0.0.1:5000/v1/chat/completions \
>   -H "Content-Type: application/json" \
>   -d '{
>     "messages": [
>       {
>         "role": "user",
>         "content": "Hello! Who are you?"
>       }
>     ],
>     "mode": "chat",
>     "character": "Example"
>   }'
works.

while llm_chain.run(question)  returns a 403 (failed handshake).

### Idea or request for content:

It would be awesome if this would be fixed.  If not, please pull the page.
Moderation: migrate to new openai api
Labels: name: auto:bug, name: area: models, name: size:M

Fixes issue #13685 : OpenAIModerationChain still uses the old OpenAI API.
### Description
The following snippet:
```python
from langchain.chains import OpenAIModerationChain
from openai import moderations

print(OpenAIModerationChain(client=moderations).run("i want to kill you"))
```
Gives the following error :
```
Traceback (most recent call last):
  File "moderate.py", line 4, in <module>
    print(OpenAIModerationChain(client=moderations).run("i want to kill you"))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\Lib\site-packages\langchain\chains\base.py", line 507, in run
    return self(args[0], callbacks=callbacks, tags=tags, metadata=metadata)[
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\Lib\site-packages\langchain\chains\base.py", line 312, in __call__
    raise e
  File "\Lib\site-packages\langchain\chains\base.py", line 306, in __call__
    self._call(inputs, run_manager=run_manager)
  File "\Lib\site-packages\langchain\chains\moderation.py", line 94, in _call
    results = self.client.create(text)
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "\Lib\site-packages\openai\lib\_old_api.py", line 39, in __call__
    raise APIRemovedInV1(symbol=self._symbol)
openai.lib._old_api.APIRemovedInV1:

You tried to access openai.Moderation, but this is no longer supported in openai>=1.0.0 - see the README at https://github.com/openai/openai-python for the API.

You can run `openai migrate` to automatically upgrade your codebase to use the 1.0.0 interface.

Alternatively, you can pin your installation to the old version, e.g. `pip install openai==0.28`

A detailed migration guide is available here: https://github.com/openai/openai-python/discussions/742
```
Create hi
Labels: name: auto:nit, name: size:XS

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Adding `BaseChatMessageHistory.__str__`
Labels: name: area: memory, name: lgtm, name: auto:improvement, name: size:S

Increasing the usability of `BaseChatMessageHistory` by implementing:
- ~`add_system_message`: for parity with human and AI messages~
- `__str__` for easy debugging

~I also added `add_system_message` to the one subclasser that changes the adders~
Mask API key for Minimax LLM
Labels: name: lgtm, name: auto:improvement, name: area: models, name: size:M

  - **Description:** Added masking for the API key for Minimax LLM + tests inspired by https://github.com/langchain-ai/langchain/pull/12418.
  - **Issue:** the issue # fixes https://github.com/langchain-ai/langchain/issues/12165
  - **Dependencies:** this fix is dependent on Minimax instantiation fix which is introduced in https://github.com/langchain-ai/langchain/pull/13439, so merge this one after.
  - **Tag maintainer:** @eyurtsev
Allow not enforcing function usage when a single function is passed to openai function executable
Labels: name: lgtm, name: auto:improvement, name: area: models, name: size:S


  - **Description:** allows not enforcing function usage when a single function is passed to an openAI function executable (or corresponding legacy chain). This is a desired feature in the case where the model does not have enough information to call a function, and needs to get back to the user. 
  - **Issue:** N/A
  - **Dependencies:** N/A
  - **Tag maintainer:** N/A
ci matrix
Labels: name: auto:refactor, name: size:L

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
read_file function implemented in tools
Labels: name: area: agent, name: auto:improvement, name: size:XS

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** Read File function implemented, 
  - **Issue:** not applicable
  - **Dependencies:** no dependencies
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Add custom async generator example
Labels: name: lgtm, name: area: doc loader, name: auto:documentation, name: size:M

<img width="1172" alt="Screenshot 2023-12-05 at 11 19 16 PM" src="https://github.com/langchain-ai/langchain/assets/10000925/6b0fbd70-9f6b-4f91-b494-9e88676b4786">
Supporting LLM Models over REST endpoints
Labels: name: auto:enhancement, name: area: models, name: size:L

**Description** 

With multiple LLMs being developed by various organizations, and in-house LLMs developed for internal consumption, we at Walmart want to expose all these LLMs over simple REST endpoint. This enables each team to independently develop their LLMs and their API signatures - which can then be easily exposed to users via LangChain. This PR adds the support to serve LLMs over REST endpoints. 

For the initial PR, this is a simple implementation of Sync HTTP REST call. Based on approval and interests, we would be enhancing this further with support for Async HTTP and Streaming HTTP as well in future PRs.

**Dependencies**
This is built on top of `requests`. If `httpx` is available on the PYTHONPATH, then httpx is preferred over requests.

**Twitter Handles**
@daichi_majumdar
Fixes issue-14295: Correctly pass along the kwargs
Labels: name: lgtm, name: area: vector store, name: auto:bug, name: size:XS


  - **Description:** Update code to correctly pass the kwargs 
  - **Issue:** #14295 
  - **Dependencies:**  - 
  - **Tag maintainer:** 

<--
If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->

#issue-14295
OpenSearchVectorSearch add_texts does wraps kwargs
Labels: name: area: vector store, name: auto:bug

### System Info

langchain==0.0.344
langchain-core==0.0.8

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [x] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

The code incorrectly calls __add with  kwargs=kwargs

```python
        return self.__add(
            texts,
            embeddings,
            metadatas=metadatas,
            ids=ids,
            bulk_size=bulk_size,
            kwargs=kwargs,
        )
```

This results that the _add method has a kwargs dict which contains a key 'kwargs' and all provided parameters (eg: engine="faiss") are not picked up...




### Expected behavior

The code does not mistakingly wraps the kwargs, it does it correctly in the add_embeddings:

```python
        return self.__add(
            list(texts),
            list(embeddings),
            metadatas=metadatas,
            ids=ids,
            bulk_size=bulk_size,
            **kwargs,
        )
```
Update docs
Labels: name: lgtm, name: area: doc loader, name: auto:documentation, name: size:S

### Description

Fixed 3 doc  issues:

1. `ConfigurableField ` needs to be imported in `docs/docs/expression_language/how_to/configure.ipynb`
2. use `error` instead of `RateLimitError()` in `docs/docs/expression_language/how_to/fallbacks.ipynb`
3. I think it might be better to output the fixed json data(when I looked at this example, I didn't understand its purpose at first, but then I suddenly realized):
<img width="1219" alt="Screenshot 2023-12-05 at 10 34 13 PM" src="https://github.com/langchain-ai/langchain/assets/10000925/7623ba13-7b56-4964-8c98-b7430fabc6de">
Mask API key for ChatOpenAI based chat_models
Labels: name: auto:improvement, name: area: models, name: size:L

  - **Description**: Mask API key for ChatOpenAi based chat_models (openai, azureopenai, anyscale, everlyai).
Made changes to all chat_models that are based on ChatOpenAI since all of them assumes that openai_api_key is str rather than SecretStr.
  - **Issue:**: #12165 
  - **Dependencies:**  N/A
  - **Tag maintainer:** @eyurtsev
  - **Twitter handle:** N/A
DOC: Cookbook entry for RAG contains older syntax
Labels: name: area: doc loader, name: auto:documentation

### Issue with current documentation:

Hey there,

the `_format_chat_history` function [in the RAG cookbook entry](https://python.langchain.com/docs/expression_language/cookbook/retrieval) contains some older syntax. As far as I understand, the cookbook is supposed to provide examples for the newest Langchain version, though.

```
    for dialogue_turn in chat_history:
        human = "Human: " + dialogue_turn[0]
        ai = "Assistant: " + dialogue_turn[1]
```

should be:

```
    for dialogue_turn in chat_history:
        human = "Human: " + dialogue_turn[0].content
        ai = "Assistant: " + dialogue_turn[1].content
```

Thank you. :)

### Idea or request for content:

_No response_
Update gitlab.py to support On Prims GitLab URL's
Labels: name: lgtm, name: auto:improvement, name: size:XS

Gitlab defaults to public URL : https://gitlab.com which is preventing to use self hosted Gilab instances. Add a Support to pass self hosted URL by setting GITLAB_URL env variable.

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
feat(embeddings): text-embeddings-inference
Labels: name: lgtm, name: area: embeddings, name: auto:improvement, name: auto:documentation, name: size:L

- **Description:** Added a notebook to illustrate how to use `text-embeddings-inference` from huggingface. As `HuggingFaceHubEmbeddings` was using a deprecated client, I made the most of this PR updating that too.

- **Issue:** #13286 

- **Dependencies**: None

- **Tag maintainer:** @baskaryan
DOC: Why use LCEL  ModuleNotFoundError: No module named 'langchain_core'
Labels: name: auto:documentation

### Issue with current documentation:
In this Announcements (https://github.com/langchain-ai/langchain/discussions/categories/announcements)  [LangChain Core #13823
](https://github.com/langchain-ai/langchain/discussions/13823) hwchase17 say he  

> TL;DR: we are splitting our core functionality to langchain-core to make LangChain more stable and reliable. This should be invisible to the eye and will happen in the background for the next two weeks, and we’d recommend not using langchain-core until then, but we’re flagging for transparency.



RunnablePassthrough moved from `langchain_core.runnables `  to from `langchain.schema.runnable`  
And the same 

langchain version : 0.0.336

### Idea or request for content:

**change** 
```
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.runnables import ConfigurableField
```

**To** 
```
from langchain.schema.runnable  import RunnablePassthrough
from langchain.schema  import StrOutputParser
from langchain.prompts import ChatPromptTemplate
from langchain.schema.runnable import RunnablePassthrough
from langchain.schema.runnable.utils import ConfigurableField
```
introduce convert_to_secret() to bananadev llm
Labels: name: auto:improvement, name: area: models, name: size:M

- **Description:** Per #12165, this PR add to BananaLLM the function convert_to_secret_str() during environment variable validation.
- **Issue:** #12165
- **Tag maintainer:** @eyurtsev
- **Twitter handle:** @treewatcha75751
Ollama: parameters and instruction templates 
Labels: name: area: doc loader, name: auto:question, name: auto:documentation

### Issue with current documentation:

It's not clear from the documentation whether when calling Ollama, langchain will take care of formatting the template correctly or if I have to supply the template by myself. 

For example, in https://ollama.ai/library/mistral:instruct

we have: 

```
parameter
stop "
"
stop "
"
stop "
"
stop "
"

template

 {{ .System }} {{ .Prompt }} 

```

Do I have to take care of formatting my instructions using these parameters and template, or langchain will take care of it? 


### Idea or request for content:

If this is not implemented, would be very useful to have definitely
fix error issue 12250
Labels: name: auto:bug, name: area: models, name: size:XS

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Jaguar vector store
Labels: name: area: vector store, name: auto:enhancement, name: size:XXL


  - **Description:** A new vector store Jaguar is being added. Class, test scripts, and documentation is added.
  - **Issue:** None -- This is the first PR contributing to LangChain
  - **Dependencies:** This depends on "pip install -U jaguardb-http-client" client http package
  - **Tag maintainer:**  @baskaryan, @eyurtsev, @hwchase1
  - **Twitter handle:** @workbot
OutputParser cheap json format Validation
Labels: name: auto:enhancement, name: area: models

### Feature request

Currently, when working with LangChain and generating JSON files which will be rendered in user facing application. We haven't found a dedicated method to check whether a JSON file adheres to the desired structure before executing subsequent operators. 
This feature request suggests the implementation of a method or utility for asserting the structure of a JSON file/format, ideally use defined schema (i.e: ResponseSchema). This would allow users to proactively identify any issues with the format.

### Motivation

In the current workflow, users have to rely on executing downstream operators (e.g., RetryWithErrorOutputParser) to discover issues with the JSON structure. This approach can be inefficient, especially when the error detection occurs after execution. Having a method to check the JSON structure beforehand would enable users to catch format issues early in the process. This would prevent users to further interact with LLM

### Your contribution

I'm willing to contribute to the implementation of this feature. I'll carefully read the CONTRIBUTING.md and follow the guidelines to submit a Pull Request once the scope is clarified.


Some simple example use case code:

```python
import os
import json
from dotenv import load_dotenv, find_dotenv
import openai
from langchain.output_parsers import StructuredOutputParser
from langchain.chat_models import ChatOpenAI
from langchain.prompts import ChatPromptTemplate
from jsonschema import validate, exceptions

_ = load_dotenv(find_dotenv())
openai.api_key = os.environ["OPENAI_API_KEY"]

expected_json_file_path = "app/files/expected_schema.json"

# LCEL - INSTANCES & STRUCTURE
prompt = ChatPromptTemplate.from_template(template=template_string)
chat_llm = ChatOpenAI(temperature=0.0)

# Define output parser based on reponse schemas
output_parser = StructuredOutputParser.from_response_schemas(
    asset_instance.response_schemas
)
# Define format instruction using get format instruction method
format_instructions = output_parser.get_format_instructions()

# LCEL DEFINE CHAIN
simple_chain_validator = prompt | chat_llm | output_parser

# LCEL INVOKE CHAIN
chain_to_parse = simple_chain_validator.invoke(
    {
        "content_to_format": asset_instance.raw_information,
        "format_instructions": format_instructions,
    }
)

# Define the expected JSON schema
with open(expected_json_file_path, "r") as schema_file:
    expected_schema = json.load(schema_file)
# Validate against the schema
try:
    validate(instance=chain_to_parse, schema=expected_schema)
    print("JSON is valid.")
except exceptions.ValidationError as e:
    print(f"Validation Error: {e}")
```