delayedkarma/langchain-issues huggingface dataset:
Fix Html2TextTransformer for shallow copy
Labels: name: lgtm, name: auto:bug, name: size:L

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Hi,
There is some unintended behavior in Html2TextTransformer.
The current code is **directly modifying the original documents that are passed as arguments to the function.**
Therefore, not only the return of the function but also the input variables are being modified simultaneously. 
**To resolve this, I added unit test code as well.**

reference link: [Shallow vs Deep Copying of Python Objects](https://realpython.com/copying-python-objects/)

Thanks! ☺️
Fix: (issue #14127) Volc Engine MaaS import error
Labels: name: lgtm, name: auto:bug, name: area: models, name: size:XS

  - **Description:** fix Volc Engine MaaS import error
  - **Issue:** [the issue # it fixes (if applicable),](https://github.com/langchain-ai/langchain/issues/14127)
  - **Dependencies:** None
  - **Tag maintainer:** @baskaryan 
  - **Twitter handle:**
Mask API key for Arcee LLM
Labels: name: lgtm, name: auto:improvement, name: area: models, name: size:M

  - **Description:** Mask API key for Arcee LLM and its associated unit tests
  - **Issue:** https://github.com/langchain-ai/langchain/issues/12165
  - **Dependencies:** N/A
  - **Tag maintainer:** @eyurtsev
  - **Twitter handle:** `eekaiboon`
Passing BaseModel type data from agent to tool
Labels: name: area: agent, name: auto:question

### Issue you'd like to raise.

We create an agent using initialize_agent, register the tool, and configure it to be selected through the agent.
However, in the agent, regardless of which tool is selected, I want to pass a data object called request of BaseModel type when executing the tool.
I would appreciate it if you could tell me how to configure and connect the agent and tool.

### Suggestion:

_No response_
How do i use system prompt template inside conversational retrieval chain?
Labels: name: auto:bug, name: area: models

@dosubot  , how do i use system prompt inside conversational retreival chain?
prompt = ChatPromptTemplate(
    messages=[
        SystemMessagePromptTemplate.from_template(
            "You are a nice chatbot named James-AI having a conversation with a human."
        ),
        MessagesPlaceholder(variable_name="chat_history"),
        HumanMessagePromptTemplate.from_template("{question}")]
)

memory = ConversationBufferWindowMemory(k=5, memory_key="chat_history", return_messages=True)
**retriever = new_db.as_retriever(search_type="similarity_score_threshold", 
                                 search_kwargs={"score_threshold": .65, 
                                                "k": 2})**


**qa = ConversationalRetrievalChain.from_llm(cfg.llm, verbose=True,retriever=retriever, memory=memory, prompt=prompt)**
                                              
if i use like that then it says 
ValidationError: 1 validation error for ConversationalRetrievalChain
prompt
  extra fields not permitted (type=value_error.extra)
Issue: How to conveniently disable langsmith calls? 
Labels: name: auto:question

### Issue you'd like to raise.

When I use an existing library that includes langsmith (such as https://github.com/langchain-ai/weblangchain), but I don't have access to langsmith API as I'm still on the waitlist. How can I quickly disable the langsmith functionality without commenting out the code?

### Suggestion:

_No response_
Add new Document Loader for JIRA
Labels: name: area: doc loader, name: auto:enhancement

### Feature request

I think Langchain should have a Document Loader that would allow reading all issues from provided project as documents.

### Motivation

Langchain currently has agent for JIRA, but there is no Document Loader that would allow to read all issues from specified project and store them in Vector database. This could be a very useful tool for RAG usecases and building internal knowledge bases in the companies. 

### Your contribution

I can create a PR for this issue.
Cloudflare Vectorize Support
Labels: name: area: vector store, name: auto:enhancement

### Feature request

Looks like it should be doable to have Cloudflare Vectorize support, as there's a HTTP API for Vectorize.

https://developers.cloudflare.com/api/operations/vectorize-list-vectorize-indexes

### Motivation

Similar to langchain-js, it would be nice if we could use Cloudflare Vectorize when using LangChain with Python. It would allow us to use the same vector store for both our TypeScript Workers and Python Lambdas.

https://js.langchain.com/docs/integrations/vectorstores/cloudflare_vectorize

### Your contribution

N/A sadly
DOC: Is there a way to conditionally create/update a Pandas DataFrame agent?
Labels: name: area: agent, name: auto:question, name: auto:documentation

### Issue with current documentation:

Is there a way to conditionally create a Pandas agent using `create_pandas_dataframe_agent` or update the DataFrame the agent has access to?

Say I have an agent that has access to tools like so:

```python
@tool()
def f(query: str) -> pd.DataFrame:
    ...

agent = initialize_agent(
    tools=[f, PythonREPLTool()],
    llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True,
    handle_parsing_errors=True,
)

```

Since `create_pandas_dataframe_agent` takes the `df` argument, I can't create this agent unless/until the tool `f` is called, producing the DataFrame. But without the specialized Pandas agent, the regular agent seems to be producing bad/bogus Python, trying to filter the DataFrame (either with syntax errors or just doing the wrong thing). Is there a way to either 1) call`create_pandas_dataframe_agent` ahead of time and then later update the contents of the `df` after the function call or 2) create a Pandas agent on-the-fly inside `f` to answer the question (I haven't found a way to pass the current `Thought` content to the nested agent in this case)?

### Idea or request for content:

It would be helpful to see how to work with Pandas data when reading dynamically created DataFrames coming from functions that agents may or may not call.
ModuleNotFoundError: No module named 'clarifai.auth'
Labels: name: auto:bug, name: area: models

### System Info

platform: Vagrant - Ubuntu 2204
python: 3.9.18
langchain version: 0.0.344
langchain core: 0.0.8
clarifai: 9.10.4

### Who can help?

@hwchase17 @agola11

### Information

- [X] The official example notebooks/scripts
- [x] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. Install the latest version of clarifai (9.10.4)
2. Run the example: https://python.langchain.com/docs/integrations/llms/clarifai

``` bash
Could not import clarifai python package. Please install it with `pip install clarifai`.
File 'clarifai.py', line 77, in validate_environment:
raise ImportError( Traceback (most recent call last):
  File "/home/vagrant/.virtualenvs/env/lib/python3.9/site-packages/langchain/llms/clarifai.py", line 74, in validate_environment
    from clarifai.auth.helper import ClarifaiAuthHelper
ModuleNotFoundError: No module named 'clarifai.auth'
```

### Expected behavior

I expect **ClarifaiAuthHelper** to import correctly.
In the latest version of clarifai **ClarifaiAuthHelper** is imported in this way:
``` python 
from clarifai.client.auth.helper import ClarifaiAuthHelper
```
Enable Multi-tenancy for Weaviate
Labels: name: needs work, name: area: vector store, name: auto:enhancement, name: size:S

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->

  - **Description:** Added support for multi-tenancy when creating a schema and taking tenant name to index and retrieve data.
  - **Dependencies:** nothing
  - **Tag maintainer:** @hwchase17 
  - **Twitter handle:** https://twitter.com/khajavi
BUG: csv loader exception, AttributeError: partially initialized module 'importlib.metadata' has no attribute 'version' (most likely due to a circular import)
Labels: name: area: doc loader, name: auto:bug

### System Info

langchain version ： 
  langchain                 0.0.344
  langchain-core            0.0.8


### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

code:
  from langchain.document_loaders.csv_loader import CSVLoader    
  loader = CSVLoader(file_path='./test.csv')
  data = loader.load()


### Expected behavior

exception:
Traceback (most recent call last):
  File "/Users/liucong/opt/miniconda3/envs/310-test/lib/python3.10/site-packages/langchain_core/__init__.py", line 4, in <module>
    __version__ = metadata.version(__package__)
AttributeError: partially initialized module 'importlib.metadata' has no attribute 'version' (most likely due to a circular import)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/liucong/opt/miniconda3/envs/310-test/lib/python3.10/site-packages/langchain/__init__.py", line 4, in <module>
    from importlib import metadata
  File "/Users/liucong/opt/miniconda3/envs/310-test/lib/python3.10/importlib/metadata/__init__.py", line 4, in <module>
    import csv
  File "/Users/liucong/PycharmProjects/llm_learn/lang_chain/data_connection/csv.py", line 1, in <module>
    from langchain.document_loaders.csv_loader import CSVLoader
  File "/Users/liucong/opt/miniconda3/envs/310-test/lib/python3.10/site-packages/langchain/document_loaders/__init__.py", line 18, in <module>
    from langchain.document_loaders.acreom import AcreomLoader
  File "/Users/liucong/opt/miniconda3/envs/310-test/lib/python3.10/site-packages/langchain/document_loaders/acreom.py", line 5, in <module>
    from langchain_core.documents import Document
  File "/Users/liucong/opt/miniconda3/envs/310-test/lib/python3.10/site-packages/langchain_core/__init__.py", line 5, in <module>
    except metadata.PackageNotFoundError:
AttributeError: partially initialized module 'importlib.metadata' has no attribute 'PackageNotFoundError' (most likely due to a circular import)
Fixes and Upgrades for the Tongyi LLM and ChatTongyi Model
Labels: name: auto:improvement, name: area: models, name: size:XL

  - **Description:** Upgrade the `Tongyi` LLM and `ChatTongyi` Model.
      - Fixed typos; it should be `Tongyi`, not `OpenAI`.
      - Fixed a bug in `stream_generate_with_retry`; it's a real stream generator now.
      - Fixed a bug in `validate_environment`; the `dashscope_api_key` should be properly handled when set by environment variables or initialization parameters.
      - Changed the `dashscope` response to incremental output by setting the parameter `incremental_output`, which eliminates the need for the prefix-removal trick.
      - Removed some unused parameters, like `n`, `prefix_messages`.
      - Added `_stream` method.
      - Added async methods, such as `_astream`, `_agenerate`.
  - **Issue:** No related issue.
  - **Dependencies:** No new dependencies.
  - **Tag maintainer:** @baskaryan 

PS: Some may be confused about the terms `dashscope`, `tongyi`, and `Qwen`:
- `dashscope`: A platform to deploy LLMs and provide APIs to invoke the LLM.
- `tongyi`: A brand name or overall term about Alibaba Cloud's LLM/AI.
- `Qwen`: An LLM that is open-sourced and deployed in `dashscope`.

We use the `dashscope` SDK to interact with the `tongyi`-`Qwen` LLM.

~~And the fix and upgrade of the chat model `ChatTongyi` will be addressed in future PRs.~~ (Already included in this PR)
Mask API key for Tongyi ChatModel and LLM
Labels: name: auto:improvement, name: area: models, name: size:L

- **Description:** Added masking of api_key for tongyi chat models and llms `langchain/chat_models/tongyi.py` &  `langchain/llms/tongyi.py`,
- **Issue:** https://github.com/langchain-ai/langchain/issues/12165
- **Dependencies:** dashscope
- **Tag maintainer:** @eyurtsev
- **Twitter handle:** FadilParves
Question about runing Langchain Extraction Demo in Chinese text.
Labels: name: auto:bug, name: auto:question, name: area: models

### Issue you'd like to raise.

```python
from langchain.chains import create_extraction_chain
from langchain.chat_models import ChatOpenAI
import dotenv


dotenv.load_dotenv()

# Schema
schema = {
    "properties": {
        "name": {"type": "string"},
        "education": {"type": "string"},
        "company": {"type": "string"},

    },
    "required": ["name", "education"],
}

# Input
inp = """小明毕业于北京大学，在微软上班\n小李毕业于清华大学在阿里巴巴上班\n小王毕业于浙江大学在烟草局上班"""

# Run chain
llm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-1106")
chain = create_extraction_chain(schema, llm)
print(chain.run(inp)[0])
```

```shell
#output
{'name': 'å°\x8fæ\x98\x8e', 'education': 'å\x8c\x97äº¬å¤§å\xad¦', 'company': 'å¾®è½¯'}
```
How to fix this Chinese `encoding/decoding` problem?

### Suggestion:

_No response_
Issue:  'BM25Retriever' has no attribute 'from_documents'
Labels: name: area: doc loader, name: auto:bug

### Issue you'd like to raise.

Code: 
retriever = BM25Retriever.from_documents(
    [
        Document(page_content="foo"),
        Document(page_content="bar"),
        Document(page_content="world"),
        Document(page_content="hello"),
        Document(page_content="foo bar"),
    ]
)

Error:
---> 30 retriever = BM25Retriever.from_documents(
     31     [
     32         Document(page_content="foo"),
     33         Document(page_content="bar"),
     34         Document(page_content="world"),
     35         Document(page_content="hello"),
     36         Document(page_content="foo bar"),
     37     ]
     38 )

AttributeError: type object 'BM25Retriever' has no attribute 'from_documents'

### Suggestion:

_No response_
Update docs to use new usage in openai>1.0.0
Labels: name: lgtm, name: area: doc loader, name: auto:documentation, name: size:L

### Description

Use new [APIs](https://github.com/openai/openai-python/blob/main/api.md#finetuning)

### Twitter handle

[lin_bob57617](https://twitter.com/lin_bob57617)
docs `AWS` platform page update
Labels: name: lgtm, name: auto:documentation, name: integration: aws, name: size:XXL

The `AWS` platform page has many missed integrations.
- added missed integration references to the `AWS` platform page
- added/updated descriptions and links in the referenced notebooks
- renamed two notebook files. They have file names != page Title, which generate unordered ToC.
- reroute the URLs for renamed files
- fixed `amazon_textract` notebook: removed failed cell outputs
create_tagging_chain_pydantic() pydantic schema validation errors
Labels: name: auto:bug, name: area: models

### System Info

[pydantic_chain_test_code.txt](https://github.com/langchain-ai/langchain/files/13532796/pydantic_chain_test_code.txt)
[error_log.txt](https://github.com/langchain-ai/langchain/files/13532799/error_log.txt)


### Who can help?

@agola11 @hwchase17 

The attached code is adopted from the official example from https://python.langchain.com/docs/use_cases/tagging. I am using the AzureChatOpenAI API, but I don't think the cause is related to that. I added an example prompt to ensure that the llm is working properly.

It appears that even when passing in a confirmed subclass of BaseModel, which is pydantic_schema, create_tagging_chain_pydantic(Tags, llm) is triggering validation errors. Please see the attached test code and error log.

### Information

- [X] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Run the attached code and match it with the attached error log, which shows the version numbers of Python, Langchain and Pydantic. I am running on WIndows 11.:

Python version: 3.11.5 | packaged by Anaconda, Inc. | (main, Sep 11 2023, 13:26:23) [MSC v.1916 64 bit (AMD64)]
Langchain version: 0.0.344
Pydantic version: 2.5.2


### Expected behavior

Expected behavior is as shown in the last example on https://python.langchain.com/docs/use_cases/tagging

    Tags(sentiment='sad', aggressiveness=5, language='spanish')
Bagatur/core lint docstring

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Add Retriever for Knowledge Base IBM Watson Discovery Service
Labels: name: area: doc loader, name: auto:enhancement

### Feature request

Adding [IBM Watson Discovery Service](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-getting-started) as an additional retriever might be a useful extension to enable RAG-use cases or supporting RetrievalChains.
IBM Watson Discovery Service can find relevant documents based on natural language queries and understand document structures such as tables.

### Motivation

Currently, there is no support for IBM Watson Discovery Service (WDS) as a retriever which requires external scripts to use WDS as a source for a RetrievalChain e.g.

### Your contribution

I'm able to contribute a IbM Watson Discovery Service retriever.
FAISS.load_local does not keep the typing of the saved Index
Labels: name: area: embeddings, name: area: vector store, name: auto:bug

### System Info

It seems the typing of my index is lost when I save and load it. This is me creating the FAISS vector store with the MAX_INNER_PRODUCT flag.

```python
# Saving
save_db = FAISS.from_texts(texts, embedding_function, metadatas=metadatas, distance_strategy="MAX_INNER_PRODUCT", normalize_L2=True)
save_db.save_local(db_directory)
print(save_db.index, save_db.distance_strategy)
# <faiss.swigfaiss_avx2.IndexFlatIP; proxy of <Swig Object of type 'faiss::IndexFlatIP *' at 0x7f4e0402e4b0> > MAX_INNER_PRODUCT

# Loading
load_db = FAISS.load_local(db_directory, embedding_function, distance_strategy="MAX_INNER_PRODUCT")
print(load_db .index, load_db .distance_strategy)
# <faiss.swigfaiss_avx2.IndexFlat; proxy of <Swig Object of type 'faiss::IndexFlat *' at 0x7f4dd0accdb0> > MAX_INNER_PRODUCT
```

I have looked through FAISS API documentation and can't tell if this is an abstracted class. 

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. Create a FAISS vector store passing distance_strategy = "MAX_INNER_PRODUCT". This should create an index with FAISS.IndexFlatIP.
2. Save the FAISS vector store.
3. Load the FAISS vector store with the distance_strategy = "MAX_INNER_PRODUCT"
4. Compare the saved.index and loaded.index objects. One is IndexFlatIP and the other is an IndexFlat class.

### Expected behavior

When loading the index from load_local, it should still be an IndexFlatIP.
Amazon DocumentDB Vector Store
Labels: name: area: vector store, name: auto:enhancement, name: integration: aws

### Feature request

Amazon DocumentDB has launched its vector search feature recently. Let us add Amazon DocumentDB as vector store in langchain.

### Motivation

This will help large customer base of Amazon DocumentDB to adopt langchain and seamlessly build AI/ML apps

### Your contribution

Amazon DocumentDB team can help with testing and technical help. 
https://aws.amazon.com/about-aws/whats-new/2023/11/vector-search-amazon-documentdb/
https://docs.aws.amazon.com/documentdb/latest/developerguide/vector-search.html