delayedkarma/langchain-issues huggingface dataset:
Error while running on Apple M1 Pro
Labels: name: bug

### System Info

Apple M1

### Who can help?

I get this error : (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')) 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Just run the model

### Expected behavior

must run on apple m1
chroma func _similarity_search_with_relevance_scores missing "kwargs" parameter
Labels: name: bug, name: needs work

### System Info

LangChain: v0.0.214

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction
https://github.com/hwchase17/langchain/blob/v0.0.214/langchain/vectorstores/base.py#L410
https://github.com/hwchase17/langchain/blob/v0.0.214/langchain/vectorstores/chroma.py#L225
https://github.com/hwchase17/langchain/blob/v0.0.214/langchain/vectorstores/chroma.py#L198

The `filter` option does not work when search_type is similarity_score_threshold

### Expected behavior

work:
```python
def _similarity_search_with_relevance_scores(
        self,
        query: str,
        k: int = 4,
        **kwargs: Any,
    ) -> List[Tuple[Document, float]]:
        return self.similarity_search_with_score(query, k, **kwargs)
```
ChatOpenAI.get_num_tokens_from_messages raises TypeError with function call response
Labels: name: bug

### System Info

- LangChain version: 0.0.214
- tiktoken version: 0.4.0

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Executing following code raises `TypeError: expected string or buffer`.

```python
from langchain.chat_models import ChatOpenAI
from langchain.schema import HumanMessage

llm = ChatOpenAI(model_name="gpt-3.5-turbo-0613", temperature=0)

functions = [
    {
      "name": "get_current_weather",
      "description": "Get the current weather in a given location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "The city and state, e.g. San Francisco, CA"
          },
          "unit": {
            "type": "string",
            "enum": ["celsius", "fahrenheit"]
          }
        },
        "required": ["location"]
      }
    }  
]

response = llm([HumanMessage(content="What is the weather like in Boston?")], functions=functions)

llm.get_num_tokens_from_messages([response])
```

`get_num_tokens_from_messages` internally converts messages to dict with `_convert_message_to_dict` and then interates all key-value pairs to count the number of tokens.
The code expects value to be a string, but when a function call is included, an exception seems to be raised because value contains a dictionary.

### Expected behavior

As far as I know, there is no officially documented way to calculate the exact token count consumption when using function call.

Someone on the OpenAI forum has [posted](https://community.openai.com/t/how-to-calculate-the-tokens-when-using-function-call/266573/10) a method for calculating the tokens, so perhaps that method could be adopted.
Fix Typo in LangChain MyScale Integration  Doc

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->

- Description: Fix Typo in LangChain MyScale Integration  Doc

@hwchase17
NotImplementedError: WebBaseLoader does not implement lazy_load()
Labels: name: bug

### Issue you'd like to raise.

when i use RecusiveUrlLoader , 
NotImplementedError: WebBaseLoader does not implement lazy_load()

![image](https://github.com/hwchase17/langchain/assets/10355060/2d1686e7-de7f-412b-a9f4-1024ec666281)


### Suggestion:

_No response_
Update arize_callback.py

Changed callbackhandler to async

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Memory seems not to be supported in create_pandas_dataframe_agent
Labels: name: enhancement

### Feature request

I am trying to add memory to create_pandas_dataframe_agent to perform post processing on a model that I trained using Langchain. I am using the following code at the moment.
```
from langchain.llms import OpenAI
import pandas as pd

df = pd.read_csv('titanic.csv')
agent = create_pandas_dataframe_agent(OpenAI(temperature=0), [df], verbose=True)
```
I tried adding memory = ConversationBufferMemory(memory_key="chat_history") but that didnt help. Tried many other methods but seems like the memory for create_pandas_dataframe_agent is not implemented

### Motivation

There is a major need in pandas processing to save models as pickle files along with adding new features to the studied dataset which alters the original dataset for the next step. It seems like langchain currently doesnt support that.

### Your contribution

I can help with the implementation if necessary.
Improve vector store onboarding exp
Labels: name: documentation

This PR
- fixes the `similarity_search_by_vector` example, makes the code run and adds the example to mirror `similarity_search`
- reverts back to chroma from faiss to remove sharp edges / create a happy path for new developers. (1) real metadata filtering, (2) expected functionality like `update`, `delete`, etc to serve beyond the most trivial use cases 

@hwchase17
Chroma.from_documents(...persist_directory=db_path) has no effect
Labels: name: bug


### System Info

$ pip freeze | grep langchain
langchain==0.0.197
langchainplus-sdk==0.0.10


### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

For some reason : 

db = Chroma.from_documents(texts, self.embeddings, persist_directory=db_path, client_settings=settings)

persist_directory=db_path, has no effect ... upon db.persist() it stores into the default directory 'db', instead of using db_path.

Only if you explicitly set Settings(persist_directory=db_path, ... ) it works.

Probable reason is that in langchain chroma.py  if you pass client_settings and 'persist_directory' is not part of the settings, it will not add 'persist_directory' as it is done in the ELSE case, but ...:

(line 77 ++)
```
        if client is not None:
            self._client = client
        else:
            if client_settings:
                self._client_settings = client_settings  <<< .... here ..........
            else:
                self._client_settings = chromadb.config.Settings()
                if persist_directory is not None:
                    self._client_settings = chromadb.config.Settings(
                        chroma_db_impl="duckdb+parquet",
                        persist_directory=persist_directory,
                    )
            self._client = chromadb.Client(self._client_settings)

        self._embedding_function = embedding_function
        self._persist_directory = persist_directory
``` 

but ... chromadb.__init__.py expects 'persist_directory' in settings i.e. (line 44), otherwise it will use the default : 

```
   elif setting == "duckdb+parquet":
        require("persist_directory")
        logger.warning(
            f"Using embedded DuckDB with persistence: data will be stored in: {settings.persist_directory}"
        )
```

### Expected behavior

db = Chroma.from_documents(texts, self.embeddings, persist_directory=db_path, client_settings=settings)


should use db_path instead of 'db'
Issue: create_csv_agent() error when loading a list of csv files
Labels: name: bug

### System Info

Langchain 0.0.214
Python 3.10
Ubuntu 22.04

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

agent = create_csv_agent(
    ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613"),
    ["titanic.csv", "titanic_age_fillna.csv"],
    verbose=True,
    agent_type=AgentType.OPENAI_FUNCTIONS,
)
agent.run("how many rows in the age column are different between the two dfs?")

Got error: ValueError: Invalid file path or buffer object type: 

### Expected behavior

According to Langchain documentation https://python.langchain.com/docs/modules/agents/toolkits/csv.html, the CSV agent "can interact with multiple csv files passed in as a list", and it should not generate an error.
sitemap loader throws error TypeError: _request() got an unexpected keyword argument 'verify', many docs refer to wrong links for sitemap as well. 
Labels: name: bug, name: documentation

### System Info

windows 11 python 3.9.16 langchain 0.0.212


### Who can help?

Code from https://python.langchain.com/docs/modules/data_connection/document_loaders/integrations/sitemap

```python
from langchain.document_loaders.sitemap import SitemapLoader

sitemap_loader = SitemapLoader(web_path="https://langchain.readthedocs.io/sitemap.xml")

docs = sitemap_loader.load()
```
throws:
```python
    self._request(hdrs.METH_GET, url, allow_redirects=allow_redirects, **kwargs)

TypeError: _request() got an unexpected keyword argument 'verify'
```python

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

from langchain.document_loaders.sitemap import SitemapLoader

sitemap_loader = SitemapLoader(web_path="https://langchain.readthedocs.io/sitemap.xml")

docs = sitemap_loader.load()

### Expected behavior

to work or get a doc update
document loader as a retriever
Labels: name: area: doc loader, name: auto:enhancement

This unbraking change allows using every Document loader as a Retriever without any additional changes.
This change effectively increased number of the available retrievers from ~20 to 80+ 

`docs/extras/modules/data_connection/retrievers/how_to/document_loaders_as_retrievers.ipynb` provides explanations and a usage example.

Maintainer responsibilities:
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
 It can be interesting for @hwchase17 and @dev2049
Issue: Can an LLM be used as a tool?
Labels: name: documentation, name: question

### Issue you'd like to raise.

I have a use case where the agent is supposed to perform certain activities (going over the metadata and telling if the currently selected column is fit for querying). This would need a `zero-shot-react-agent` to use several LLMs as tools instead of the present ones (like search being shown everywhere). The [documentation](https://python.langchain.com/docs/modules/agents/how_to/custom_mrkl_agent#multiple-inputs) shows that this is possible but is in itself quite ambiguous. 
How do I create an LLMChain as a tool if it always needs a prompt while initialisation? And the prompt can be created only after mentioning this LLMChain as a tool in the `create_prompt` function?

### Suggestion:

_No response_
align chroma vectorstore get with chromadb to enable where filtering
Labels: name: lgtm

allows for where filtering on collection via get

  - Description: aligns langchain chroma vectorstore get with underlying [chromadb collection get](https://github.com/chroma-core/chroma/blob/main/chromadb/api/models/Collection.py#L103) allowing for where filtering, etc.
  - Issue: NA
  - Dependencies: none
  - Tag maintainer: @rlancemartin, @eyurtsev
  - Twitter handle: @pappanaka
Fix conversational_retrieval combining with chat history affects the question and answer experience

Fixes # (issue)
While using the ConversationalRetrievalChain, and specifically applying both condense_question_prompt and combine_docs_chain_kwargs parameters, I followed the official documentation to pass chat_history when invoking this chain. The program runs as expected, but the Q&A experience is less than optimal. I've identified two main issues:

When summarizing the question using condense_question, the response prompt generated by load_qa_chain is treated as 'Human' rather than 'AI'. Currently, I am remedying this by determining whether to pass the question to human_message or ai_message, based on whether chat_history is provided through an external interface.

Upon reviewing the code, I noticed that the answer is first generated by self.question_generator, but in reality, the desired answer initially comes from the prompt in combine_docs_chain_kwargs. However, this response is then re-fed into self.combine_docs_chain, leading to a situation of self-posed questions and self-given answers, which isn't ideal. The final answer from combine_docs_chain isn't what I'm looking for, and I see no need to go through another loop only to end up with an unsatisfactory answer. Thus, I wish to adjust the logic for obtaining the answer in the call and acall methods.

While my implementation might not be the most ideal, it considerably enhances the Q&A experience based on chat history in my project. I hope for official support on this issue and appreciate your help. Thank you!

Before submitting
Help for an error that appears in the CONVERSATIONAL Agent
Labels: name: bug, name: question

### Issue you'd like to raise.

This problem occurred when I tried to use the tool support for a conversation-type agent using an agent of type SRUCTURED_CHAT_SHOT_REACT_DESCRIPTION. Here is some of the source code and errors
![IMG_20230624_134959](https://github.com/hwchase17/langchain/assets/64879324/17529dfe-68bc-49b3-9f99-96e588e323b6)

env:
Python 3.10
Langchain Lastest
WIN 10 Profession Lastest

### Suggestion:

_No response_
crash because of missing bs4 dependency in version 2.12
Labels: name: bug

### System Info

Langchain: 2.12
commit: #6455
version 2.11 does niet have this

### Who can help?

@rlancemartin, @eyurtsev

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

workaround: install bs4 manually (pip install bs4)

`from langchain.agents import initialize_agent, AgentType`
leads to:
```
  File "//main.py", line 17, in <module>
    from langchain.agents import initialize_agent, AgentType
  File "/usr/local/lib/python3.11/site-packages/langchain/__init__.py", line 6, in <module>
    from langchain.agents import MRKLChain, ReActChain, SelfAskWithSearchChain
  File "/usr/local/lib/python3.11/site-packages/langchain/agents/__init__.py", line 2, in <module>
    from langchain.agents.agent import (
  File "/usr/local/lib/python3.11/site-packages/langchain/agents/agent.py", line 16, in <module>
    from langchain.agents.tools import InvalidTool
  File "/usr/local/lib/python3.11/site-packages/langchain/agents/tools.py", line 8, in <module>
    from langchain.tools.base import BaseTool, Tool, tool
  File "/usr/local/lib/python3.11/site-packages/langchain/tools/__init__.py", line 3, in <module>
    from langchain.tools.arxiv.tool import ArxivQueryRun
  File "/usr/local/lib/python3.11/site-packages/langchain/tools/arxiv/tool.py", line 12, in <module>
    from langchain.utilities.arxiv import ArxivAPIWrapper
  File "/usr/local/lib/python3.11/site-packages/langchain/utilities/__init__.py", line 3, in <module>
    from langchain.utilities.apify import ApifyWrapper
  File "/usr/local/lib/python3.11/site-packages/langchain/utilities/apify.py", line 5, in <module>
    from langchain.document_loaders import ApifyDatasetLoader
  File "/usr/local/lib/python3.11/site-packages/langchain/document_loaders/__init__.py", line 97, in <module>
    from langchain.document_loaders.recursive_url_loader import RecusiveUrlLoader
  File "/usr/local/lib/python3.11/site-packages/langchain/document_loaders/recursive_url_loader.py", line 5, in <module>
    from bs4 import BeautifulSoup
ModuleNotFoundError: No module named 'bs4'

```
requirements.txt:
```
openai==0.27.8
fastapi==0.97.0
websockets==11.0.3
pydantic==1.10.9
langchain==0.0.212
uvicorn[standard]
jinja2
lancedb==0.1.8
itsdangerous
tiktoken==0.4.0
```


### Expected behavior

I think
from bs4 import BeautifulSoup
in recursive_url_loader.py
should have been a local import
Problem to run on linux but not on windows 
Labels: name: bug

### System Info

see: https://github.com/hwchase17/langchain/discussions/1533

### Who can help?

@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

use FAISS on windows and next try to reuse those embeddings on lunix

### Expected behavior

```
I have a problem, when I want to run a project with langchain on windows, everything works perfectly, and with the same conditions on linux (libraries, python version, etc.) it doesn't work and throws this error, does anyone know what it could be?

2023-03-08 15:13:24 Failed to run listener function (error: search() missing 3 required positional arguments: 'k', 'distances', and 'labels')
2023-03-08 15:13:24 Traceback (most recent call last):
2023-03-08 15:13:24 File "/usr/local/lib/python3.9/site-packages/slack_bolt/listener/thread_runner.py", line 120, in run_ack_function_asynchronously
2023-03-08 15:13:24 listener.run_ack_function(request=request, response=response)
2023-03-08 15:13:24 File "/usr/local/lib/python3.9/site-packages/slack_bolt/listener/custom_listener.py", line 50, in run_ack_function
2023-03-08 15:13:24 return self.ack_function(
2023-03-08 15:13:24 File "//./main.py", line 28, in question
2023-03-08 15:13:24 response = processQuestion(query)
2023-03-08 15:13:24 File "/api.py", line 42, in processQuestion
2023-03-08 15:13:24 sources = doSimilaritySearch(index, query)
2023-03-08 15:13:24 File "/utils.py", line 87, in doSimilaritySearch
2023-03-08 15:13:24 docs = indexFaiss.similarity_search(query, k=5)
2023-03-08 15:13:24 File "/usr/local/lib/python3.9/site-packages/langchain/vectorstores/faiss.py", line 166, in similarity_search
2023-03-08 15:13:24 docs_and_scores = self.similarity_search_with_score(query, k)
2023-03-08 15:13:24 File "/usr/local/lib/python3.9/site-packages/langchain/vectorstores/faiss.py", line 136, in similarity_search_with_score
2023-03-08 15:13:24 docs = self.similarity_search_with_score_by_vector(embedding, k)
2023-03-08 15:13:24 File "/usr/local/lib/python3.9/site-packages/langchain/vectorstores/faiss.py", line 110, in similarity_search_with_score_by_vector
2023-03-08 15:13:24 scores, indices = self.index.search(np.array([embedding], dtype=np.float32), k)
2023-03-08 15:13:24 TypeError: search() missing 3 required positional arguments: 'k', 'distances', and 'labels'
```