delayedkarma/langchain-issues huggingface dataset:
Add test for svm retriever
Labels: name: lgtm, name: auto:improvement

# What
- This is to add unit test for svm retriever.

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: add test for svm retriever
  - Issue: None
  - Dependencies: None
  - Tag maintainer: @baskaryan
  - Twitter handle: @MlopsJ

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
update LangSmith notebook
Labels: name: auto:documentation

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
feat: moving azure OpenAI API version to the latest 2023-05-15
Labels: name: lgtm, name: area: embeddings, name: auto:improvement

Moving to the latest non-preview Azure OpenAI API version=2023-05-15. The previous 2023-03-15-preview doesn't have support, SLA etc. For instance, OpenAI SDK has moved to this version https://github.com/openai/openai-python/releases/tag/v0.27.7

@baskaryan
Google Palm 2 as underlying LLM for Agent
Labels: name: area: agent, name: auto:question, name: auto:enhancement, name: stale

### Feature request

I want to be able to use Google Palm (bison) as the underlying LLM for agents. Currently I'm using chatGPT, but I also want to experiment how Google Palm performs with tool-picking. I have API access to I can experiment with it. 

### Motivation

I propose Google's Palm because I've anecdotally seen examples where it performed well in summarization and decision-making tasks. That makes me think it would be a very compelling candidate for 

### Your contribution

Will I need to over-ride some class within langchain? If you can give me step-by-step instructions, I might be able to help and submit a PR. Thanks!
Fixed sparql SELECT and UPDATE query function
Labels: name: lgtm, name: auto:improvement

  - Description: Changed "SELECT" and "UPDTAE" intent check from "=" to "in",
  - Issue: Based on my own testing, most of the LLM (StarCoder, NeoGPT3, etc..) doesn't return a single word response ("SELECT" / "UPDATE") through this modification, we can accomplish the same output without curated prompt engineering.
  - Dependencies: None
  - Tag maintainer: @baskaryan
  - Twitter handle: @aditya_0290


Thank you for maintaining this library, Keep up the good efforts.
remove unused import in voice assistant doc
Labels: name: auto:documentation

Description: Removed unused import in voice_assistant doc. 
Tag maintainer: @baskaryan 
<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Async recursive url loader
Labels: name: area: doc loader, name: auto:enhancement

Description: This PR implements a new document loader, async_recursive_url_loader, based on the recursive_loader. It not only made the original loader async but also added a few features, such as limiting the depth of the access, and customizable extractors(from the raw webpage to the text of the Document object), so that users can use other tools to extract the webpage. This PR also includes the document and test for the new loader.
Issue: N/A
Dependencies: asyncio, aiohttp
Tag maintainer: @rlancemartin, @eyurtsev
Twitter handle: @ Zend_Nihility
Notebook update: sales agent with tools
Labels: name: area: agent, name: auto:documentation

  - Description: This is an update to a previously published notebook. 
  Sales Agent now has access to tools, and this notebook shows how to use a Product Knowledge base
  to reduce hallucinations and act as a better sales person!
  - Issue: N/A
  - Dependencies: `chromadb openai tiktoken`
  - Tag maintainer:  @baskaryan @hinthornw
  - Twitter handle: @FilipMichalsky
OpenAIEmbeddings
Labels: name: lgtm, name: area: embeddings, name: auto:improvement

(#7654)

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
@hwchase17, @baskaryan
Document loader/debug
Labels: name: area: doc loader, name: auto:improvement

Description: Added debugging output in DirectoryLoader to identify the file being processed.
Issue: [Need a trace or debug feature in Lanchain DirectoryLoader #7725](https://github.com/hwchase17/langchain/issues/7725)
Dependencies: No additional dependencies are required.
Tag maintainer: @rlancemartin, @eyurtsev
This PR enhances the DirectoryLoader with debugging output to help diagnose issues when loading documents. This new feature does not add any dependencies and has been tested on a local machine.
upgrade chroma to 0.4.0
Labels: name: lgtm, name: area: vector store, name: auto:improvement

** This should land Monday the 17th ** 

Chroma is upgrading from `0.3.29` to `0.4.0`. `0.4.0` is easier to build, more durable, faster, smaller, and more extensible. This comes with a few changes:

1. A simplified and improved client setup. Instead of having to remember weird settings, users can just do `EphemeralClient`, `PersistentClient` or `HttpClient` (the underlying direct `Client` implementation is also still accessible)

2. We migrated data stores away from `duckdb` and `clickhouse`. This changes the api for the `PersistentClient` that used to reference `chroma_db_impl="duckdb+parquet"`. Now we simply set `is_persistent=true`. `is_persistent` is set for you to `true` if you use `PersistentClient`. 

3. Because we migrated away from `duckdb` and `clickhouse` - this also means that users need to migrate their data into the new layout and schema. Chroma is committed to providing extension notification and tooling around any schema and data migrations (for example - this PR!). 

After upgrading to `0.4.0` - if users try to access their data that was stored in the previous regime, the system will throw an `Exception` and instruct them how to use the migration assistant to migrate their data. The migration assitant is a pip installable CLI: `pip install chroma_migrate`. And is runnable by calling `chroma_migrate` 

-- TODO ADD here is a short video demonstrating how it works. 

Please reference the readme at [chroma-core/chroma-migrate](https://github.com/chroma-core/chroma-migrate) to see a full write-up of our philosophy on migrations as well as more details about this particular migration. 

Please direct any users facing issues upgrading to our Discord channel called [#get-help](https://discord.com/channels/1073293645303795742/1129200523111841883). We have also created a [email listserv](https://airtable.com/shrHaErIs1j9F97BE) to notify developers directly in the future about breaking changes. 

TODO
- [x] Migrated any `duckdb+parquet` strings to the new format
- [ ] Notified users about the breaking change (this PR, other suggestions?)
- [x] Move pypi target away from feature-branch and to `0.4.0` after `0.4.0` is released. 
- [x] We need to merge in a more flexible range for `fastapi` to the chroma branch - as is in-progress here https://github.com/chroma-core/chroma/pull/807
Improved GPT4All model streaming tokens without adding callbacks
Labels: name: auto:improvement

 
  - **Description:** Right now token streaming is enabled by adding callbacks. But there was no streaming of tokens occuring after adding callbacks.Latest version of gpt4all: version `1.0.3` addresses this issue and hence I tried to integrate that in the existing function.I have kept the code for supporting callbacks for streaming,  however I think that can be removed in coming versions. Also added docs for the same.
  
  - **Issue:** This commit fixes issue #7747.,
  - Dependencies: No dependency change,
  - Tag maintainer: @hwchase17 @agola11,
  - Twitter handle: @AnindyadeepS
Streaming does not work using streaming callbacks for gpt4all model
Labels: name: auto:bug

### System Info

**System Information**

System: `Linux`
OS: `Pop OS`
Langchain version: `0.0.232`
Python version: `3.9.17`
gpt4all version: used for both version `1.0.1` and version `1.0.3`.

### Who can help?

Models:

@hwchase17 

Streaming Callbacks:

@agola11 

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [X] Callbacks/Tracing
- [ ] Async

### Reproduction


```python3
import gpt4all as gpt 
from langchain import PromptTemplate, LLMChain
from langchain.llms import GPT4All
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler

template = """
Let's think step by step of the question: {question}
Based on all the thought the final answer becomes:
"""

prompt = PromptTemplate(template=template, input_variables=["question"])

local_path = (
    "./model/ggml-gpt4all-j-v1.3-groovy.bin" 
)

callbacks = [StreamingStdOutCallbackHandler()]

llm = GPT4All(
    model=local_path, 
    backend="llama", 
    verbose=True, callbacks=callbacks
)

llm_chain = LLMChain(prompt=prompt, llm=llm, verbose=True)

question = "What NFL team won the Super Bowl in the year Justin Bieber was born?"
llm_chain.run(question)
```

### Expected behavior

I should tokens streaming and printing in the terminal one at a time, but everything I get all at once.
feat: temp fixed Could not parse LLM output on agents folder
Labels: name: area: agent, name: lgtm, name: auto:bug

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Added search parameters to qdrant max_marginal_relevance_search
Labels: name: area: vector store, name: auto:improvement

Adds the qdrant search filter/params to the `max_marginal_relevance_search` method, which is present on others. I did not add `offset` for pagination, because it's behavior would be ambiguous in this setting (since we fetch extra and down-select). 

@rlancemartin, @eyurtsev
DOC: Arxiv API Tool code snippet very instable and produces very often an OutputParserException
Labels: name: area: agent, name: auto:bug, name: auto:documentation

### Issue with current documentation:

```
pip install arxiv
from langchain.chat_models import ChatOpenAI
from langchain.agents import load_tools, initialize_agent, AgentType

```

```
llm = ChatOpenAI(temperature=0.0)
tools = load_tools(
    ["arxiv"],
)

```
```
agent_chain = initialize_agent(
    tools,
    llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True,
)

agent_chain.run(
    "What's the paper 1605.08386 about?",
)
```

very often causes an OutputParserError due to the failure of parsing the final thought.

`OutputParserException: Could not parse LLM output`

### Idea or request for content:

It seems that just changing the llm to :

`llm = OpenAI(temperature=0.0)`

helps a lot with the output success completion.
fix: make error message clear when input_key arg is required
Labels: name: area: memory, name: auto:bug

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: Makes it more clear that input_key argument is required when there are multiple input variables in a prompt, 
  - Issue: #2013,
  - Tag maintainer: @hwchase17,

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
 
 - Description: Makes it more clear that input_key argument is required with multiple input variables in a prompt, 
 - Issue: #2013,
 - Tag maintainer: @hwchase17,
fix: absolute URLs from links in `RecursiveUrlLoader`
Labels: name: area: doc loader, name: auto:bug

## Description

This PR solves an absolute link problem in `RecursiveUrlLoader`. When the "href" of a link on the page contains an absolute URL, the link is ignored.
Example: `<a href="http://test.com/example"></a>` will be ignored when base_url is `http://test.com/`.

@rlancemartin @eyurtsev
Vulnerabilities: CVE-2023-36258, CVE-2023-3454, CVE-2023-34541, CVE-2023-36188, CVE-2023-36189
Labels: name: auto:bug

### System Info

Hi. 

I wanted to deploy application with Langchain but I am unable to pass security scans because of the following vulnerabilities: 

[CVE-2023-36258](https://nvd.nist.gov/vuln/detail/CVE-2023-36258)

[CVE-2023-34540](https://nvd.nist.gov/vuln/detail/CVE-2023-34540)

[CVE-2023-34541](https://nvd.nist.gov/vuln/detail/CVE-2023-34541)

[CVE-2023-36188](https://nvd.nist.gov/vuln/detail/CVE-2023-36188)

[CVE-2023-36189](https://nvd.nist.gov/vuln/detail/CVE-2023-36189)

I am unable to disable security scans. Are there any temporal fixes? 

@hwchase17 
@JamalRahman 


### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

pip install langchain

### Expected behavior

be able to pass Aqua Scanner
Document loader/debug
Labels: name: area: doc loader, name: auto:improvement


- Description: Added debugging output in DirectoryLoader to identify the file being processed.
- Issue: https://github.com/hwchase17/langchain/issues/7725
- Dependencies: No additional dependencies are required.
- Tag maintainer: @rlancemartin, @eyurtsev

This PR enhances the DirectoryLoader with debugging output to help diagnose issues when loading documents. This new feature does not add any dependencies and has been tested on a local machine.
WIKIPEDIA Language setting not applied in run method
Labels: name: area: agent, name: auto:bug

### System Info

```
langchain==0.0.228
langchain/utilities/wikipedia.py



```

The run method does not call the language change before execution, so it always runs with lang="en"
```


    def run(self, query: str) -> str:
        """Run Wikipedia search and get page summaries."""
       
       #define languaje 
        self.wiki_client.set_lang(self.lang)  #<-- this line fixes the issue
        
        page_titles = self.wiki_client.search(query[:WIKIPEDIA_MAX_QUERY_LENGTH])
        summaries = []
        for page_title in page_titles[: self.top_k_results]:
            if wiki_page := self._fetch_page(page_title):
                if summary := self._formatted_page_summary(page_title, wiki_page):
                    summaries.append(summary)
        if not summaries:
            return "No good Wikipedia Search Result was found"
        return "\n\n".join(summaries)[: self.doc_content_chars_max]
```
```

### Who can help?

@nfcampos @leo-gan @hwc

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

...
llm = ChatOpenAI(openai_api_key=openai_api_key,temperature=0)
tools = load_tools(["wikipedia"])
tools[0].api_wrapper.top_k_results=1
tools[0].api_wrapper.lang="es"

agent= initialize_agent(
    tools, 
    llm, 
    agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION,
    handle_parsing_errors=True,
    verbose = True)

agent("Mi primer Issue en Github")


### Expected behavior

http://**es**.wikipedia.com....
rfc
Labels: name: auto:refactor

exploring pkg per chain. seeing what moving PALChain would look like. imagine most top level stuff moved down into another langchain dir. so repo struct something like. not langchain package would still import PalChain

langchain
&emsp; | - langchain (could rename src)
&emsp;&emsp; | - ...
&emsp; | - pyproject.toml
&emsp;  | - poetry.lock
&emsp;  | tests
&emsp;&emsp; | - ...
&emsp;  | ...
langchain-pal
&emsp;  | ...
langchain-core
&emsp;  | ...
Makefile
...
Add ability to load (deserialize) objects from other namespaces
Labels: name: area: doc loader, name: auto:improvement

I have some Prompt subclasses in my project that I'd like to be able to deserialize in callbacks. Right now `loads()`/`load()` will bail when it encounters my object, but I know I can trust the objects because they're in my own projects.

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->