delayedkarma/langchain-issues huggingface dataset:
Add session-based context storage to VectorStoreRetrieverMemory
Labels: name: area: memory, name: auto:enhancement

### Title: Add Session-Based Context Storage to SessionVectorStoreRetrieverMemory

#### Description:

This PR introduces a new feature to the `SessionVectorStoreRetrieverMemory` class that allows developers to store and retrieve the history of a particular chat session. It proposes a new class, `SessionVectorStoreRetrieverMemory`, which extends `VectorStoreRetrieverMemory` and adds a `session_id` field. 

The key changes include:

- The addition of a `session_id` field, which will be used to uniquely identify a chat session. This will allow us to store and retrieve the history of a particular session.

- Modifying the `_form_documents` method to attach the `session_id` to each document created. This means that each document in our vector store will now include information about the session it belongs to.

This feature requires the `VectorStoreRetriever` to support metadata filtering. For instance, you can create a retriever that filters on the `session_id` field as follows:

```python
import uuid
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.llms import OpenAI
from langchain.memory import SessionVectorStoreRetrieverMemory

# generate a random session id
session_id = uuid.uuid4().hex

embedding_fn = OpenAIEmbeddings()

vec_store = Pinecone.from_existing_index(PINECONE_INDEX, embedding_fn, text_key="chat_memory")

# include session_id as param in retriever and memory object
retriever = vec_store.as_retriever(search_kwargs={'k': 1, 'filter': {'session_id': session_id}})
memory = SessionVectorStoreRetrieverMemory(retriever=retriever, session_id=session_id)  

memory.save_context({"input": "My favorite food is pizza"}, {"output": "thats good to know"})
memory.save_context({"input": "My favorite sport is soccer"}, {"output": "..."})
memory.save_context({"input": "I don't the Celtics"}, {"output": "ok"}) 

print(memory.load_memory_variables({"prompt": "what sport should i watch?"})["history"])


```
This will ensure that only documents with the specified session_id are returned when a query is made.

The rationale behind these changes is to enable a finer granularity of data retrieval. Currently, the SessionVectorStoreRetrieverMemory class retrieves relevant documents based on a query without considering the context of a chat session. By adding the session_id, we can retrieve documents that are not only relevant to the query but also belong to the same chat session. This makes it possible to have more context-aware conversations and could improve the quality of our chatbot's responses, while maintaining privacy between sessions of different users.

The change is backward-compatible and does not affect the existing functionality of the SessionVectorStoreRetrieverMemory class. Developers can choose to use the session-based version or the original version depending on their needs.

Contributor: Adi Sidapara (Github: @asidapara, Twitter: @AdiSidapara)

Could you review? @hwchase17 @dev2049 Thanks in advance!
Fix serialization issue with W&B
Labels: name: lgtm

The chain input_documents are not displaying properly in W&B, due to serialization issue:

<img width="1164" alt="Screenshot 2023-06-04 at 11 58 26 AM" src="https://github.com/hwchase17/langchain/assets/134809928/f31f14f6-0935-4cca-9913-6760cd40eadf">
feat: make YoutubeLoader support multi language transcript
Labels: name: lgtm

Adding support multi language transcript for YoutubeLoader.

#### Example
<img width="1033" alt="Screenshot 2023-06-05 at 01 10 31" src="https://github.com/hwchase17/langchain/assets/42396310/34b99e7e-4e19-4946-85fe-20c3bde04007">

#### Who can review?

Tag maintainers/contributors who might be interested:
@eyurtsev
prompt template not reset when reinstantiating SQLDatabase object

### System Info

python 3.8, langchain 0.0.187

### Who can help?

@hwchase17 @vowelparrot 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```python
from langchain import SQLDatabase, SQLDatabaseChain
rom langchain.llms.openai import OpenAI

db_fp = 'test.db'
db = SQLDatabase.from_uri(f"sqlite:///{db_fp}")
llm = OpenAI(temperature=0, max_tokens=1000)
db_chain = SQLDatabaseChain.from_llm(llm, db, verbose=True)

# reading default prompt
db_chain.llm_chain.prompt.template
# set to sth else
db_chain.llm_chain.prompt.template = 0
# do something, prompt not good, so want to reset
db_chain = SQLDatabaseChain.from_llm(llm, db, verbose=True)
db_chain.llm_chain.prompt.template
# BUG: returns 0, but should be default
```

### Expected behavior

should reinstantiate SQL chain object, but state from template reassignment is kept
support local GPU server without runhouse !!!

### Feature request

Hi guys, can we just launch SelfHostedEmbedding without runhouse, as a matter of fact, runhouse its self is still in the early development stage and has some issues, why does langchain depends on it for selfhosted instead of call cuda directly?

for example:

device = torch.device('cuda')

embeddings = SelfHostedEmbeddings(
    model_load_fn=get_pipeline, 
    hardware=device,
    model_reqs=["./", "torch", "transformers"],
    inference_fn=inference_fn
)


### Motivation

use GPU local server directly instead of through runhouse.

### Your contribution

*
Add Support for Hugging Face Question Answering Models
Labels: name: auto:enhancement

Working on Adding HuggingFace Question Answering Model support to Langchain  #5660 

First Commit is just a proof on concept, that simply adds the functionality to the HuggingFacePipeline Class and its _call method.
This implementation is however in conflict with its Base Classes, since Q&A Models require an additional input (at least before the input gets tokenized).
 
Since this is about adding new models it might be of interest to you:
  - @hwchase17
  - @agola11

I'm not sure if it's best to modify the base classes to add the support for an additional argument (the question / context), or to add a new class for question answering models.
I am happy to discuss the best way forward / your ideas.
ValueError: fix_text not implemented for this agent.

### System Info

langchain==0.0.100

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Hey, I am working on a Toolformer Agent similar to ReAct.
The prompts given to the agent is a few-shot prompt with tool usage.
An example can be seen on this website:

https://tsmatz.wordpress.com/2023/03/07/react-with-openai-gpt-and-langchain/#comments 

I can define the agent without any bugs:
```python
SUFFIX = """\nQuestion: {input}
{agent_scratchpad}"""

TEST_PROMPT = PromptTemplate.from_examples(
  EXAMPLES, SUFFIX, ["input", "agent_scratchpad"]
)

class ReActTestAgent(Agent):
  @classmethod
  def create_prompt(cls, tools: Sequence[BaseTool]) -> BasePromptTemplate:
    return TEST_PROMPT

  @classmethod
  def _validate_tools(cls, tools: Sequence[BaseTool]) -> None:
    if len(tools) != 4:
      raise ValueError("The number of tools is invalid.")
    tool_names = {tool.name for tool in tools}
    if tool_names != {"GetInvoice", "Diff", "Total", "python_repl"}:
      raise ValueError("The name of tools is invalid.")

  @property
  def _agent_type(self) -> str:
    return "react-test"

  @property
  def finish_tool_name(self) -> str:
    return "Finish"

  @property
  def observation_prefix(self) -> str:
    return f"Observation : "

  @property
  def llm_prefix(self) -> str:
    return f"Thought : "

  # This method is called by framework to parse text
  def _extract_tool_and_input(self, text: str) -> Optional[Tuple[str, str]]:
    action_prefix = f"Action : "
    if not text.split("\n")[1].startswith(action_prefix):
      return None
    action_block = text.split("\n")[1]

    action_str = action_block[len(action_prefix) :]
    re_matches = re.search(r"(.*?)\[(.*?)\]", action_str)
    if re_matches is None:
      raise ValueError(f"Could not parse action directive: {action_str}")
    return re_matches.group(1), re_matches.group(2)

##########
# run agent
##########

# llm = AzureOpenAI(
#   deployment_name="davinci003-deploy",
#   model_name="text-davinci-003",
#   temperature=0,
# )
llm = OpenAI(temperature=0, model_name="gpt-3.5-turbo")

agent = ReActTestAgent.from_llm_and_tools(
  llm,
  tools,
)
agent_executor = AgentExecutor.from_agent_and_tools(
  agent=agent,
  tools=tools,
  verbose=True,
)
```
but When I give instruction to the agent 
```python
question = "hi"
agent_executor.run(question)
```
Even the simplest one "hi" will return me the error saying: **ValueError: fix_text not implemented for this agent.**
So my question is Why did this error emerge? How to avoid the error?



### Expected behavior

The expected output should be similar to the few-shot prompts
Added Dependencies Status, Open issues and releases badges in Readme.md

[![Dependency Status](https://img.shields.io/librariesio/github/hwchase17/langchain)](https://libraries.io/github/hwchase17/langchain)
[![Open Issues](https://img.shields.io/github/issues-raw/hwchase17/langchain)](https://github.com/hwchase17/langchain/issues)
[![Release Notes](https://img.shields.io/github/release/hwchase17/langchain)](https://github.com/hwchase17/langchain/releases)
the first paragraph of code in the quick start can not run in jupyter

### System Info

Hi everyone, 
I am a beginner of Python and I can not run the first part of the code 
![1685862362254](https://github.com/hwchase17/langchain/assets/114320010/0cd40666-fc4a-431d-8c45-5a2fa3956921)

but those code is supposed to output feetful of fun
![image](https://github.com/hwchase17/langchain/assets/114320010/439335dc-d3de-4f7d-8e8d-a859f8601750)

Is there anybody who knows why? 

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

set the environment variable with openai API key and then run the code in the picture

### Expected behavior

it supposed to output something  OR  why is this problem? what should I add into the code?
Improve the recursive text splitting algorithm to minimize the number of chunks
Labels: name: auto:improvement

I notice `RecursiveCharacterTextSplitter` often produce chunks that are much smaller than the specified chunk_size. 

The problem happens because we merge the `_good_splits` and add it to the `final_chunks` as soon as we encounter a large next chunk. We can improve chunk space utilization by putting `splits` in a stack, so that during iteration we can further split the next chunk and push the smaller chunks into the stack. 

@hwchase17  What do you think about this approach?
sqlalchemy MovedIn20Warning declarative_base DEPRICATION fix 
Labels: name: lgtm

fix for the sqlalchemy deprecated declarative_base import :

```
MovedIn20Warning: The ``declarative_base()`` function is now available as sqlalchemy.orm.declarative_base(). (deprecated since: 2.0) (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
  Base = declarative_base()  # type: Any
```

Import is wrapped in an try catch Block to fallback to the old import if needed.
Fix code issue

<!--
Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.

Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.

After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
-->

<!-- Remove if not applicable -->

Fixes code issue

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->

#### Who can review?

  - @hwchase17
  - @agola11

<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @vowelparrot

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
Feature: Allows for structured output parser to parse lists of json objects 
Labels: name: auto:improvement

# Description
Enables structured output parser to parse lists of json objects

Prior it would throw an error when parsing a list of json objects about missing keys. Now it evaluates if the parsed json object is a list then uses a difference in the set of keys from each record in the json list and the set of original expected keys.

This allows for the `StructuredOutputParser` to parse multiple objects of the same schema at once

#### Who can review?

Tag maintainers/contributors who might be interested:

@hwchase17
@eyurtsev
@vowelparrot
Fix transposed properties

~Args given to ChatVertexAI were not being transmitted to the google library.~
In the course of writing the unit test, I found that top_k and top_p were transposed.
~The unit tests uses pytest's monkeypatch because I could not get mock.patch to work (I'm not used to using it, the Python doc, and gpt4 were all unhelpful, sorry).~
~If you would like to rewrite the monkeypatch using patch: by all means. I would love to see how it's done!~
~If you would like me to rewrite the patch with monkeypatch (for consistency) I would be happy to do that.~

~Fixes #5667~

#### Who can review?

Tag maintainers/contributors who might be interested:
@hwchase17
@agola11

Thanks for the amazing work you've all done on this project. I've been using it every day for months and I'm incredibly impressed!
Model settings (temp, max tokens, topk, topp) ignored with vertexai

### System Info

langchain-0.0.189, windows, python 3.11.3

### Who can help?

Will submit PR to resolve.

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

When calling ConversationChain.predict with VertexAI, the settings to the LLM are overriden.

To repro, run the following (adapted from tests.integration_tests.chat_models.test_vertexai.test_vertexai_single_call) :
```python
from langchain.chat_models import ChatVertexAI
from langchain.schema import (
    HumanMessage,
)

model = ChatVertexAI(
    temperature=1,
    max_output_tokens=1,
    top_p=1,
    top_k=1,
)
message = HumanMessage(content="Tell me everything you know about science.")
response = model([message])
assert len(response.content) < 50
```

### Expected behavior

Expected: the assertion should be true if only 1 token was sent back. Allowing for mistakes in the LLM, I set very generous threshold of 50 characters.
Add aviary support
Labels: name: enhancement, name: llms

Aviary is an open source toolkit for evaluating and deploying open source LLMs. You can find out more about it on [http://github.com/ray-project/aviary). You can try it out at [http://aviary.anyscale.com](aviary.anyscale.com). 

This code adds support for Aviary in LangChain. To minimize dependencies, it connects directly to the HTTP endpoint. 

The current implementation is not accelerated and uses the default implementation of `predict` and `generate`. 

It includes a test and a simple example. 

@hwchase17 and @agola11 could you have a look at this?
Support for Hugging Face Question Answering Models

### Feature request

Support for Hugging Face's Question Answering Models would be great.

### Motivation

It would allow for the construction of extractive Question Answering Pipelines, making use of the vast amount of open Q&A Models on Hugging Face.

### Your contribution

I would be happy to tackle the implementation and submitt a PR. Have you thought about integrating this already?
If so, I'm open for a discussion on the best way to to this. (New Class vs extending 'HuggingFacePipeline').
Created fix for 5475

Created fix for 5475
Currently in PGvector, we do not have any function that returns the instance of an existing store. The from_documents always adds embeddings and then returns the store. This fix is to add a function that will return the instance of an existing store

Also changed the jupyter example for PGVector to show the example of using the function

<!-- Remove if not applicable -->

Fixes # 5475

#### Before submitting

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests, lint
etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->

#### Who can review?
@dev2049
@hwchase17 

Tag maintainers/contributors who might be interested:

<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @vowelparrot

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
DOC: Annotation error in the CAMELAgent helper class

### Issue with current documentation:

**Severity:** Minor

**Where:** 
Use Cases -> Agent Simulations -> Simulations with two agents -> CAMEL

Direct link:
[CAMEL Role-Playing Autonomous Cooperative Agents](https://python.langchain.com/en/latest/use_cases/agent_simulations/camel_role_playing.html)

**What:**
In the code of the CAMELAgent, the reset_method has the wrong type annotation.

Should be:

```python

def reset(self) -> List[BaseMessage]:
    self.init_messages()
    return self.stored_messages

```

### Idea or request for content:

_No response_
Issue: When using agent_chain.run to query documents, I want to return both the LLM answer and Source documents, however agent_chain.run only allows for exactly 1 output

### Issue you'd like to raise.

agent_chain.run does not allow for more than a single output, when calling RetrievalQA.from_chain_type directly with the same query it has no issues returning multiple outputs, however agent_chain is where all of my custom tools and configurations are.

agent_chain.run in my code at the moment refers to AgentExecutor.from_agents_and_tools

Is there a current workaround for this or is there a strong reason why .run can only support a single output?

### Suggestion:

_No response_
Support similarity_score_threshold retrieval with Chroma
Labels: name: lgtm

Fixes https://github.com/hwchase17/langchain/issues/5067

Verified the following code now works correctly:
```
db = Chroma(persist_directory=index_directory(index_name), embedding_function=embeddings)
retriever = db.as_retriever(search_type="similarity_score_threshold", search_kwargs={"score_threshold": 0.4})
docs = retriever.get_relevant_documents(query)
```
New features and tests for MW Dump Loader

Adds new paramerters for to the MWDumpLoader class:
* **skip_redirecst** (_bool_) Tells the loader to skip articles that redirect to other articles. `False` by default.
* **stop_on_error** (_bool_) Tells the parser to skip any page that causes a parse error. `True` by default.
* **namespaces** (_List[int]_) Tells the parser which namespaces to parse. Contains namespaces from -2 to 15 by default.

Default values are chosen to preserve backwards compatability.

Sample dump XML and tests are also included!

 
Tagging people for checks / review:
  - @eyurtsev
Extend `langchain.callbacks` documentation

Hi to whoever is reading this! 🤗

# What's in this PR?

As of the recent addition of `ArgillaCallbackHandler` in the `master` branch of `LangChain`, merged by @dev2049 (but maybe also relevant to @agola11 since it's callback-related), @davidberenstein1957 and @dvsrepo noticed that the `Callbacks -> How to guides -> Argilla` page was missing, so I've extended the `langchain.callbacks` documentation a little bit to include the "How to guides" to also allow any other callback integration to live there, and add the missing API Reference for the callbacks.
SVM-based SKLearnVectorStore and filtering options for SKLearnVectorStores
Labels: name: area: vector store, name: auto:improvement

In this PR I have made two additions to `SKLearnVectorStore`

## SVM based VectorStore: 

Instead of kNN, an SVM can be used for the retrieval step. This "trick" recently got some attention due to [this tweet](https://twitter.com/karpathy/status/1647025230546886658?s=2). The tweet also discusses possible advantages (better retrieval quality) and disadvantages (slower than kNN).

For the implementation, I split the `SKLearnVectorStore` class and created an abstract class `SKLearnVectorStoreBase` for which `_similarity_index_search_with_score` needs to be implemented. This abstract class is used to implement a kNN- and SVM-based vector store.

## Filtering

Both the kNN and SVM vector stores can now be used with metadata filters. The filters are either strings, where a `==` comparison is performed, or a callable that is called with the value of the metadata entry and should return a bool. When using filters, the retrieval algorithm must always be retrained, which slows down the retrieval step in the case of kNN.


Edit: I think  @dev2049 is the maintainer for this part of the code.