delayedkarma/langchain-issues huggingface dataset:
Issue: UnicodeDecodeError when running .txt files using TextLoader
Labels: name: area: doc loader, name: auto:bug

### Issue you'd like to raise.

Hello - I receive a UnicodeDecodeError when running the below code:

from dotenv import load_dotenv
load_dotenv()
from langchain.llms import OpenAI
from langchain.vectorstores import Chroma
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.chains import RetrievalQA
llm = OpenAI(temperature=0.1)
from langchain.document_loaders import TextLoader
loader = TextLoader("./Training/test2.txt")
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(documents)
embeddings = OpenAIEmbeddings()
docsearch = Chroma.from_documents(texts, embeddings)
qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=docsearch.as_retriever(search_kwargs={"k": 1}))
query = "How much did the judge fine Twitter?"
qa.run(query)

The test2.txt is a 1,600-word UTF-8 encoded file. Here is the error text that I receive:

**Traceback (most recent call last):
  File "C:\Users\Admin\PycharmProjects\pythonProject\Misc\Testing.py", line 17, in <module>
    documents = loader.load()
                ^^^^^^^^^^^^^
  File "C:\Users\Admin\PycharmProjects\pythonProject\Misc\venv\Lib\site-packages\langchain\document_loaders\text.py", line 18, in load
    text = f.read()
           ^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1079: character maps to <undefined>**

Any advice would be greatly appreciated.

### Suggestion:

_No response_
Chat LLM adds an extra sentence in front of SQL queries, produces Output Parsing Error
Labels: name: area: agent, name: auto:bug, name: area: models

### System Info

python==3.11
langchain==0.0.246


### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. db_chain = SQLDatabaseChain.from_llm(chat_llm, db, use_query_checker=True, verbose=True)#, query_prompt=PROMPT)
2. chat_llm=AzureChatOpenAI(deployment_name="gpt-35-turbo-16k",
                                         model_name="gpt-35-turbo-16k" )
3. tools=[Tool(
            func=db_chain.run,
            name="Database Search",
            description="useful for when you need to lookup specific queries on the ATLASIQ Schema"
            )]
4. prefix = """ You are an expert Snowflake SQL data analyst, who writes queries with perfect syntax, 
and performs necessary computations on that data in the AtlasIQ Schema. Your goal is to answer the following questions as best you can. 
When there are multiple results for the same quantity, return all of them. DO NOT hallucinate an answer if there is no result."""
suffix = """Begin!"

{chat_history}
Question: {input}
{agent_scratchpad}"""

prompt = ZeroShotAgent.create_prompt(
            tools,
                prefix=prefix,
                    suffix=suffix,
                        input_variables=["input", "chat_history", "agent_scratchpad"],
                        )
memory = ConversationBufferMemory(memory_key="chat_history")

llm_chain = LLMChain(llm=chat_llm, prompt=prompt)#,  callbacks=[custom_handler.short_chain(

new_zero_agent = ZeroShotAgent(llm_chain=llm_chain, tools=tools, verbose=True)
agent_chain = AgentExecutor.from_agent_and_tools(
            agent=new_zero_agent, tools=tools, verbose=True, memory=memory, handle_parsing_errors=True)

agent_chain.run(query)


### Expected behavior

I expect the query to be synthesized and executed but instead, I get an unnecessary explanation of why the query is correct "The original query does not contain any of the mentioned mistakes. Therefore, here is the reproduced original query:

SELECT SUM(total_operating_expenses) 
FROM expense_income_table 
WHERE market = 'Seattle' 
AND period = '202305'The original query provided does not contain any of the mentioned mistakes. Hence, the original query is reproduced below:

```sql
SELECT SUM(total_operating_expenses) 
FROM expense_income_table 
WHERE market = 'Seattle"<- which causes the output parsing error
DOC: What are the default values to RecursiveCharacterTextSplitter
Labels: name: area: doc loader, name: auto:question, name: auto:documentation

### Issue with current documentation:

I am using the default configuration for RecursiveCharacterTextSplitter

    text_splitter = RecursiveCharacterTextSplitter()
    chunks = text_splitter.create_documents([full_text])

However, I would like to know what values are being used in the defaults. I would like to explicitly pass values, because for certain documents it seems to be not splitting and downstream I am getting from OpenAI's embeddings endpoint `This model's maximum context length is 8191 tokens, however you requested 11451 tokens (11451 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.`



### Idea or request for content:

 I would like to know what values are being used as default for RecursiveCharacterTextSplitter
`AirbyteStripeLoader` throws an error
Labels: name: area: doc loader, name: auto:bug

### System Info

When importing the `AirbyteStripeLoader` introduced in v0.0.259 it throws an `module not found error`. 

```
from libs.langchain.langchain.utils.utils import guard_import

ModuleNotFoundError: No module named 'libs'
```

### Who can help?

@eyurtsev 

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
from langchain.document_loaders.airbyte import AirbyteStripeLoader

config = {
    # your stripe configuration
}

loader = AirbyteStripeLoader(config=config, stream_name="invoices") # check the documentation linked above for a list of all streams
```

### Expected behavior

Import the document should not throw an error.
Include tokens from function output in streaming callbacks
Labels: name: auto:improvement, name: area: models

Description
---
This adds a small bit of logic so that `on_new_llm_token` callbacks will be properly passed a valid token (instead of `''`) when a streaming model is called as part of a function call or tool run.

Issue
---
This is a *partial* fix for #7385, having only implemented this logic for `ChatOpenAI`. As @rasca mentions in that issue:

> We need to add this to every llm/

I haven't gone this far, as I don't have reliable (or any) access to many of the LLMs that are integrated, but am using OpenAI's models for my own development.

Dependencies
---
There are no extra dependencies required for this fix.

Tag Maintainer
---
I have a feeling this one should probably be looked at by @agola11.

Twitter Handle
---
What's Twitter? 🤣 

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
run() got an unexpected keyword argument 'name'
Labels: name: auto:bug, name: area: models

### Issue you'd like to raise.

Hey y'all. i have this code. i wrote it one month back and when i am running it now, it is giving me an error.

**CODE**:

<code>import os
from langchain.llms import OpenAI
from langchain import PromptTemplate
from langchain.chains import LLMChain
from constant import openai_key
import streamlit as st
os.environ['OPENAI_API_KEY'] = openai_key
llm = OpenAI(temperature=0.8)
multiple_inputs = PromptTemplate(
    input_variables=["name", "age", "location"],
    template="My name is {name}, I am {age} years old and live in {location}."
)
chain = LLMChain(llm=llm, prompt=multiple_inputs)
output = chain.run(
    name="John", 
    age=30,
    location="New York"
)
print(output)
</code>


**ERROR**:

<code>Traceback (most recent call last):
  File ".\questions.py", line 19, in <module>
    output = chain.run(
TypeError: run() got an unexpected keyword argument 'name'</code>

### Suggestion:

_No response_
Out of dataset answer and reference link provided for RAG example
Labels: name: area: vector store, name: auto:bug

### System Info

Ubuntu 22
Langchain version 

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Current behaviour: 

I'm using the [RAG example](https://github.com/pinecone-io/examples/blob/master/docs/langchain-retrieval-augmentation.ipynb) and feeding my own database of [1 football article](https://www.bbc.co.uk/sport/football/65984561).
The Pinecone DB is a brand new database and only contains vectors from the football article.

When I do qa_with_sources(query="Who is Sachin Tendulkar") it provides me an answer and a link as a reference. This is not the expected behavior.

I have not fed any article about Sachin Tendulkar to the database. How and why/where from is it getting the answer and the link?

Now, If I add more articles only about football, push the vector count in the database to around 90. And then I ask the same question, query="Who is Sachin Tendulkar", it is not able to give the answer, which is the expected behavior.

I wonder if the fullness of the vector db makes it more accurate? Has anyone else seen this?

Repro: 

Create a new Vector DB on pinecone. Use [this example](https://github.com/pinecone-io/examples/blob/master/docs/langchain-retrieval-augmentation.ipynb) to feed in a [football](https://www.bbc.com/sport/football/65984561) article.

Run query="Who is Sachin Tendulkar". Note the result contains a reference and an answer. (Unexpected)

Now, create a more full db, with more articles and ask the same query. Note that the results is empty as expected.

### Expected behavior

Since the database does not contain any article or mention of Sachin Tendulkar, it should not provide any answer, and instead say "This is not mentioned in the database".
Issue in running ConversationalRetrievalChain query across multiple Opensearch indices with wildcard specification
Labels: name: area: vector store, name: auto:bug, name: area: models

### System Info

Langchain version: 0.0.257
Python version: 3.11
Opensearch-py version: 2.3.0

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I have this `Opensearch Vector DB `and I maintain multiple indices that start with "index-" (for example index-pdf, index-html). I have indexed sets of documents to each of the indices using Langchain's OpenSearchVectorSearch.from_documents() function.

Now, I want to run some queries which I want them to be run across multiple indices. An example would be "What is the title of each document?". When I execute below code, it either just outputs answer from first or last matching index, or says it cannot find the answer. Here is my current code:

```
from langchain.vectorstores import OpenSearchVectorSearch
from langchain.chains import RetrievalQA, ConversationalRetrievalChain
import os
from langchain.embeddings import OpenAIEmbeddings
from langchain.chat_models import ChatOpenAI


embeddings = OpenAIEmbeddings()

def get_llm(model):
    llm = ChatOpenAI(model_name=model.lower(), verbose=False, temperature=0)
    return llm


docsearch = OpenSearchVectorSearch(opensearch_url="http://localhost:9200",
                                           
                                            index_name="index-*",
                                            embedding_function=embeddings)



chain = ConversationalRetrievalChain.from_llm(
    llm=get_llm("gpt-3.5-turbo"),
    retriever=docsearch.as_retriever(),
)


result = chain({'question': 'What is the title of each document?', "chat_history": []})
response = result['answer']

print(response)
```
The response I get is either of the format "The document provided does not list different titles..."

### Expected behavior

Response should be span across multiple indices
Add convenience methods to ConversationBufferMemory and ConversationB…
Labels: name: area: memory, name: lgtm, name: auto:improvement

Add convenience methods to `ConversationBufferMemory` and `ConversationBufferWindowMemory` to get buffer either as messages or as string.

Helps when `return_messages` is set to `True` but you want access to the messages as a string, and vice versa.

@hwchase17

One use case: Using a `MultiPromptRouter` where `default_chain` is `ConversationChain`, but destination chains are `LLMChains`. Injecting chat memory into prompts for destination chains prints a stringified `List[Messages]` in the prompt, which creates a lot of noise. These convenience methods allow caller to choose either as needed.
Force LangChain Agent to Use a Tool
Labels: name: area: agent, name: auto:question, name: auto:improvement

### Issue you'd like to raise.

How do you force a langchain agent to use a tool and not use any information outside the tool? Currently, if the question is not related to the Tool it uses its own data to generate answer which I don't want it to be.

Use Case: I have an index which has a lot of documents. I want to make sure the agent will use the index ALWAYS to get information. If the question is not in the index it should return "I don't know" or similar.

P.S I am using OPENAI_FUNCTION agent.

### Suggestion:

_No response_
issues loading the prompt with load_prompt

### System Info

colab

```
!pip install -q langchain tiktoken openai chromadb
```


### Who can help?

@eyurtsev  @aga

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
%%writefile adjective_joke_prompt.json

{
	# what type of prompt. Currently supports "prompt" and "few_shot"
	"_type": "prompt",

	# the input variables used in the template
	"input_variables": ["adjective", "content"],
	
	# the template text of the prompt, including variable placeholders
	"template": "Tell me a {{ adjective }} joke about {{ content }}",
	# alternatively the template text can be loaded from a file
	"template_path": "adjective_joke_prompt_template.txt"
	# NOTE: both "template" and "template_path" cannot be used at the same time!
	
	# the format of the template 
	"template_format": "jinja2",
	
	# currently only the "RegexParser" is supported for "output_parser"
	# this is example of a date parser
	"output_parser": {
		"_type": "regex_parser",
		"regex": "(\\d{4})-(\\d{2})-(\\d{2})",
		"output_keys": ["year", "month", "day"]
	}
	
}
```
```
# load the prompt using a file
prompt_template = load_prompt("adjective_joke_prompt.json")

# create a prompt using the variables
prompt_template.format(adjective="funny", content="chickens")
```
gives error:
```
---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
[<ipython-input-41-7755c281a03b>](https://localhost:8080/#) in <cell line: 2>()
      1 # load the prompt using a file
----> 2 prompt_template = load_prompt("adjective_joke_prompt.json")
      3 
      4 # create a prompt using the variables
      5 prompt_template.format(adjective="funny", content="chickens")

5 frames
[/usr/lib/python3.10/json/decoder.py](https://localhost:8080/#) in raw_decode(self, s, idx)
    351         """
    352         try:
--> 353             obj, end = self.scan_once(s, idx)
    354         except StopIteration as err:
    355             raise JSONDecodeError("Expecting value", s, err.value) from None

JSONDecodeError: Expecting property name enclosed in double quotes: line 3 column 2 (char 4)
```

### Expected behavior

gives response based on template
Issue: Can't pass embedded documents with chat
Labels: name: area: memory, name: auto:bug, name: area: models

### Issue you'd like to raise.

I am trying to have a conversation with my documents like this

```typescript
const index = client.Index(indexName); //pinecode index
const queryEmbedding = await new OpenAIEmbeddings().embedQuery(question);
const queryResponse = await index.query({
  queryRequest: { topK: 10, vector: queryEmbedding, includeMetadata: true },
});
const llm = new OpenAI();
const chatHistory = new ChatMessageHistory();
const memory = new BufferMemory({ chatHistory, inputKey: "my_chat_history", memoryKey: "chat_history" });
const chain = new ConversationChain({ llm, memory, verbose: true });
const concatenatedPageContent = queryResponse.matches.map(match => (<any>match?.metadata)?.pageContent).join(" ");
const result = await chain.call({
  input_documents: [new Document({ pageContent: concatenatedPageContent })],
  input: question,
});
```

But gettiing
```
Error: Missing value for input history
```
It seems to be impossible to marry embedded docs with chatting
Human as a Tool in Production 
Labels: name: auto:question, name: area: models

### Issue you'd like to raise.

Can we use Human as a tool in product?
def get_input() -> str:
    print("Insert your text. Enter 'q' or press Ctrl-D (or Ctrl-Z on Windows) to end.")
    contents = []
    while True:
        try:
            line = input()
        except EOFError:
            break
        if line == "q":
            break
        contents.append(line)
    return "\n".join(contents)


# You can modify the tool when loading
tools = load_tools(["human", "ddg-search"], llm=math_llm, input_func=get_input)

what to do with input() in get_input() function if we need to use this tool in production?

reference:-https://python.langchain.com/docs/integrations/tools/human_tools

### Suggestion:

_No response_
qa_with_source it returns all documents, not just the ones used.
Labels: name: area: doc loader, name: auto:improvement

### Feature request

### First
The aim of qa_with_sources is to find only the documents used in the answer.

The attribute `return_source_documents` in `qa_with_sources` chain, returns all the documents. Not just the documents used to provide the answer.

I think it's not necessary because `retriever.get_relevant_documents(question)` returns the same documents list.
It has no added value, and this is not in the spirit of the chain.

I propose to add an attribute `return_used_documents` or change the semantic of `return_source_documents` to limit the result to the documents used to provide the answer.

### Second
With a long list of documents, with a big URL for each document (like a document come from google drive or Microsoft Azure), the number of tokens used is exploding.
The recursive map-reduce must be activated. At each reduction, **some URLs disappear**, and space for documents is shrinking.


### Motivation

When we use the qa_with_source, we want to be able to justify the response.
Actually, we can return a correct list of url, but not the list of associated documents.
Sometime, when the original document is split into multiple documents, all part have the same URL. It's not possible to find the corresponding documents with a list of URL.

### Your contribution

I propose a new chain [qa_with_reference](https://github.com/langchain-ai/langchain/pull/7278) without these problems, but for the moment, nothing is moving.
BagelDB (bageldb.ai), VectorStore integration.
Labels: name: needs documentation, name: area: vector store, name: auto:enhancement

-   **Description**: [BagelDB](bageldb.ai) a collaborative vector database. Integrated the bageldb PyPi package with langchain with related tests and code.

  - **Issue**: Not applicable.
  - **Dependencies**: `betabageldb` PyPi package.
  - **Tag maintainer**: @rlancemartin, @eyurtsev, @baskaryan
  - **Twitter handle**: bageldb_ai (https://twitter.com/BagelDB_ai)
  
We ran `make format`, `make lint` and `make test` locally.

Followed the contribution guideline thoroughly https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md