delayedkarma/langchain-issues huggingface dataset:
FileCallbackHandler should open file in UTF-8 encoding
Labels: name: bug, name: enhancement

### System Info

Langchain 0.0.200
Python 3.11.4
Windows 10

### Who can help?

@agola11

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [X] Callbacks/Tracing
- [ ] Async

### Reproduction

Use the handler with a prompt containing characters that aren't represented on the platform charset (e.g. cp1252)

Error appears in console:
```
[manager.py:207] Error in on_text callback: 'charmap' codec can't encode character '\u2010' in position 776: character maps to <undefined>
```

Target logfile only has "entering new chain" and "finished chain" lines.

### Expected behavior

1. No error
2. Log file has usual output

Fix: in constructor: ```self.file = cast(TextIO, open(file_path, "a", encoding="utf-8"))```
How to add memory to SQLDatabaseChain?
Labels: name: question, name: needs work

### Issue you'd like to raise.

I want to create a chain to make query against my database. Also I want to add memory to this chain.

Example of dialogue I want to see:

Query: Who is an owner of website with domain domain.com? 
Answer: Boba Bobovich 
Query: Tell me his email Answer: 
Boba Bobovich's email is boba@boba.com

I have this code:

```
import os
from langchain import OpenAI, SQLDatabase, SQLDatabaseChain, PromptTemplate
from langchain.memory import ConversationBufferMemory

memory = ConversationBufferMemory()
db = SQLDatabase.from_uri(os.getenv("DB_URI"))
llm = OpenAI(temperature=0, verbose=True)
db_chain = SQLDatabaseChain.from_llm(llm, db, verbose=True, memory=memory)

db_chain.run("Who is owner of the website with domain https://damon.name")
db_chain.run("Tell me his email")
print(memory.load_memory_variables({}))
```

It gives:


```
> Entering new  chain...
Who is owner of the website with domain https://damon.name
SQLQuery:SELECT first_name, last_name FROM owners JOIN websites ON owners.id = websites.owner_id WHERE domain = 'https://damon.name' LIMIT 5;
SQLResult: [('Geo', 'Mertz')]
Answer:Geo Mertz is the owner of the website with domain https://damon.name.
> Finished chain.

> Entering new  chain...
Tell me his email
SQLQuery:SELECT email FROM owners WHERE first_name = 'Westley' AND last_name = 'Waters'
SQLResult: [('Ken70@hotmail.com',)]
Answer:Westley Waters' email is Ken70@hotmail.com.
> Finished chain.
{'history': "Human: Who is owner of the website with domain https://damon.name\nAI: Geo Mertz is the owner of the website with domain https://damon.name.\nHuman: Tell me his email\nAI: Westley Waters' email is Ken70@hotmail.com."}
```

Well, it saves context to memory but chain doesn't use it to give a proper answer (wrong email). How to fix it?

Also I don't want to use an agent because I want to manage to do this with a simple chain first. Tell me if it's impossible with simple chain.

### Suggestion:

_No response_
make - ModuleNotFoundError: No module named 'scripts'
Labels: name: bug, name: question

### System Info

OS: `Ubuntu 22.04`
Langchain Version: `0.0.219`
Poetry Version: `1.5.1`
Python Version: `3.10.6`

### Who can help?

@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I would like to contribute to the project, I followed instructions for [contributing](https://github.com/hwchase17/langchain/blob/8502117f62fc4caa53d504ccc9d4e6a512006e7f/.github/CONTRIBUTING.md) and have installed poetry.

After setting up the virtual env, the command: `poetry install -E all` runs successfully :heavy_check_mark: 
However when trying to do a `make test` I get the following error:

```
Traceback (most recent call last):
  File "[redacted]/langchain/.venv/bin/make", line 5, in <module>
    from scripts.proto import main
ModuleNotFoundError: No module named 'scripts'
```
I have the virtual env activated and the command `make` is using the one installed in the virtual env. (see above error)

I'm assuming I'm missing a dependency but it's not obvious to determine, can you help please?

### Expected behavior

`make` should run properly
Fix run_inline to run hooks in the same context
Labels: name: bug, name: callbacks

Since https://github.com/hwchase17/langchain/commit/e1b801be36d2d492eea071dcb594e966414b31cf, run_inline flags on callbacks (introduced in [PR](https://github.com/hwchase17/langchain/pull/6424)) are broken.

Because higher-level on_llm_start uses asyncio.gather, it's creates a task for given coroutines, and changes the hooks' context. Hence, it's not run inline and context is lost. Added test shows that if ran without changes in the manager.py.

This approach changes the interface of `_ahandle_event` to be synchronous method that returns an awaitable (I think it doesn't change much), which is a result of one `asyncio.wait` statement.

This ensures that `run_inline` hooks will be run before async hooks, but it doesn't ensure that relative order of inline hooks is preserved.

I'm not sure if that's a good approach, but posting it anyway so we can discuss the direction.
Add JSON Lines support to JSONLoader
Labels: name: area: doc loader

**Description**:

The JSON Lines format is used by some services such as OpenAI and HuggingFace. It's also a convenient alternative to CSV.

This PR adds JSON Lines support to `JSONLoader` and also updates related tests.

**Tag maintainer**: @rlancemartin, @eyurtsev.

PS I was not able to build docs locally so didn't update related section.
Stream Mode does not recognize openai.error
Labels: name: bug

### System Info

langchain-0.0.219
python-3.9
macos-13.4.1 (22F82)

### Who can help?

@agola11 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [X] Async

### Reproduction

1. use async style request in stream mode
```python
ChatOpenAI(streaming=True).agenerate([...])
```
2. Got an error if openai sdk raises `openai.error.APIError`
![image](https://github.com/hwchase17/langchain/assets/18662395/efacc3e8-f075-4b36-877e-afd23bb9ef88)
3. `acompletion_with_retry` not effective because the exception was thrown by `"openai/api_requestor.py", line 763, in _interpret_response_line` which is after “request”
in: tenacity/_asyncio.py
![tenacity/_asyncio.py](https://github.com/hwchase17/langchain/assets/18662395/be78d17a-b0f3-4f25-941b-f1a50cd0708d)


### Expected behavior

Catch exception, and do retry.
Create repo1

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Extend the QA use cases by DataChad example

  - Description: Add an use case example to question_answering section
  - Issue: tN/A
  - Dependencies: N/A
  - Tag maintainer: @hwchase17 
  - Twitter handle: @thegreatgustby
  
  @hwchase17 we had spoken in twitter dms (12th May) to include the project in the use-cases section.
  I forgot to do it since, so wanted to do it now. Hope it's still accepted.
  Thanks!
Add API Header for Amazon API Gateway Authentication

Add API Headers support for Amazon API Gateway to enable Authentication using DynamoDB.

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
load_prompt Unable to set encoding for JSON files
Labels: name: bug, name: enhancement

### System Info

llm = OpenAI(temperature=0)
chain = LLMChain(llm=llm, prompt=load_prompt("coolapk_prompt.json"))

config = json.load(f)
UnicodeDecodeError: 'gbk' codec can't decode byte 0xaf in position 108: illegal multibyte sequence

### Who can help?


llm = OpenAI(temperature=0)
chain = LLMChain(llm=llm, prompt=load_prompt("coolapk_prompt.json"))

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

 
llm = OpenAI(temperature=0)
chain = LLMChain(llm=llm, prompt=load_prompt("coolapk_prompt.json"))

### Expected behavior

add set encoding for JSON files
The ReDOS Attack for the regex in langchain
Labels: name: bug, name: enhancement

### System Info

python 3.9
langchain 0.0.199

### Who can help?

@hwchase17 @agola11 @eyu

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

The regex such as https://github.com/hwchase17/langchain/blob/6370808d41eda0d056375015fda9284e9f01280c/langchain/agents/mrkl/output_parser.py#L18
will raise a ReDOS when the model output something below
`'Action' + '\n'*n + ':' + '\t'*n + '\t0Input:' + '\t'*n + '\nAction' + '\t'*n + 'Input' + '\t'*n + '\t0Input:ActionInput:'`
full code:
```
import re
n=780
llm_output='Action' + '\n'*n + ':' + '\t'*n + '\t0Input:' + '\t'*n + '\nAction' + '\t'*n + 'Input' + '\t'*n + '\t0Input:ActionInput:'
regex = (
            r"Action\s*\d*\s*:[\s]*(.*?)[\s]*Action\s*\d*\s*Input\s*\d*\s*:[\s]*(.*)"
        )
match = re.search(regex, llm_output, re.DOTALL)
action = match.group(1).strip()
print(action)
action_input = match.group(2)
```
To slow the regex

### Expected behavior

```
Protection
There are several things you can do to protect yourself from ReDoS attacks.

1.Look at safer alternative libraries such as Facebook’s [pyre2](https://pypi.org/project/pyre2/), which is a python wrapper around Google’s C++ Regex Library, [re2](https://github.com/google/re2/).
2.Always double check all regex you add to your application and never blindly trust regex patterns you find online.
3.Utilize SAST and fuzzing tools to test your own code, and check out [Ochrona](https://ochrona.dev/) to make sure your dependencies are not vulnerable to ReDoS attacks.
4.If possible, limit the length of your input to avoid longer than necessary strings.
```
from  https://medium.com/ochrona/python-dos-prevention-the-redos-attack-7267a8fa2d5c

Also you can check your regex online https://devina.io/redos-checker.
chore(openllm): make sure to download the model if not exists
Labels: name: lgtm

OpenLLM Runner now include a ensure_available argument to separate the process
of automatic setup of the given model. This was introduced to allow better UX
with openllm.

I updated the OpenLLM accordingly, since we are just loading the model into
memory in "local" mode, `ensure_available` should set to `True`

cc @dev2049

Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change,
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Re-add Support for SQLAlchemy <1.4
Labels: name: enhancement, name: breaking

Support for SQLAlchemy 1.3 was removed in version 0.0.203 by change #6086. Re-adding support.

  - Description: Imports SQLAlchemy Row at class creation time instead of at init to support SQLAlchemy <1.4. This is the only breaking change and was introduced in version 0.0.203 #6086.
  
  A similar change was merged before: https://github.com/hwchase17/langchain/pull/4647
  
  - Dependencies: Reduces SQLAlchemy dependency to > 1.3
  - Tag maintainer: @rlancemartin, @eyurtsev, @hwchase17, @wangxuqi
Add pg_hnsw vectorstore integration
Labels: name: enhancement, name: needs documentation

Hi @rlancemartin, @eyurtsev!

  - Description: Adding HNSW extension support for Postgres. Similar to pgvector vectorstore, with 3 differences 
      1. it uses HNSW extension for exact and ANN searches, 
      2. Vectors are of type array of real
      3. Only supports L2
      
  - Dependencies: [HNSW](https://github.com/knizhnik/hnsw) extension for Postgres
  
  - Example:
  ```python
    db = HNSWVectoreStore.from_documents(
      embedding=embeddings,
      documents=docs,
      collection_name=collection_name,
      connection_string=connection_string
  )
  
  query = "What did the president say about Ketanji Brown Jackson"
  docs_with_score: List[Tuple[Document, float]] = db.similarity_search_with_score(query) 
  ```

The example notebook is in the PR too.
SQLDatabaseChain cannot be loaded
Labels: name: bug

### System Info

langchain==0.0.217

### Who can help?

@hwchase17 

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

The following code fails:
```python
def create_sqlite_db_file(db_dir):
    # Connect to SQLite database (or create it if it doesn't exist)
    conn = sqlite3.connect(db_dir)

    # Create a cursor
    c = conn.cursor()

    # Create a dummy table
    c.execute('''
        CREATE TABLE IF NOT EXISTS employees(
            id INTEGER PRIMARY KEY,
            name TEXT,
            salary REAL,
            department TEXT,
            position TEXT,
            hireDate TEXT);
        ''')

    # Insert dummy data into the table
    c.execute('''
        INSERT INTO employees (name, salary, department, position, hireDate)
        VALUES ('John Doe', 80000, 'IT', 'Engineer', '2023-06-26');
        ''')

    # Commit the transaction
    conn.commit()

    # Close the connection
    conn.close()

def test_log_and_load_sql_database_chain(tmp_path):
    # Create the SQLDatabaseChain
    db_file_path = tmp_path / "my_database.db"
    sqlite_uri = f"sqlite:///{db_file_path}"
    llm = OpenAI(temperature=0)
    create_sqlite_db_file(db_file_path)
    db = SQLDatabase.from_uri(sqlite_uri)
    db_chain = SQLDatabaseChain.from_llm(llm, db)

    db_chain.save('/path/to/test_chain.yaml')
    from langchain.chains import load_chain
    loaded_chain = load_chain('/path/to/test_chain.yaml', database=db)
```
Error:
```
    def load_llm_from_config(config: dict) -> BaseLLM:
        """Load LLM from Config Dict."""
>       if "_type" not in config:
E       TypeError: argument of type 'NoneType' is not iterable
```

### Expected behavior

No error should occur.
Update `langchain.chains.create_extraction_chain_pydantic` to parse results successfully
Labels: name: bug, name: lgtm

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @dev2049
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @dev2049
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @vowelparrot
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
 
- Description: 
  - The current code uses `PydanticSchema.schema()` and `_get_extraction_function` at the same time. As a result, a response from OpenAI has two nested `info`, and `PydanticAttrOutputFunctionsParser` fails to parse it. This PR will use the pydantic class given as an arg instead.
- Issue: no related issue yet
- Dependencies: no dependency change
- Tag maintainer: @dev2049
- Twitter handle: @shotarok28
MultiQueryReceiver does not work
Labels: name: bug

### System Info

 langchain-0.0.218
Python 3.10.10


### Who can help?


@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Cannot import MultiQueryRetriever, because when i run it, it says:

Traceback (most recent call last):
  File "C:\langchain\JETPACK-FLOW-MULTIQUERY.py", line 14, in <module>
    from langchain.retrievers.multi_query import MultiQueryRetriever
  File "C:\Users\roman\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain\retrievers\__init__.py", line 21, in <module>
    from langchain.retrievers.self_query.base import SelfQueryRetriever
  File "C:\Users\roman\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain\retrievers\self_query\base.py", line 9, in <module>
    from langchain.chains.query_constructor.base import load_query_constructor_chain
  File "C:\Users\roman\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain\chains\query_constructor\base.py", line 14, in <module>
    from langchain.chains.query_constructor.parser import get_parser
  File "C:\Users\roman\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain\chains\query_constructor\parser.py", line 9, in <module>
    raise ValueError(
ValueError: Lark should be at least version 1.1.5, got 0.12.0

There was same error reported in this bug report:
https://github.com/hwchase17/langchain/discussions/6068
Their solution was simply to install langchain version 0.0.201, but for me it does not help, because there is no MultiQueryReceiver in this version. 

I am using: langchain-0.0.218



### Expected behavior

Does not crash, when i try to run it.
Include pruning on VectorStoreRetrieverMemory
Labels: name: enhancement

### Feature request

Several memory storage options have a "buffer" type which includes a pruning function that allows users to specify a max_tokens_limit which stops token overflow issues. I'm requesting this be included with the VectorStoreRetrieverMemory as well.

### Motivation

Running into token overflow issues is annoying and a pruning option is a simple way to get around this when using vector store based memory.

### Your contribution

I'm happy to attempt a solution on this given I'm able to use my company computer this weekend. Otherwise I would appreciate any support people are able to give.