delayedkarma/langchain-issues huggingface dataset:
SQL Chain Result - Error
Labels: name: auto:question, name: area: models

### System Info

**### I´m directly providing these arguments to the LLMs via the prompt:**

**SOURCE CODE:**

from langchain import OpenAI, SQLDatabase
from langchain_experimental.sql import SQLDatabaseChain
from langchain.chains import create_sql_query_chain
from langchain.prompts.prompt import PromptTemplate

import environ
env = environ.Env()
environ.Env.read_env()

API_KEY = env('OPENAI_API_KEY')

db = SQLDatabase.from_uri(
    f"postgresql+psycopg2://postgres:{env('DBPASS')}@localhost:5432/{env('DATABASE')}",
)

llm = OpenAI(temperature=0, openai_api_key=API_KEY)

_DEFAULT_TEMPLATE = """Given an input question, first create a syntactically correct {dialect} query to run, then look at the results of the query and return the answer.
Use the following format:

Question:"Question here"
SQLQuery:"SQL Query to run"
SQLResult:"Result of the SQLQuery"
Answer:"Final answer here"

Only use the following tables:

{table_info}

If someone asks for the table foobar, they really mean the tasks table.

Question: {input}"""

PROMPT = PromptTemplate(
    input_variables=["input", "table_info", "dialect"], template=_DEFAULT_TEMPLATE
)
db_chain = SQLDatabaseChain.from_llm(llm, db, prompt=PROMPT, verbose=True, use_query_checker=True)

def get_prompt():
    print("Digite 'exit' para sair")

    while True:
      
        prompt = input("Entre c/ uma pergunta (prompt): ")

        if prompt.lower() == 'exit':
            print('Saindo...')
            break
        else:
            try:
                result = db_chain(prompt)
                print(result)

            except Exception as e:
                print(e)

get_prompt()

**BUT THE RESULT IS WITH [ ] AND NUMBERS:**

←[1m> Entering new SQLDatabaseChain chain...←[0m
how many tasks do we have?
SQLQuery:←[32;1m←[1;3mSELECT COUNT(*) FROM tasks;←[0m
SQLResult: ←[33;1m←[1;3m[(6,)]←[0m
Answer:←[32;1m←[1;3mWe have 6 tasks.←[0m
←[1m> Finished chain.←[0m

**HOW CAN I FIX IT?**

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I need the answes without these numbers:  [32;1m←[1;3m

### Expected behavior

I need the answes without these numbers:  [32;1m←[1;3m
Multi Tenant Support for Weaviate
Labels: name: area: vector store, name: auto:enhancement, name: auto:improvement

### Feature request

Weaviate introduced multi-tenancy support in version 1.20
https://weaviate.io/blog/multi-tenancy-vector-search

### Motivation

This can help users using Langchain + Weaviate at scale, ingesting documents and attaching tenants to it.

### Your contribution

I have implemented, but would need some help to check if everything is ok and in accordance with LangChain.

Also, I would like help on the as_retriver, as I was not able to implement multitenant on it, Yet.

the code is living here: https://github.com/dudanogueira/langchain/tree/weaviate-multitenant
Update __init__.py
Labels: name: lgtm, name: auto:improvement

Add SQLDatabaseSequentialChain Class to __init__.py so it can be accessed and used

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: SQLDatabaseSequentialChain is not found when importing Langchain_experimental package, when I open __init__.py Langchain_expermental.sql, I found that  SQLDatabaseSequentialChain is imported and add to __all__ list
  - Issue: SQLDatabaseSequentialChain is not found in Langchain_experimental package
  - Dependencies: None,
  - Tag maintainer: None,
  - Twitter handle: None,

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
mv base cache to schema
Labels: name: auto:refactor

if you remove all other imports from langchain.init it exposes this circular dep

```console
  File "/Users/bagatur/langchain/libs/langchain/langchain/__init__.py", line 6, in <module>
    from langchain.cache import BaseCache
  File "/Users/bagatur/langchain/libs/langchain/langchain/cache.py", line 56, in <module>
    from langchain.embeddings.base import Embeddings
  File "/Users/bagatur/langchain/libs/langchain/langchain/embeddings/__init__.py", line 51, in <module>
    from langchain.embeddings.sagemaker_endpoint import SagemakerEndpointEmbeddings
  File "/Users/bagatur/langchain/libs/langchain/langchain/embeddings/sagemaker_endpoint.py", line 4, in <module>
    from langchain.llms.sagemaker_endpoint import ContentHandlerBase
  File "/Users/bagatur/langchain/libs/langchain/langchain/llms/__init__.py", line 22, in <module>
    from langchain.llms.ai21 import AI21
  File "/Users/bagatur/langchain/libs/langchain/langchain/llms/ai21.py", line 6, in <module>
    from langchain.llms.base import LLM
  File "/Users/bagatur/langchain/libs/langchain/langchain/llms/base.py", line 50, in <module>
    from langchain.prompts.base import StringPromptValue
  File "/Users/bagatur/langchain/libs/langchain/langchain/prompts/__init__.py", line 40, in <module>
    from langchain.prompts.example_selector import (
  File "/Users/bagatur/langchain/libs/langchain/langchain/prompts/example_selector/__init__.py", line 4, in <module>
    from langchain.prompts.example_selector.semantic_similarity import (
  File "/Users/bagatur/langchain/libs/langchain/langchain/prompts/example_selector/semantic_similarity.py", line 9, in <module>
    from langchain.vectorstores.base import VectorStore
  File "/Users/bagatur/langchain/libs/langchain/langchain/vectorstores/__init__.py", line 50, in <module>
    from langchain.vectorstores.matching_engine import MatchingEngine
  File "/Users/bagatur/langchain/libs/langchain/langchain/vectorstores/matching_engine.py", line 10, in <module>
    from langchain.embeddings import TensorflowHubEmbeddings
ImportError: cannot import name 'TensorflowHubEmbeddings' from partially initialized module 'langchain.embeddings' (most likely due to a circular import) (/Users/bagatur/langchain/libs/langchain/langchain/embeddings/__init__.py)
```
ConversationalRetrievalChain [got multiple argument for question_generator]
Labels: name: auto:bug, name: area: models

### System Info

Getting error: got multiple values for keyword argument- question_generator .

return cls(\nTypeError: langchain.chains.conversational_retrieval.base.ConversationalRetrievalChain() got multiple values for keyword argument \'question_generator\'', 'SystemError'

`Qtemplate = (
"Combine the chat history and follow up question into "
"a standalone question. Chat History: {chat_history}"
"Follow up question: {question} withoud changing the real meaning of the question itself."
)
CONDENSE_QUESTION_PROMPT = PromptTemplate.from_template(Qtemplate)
question_generator_chain = LLMChain(llm=OpenAI(openai_api_key=openai.api_key), prompt=CONDENSE_QUESTION_PROMPT)
chain = ConversationalRetrievalChain.from_llm(
llm=llm,
retriever=self.vector_store.as_retriever(),
combine_docs_chain_kwargs=chain_type_kwargs,
verbose=True,
return_source_documents=True,
get_chat_history=lambda h: h,
memory=window_memory,
question_generator=question_generator_chain

)`

### Who can help?

@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Steps to reproduct the behaviour:
1. Generate standalone question, which does not change the meaning of question, if it is changing meaning of question, keep question as it is.
2. Generate output using memory and get most accurate answer.


### Expected behavior

Expecting the right code for implementing same functionality.
Fix json parsing error of MULTI_PROMPT_ROUTER_TEMPLATE
Labels: name: auto:bug

The output at times lacks the closing markdown code block. The prompt is changed to explicitly request the closing backticks.

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
Optional Arguments Treated as Required by "format_tool_to_openai_function"
Labels: name: auto:bug, name: area: models

### System Info

I have a tool with one required argument _chart_data_ and one optional argument _chart_title_. The tool is defined using the BaseModel class from Pydantic and is decorated with @tool("charts", args_schema=ChartInput).

However, optional arguments are pushed into the 'required' list that is being passed to OpenAI.

Do you have any suggestions for resolving this issue? GPT-3.5 consistently prompts for the chart_title argument, even though it's supposed to be optional.

### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Here's the code snippet:

langchain==0.0.274
```
class ChartInput(BaseModel):
    chart_data: list = Field(..., description="Data for chart")
    chart_title: Optional[str] = Field(None, description="The title for the chart.")


@tool("charts", args_schema=ChartInput)
def charts_tool(chart_data: list, chart_title: Optional[str]=None):
    '''useful when creating charts'''

    return 'chart image url'
```



### Expected behavior

When printing out the output of: format_tool_to_openai_function(charts_tool) , I see can chart_title which is an optional argument is pushed into the required args list.

`{'name': 'charts', 'description': 'charts(chart_data: list, chart_title: Optional[str] = None) - useful when creating charts', 'parameters': {'type': 'object', 'properties': {'chart_data': {'title': 'Chart Data', 'description': 'data for chart', 'type': 'array', 'items': {}}, 'chart_title': {'title': 'Chart Title', 'description': 'The title for the chart.', 'type': 'string'}}, 'required': ['chart_data', 'chart_title']}}`
perf: update repositories
Labels: name: area: agent, name: auto:improvement

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
addition of cosine distance function for faiss
Labels: name: area: vector store, name: auto:improvement

  - Description: added the _cosine_relevance_score_fn to _select_relevance_score_fn of faiss.py to enable the use of cosine distance for similarity for this vector store and to comply with the Error Message, that implies, that cosine should be a valid distance strategy
  - Issue: no relevant Issue found, but needed this function myself and tested it in a private repo
  - Dependencies: none
Add security notices on PAL and CPAL experimental chains.
Labels: name: auto:documentation

Clearly document that the PAL and CPAL techniques involve generating code, and that such code must be properly sandboxed and given appropriate narrowly-scoped credentials in order to ensure security.

While our implementations include some mitigations, Python and SQL sandboxing is well-known to be a very hard problem and our mitigations are no replacement for proper sandboxing and permissions management. The implementation of such techniques must be performed outside the scope of the Python process where this package's code runs, so its correct setup and administration must therefore be the responsibility of the user of this code.
Add notice about security-sensitive experimental code to experimental README.
Labels: name: lgtm, name: auto:documentation

It renders like this: https://github.com/langchain-ai/langchain/tree/pg/experimental-readme/libs/experimental

![image](https://github.com/langchain-ai/langchain/assets/2348618/a5f9569d-96f6-44c6-8559-921adb3e337d)
Issue: ParentDocumentRetriever is slow with FAISS because add_documents uses embed_query without batching
Labels: name: area: embeddings, name: area: vector store, name: auto:bug, name: auto:improvement

### Issue you'd like to raise.

I wanted to use the new `ParentDocumentRetriever` but found the indexing extremely slow. I think the reason
is this line here https://github.com/langchain-ai/langchain/blob/e80834d783c6306a68df54e6251d9fc307aee87c/libs/langchain/langchain/retrievers/parent_document_retriever.py#L112

the `add_documents` for FAISS calls the `embed_texts` function in a list comprehension here https://github.com/langchain-ai/langchain/blob/e80834d783c6306a68df54e6251d9fc307aee87c/libs/langchain/langchain/vectorstores/faiss.py#L166

This only embeds a single chunk of text at a time which is really slow, especially when using OpenAIEmbeddings. Replacing this with a call to OpenAIEmbeddings.embed_documents(docs) would result in a huge speedup as it batches things up per API call (default batch size of a 1000). 

I replaced the `self.vectorstore.add_documents(docs)` with 

```python
texts = [doc.page_content for doc in docs]
metadatas = [doc.metadata for doc in docs]
embeddings = OpenAIEmbeddings().embed_documents([doc.page_content for doc in docs])
self.vectorstore._FAISS__add(texts, embeddings, metadatas)
```

But a more general solution would be needed where because on initialisation only the `embed_function` is stored and not the underlying embedding model itself.


### Suggestion:

_No response_
Cube semantic loader: allow cubes processing
Labels: name: area: doc loader, name: auto:improvement

We've started to receive feedback (after launch) that using only views is confusing.
We're considering this as a good practice, as a view serves as a "facade" for your data - however, we decided to let users decide this on their own.

Solves the questions from:
- https://github.com/cube-js/cube/issues/7028
- https://github.com/langchain-ai/langchain/pull/9690
WebBaseLoader encoding issue
Labels: name: area: doc loader, name: auto:bug

### System Info

LangChain 0.0.275
Python 3.10.12
(Google Colaboratory)

### Who can help?

Hello, @eyurtsev!
We found an issue related to `WebBaseLoader`.

I guess the problem is related to `Response.apparent_encoding` leveraged by `WebBaseLoader. chardet.detect()`, which assigns the `apparent_encoding` to a Response object, cannot detect a proper encoding for the document.

Please find the details below.

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. Import WebBaseLoader
2. Load the test url with WebBaseLoader
3. Check the content
```
from langchain.document_loaders import WebBaseLoader

url = 'https://huggingface.co/docs/transformers/v4.32.0/ko/tasks/sequence_classification'
loader = WebBaseLoader(url)
data = loader.load()
for k, v in data[0].metadata.items():
    print(f"{k} : {v}")
content = data[0].page_content
print("content : ")
content[5000:5300] # to truncate the front of the documents with many new lines.
```
then you can see the output as below:
```
source : https://huggingface.co/docs/transformers/v4.32.0/ko/tasks/sequence_classification
title : í…�ìŠ¤íŠ¸ ë¶„ë¥˜
description : Weâ€™re on a journey to advance and democratize artificial intelligence through open source and open science.
language : No language found.
content : 
‹¤.
ì�´ ê°€ì�´ë“œì—�ì„œ í•™ìŠµí•  ë‚´ìš©ì�€:
IMDb ë�°ì�´í„°ì…‹ì—�ì„œ DistilBERTë¥¼ íŒŒì�¸ íŠœë‹�í•˜ì—¬ ì˜�í™” ë¦¬ë·°ê°€ ê¸�ì •ì �ì�¸ì§€ ë¶€ì •ì �ì�¸ì§€ íŒ�ë‹¨í•©ë‹ˆë‹¤.
ì¶”ë¡ ì�„ ìœ„í•´ íŒŒì�¸ íŠœë‹� ëª¨ë�¸ì�„ ì‚¬ìš©í•©ë‹ˆë‹¤.
ì�´ íŠœí† ë¦¬ì–¼ì—�ì„œ ì„¤ëª…í•˜ëŠ” ì�‘ì—…ì�€ ë‹¤ì�Œ ëª¨ë�¸ ì•„í‚¤í…�ì²˜ì
```
To our knowledge, this is the only case that suffers from this issue.

### Expected behavior

We want to work like the below(another webpage):

```
source : https://www.tensorflow.org/versions?hl=ko
title : TensorFlow API 버전  |  TensorFlow v2.13.0
language : ko-x-mtfrom-en
content : 
TensorFlow API 버전  |  TensorFlow v2.13.0
    설치
    학습
                      소개          
                      TensorFlow를 처음 사용하시나요?
                      TensorFlow
                      핵심 오픈소스 ML 라이브러리
```
WebBaseLoader can detect encoding properly for almost all webpages that we know of.
LangChain cannot deal with new OpenLLM Version (0.2.27)
Labels: name: auto:bug, name: area: models

### System Info

LangChain: 0.0.275
OpenLLM: 0.2.27
Python: 3.11.1

on Ubuntu 22.04 / Windows 11

### Who can help?

@agola11, @hwchase17

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

### On the server side:
1. `conda create --name openllm python=3.11`
2. `conda activate openllm`
3. `pip install openllm`
4. `openllm start llama --model_id meta-llama/Llama-2-13b-chat-hf`


### On the client side / my local machine:
1. `conda create --name openllm python=3.11`
2. `conda activate openllm`
3. `pip install openllm`
4. Execute the following script:

```python
from langchain.llms import OpenLLM

llm = OpenLLM(server_url='http://<server-ip>:3000')
print(llm("What is the difference between a duck and a goose?"))
```

Then, the following error comes up (similar scripts produce the same error):
```bash
File "C:\Users\<User>\.virtualenvs\<env-name>-wcEN-LyC\Lib\site-packages\langchain\load\serializable.py", line 74, in __init__
    super().__init__(**kwargs)
  File "pydantic\main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Generation
text
  str type expected (type=type_error.str)
```

I could solve the error manually by adapting `langchain/llms/base.py` at line 985 to the following line:
```python
generations.append([Generation(text=text["text"], generation_info=text)])
```

### Expected behavior

I would expect, the provided example script works fine, successfully requests text generation from the deployed server, and returns that text to the user / program.
Model Timeout after 3 requests to endpoint (Amazon Bedrock AI21 models)
Labels: name: auto:bug, name: area: models

### System Info

langchain==0.0.249
macOS Ventura v13.5.1
Python 3.11.0rc2


### Who can help?

@3coins, @hwchase17, @agola11 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. Setup AWS access and import all necessary dependencies 
2. Initialize LLMs
`llm_ai21_j2_mid = Bedrock(
    model_id= "ai21.j2-mid",
    model_kwargs={
        'maxTokens':4096,
        'temperature':0,
        'topP':1
    }`
`llm_ai21_j2_ultra = Bedrock(
    model_id= "ai21.j2-ultra",
    model_kwargs={
        'maxTokens':4096,
        'temperature':0,
        'topP':1
    }`

3. Run inference on `ai21.j2-mid` and `ai21.j2-ultra` models in a loop of 10 for example.

### Expected behavior

One of the models will throw a timeout error.
<img width="963" alt="image" src="https://github.com/langchain-ai/langchain/assets/73419491/e3faabe3-6543-442a-86f5-efbe5ce009d1">