delayedkarma/langchain-issues huggingface dataset:
Issue: <Not able to filter OpenSearch vectorstore  using Filter in search_kwargs>
Labels: name: area: vector store, name: auto:question

### Issue you'd like to raise.

Hello Team,
We have a metadata field in our opensearch index which I would like use to filter the passages before sending it to LLM to generate the answer. Tried many options suggested in chat.langchain.com but none of them seems to work.

Here is an example:
I have an user with associated groups ['user_group1', 'user_group2'] and the OpenSearch index with data like 
{
'text': 'Page Content......',
'metadata': {
     'source_acl': [ {'group_name': 'user_group1',  'group_type': 'type1'}, {}, {}] 
}
}

So far I have been trying this

```
chain = ConversationalRetrievalChain.from_llm(
                llm=SagemakerEndpoint(
                    endpoint_name=constants.EMBEDDING_ENDPOINT_ARN_ALPHA,
                    region_name=constants.REGION,
                    content_handler=self.content_handler_llm,
                    client=self.sg_client
                ),
                # for experimentation, you can change number of documents to retrieve here
                retriever=opensearch_vectorstore.as_retriever(search_kwargs={'k': 3, 'filter': 'source_acl.group_name:user_group1'}),
                memory=memory,
                condense_question_prompt=custom_question_prompt,
                return_source_documents=True
            )
```

Am I missing anything here?

Thanks in Advance.

### Suggestion:

_No response_
Integrate RetryOutputParser with ConversationalRetrievalChain
Labels: name: auto:enhancement, name: area: models

### Feature request

Is there a way to integrate `RetryOutputParser` with `ConversationalRetrievalChain`? Specifically, if I use OpenAI's function calling to generate the response in a specific format, the LLM model does not always follow the instructions and then it would fail the Pydantic model validation.

Therefore, it's important to build the retry mechanism for the `LLMChain` (aka, `doc_chain` argument).

### Motivation

Specifically, if I use OpenAI's function calling to generate the response in a specific format, the LLM model does not always follow the instructions and then it would fail the Pydantic model validation.

Therefore, it's important to build the retry mechanism for the `LLMChain` (aka, `doc_chain` argument).

### Your contribution

TBD
`poetry install --with test`: Group(s) not found: test (via --with)

### System Info

I am trying to setup the local development environment with `poetry install --with test` but I am running into the error `Group(s) not found: test (via --with)`

```
> origin/master:master via 🐍 v3.11.6
> 57% langchain ❯ poetry install --with test

Group(s) not found: test (via --with)
```

Could you advise on what might be happening here?

### Who can help?

@eyurtsev, I think you might be able to advise here? 🙏🏼 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

- Clone repository
- With poetry installed, run `poetry install --with test`

### Expected behavior

Local development setup works without an error.
feat: document_loader Textract linearized output
Labels: name: area: doc loader, name: auto:enhancement, name: integration: aws

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** linearizing the output from Amazon Textract into a layout with tables and key/value pairs when those features are passed in as feature parameters 
  - **Issue:** fixes #12068
  - **Dependencies:** adding amazon-textract-textractor as dependency
  - **Tag maintainer:** @3coins  
  - **Twitter handle:**
feat: Add Client Info to available Google Cloud Clients
Labels: name: area: doc loader, name: auto:improvement

- This is used internally to gather aggregate usage metrics for the LangChain integrations

- Note: This cannot be added to some of the Vertex AI integrations at this time because the SDK doesn't allow overriding the [`ClientInfo`](https://googleapis.dev/python/google-api-core/latest/client_info.html#module-google.api_core.client_info)

- Added to:
  - BigQuery
  - Google Cloud Storage
  - Document AI
  - Vertex AI Model Garden
  - Document AI Warehouse
  - Vertex AI Search
  - Vertex AI Matching Engine (Cloud Storage Client)
 
@baskaryan, @eyurtsev, @hwchase17
Issue: View context used in query in ConversationalRetrievalChain
Labels: name: area: memory, name: auto:question, name: area: models

### Issue you'd like to raise.

How can I see the context that was used to answer a specific question? 


from langchain.memory import ConversationBufferWindowMemory
memory = ConversationBufferWindowMemory(k=3, memory_key="chat_history", return_messages=True)

qa = ConversationalRetrievalChain.from_llm(llm_model, vector_store.as_retriever(search_type='similarity', search_kwargs={'k': 2}), memory=memory)
query = "Can I as a manager fly business class?"

##predict with the model
text = f"""
 
 You are a HR-assistant that answers questions:

 
{query} 

"""
result = qa({"question": text})
result

### Suggestion:

_No response_
mypy linting error in PyPDFParser with pypdf 3.16.4
Labels: name: area: doc loader, name: auto:bug

### System Info

make lint errors

```
make lint
./scripts/check_pydantic.sh .
./scripts/check_imports.sh
poetry run ruff .
[ "." = "" ] || poetry run black . --check
All done! ✨ 🍰 ✨
1866 files would be left unchanged.
[ "." = "" ] || poetry run mypy .
langchain/document_loaders/parsers/pdf.py:95: error: "PdfObject" has no attribute "keys"  [attr-defined]
langchain/document_loaders/parsers/pdf.py:98: error: Value of type "PdfObject" is not indexable  [index]
Found 2 errors in 1 file (checked 1863 source files)
make: *** [lint] Error 1```

for line 95:
```
        if not self.extract_images or "/XObject" not in page["/Resources"].keys():
```

and line 98:
```
        xObject = page["/Resources"]["/XObject"].get_object()
```

on master 7c4f340cc, v0.320

commented with # type: ignore
 for my PR, but keeping here as reference to fix


### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```make lint```



### Expected behavior

no mypy error
OpenAPI Spec - reduce_openapi_spec - maximum recursion depth exceeded
Labels: name: area: agent, name: auto:bug, name: auto:question

### Issue you'd like to raise.

Hello team!

I am new to the OpenAPI Specs Toolkit. I am trying to build an OpenAPI Agent Planner for the Microsoft Graph API.  The OpenAPI specs can be found here: https://github.com/microsoftgraph/msgraph-metadata/blob/master/openapi/v1.0/openapi.yaml . I downloaded the YAML file and followed this notebook: https://python.langchain.com/docs/integrations/toolkits/openapi#1st-example-hierarchical-planning-agent .

```python
import os
import yaml
from langchain.agents.agent_toolkits.openapi.spec import reduce_openapi_spec

current_directory = os.path.abspath('')
data_directory = os.path.join(current_directory, "data")
msgraph_api_file = os.path.join(data_directory, "msgraph-openapi.yaml")

raw_msgraph_api_spec = yaml.load(open(msgraph_api_file,encoding='utf-8').read(), Loader=yaml.Loader)
msgraph_api_spec = reduce_openapi_spec(raw_msgraph_api_spec)
```

Does anyone know how to handle large OpenAPI Specs? I ran the following to read my OpenAPI YAML spec and when using the `reduce_openapi_spec` module, I get the following error below:

`RecursionError: maximum recursion depth exceeded while calling a Python object` 

Is there a setting I need to change in LangChain? Please and thank you for your help in advance. I believe this issue is before I run into the following token limit, right? https://github.com/langchain-ai/langchain/issues/2786

### Suggestion:

_No response_
JsonSpec `keys` method doesn't handle indexing lists properly
Labels: name: auto:bug

There's an issue when using JsonSpec that it can't handle the case where it indexes a list with `0` because in [line 52](https://github.com/langchain-ai/langchain/blob/d0505c0d4755d8cbe62a8ddee68f53a5cb330892/libs/langchain/langchain/tools/json/tool.py#L52) it checks `if i` and since `i==0` it returns `False`. 

Instead, I believe this should check for purely existence of `i` as a string or int.
Expose configuration options in GraphCypherQAChain
Labels: name: auto:improvement, name: area: models

Allows for passing arguments into the LLM chains used by the GraphCypherQAChain. This is to address a request by a user to include memory in the Cypher creating chain. Will keep the prompt variables as-is to be backward compatible. But, would be a good idea to deprecate them and use the **kwargs variables. Added a test case. 

In general, I think it would be good for any chain to automatically pass in a readonlymemory(of its input) to its subchains whilist allowing for an override. But, this would be a different change.
StructuredChatOutputParser regex not accounting for space
Labels: name: area: agent, name: auto:bug, name: area: models

### System Info

Langchain version: 0.0.320
Python 3.9.6

Issue:
The LLM output ` ```json  \n{\n  \"action\": \"...``` ` for a StructuredChatAgent using StructuredChatOutputParser.
The regex in the parser `re.compile(r"```(?:json)?\n(.*?)```", re.DOTALL)` does not allow for a space after `json`.

LLM: Claude v2

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

create an agent using StructuredChatAgent and a tool that is called by the agent.
If the output from the LLM contains a space after `json`, it will call `AgentFinish`

### Expected behavior

The expected behaviour is that the output should match the regex.
Langchain crashes when retrieving results from vertexai codey models
Labels: name: auto:bug, name: area: models

### System Info

Hey folks, I think i stumbled on a bug (or i'm using langchain wrong)

Langchain version : 0.0.320
Platform Ubuntu 23.04
Python: 3.11.4


### Who can help?
@hwchase17 , @agola11 

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Run the following:
`
llm = llms.VertexAI(model_name="code-bison@001", max_output_tokens=1000, temperature=0.0)
prediction = llm.predict(""" write a fibonacci sequence in python""")

from pprint import pprint
pprint(prediction)`


### Expected behavior

We get a prediction


(adding more info since the form has ran out of fields)

I think the bug is in [llms/vertexai.py:301](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/llms/vertexai.py#L301C23-L301C23) . Variable res is a TextGenerationResponse as opposed to MultiCandidateTextGenerationResponse

Hence there are no "candidates" as you would expect from a chat model.

This happens because:  Google's sdk (vertexai/language_models_language_models.py)
method:
>_ChatSessionBase. _parse_chat_prediction_response

returns a `MultiCandidateTextGenerationResponse`
but both  `CodeChatSession`  and `CodeGenerationModel` return a `TextGenerationResponse`


I think the fix might be replacing 

`generations.append([_response_to_generation(r) for r in res.candidates])`
with
```

                if self.is_codey_model:
                    generations.append([_response_to_generation(res)])
                else:
                    generations.append([_response_to_generation(r) for r in res.candidates])
```

happy to send a pr if I helps
MatchingEngine.add_texts() doesn't work
Labels: name: area: vector store, name: auto:bug

### System Info

`langchain==0.0.320`

### Who can help?

_No response_

### Information

- [x] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```py
from langchain.vectorstores import MatchingEngine

texts = [
    "The cat sat on",
    "the mat.",
    "I like to",
    "eat pizza for",
    "dinner.",
    "The sun sets",
    "in the west.",
]


vector_store = MatchingEngine.from_components(
    texts=texts,
    project_id="<my_project_id>",
    region="<my_region>",
    gcs_bucket_uri="<my_gcs_bucket>",
    index_id="<my_matching_engine_index_id>",
    endpoint_id="<my_matching_engine_endpoint_id>",
)

vector_store.add_texts(texts=texts)
```

### Expected behavior

I think line 136 in matching_engine.py should be
```py
jsons.append(json_)
```
Dev qf

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
CharacterTextSplitter return incorrect chunks
Labels: name: area: doc loader, name: auto:bug, name: area: models

### System Info

`langchain==0.0.320`

Example file: https://drive.google.com/file/d/1zDj3VXohUO7x4udu9RY9KxWbzBUIrsvb/view?usp=sharing

```
from PyPDF2 import PdfReader
from langchain.text_splitter import CharacterTextSplitter

# Specify the PDF file to be processed
pdf = "Deep Learning.pdf"

# Initialize a PdfReader object to read the PDF file
pdf_reader = PdfReader(pdf)

# Initialize an empty string to store the extracted text from the PDF
text = ""

for i, page in enumerate(pdf_reader.pages):
    text += f" ### Page {i}:\n\n" + page.extract_text()

assert text.count(" ### ") == 100

text_splitter = CharacterTextSplitter(
    separator=" ### ",
)
chunks = text_splitter.split_text(text)
len(chunks) ## = 76
```

Length chunks = 76 while there are 100 counts of the separator ` ### `. All the splits by ` ### ` look decent, so not sure why they are being merged strangely. Please help advice! Thanks!

Thank you so much!

### Who can help?

@hwchase17

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
from PyPDF2 import PdfReader
from langchain.text_splitter import CharacterTextSplitter

# Specify the PDF file to be processed
pdf = "Deep Learning.pdf"

# Initialize a PdfReader object to read the PDF file
pdf_reader = PdfReader(pdf)

# Initialize an empty string to store the extracted text from the PDF
text = ""

for i, page in enumerate(pdf_reader.pages):
    text += f" ### Page {i}:\n\n" + page.extract_text()

assert text.count(" ### ") == 100

text_splitter = CharacterTextSplitter(
    separator=" ### ",
)
chunks = text_splitter.split_text(text)
len(chunks) ## = 76
```

### Expected behavior

`len(chunks) == 100`
LCEL not working with MultiPromptChain
Labels: name: auto:bug, name: area: models, name: area: lcel

### System Info

python == 3.10.12
langchain == 0.0.312

### Who can help?

@hwchase17 @agola11

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```open_llm = ChatOpenAI(temperature=0, openai_api_key=OPENAI_API_KEY, model_name='gpt-3.5-turbo-16k', max_tokens=3000)

# test chain (LCEL)
human_message = HumanMessagePromptTemplate.from_template('test prompt: {input}')
system_message = SystemMessagePromptTemplate.from_template('system')
all_messages = ChatPromptTemplate.from_messages([system_message, human_message])
test_chain = all_messages | open_llm | StrOutputParser()

# LLMChain
class MyLLMChain(LLMChain):
    output_key: str = "output"  #: :meta private

prompt_infos  = [
    {
        "name": "mastermind", 
        "description": "for general questions.",
        "chain": MyLLMChain(llm=openai_chatbot, prompt=PromptTemplate(template='''{input}''', 
                                                input_variables=["input"]
                                               )) 
    },
    {
        "name": "test", 
        "description": "for testing",
        "chain": test_chain
    }]

destinations = [f"{p['name']}: {p['description']}" for p in prompt_infos]
destinations_str = "\n".join(destinations)
router_template = MULTI_PROMPT_ROUTER_TEMPLATE.format(destinations=destinations_str)

router_prompt = PromptTemplate(
    template=router_template,
    input_variables=["input"],
    # output_parser=OpenAIFunctionsAgentOutputParser()
    # output_parser=RouterOutputParser(),
    output_parser=OutputFixingParser.from_llm(parser=RouterOutputParser(), llm=openai_chatbot),
)
router_chain = LLMRouterChain.from_llm(openai_chatbot, router_prompt)

destination_chains = {}
for p_info in prompt_infos:
    name = p_info["name"]
    chain = p_info["chain"]
    destination_chains[name] = chain
    
default_chain = ConversationChain(llm=openai_chatbot, output_key="output") 


agent_yan_chain = MultiPromptChain(
                        router_chain=router_chain,
                        destination_chains=destination_chains,
                        default_chain=default_chain,
                        verbose=True
                    )
```

Error message:
```
ValidationError: 1 validation error for MultiPromptChain
destination_chains -> test
  Can't instantiate abstract class Chain with abstract methods _call, input_keys, output_keys (type=type_error)
```

### Expected behavior

LCEL should work with MultiPromptChain
Langchain Document schema explodes with FastAPI
Labels: name: area: vector store, name: auto:bug, name: area: models

### System Info

Python 3.11, langchain 0.0.315 and mac

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Create a FastAPI endpoint/router


```python
from fastapi import APIRouter, Response
from pydantic import BaseModel
from typing import Optional
from langchain.schema import Document

router = APIRouter()



class QueryOut(BaseModel):
    answer: str
    sources: list[Document]


@router.post("/query")
async def get_answer():
    sources = [Document(page_content="foo", metadata={"baa": "baaaa"})]
    return QueryOut(answer="answer", sources=sources)

```

Then call it and boom 

```bash
 File "/Users/FRANCESCO.ZUPPICHINI/Documents/dbi_alphaai_knowledge_bot/.venv/lib/python3.11/site-packages/pydantic/main.py", line 164, in __init__
    __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__)
TypeError: BaseModel.validate() takes 2 positional arguments but 3 were given
```

Is this linked to the fact that langchain is still using pydantic v1? If so, I love you guys but put that 30M to good use 😄 

### Expected behavior

Should not explode
10634: Added the capability to inject boto3 client in SagemakerEndpointEmbeddings
Labels: name: area: embeddings, name: auto:improvement, name: integration: aws

**Description: Allow to inject boto3 client for Cross account access type of scenarios in using SagemakerEndpointEmbeddings and also updated the documentation for same in the sample notebook**

**Issue:SagemakerEndpointEmbeddings cross account capability #10634 #10184**

Dependencies: None
Tag maintainer:
Twitter handle:lethargicoder
Fix Typo in CONTRIBUTING.md file
Labels: name: lgtm, name: auto:documentation

Fix Type & add suitable pronoun in CONTRIBUTING.md file

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->