delayedkarma/langchain-issues huggingface dataset:
How to show images from PDFs, PPT, DOCs documents as part of answer?
Labels: name: area: doc loader, name: auto:question, name: auto:improvement

### Issue you'd like to raise.

Hello, I have this use-case where there are different types of documents. I can parse documents using document loaders using langchain. But, there are images also in these documents. I want to store them as metadata and if answer generated from a context chunk it show the image also. Please help.

### Suggestion:

_No response_
AttributeError: partially initialized module 'arxiv' has no attribute 'Search' (most likely due to a circular import)
Labels: name: area: doc loader, name: area: vector store, name: auto:bug

### System Info

version:
- langchain-0.0.320
- py311

### Who can help?

@eyurtsev 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Running the [example code](https://python.langchain.com/docs/integrations/retrievers/arxiv)

```python
from langchain.retrievers import ArxivRetriever

retriever = ArxivRetriever(load_max_docs=2)

docs = retriever.get_relevant_documents(query="1605.08386")
```

The above code gives the error: 

`AttributeError: partially initialized module 'arxiv' has no attribute 'Search' (most likely due to a circular import)`


### Expected behavior


Should print out 
```
    {'Published': '2016-05-26',
     'Title': 'Heat-bath random walks with Markov bases',
     'Authors': 'Caprice Stanley, Tobias Windisch',
     'Summary': 'Graphs on lattice points are studied whose edges come from a finite set of\nallowed moves of arbitrary length. We show that the diameter of these graphs on\nfibers of a fixed integer matrix can be bounded from above by a constant. We\nthen study the mixing behaviour of heat-bath random walks on these graphs. We\nalso state explicit conditions on the set of moves so that the heat-bath random\nwalk, a generalization of the Glauber dynamics, is an expander in fixed\ndimension.'}
```
latest version langchain locks peer dependency: "googleapis" version to 126.0.1 
Labels: name: auto:bug

### System Info

latest version langchain locks peer dependency: "googleapis" version to 126.0.1, wondering why it has to be googleapis v126.0.1. it introduces tech debts.

Please fix the peer dep version for `googleapis` if there is no specific reasons.

### Who can help?

@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

when try to install both latest version langchain and googleapis latest version in nodejs via npm.

### Expected behavior

langchain shouldn't prevent latest googleapis to be installed.
updated API Reference `langchain.agents` functions table
Labels: name: area: doc loader, name: auto:documentation

The `agents.agent_toolkits.conversational_retrieval.openai_functions.create_conversational_retrieval_agent` is a too long name.
It makes the ['agents/functions' table](https://api.python.langchain.com/en/latest/api_reference.html#id2) inside API Reference unreadable.
I've hidden this function from the API Reference.
Note: This is a temporary solution! There should be more elegant solution.
DOC: Example from Custom Tool documentation fails
Labels: name: area: doc loader, name: auto:bug, name: auto:documentation

### Issue with current documentation:

On this page

This example:
```
class SearchInput(BaseModel):
    query: str = Field(description="should be a search query")


@tool("search", return_direct=True, args_schema=SearchInput)
def search_api(query: str) -> str:
    """Searches the API for the query."""
    return "Results"

search_api
```

Fails with:
ValidationError: 1 validation error for StructuredTool
args_schema
  subclass of BaseModel expected (type=type_error.subclass; expected_class=BaseModel)

### Idea or request for content:

nothing missing, but example fails
Name: langchain
Version: 0.0.311
Fix bug where types were lost when calling with_cconfig or bind
Labels: name: auto:bug

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Add how-to guide on runnable generators
Labels: name: auto:documentation

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Update default recursion_limit, update docs
Labels: name: auto:improvement

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Fix type hint for older py versions
Labels: name: auto:nit, name: area: models

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Add optional config arg to RunnablePassthrough func arg
Labels: name: auto:improvement

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
DOC: Issue Connecting to Docusaurus in Local Container Environment
Labels: name: area: doc loader, name: auto:improvement, name: auto:documentation

### Issue with current documentation:

When starting Docusaurus in a local container environment using the `make docs_build` command, it runs at `localhost:3000`, but a connection from the host machine is not possible.
The container has both `poetry` and `yarn` installed.
Even though port 3000 is forwarded to the host, the connection is not established.

https://github.com/langchain-ai/langchain/blob/master/docs/package.json#L7

I am proficient with Python but struggle with React, which might be influencing my troubleshooting.
Thank you!!

### Idea or request for content:

By adding the `--host 0.0.0.0` option to the `start` command in `package.json`, the issue might be resolved.

```diff
-"start": "rm -rf ./docs/api && docusaurus start",
+"start": "rm -rf ./docs/api && docusaurus start --host 0.0.0.0",
```
- With this change, it would be possible to access the port that's being forwarded from within the container to the host environment.
- However, using the `--host 0.0.0.0` option can have security implications, so caution is advised especially in production environments.
Fixed hardcoded "vector" and replaced with vector_query_field variable
Labels: name: area: vector store, name: auto:bug, name: integration: elasticsearch

  - **Description:** In the max_marginal_relevance_search function of the ElasticsearchStore vector store, the name of the field corresponding to the vector embedding of the document is hard coded in the delete statement that drops the field from the document metadata. This results in an exception if the vector embedding field is customized. This PR changes the hard-coded "vector" into the vector_query_field variable.
  - **Issue:** None
  - **Dependencies:** None
  - **Tag maintainer:** @hwchase17
Issue: `VectorDBQA` is a better chain than `RetrievalQA` when it comes to streaming model response
Labels: name: auto:bug, name: auto:question, name: area: models, name: integration: chroma

### Issue you'd like to raise.

I found that using a `RetrievalQA` for streaming outputs gibberish response. For example, using a `RetrievalQA` with code below on the `state_of_the_union.txt` example:
```
doc_chain = load_qa_chain(
    llm=ChatOpenAI(
        streaming=True,
        openai_api_key=api_key,
        callbacks=[StreamingStdOutCallbackHandler()]
    ),
    chain_type="map_reduce",
    verbose=False,
)

retrieval_qa = RetrievalQA(
    combine_documents_chain=doc_chain,
    retriever=retriever,
)
``` 

With this call: `retrieval_qa.run("What did the president say about Ketanji Brown Jackson")` outputs this streamed response:
```
"And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.""And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.""And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence."The given portion of the document does not mention anything about what the president said about Ketanji Brown Jackson.The given portion of the document does not provide any information about what the president said about Ketanji Brown Jackson.
```

While using `VectorDBChain`:
```
qa = VectorDBQA.from_chain_type(
    llm=ChatOpenAI(
        streaming=True,
        openai_api_key=api_key,
        callbacks=[StreamingStdOutCallbackHandler()]
    ), 
    chain_type="map_reduce", 
    vectorstore=db
)
```

Outputs this:
```
The President said that he nominated Circuit Court of Appeals Judge Ketanji Brown Jackson, who is one of our nation's top legal minds and will continue Justice Breyer's legacy of excellence.
```

Is there any reason as to why the `VectorDBQA` chains have been deprecated in favour of the `RetrievalQA` chains? At first I thought the gibbering streaming was related to using PGVector, but I even tried it with Chroma and am having the same issue.

### Suggestion:

_No response_
a bug fix for sagemakerendpoint streaming
Labels: name: auto:bug, name: area: models, name: integration: aws

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a bug fix for sagemaker endpoint streaming, 
  - **Issue:** the issue # it fixed _call function so that it can now stream from TGI container as well and added _acall stream function for both LMI or TGI containers,
  - **Dependencies:** no dependencies,
  - **Tag maintainer:** @3coins ,
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Feature: Steam game recommendation tool
Labels: name: auto:enhancement, name: area: models

### Feature request

Feature request to Integrate a new Langchain tool that make recommandtions on steam games based on user's Steam ID and provide game information based on given Steam game name.

### Motivation

We recognize the current challenges users face when discovering games on Steam—cluttered search results and lack of tailored information. To revolutionize this experience, we propose integrating Langchain, offering personalized game recommendations based on a user's Steam ID and comprehensive game details based on specific titles.

Langchain's algorithm will provide personalized game suggestions aligned with users' preferences, eliminating irrelevant options. It'll also furnish detailed game insights—prices, discounts, popularity, and latest news—for informed decision-making.

This integration aims to streamline game discovery, enhance user satisfaction, and foster a vibrant gaming community on Steam. We're eager to discuss this enhancement and its potential to transform the Steam experience.


### Your contribution

We are a group of 4 looking to contribute to Langchain as a course project and will be creating a PR for this issue in mid November.
ConversationalRetrievalChain cannot return source documents when using ConversationBufferWindowMemory
Labels: name: area: memory, name: auto:bug, name: area: models, name: integration: chroma

### System Info

Langchain 0.0.285, python 3.11.5, Windows 11

### Who can help?

@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [X] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

``` python
from langchain.embeddings import HuggingFaceInstructEmbeddings
from langchain.vectorstores import Chroma
from langchain.text_splitter import CharacterTextSplitter
from langchain.llms import OpenAI
from langchain.chains import ConversationalRetrievalChain
from langchain.document_loaders import TextLoader
from langchain.memory import ConversationBufferMemory
from mysecrets import secrets


loader = TextLoader("./sotu.txt")
documents = loader.load()

text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
documents = text_splitter.split_documents(documents)
model_name = "hkunlp/instructor-large"
model_kwargs = {'device': 'cpu'}
encode_kwargs = {'normalize_embeddings': True}
embed_instruction = "Represent the document for summary"

embedding_function =  HuggingFaceInstructEmbeddings(
    model_name=model_name,
    model_kwargs=model_kwargs,
    encode_kwargs=encode_kwargs,
    embed_instruction = embed_instruction
)

vectorstore = Chroma.from_documents(documents, embedding_function)


memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
llm = OpenAI(openai_api_key=secrets["OPENAI-APIKEY"],temperature=0)
qa = ConversationalRetrievalChain.from_llm(llm=llm, retriever=vectorstore.as_retriever(), memory=memory, return_source_documents=True)

query = "What did the president say about Ketanji Brown Jackson"
result = qa({"question": query})
```

### Expected behavior

When `return_source_documents` is set to False, code runs as intended. When ConversationalRetrievalChain uses the default value for `memory`, code runs as intended. If ConversationalRetrievalChain is used with memory and source documents are to be returned, the code fails since chat_memory.py (https://github.com/langchain-ai/langchain/blame/master/libs/langchain/langchain/memory/chat_memory.py) is expecting only one key.
Architecture: Decouple Configuration from Inheritance for better extensibility
Labels: name: area: agent, name: auto:improvement, name: auto:refactor

### Feature request

There seems to be a conflation between configuration code and inheritance trees that make it nearly impossible to extend langchain functionality without recreating entire class inheritance. I ran into this almost immediately when trying to customize agents and executors to my application needs, but also take advantage of the pre-assembled styles (e.g. Zero-Shot react). I had to recreate my style of zero shot executor and agent by copying the parameters from the subclasses into my forked instance of the bases.

My proposal is to decouple these two concepts and have a single inheritance tree for functionality (e.g. agents, chains) and a separate notion of a “config” class that can hold the parameterization for an object. The factory methods on the core classes can then accept a config parameter to create an instance with the desired parameterization. This enables extension in two dimensions - any core object (agent, chain, etc) x any configuration. Furthermore new configurations can easily be created by end users or even extensions of existing ones (maybe you want to swap out one parameter from a predefined config without copy pasting the whole code).

### Motivation

Architectural improvement and future extensibility of the framework

### Your contribution

PR forthcoming with a POC when I have some time to abstract what I did for my projects into the langchain source but wanted to seed this for discussion!
SendGrid API Integration For Enhanced Email Communication
Labels: name: auto:enhancement

### Feature request

Feature request to integrate the SendGrid API into Langchain for enhanced email communication and notifications.

**Relevant links:**
1. [SendGrid API Documentation](https://sendgrid.com/docs/API_Reference/index.html): The official documentation for the SendGrid API, offering comprehensive information on how to use the API for sending emails and managing email communication.
2. [SendGrid GitHub Repository](https://github.com/sendgrid/sendgrid-python): The GitHub repository for SendGrid's official Python library, which can be utilized for API integration.

### Motivation

Langchain is committed to offering a seamless platform for language and communication-related tools. By integrating the SendGrid API, we aim to bring advanced email communication capabilities to the platform, enhancing its utility and user experience.

SendGrid is a renowned email platform that provides a reliable and efficient way to send, receive, and manage emails. By incorporating the SendGrid API into Langchain, we enable users to send emails, manage notifications, and enhance their communication within the platform.

Our goal is to provide Langchain users with a feature-rich email system that allows them to send notifications, updates, and alerts directly from Langchain. This integration will streamline communication, offering users a seamless experience and the ability to keep stakeholders informed and engaged through email.

In summary, the SendGrid API integration project is designed to extend Langchain's capabilities by incorporating a powerful email communication tool. This will help users effectively manage email notifications and stay connected with stakeholders within the platform.

### Your contribution

We are 4 University of Toronto students and are very interested in contributing to Langchain for a course project. We will be
creating a PR that implements this feature sometime in mid November.