delayedkarma/langchain-issues huggingface dataset:
Stack Exchange API Integration
Labels: name: auto:enhancement

### Feature request

Request to Integrate Stack Exchange API into Langchain for enhanced information access and interaction.

**Relevant links:**
1. **Stack Exchange API Documentation**: The official documentation for the Stack Exchange API, providing detailed information on how to interact with the API and retrieve data.
[Stack Exchange API Documentation](https://api.stackexchange.com/docs)
2. **Stack Exchange API Client GitHub Repository**: Community-supported libraries or tools that interact with the API.
[Stack Exchange API Client GitHub Repository](https://github.com/benatespina/StackExchangeApiClient)

### Motivation

Langchain aims to provide a comprehensive and seamless platform for accessing and interacting with a wide array of linguistic and language-related resources. In this context, integrating the Stack Exchange API would greatly enhance the utility of the platform.

Stack Exchange hosts a plethora of specialized communities, each dedicated to specific domains of knowledge and expertise. These communities generate valuable content in the form of questions, answers, and discussions. By integrating the Stack Exchange API into Langchain, we can enable users to access this wealth of information directly from within the platform.

What we aim to provide is a convenient gateway for Langchain users to explore Stack Exchange communities, search for relevant questions and answers, and actively participate in discussions. This integration would not only simplify the process of finding authoritative information on various topics but also facilitate direct engagement with experts and enthusiasts in their respective fields.

In summary, our Stack Exchange API integration project is designed to enrich the Langchain experience by offering a direct link to the vast knowledge repositories of Stack Exchange communities. It will empower users to seamlessly navigate between these platforms, harnessing the collective wisdom of these communities for their linguistic and language-related endeavors.


### Your contribution

We are 4 University of Toronto students and are very interested in contributing to Langchain for a course project. We will be 
creating a PR that implements this feature sometime in mid November.
Issue with Exception Handling: UnstructuredURLLoader Does Not Raise Exceptions Inheriting from Base Class Exception
Labels: name: area: doc loader, name: auto:bug

### System Info

#### Description:
I encountered an issue with the UnstructuredURLLoader class from the "langchain" library, specifically in the libs/langchain/langchain/document_loaders/url.py module. When trying to handle exceptions for a failed request, I observed that the exception raised by the library doesn't inherit from the base class Exception. This makes it challenging to handle exceptions properly in my code.

#### **Actual Behavior**:
The exception raised by the **`UnstructuredURLLoader`** doesn't inherit from the base class **`Exception`**, causing difficulty in handling exceptions effectively.

#### **Error Message**:
```Error fetching or processing https://en.wikipesdfdia.org/, exception: HTTPSConnectionPool(host='en.wikipesdfdia.org', port=443): Max retries exceeded with url: / (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x000002113AD12F10>: Failed to resolve 'en.wikipesdfdia.org' ([Errno 11001] getaddrinfo failed)"))```

### Who can help?

@eyurtsev @agola11 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [X] Callbacks/Tracing
- [ ] Async

### Reproduction

```py
from langchain.document_loaders import UnstructuredURLLoader
from urllib3 import HTTPSConnectionPool

url = 'https://en.wikipesdfdia.org/'

try:
    unstructured_loader = UnstructuredURLLoader([url])
    all_content = unstructured_loader.load()
    print("Request was successful")
except HTTPSConnectionPool as e:
    # Handle HTTPConnectionPool exceptions
    print(f"An HTTPConnectionPool exception occurred: {e}")
except Exception as e:
    # Handle other exceptions
    print(f"An unexpected error occurred: {e}")
```

### Expected behavior

I expected the exception raised by the **`UnstructuredURLLoader`** to be derived from the base class **`Exception`**, which would allow for proper exception handling.
API Key Leakage in the BaseLanguageModel 
Labels: name: auto:bug, name: area: models

### System Info

Langchain:0.0.319
Python:3.11.2
System: macOS 13.5.2 arm64

### Who can help?

@hwchase17
@agola11

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I implemented a generic module to handle many types of chat models. I executed the following script after I set `OPENAI_API_KEY=xxx` in `.env` file.


```python
from dataclasses import dataclass, field

from langchain.schema.language_model import BaseLanguageModel
from langchain.chat_models import ChatOpenAI
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

@dataclass
class ModelConfig:
    """Model Configuration"""

    model: BaseLanguageModel
    arguments: field(
        default_factory=dict
    )  # This could be any arguments for BaseLanguageModel child class

    def init_model(self) -> BaseLanguageModel:
        return self.model(**self.arguments)

if __name__ == "__main__":
    model = ModelConfig(model=ChatOpenAI, arguments=dict())
    print(model)

```

And then the output contains **a leaked API key**!

```sh
<class 'openai.api_resources.chat_completion.ChatCompletion'> openai_api_key=xxxx>
```





### Expected behavior

My idea off the top of my head is to use, say,  `__str__` directive inside the original class. But this seems just a workaround for ChatOpenAI class. So something more generic feature in `BaseLanguageModel` would be more decent. Like should have a common property in the base class, which handle API token across any LLM models and this property might be a special class that can hide the value when other class and functions call this.

https://github.com/langchain-ai/langchain/blob/08f5e6b8012f5eda2609103f33676199a3781a15/libs/langchain/langchain/chat_models/openai.py#L181-L238

## Similar issue.

#8499
10634: Added the capability to inject boto3 client in SagemakerEndpointEmbeddings
Labels: name: area: embeddings, name: auto:improvement, name: integration: aws

**Description: Allow to inject boto3 client for Cross account access type of scenarios in using SagemakerEndpointEmbeddings and also updated the documentation for same in the sample notebook**
Issue:SSagemakerEndpointEmbeddings cross account capability #10634 #10184
Dependencies: None
Tag maintainer:
Twitter handle:lethargicoder
Retriever with score  #11062
Labels: name: auto:enhancement, name: area: models

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
"GoogleVertexAISearchRetriever" object has no field "_serving_config". It's a Pydantic-related bug.
Labels: name: auto:bug, name: area: models

### System Info

When upgrading from version 0.317 to 0.318, there is a bug regarding the Pydantic model validations. This issue doesn't happen in versions before 0.318.

File "...venv/lib/python3.10/site-packages/langchain/retrievers/google_vertex_ai_search.py", line 268, in __init__
    self._serving_config = self._client.serving_config_path(
  File "pydantic/main.py", line 357, in pydantic.main.BaseModel.__setattr__
ValueError: "GoogleVertexAISearchRetriever" object has no field "_serving_config"

### Who can help?

I tagged both of you, @eyurtsev and @kreneskyp, regarding this version 0.318 change, in case it is related: https://github.com/langchain-ai/langchain/pull/11936.

Thanks for this amazing library!

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Execute the following code:

```
def main():
    aiplatform.init(project=PROJECT_ID)

    llm = VertexAI(model_name=MODEL, temperature=0.0)

    retriever = GoogleVertexAISearchRetriever(
        project_id=PROJECT_ID, search_engine_id=DATA_STORE_ID
    )

    search_query = "Who was the CEO of DeepMind in 2021?"

    retrieval_qa = RetrievalQA.from_chain_type(
        llm=llm, chain_type="stuff", retriever=retriever
    )
    answer = retrieval_qa.run(search_query)

    print(answer)


if __name__ == "__main__":
    main()
```

### Expected behavior

To output the answer from the model, as it does in versions before 0.318
Fix typo in jsonformer_experimental.ipynb
Labels: name: lgtm, name: auto:documentation

HuggingFace -> Hugging Face

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
feature: Reddit API Tool
Labels: name: auto:enhancement

### Feature request

Feature request to Integrate a new Langchain tool that summarizes reddit user interactions/discussion based on user input.

### Motivation

Reddit is a vast platform with numerous discussions and threads covering a wide range of topics. Users often seek information, insights, and opinions on specific subjects of interest within these discussions. However, navigating through these discussions and identifying the most valuable insights can be a time-consuming and sometimes overwhelming process.

What we aim to provide is a summarization toolkit that simplifies the experience of exploring Reddit. With this toolkit, users can input a topic or subject they are interested in, and it will not only aggregate relevant discussions (posts) from Reddit but also generate concise and coherent summaries of these discussions. This tool streamlines the process of distilling key information, popular opinions, and diverse perspectives from Reddit threads, making it easier for users to stay informed and engaged with the platform's wealth of content.

In summary, our Reddit summarization toolkit is designed to save users time and effort by delivering informative and easily digestible summaries of Reddit discussions, ultimately enhancing their experience in accessing and comprehending the vast amount of information and opinions present on the platform.


### Your contribution

We are a group of 4 looking to contribute to Langchain as a course project and will be creating a PR for this issue in mid/late November.
How to run Semi_structured_and_multi_modal_RAG.ipynb
Labels: name: auto:bug, name: area: models

When running on colab, it has following error:
```
[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /root/nltk_data...
[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-2-3092bb1466a5>](https://localhost:8080/#) in <cell line: 7>()
      5 
      6 # Get elements
----> 7 raw_pdf_elements = partition_pdf(filename=path+"LLaVA.pdf",
      8                                  # Using pdf format to find embedded image blocks
      9                                  extract_images_in_pdf=True,

8 frames
[/usr/local/lib/python3.10/dist-packages/unstructured_inference/models/base.py](https://localhost:8080/#) in get_model(model_name, **kwargs)
     70     else:
     71         raise UnknownModelException(f"Unknown model type: {model_name}")
---> 72     model.initialize(**initialize_params)
     73     models[model_name] = model
     74     return model

TypeError: UnstructuredYoloXModel.initialize() got an unexpected keyword argument 'extract_images_in_pdf'
```

https://github.com/langchain-ai/langchain/blob/5dbe456aae755e3190c46316102e772dfcb6e148/cookbook/Semi_structured_and_multi_modal_RAG.ipynb#L103
Chat API Endpoints with RAG 
Labels: name: auto:enhancement, name: area: models

### Feature request

Cohere's upcoming (currently in BETA) Co.Chat + RAG API Endpoint will offer additional parameters such as "documents" and "connectors" that can be used for RAG. This is the major new feature of the Cohere Coral models (RAG)

See https://docs.cohere.com/reference/chat

GPT-4 (and the Other Chat Models following soon after) I am sure will have similar ability to offer RAG with its various new chat models. 

Would the ability to add additional parameters to API chat endpoints (such as a document folder or a url etc...) be able to be supported via the Langchain Runnable Interface once these new Chat endpoints API's are GA? 

This support would be needed for the external RAG for both the Input Type (_PromptValue?_) and the Output Type. (ChatMessage)


### Motivation

The main motivation for this proposal for updating the Input and Output Types in the Langchain Runnable Interface _(if that is indeed the correct place from a code standpoint this would be supported_) is to support RAG inputs and outputs (HumanMessage and AIMessage)  with the newer Chat Models from Cohere, OpenAI etc. and in very near future support multimodal LLMs which are fast becoming the norm.

### Your contribution

I would need a bit of help with the PR.
PyPDFLoader use url in metadata source if file is a web path
Labels: name: lgtm, name: area: doc loader, name: auto:improvement

**Description:** Update `langchain.document_loaders.pdf.PyPDFLoader` to store url in metadata (instead of a temporary file path) if user provides a web path to a pdf

  - **Issue:** Related to #7034; the reporter on that issue submitted a PR updating `PyMuPDFParser` for this behavior, but it has unresolved merge issues as of 20 Oct 2023 #7077
      - In addition to `PyPDFLoader` and `PyMuPDFParser`, these other classes in `langchain.document_loaders.pdf` exhibit similar behavior and could benefit from an update: `PyPDFium2Loader`, `PDFMinerLoader`, `PDFMinerPDFasHTMLLoader`, `PDFPlumberLoader` (I'm happy to contribute to some/all of that, including assisting with `PyMuPDFParser`, if my work is agreeable)
      - The root cause is that the underlying pdf parser classes, e.g. `langchain.document_loaders.parsers.pdf.PyPDFParser`, never receive information about the url; the parsers receive a `langchain.document_loaders.blob_loaders.blob`, which contains the pdf contents and local file path, but not the url
      - This update passes the web path directly to the parser since it's minimally invasive and doesn't require further changes to maintain existing behavior for local files... bigger picture, I'd consider extending `blob` so that extra information like this can be communicated, but that has much bigger implications on the codebase which I think warrants maintainer input

  - **Dependencies:** None

```python
# old behavior
>>> from langchain.document_loaders import PyPDFLoader
>>> loader = PyPDFLoader('https://arxiv.org/pdf/1706.03762.pdf')
>>> docs = loader.load()
>>> docs[0].metadata
{'source': '/var/folders/w2/zx77z1cs01s1thx5dhshkd58h3jtrv/T/tmpfgrorsi5/tmp.pdf', 'page': 0}

# new behavior
>>> from langchain.document_loaders import PyPDFLoader
>>> loader = PyPDFLoader('https://arxiv.org/pdf/1706.03762.pdf')
>>> docs = loader.load()
>>> docs[0].metadata
{'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 0}
```
Integrate MemGPT
Labels: name: area: memory, name: auto:enhancement, name: area: models

### Feature request

Implement MemGPT for better memory management with local and hosted models:
https://github.com/cpacker/MemGPT


### Motivation

This can help automate the manually creation and retrieval of vector stores.

### Your contribution

I'm not really familiar with the langchain repo at the moment, I could provide feedback on the implementation from a users perspective.
Improve flexibility of ReAct agent on the first iteration
Labels: name: area: agent, name: auto:improvement

### Feature request

Many of the non-OpenAI LLMs struggle to complete the first "Thought" part of the prompt. Falcon, Llama, Mistral, Zephyr, and many others will successfully choose the action and action input, but leave out an initial thought.

### Motivation

For the very first interaction, the agent scratchpad is empty, and the first thought isn't always necessarily helpful, especially if the first action and action input are valid.

Interestingly, removing the "thought" portion of the prompt often results in the LLM properly completing the thought. However, doing so breaks Langchain because Langchain is expecting the agent_scratchpad variable to be present in the template for completion.

An alternative could be to make the entire "Thought" part of the prompt only present if agent_scratchpad isn't empty. However, that would require "building" the prompt template conditionally.

### Your contribution

Most of this takes place in either the MRKL agent:
https://github.com/langchain-ai/langchain/tree/master/libs/langchain/langchain/agents/mrkl

Or potentially the ReAct Docstore agent:
https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/agents/react/base.py
Add multiple models and streaming support to OpenAICallbackHandler
Labels: name: auto:improvement, name: area: models

### Description

Old OpenAICalbackHandler does not support streaming mode, which is used widely today. Also, it is hard to differentiate between models when using the same callback with different models. It is particularly useful when used in agents combinig "smart" and "fast" models. This PR is an attempt to solve these problems.

If tiktoken is not installed, my solution will work as it previously did, except that it will show a warning if used in conjunction with streaming model.

Additionally it adds per-model methods to get prompt, completion and total tokens, as well as total cost.
If a user will try to use total_tokens, prompt_tokens, completion_tokens after using the callback with multiple models, a warning will be displayed.

### Issue
#3114 #4583

### Dependencies
N/A

### Tag maintainer
Not yet :)

### Twitter handle
N/A
fix: Add `_serving_config` property to `GoogleVertexAISearchRetriever`
Labels: name: auto:bug

- Fixes error:

```
ValueError: "GoogleVertexAISearchRetriever" object has no field "_serving_config"
```

Introduced in #11736

@baskaryan, @eyurtsev, @hwchase17 if you could review and merge quickly, that would be appreciated :)
Allow specifying custom input/output schemas for runnables with .with_types()
Labels: name: auto:improvement



<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->