delayedkarma/langchain-issues huggingface dataset:
Add OpenAI Whisper Audio Loader
Labels: name: area: doc loader, name: auto:enhancement

Introducing an ability to load an audi transcription document via OpenAI Whisper.
<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
How to use langchain to create my own databricks dolly chat robot
Labels: name: area: embeddings, name: area: vector store, name: auto:question

I have a operation manual pdf file about a website and i want to use langchain let dolly can responed the question about the website
below is my code:
```
from langchain.embeddings import HuggingFaceEmbeddings
from PyPDF2 import PdfReader
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import ElasticVectorSearch, Pinecone, Weaviate, FAISS
from langchain.chains.question_answering import load_qa_chain
from langchain.llms import HuggingFacePipeline
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
from langchain.prompts import PromptTemplate
import torch

hf_embed = HuggingFaceEmbeddings(model_name="sentence-transformers/all-mpnet-base-v2")
from google.colab import drive
drive.mount('/content/gdrive', force_remount=True)
root_dir = "/content/gdrive/My Drive/"
reader = PdfReader('/content/gdrive/My Drive/data/operation Manual.pdf')
raw_text = ''
for i, page in enumerate(reader.pages):
    text = page.extract_text()
    if text:
        raw_text += text
text_splitter = CharacterTextSplitter(        
    separator = "\n",
    chunk_size = 1000,
    chunk_overlap  = 200,
    length_function = len,
)
texts = text_splitter.split_text(raw_text)
docsearch = FAISS.from_texts(texts, hf_embed)
model_name = "databricks/dolly-v2-3b"
instruct_pipeline = pipeline(model=model_name, torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto", 
                               return_full_text=True, max_new_tokens=256, top_p=0.95, top_k=50)
hf_pipe = HuggingFacePipeline(pipeline=instruct_pipeline)
prompt_template = """Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

{context}

Question: {question}
"""
PROMPT = PromptTemplate(
    template=prompt_template, input_variables=["context", "question"]
)
query = "I forgot my login password."          
docs = docsearch.similarity_search(query)
chain = load_qa_chain(llm = hf_pipe, chain_type="stuff", prompt=PROMPT)
chain({"input_documents": docs, "question": query}, return_only_outputs=True)
```
I am refer to the article in the databricks website [https://www.databricks.com/resources/demos/tutorials/data-science-and-ai/build-your-chat-bot-with-dolly?itm_data=demo_center]

the output is not the answer that in the operation Manual and it have to spend much time (about 2.5 hours) to output answer. where did i do wrong?
Async Recursive URL loader
Labels: name: area: doc loader, name: auto:improvement

Description: This PR improves the function of recursive_url_loader, such as limiting the depth of the access, and customizable extractors(from the raw webpage to the text of the Document object), so that users can use other tools to extract the webpage. This PR also includes the document and test for the new loader.
Old PR closed due to project structure change. #7756

Because socket requests are not allowed, the old unit test was removed.
Issue: N/A
Dependencies: asyncio, aiohttp
Tag maintainer: @rlancemartin
Twitter handle: @ Zend_Nihility
Fixed typo: papaer -> paper
Labels: name: lgtm, name: auto:documentation

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
API Key Leakage in the PromptLayerChatOpenAI Model
Labels: name: area: embeddings, name: auto:bug

### System Info

**Langchain**:0.0.247
**Python**:3.10.5
**System**: macOS 13.4 arm64

### Who can help?

@hwchase17

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

There is the minimal produce demo 
```python
from langchain.chat_models import PromptLayerChatOpenAI
from dotenv import load_dotenv

load_dotenv()
import promptlayer
from langchain.schema import HumanMessage

promptlayer.api_key = "xxxxxxxxxxx"
llm = PromptLayerChatOpenAI(model="gpt-3.5-turbo", verbose=True, pl_tags=["test"])

print(llm([HumanMessage(content="I am a cat and I want")]))
```
I found it on the [**PromptLayer**](https://promptlayer.com/)
<img width="978" alt="image" src="https://github.com/langchain-ai/langchain/assets/3949397/770f6c3f-ec8a-42df-967d-8e207b3208d4">


### Expected behavior

Why is the API key carried in this PromptLayer? This is insecure, it should be removed. Then, I discovered the root of the problem
The paras are generated from this function (**_generate** and  **_agenerate**).
And I got it inherit from **ChatOpenAI** model function __create_message_dicts_ and __client_params_, then I found the api_key
https://github.com/langchain-ai/langchain/blob/08f5e6b8012f5eda2609103f33676199a3781a15/libs/langchain/langchain/chat_models/openai.py#L487
but the api_key is necessary for ChatOpenAI model to get the result. 
so it should be changed with this two line
https://github.com/langchain-ai/langchain/blob/08f5e6b8012f5eda2609103f33676199a3781a15/libs/langchain/langchain/chat_models/promptlayer_openai.py#L59
https://github.com/langchain-ai/langchain/blob/08f5e6b8012f5eda2609103f33676199a3781a15/libs/langchain/langchain/chat_models/promptlayer_openai.py#L98
Loading documents from a YouTube url doesn't work.
Labels: name: area: doc loader, name: auto:bug

### System Info

LangChain 0.0.247, Python 3.11, Linux.

### Who can help?

@rlancemartin

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Described [here](https://python.langchain.com/docs/integrations/document_loaders/youtube_audio).
```python
from langchain.document_loaders.generic import GenericLoader
from langchain.document_loaders.parsers import OpenAIWhisperParser
from langchain.document_loaders.blob_loaders.youtube_audio import YoutubeAudioLoader

# Two Karpathy lecture videos
urls = ["https://youtu.be/kCc8FmEb1nY", "https://youtu.be/VMj-3S1tku0"]

# Directory to save audio files
save_dir = "~/Downloads/YouTube"

# Transcribe the videos to text
loader = GenericLoader(YoutubeAudioLoader(urls, save_dir), OpenAIWhisperParser())
docs = loader.load()
print(docs)
```
I get an empty list.

```log
[youtube] Extracting URL: https://youtu.be/kCc8FmEb1nY
[youtube] kCc8FmEb1nY: Downloading webpage
[youtube] kCc8FmEb1nY: Downloading ios player API JSON
[youtube] kCc8FmEb1nY: Downloading android player API JSON
[youtube] kCc8FmEb1nY: Downloading m3u8 information
[info] kCc8FmEb1nY: Downloading 1 format(s): 140
[download] Destination: /home/dm/Downloads/YouTube/Let's build GPT： from scratch, in code, spelled out..m4a
[download] 100% of  107.73MiB in 00:00:11 at 9.19MiB/s
[FixupM4a] Correcting container of "/home/dm/Downloads/YouTube/Let's build GPT： from scratch, in code, spelled out..m4a"
[ExtractAudio] Not converting audio /home/dm/Downloads/YouTube/Let's build GPT： from scratch, in code, spelled out..m4a; file is already in target format m4a
[youtube] Extracting URL: https://youtu.be/VMj-3S1tku0
[youtube] VMj-3S1tku0: Downloading webpage
[youtube] VMj-3S1tku0: Downloading ios player API JSON
[youtube] VMj-3S1tku0: Downloading android player API JSON
[youtube] VMj-3S1tku0: Downloading m3u8 information
[info] VMj-3S1tku0: Downloading 1 format(s): 140
[download] Destination: /home/dm/Downloads/YouTube/The spelled-out intro to neural networks and backpropagation： building micrograd.m4a
[download] 100% of  135.08MiB in 00:00:13 at 9.65MiB/s
[FixupM4a] Correcting container of "/home/dm/Downloads/YouTube/The spelled-out intro to neural networks and backpropagation： building micrograd.m4a"
[ExtractAudio] Not converting audio /home/dm/Downloads/YouTube/The spelled-out intro to neural networks and backpropagation： building micrograd.m4a; file is already in target format m4a
[]
```

### Expected behavior

A non-empty list of documents is expected.
Add StreamlitChatMessageHistory
Labels: name: area: memory, name: lgtm, name: auto:enhancement

Add a StreamlitChatMessageHistory class that stores chat messages in [Streamlit's Session State](https://docs.streamlit.io/library/api-reference/session-state).

Note: The integration test uses a currently-experimental Streamlit testing framework to simulate the execution of a Streamlit app. Marking this PR as draft until I confirm with the Streamlit team that we're comfortable supporting it.
RuntimeError: asyncio.run() cannot be called from a running event loop
Labels: name: area: doc loader, name: area: vector store, name: auto:bug

### System Info

python 3.11


### Who can help?

@rlancemartin 
File langchain/document_loaders/async_html.py:136-->  results = asyncio.run(self.fetch_all(self.web_paths))

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction


from langchain.retrievers.web_research import WebResearchRetriever

import pinecone
import os
from langchain.vectorstores import Pinecone
from langchain.utilities import GoogleSearchAPIWrapper
from common_util.llms import LLM_FACT, EMBEDDINGS
from dotenv import load_dotenv
from common_util.namespaceEnum import PineconeNamespaceEnum

load_dotenv()
index_name = "langchain-demo"

pinecone.init(api_key=os.getenv("PINECONE_API_KEY"), environment=os.getenv("PINECONE_ENV"))

# Vectorstore
vectorstore = Pinecone.from_existing_index(index_name, EMBEDDINGS, namespace=PineconeNamespaceEnum.WEB_SEARCH.value)


# LLM
llm = LLM_FACT

# Search 
os.environ["GOOGLE_CSE_ID"] = os.getenv("GOOGLE_CSE_ID")
os.environ["GOOGLE_API_KEY"] = os.getenv("GOOGLE_API_KEY")
search = GoogleSearchAPIWrapper()

web_research_retriever = WebResearchRetriever.from_llm(
    vectorstore=vectorstore,
    llm=llm, 
    search=search, 
)

from langchain.chains import RetrievalQAWithSourcesChain
user_input = "How do LLM Powered Autonomous Agents work?"
qa_chain = RetrievalQAWithSourcesChain.from_chain_type(llm,retriever=web_research_retriever)
result = qa_chain({"question": user_input})
result

### Expected behavior

should search the web async
Agent stopped due to iteration limit or time limit
Labels: name: area: agent, name: auto:bug

### Issue you'd like to raise.

I am getting "Agent stopped due to iteration limit or time limit" as the error , As my max_iteration is also 15 then too i am getting this error. 
I need some  particular output from the model.
Following is my code 
agent_executor = AgentExecutor.from_agent_and_tools(agent=agent,
                                                tools=tools,
                                                verbose=True, memory = memory, max_iterations = 15)

 

### Suggestion:

_No response_
Added Proper Documentation for `faiss-gpu` Installation
Labels: name: lgtm, name: area: vector store, name: auto:documentation

### Description
 In the LangChain Documentation and Comments, I've Noticed that `pip install faiss` was mentioned, instead of `pip install faiss-gpu`, since installing `pip install faiss` results in an error. I've gone ahead and updated the Documentation, and `faiss.ipynb`. This Change will ensure ease of use for the end user, trying to install `faiss-gpu`.

### Issue: 
Documentation / Comments Related.

### Dependencies:
No Dependencies we're changed only updated the files with the wrong reference.

### Tag maintainer:
 @rlancemartin, @eyurtsev (Thank You for your contributions :smile: )
Add new run types for Runnables
Labels: name: auto:improvement

- allow overriding run_type in on_chain_start

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
no support for loading 70B models via llamacpp
Labels: name: auto:bug

### System Info

LangChain 0.0.242

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

tried to load https://huggingface.co/TheBloke/StableBeluga2-70B-GGML model to work with lanchain's Llamacpp : 

`llm = LlamaCpp(model_path="./stablebeluga2-70b.ggmlv3.q4_0.bin", n_gpu_layers=n_gpu_layers,
 n_batch=n_batch, n_ctx=8192, input={"temperature": 0.01},n_threads=8)
llm_chain = LLMChain(llm=llm, prompt=prompt)`

i see that there is no support in passing  n_gqa=8 parameter that according to [https://github.com/abetlen/llama-cpp-python](https://github.com/abetlen/llama-cpp-python) should be used for 70B models
the error i get is :
error loading model: llama.cpp: tensor 'layers.0.attention.wk.weight' has wrong shape; expected  8192 x  8192, got  8192 x  1024

### Expected behavior

model should be loaded successfuly
add llama index retriever test
Labels: name: auto:improvement

# What
- add llama index retriever test

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: add llama index retriever test
  - Issue: None
  - Dependencies: None
  - Tag maintainer: @rlancemartin, @eyurtsev
  - Twitter handle: @MlopsJ


Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
Fix documentation for from_documents signature
Labels: name: lgtm, name: area: vector store, name: auto:documentation

Docs for from_documents() were outdated as seen in https://github.com/langchain-ai/langchain/issues/8457 .

fixes #8457 

<!-- Thank you for contributing to LangChain!

Replace this comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the same people again.

See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->
FallbackChatOpenAI
Labels: name: auto:improvement

# Description

This PR presents the class `GptTurboWith16kFallback` that extends `ChatOpenAI` with a fallback mechanism for larger language models when encountering a "maximum context length" error.
## Issue

This addition aligns with the discussion [here](https://github.com/langchain-ai/langchain/discussions/1802). If an existing solution or further discussions regarding token-specific fallback mechanisms are available, I'd appreciate being directed to them.
## Dependencies

No additional dependencies.
## Additional context


While this PR provides a solution for automatic fallbacks when handling larger input contexts, it is merely a stepping stone. The goal is to broaden this concept and allow for fallback models with larger context capabilities across different models, not just ChatOpenAI. We aim to refine this idea and warmly welcome further contributions.
## Maintainers

@baskaryan, @hwchase17
## Proposed changes

`GptTurboWith16kFallback` introduces a `fallback_model`attribute. This attribute specifies a larger language model to switch to when encountering a "maximum context length" error. The default fallback_model is "gpt-3.5-turbo-16k-0613", and it can be customized during class initialization.

Overriding of the `_generate `method from `ChatOpenAI`. If a "maximum context length" error occurs during the generation, a new `GptTurboWith16kFallback` instance is created with the fallback_model and a new generation attempt is made.

Moreover, it's plausible that the existing retry logic can handle this situation. If there's a way to leverage that, I'd love to understand it better.

I highly appreciate your feedback and contributions to enhancing this concept. As a new OSS contributor, I'm seeking your understanding and assistance in making this contribution more effective. Thank you.