delayedkarma/langchain-issues huggingface dataset:
Issue: getting not implemented error when using async chromium loader
Labels: name: area: doc loader, name: auto:bug

### Issue you'd like to raise.

I tried to use webscraping by using langchain and used the async chromium loader as per the lanchain documentation but I was getting error asyncio.run() error so I made some chnages that i found on web but its still not working .
below is the code:
import asyncio
import playwright
from langchain.document_loaders import AsyncChromiumLoader
from langchain.document_transformers import BeautifulSoupTransformer
import nest_asyncio
nest_asyncio.apply()


async def main():
    # Create an instance of AsyncChromiumLoader with a list of URLs
    urls = ["https://www.hdfcbank.com/personal/borrow/popular-loans/personal-loan/eligibility"]  # Replace with your desired URLs
    loader = AsyncChromiumLoader(urls)

    # Load the URLs
    html = await loader.load()

    # Print the HTML content
    for url, content in html.items():
        print(f"URL: {url}\nHTML Content: {content}\n")

if __name__ == "__main__":
    asyncio.run(main())


### Suggestion:

_No response_
Issue: Why did the agent provide the action and action input, but instead of executing it, it directly finished the chain
Labels: name: area: agent, name: auto:bug

### Issue you'd like to raise.

I provide the agent a get_date function as a tool，but when I ask the agent what is today's date， the agent provide the action and action input, but instead of executing it, it directly finished the chain。
![image](https://github.com/langchain-ai/langchain/assets/149214596/737293ae-ef41-4c66-aef2-fe1267217c37)
![image](https://github.com/langchain-ai/langchain/assets/149214596/e1644d5e-9e52-4322-904a-b332f81d32c8)
![image](https://github.com/langchain-ai/langchain/assets/149214596/eaadad2c-0bc9-43ef-8b5e-d5a12626a3ea)


### Suggestion:

_No response_
Fixes to the docs for timescale vector template
Labels: name: auto:documentation

**Description:** Changes to timescale vector template docs and index description.

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Added ability to use tuned model with ChatVertexAI
Labels: name: auto:improvement, name: area: models

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** Was not able to use a tuned model when using "ChatVertexAI" added that based on how it was done in non chat VertexAI class.
  - **Issue:** N/A
  - **Dependencies:** N/A
  - **Tag maintainer:** N/A
  - **Twitter handle:** @arian_jahiri

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally. (**DONE**)

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->

  - **Description:** Was not able to use a tuned model when using "ChatVertexAI" added that based on how it was done in non chat VertexAI class.
  - **Issue:** N/A
  - **Dependencies:** N/A
  - **Tag maintainer:** @baskaryan 
  - **Twitter handle:** @arian_jahiri
[Fine tuning template] Fine-tune embedding model
Labels: name: area: embeddings, name: auto:enhancement

### Feature request

Set up template to simplify fine-tuning of embedding models.

### Motivation

[Fine-tuning](https://blog.langchain.dev/using-langsmith-to-support-fine-tuning-of-open-source-llms/) of embedding models can improve RAG.

### Your contribution

Add template
[Evaluation template] Runs evaluation against LangSmith (public) dataset
Labels: name: auto:enhancement, name: area: models

### Feature request

Given a runnable (e.g., RAG pipeline) automatically run eval against a given (public) dataset

### Motivation

Made evaluation simple by abstracting w/ a template and passing in only a runnable. 

(Template can be called as RemoteRunnable.)

Benchmark your runnable on an existing public LangSmith dataset.

Will differ depending on the use-case / dataset of choice.

### Your contribution

Add template
[Data generation template] Synthetic QA pairs from input document
Labels: name: auto:enhancement, name: area: models

### Feature request

Synthetic dataset generation and extraction (to JSON or similar)

### Motivation

Datasets (for eval or fine-tuning) are challenging to generate. Some prompts have used LLMs to [generate](https://smith.langchain.com/hub/homanp/question-answer-pair?ref=blog.langchain.dev&organizationId=1fa8b1f4-fcb9-4072-9aa9-983e35ad61b8) them. 

### Your contribution

Add template
[Summarization template] Audio
Labels: name: auto:enhancement, name: area: models

### Feature request

Summarization on audio content (podcasts, YT videos, lectures) would be useful.

### Motivation

Prior work by [AssemblyAI](https://github.com/AssemblyAI/youtube-tutorials/blob/main/podcast-summary-app/podcasts.py) has focused on this.

### Your contribution

Add template
[Extraction template] NER
Labels: name: auto:enhancement, name: area: models

### Feature request

Name Entity Recognition is the process of identifying and classifying named entities (like people, organizations, dates, and locations) in text.



### Motivation

NER is useful in various areas:

1. Information Retrieval: Helps in improving the efficiency of search engines. For example, if someone searches for "Apple," understanding whether they mean the fruit or the tech company can be crucial.

2. Content Recommendation: For news or media agencies, knowing what entities are mentioned in articles can help in recommending related content to users.

3. Data Mining: For extracting structured information from unstructured datasets.

4. Question Answering: Helps chatbots and virtual assistants understand and respond to user queries more accurately.

5. Knowledge Graph Construction: Crucial for building structured databases of information extracted from text.

### Your contribution

Add template
[Agent template] Local / oss agent
Labels: name: area: agent, name: auto:enhancement, name: area: models

### Feature request

Use oss LLM w/ agent (e.g., [Zephyr](https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha))

### Motivation

See notes [here](https://blog.llamaindex.ai/dumber-llm-agents-need-more-constraints-and-better-tools-17a524c59e12), [here](https://betterprogramming.pub/exploring-react-agent-for-better-prompting-in-rag-pipeline-b231aae0ca7c), [here](https://www.pinecone.io/learn/series/langchain/langchain-agents/) on use-cases that admit well to oss LLM agents.

### Your contribution

Add template
[Query construction template] Semantic + SQL with pgvector
Labels: name: area: embeddings, name: area: vector store, name: auto:enhancement

### Feature request

SQL + semantic searching w/ pgvector

### Motivation

Follow the format of [this cookbook](https://github.com/langchain-ai/langchain/pull/12365/files) to enable unified SQL + semantic search (on a column with embeddings).

### Your contribution

Template
[Extraction template] Tables from unstructured text
Labels: name: auto:enhancement, name: area: models

### Feature request

Extraction a structured (e.g., csv, pandas DF, etc) table from unstructured text 

### Motivation

LLMs can infer table structure from text input. 

A template that performs this and output a dataq (e.g., csv, pandas DF, etc) object would be useful.

### Your contribution

Add template
[RAG template] Multi-modal
Labels: name: area: embeddings, name: area: vector store, name: auto:enhancement, name: area: models

### Feature request

Template that features multi-modal RAG

### Motivation

* [Unstructured for document processing](https://github.com/langchain-ai/langchain/blob/master/cookbook/Semi_structured_and_multi_modal_RAG.ipynb)
* Multimodal embeddings
* Showcase vectorstore(s) that support img+text
* Multi-modal LLM for answer synthesis

### Your contribution

Multi-modal RAG
enable the device_map parameter in huggingface pipeline
Labels: name: auto:improvement, name: area: models

### Enabling `device_map` in HuggingFacePipeline 

For multi-gpu settings with large models, the [accelerate](https://huggingface.co/docs/accelerate/usage_guides/big_modeling#using--accelerate) library provides the `device_map` parameter to automatically distribute the model across GPUs / disk. 

The [Transformers pipeline](https://github.com/huggingface/transformers/blob/3520e37e86913715959ff14fef76340010c8de57/src/transformers/pipelines/__init__.py#L543) enables users to specify `device` (or) `device_map`, and handles cases (with warnings) when both are specified. 

However, Langchain's HuggingFacePipeline only supports specifying `device` when calling transformers which limits large models and multi-gpu use-cases. 
Additionally, the [default value](https://github.com/praveenv/langchain/blob/8bd3ce59cd4ef80db5eb52922bb31588596939e1/libs/langchain/langchain/llms/huggingface_pipeline.py#L72) of `device` is initialized to `-1` , which is incompatible with the transformers pipeline when `device_map` is specified. 

This PR addresses the addition of `device_map` as a parameter , and solves the incompatibility of `device = -1` when `device_map` is also specified. 
An additional test has been added for this feature. 

Additionally, some existing tests no longer work since 
1. `max_new_tokens` has to be specified under `pipeline_kwargs` and not `model_kwargs`
2. The GPT2 tokenizer raises a `ValueError: Pipeline with tokenizer without pad_token cannot do batching`, since the `tokenizer.pad_token` is `None` ([related issue](https://github.com/huggingface/transformers/issues/19853) on the transformers repo).

This PR handles fixing these tests as well.
Fixes 'Nonetype' not iterable for ObsidianLoader
Labels: name: area: doc loader, name: auto:bug

When the frontmatter of an Obsidian note contains a tags key which has an empty value an error is raised as a 'Nonetype' is being iterated. This leads to the Loader failing. 

``` markdown
---
date: 2023-10-17 13:59
modification_date: 2023-10-17 13:59
tags: 
---
```

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
Add support for vectorstores that return cosine similarity (and not d…
Labels: name: area: vector store, name: auto:improvement, name: integration: pinecone

Issue: https://github.com/langchain-ai/langchain/issues/12707

Pinecone's default distance strategy on index creation is cosine similarity and not cosine distance. While, we can have a custom _cosine_similarity_relevance_score_fn for Pinecone, I think other vectorstores may also return cosine similarity (and/or distance),i.e. we should expose this to the user so that we can choose the appropriate relevance function
correcting a bug in VertexAIModelGarden class
Labels: name: auto:bug, name: area: models

**Description:** The PR is fixing the following:

- As it stands the VertexAIModelGarden class is not working as the arguments in the code for Generation method is wrong. Corrected for both generate and agenerate
- The location is hardcoded as us-central1. Removed this as the models in Model garden can be deployed in other locations
- Added documentation to tuned_model_name to give an example of it should look like.
 **Tag maintainer:** @baskaryan
 **Dependencies:** No dependency required for this change
Issue: How to implement validation on number of arguments in tools?
Labels: name: area: agent, name: auto:question

### Issue you'd like to raise.

I am using the STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION agent and I am trying to implement tools that take varying number of arguments (e.g. tool1 takes a single argument and tool2 takes 3 arguments). I am using arg_schema to define the arguments and their types. I am wondering how can I return a message to the agent in case the input arguments do not satisfy the arg_schema without erroring out the service. My objective with this is that the agent will try again after modifying the "Thought". How do I do that?


### Suggestion:

_No response_
Issue: Agent doesn't actually utilize custom tool
Labels: name: area: agent, name: auto:bug

### Issue you'd like to raise.

Agent calls get_date_time custom tool but does not actually use it. The following was its output

Action: Amadeus\_flight\_offers\_search,
{
"originLocationCode": "LA",
"destinationLocationCode": "NYC",
"departureDate": "$(get\_date\_time)",
"returnDate": "$(get\_date\_time)",
"adults": 1,
"children": 0,
"max": 1,
"nonStop": false,
"maxPrice": 0,
"currencyCode": "USD",
"travelClass": "ECONOMY"
}
Observation: The flight offers search has been completed. Please review the results for your desired flight.

```
# Create Amadeus toolkit
toolkit = AmadeusToolkit()
tools = toolkit.get_tools()

# Set up the LLMChain, passing in memory
template = """You are an AI chatbot having a conversation with a human.

{history}
Human: {human_input}
AI: """

prompt = PromptTemplate(input_variables=["history", "human_input"], template=template)
                           
llm = ChatOllama(model="mistral:7b-instruct",
                        verbose=True,
                        callback_manager = CallbackManager([StreamingStdOutCallbackHandler()]))

# Define custom tool
datetime_tool = Tool(
    name="get_date_time",
    func=lambda x: datetime.now().isoformat(), # '%Y-%m-%dT%H:%M:%S'
    description="Use the system's default time zone to return the current date and time",
)

# Add the new tool to tool list
tools.append(datetime_tool)

system_message = SystemMessage(content="""
You are a helpful, respectful and honest travel agent.
...
""")

agent = initialize_agent(
    tools=tools,
    llm=llm,
    verbose=True,
    early_stopping_method="generate",
    memory=memory,
    agent_kwargs={"system_message": system_message,
                  "extra_prompt_messages": [MessagesPlaceholder(variable_name="memory")]},
    agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,
)
```


### Suggestion:

_No response_
feat: implement support for different type of containers for AWS Sagemaker LLM Streaming 
Labels: name: auto:enhancement, name: area: models, name: integration: aws

PR adds the ability to have multiple types of content handling on streams, instead of the current hardcoded solution for LMI containers (https://aws.amazon.com/blogs/machine-learning/elevating-the-generative-ai-experience-introducing-streaming-support-in-amazon-sagemaker-hosting/)

It iterates upon #10535 , where I've added the `_stream` implementation  as well as added some Jupyter examples on how to implement.
feat: Supprt pgvecto.rs as a VectorStore
Labels: name: lgtm, name: area: vector store, name: auto:enhancement

Supprt [pgvecto.rs](https://github.com/tensorchord/pgvecto.rs) as a new VectorStore type.

This introduces a new dependency [pgvecto_rs](https://pypi.org/project/pgvecto_rs/) and upgrade SQLAlchemy to ^2.

Relate to https://github.com/tensorchord/pgvecto.rs/issues/11
Update google_vertex_ai_palm.ipynb
Labels: name: lgtm, name: auto:documentation

Fixed a typo

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** Fixed a typo on the code
  - **Issue:** the issue # it fixes (if applicable),


Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. It lives in `docs/extras` directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
 -->
OpenAIEmbeddings fails to connect
Labels: name: area: embeddings, name: auto:bug

### System Info

Ubuntu 22.04
python = "3.11"
langchain = "0.0.320"
openai = "0.28.1"

### Who can help?

https://github.com/hwchase17
https://github.com/agola11

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction



Surprisingly, this very basic call:

```
    def testing_langchain_interface(self):
        openai_key = os.getenv('OPENAI_API_KEY')
        text = "This is the text to be embedded"
        result = OpenAIEmbeddings(openai_api_key=openai_key).embed_documents([text])
```

Results in this error:
```
[2023-11-01 10:10:02] [WARNING] TESTING - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 400 Bad Request'))': /v1/embeddings
[2023-11-01 10:10:02] [WARNING] TESTING - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 400 Bad Request'))': /v1/embeddings
[2023-11-01 10:10:02] [WARNING] TESTING - Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised APIConnectionError: Error communicating with OpenAI: HTTPSConnectionPool(host='api.openai.com', port=443): Max retries exceeded with url: /v1/embeddings (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 400 Bad Request'))).
```
Kwargs values at call site in _embed_with_retry:
```
{'input': [[2028, 374, 279, 1495, 311, 387, 23711]], 'model': 'text-embedding-ada-002', 'request_timeout': None, 'headers': None, 'api_key': '***', , 'organization': '', 'api_base': '', 'api_type': 'openai', 'api_version': ''}
```

While this code:
```
    def testing_openai_interface(self):
        openai.api_key = os.getenv('OPENAI_API_KEY')
        model = "text-embedding-ada-002"
        text = "This is the text to be embedded"
        result = openai.Embedding.create(input=text, model=model, api_type='openai')
```
executes correctly and connects to the OpenAI API.

### Expected behavior

OpenAIEmbeddings should connect without error to OpenAI api.
Correct number of elements in config list in `batch()` and `abatch()` of `BaseLLM`
Labels: name: lgtm, name: auto:bug, name: area: models

- **Description:** Correct number of elements in config list in `batch()` and `abatch()` of `BaseLLM` in case `max_concurrency` is not None.
- **Issue:** #12643
- **Twitter handle:** @akionux
OpenAIEmbeddings timeout parameter doesn't work
Labels: name: area: embeddings, name: auto:bug, name: integration: chroma

### System Info

langchain==0.0.327, MacOS, Python 3.11.6

### Who can help?

@netseye 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

I'm trying to use OpenAIEmbeddings with Chroma

```
embedding_function = OpenAIEmbeddings(
    openai_api_type="azure",
    deployment="text-embedding-ada-002",
    model="text-embedding-ada-002",
    openai_api_base="https://something.openai.azure.com/",
    openai_api_key=os.getenv("AZURE_OPENAI_KEY"),
)

vector_store = Chroma.from_documents(
    docs, embedding_function, persist_directory=os.getenv("CHROMA_DB")
)
```

and I'm getting 
`Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 10.0 seconds as it raised RateLimitError: Requests to the Embeddings_Create Operation under Azure OpenAI API version 2022-12-01 have exceeded call rate limit of your current OpenAI S0 pricing tier. Please retry after 20 seconds. Please go here: https://aka.ms/oai/quotaincrease if you would like to further increase the default rate limit..`

I tried to set request_timeout to different values, but it seems to be ignored, because behaviour doesn't change, I still see this requests to OpenAI endpoint with the same frequency.

### Expected behavior

I would expect having this timeout invocated when there is an Exception from OpenAI is caught.