delayedkarma/langchain-issues huggingface dataset:
Fix typo in tigris
Labels: name: auto:documentation

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->

The link has a **typo** in [tigirs docs](https://python.langchain.com/docs/integrations/providers/tigris), so I couldn't access it. So, I have corrected it.
Thanks! ☺️
PipelinePromptTemplate does not respect partial_variables
Labels: name: auto:bug, name: area: models

### System Info

langchain 0.0.271

### Who can help?

@hwchase17 

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

This raises a key error because `word` is treated like an input variable even though it's a partial variables

```python

p = PipelinePromptTemplate(
    final_prompt=PromptTemplate.from_template("this {word} work"),
    pipeline_prompts=[],
    input_variables=[],
    partial_variables={"word": "does"}
)

print(p.format())
# or
print(p.partial().format())
```

### Expected behavior

partial_variables should be interpolated properly
Allow specifying dtype in `langchain.llms.VLLM`
Labels: name: auto:improvement, name: area: models

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->

  - Description: add `dtype` argument for VLLM 
  - Issue: #9593 
  - Dependencies: none
  - Tag maintainer: @hwchase17, @baskaryan
Fixed the error on ConfluenceLoader when content_format=VIEW and `keep_markdown_format`=True
Labels: name: lgtm, name: area: doc loader, name: auto:bug


  - Description: a description of the change

 when I set `content_format=ContentFormat.VIEW` and `keep_markdown_format=True` on ConfluenceLoader, it shows the following error:
```
langchain/document_loaders/confluence.py", line 459, in process_page
    page["body"]["storage"]["value"], heading_style="ATX"
KeyError: 'storage'
```
The reason is because the content format was set to `view` but it was still trying to get the content from `page["body"]["storage"]["value"]`.

Also added the other content formats which are supported by Atlassian API
https://stackoverflow.com/questions/34353955/confluence-rest-api-expanding-page-body-when-retrieving-page-by-title/34363386#34363386

  - Issue: the issue # it fixes (if applicable),

Not applicable.

  - Dependencies: any dependencies required for this change,

Added optional dependency `markdownify` if anyone wants to extract in markdown format.
Monitoring the progress of long running vectorDB index updates
Labels: name: area: vector store, name: auto:enhancement

### Feature request

Add the option to output progress bar Methods that add documents to an index, e.g., `from_documents`. 

### Motivation

Methods that add documents to an index can take a long time to execute, e.g., `from_documents`.  When executing from a notebook or CLI it would be very convenient to be able to track the progressing using something like tqdm.

### Your contribution

No
Cache poetry install + unify Python/Poetry setup for lint and test jobs.
Labels: name: auto:improvement

With this PR:
- All lint and test jobs use the exact same Python + Poetry installation approach, instead of lints doing it one way and tests doing it another way.
- The Poetry installation itself is cached, which saves ~15s per run.
- We no longer pass shell commands as workflow arguments to a workflow that just runs them in a shell. This makes our actions more resilient to shell code injection.

If y'all like this approach, I can modify the scheduled tests workflow and the release workflow to use this too.
DOC: Langchain Supabase Vectorstore ID Incompatability
Labels: name: area: vector store, name: auto:bug, name: auto:documentation

### Issue with current documentation:

The current documentation recommends running the following SQL function to create a table and function for using a supabase postgres database as a vector store. 

```postgres
-- Enable the pgvector extension to work with embedding vectors
create extension vector;

-- Create a table to store your documents
create table documents (
  id bigserial primary key,
  content text, -- corresponds to Document.pageContent
  metadata jsonb, -- corresponds to Document.metadata
  embedding vector(1536) -- 1536 works for OpenAI embeddings, change if needed
);

-- Create a function to search for documents
create function match_documents (
  query_embedding vector(1536),
  match_count int default null,
  filter jsonb DEFAULT '{}'
) returns table (
  id bigint,
  content text,
  metadata jsonb,
  similarity float
)
language plpgsql
as $$
#variable_conflict use_column
begin
  return query
  select
    id,
    content,
    metadata,
    1 - (documents.embedding <=> query_embedding) as similarity
  from documents
  where metadata @> filter
  order by documents.embedding <=> query_embedding
  limit match_count;
end;
$$;
```

This creates a table with a column named id of type `bigint`

However when using the `SupabaseVectorStore` class and trying to run methods such as `add_documents` or `add_texts`, an attempt is made to insert `Document` objects into the database. However it uses the uuid generated as the id which is incompatible with the `bigint`  column type resulting in errors such as 

```bash
postgrest.exceptions.APIError: {'code': '22P02', 'details': None, 'hint': None, 'message': 'invalid input syntax for type bigint: "dbf8aa60-8295-450c-83bc-7395e2836a6a"'}
```


### Idea or request for content:

Instead the documentation should recommend using the column type of uuid which is natively supported within supabase
Test experimental package with `langchain` on `master` branch.
Labels: name: auto:improvement

It's possible that langchain-experimental works fine with the latest *published* langchain, but is broken with the langchain on `master`. Unfortunately, you can see this is currently the case — this is why this PR also includes a minor fix for the `langchain` package itself.

We want to catch situations like that *before* releasing a new langchain, hence this test.
Reduce cache timeouts to ensure faster builds on timeout.
Labels: name: auto:improvement

The current timeouts are too long, and mean that if the GitHub cache decides to act up, jobs get bogged down for 15min at a time. This has happened 2-3 times already this week -- a tiny fraction of our total workflows but really annoying when it happens to you. We can do better.

Installing deps on cache miss takes about ~4min, so it's not worth waiting more than 4min for the deps cache. The black and mypy caches save 1 and 2min, respectively, so wait only up to that long to download them.
duckdb too small to be a Parquet file
Labels: name: area: vector store, name: auto:bug

### System Info

langchain               0.0.193 
python 3.8.10


### Who can help?

@agola11 
@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [x] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [x] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. Im creating multiple vector stores using Chroma DB and saved these inside subdirectories inside a directory DB/ , all occurring inside of a docker container. I'm using persist method as you detail in the docs.
2.  save and load work as expected, however, after container restart (an expected operation during development), I got the following error related to parquet file in the DB subdirectory: Invalid Input Error: File '/app/DB/SIC/chroma-embeddings.parquet' too small to be a Parquet file. During interaction with Chroma DB app only reads data not adds or overwrites data. 

it seems that the reading operation overwrites the chroma-embeddings.parquet file.


### Expected behavior

I expect that after the container restarts the DB could be loaded and not require to be rebuilt.
Add indexing support
Labels: name: area: vector store, name: auto:enhancement

This PR introduces a persistence layer to help with indexing workflows into
vectostores.

The indexing code helps users to:

1. Avoid writing duplicated content into the vectostore
2. Avoid over-writing content if it's unchanged

Importantly, this keeps on working even if the content being written is derived
via a set of transformations from some source content (e.g., indexing children
documents that were derived from parent documents by chunking.)

The two main components are:

1. Persistence layer that keeps track of which keys were updated and when.
   Keeping track of the timestamp of updates, allows to clean up old content
   safely, and with minimal complexity.
2. HashedDocument which is used to hash the contents (including metadata) of
   the documents. We rely on the hashes for identifying duplicates.


The indexing code works with **ANY** document loader. To add transformations
to the documents, users for now can add a custom document loader
that composes an existing loader together with document transformers.
Croma Db Wrapper not considering the **kwargs for quering the collection
Labels: name: area: vector store, name: auto:improvement

### System Info

langchain-0.0.270 

Croma Db Wrapper not considering the **kwargs for querying the collection

Method: similarity_search_with_score
Code : 
results = self.__query_collection(
                query_embeddings=[query_embedding], n_results=k, where=filter
            )

the query collection doesn't uses **kwargs which will be helpful to query the document that contains "string"

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

langchain-0.0.270 

Croma Db Wrapper not considering the **kwargs for querying the collection

Method: similarity_search_with_score
Code : 
results = self.__query_collection(
                query_embeddings=[query_embedding], n_results=k, where=filter
            )

the query collection doesn't uses **kwargs which will be helpful to query the document that contains "string"

### Expected behavior

langchain-0.0.270 

Croma Db Wrapper not considering the **kwargs for querying the collection

Method: similarity_search_with_score
Code : 
results = self.__query_collection(
                query_embeddings=[query_embedding], n_results=k, where=filter
            )

the query collection doesn't uses **kwargs which will be helpful to query the document that contains "string"
Add to support polars
Labels: name: lgtm, name: area: doc loader, name: auto:enhancement

### Description
Polars is a DataFrame interface on top of an OLAP Query Engine implemented in Rust.
Polars is faster to read than pandas, so I'm looking forward to seeing it added to the document loader.

### Issue
N/A

### Dependencies
polars (https://pola-rs.github.io/polars-book/user-guide/)

### Tag maintainer
x

### Twitter handle
N/A
TypeError: Object of type ConversationalRetrievalChain is not JSON serializable
Labels: name: auto:bug, name: area: models

### System Info

langchain==0.0.249


### Who can help?

@hwchase17 @agola11 @eyurtsev 

### Information

- [X] The official example notebooks/scripts
- [x] My own modified scripts

### Related Components

- [x] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

from langchain.chains import  ConversationalRetrievalChain
qa = ConversationalRetrievalChain.from_llm(with some values such as vector store, memory etc)
now i want to serialize qa, or store it, the key is to be able to store the qa and pass it wherever i want, ideally store it in one endpoint and pass to another endpoint
but i get errors, about not being able to serialize, which makes it difficult
i have tried flask session, json.dumps, pickle, cloud pickle, tried to store in a database etc, still get an error


### Expected behavior

I should be able to serialize or save the QA, so i can pass it from one endpoint to another, to keep user in session, and isolate a user's QA from others
Upload ChromDB vectordb from s3 
Labels: name: area: embeddings, name: area: doc loader, name: area: vector store, name: auto:enhancement

### Feature request


Greetings everyone,

I'm interested in directly saving the chromaDB vector store to an S3 bucket. Is there a way to accomplish this?

### Motivation

I want to run my LLM directly in the AWS cloud, but firstly I need to deal with implementation of efficient loading and saving of my db. 

### Your contribution

I already implemented function to load data from s3 and creating the vector store.

```
import boto3

from langchain.document_loaders import S3DirectoryLoader
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings

# Initialize the S3 client
s3 = boto3.client('s3')

# Specify the S3 bucket and directory path
bucket_name = 'bucket_name'
directory_key = 's3_path'

# List objects with a delimiter to get only common prefixes (directories)
response = s3.list_objects_v2(Bucket=bucket_name, Prefix=directory_path, Delimiter='/')

# Extract the common prefixes (directories) from the response
common_prefixes = response.get('CommonPrefixes', [])

# Print the directory names
for prefix in common_prefixes:
    print(prefix['Prefix'])

def create_chromadb_from_s3():
    # Load data from s3
    docs = []
    for key in s3.list_objects_v2(Bucket=bucket_name, Prefix=directory_path, Delimiter='/').get('CommonPrefixes', []):
        loader = S3DirectoryLoader(bucket_name, key['Prefix'])
        docs.extend(loader.load())    

    # Split documents 
    text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
    texts = text_splitter.split_documents(docs)

    embeddings = OpenAIEmbeddings(openai_api_key=open_ai_secret)

    db = Chroma.from_documents(
        texts, embedding=embeddings
    )
    return db 
    
```
Unfortunately  there is no way to set "persist_dir" as connection to the s3 bucket. Any idea how we can implement it?
Add del vector pgvector + adding modification time to confluence and google drive docs
Labels: name: lgtm, name: area: doc loader, name: auto:improvement

Description:
- adding implementation of delete for pgvector
- adding modification time in docs metadata for confluence and google drive.

Issue:
https://github.com/langchain-ai/langchain/issues/9312

Tag maintainer: @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
Expose x-ratelimit-* headers from OpenAI API with Langchain
Labels: name: area: embeddings, name: auto:improvement, name: area: models

### Describe the feature or improvement you're requesting

When making a direct call to OpenAI's POST /v1/chat/completions endpoint, we receive valuable headers that provide information about the rate limiting, including:

```
x-ratelimit-limit-requests: 3500
x-ratelimit-remaining-requests: 3499
x-ratelimit-reset-requests: 17ms
```
However, when using the `ChatOpenAI` and `OpenAIEmbeddings` APIs, these rate limit headers are not returned in the response or as part of the `OpenAICallbackHandler` object (as total_tokens or total_cost values). This makes it challenging to track and manage rate limits programmatically.

Can we get rate headers values using `ChatOpenAI` and `OpenAIEmbeddings`?

It would be nice if we could get access to these headers, or have them returned in a helpful format along with the response.

### Motivation

Having access to rate limit information is crucial for developers to effectively manage their API usage, especially when using these APIs in production environments. This addition would greatly enhance the usability and monitoring capabilities of the OpenAI API.
List in ChatMessageHistory is not correctly initialized
Labels: name: auto:bug, name: area: models

### System Info

langchain version 0.0.270

### Who can help?

@hwchase17

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Create multiple instances of ChatMessageHistory and add history. They shared the list.

### Expected behavior

`messages: List[BaseMessage] = []
`
to

`messages: List[BaseMessage] = Field(default_factory=list)`

Fix [here](https://github.com/langchain-ai/langchain/pull/9594)
Allow specifying dtype in `langchain.llms.VLLM`
Labels: name: auto:enhancement, name: area: models

### Feature request

Add option to specify dtype (e.g. `float16`) in `langchain.llms.VLLM()` constructor, 

### Motivation

Currently, `vllm` library provides an argument to use `float16` dtype, but langchain doesn't. 

There is no way to use langchain VLLM with a GPU < 8.0 compute capability.

![image](https://github.com/langchain-ai/langchain/assets/22888849/4ff6d075-58a7-44f0-bc93-812c3e95d45f)


![image](https://github.com/langchain-ai/langchain/assets/22888849/68d4a3f0-5056-47a4-9d44-742603c53f1f)



### Your contribution

I can contribute with guideline from langchain team
Merge from original langchain
Labels: name: auto:refactor, name: area: models

<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc: 
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
  1. a test for the integration, preferably unit tests that do not rely on network access,
  2. an example notebook showing its use. These live is docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->