Tony20100/langchain_full huggingface dataset:
Kinetica Vectorstore API
Kinetica Vectorstore based Retriever
LLMLingua Document Compressor
LanceDB
Lantern
ManticoreSearch VectorStore
Marqo
Meilisearch
Milvus
Momento Vector Index (MVI)
MyScale
Neo4j Vector Index
OpenSearch
OpenVINO Reranker
PGVecto.rs
Pinecone
Postgres Embedding
Qdrant
RankLLM Reranker
Relyt
Rockset
SAP HANA Cloud Vector Engine
SQLite-VSS
ScaNN
SemaDB
SingleStoreDB
Supabase (Postgres)
SurrealDB
Tair
Tencent Cloud VectorDB
TiDB Vector
Tigris
TileDB
Timescale Vector (Postgres)
Typesense
USearch
UpTrain
Upstash Vector
VDMS
Vald
Vearch
Vectara
Vectara 
Vespa
VoyageAI Reranker
Weaviate
Xata
Zilliz
scikit-learn
viking DB
vlite
langchain_community.document_loaders.rtf.UnstructuredRTFLoader¶
class langchain_community.document_loaders.rtf.UnstructuredRTFLoader(file_path: Union[str, Path], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load RTF files using Unstructured.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain_community.document_loaders import UnstructuredRTFLoader
loader = UnstructuredRTFLoader(“example.rtf”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-rtf
Initialize with a file path.
Parameters
file_path (Union[str, Path]) – The path to the file to load.
mode (str) – The mode to use for partitioning. See unstructured for details.
Defaults to “single”.
**unstructured_kwargs (Any) – Additional keyword arguments to pass
to unstructured.
Methods
__init__(file_path[, mode])
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Initialize with a file path.
Parameters
file_path (Union[str, Path]) – The path to the file to load.
mode (str) – The mode to use for partitioning. See unstructured for details.
Defaults to “single”.
**unstructured_kwargs (Any) – Additional keyword arguments to pass
to unstructured.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using UnstructuredRTFLoader¶
Unstructured
langchain_community.document_loaders.pdf.PyPDFium2Loader¶
class langchain_community.document_loaders.pdf.PyPDFium2Loader(file_path: str, *, headers: Optional[Dict] = None, extract_images: bool = False)[source]¶
Load PDF using pypdfium2 and chunks at character level.
Initialize with a file path.
Attributes
source
Methods
__init__(file_path, *[, headers, extract_images])
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load given path as pages.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (str) – 
headers (Optional[Dict]) – 
extract_images (bool) – 
__init__(file_path: str, *, headers: Optional[Dict] = None, extract_images: bool = False)[source]¶
Initialize with a file path.
Parameters
file_path (str) – 
headers (Optional[Dict]) – 
extract_images (bool) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load given path as pages.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using PyPDFium2Loader¶
How to load PDFs
langchain_community.document_loaders.modern_treasury.ModernTreasuryLoader¶
class langchain_community.document_loaders.modern_treasury.ModernTreasuryLoader(resource: str, organization_id: Optional[str] = None, api_key: Optional[str] = None)[source]¶
Load from Modern Treasury.
Parameters
resource (str) – The Modern Treasury resource to load.
organization_id (Optional[str]) – The Modern Treasury organization ID. It can also be
specified via the environment variable
“MODERN_TREASURY_ORGANIZATION_ID”.
api_key (Optional[str]) – The Modern Treasury API key. It can also be specified via
the environment variable “MODERN_TREASURY_API_KEY”.
Methods
__init__(resource[, organization_id, api_key])
param resource
The Modern Treasury resource to load.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(resource: str, organization_id: Optional[str] = None, api_key: Optional[str] = None) → None[source]¶
Parameters
resource (str) – The Modern Treasury resource to load.
organization_id (Optional[str]) – The Modern Treasury organization ID. It can also be
specified via the environment variable
“MODERN_TREASURY_ORGANIZATION_ID”.
api_key (Optional[str]) – The Modern Treasury API key. It can also be specified via
the environment variable “MODERN_TREASURY_API_KEY”.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using ModernTreasuryLoader¶
Modern Treasury
langchain_community.document_loaders.yuque.YuqueLoader¶
class langchain_community.document_loaders.yuque.YuqueLoader(access_token: str, api_url: str = 'https://www.yuque.com')[source]¶
Load documents from Yuque.
Initialize with Yuque access_token and api_url.
Parameters
access_token (str) – Personal access token - see https://www.yuque.com/settings/tokens.
api_url (str) – Yuque API url.
Attributes
headers
Methods
__init__(access_token[, api_url])
Initialize with Yuque access_token and api_url.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
get_books(user_id)
get_document(book_id, document_id)
get_document_ids(book_id)
get_documents()
get_user_id()
http_get(url)
lazy_load()
A lazy loader for Documents.
load()
Load documents from Yuque.
load_and_split([text_splitter])
Load Documents and split into chunks.
parse_document(document)
parse_document_body(body)
__init__(access_token: str, api_url: str = 'https://www.yuque.com')[source]¶
Initialize with Yuque access_token and api_url.
Parameters
access_token (str) – Personal access token - see https://www.yuque.com/settings/tokens.
api_url (str) – Yuque API url.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
get_books(user_id: int) → List[Dict][source]¶
Parameters
user_id (int) – 
Return type
Parameters
user_id (int) – 
Return type
List[Dict]
get_document(book_id: int, document_id: int) → Dict[source]¶
Parameters
book_id (int) – 
document_id (int) – 
Return type
Dict
get_document_ids(book_id: int) → List[int][source]¶
Parameters
book_id (int) – 
Return type
List[int]
get_documents() → Iterator[Document][source]¶
Return type
Iterator[Document]
get_user_id() → int[source]¶
Return type
int
http_get(url: str) → Dict[source]¶
Parameters
url (str) – 
Return type
Dict
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents from Yuque.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
parse_document(document: Dict) → Document[source]¶
Parameters
document (Dict) – 
Return type
Document
static parse_document_body(body: str) → str[source]¶
Parameters
body (str) – 
Return type
str
Examples using YuqueLoader¶
Yuque
langchain_community.document_loaders.obs_file.OBSFileLoader¶
class langchain_community.document_loaders.obs_file.OBSFileLoader(bucket: str, key: str, client: Optional[Any] = None, endpoint: str = '', config: Optional[dict] = None)[source]¶
Load from the Huawei OBS file.
Initialize the OBSFileLoader with the specified settings.
Parameters
bucket (str) – The name of the OBS bucket to be used.
key (str) – The name of the object in the OBS bucket.
client (ObsClient, optional) – An instance of the ObsClient to connect to OBS.
endpoint (str, optional) – The endpoint URL of your OBS bucket. This parameter is mandatory if client is not provided.
config (dict, optional) – The parameters for connecting to OBS, provided as a dictionary. This parameter is ignored if client is provided. The dictionary could have the following keys:
- “ak” (str, optional): Your OBS access key (required if get_token_from_ecs is False and bucket policy is not public read).
- “sk” (str, optional): Your OBS secret key (required if get_token_from_ecs is False and bucket policy is not public read).
- “token” (str, optional): Your security token (required if using temporary credentials).
- “get_token_from_ecs” (bool, optional): Whether to retrieve the security token from ECS. Defaults to False if not provided. If set to True, ak, sk, and token will be ignored.
Raises
ValueError – If the esdk-obs-python package is not installed.
TypeError – If the provided client is not an instance of ObsClient.
ValueError – If client is not provided, but endpoint is missing.
Note
ValueError – If client is not provided, but endpoint is missing.
Note
Before using this class, make sure you have registered with OBS and have the necessary credentials. The ak, sk, and endpoint values are mandatory unless get_token_from_ecs is True or the bucket policy is public read. token is required when using temporary credentials.
Example
To create a new OBSFileLoader with a new client:
```
config = {
“ak”: “your-access-key”,
“sk”: “your-secret-key”
}
obs_loader = OBSFileLoader(“your-bucket-name”, “your-object-key”, config=config)
```
To create a new OBSFileLoader with an existing client:
```
from obs import ObsClient
# Assuming you have an existing ObsClient object ‘obs_client’
obs_loader = OBSFileLoader(“your-bucket-name”, “your-object-key”, client=obs_client)
```
To create a new OBSFileLoader without an existing client:
`
obs_loader = OBSFileLoader("your-bucket-name", "your-object-key", endpoint="your-endpoint-url")
`
Methods
__init__(bucket, key[, client, endpoint, config])
Initialize the OBSFileLoader with the specified settings.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(bucket: str, key: str, client: Optional[Any] = None, endpoint: str = '', config: Optional[dict] = None) → None[source]¶
Initialize the OBSFileLoader with the specified settings.
Parameters
bucket (str) – The name of the OBS bucket to be used.
Parameters
bucket (str) – The name of the OBS bucket to be used.
key (str) – The name of the object in the OBS bucket.
client (ObsClient, optional) – An instance of the ObsClient to connect to OBS.
endpoint (str, optional) – The endpoint URL of your OBS bucket. This parameter is mandatory if client is not provided.
config (dict, optional) – The parameters for connecting to OBS, provided as a dictionary. This parameter is ignored if client is provided. The dictionary could have the following keys:
- “ak” (str, optional): Your OBS access key (required if get_token_from_ecs is False and bucket policy is not public read).
- “sk” (str, optional): Your OBS secret key (required if get_token_from_ecs is False and bucket policy is not public read).
- “token” (str, optional): Your security token (required if using temporary credentials).
- “get_token_from_ecs” (bool, optional): Whether to retrieve the security token from ECS. Defaults to False if not provided. If set to True, ak, sk, and token will be ignored.
Raises
ValueError – If the esdk-obs-python package is not installed.
TypeError – If the provided client is not an instance of ObsClient.
ValueError – If client is not provided, but endpoint is missing.
Return type
None
Note
Before using this class, make sure you have registered with OBS and have the necessary credentials. The ak, sk, and endpoint values are mandatory unless get_token_from_ecs is True or the bucket policy is public read. token is required when using temporary credentials.
Example
To create a new OBSFileLoader with a new client:
```
config = {
“ak”: “your-access-key”,
“sk”: “your-secret-key”
}
“ak”: “your-access-key”,
“sk”: “your-secret-key”
}
obs_loader = OBSFileLoader(“your-bucket-name”, “your-object-key”, config=config)
```
To create a new OBSFileLoader with an existing client:
```
from obs import ObsClient
# Assuming you have an existing ObsClient object ‘obs_client’
obs_loader = OBSFileLoader(“your-bucket-name”, “your-object-key”, client=obs_client)
```
To create a new OBSFileLoader without an existing client:
`
obs_loader = OBSFileLoader("your-bucket-name", "your-object-key", endpoint="your-endpoint-url")
`
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using OBSFileLoader¶
Huawei
Huawei OBS File
langchain_community.document_loaders.async_html.AsyncHtmlLoader¶
class langchain_community.document_loaders.async_html.AsyncHtmlLoader(web_path: Union[str, List[str]], header_template: Optional[dict] = None, verify_ssl: Optional[bool] = True, proxies: Optional[dict] = None, autoset_encoding: bool = True, encoding: Optional[str] = None, default_parser: str = 'html.parser', requests_per_second: int = 2, requests_kwargs: Optional[Dict[str, Any]] = None, raise_for_status: bool = False, ignore_load_errors: bool = False, *, preserve_order: bool = True, trust_env: bool = False)[source]¶
Load HTML asynchronously.
Initialize with a webpage path.
Methods
__init__(web_path[, header_template, ...])
Initialize with a webpage path.
alazy_load()
Lazy load text from the url(s) in web_path.
aload()
Load data into Document objects.
fetch_all(urls)
Fetch all urls concurrently with rate limiting.
lazy_load()
Lazy load text from the url(s) in web_path.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
web_path (Union[str, List[str]]) – 
header_template (Optional[dict]) – 
verify_ssl (Optional[bool]) – 
proxies (Optional[dict]) – 
autoset_encoding (bool) – 
encoding (Optional[str]) – 
default_parser (str) – 
requests_per_second (int) – 
requests_kwargs (Optional[Dict[str, Any]]) – 
raise_for_status (bool) – 
ignore_load_errors (bool) – 
preserve_order (bool) – 
trust_env (bool) –
preserve_order (bool) – 
trust_env (bool) – 
__init__(web_path: Union[str, List[str]], header_template: Optional[dict] = None, verify_ssl: Optional[bool] = True, proxies: Optional[dict] = None, autoset_encoding: bool = True, encoding: Optional[str] = None, default_parser: str = 'html.parser', requests_per_second: int = 2, requests_kwargs: Optional[Dict[str, Any]] = None, raise_for_status: bool = False, ignore_load_errors: bool = False, *, preserve_order: bool = True, trust_env: bool = False)[source]¶
Initialize with a webpage path.
Parameters
web_path (Union[str, List[str]]) – 
header_template (Optional[dict]) – 
verify_ssl (Optional[bool]) – 
proxies (Optional[dict]) – 
autoset_encoding (bool) – 
encoding (Optional[str]) – 
default_parser (str) – 
requests_per_second (int) – 
requests_kwargs (Optional[Dict[str, Any]]) – 
raise_for_status (bool) – 
ignore_load_errors (bool) – 
preserve_order (bool) – 
trust_env (bool) – 
async alazy_load() → AsyncIterator[Document][source]¶
Lazy load text from the url(s) in web_path.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
async fetch_all(urls: List[str]) → List[str][source]¶
Fetch all urls concurrently with rate limiting.
Parameters
urls (List[str]) – 
Return type
List[str]
lazy_load() → Iterator[Document][source]¶
Lazy load text from the url(s) in web_path.
Lazy load text from the url(s) in web_path.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using AsyncHtmlLoader¶
Activeloop Deep Memory
AsyncHtml
HTML to text
Markdownify
langchain_community.document_loaders.tidb.TiDBLoader¶
class langchain_community.document_loaders.tidb.TiDBLoader(connection_string: str, query: str, page_content_columns: Optional[List[str]] = None, metadata_columns: Optional[List[str]] = None, engine_args: Optional[Dict[str, Any]] = None)[source]¶
Load documents from TiDB.
Initialize TiDB document loader.
Parameters
connection_string (str) – The connection string for the TiDB database,
format: “mysql+pymysql://root@127.0.0.1:4000/test”.
query (str) – The query to run in TiDB.
page_content_columns (Optional[List[str]]) – Optional. Columns written to Document page_content,
default(None) to all columns.
metadata_columns (Optional[List[str]]) – Optional. Columns written to Document metadata,
default(None) to no columns.
engine_args (Optional[Dict[str, Any]]) – Optional. Additional arguments to pass to sqlalchemy engine.
Methods
__init__(connection_string, query[, ...])
Initialize TiDB document loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load TiDB data into document objects.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(connection_string: str, query: str, page_content_columns: Optional[List[str]] = None, metadata_columns: Optional[List[str]] = None, engine_args: Optional[Dict[str, Any]] = None) → None[source]¶
Initialize TiDB document loader.
Parameters
connection_string (str) – The connection string for the TiDB database,
Parameters
connection_string (str) – The connection string for the TiDB database,
format: “mysql+pymysql://root@127.0.0.1:4000/test”.
query (str) – The query to run in TiDB.
page_content_columns (Optional[List[str]]) – Optional. Columns written to Document page_content,
default(None) to all columns.
metadata_columns (Optional[List[str]]) – Optional. Columns written to Document metadata,
default(None) to no columns.
engine_args (Optional[Dict[str, Any]]) – Optional. Additional arguments to pass to sqlalchemy engine.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load TiDB data into document objects.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using TiDBLoader¶
TiDB
langchain_community.document_loaders.powerpoint.UnstructuredPowerPointLoader¶
class langchain_community.document_loaders.powerpoint.UnstructuredPowerPointLoader(file_path: Optional[Union[str, List[str], Path, List[Path]]], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load Microsoft PowerPoint files using Unstructured.
Works with both .ppt and .pptx files.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain_community.document_loaders import UnstructuredPowerPointLoader
loader = UnstructuredPowerPointLoader(“example.pptx”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-pptx
Initialize with file path.
Methods
__init__(file_path[, mode])
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (Optional[Union[str, List[str], Path, List[Path]]]) – 
mode (str) – 
unstructured_kwargs (Any) – 
__init__(file_path: Optional[Union[str, List[str], Path, List[Path]]], mode: str = 'single', **unstructured_kwargs: Any)¶