Tony20100/langchain_full huggingface dataset:
langchain_community.document_loaders.unstructured.UnstructuredFileIOLoader¶
class langchain_community.document_loaders.unstructured.UnstructuredFileIOLoader(file: Union[IO, Sequence[IO]], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load files using Unstructured.
The file loader
uses the unstructured partition function and will automatically detect the file
type. You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain_community.document_loaders import UnstructuredFileIOLoader
with open(“example.pdf”, “rb”) as f:
loader = UnstructuredFileIOLoader(f, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition
Initialize with file path.
Methods
__init__(file[, mode])
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file (Union[IO, Sequence[IO]]) – 
mode (str) – 
unstructured_kwargs (Any) – 
__init__(file: Union[IO, Sequence[IO]], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Initialize with file path.
Parameters
Initialize with file path.
Parameters
file (Union[IO, Sequence[IO]]) – 
mode (str) – 
unstructured_kwargs (Any) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using UnstructuredFileIOLoader¶
Google Drive
Unstructured
langchain_community.document_loaders.couchbase.CouchbaseLoader¶
class langchain_community.document_loaders.couchbase.CouchbaseLoader(connection_string: str, db_username: str, db_password: str, query: str, *, page_content_fields: Optional[List[str]] = None, metadata_fields: Optional[List[str]] = None)[source]¶
Load documents from Couchbase.
Each document represents one row of the result. The page_content_fields are
written into the page_content`of the document. The `metadata_fields are written
into the metadata of the document. By default, all columns are written into
the page_content and none into the metadata.
Initialize Couchbase document loader.
Parameters
connection_string (str) – The connection string to the Couchbase cluster.
db_username (str) – The username to connect to the Couchbase cluster.
db_password (str) – The password to connect to the Couchbase cluster.
query (str) – The SQL++ query to execute.
page_content_fields (Optional[List[str]]) – The columns to write into the
page_content field of the document. By default, all columns are
written.
metadata_fields (Optional[List[str]]) – The columns to write into the
metadata field of the document. By default, no columns are written.
Methods
__init__(connection_string, db_username, ...)
Initialize Couchbase document loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load Couchbase data into Document objects lazily.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(connection_string: str, db_username: str, db_password: str, query: str, *, page_content_fields: Optional[List[str]] = None, metadata_fields: Optional[List[str]] = None) → None[source]¶
Initialize Couchbase document loader.
Parameters
connection_string (str) – The connection string to the Couchbase cluster.
db_username (str) – The username to connect to the Couchbase cluster.
db_password (str) – The password to connect to the Couchbase cluster.
query (str) – The SQL++ query to execute.
page_content_fields (Optional[List[str]]) – The columns to write into the
page_content field of the document. By default, all columns are
written.
metadata_fields (Optional[List[str]]) – The columns to write into the
metadata field of the document. By default, no columns are written.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load Couchbase data into Document objects lazily.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
langchain_community.document_loaders.dropbox.DropboxLoader¶
class langchain_community.document_loaders.dropbox.DropboxLoader[source]¶
Bases: BaseLoader, BaseModel
Load files from Dropbox.
In addition to common files such as text and PDF files, it also supports
Dropbox Paper files.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param dropbox_access_token: str [Required]¶
Dropbox access token.
param dropbox_file_paths: Optional[List[str]] = None¶
The file paths to load from.
param dropbox_folder_path: Optional[str] = None¶
The folder path to load from.
param recursive: bool = False¶
Flag to indicate whether to load files recursively from subfolders.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Parameters
obj (Any) – 
Return type
Model
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
by_alias (bool) – 
skip_defaults (Optional[bool]) – 
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
encoder (Optional[Callable[[Any], Any]]) – 
models_as_dict (bool) – 
dumps_kwargs (Any) – 
Return type
unicode
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
path (Union[str, Path]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod parse_obj(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
b (Union[str, bytes]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
Return type
DictStrAny
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
dumps_kwargs (Any) – 
Return type
unicode
classmethod update_forward_refs(**localns: Any) → None¶
langchain_community.document_loaders.airbyte.AirbyteHubspotLoader¶
class langchain_community.document_loaders.airbyte.AirbyteHubspotLoader(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None)[source]¶
Load from Hubspot using an Airbyte source connector.
Initializes the loader.
Parameters
config (Mapping[str, Any]) – The config to pass to the source connector.
stream_name (str) – The name of the stream to load.
record_handler (Optional[Callable[[Any, Optional[str]], Document]]) – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state (Optional[Any]) – The state to pass to the source connector. Defaults to None.
Attributes
last_state
Methods
__init__(config, stream_name[, ...])
Initializes the loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None) → None[source]¶
Initializes the loader.
Parameters
config (Mapping[str, Any]) – The config to pass to the source connector.
stream_name (str) – The name of the stream to load.
record_handler (Optional[Callable[[Any, Optional[str]], Document]]) – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state (Optional[Any]) – The state to pass to the source connector. Defaults to None.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using AirbyteHubspotLoader¶
Airbyte Hubspot (Deprecated)
langchain_community.document_loaders.hugging_face_model.HuggingFaceModelLoader¶
class langchain_community.document_loaders.hugging_face_model.HuggingFaceModelLoader(*, search: Optional[str] = None, author: Optional[str] = None, filter: Optional[str] = None, sort: Optional[str] = None, direction: Optional[str] = None, limit: Optional[int] = 3, full: Optional[bool] = None, config: Optional[bool] = None)[source]¶
Load model information from Hugging Face Hub, including README content.
This loader interfaces with the Hugging Face Models API to fetch and load
model metadata and README files.
The API allows you to search and filter models based on specific criteria
such as model tags, authors, and more.
API URL: https://huggingface.co/api/models
DOC URL: https://huggingface.co/docs/hub/en/api
Examples
from langchain_community.document_loaders import HuggingFaceModelLoader
# Initialize the loader with search criteria
loader = HuggingFaceModelLoader(search="bert", limit=10)
# Load models
documents = loader.load()
# Iterate through the fetched documents
for doc in documents:
    print(doc.page_content)  # README content of the model
    print(doc.metadata)      # Metadata of the model
Initialize the HuggingFaceModelLoader.
Parameters
search (Optional[str]) – Filter based on substrings for repos and their usernames.
author (Optional[str]) – Filter models by an author or organization.
filter (Optional[str]) – Filter based on tags.
sort (Optional[str]) – Property to use when sorting.
direction (Optional[str]) – Direction in which to sort.
limit (Optional[int]) – Limit the number of models fetched.
full (Optional[bool]) – Whether to fetch most model data.
full (Optional[bool]) – Whether to fetch most model data.
config (Optional[bool]) – Whether to also fetch the repo config.
Attributes
BASE_URL
README_BASE_URL
Methods
__init__(*[, search, author, filter, sort, ...])
Initialize the HuggingFaceModelLoader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
fetch_models()
Fetch model information from Hugging Face Hub.
fetch_readme_content(model_id)
Fetch the README content for a given model.
lazy_load()
Load model information lazily, including README content.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(*, search: Optional[str] = None, author: Optional[str] = None, filter: Optional[str] = None, sort: Optional[str] = None, direction: Optional[str] = None, limit: Optional[int] = 3, full: Optional[bool] = None, config: Optional[bool] = None)[source]¶
Initialize the HuggingFaceModelLoader.
Parameters
search (Optional[str]) – Filter based on substrings for repos and their usernames.
author (Optional[str]) – Filter models by an author or organization.
filter (Optional[str]) – Filter based on tags.
sort (Optional[str]) – Property to use when sorting.
direction (Optional[str]) – Direction in which to sort.
limit (Optional[int]) – Limit the number of models fetched.
full (Optional[bool]) – Whether to fetch most model data.
config (Optional[bool]) – Whether to also fetch the repo config.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
fetch_models() → List[dict][source]¶
Fetch model information from Hugging Face Hub.
Return type
List[dict]
fetch_readme_content(model_id: str) → str[source]¶
Fetch the README content for a given model.
Parameters
model_id (str) – 
Return type
str
lazy_load() → Iterator[Document][source]¶
Load model information lazily, including README content.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
langchain_community.document_loaders.docugami.DocugamiLoader¶
class langchain_community.document_loaders.docugami.DocugamiLoader[source]¶
Bases: BaseLoader, BaseModel
[Deprecated] Load from Docugami.
To use, you should have the dgml-utils python package installed.
Notes
Deprecated since version 0.0.24.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param access_token: Optional[str] = None¶
The Docugami API access token to use.
param api: str = 'https://api.docugami.com/v1preview1'¶
The Docugami API endpoint to use.
param docset_id: Optional[str] = None¶
The Docugami API docset ID to use.
param document_ids: Optional[Sequence[str]] = None¶
The Docugami API document IDs to use.
param file_paths: Optional[Sequence[Union[Path, str]]] = None¶
The local file paths to use.
param include_project_metadata_in_doc_metadata: bool = True¶
Set to True if you want to include the project metadata in the doc metadata.
param include_xml_tags: bool = False¶
Set to true for XML tags in chunk output text.
param max_metadata_length = 512¶
Max length of metadata text returned.
param max_text_length = 4096¶
Max length of chunk text returned.
param min_text_length: int = 32¶
Threshold under which chunks are appended to next to avoid over-chunking.
param parent_hierarchy_levels: int = 0¶
Set appropriately to get parent chunks using the chunk hierarchy.
param parent_id_key: str = 'doc_id'¶
Metadata key for parent doc ID.
Metadata key for parent doc ID.
param sub_chunk_tables: bool = False¶
Set to True to return sub-chunks within tables.
param whitespace_normalize_text: bool = True¶
Set to False if you want to full whitespace formatting in the original
XML doc, including indentation.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to include in new model
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to exclude from new model, as with values this takes precedence over include
update (Optional[DictStrAny]) – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
by_alias (bool) – 
skip_defaults (Optional[bool]) – 
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
encoder (Optional[Callable[[Any], Any]]) – 
models_as_dict (bool) – 
dumps_kwargs (Any) – 
Return type
unicode
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
path (Union[str, Path]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod parse_obj(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
b (Union[str, bytes]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
Return type
DictStrAny
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
dumps_kwargs (Any) – 
Return type
unicode
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
Parameters
localns (Any) – 
Return type
None
classmethod validate(value: Any) → Model¶
Parameters
value (Any) – 
Return type
Model
langchain_community.document_loaders.whatsapp_chat.WhatsAppChatLoader¶
class langchain_community.document_loaders.whatsapp_chat.WhatsAppChatLoader(path: str)[source]¶
Load WhatsApp messages text file.
Initialize with path.
Methods
__init__(path)
Initialize with path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
path (str) – 
__init__(path: str)[source]¶
Initialize with path.
Parameters
path (str) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using WhatsAppChatLoader¶
Facebook - Meta
WhatsApp
WhatsApp Chat