Tony20100/langchain_full huggingface dataset:
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(bucket: str, prefix: str = '', *, region_name: Optional[str] = None, api_version: Optional[str] = None, use_ssl: Optional[bool] = True, verify: Union[str, bool, None] = None, endpoint_url: Optional[str] = None, aws_access_key_id: Optional[str] = None, aws_secret_access_key: Optional[str] = None, aws_session_token: Optional[str] = None, boto_config: Optional[botocore.client.Config] = None)[source]¶
Initialize with bucket and key name.
Parameters
bucket (str) – The name of the S3 bucket.
prefix (str) – The prefix of the S3 key. Defaults to “”.
region_name (Optional[str]) – The name of the region associated with the client.
A client is associated with a single region.
api_version (Optional[str]) – The API version to use.  By default, botocore will
use the latest API version when creating a client.  You only need
to specify this parameter if you want to use a previous API version
of the client.
use_ssl (Optional[bool]) – Whether to use SSL.  By default, SSL is used.
Note that not all services support non-ssl connections.
verify (Union[str, bool, None]) – Whether to verify SSL certificates.
By default SSL certificates are verified.  You can provide the
following values:
False - do not validate SSL certificates.  SSL will still be
used (unless use_ssl is False), but SSL certificates
will not be verified.
used (unless use_ssl is False), but SSL certificates
will not be verified.
path/to/cert/bundle.pem - A filename of the CA cert bundle to
uses.  You can specify this argument if you want to use a
different CA cert bundle than the one used by botocore.
endpoint_url (Optional[str]) – The complete URL to use for the constructed
client.  Normally, botocore will automatically construct the
appropriate URL to use when communicating with a service.  You can
specify a complete URL (including the “http/https” scheme) to
override this behavior.  If this value is provided, then
use_ssl is ignored.
aws_access_key_id (Optional[str]) – The access key to use when creating
the client.  This is entirely optional, and if not provided,
the credentials configured for the session will automatically
be used.  You only need to provide this argument if you want
to override the credentials used for this specific client.
aws_secret_access_key (Optional[str]) – The secret key to use when creating
the client.  Same semantics as aws_access_key_id above.
aws_session_token (Optional[str]) – The session token to use when creating
the client.  Same semantics as aws_access_key_id above.
boto_config (botocore.client.Config) – Advanced boto3 client configuration options. If a value
is specified in the client config, its value will take precedence
over environment variables and configuration values, but not over
a value passed explicitly to the method. If a default config
object is set on the session, the config object used when creating
the client will be the result of calling merge() on the
default config with the config provided to this call.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using S3DirectoryLoader¶
AWS
AWS S3 Directory
langchain_community.document_loaders.imsdb.IMSDbLoader¶
class langchain_community.document_loaders.imsdb.IMSDbLoader(web_path: Union[str, Sequence[str]] = '', header_template: Optional[dict] = None, verify_ssl: bool = True, proxies: Optional[dict] = None, continue_on_failure: bool = False, autoset_encoding: bool = True, encoding: Optional[str] = None, web_paths: Sequence[str] = (), requests_per_second: int = 2, default_parser: str = 'html.parser', requests_kwargs: Optional[Dict[str, Any]] = None, raise_for_status: bool = False, bs_get_text_kwargs: Optional[Dict[str, Any]] = None, bs_kwargs: Optional[Dict[str, Any]] = None, session: Optional[Any] = None)[source]¶
Load IMSDb webpages.
Initialize loader.
Parameters
web_paths (Sequence[str]) – Web paths to load from.
requests_per_second (int) – Max number of concurrent requests to make.
default_parser (str) – Default parser to use for BeautifulSoup.
requests_kwargs (Optional[Dict[str, Any]]) – kwargs for requests
raise_for_status (bool) – Raise an exception if http status code denotes an error.
bs_get_text_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 get_text
bs_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 web page parsing
web_path (Union[str, Sequence[str]]) – 
header_template (Optional[dict]) – 
verify_ssl (bool) – 
proxies (Optional[dict]) – 
continue_on_failure (bool) – 
autoset_encoding (bool) – 
encoding (Optional[str]) – 
session (Any) – 
Attributes
web_path
Methods
session (Any) – 
Attributes
web_path
Methods
__init__([web_path, header_template, ...])
Initialize loader.
alazy_load()
A lazy loader for Documents.
aload()
Load text from the urls in web_path async into Documents.
fetch_all(urls)
Fetch all urls concurrently with rate limiting.
lazy_load()
Lazy load text from the url(s) in web_path.
load()
Load webpage.
load_and_split([text_splitter])
Load Documents and split into chunks.
scrape([parser])
Scrape data from webpage and return it in BeautifulSoup format.
scrape_all(urls[, parser])
Fetch all urls, then return soups for all results.
__init__(web_path: Union[str, Sequence[str]] = '', header_template: Optional[dict] = None, verify_ssl: bool = True, proxies: Optional[dict] = None, continue_on_failure: bool = False, autoset_encoding: bool = True, encoding: Optional[str] = None, web_paths: Sequence[str] = (), requests_per_second: int = 2, default_parser: str = 'html.parser', requests_kwargs: Optional[Dict[str, Any]] = None, raise_for_status: bool = False, bs_get_text_kwargs: Optional[Dict[str, Any]] = None, bs_kwargs: Optional[Dict[str, Any]] = None, session: Optional[Any] = None) → None¶
Initialize loader.
Parameters
web_paths (Sequence[str]) – Web paths to load from.
requests_per_second (int) – Max number of concurrent requests to make.
default_parser (str) – Default parser to use for BeautifulSoup.
requests_kwargs (Optional[Dict[str, Any]]) – kwargs for requests
raise_for_status (bool) – Raise an exception if http status code denotes an error.
raise_for_status (bool) – Raise an exception if http status code denotes an error.
bs_get_text_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 get_text
bs_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 web page parsing
web_path (Union[str, Sequence[str]]) – 
header_template (Optional[dict]) – 
verify_ssl (bool) – 
proxies (Optional[dict]) – 
continue_on_failure (bool) – 
autoset_encoding (bool) – 
encoding (Optional[str]) – 
session (Optional[Any]) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
aload() → List[Document]¶
Load text from the urls in web_path async into Documents.
Return type
List[Document]
async fetch_all(urls: List[str]) → Any¶
Fetch all urls concurrently with rate limiting.
Parameters
urls (List[str]) – 
Return type
Any
lazy_load() → Iterator[Document]¶
Lazy load text from the url(s) in web_path.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load webpage.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Returns
List of Documents.
Return type
List[Document]
scrape(parser: Optional[str] = None) → Any¶
Scrape data from webpage and return it in BeautifulSoup format.
Parameters
parser (Optional[str]) – 
Return type
Any
scrape_all(urls: List[str], parser: Optional[str] = None) → List[Any]¶
Fetch all urls, then return soups for all results.
Parameters
urls (List[str]) – 
parser (Optional[str]) – 
Return type
List[Any]
Examples using IMSDbLoader¶
IMSDb
langchain_community.document_loaders.email.OutlookMessageLoader¶
class langchain_community.document_loaders.email.OutlookMessageLoader(file_path: Union[str, Path])[source]¶
Loads Outlook Message files using extract_msg.
https://github.com/TeamMsgExtractor/msg-extractor
Initialize with a file path.
Parameters
file_path (Union[str, Path]) – The path to the Outlook Message file.
Methods
__init__(file_path)
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path])[source]¶
Initialize with a file path.
Parameters
file_path (Union[str, Path]) – The path to the Outlook Message file.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
langchain_community.document_loaders.psychic.PsychicLoader¶
class langchain_community.document_loaders.psychic.PsychicLoader(api_key: str, account_id: str, connector_id: Optional[str] = None)[source]¶
Load from Psychic.dev.
Initialize with API key, connector id, and account id.
Parameters
api_key (str) – The Psychic API key.
account_id (str) – The Psychic account id.
connector_id (Optional[str]) – The Psychic connector id.
Methods
__init__(api_key, account_id[, connector_id])
Initialize with API key, connector id, and account id.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(api_key: str, account_id: str, connector_id: Optional[str] = None)[source]¶
Initialize with API key, connector id, and account id.
Parameters
api_key (str) – The Psychic API key.
account_id (str) – The Psychic account id.
connector_id (Optional[str]) – The Psychic connector id.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using PsychicLoader¶
Psychic
langchain_community.document_loaders.pebblo.PebbloSafeLoader¶
class langchain_community.document_loaders.pebblo.PebbloSafeLoader(langchain_loader: BaseLoader, name: str, owner: str = '', description: str = '', api_key: Optional[str] = None, load_semantic: bool = False, classifier_url: Optional[str] = None)[source]¶
Pebblo Safe Loader class is a wrapper around document loaders enabling the data
to be scrutinized.
Methods
__init__(langchain_loader, name[, owner, ...])
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
calculate_content_size(page_content)
Calculate the content size in bytes: - Encode the string to bytes using a specific encoding (e.g., UTF-8) - Get the length of the encoded bytes.
get_file_owner_from_path(file_path)
Fetch owner of local file path.
get_source_size(source_path)
Fetch size of source path.
lazy_load()
Load documents in lazy fashion.
load()
Load Documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
set_discover_sent()
set_loader_sent()
Parameters
langchain_loader (BaseLoader) – 
name (str) – 
owner (str) – 
description (str) – 
api_key (Optional[str]) – 
load_semantic (bool) – 
classifier_url (Optional[str]) – 
__init__(langchain_loader: BaseLoader, name: str, owner: str = '', description: str = '', api_key: Optional[str] = None, load_semantic: bool = False, classifier_url: Optional[str] = None)[source]¶
Parameters
langchain_loader (BaseLoader) – 
name (str) –
Parameters
langchain_loader (BaseLoader) – 
name (str) – 
owner (str) – 
description (str) – 
api_key (Optional[str]) – 
load_semantic (bool) – 
classifier_url (Optional[str]) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
static calculate_content_size(page_content: str) → int[source]¶
Calculate the content size in bytes:
- Encode the string to bytes using a specific encoding (e.g., UTF-8)
- Get the length of the encoded bytes.
Parameters
page_content (str) – Data string.
Returns
Size of string in bytes.
Return type
int
static get_file_owner_from_path(file_path: str) → str[source]¶
Fetch owner of local file path.
Parameters
file_path (str) – Local file path.
Returns
Name of owner.
Return type
str
get_source_size(source_path: str) → int[source]¶
Fetch size of source path. Source can be a directory or a file.
Parameters
source_path (str) – Local path of data source.
Returns
Source size in bytes.
Return type
int
lazy_load() → Iterator[Document][source]¶
Load documents in lazy fashion.
Raises
NotImplementedError – raised when lazy_load id not implemented
within wrapped loader. – 
Yields
list – Documents from loader’s lazy loading.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load Documents.
Returns
Documents fetched from load method of the wrapped loader.
Return type
list
Returns
Documents fetched from load method of the wrapped loader.
Return type
list
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
classmethod set_discover_sent() → None[source]¶
Return type
None
classmethod set_loader_sent() → None[source]¶
Return type
None
Examples using PebbloSafeLoader¶
Pebblo Safe DocumentLoader
langchain_community.document_loaders.tsv.UnstructuredTSVLoader¶
class langchain_community.document_loaders.tsv.UnstructuredTSVLoader(file_path: Union[str, Path], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load TSV files using Unstructured.
Like other
Unstructured loaders, UnstructuredTSVLoader can be used in both
“single” and “elements” mode. If you use the loader in “elements”
mode, the TSV file will be a single Unstructured Table element.
If you use the loader in “elements” mode, an HTML representation
of the table will be available in the “text_as_html” key in the
document metadata.
Examples
from langchain_community.document_loaders.tsv import UnstructuredTSVLoader
loader = UnstructuredTSVLoader(“stanley-cups.tsv”, mode=”elements”)
docs = loader.load()
Initialize with file path.
Methods
__init__(file_path[, mode])
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (Union[str, Path]) – 
mode (str) – 
unstructured_kwargs (Any) – 
__init__(file_path: Union[str, Path], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Initialize with file path.
Parameters
file_path (Union[str, Path]) – 
mode (str) – 
unstructured_kwargs (Any) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using UnstructuredTSVLoader¶
TSV
Unstructured
langchain_community.document_loaders.parsers.txt.TextParser¶
class langchain_community.document_loaders.parsers.txt.TextParser[source]¶
Parser for text blobs.
Methods
__init__()
lazy_parse(blob)
Lazily parse the blob.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__()¶
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazily parse the blob.
Parameters
blob (Blob) – 
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
langchain_astradb.document_loaders.AstraDBLoader¶
class langchain_astradb.document_loaders.AstraDBLoader(collection_name: str, *, token: ~typing.Optional[str] = None, api_endpoint: ~typing.Optional[str] = None, astra_db_client: ~typing.Optional[~astrapy.core.db.AstraDB] = None, async_astra_db_client: ~typing.Optional[~astrapy.core.db.AsyncAstraDB] = None, namespace: ~typing.Optional[str] = None, filter_criteria: ~typing.Optional[~typing.Dict[str, ~typing.Any]] = None, projection: ~typing.Optional[~typing.Dict[str, ~typing.Any]] = <object object>, find_options: ~typing.Optional[~typing.Dict[str, ~typing.Any]] = None, nb_prefetched: int = 1000, page_content_mapper: ~typing.Callable[[~typing.Dict], str] = <function dumps>, metadata_mapper: ~typing.Optional[~typing.Callable[[~typing.Dict], ~typing.Dict[str, ~typing.Any]]] = None)[source]¶
Load DataStax Astra DB documents.
Parameters
collection_name (str) – name of the Astra DB collection to use.
token (Optional[str]) – API token for Astra DB usage. If not provided, the environment
variable ASTRA_DB_APPLICATION_TOKEN is inspected.
api_endpoint (Optional[str]) – full URL to the API endpoint, such as
https://<DB-ID>-us-east1.apps.astra.datastax.com. If not provided,
the environment variable ASTRA_DB_API_ENDPOINT is inspected.
astra_db_client (Optional[AstraDB]) – alternative to token+api_endpoint,
you can pass an already-created ‘astrapy.db.AstraDB’ instance.
async_astra_db_client (Optional[AsyncAstraDB]) – alternative to token+api_endpoint,
you can pass an already-created ‘astrapy.db.AsyncAstraDB’ instance.
namespace (Optional[str]) – namespace (aka keyspace) where the collection resides.
If not provided, the environment variable ASTRA_DB_KEYSPACE is
inspected. Defaults to the database’s “default namespace”.
filter_criteria (Optional[Dict[str, Any]]) – Criteria to filter documents.
projection (Optional[Dict[str, Any]]) – Specifies the fields to return. If not provided, reads
fall back to the Data API default projection.
find_options (Optional[Dict[str, Any]]) – Additional options for the query.
nb_prefetched (int) – Max number of documents to pre-fetch. Defaults to 1000.
page_content_mapper (Callable[[Dict], str]) – Function applied to collection documents to create
the page_content of the LangChain Document. Defaults to json.dumps.
metadata_mapper (Optional[Callable[[Dict], Dict[str, Any]]]) – 
Methods
__init__(collection_name, *[, token, ...])
Load DataStax Astra DB documents.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(collection_name: str, *, token: ~typing.Optional[str] = None, api_endpoint: ~typing.Optional[str] = None, astra_db_client: ~typing.Optional[~astrapy.core.db.AstraDB] = None, async_astra_db_client: ~typing.Optional[~astrapy.core.db.AsyncAstraDB] = None, namespace: ~typing.Optional[str] = None, filter_criteria: ~typing.Optional[~typing.Dict[str, ~typing.Any]] = None, projection: ~typing.Optional[~typing.Dict[str, ~typing.Any]] = <object object>, find_options: ~typing.Optional[~typing.Dict[str, ~typing.Any]] = None, nb_prefetched: int = 1000, page_content_mapper: ~typing.Callable[[~typing.Dict], str] = <function dumps>, metadata_mapper: ~typing.Optional[~typing.Callable[[~typing.Dict], ~typing.Dict[str, ~typing.Any]]] = None) → None[source]¶
Load DataStax Astra DB documents.
Parameters
collection_name (str) – name of the Astra DB collection to use.
token (Optional[str]) – API token for Astra DB usage. If not provided, the environment
variable ASTRA_DB_APPLICATION_TOKEN is inspected.
api_endpoint (Optional[str]) – full URL to the API endpoint, such as
https://<DB-ID>-us-east1.apps.astra.datastax.com. If not provided,
the environment variable ASTRA_DB_API_ENDPOINT is inspected.
astra_db_client (Optional[AstraDB]) – alternative to token+api_endpoint,
you can pass an already-created ‘astrapy.db.AstraDB’ instance.
async_astra_db_client (Optional[AsyncAstraDB]) – alternative to token+api_endpoint,