Tony20100/langchain_full huggingface dataset:
langchain_community.document_loaders.azure_ai_data.AzureAIDataLoader¶
class langchain_community.document_loaders.azure_ai_data.AzureAIDataLoader(url: str, glob: Optional[str] = None)[source]¶
Load from Azure AI Data.
Initialize with URL to a data asset or storage location
.
Attributes
url
URL to the data asset or storage location.
glob_pattern
Optional glob pattern to select files.
Methods
__init__(url[, glob])
Initialize with URL to a data asset or storage location
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
url (str) – 
glob (Optional[str]) – 
__init__(url: str, glob: Optional[str] = None)[source]¶
Initialize with URL to a data asset or storage location
.
Parameters
url (str) – 
glob (Optional[str]) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
langchain_community.document_loaders.directory.DirectoryLoader¶
class langchain_community.document_loaders.directory.DirectoryLoader(path: str, glob: str = '**/[!.]*', silent_errors: bool = False, load_hidden: bool = False, loader_cls: ~typing.Union[~typing.Type[~langchain_community.document_loaders.unstructured.UnstructuredFileLoader], ~typing.Type[~langchain_community.document_loaders.text.TextLoader], ~typing.Type[~langchain_community.document_loaders.html_bs.BSHTMLLoader], ~typing.Type[~langchain_community.document_loaders.csv_loader.CSVLoader]] = <class 'langchain_community.document_loaders.unstructured.UnstructuredFileLoader'>, loader_kwargs: ~typing.Optional[dict] = None, recursive: bool = False, show_progress: bool = False, use_multithreading: bool = False, max_concurrency: int = 4, *, exclude: ~typing.Union[~typing.Sequence[str], str] = (), sample_size: int = 0, randomize_sample: bool = False, sample_seed: ~typing.Optional[int] = None)[source]¶
Load from a directory.
Initialize with a path to directory and how to glob over it.
Parameters
path (str) – Path to directory.
glob (str) – Glob pattern to use to find files. Defaults to “**/[!.]*”
(all files except hidden).
exclude (Union[Sequence[str], str]) – A pattern or list of patterns to exclude from results.
Use glob syntax.
silent_errors (bool) – Whether to silently ignore errors. Defaults to False.
load_hidden (bool) – Whether to load hidden files. Defaults to False.
loader_cls (Union[Type[UnstructuredFileLoader], Type[TextLoader], Type[BSHTMLLoader], Type[CSVLoader]]) – Loader class to use for loading files.
Defaults to UnstructuredFileLoader.
Defaults to UnstructuredFileLoader.
loader_kwargs (Optional[dict]) – Keyword arguments to pass to loader_cls. Defaults to None.
recursive (bool) – Whether to recursively search for files. Defaults to False.
show_progress (bool) – Whether to show a progress bar. Defaults to False.
use_multithreading (bool) – Whether to use multithreading. Defaults to False.
max_concurrency (int) – The maximum number of threads to use. Defaults to 4.
sample_size (int) – The maximum number of files you would like to load from the
directory.
randomize_sample (bool) – Shuffle the files to get a random sample.
sample_seed (Optional[int]) – set the seed of the random shuffle for reproducibility.
Examples
Methods
__init__(path[, glob, silent_errors, ...])
Initialize with a path to directory and how to glob over it.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load documents lazily.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(path: str, glob: str = '**/[!.]*', silent_errors: bool = False, load_hidden: bool = False, loader_cls: ~typing.Union[~typing.Type[~langchain_community.document_loaders.unstructured.UnstructuredFileLoader], ~typing.Type[~langchain_community.document_loaders.text.TextLoader], ~typing.Type[~langchain_community.document_loaders.html_bs.BSHTMLLoader], ~typing.Type[~langchain_community.document_loaders.csv_loader.CSVLoader]] = <class 'langchain_community.document_loaders.unstructured.UnstructuredFileLoader'>, loader_kwargs: ~typing.Optional[dict] = None, recursive: bool = False, show_progress: bool = False, use_multithreading: bool = False, max_concurrency: int = 4, *, exclude: ~typing.Union[~typing.Sequence[str], str] = (), sample_size: int = 0, randomize_sample: bool = False, sample_seed: ~typing.Optional[int] = None)[source]¶
Initialize with a path to directory and how to glob over it.
Parameters
path (str) – Path to directory.
glob (str) – Glob pattern to use to find files. Defaults to “**/[!.]*”
(all files except hidden).
exclude (Union[Sequence[str], str]) – A pattern or list of patterns to exclude from results.
Use glob syntax.
silent_errors (bool) – Whether to silently ignore errors. Defaults to False.
load_hidden (bool) – Whether to load hidden files. Defaults to False.
loader_cls (Union[Type[UnstructuredFileLoader], Type[TextLoader], Type[BSHTMLLoader], Type[CSVLoader]]) – Loader class to use for loading files.
Defaults to UnstructuredFileLoader.
Defaults to UnstructuredFileLoader.
loader_kwargs (Optional[dict]) – Keyword arguments to pass to loader_cls. Defaults to None.
recursive (bool) – Whether to recursively search for files. Defaults to False.
show_progress (bool) – Whether to show a progress bar. Defaults to False.
use_multithreading (bool) – Whether to use multithreading. Defaults to False.
max_concurrency (int) – The maximum number of threads to use. Defaults to 4.
sample_size (int) – The maximum number of files you would like to load from the
directory.
randomize_sample (bool) – Shuffle the files to get a random sample.
sample_seed (Optional[int]) – set the seed of the random shuffle for reproducibility.
Examples
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load documents lazily.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using DirectoryLoader¶
Apache Doris
Azure AI Search
How to load documents from a directory
StarRocks
langchain_community.document_loaders.hn.HNLoader¶
class langchain_community.document_loaders.hn.HNLoader(web_path: Union[str, Sequence[str]] = '', header_template: Optional[dict] = None, verify_ssl: bool = True, proxies: Optional[dict] = None, continue_on_failure: bool = False, autoset_encoding: bool = True, encoding: Optional[str] = None, web_paths: Sequence[str] = (), requests_per_second: int = 2, default_parser: str = 'html.parser', requests_kwargs: Optional[Dict[str, Any]] = None, raise_for_status: bool = False, bs_get_text_kwargs: Optional[Dict[str, Any]] = None, bs_kwargs: Optional[Dict[str, Any]] = None, session: Optional[Any] = None)[source]¶
Load Hacker News data.
It loads data from either main page results or the comments page.
Initialize loader.
Parameters
web_paths (Sequence[str]) – Web paths to load from.
requests_per_second (int) – Max number of concurrent requests to make.
default_parser (str) – Default parser to use for BeautifulSoup.
requests_kwargs (Optional[Dict[str, Any]]) – kwargs for requests
raise_for_status (bool) – Raise an exception if http status code denotes an error.
bs_get_text_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 get_text
bs_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 web page parsing
web_path (Union[str, Sequence[str]]) – 
header_template (Optional[dict]) – 
verify_ssl (bool) – 
proxies (Optional[dict]) – 
continue_on_failure (bool) – 
autoset_encoding (bool) – 
encoding (Optional[str]) – 
session (Any) – 
Attributes
web_path
Methods
session (Any) – 
Attributes
web_path
Methods
__init__([web_path, header_template, ...])
Initialize loader.
alazy_load()
A lazy loader for Documents.
aload()
Load text from the urls in web_path async into Documents.
fetch_all(urls)
Fetch all urls concurrently with rate limiting.
lazy_load()
Lazy load text from the url(s) in web_path.
load()
Get important HN webpage information.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_comments(soup_info)
Load comments from a HN post.
load_results(soup)
Load items from an HN page.
scrape([parser])
Scrape data from webpage and return it in BeautifulSoup format.
scrape_all(urls[, parser])
Fetch all urls, then return soups for all results.
__init__(web_path: Union[str, Sequence[str]] = '', header_template: Optional[dict] = None, verify_ssl: bool = True, proxies: Optional[dict] = None, continue_on_failure: bool = False, autoset_encoding: bool = True, encoding: Optional[str] = None, web_paths: Sequence[str] = (), requests_per_second: int = 2, default_parser: str = 'html.parser', requests_kwargs: Optional[Dict[str, Any]] = None, raise_for_status: bool = False, bs_get_text_kwargs: Optional[Dict[str, Any]] = None, bs_kwargs: Optional[Dict[str, Any]] = None, session: Optional[Any] = None) → None¶
Initialize loader.
Parameters
web_paths (Sequence[str]) – Web paths to load from.
requests_per_second (int) – Max number of concurrent requests to make.
default_parser (str) – Default parser to use for BeautifulSoup.
default_parser (str) – Default parser to use for BeautifulSoup.
requests_kwargs (Optional[Dict[str, Any]]) – kwargs for requests
raise_for_status (bool) – Raise an exception if http status code denotes an error.
bs_get_text_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 get_text
bs_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 web page parsing
web_path (Union[str, Sequence[str]]) – 
header_template (Optional[dict]) – 
verify_ssl (bool) – 
proxies (Optional[dict]) – 
continue_on_failure (bool) – 
autoset_encoding (bool) – 
encoding (Optional[str]) – 
session (Optional[Any]) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
aload() → List[Document]¶
Load text from the urls in web_path async into Documents.
Return type
List[Document]
async fetch_all(urls: List[str]) → Any¶
Fetch all urls concurrently with rate limiting.
Parameters
urls (List[str]) – 
Return type
Any
lazy_load() → Iterator[Document]¶
Lazy load text from the url(s) in web_path.
Return type
Iterator[Document]
load() → List[Document][source]¶
Get important HN webpage information.
HN webpage components are:
title
content
source url,
time of post
author of the post
number of comments
rank of the post
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
load_comments(soup_info: Any) → List[Document][source]¶
Load comments from a HN post.
Parameters
soup_info (Any) – 
Return type
List[Document]
load_results(soup: Any) → List[Document][source]¶
Load items from an HN page.
Parameters
soup (Any) – 
Return type
List[Document]
scrape(parser: Optional[str] = None) → Any¶
Scrape data from webpage and return it in BeautifulSoup format.
Parameters
parser (Optional[str]) – 
Return type
Any
scrape_all(urls: List[str], parser: Optional[str] = None) → List[Any]¶
Fetch all urls, then return soups for all results.
Parameters
urls (List[str]) – 
parser (Optional[str]) – 
Return type
List[Any]
Examples using HNLoader¶
Google Spanner
Hacker News
langchain_community.document_loaders.chromium.AsyncChromiumLoader¶
class langchain_community.document_loaders.chromium.AsyncChromiumLoader(urls: List[str], *, headless: bool = True, user_agent: Optional[str] = None)[source]¶
Scrape HTML pages from URLs using a
headless instance of the Chromium.
Initialize the loader with a list of URL paths.
Parameters
urls (List[str]) – A list of URLs to scrape content from.
headless (bool) – Whether to run browser in headless mode.
user_agent (Optional[str]) – The user agent to use for the browser
Raises
ImportError – If the required ‘playwright’ package is not installed.
Methods
__init__(urls, *[, headless, user_agent])
Initialize the loader with a list of URL paths.
alazy_load()
Asynchronously load text content from the provided URLs.
aload()
Load data into Document objects.
ascrape_playwright(url)
Asynchronously scrape the content of a given URL using Playwright's async API.
lazy_load()
Lazily load text content from the provided URLs.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(urls: List[str], *, headless: bool = True, user_agent: Optional[str] = None)[source]¶
Initialize the loader with a list of URL paths.
Parameters
urls (List[str]) – A list of URLs to scrape content from.
headless (bool) – Whether to run browser in headless mode.
user_agent (Optional[str]) – The user agent to use for the browser
Raises
ImportError – If the required ‘playwright’ package is not installed.
async alazy_load() → AsyncIterator[Document][source]¶
async alazy_load() → AsyncIterator[Document][source]¶
Asynchronously load text content from the provided URLs.
This method leverages asyncio to initiate the scraping of all provided URLs
simultaneously. It improves performance by utilizing concurrent asynchronous
requests. Each Document is yielded as soon as its content is available,
encapsulating the scraped content.
Yields
Document – A Document object containing the scraped content, along with its
source URL as metadata.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
async ascrape_playwright(url: str) → str[source]¶
Asynchronously scrape the content of a given URL using Playwright’s async API.
Parameters
url (str) – The URL to scrape.
Returns
The scraped HTML content or an error message if an exception occurs.
Return type
str
lazy_load() → Iterator[Document][source]¶
Lazily load text content from the provided URLs.
This method yields Documents one at a time as they’re scraped,
instead of waiting to scrape all URLs before returning.
Yields
Document – The scraped content encapsulated within a Document object.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
langchain_community.document_loaders.hugging_face_dataset.HuggingFaceDatasetLoader¶
class langchain_community.document_loaders.hugging_face_dataset.HuggingFaceDatasetLoader(path: str, page_content_column: str = 'text', name: Optional[str] = None, data_dir: Optional[str] = None, data_files: Optional[Union[str, Sequence[str], Mapping[str, Union[str, Sequence[str]]]]] = None, cache_dir: Optional[str] = None, keep_in_memory: Optional[bool] = None, save_infos: bool = False, use_auth_token: Optional[Union[bool, str]] = None, num_proc: Optional[int] = None)[source]¶
Load from Hugging Face Hub datasets.
Initialize the HuggingFaceDatasetLoader.
Parameters
path (str) – Path or name of the dataset.
page_content_column (str) – Page content column name. Default is “text”.
name (Optional[str]) – Name of the dataset configuration.
data_dir (Optional[str]) – Data directory of the dataset configuration.
data_files (Optional[Union[str, Sequence[str], Mapping[str, Union[str, Sequence[str]]]]]) – Path(s) to source data file(s).
cache_dir (Optional[str]) – Directory to read/write data.
keep_in_memory (Optional[bool]) – Whether to copy the dataset in-memory.
save_infos (bool) – Save the dataset information (checksums/size/splits/…).
Default is False.
use_auth_token (Optional[Union[bool, str]]) – Bearer token for remote files on the Dataset Hub.
num_proc (Optional[int]) – Number of processes.
Methods
__init__(path[, page_content_column, name, ...])
Initialize the HuggingFaceDatasetLoader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load documents lazily.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
parse_obj(page_content)
__init__(path: str, page_content_column: str = 'text', name: Optional[str] = None, data_dir: Optional[str] = None, data_files: Optional[Union[str, Sequence[str], Mapping[str, Union[str, Sequence[str]]]]] = None, cache_dir: Optional[str] = None, keep_in_memory: Optional[bool] = None, save_infos: bool = False, use_auth_token: Optional[Union[bool, str]] = None, num_proc: Optional[int] = None)[source]¶
Initialize the HuggingFaceDatasetLoader.
Parameters
path (str) – Path or name of the dataset.
page_content_column (str) – Page content column name. Default is “text”.
name (Optional[str]) – Name of the dataset configuration.
data_dir (Optional[str]) – Data directory of the dataset configuration.
data_files (Optional[Union[str, Sequence[str], Mapping[str, Union[str, Sequence[str]]]]]) – Path(s) to source data file(s).
cache_dir (Optional[str]) – Directory to read/write data.
keep_in_memory (Optional[bool]) – Whether to copy the dataset in-memory.
save_infos (bool) – Save the dataset information (checksums/size/splits/…).
Default is False.
use_auth_token (Optional[Union[bool, str]]) – Bearer token for remote files on the Dataset Hub.
num_proc (Optional[int]) – Number of processes.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load documents lazily.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
parse_obj(page_content: Union[str, object]) → str[source]¶
Parameters
page_content (Union[str, object]) – 
Return type
str
Examples using HuggingFaceDatasetLoader¶
Hugging Face
HuggingFace dataset
langchain_community.document_loaders.assemblyai.TranscriptFormat¶
class langchain_community.document_loaders.assemblyai.TranscriptFormat(value)[source]¶
Transcript format to use for the document loader.
TEXT = 'text'¶
One document with the transcription text
SENTENCES = 'sentences'¶
Multiple documents, splits the transcription by each sentence
PARAGRAPHS = 'paragraphs'¶
Multiple documents, splits the transcription by each paragraph
SUBTITLES_SRT = 'subtitles_srt'¶
One document with the transcript exported in SRT subtitles format
SUBTITLES_VTT = 'subtitles_vtt'¶
One document with the transcript exported in VTT subtitles format
Examples using TranscriptFormat¶
AssemblyAI Audio Transcripts
langchain_community.document_loaders.s3_directory.S3DirectoryLoader¶
class langchain_community.document_loaders.s3_directory.S3DirectoryLoader(bucket: str, prefix: str = '', *, region_name: Optional[str] = None, api_version: Optional[str] = None, use_ssl: Optional[bool] = True, verify: Union[str, bool, None] = None, endpoint_url: Optional[str] = None, aws_access_key_id: Optional[str] = None, aws_secret_access_key: Optional[str] = None, aws_session_token: Optional[str] = None, boto_config: Optional[botocore.client.Config] = None)[source]¶
Load from Amazon AWS S3 directory.
Initialize with bucket and key name.
Parameters
bucket (str) – The name of the S3 bucket.
prefix (str) – The prefix of the S3 key. Defaults to “”.
region_name (Optional[str]) – The name of the region associated with the client.
A client is associated with a single region.
api_version (Optional[str]) – The API version to use.  By default, botocore will
use the latest API version when creating a client.  You only need
to specify this parameter if you want to use a previous API version
of the client.
use_ssl (Optional[bool]) – Whether to use SSL.  By default, SSL is used.
Note that not all services support non-ssl connections.
verify (Union[str, bool, None]) – Whether to verify SSL certificates.
By default SSL certificates are verified.  You can provide the
following values:
False - do not validate SSL certificates.  SSL will still be
used (unless use_ssl is False), but SSL certificates
will not be verified.
path/to/cert/bundle.pem - A filename of the CA cert bundle to
path/to/cert/bundle.pem - A filename of the CA cert bundle to
uses.  You can specify this argument if you want to use a
different CA cert bundle than the one used by botocore.
endpoint_url (Optional[str]) – The complete URL to use for the constructed
client.  Normally, botocore will automatically construct the
appropriate URL to use when communicating with a service.  You can
specify a complete URL (including the “http/https” scheme) to
override this behavior.  If this value is provided, then
use_ssl is ignored.
aws_access_key_id (Optional[str]) – The access key to use when creating
the client.  This is entirely optional, and if not provided,
the credentials configured for the session will automatically
be used.  You only need to provide this argument if you want
to override the credentials used for this specific client.
aws_secret_access_key (Optional[str]) – The secret key to use when creating
the client.  Same semantics as aws_access_key_id above.
aws_session_token (Optional[str]) – The session token to use when creating
the client.  Same semantics as aws_access_key_id above.
boto_config (botocore.client.Config) – Advanced boto3 client configuration options. If a value
is specified in the client config, its value will take precedence
over environment variables and configuration values, but not over
a value passed explicitly to the method. If a default config
object is set on the session, the config object used when creating
the client will be the result of calling merge() on the
default config with the config provided to this call.
Methods
__init__(bucket[, prefix, region_name, ...])
Initialize with bucket and key name.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.