Tony20100/langchain_full huggingface dataset:
of the root url.
link_regex (Union[str, re.Pattern, None]) – Regex for extracting sub-links from the raw html of a web page.
headers (Optional[dict]) – Default request headers to use for all requests.
check_response_status (bool) – If True, check HTTP response status and skip
URLs with error responses (400-599).
continue_on_failure (bool) – If True, continue if getting or parsing a link raises
an exception. Otherwise, raise the exception.
base_url (Optional[str]) – The base url to check for outside links against.
autoset_encoding (bool) – Whether to automatically set the encoding of the response.
If True, the encoding of the response will be set to the apparent
encoding, unless the encoding argument has already been explicitly set.
encoding (Optional[str]) – The encoding of the response. If manually set, the encoding will be
set to given value, regardless of the autoset_encoding argument.
Methods
__init__(url[, max_depth, use_async, ...])
Initialize with URL to crawl and any subdirectories to exclude.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load web pages.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(url: str, max_depth: Optional[int] = 2, use_async: Optional[bool] = None, extractor: Optional[Callable[[str], str]] = None, metadata_extractor: Optional[Union[Callable[[str, str], dict], Callable[[str, str, Union[Response, ClientResponse]], dict]]] = None, exclude_dirs: Optional[Sequence[str]] = (), timeout: Optional[int] = 10, prevent_outside: bool = True, link_regex: Optional[Union[str, Pattern]] = None, headers: Optional[dict] = None, check_response_status: bool = False, continue_on_failure: bool = True, *, base_url: Optional[str] = None, autoset_encoding: bool = True, encoding: Optional[str] = None) → None[source]¶
Initialize with URL to crawl and any subdirectories to exclude.
Parameters
url (str) – The URL to crawl.
max_depth (Optional[int]) – The max depth of the recursive loading.
use_async (Optional[bool]) – Whether to use asynchronous loading.
If True, this function will not be lazy, but it will still work in the
expected way, just not lazy.
extractor (Optional[Callable[[str], str]]) – A function to extract document contents from raw html.
When extract function returns an empty string, the document is
ignored.
metadata_extractor (Optional[Union[Callable[[str, str], dict], Callable[[str, str, Union[Response, ClientResponse]], dict]]]) – A function to extract metadata from args: raw html, the
source url, and the requests.Response/aiohttp.ClientResponse object
(args in that order).
Default extractor will attempt to use BeautifulSoup4 to extract the
title, description and language of the page.
title, description and language of the page.
..code-block:: python
import requests
import aiohttp
def simple_metadata_extractor(raw_html: str, url: str, response: Union[requests.Response, aiohttp.ClientResponse]
) -> dict:content_type = getattr(response, “headers”).get(“Content-Type”, “”)
return {“source”: url, “content_type”: content_type}
exclude_dirs (Optional[Sequence[str]]) – A list of subdirectories to exclude.
timeout (Optional[int]) – The timeout for the requests, in the unit of seconds. If None then
connection will not timeout.
prevent_outside (bool) – If True, prevent loading from urls which are not children
of the root url.
link_regex (Optional[Union[str, Pattern]]) – Regex for extracting sub-links from the raw html of a web page.
headers (Optional[dict]) – Default request headers to use for all requests.
check_response_status (bool) – If True, check HTTP response status and skip
URLs with error responses (400-599).
continue_on_failure (bool) – If True, continue if getting or parsing a link raises
an exception. Otherwise, raise the exception.
base_url (Optional[str]) – The base url to check for outside links against.
autoset_encoding (bool) – Whether to automatically set the encoding of the response.
If True, the encoding of the response will be set to the apparent
encoding, unless the encoding argument has already been explicitly set.
encoding (Optional[str]) – The encoding of the response. If manually set, the encoding will be
set to given value, regardless of the autoset_encoding argument.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load web pages.
When use_async is True, this function will not be lazy,
but it will still work in the expected way, just not lazy.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using RecursiveUrlLoader¶
Recursive URL
langchain_community.document_loaders.pdf.PDFMinerLoader¶
class langchain_community.document_loaders.pdf.PDFMinerLoader(file_path: str, *, headers: Optional[Dict] = None, extract_images: bool = False, concatenate_pages: bool = True)[source]¶
Load PDF files using PDFMiner.
Initialize with file path.
Parameters
extract_images (bool) – Whether to extract images from PDF.
concatenate_pages (bool) – If True, concatenate all PDF pages into one a single
document. Otherwise, return one document per page.
file_path (str) – 
headers (Optional[Dict]) – 
Attributes
source
Methods
__init__(file_path, *[, headers, ...])
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazily load documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, *, headers: Optional[Dict] = None, extract_images: bool = False, concatenate_pages: bool = True) → None[source]¶
Initialize with file path.
Parameters
extract_images (bool) – Whether to extract images from PDF.
concatenate_pages (bool) – If True, concatenate all PDF pages into one a single
document. Otherwise, return one document per page.
file_path (str) – 
headers (Optional[Dict]) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazily load documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using PDFMinerLoader¶
How to load PDFs
langchain_community.document_loaders.youtube.GoogleApiYoutubeLoader¶
class langchain_community.document_loaders.youtube.GoogleApiYoutubeLoader(google_api_client: GoogleApiClient, channel_name: Optional[str] = None, video_ids: Optional[List[str]] = None, add_video_info: bool = True, captions_language: str = 'en', continue_on_failure: bool = False)[source]¶
Load all Videos from a YouTube Channel.
To use, you should have the googleapiclient,youtube_transcript_api
python package installed.
As the service needs a google_api_client, you first have to initialize
the GoogleApiClient.
Additionally you have to either provide a channel name or a list of videoids
“https://developers.google.com/docs/api/quickstart/python”
Example
from langchain_community.document_loaders import GoogleApiClient
from langchain_community.document_loaders import GoogleApiYoutubeLoader
google_api_client = GoogleApiClient(
    service_account_path=Path("path_to_your_sec_file.json")
)
loader = GoogleApiYoutubeLoader(
    google_api_client=google_api_client,
    channel_name = "CodeAesthetic"
)
load.load()
Attributes
add_video_info
captions_language
channel_name
continue_on_failure
video_ids
google_api_client
Methods
__init__(google_api_client[, channel_name, ...])
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
validate_channel_or_videoIds_is_set(values)
Validate that either folder_id or document_ids is set, but not both.
Parameters
google_api_client (GoogleApiClient) – 
channel_name (Optional[str]) –
google_api_client (GoogleApiClient) – 
channel_name (Optional[str]) – 
video_ids (Optional[List[str]]) – 
add_video_info (bool) – 
captions_language (str) – 
continue_on_failure (bool) – 
__init__(google_api_client: GoogleApiClient, channel_name: Optional[str] = None, video_ids: Optional[List[str]] = None, add_video_info: bool = True, captions_language: str = 'en', continue_on_failure: bool = False) → None¶
Parameters
google_api_client (GoogleApiClient) – 
channel_name (Optional[str]) – 
video_ids (Optional[List[str]]) – 
add_video_info (bool) – 
captions_language (str) – 
continue_on_failure (bool) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Returns
List of Documents.
Return type
List[Document]
classmethod validate_channel_or_videoIds_is_set(values: Dict[str, Any]) → Dict[str, Any][source]¶
Validate that either folder_id or document_ids is set, but not both.
Parameters
values (Dict[str, Any]) – 
Return type
Dict[str, Any]
Examples using GoogleApiYoutubeLoader¶
YouTube
YouTube transcripts
langchain_community.document_loaders.html.UnstructuredHTMLLoader¶
class langchain_community.document_loaders.html.UnstructuredHTMLLoader(file_path: Optional[Union[str, List[str], Path, List[Path]]], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load HTML files using Unstructured.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain_community.document_loaders import UnstructuredHTMLLoader
loader = UnstructuredHTMLLoader(“example.html”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-html
Initialize with file path.
Methods
__init__(file_path[, mode])
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (Optional[Union[str, List[str], Path, List[Path]]]) – 
mode (str) – 
unstructured_kwargs (Any) – 
__init__(file_path: Optional[Union[str, List[str], Path, List[Path]]], mode: str = 'single', **unstructured_kwargs: Any)¶
Initialize with file path.
Parameters
Initialize with file path.
Parameters
file_path (Optional[Union[str, List[str], Path, List[Path]]]) – 
mode (str) – 
unstructured_kwargs (Any) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using UnstructuredHTMLLoader¶
How to load HTML
Unstructured
langchain_community.document_loaders.git.GitLoader¶
class langchain_community.document_loaders.git.GitLoader(repo_path: str, clone_url: Optional[str] = None, branch: Optional[str] = 'main', file_filter: Optional[Callable[[str], bool]] = None)[source]¶
Load Git repository files.
The Repository can be local on disk available at repo_path,
or remote at clone_url that will be cloned to repo_path.
Currently, supports only text files.
Each document represents one file in the repository. The path points to
the local Git repository, and the branch specifies the branch to load
files from. By default, it loads from the main branch.
Parameters
repo_path (str) – The path to the Git repository.
clone_url (Optional[str]) – Optional. The URL to clone the repository from.
branch (Optional[str]) – Optional. The branch to load files from. Defaults to main.
file_filter (Optional[Callable[[str], bool]]) – Optional. A function that takes a file path and returns
a boolean indicating whether to load the file. Defaults to None.
Methods
__init__(repo_path[, clone_url, branch, ...])
param repo_path
The path to the Git repository.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(repo_path: str, clone_url: Optional[str] = None, branch: Optional[str] = 'main', file_filter: Optional[Callable[[str], bool]] = None)[source]¶
Parameters
repo_path (str) – The path to the Git repository.
Parameters
repo_path (str) – The path to the Git repository.
clone_url (Optional[str]) – Optional. The URL to clone the repository from.
branch (Optional[str]) – Optional. The branch to load files from. Defaults to main.
file_filter (Optional[Callable[[str], bool]]) – Optional. A function that takes a file path and returns
a boolean indicating whether to load the file. Defaults to None.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using GitLoader¶
Git
langchain_community.document_loaders.notebook.NotebookLoader¶
class langchain_community.document_loaders.notebook.NotebookLoader(path: Union[str, Path], include_outputs: bool = False, max_output_length: int = 10, remove_newline: bool = False, traceback: bool = False)[source]¶
Load Jupyter notebook (.ipynb) files.
Initialize with a path.
Parameters
path (Union[str, Path]) – The path to load the notebook from.
include_outputs (bool) – Whether to include the outputs of the cell.
Defaults to False.
max_output_length (int) – Maximum length of the output to be displayed.
Defaults to 10.
remove_newline (bool) – Whether to remove newlines from the notebook.
Defaults to False.
traceback (bool) – Whether to return a traceback of the error.
Defaults to False.
Methods
__init__(path[, include_outputs, ...])
Initialize with a path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(path: Union[str, Path], include_outputs: bool = False, max_output_length: int = 10, remove_newline: bool = False, traceback: bool = False)[source]¶
Initialize with a path.
Parameters
path (Union[str, Path]) – The path to load the notebook from.
include_outputs (bool) – Whether to include the outputs of the cell.
Defaults to False.
max_output_length (int) – Maximum length of the output to be displayed.
Defaults to 10.
remove_newline (bool) – Whether to remove newlines from the notebook.
remove_newline (bool) – Whether to remove newlines from the notebook.
Defaults to False.
traceback (bool) – Whether to return a traceback of the error.
Defaults to False.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using NotebookLoader¶
Jupyter Notebook
langchain_community.document_loaders.epub.UnstructuredEPubLoader¶
class langchain_community.document_loaders.epub.UnstructuredEPubLoader(file_path: Optional[Union[str, List[str], Path, List[Path]]], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load EPub files using Unstructured.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain_community.document_loaders import UnstructuredEPubLoader
loader = UnstructuredEPubLoader(“example.epub”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-epub
Initialize with file path.
Methods
__init__(file_path[, mode])
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (Optional[Union[str, List[str], Path, List[Path]]]) – 
mode (str) – 
unstructured_kwargs (Any) – 
__init__(file_path: Optional[Union[str, List[str], Path, List[Path]]], mode: str = 'single', **unstructured_kwargs: Any)¶
Initialize with file path.
Parameters
Initialize with file path.
Parameters
file_path (Optional[Union[str, List[str], Path, List[Path]]]) – 
mode (str) – 
unstructured_kwargs (Any) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using UnstructuredEPubLoader¶
EPub 
Unstructured
langchain_community.document_loaders.open_city_data.OpenCityDataLoader¶
class langchain_community.document_loaders.open_city_data.OpenCityDataLoader(city_id: str, dataset_id: str, limit: int)[source]¶
Load from Open City.
Initialize with dataset_id.
Example: https://dev.socrata.com/foundry/data.sfgov.org/vw6y-z8j6
e.g., city_id = data.sfgov.org
e.g., dataset_id = vw6y-z8j6
Parameters
city_id (str) – The Open City city identifier.
dataset_id (str) – The Open City dataset identifier.
limit (int) – The maximum number of documents to load.
Methods
__init__(city_id, dataset_id, limit)
Initialize with dataset_id.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load records.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(city_id: str, dataset_id: str, limit: int)[source]¶
Initialize with dataset_id.
Example: https://dev.socrata.com/foundry/data.sfgov.org/vw6y-z8j6
e.g., city_id = data.sfgov.org
e.g., dataset_id = vw6y-z8j6
Parameters
city_id (str) – The Open City city identifier.
dataset_id (str) – The Open City dataset identifier.
limit (int) – The maximum number of documents to load.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load records.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using OpenCityDataLoader¶
Geopandas
Open City Data
langchain_community.document_loaders.parsers.language.ruby.RubySegmenter¶
class langchain_community.document_loaders.parsers.language.ruby.RubySegmenter(code: str)[source]¶
Code segmenter for Ruby.
Methods
__init__(code)
extract_functions_classes()
get_chunk_query()
get_language()
get_parser()
is_valid()
make_line_comment(text)
simplify_code()
Parameters
code (str) – 
__init__(code: str)¶
Parameters
code (str) – 
extract_functions_classes() → List[str]¶
Return type
List[str]
get_chunk_query() → str[source]¶
Return type
str
get_language() → Language[source]¶
Return type
Language
get_parser() → Parser¶
Return type
Parser
is_valid() → bool¶
Return type
bool
make_line_comment(text: str) → str[source]¶
Parameters
text (str) – 
Return type
str
simplify_code() → str¶
Return type
str