Tony20100/langchain_full huggingface dataset:
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(query: str, database: str, s3_output_uri: str, profile_name: str, metadata_columns: Optional[List[str]] = None)[source]¶
Initialize Athena document loader.
Parameters
query (str) – The query to run in Athena.
database (str) – Athena database
s3_output_uri (str) – Athena output path
metadata_columns (Optional[List[str]]) – Optional. Columns written to Document metadata.
profile_name (str) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using AthenaLoader¶
AWS
Athena
langchain_community.document_loaders.glue_catalog.GlueCatalogLoader¶
class langchain_community.document_loaders.glue_catalog.GlueCatalogLoader(database: str, *, session: Optional[Session] = None, profile_name: Optional[str] = None, table_filter: Optional[List[str]] = None)[source]¶
Load table schemas from AWS Glue.
This loader fetches the schema of each table within a specified AWS Glue database.
The schema details include column names and their data types, similar to pandas
dtype representation.
AWS credentials are automatically loaded using boto3, following the standard AWS
method:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
If a specific AWS profile is required, it can be specified and will be used to
establish the session.
Initialize Glue database loader.
Parameters
database (str) – The name of the Glue database from which to load table schemas.
session (Optional[Session]) – Optional. A boto3 Session object. If not provided, a new
session will be created.
profile_name (Optional[str]) – Optional. The name of the AWS profile to use for credentials.
table_filter (Optional[List[str]]) – Optional. List of table names to fetch schemas for,
fetching all if None.
Methods
__init__(database, *[, session, ...])
Initialize Glue database loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazily load table schemas as Document objects.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(database: str, *, session: Optional[Session] = None, profile_name: Optional[str] = None, table_filter: Optional[List[str]] = None)[source]¶
Initialize Glue database loader.
Parameters
database (str) – The name of the Glue database from which to load table schemas.
session (Optional[Session]) – Optional. A boto3 Session object. If not provided, a new
session will be created.
profile_name (Optional[str]) – Optional. The name of the AWS profile to use for credentials.
table_filter (Optional[List[str]]) – Optional. List of table names to fetch schemas for,
fetching all if None.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazily load table schemas as Document objects.
Yields
Document objects, each representing the schema of a table.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using GlueCatalogLoader¶
Glue Catalog
langchain_community.document_loaders.url_playwright.PlaywrightURLLoader¶
class langchain_community.document_loaders.url_playwright.PlaywrightURLLoader(urls: List[str], continue_on_failure: bool = True, headless: bool = True, remove_selectors: Optional[List[str]] = None, evaluator: Optional[PlaywrightEvaluator] = None, proxy: Optional[Dict[str, str]] = None)[source]¶
Load HTML pages with Playwright and parse with Unstructured.
This is useful for loading pages that require javascript to render.
Parameters
urls (List[str]) – 
continue_on_failure (bool) – 
headless (bool) – 
remove_selectors (Optional[List[str]]) – 
evaluator (Optional[PlaywrightEvaluator]) – 
proxy (Optional[Dict[str, str]]) – 
urls¶
List of URLs to load.
Type
List[str]
continue_on_failure¶
If True, continue loading other URLs on failure.
Type
bool
headless¶
If True, the browser will run in headless mode.
Type
bool
proxy¶
If set, the browser will access URLs
through the specified proxy.
Type
Optional[Dict[str, str]]
Example
from langchain_community.document_loaders import PlaywrightURLLoader
urls = ["https://api.ipify.org/?format=json",]
proxy={
    "server": "https://xx.xx.xx:15818", # https://<host>:<port>
    "username": "username",
    "password": "password"
}
loader = PlaywrightURLLoader(urls, proxy=proxy)
data = loader.load()
Load a list of URLs using Playwright.
Methods
__init__(urls[, continue_on_failure, ...])
Load a list of URLs using Playwright.
alazy_load()
Load a list of URLs using Playwright.
alazy_load()
Load the specified URLs with Playwright and create Documents asynchronously.
aload()
Load the specified URLs with Playwright and create Documents asynchronously.
lazy_load()
Load the specified URLs using Playwright and create Document instances.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(urls: List[str], continue_on_failure: bool = True, headless: bool = True, remove_selectors: Optional[List[str]] = None, evaluator: Optional[PlaywrightEvaluator] = None, proxy: Optional[Dict[str, str]] = None)[source]¶
Load a list of URLs using Playwright.
Parameters
urls (List[str]) – 
continue_on_failure (bool) – 
headless (bool) – 
remove_selectors (Optional[List[str]]) – 
evaluator (Optional[PlaywrightEvaluator]) – 
proxy (Optional[Dict[str, str]]) – 
async alazy_load() → AsyncIterator[Document][source]¶
Load the specified URLs with Playwright and create Documents asynchronously.
Use this function when in a jupyter notebook environment.
Returns
A list of Document instances with loaded content.
Return type
AsyncIterator[Document]
async aload() → List[Document][source]¶
Load the specified URLs with Playwright and create Documents asynchronously.
Use this function when in a jupyter notebook environment.
Returns
A list of Document instances with loaded content.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load the specified URLs using Playwright and create Document instances.
Returns
A list of Document instances with loaded content.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using PlaywrightURLLoader¶
URL
langchain_community.document_loaders.parsers.language.scala.ScalaSegmenter¶
class langchain_community.document_loaders.parsers.language.scala.ScalaSegmenter(code: str)[source]¶
Code segmenter for Scala.
Methods
__init__(code)
extract_functions_classes()
get_chunk_query()
get_language()
get_parser()
is_valid()
make_line_comment(text)
simplify_code()
Parameters
code (str) – 
__init__(code: str)¶
Parameters
code (str) – 
extract_functions_classes() → List[str]¶
Return type
List[str]
get_chunk_query() → str[source]¶
Return type
str
get_language() → Language[source]¶
Return type
Language
get_parser() → Parser¶
Return type
Parser
is_valid() → bool¶
Return type
bool
make_line_comment(text: str) → str[source]¶
Parameters
text (str) – 
Return type
str
simplify_code() → str¶
Return type
str
langchain_community.document_loaders.conllu.CoNLLULoader¶
class langchain_community.document_loaders.conllu.CoNLLULoader(file_path: Union[str, Path])[source]¶
Load CoNLL-U files.
Initialize with a file path.
Methods
__init__(file_path)
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load from a file path.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (Union[str, Path]) – 
__init__(file_path: Union[str, Path])[source]¶
Initialize with a file path.
Parameters
file_path (Union[str, Path]) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load from a file path.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using CoNLLULoader¶
CoNLL-U
langchain_community.document_loaders.youtube.YoutubeLoader¶
class langchain_community.document_loaders.youtube.YoutubeLoader(video_id: str, add_video_info: bool = False, language: Union[str, Sequence[str]] = 'en', translation: Optional[str] = None, transcript_format: TranscriptFormat = TranscriptFormat.TEXT, continue_on_failure: bool = False)[source]¶
Load YouTube transcripts.
Initialize with YouTube video ID.
Methods
__init__(video_id[, add_video_info, ...])
Initialize with YouTube video ID.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
extract_video_id(youtube_url)
Extract video id from common YT urls.
from_youtube_url(youtube_url, **kwargs)
Given youtube URL, load video.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
video_id (str) – 
add_video_info (bool) – 
language (Union[str, Sequence[str]]) – 
translation (Optional[str]) – 
transcript_format (TranscriptFormat) – 
continue_on_failure (bool) – 
__init__(video_id: str, add_video_info: bool = False, language: Union[str, Sequence[str]] = 'en', translation: Optional[str] = None, transcript_format: TranscriptFormat = TranscriptFormat.TEXT, continue_on_failure: bool = False)[source]¶
Initialize with YouTube video ID.
Parameters
video_id (str) – 
add_video_info (bool) – 
language (Union[str, Sequence[str]]) – 
translation (Optional[str]) – 
transcript_format (TranscriptFormat) – 
continue_on_failure (bool) –
transcript_format (TranscriptFormat) – 
continue_on_failure (bool) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
static extract_video_id(youtube_url: str) → str[source]¶
Extract video id from common YT urls.
Parameters
youtube_url (str) – 
Return type
str
classmethod from_youtube_url(youtube_url: str, **kwargs: Any) → YoutubeLoader[source]¶
Given youtube URL, load video.
Parameters
youtube_url (str) – 
kwargs (Any) – 
Return type
YoutubeLoader
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using YoutubeLoader¶
Build a Query Analysis System
Google
YouTube
YouTube transcripts
langchain_community.document_loaders.parsers.audio.OpenAIWhisperParser¶
class langchain_community.document_loaders.parsers.audio.OpenAIWhisperParser(api_key: Optional[str] = None, *, chunk_duration_threshold: float = 0.1, base_url: Optional[str] = None, language: Optional[str] = None, prompt: Optional[str] = None, response_format: Optional[Literal['json', 'text', 'srt', 'verbose_json', 'vtt']] = None, temperature: Optional[float] = None)[source]¶
Transcribe and parse audio files.
Audio transcription is with OpenAI Whisper model.
Parameters
api_key (Optional[str]) – OpenAI API key
chunk_duration_threshold (float) – minimum duration of a chunk in seconds
NOTE: According to the OpenAI API, the chunk duration should be at least 0.1
seconds. If the chunk duration is less or equal than the threshold,
it will be skipped.
base_url (Optional[str]) – 
language (Optional[str]) – 
prompt (Optional[str]) – 
response_format (Optional[Literal['json', 'text', 'srt', 'verbose_json', 'vtt']]) – 
temperature (Optional[float]) – 
Methods
__init__([api_key, ...])
lazy_parse(blob)
Lazily parse the blob.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__(api_key: Optional[str] = None, *, chunk_duration_threshold: float = 0.1, base_url: Optional[str] = None, language: Optional[str] = None, prompt: Optional[str] = None, response_format: Optional[Literal['json', 'text', 'srt', 'verbose_json', 'vtt']] = None, temperature: Optional[float] = None)[source]¶
Parameters
Parameters
api_key (Optional[str]) – 
chunk_duration_threshold (float) – 
base_url (Optional[str]) – 
language (Optional[str]) – 
prompt (Optional[str]) – 
response_format (Optional[Literal['json', 'text', 'srt', 'verbose_json', 'vtt']]) – 
temperature (Optional[float]) – 
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazily parse the blob.
Parameters
blob (Blob) – 
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
Examples using OpenAIWhisperParser¶
Google
YouTube audio
langchain_community.document_loaders.xml.UnstructuredXMLLoader¶
class langchain_community.document_loaders.xml.UnstructuredXMLLoader(file_path: Union[str, Path], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load XML file using Unstructured.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain_community.document_loaders import UnstructuredXMLLoader
loader = UnstructuredXMLLoader(“example.xml”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-xml
Initialize with file path.
Methods
__init__(file_path[, mode])
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (Union[str, Path]) – 
mode (str) – 
unstructured_kwargs (Any) – 
__init__(file_path: Union[str, Path], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Initialize with file path.
Parameters
file_path (Union[str, Path]) – 
mode (str) – 
unstructured_kwargs (Any) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using UnstructuredXMLLoader¶
Unstructured
XML
langchain_community.document_loaders.pdf.PDFMinerPDFasHTMLLoader¶
class langchain_community.document_loaders.pdf.PDFMinerPDFasHTMLLoader(file_path: str, *, headers: Optional[Dict] = None)[source]¶
Load PDF files as HTML content using PDFMiner.
Initialize with a file path.
Attributes
source
Methods
__init__(file_path, *[, headers])
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (str) – 
headers (Optional[Dict]) – 
__init__(file_path: str, *, headers: Optional[Dict] = None)[source]¶
Initialize with a file path.
Parameters
file_path (str) – 
headers (Optional[Dict]) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
langchain_community.document_loaders.college_confidential.CollegeConfidentialLoader¶
class langchain_community.document_loaders.college_confidential.CollegeConfidentialLoader(web_path: Union[str, Sequence[str]] = '', header_template: Optional[dict] = None, verify_ssl: bool = True, proxies: Optional[dict] = None, continue_on_failure: bool = False, autoset_encoding: bool = True, encoding: Optional[str] = None, web_paths: Sequence[str] = (), requests_per_second: int = 2, default_parser: str = 'html.parser', requests_kwargs: Optional[Dict[str, Any]] = None, raise_for_status: bool = False, bs_get_text_kwargs: Optional[Dict[str, Any]] = None, bs_kwargs: Optional[Dict[str, Any]] = None, session: Optional[Any] = None)[source]¶
Load College Confidential webpages.
Initialize loader.
Parameters
web_paths (Sequence[str]) – Web paths to load from.
requests_per_second (int) – Max number of concurrent requests to make.
default_parser (str) – Default parser to use for BeautifulSoup.
requests_kwargs (Optional[Dict[str, Any]]) – kwargs for requests
raise_for_status (bool) – Raise an exception if http status code denotes an error.
bs_get_text_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 get_text
bs_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 web page parsing
web_path (Union[str, Sequence[str]]) – 
header_template (Optional[dict]) – 
verify_ssl (bool) – 
proxies (Optional[dict]) – 
continue_on_failure (bool) – 
autoset_encoding (bool) – 
encoding (Optional[str]) – 
session (Any) – 
Attributes
web_path
Methods
session (Any) – 
Attributes
web_path
Methods
__init__([web_path, header_template, ...])
Initialize loader.
alazy_load()
A lazy loader for Documents.
aload()
Load text from the urls in web_path async into Documents.
fetch_all(urls)
Fetch all urls concurrently with rate limiting.
lazy_load()
Lazy load text from the url(s) in web_path.
load()
Load webpages as Documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
scrape([parser])
Scrape data from webpage and return it in BeautifulSoup format.
scrape_all(urls[, parser])
Fetch all urls, then return soups for all results.
__init__(web_path: Union[str, Sequence[str]] = '', header_template: Optional[dict] = None, verify_ssl: bool = True, proxies: Optional[dict] = None, continue_on_failure: bool = False, autoset_encoding: bool = True, encoding: Optional[str] = None, web_paths: Sequence[str] = (), requests_per_second: int = 2, default_parser: str = 'html.parser', requests_kwargs: Optional[Dict[str, Any]] = None, raise_for_status: bool = False, bs_get_text_kwargs: Optional[Dict[str, Any]] = None, bs_kwargs: Optional[Dict[str, Any]] = None, session: Optional[Any] = None) → None¶
Initialize loader.
Parameters
web_paths (Sequence[str]) – Web paths to load from.
requests_per_second (int) – Max number of concurrent requests to make.
default_parser (str) – Default parser to use for BeautifulSoup.
requests_kwargs (Optional[Dict[str, Any]]) – kwargs for requests
raise_for_status (bool) – Raise an exception if http status code denotes an error.
raise_for_status (bool) – Raise an exception if http status code denotes an error.
bs_get_text_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 get_text
bs_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 web page parsing
web_path (Union[str, Sequence[str]]) – 
header_template (Optional[dict]) – 
verify_ssl (bool) – 
proxies (Optional[dict]) – 
continue_on_failure (bool) – 
autoset_encoding (bool) – 
encoding (Optional[str]) – 
session (Optional[Any]) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
aload() → List[Document]¶
Load text from the urls in web_path async into Documents.
Return type
List[Document]
async fetch_all(urls: List[str]) → Any¶
Fetch all urls concurrently with rate limiting.
Parameters
urls (List[str]) – 
Return type
Any
lazy_load() → Iterator[Document]¶
Lazy load text from the url(s) in web_path.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load webpages as Documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]