Tony20100/langchain_full huggingface dataset:
langchain_community.document_loaders.url_playwright.PlaywrightEvaluator¶
class langchain_community.document_loaders.url_playwright.PlaywrightEvaluator[source]¶
Abstract base class for all evaluators.
Each evaluator should take a page, a browser instance, and a response
object, process the page as necessary, and return the resulting text.
Methods
__init__()
evaluate(page, browser, response)
Synchronously process the page and return the resulting text.
evaluate_async(page, browser, response)
Asynchronously process the page and return the resulting text.
__init__()¶
abstract evaluate(page: Page, browser: Browser, response: Response) → str[source]¶
Synchronously process the page and return the resulting text.
Parameters
page (Page) – The page to process.
browser (Browser) – The browser instance.
response (Response) – The response from page.goto().
Returns
The text content of the page.
Return type
text
abstract async evaluate_async(page: AsyncPage, browser: AsyncBrowser, response: AsyncResponse) → str[source]¶
Asynchronously process the page and return the resulting text.
Parameters
page (AsyncPage) – The page to process.
browser (AsyncBrowser) – The browser instance.
response (AsyncResponse) – The response from page.goto().
Returns
The text content of the page.
Return type
text
langchain_community.document_loaders.parsers.doc_intelligence.AzureAIDocumentIntelligenceParser¶
class langchain_community.document_loaders.parsers.doc_intelligence.AzureAIDocumentIntelligenceParser(api_endpoint: str, api_key: str, api_version: Optional[str] = None, api_model: str = 'prebuilt-layout', mode: str = 'markdown', analysis_features: Optional[List[str]] = None)[source]¶
Loads a PDF with Azure Document Intelligence
(formerly Forms Recognizer).
Methods
__init__(api_endpoint, api_key[, ...])
lazy_parse(blob)
Lazily parse the blob.
parse(blob)
Eagerly parse the blob into a document or documents.
parse_url(url)
Parameters
api_endpoint (str) – 
api_key (str) – 
api_version (Optional[str]) – 
api_model (str) – 
mode (str) – 
analysis_features (Optional[List[str]]) – 
__init__(api_endpoint: str, api_key: str, api_version: Optional[str] = None, api_model: str = 'prebuilt-layout', mode: str = 'markdown', analysis_features: Optional[List[str]] = None)[source]¶
Parameters
api_endpoint (str) – 
api_key (str) – 
api_version (Optional[str]) – 
api_model (str) – 
mode (str) – 
analysis_features (Optional[List[str]]) – 
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazily parse the blob.
Parameters
blob (Blob) – 
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
langchain_community.document_loaders.airbyte.AirbyteZendeskSupportLoader¶
class langchain_community.document_loaders.airbyte.AirbyteZendeskSupportLoader(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None)[source]¶
Load from Zendesk Support using an Airbyte source connector.
Initializes the loader.
Parameters
config (Mapping[str, Any]) – The config to pass to the source connector.
stream_name (str) – The name of the stream to load.
record_handler (Optional[Callable[[Any, Optional[str]], Document]]) – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state (Optional[Any]) – The state to pass to the source connector. Defaults to None.
Attributes
last_state
Methods
__init__(config, stream_name[, ...])
Initializes the loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None) → None[source]¶
Initializes the loader.
Parameters
config (Mapping[str, Any]) – The config to pass to the source connector.
stream_name (str) – The name of the stream to load.
record_handler (Optional[Callable[[Any, Optional[str]], Document]]) – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state (Optional[Any]) – The state to pass to the source connector. Defaults to None.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using AirbyteZendeskSupportLoader¶
Airbyte Zendesk Support (Deprecated)
langchain_community.document_loaders.oracleai.ParseOracleDocMetadata¶
class langchain_community.document_loaders.oracleai.ParseOracleDocMetadata[source]¶
Parse Oracle doc metadata…
Initialize and reset this instance.
If convert_charrefs is True (the default), all character references
are automatically converted to the corresponding Unicode characters.
Attributes
CDATA_CONTENT_ELEMENTS
Methods
__init__()
Initialize and reset this instance.
check_for_whole_start_tag(i)
clear_cdata_mode()
close()
Handle any buffered data.
feed(data)
Feed data to the parser.
get_metadata()
get_starttag_text()
Return full source of start tag: '<...>'.
getpos()
Return current line number and offset.
goahead(end)
handle_charref(name)
handle_comment(data)
handle_data(data)
handle_decl(decl)
handle_endtag(tag)
handle_entityref(name)
handle_pi(data)
handle_startendtag(tag, attrs)
handle_starttag(tag, attrs)
parse_bogus_comment(i[, report])
parse_comment(i[, report])
parse_declaration(i)
parse_endtag(i)
parse_html_declaration(i)
parse_marked_section(i[, report])
parse_pi(i)
parse_starttag(i)
reset()
Reset this instance.
set_cdata_mode(elem)
unknown_decl(data)
updatepos(i, j)
__init__() → None[source]¶
Initialize and reset this instance.
If convert_charrefs is True (the default), all character references
are automatically converted to the corresponding Unicode characters.
Return type
None
check_for_whole_start_tag(i)¶
clear_cdata_mode()¶
close()¶
Handle any buffered data.
feed(data)¶
Feed data to the parser.
Handle any buffered data.
feed(data)¶
Feed data to the parser.
Call this as often as you want, with as little or as much text
as you want (may include ‘n’).
get_metadata() → Dict[str, Any][source]¶
Return type
Dict[str, Any]
get_starttag_text()¶
Return full source of start tag: ‘<…>’.
getpos()¶
Return current line number and offset.
goahead(end)¶
handle_charref(name)¶
handle_comment(data)¶
handle_data(data: str) → None[source]¶
Parameters
data (str) – 
Return type
None
handle_decl(decl)¶
handle_endtag(tag)¶
handle_entityref(name)¶
handle_pi(data)¶
handle_startendtag(tag, attrs)¶
handle_starttag(tag: str, attrs: List[Tuple[str, Optional[str]]]) → None[source]¶
Parameters
tag (str) – 
attrs (List[Tuple[str, Optional[str]]]) – 
Return type
None
parse_bogus_comment(i, report=1)¶
parse_comment(i, report=1)¶
parse_declaration(i)¶
parse_endtag(i)¶
parse_html_declaration(i)¶
parse_marked_section(i, report=1)¶
parse_pi(i)¶
parse_starttag(i)¶
reset()¶
Reset this instance.  Loses all unprocessed data.
set_cdata_mode(elem)¶
unknown_decl(data)¶
updatepos(i, j)¶
langchain_community.document_loaders.onedrive_file.OneDriveFileLoader¶
class langchain_community.document_loaders.onedrive_file.OneDriveFileLoader[source]¶
Bases: BaseLoader, BaseModel
Load a file from Microsoft OneDrive.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param file: File [Required]¶
The file to load.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to include in new model
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to exclude from new model, as with values this takes precedence over include
Parameters
obj (Any) – 
Return type
Model
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
by_alias (bool) – 
skip_defaults (Optional[bool]) – 
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
encoder (Optional[Callable[[Any], Any]]) – 
models_as_dict (bool) – 
dumps_kwargs (Any) – 
Return type
unicode
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load Documents
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
langchain_community.document_loaders.docusaurus.DocusaurusLoader¶
class langchain_community.document_loaders.docusaurus.DocusaurusLoader(url: str, custom_html_tags: Optional[List[str]] = None, **kwargs: Any)[source]¶
Load from Docusaurus Documentation.
It leverages the SitemapLoader to loop through the generated pages of a
Docusaurus Documentation website and extracts the content by looking for specific
HTML tags. By default, the parser searches for the main content of the Docusaurus
page, which is normally the <article>. You can also define your own
custom HTML tags by providing them as a list, for example: [“div”, “.main”, “a”].
Initialize DocusaurusLoader
Parameters
url (str) – The base URL of the Docusaurus website.
custom_html_tags (Optional[List[str]]) – Optional custom html tags to extract content from pages.
kwargs (Any) – Additional args to extend the underlying SitemapLoader, for example:
filter_urls, blocksize, meta_function, is_local, continue_on_failure
Attributes
web_path
Methods
__init__(url[, custom_html_tags])
Initialize DocusaurusLoader
alazy_load()
A lazy loader for Documents.
aload()
Load text from the urls in web_path async into Documents.
fetch_all(urls)
Fetch all urls concurrently with rate limiting.
lazy_load()
Load sitemap.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
parse_sitemap(soup)
Parse sitemap xml and load into a list of dicts.
scrape([parser])
Scrape data from webpage and return it in BeautifulSoup format.
scrape_all(urls[, parser])
Fetch all urls, then return soups for all results.
Fetch all urls, then return soups for all results.
__init__(url: str, custom_html_tags: Optional[List[str]] = None, **kwargs: Any)[source]¶
Initialize DocusaurusLoader
Parameters
url (str) – The base URL of the Docusaurus website.
custom_html_tags (Optional[List[str]]) – Optional custom html tags to extract content from pages.
kwargs (Any) – Additional args to extend the underlying SitemapLoader, for example:
filter_urls, blocksize, meta_function, is_local, continue_on_failure
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
aload() → List[Document]¶
Load text from the urls in web_path async into Documents.
Return type
List[Document]
async fetch_all(urls: List[str]) → Any¶
Fetch all urls concurrently with rate limiting.
Parameters
urls (List[str]) – 
Return type
Any
lazy_load() → Iterator[Document]¶
Load sitemap.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
parse_sitemap(soup: Any) → List[dict]¶
Parse sitemap xml and load into a list of dicts.
Parameters
Parse sitemap xml and load into a list of dicts.
Parameters
soup (Any) – BeautifulSoup object.
Returns
List of dicts.
Return type
List[dict]
scrape(parser: Optional[str] = None) → Any¶
Scrape data from webpage and return it in BeautifulSoup format.
Parameters
parser (Optional[str]) – 
Return type
Any
scrape_all(urls: List[str], parser: Optional[str] = None) → List[Any]¶
Fetch all urls, then return soups for all results.
Parameters
urls (List[str]) – 
parser (Optional[str]) – 
Return type
List[Any]
Examples using DocusaurusLoader¶
Docusaurus
langchain_community.document_loaders.blob_loaders.youtube_audio.YoutubeAudioLoader¶
class langchain_community.document_loaders.blob_loaders.youtube_audio.YoutubeAudioLoader(urls: List[str], save_dir: str)[source]¶
Load YouTube urls as audio file(s).
Methods
__init__(urls, save_dir)
yield_blobs()
Yield audio blobs for each url.
Parameters
urls (List[str]) – 
save_dir (str) – 
__init__(urls: List[str], save_dir: str)[source]¶
Parameters
urls (List[str]) – 
save_dir (str) – 
yield_blobs() → Iterable[Blob][source]¶
Yield audio blobs for each url.
Return type
Iterable[Blob]
Examples using YoutubeAudioLoader¶
Google
YouTube audio
langchain_community.document_loaders.stripe.StripeLoader¶
class langchain_community.document_loaders.stripe.StripeLoader(resource: str, access_token: Optional[str] = None)[source]¶
Load from Stripe API.
Initialize with a resource and an access token.
Parameters
resource (str) – The resource.
access_token (Optional[str]) – The access token.
Methods
__init__(resource[, access_token])
Initialize with a resource and an access token.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(resource: str, access_token: Optional[str] = None) → None[source]¶
Initialize with a resource and an access token.
Parameters
resource (str) – The resource.
access_token (Optional[str]) – The access token.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
langchain_community.document_loaders.rst.UnstructuredRSTLoader¶
class langchain_community.document_loaders.rst.UnstructuredRSTLoader(file_path: Union[str, Path], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load RST files using Unstructured.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain_community.document_loaders import UnstructuredRSTLoader
loader = UnstructuredRSTLoader(“example.rst”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-rst
Initialize with a file path.
Parameters
file_path (Union[str, Path]) – The path to the file to load.
mode (str) – The mode to use for partitioning. See unstructured for details.
Defaults to “single”.
**unstructured_kwargs (Any) – Additional keyword arguments to pass
to unstructured.
Methods
__init__(file_path[, mode])
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Initialize with a file path.
Initialize with a file path.
Parameters
file_path (Union[str, Path]) – The path to the file to load.
mode (str) – The mode to use for partitioning. See unstructured for details.
Defaults to “single”.
**unstructured_kwargs (Any) – Additional keyword arguments to pass
to unstructured.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using UnstructuredRSTLoader¶
RST
Unstructured
langchain_community.document_loaders.merge.MergedDataLoader¶
class langchain_community.document_loaders.merge.MergedDataLoader(loaders: List)[source]¶
Merge documents from a list of loaders
Initialize with a list of loaders
Methods
__init__(loaders)
Initialize with a list of loaders
alazy_load()
Lazy load docs from each individual loader.
aload()
Load data into Document objects.
lazy_load()
Lazy load docs from each individual loader.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
loaders (List) – 
__init__(loaders: List)[source]¶
Initialize with a list of loaders
Parameters
loaders (List) – 
async alazy_load() → AsyncIterator[Document][source]¶
Lazy load docs from each individual loader.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load docs from each individual loader.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using MergedDataLoader¶
Merge Documents Loader
langchain_community.document_loaders.mongodb.MongodbLoader¶
class langchain_community.document_loaders.mongodb.MongodbLoader(connection_string: str, db_name: str, collection_name: str, *, filter_criteria: Optional[Dict] = None, field_names: Optional[Sequence[str]] = None)[source]¶
Load MongoDB documents.
Methods
__init__(connection_string, db_name, ...[, ...])
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
connection_string (str) – 
db_name (str) – 
collection_name (str) – 
filter_criteria (Optional[Dict]) – 
field_names (Optional[Sequence[str]]) – 
__init__(connection_string: str, db_name: str, collection_name: str, *, filter_criteria: Optional[Dict] = None, field_names: Optional[Sequence[str]] = None) → None[source]¶
Parameters
connection_string (str) – 
db_name (str) – 
collection_name (str) – 
filter_criteria (Optional[Dict]) – 
field_names (Optional[Sequence[str]]) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Attention:
Load data into Document objects.
Attention:
This implementation starts an asyncio event loop which
will only work if running in a sync env. In an async env, it should
fail since there is already an event loop running.
This code should be updated to kick off the event loop from a separate
thread if running within an async context.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using MongodbLoader¶
MongoDB