Tony20100/langchain_full huggingface dataset:
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
Return type
DictStrAny
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
dumps_kwargs (Any) – 
Return type
unicode
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
Parameters
localns (Any) – 
Return type
None
classmethod validate(value: Any) → Model¶
Parameters
value (Any) – 
Return type
Model
Examples using OneNoteLoader¶
Microsoft
Microsoft OneNote
langchain_community.document_loaders.arcgis_loader.ArcGISLoader¶
class langchain_community.document_loaders.arcgis_loader.ArcGISLoader(layer: Union[str, arcgis.features.FeatureLayer], gis: Optional[arcgis.gis.GIS] = None, where: str = '1=1', out_fields: Optional[Union[List[str], str]] = None, return_geometry: bool = False, result_record_count: Optional[int] = None, lyr_desc: Optional[str] = None, **kwargs: Any)[source]¶
Load records from an ArcGIS FeatureLayer.
Methods
__init__(layer[, gis, where, out_fields, ...])
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load records from FeatureLayer.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
layer (Union[str, arcgis.features.FeatureLayer]) – 
gis (Optional[arcgis.gis.GIS]) – 
where (str) – 
out_fields (Optional[Union[List[str], str]]) – 
return_geometry (bool) – 
result_record_count (Optional[int]) – 
lyr_desc (Optional[str]) – 
kwargs (Any) – 
__init__(layer: Union[str, arcgis.features.FeatureLayer], gis: Optional[arcgis.gis.GIS] = None, where: str = '1=1', out_fields: Optional[Union[List[str], str]] = None, return_geometry: bool = False, result_record_count: Optional[int] = None, lyr_desc: Optional[str] = None, **kwargs: Any)[source]¶
Parameters
layer (Union[str, arcgis.features.FeatureLayer]) –
Parameters
layer (Union[str, arcgis.features.FeatureLayer]) – 
gis (Optional[arcgis.gis.GIS]) – 
where (str) – 
out_fields (Optional[Union[List[str], str]]) – 
return_geometry (bool) – 
result_record_count (Optional[int]) – 
lyr_desc (Optional[str]) – 
kwargs (Any) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load records from FeatureLayer.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using ArcGISLoader¶
ArcGIS
langchain_community.document_loaders.web_base.WebBaseLoader¶
class langchain_community.document_loaders.web_base.WebBaseLoader(web_path: Union[str, Sequence[str]] = '', header_template: Optional[dict] = None, verify_ssl: bool = True, proxies: Optional[dict] = None, continue_on_failure: bool = False, autoset_encoding: bool = True, encoding: Optional[str] = None, web_paths: Sequence[str] = (), requests_per_second: int = 2, default_parser: str = 'html.parser', requests_kwargs: Optional[Dict[str, Any]] = None, raise_for_status: bool = False, bs_get_text_kwargs: Optional[Dict[str, Any]] = None, bs_kwargs: Optional[Dict[str, Any]] = None, session: Optional[Any] = None)[source]¶
Load HTML pages using urllib and parse them with `BeautifulSoup’.
Initialize loader.
Parameters
web_paths (Sequence[str]) – Web paths to load from.
requests_per_second (int) – Max number of concurrent requests to make.
default_parser (str) – Default parser to use for BeautifulSoup.
requests_kwargs (Optional[Dict[str, Any]]) – kwargs for requests
raise_for_status (bool) – Raise an exception if http status code denotes an error.
bs_get_text_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 get_text
bs_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 web page parsing
web_path (Union[str, Sequence[str]]) – 
header_template (Optional[dict]) – 
verify_ssl (bool) – 
proxies (Optional[dict]) – 
continue_on_failure (bool) – 
autoset_encoding (bool) – 
encoding (Optional[str]) – 
session (Any) – 
Attributes
web_path
Methods
session (Any) – 
Attributes
web_path
Methods
__init__([web_path, header_template, ...])
Initialize loader.
alazy_load()
A lazy loader for Documents.
aload()
Load text from the urls in web_path async into Documents.
fetch_all(urls)
Fetch all urls concurrently with rate limiting.
lazy_load()
Lazy load text from the url(s) in web_path.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
scrape([parser])
Scrape data from webpage and return it in BeautifulSoup format.
scrape_all(urls[, parser])
Fetch all urls, then return soups for all results.
__init__(web_path: Union[str, Sequence[str]] = '', header_template: Optional[dict] = None, verify_ssl: bool = True, proxies: Optional[dict] = None, continue_on_failure: bool = False, autoset_encoding: bool = True, encoding: Optional[str] = None, web_paths: Sequence[str] = (), requests_per_second: int = 2, default_parser: str = 'html.parser', requests_kwargs: Optional[Dict[str, Any]] = None, raise_for_status: bool = False, bs_get_text_kwargs: Optional[Dict[str, Any]] = None, bs_kwargs: Optional[Dict[str, Any]] = None, session: Optional[Any] = None) → None[source]¶
Initialize loader.
Parameters
web_paths (Sequence[str]) – Web paths to load from.
requests_per_second (int) – Max number of concurrent requests to make.
default_parser (str) – Default parser to use for BeautifulSoup.
requests_kwargs (Optional[Dict[str, Any]]) – kwargs for requests
raise_for_status (bool) – Raise an exception if http status code denotes an error.
raise_for_status (bool) – Raise an exception if http status code denotes an error.
bs_get_text_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 get_text
bs_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 web page parsing
web_path (Union[str, Sequence[str]]) – 
header_template (Optional[dict]) – 
verify_ssl (bool) – 
proxies (Optional[dict]) – 
continue_on_failure (bool) – 
autoset_encoding (bool) – 
encoding (Optional[str]) – 
session (Optional[Any]) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
aload() → List[Document][source]¶
Load text from the urls in web_path async into Documents.
Return type
List[Document]
async fetch_all(urls: List[str]) → Any[source]¶
Fetch all urls concurrently with rate limiting.
Parameters
urls (List[str]) – 
Return type
Any
lazy_load() → Iterator[Document][source]¶
Lazy load text from the url(s) in web_path.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Returns
List of Documents.
Return type
List[Document]
scrape(parser: Optional[str] = None) → Any[source]¶
Scrape data from webpage and return it in BeautifulSoup format.
Parameters
parser (Optional[str]) – 
Return type
Any
scrape_all(urls: List[str], parser: Optional[str] = None) → List[Any][source]¶
Fetch all urls, then return soups for all results.
Parameters
urls (List[str]) – 
parser (Optional[str]) – 
Return type
List[Any]
Examples using WebBaseLoader¶
Build a Local RAG Application
Build a Retrieval Augmented Generation (RAG) App
Build an Agent
Conversational RAG
Google Cloud Vertex AI Reranker
How to add chat history
How to add retrieval to chatbots
How to get your RAG application to return sources
How to stream results from your RAG application
How to use the MultiQueryRetriever
Infino
Merge Documents Loader
RePhraseQuery
Summarize Text
WebBaseLoader
Zep
Zep Cloud
langchain_community.document_loaders.parsers.pdf.extract_from_images_with_rapidocr¶
langchain_community.document_loaders.parsers.pdf.extract_from_images_with_rapidocr(images: Sequence[Union[Iterable[ndarray], bytes]]) → str[source]¶
Extract text from images with RapidOCR.
Parameters
images (Sequence[Union[Iterable[ndarray], bytes]]) – Images to extract text from.
Returns
Text extracted from images.
Raises
ImportError – If rapidocr-onnxruntime package is not installed.
Return type
str
langchain_community.document_loaders.notion.NotionDirectoryLoader¶
class langchain_community.document_loaders.notion.NotionDirectoryLoader(path: Union[str, Path], *, encoding: str = 'utf-8')[source]¶
Load Notion directory dump.
Initialize with a file path.
Methods
__init__(path, *[, encoding])
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
path (Union[str, Path]) – 
encoding (str) – 
__init__(path: Union[str, Path], *, encoding: str = 'utf-8') → None[source]¶
Initialize with a file path.
Parameters
path (Union[str, Path]) – 
encoding (str) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
langchain_community.document_loaders.quip.QuipLoader¶
class langchain_community.document_loaders.quip.QuipLoader(api_url: str, access_token: str, request_timeout: Optional[int] = 60, *, allow_dangerous_xml_parsing: bool = False)[source]¶
Load Quip pages.
Port of https://github.com/quip/quip-api/tree/master/samples/baqup
Parameters
api_url (str) – https://platform.quip.com
access_token (str) – token of access quip API. Please refer:
https://quip.com/dev/automation/documentation/current#section/Authentication/Get-Access-to-Quip’s-APIs
request_timeout (Optional[int]) – timeout of request, default 60s.
allow_dangerous_xml_parsing (bool) – Allow dangerous XML parsing, defaults to False
Methods
__init__(api_url, access_token[, ...])
param api_url
https://platform.quip.com
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
get_thread_ids_by_folder_id(folder_id, ...)
Get thread ids by folder id and update in thread_ids
lazy_load()
A lazy loader for Documents.
load([folder_ids, thread_ids, max_docs, ...])
:param : param folder_ids: List of specific folder IDs to load, defaults to None :param : param thread_ids: List of specific thread IDs to load, defaults to None :param : param max_docs: Maximum number of docs to retrieve in total, defaults 1000 :param : param include_all_folders: Include all folders that your access_token          can access, but doesn't include your private folder :param : param include_comments: Include comments, defaults to False :param : param include_images: Include images, defaults to False
load_and_split([text_splitter])
load_and_split([text_splitter])
Load Documents and split into chunks.
process_thread(thread_id, include_images, ...)
process_thread_images(tree)
process_thread_messages(thread_id)
process_threads(thread_ids, include_images, ...)
Process a list of thread into a list of documents.
__init__(api_url: str, access_token: str, request_timeout: Optional[int] = 60, *, allow_dangerous_xml_parsing: bool = False)[source]¶
Parameters
api_url (str) – https://platform.quip.com
access_token (str) – token of access quip API. Please refer:
https://quip.com/dev/automation/documentation/current#section/Authentication/Get-Access-to-Quip’s-APIs
request_timeout (Optional[int]) – timeout of request, default 60s.
allow_dangerous_xml_parsing (bool) – Allow dangerous XML parsing, defaults to False
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
get_thread_ids_by_folder_id(folder_id: str, depth: int, thread_ids: List[str]) → None[source]¶
Get thread ids by folder id and update in thread_ids
Parameters
folder_id (str) – 
depth (int) – 
thread_ids (List[str]) – 
Return type
None
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
A lazy loader for Documents.
Return type
Iterator[Document]
load(folder_ids: Optional[List[str]] = None, thread_ids: Optional[List[str]] = None, max_docs: Optional[int] = 1000, include_all_folders: bool = False, include_comments: bool = False, include_images: bool = False) → List[Document][source]¶
:param : param folder_ids: List of specific folder IDs to load, defaults to None
:param : param thread_ids: List of specific thread IDs to load, defaults to None
:param : param max_docs: Maximum number of docs to retrieve in total, defaults 1000
:param : param include_all_folders: Include all folders that your access_token
can access, but doesn’t include your private folder
:param : param include_comments: Include comments, defaults to False
:param : param include_images: Include images, defaults to False
Parameters
folder_ids (Optional[List[str]]) – 
thread_ids (Optional[List[str]]) – 
max_docs (Optional[int]) – 
include_all_folders (bool) – 
include_comments (bool) – 
include_images (bool) – 
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
process_thread(thread_id: str, include_images: bool, include_messages: bool) → Optional[Document][source]¶
Parameters
thread_id (str) – 
include_images (bool) –
Parameters
thread_id (str) – 
include_images (bool) – 
include_messages (bool) – 
Return type
Optional[Document]
process_thread_images(tree: ElementTree) → str[source]¶
Parameters
tree (ElementTree) – 
Return type
str
process_thread_messages(thread_id: str) → str[source]¶
Parameters
thread_id (str) – 
Return type
str
process_threads(thread_ids: Sequence[str], include_images: bool, include_messages: bool) → List[Document][source]¶
Process a list of thread into a list of documents.
Parameters
thread_ids (Sequence[str]) – 
include_images (bool) – 
include_messages (bool) – 
Return type
List[Document]
Examples using QuipLoader¶
Quip
langchain_community.document_loaders.unstructured.satisfies_min_unstructured_version¶
langchain_community.document_loaders.unstructured.satisfies_min_unstructured_version(min_version: str) → bool[source]¶
Check if the installed Unstructured version exceeds the minimum version
for the feature in question.
Parameters
min_version (str) – 
Return type
bool
langchain_community.document_loaders.airbyte.AirbyteCDKLoader¶
class langchain_community.document_loaders.airbyte.AirbyteCDKLoader(config: Mapping[str, Any], source_class: Any, stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None)[source]¶
Load with an Airbyte source connector implemented using the CDK.
Initializes the loader.
Parameters
config (Mapping[str, Any]) – The config to pass to the source connector.
source_class (Any) – The source connector class.
stream_name (str) – The name of the stream to load.
record_handler (Optional[Callable[[Any, Optional[str]], Document]]) – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state (Optional[Any]) – The state to pass to the source connector. Defaults to None.
Attributes
last_state
Methods
__init__(config, source_class, stream_name)
Initializes the loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(config: Mapping[str, Any], source_class: Any, stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None) → None[source]¶
Initializes the loader.
Parameters
config (Mapping[str, Any]) – The config to pass to the source connector.
source_class (Any) – The source connector class.
source_class (Any) – The source connector class.
stream_name (str) – The name of the stream to load.
record_handler (Optional[Callable[[Any, Optional[str]], Document]]) – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state (Optional[Any]) – The state to pass to the source connector. Defaults to None.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using AirbyteCDKLoader¶
Airbyte CDK (Deprecated)
langchain_community.document_loaders.parsers.pdf.PDFPlumberParser¶
class langchain_community.document_loaders.parsers.pdf.PDFPlumberParser(text_kwargs: Optional[Mapping[str, Any]] = None, dedupe: bool = False, extract_images: bool = False)[source]¶
Parse PDF with PDFPlumber.
Initialize the parser.
Parameters
text_kwargs (Optional[Mapping[str, Any]]) – Keyword arguments to pass to pdfplumber.Page.extract_text()
dedupe (bool) – Avoiding the error of duplicate characters if dedupe=True.
extract_images (bool) – 
Methods
__init__([text_kwargs, dedupe, extract_images])
Initialize the parser.
lazy_parse(blob)
Lazily parse the blob.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__(text_kwargs: Optional[Mapping[str, Any]] = None, dedupe: bool = False, extract_images: bool = False) → None[source]¶
Initialize the parser.
Parameters
text_kwargs (Optional[Mapping[str, Any]]) – Keyword arguments to pass to pdfplumber.Page.extract_text()
dedupe (bool) – Avoiding the error of duplicate characters if dedupe=True.
extract_images (bool) – 
Return type
None
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazily parse the blob.
Parameters
blob (Blob) – 
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
langchain_community.document_loaders.browserless.BrowserlessLoader¶
class langchain_community.document_loaders.browserless.BrowserlessLoader(api_token: str, urls: Union[str, List[str]], text_content: bool = True)[source]¶
Load webpages with Browserless /content endpoint.
Initialize with API token and the URLs to scrape
Attributes
api_token
Browserless API token.
urls
List of URLs to scrape.
Methods
__init__(api_token, urls[, text_content])
Initialize with API token and the URLs to scrape
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load Documents from URLs.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
api_token (str) – 
urls (Union[str, List[str]]) – 
text_content (bool) – 
__init__(api_token: str, urls: Union[str, List[str]], text_content: bool = True)[source]¶
Initialize with API token and the URLs to scrape
Parameters
api_token (str) – 
urls (Union[str, List[str]]) – 
text_content (bool) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load Documents from URLs.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
langchain_community.document_loaders.spider.SpiderLoader¶
class langchain_community.document_loaders.spider.SpiderLoader(url: str, *, api_key: Optional[str] = None, mode: Literal['scrape', 'crawl'] = 'scrape', params: Optional[dict] = None)[source]¶
Load web pages as Documents using Spider AI.
Must have the Python package spider-client installed and a Spider API key.
See https://spider.cloud for more.
Initialize with API key and URL.
Parameters
url (str) – The URL to be processed.
api_key (Optional[str]) – The Spider API key. If not specified, will be read from env
SPIDER_API_KEY. (var) – 
mode (Literal['scrape', 'crawl']) – The mode to run the loader in. Default is “scrape”.
Options include “scrape” (single page) and “crawl” (with deeper
crawling following subpages).
params (Optional[dict]) – Additional parameters for the Spider API.
Methods
__init__(url, *[, api_key, mode, params])
Initialize with API key and URL.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load documents based on the specified mode.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(url: str, *, api_key: Optional[str] = None, mode: Literal['scrape', 'crawl'] = 'scrape', params: Optional[dict] = None)[source]¶
Initialize with API key and URL.
Parameters
url (str) – The URL to be processed.