Tony20100/langchain_full huggingface dataset:
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load file.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using PDFPlumberLoader¶
How to load PDFs
langchain_community.document_loaders.diffbot.DiffbotLoader¶
class langchain_community.document_loaders.diffbot.DiffbotLoader(api_token: str, urls: List[str], continue_on_failure: bool = True)[source]¶
Load Diffbot json file.
Initialize with API token, ids, and key.
Parameters
api_token (str) – Diffbot API token.
urls (List[str]) – List of URLs to load.
continue_on_failure (bool) – Whether to continue loading other URLs if one fails.
Defaults to True.
Methods
__init__(api_token, urls[, continue_on_failure])
Initialize with API token, ids, and key.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Extract text from Diffbot on all the URLs and return Documents
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(api_token: str, urls: List[str], continue_on_failure: bool = True)[source]¶
Initialize with API token, ids, and key.
Parameters
api_token (str) – Diffbot API token.
urls (List[str]) – List of URLs to load.
continue_on_failure (bool) – Whether to continue loading other URLs if one fails.
Defaults to True.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Extract text from Diffbot on all the URLs and return Documents
Return type
Extract text from Diffbot on all the URLs and return Documents
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using DiffbotLoader¶
Diffbot
langchain_community.document_loaders.joplin.JoplinLoader¶
class langchain_community.document_loaders.joplin.JoplinLoader(access_token: Optional[str] = None, port: int = 41184, host: str = 'localhost')[source]¶
Load notes from Joplin.
In order to use this loader, you need to have Joplin running with the
Web Clipper enabled (look for “Web Clipper” in the app settings).
To get the access token, you need to go to the Web Clipper options and
under “Advanced Options” you will find the access token.
You can find more information about the Web Clipper service here:
https://joplinapp.org/clipper/
Parameters
access_token (Optional[str]) – The access token to use.
port (int) – The port where the Web Clipper service is running. Default is 41184.
host (str) – The host where the Web Clipper service is running.
Default is localhost.
Methods
__init__([access_token, port, host])
param access_token
The access token to use.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(access_token: Optional[str] = None, port: int = 41184, host: str = 'localhost') → None[source]¶
Parameters
access_token (Optional[str]) – The access token to use.
port (int) – The port where the Web Clipper service is running. Default is 41184.
host (str) – The host where the Web Clipper service is running.
Default is localhost.
Default is localhost.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using JoplinLoader¶
Joplin
langchain_community.document_loaders.parsers.docai.DocAIParser¶
class langchain_community.document_loaders.parsers.docai.DocAIParser(*, client: Optional[DocumentProcessorServiceClient] = None, location: Optional[str] = None, gcs_output_path: Optional[str] = None, processor_name: Optional[str] = None)[source]¶
[Deprecated] Google Cloud Document AI parser.
For a detailed explanation of Document AI, refer to the product documentation.
https://cloud.google.com/document-ai/docs/overview
Notes
Deprecated since version 0.0.32.
Initializes the parser.
Parameters
client (Optional[DocumentProcessorServiceClient]) – a DocumentProcessorServiceClient to use
location (Optional[str]) – a Google Cloud location where a Document AI processor is located
gcs_output_path (Optional[str]) – a path on Google Cloud Storage to store parsing results
processor_name (Optional[str]) – full resource name of a Document AI processor or processor
version
You should provide either a client or location (and then a clientwould be instantiated).
Methods
__init__(*[, client, location, ...])
Initializes the parser.
batch_parse(blobs[, gcs_output_path, ...])
Parses a list of blobs lazily.
docai_parse(blobs, *[, gcs_output_path, ...])
Runs Google Document AI PDF Batch Processing on a list of blobs.
get_results(operations)
is_running(operations)
lazy_parse(blob)
Parses a blob lazily.
online_process(blob[, ...])
Parses a blob lazily using online processing.
operations_from_names(operation_names)
Initializes Long-Running Operations from their names.
parse(blob)
Eagerly parse the blob into a document or documents.
parse_from_results(results)
Eagerly parse the blob into a document or documents.
parse_from_results(results)
__init__(*, client: Optional[DocumentProcessorServiceClient] = None, location: Optional[str] = None, gcs_output_path: Optional[str] = None, processor_name: Optional[str] = None)[source]¶
Initializes the parser.
Parameters
client (Optional[DocumentProcessorServiceClient]) – a DocumentProcessorServiceClient to use
location (Optional[str]) – a Google Cloud location where a Document AI processor is located
gcs_output_path (Optional[str]) – a path on Google Cloud Storage to store parsing results
processor_name (Optional[str]) – full resource name of a Document AI processor or processor
version
You should provide either a client or location (and then a clientwould be instantiated).
batch_parse(blobs: Sequence[Blob], gcs_output_path: Optional[str] = None, timeout_sec: int = 3600, check_in_interval_sec: int = 60) → Iterator[Document][source]¶
Parses a list of blobs lazily.
Parameters
blobs (Sequence[Blob]) – a list of blobs to parse.
gcs_output_path (Optional[str]) – a path on Google Cloud Storage to store parsing results.
timeout_sec (int) – a timeout to wait for Document AI to complete, in seconds.
check_in_interval_sec (int) – an interval to wait until next check
whether parsing operations have been completed, in seconds
Return type
Iterator[Document]
This is a long-running operation. A recommended way is to decoupleparsing from creating LangChain Documents:
>>> operations = parser.docai_parse(blobs, gcs_path)
>>> parser.is_running(operations)
You can get operations names and save them:
>>> names = [op.operation.name for op in operations]
>>> names = [op.operation.name for op in operations]
And when all operations are finished, you can use their results:
>>> operations = parser.operations_from_names(operation_names)
>>> results = parser.get_results(operations)
>>> docs = parser.parse_from_results(results)
docai_parse(blobs: Sequence[Blob], *, gcs_output_path: Optional[str] = None, processor_name: Optional[str] = None, batch_size: int = 1000, enable_native_pdf_parsing: bool = True, field_mask: Optional[str] = None) → List[Operation][source]¶
Runs Google Document AI PDF Batch Processing on a list of blobs.
Parameters
blobs (Sequence[Blob]) – a list of blobs to be parsed
gcs_output_path (Optional[str]) – a path (folder) on GCS to store results
processor_name (Optional[str]) – name of a Document AI processor.
batch_size (int) – amount of documents per batch
enable_native_pdf_parsing (bool) – a config option for the parser
field_mask (Optional[str]) – a comma-separated list of which fields to include in the
Document AI response.
suggested: “text,pages.pageNumber,pages.layout”
Return type
List[Operation]
Document AI has a 1000 file limit per batch, so batches larger than that need
to be split into multiple requests.
Batch processing is an async long-running operation
and results are stored in a output GCS bucket.
get_results(operations: List[Operation]) → List[DocAIParsingResults][source]¶
Parameters
operations (List[Operation]) – 
Return type
List[DocAIParsingResults]
is_running(operations: List[Operation]) → bool[source]¶
Parameters
operations (List[Operation]) – 
Return type
bool
Parameters
operations (List[Operation]) – 
Return type
bool
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Parses a blob lazily.
Parameters
blobs – a Blob to parse
blob (Blob) – 
Return type
Iterator[Document]
This is a long-running operation. A recommended way is to batchdocuments together and use the batch_parse() method.
online_process(blob: Blob, enable_native_pdf_parsing: bool = True, field_mask: Optional[str] = None, page_range: Optional[List[int]] = None) → Iterator[Document][source]¶
Parses a blob lazily using online processing.
Parameters
blob (Blob) – a blob to parse.
enable_native_pdf_parsing (bool) – enable pdf embedded text extraction
field_mask (Optional[str]) – a comma-separated list of which fields to include in the
Document AI response.
suggested: “text,pages.pageNumber,pages.layout”
page_range (Optional[List[int]]) – list of page numbers to parse. If None,
entire document will be parsed.
Return type
Iterator[Document]
operations_from_names(operation_names: List[str]) → List[Operation][source]¶
Initializes Long-Running Operations from their names.
Parameters
operation_names (List[str]) – 
Return type
List[Operation]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
parse_from_results(results: List[DocAIParsingResults]) → Iterator[Document][source]¶
langchain_community.document_loaders.excel.UnstructuredExcelLoader¶
class langchain_community.document_loaders.excel.UnstructuredExcelLoader(file_path: Union[str, Path], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load Microsoft Excel files using Unstructured.
Like other
Unstructured loaders, UnstructuredExcelLoader can be used in both
“single” and “elements” mode. If you use the loader in “elements”
mode, each sheet in the Excel file will be an Unstructured Table
element. If you use the loader in “single” mode, an
HTML representation of the table will be available in the
“text_as_html” key in the document metadata.
Examples
from langchain_community.document_loaders.excel import UnstructuredExcelLoader
loader = UnstructuredExcelLoader(“stanley-cups.xlsx”, mode=”elements”)
docs = loader.load()
Parameters
file_path (Union[str, Path]) – The path to the Microsoft Excel file.
mode (str) – The mode to use when partitioning the file. See unstructured docs
for more info. Optional. Defaults to “single”.
**unstructured_kwargs (Any) – Keyword arguments to pass to unstructured.
Methods
__init__(file_path[, mode])
param file_path
The path to the Microsoft Excel file.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Parameters
file_path (Union[str, Path]) – The path to the Microsoft Excel file.
Parameters
file_path (Union[str, Path]) – The path to the Microsoft Excel file.
mode (str) – The mode to use when partitioning the file. See unstructured docs
for more info. Optional. Defaults to “single”.
**unstructured_kwargs (Any) – Keyword arguments to pass to unstructured.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using UnstructuredExcelLoader¶
Microsoft
Microsoft Excel
Unstructured
langchain_community.document_loaders.parsers.docai.DocAIParsingResults¶
class langchain_community.document_loaders.parsers.docai.DocAIParsingResults(source_path: str, parsed_path: str)[source]¶
Dataclass to store Document AI parsing results.
Attributes
source_path
parsed_path
Methods
__init__(source_path, parsed_path)
Parameters
source_path (str) – 
parsed_path (str) – 
__init__(source_path: str, parsed_path: str) → None¶
Parameters
source_path (str) – 
parsed_path (str) – 
Return type
None
langchain_community.document_loaders.pdf.PyPDFDirectoryLoader¶
class langchain_community.document_loaders.pdf.PyPDFDirectoryLoader(path: Union[str, Path], glob: str = '**/[!.]*.pdf', silent_errors: bool = False, load_hidden: bool = False, recursive: bool = False, extract_images: bool = False)[source]¶
Load a directory with PDF files using pypdf and chunks at character level.
Loader also stores page numbers in metadata.
Methods
__init__(path[, glob, silent_errors, ...])
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
path (Union[str, Path]) – 
glob (str) – 
silent_errors (bool) – 
load_hidden (bool) – 
recursive (bool) – 
extract_images (bool) – 
__init__(path: Union[str, Path], glob: str = '**/[!.]*.pdf', silent_errors: bool = False, load_hidden: bool = False, recursive: bool = False, extract_images: bool = False)[source]¶
Parameters
path (Union[str, Path]) – 
glob (str) – 
silent_errors (bool) – 
load_hidden (bool) – 
recursive (bool) – 
extract_images (bool) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using PyPDFDirectoryLoader¶
How to load PDFs
langchain_community.document_loaders.chm.CHMParser¶
class langchain_community.document_loaders.chm.CHMParser(path: str)[source]¶
Microsoft Compiled HTML Help (CHM) Parser.
Attributes
encoding
path
file
Methods
__init__(path)
index()
load(path)
load_all()
Parameters
path (str) – 
__init__(path: str)[source]¶
Parameters
path (str) – 
index() → List[Dict[str, str]][source]¶
Return type
List[Dict[str, str]]
load(path: Union[str, bytes]) → str[source]¶
Parameters
path (Union[str, bytes]) – 
Return type
str
load_all() → List[Dict[str, str]][source]¶
Return type
List[Dict[str, str]]
langchain_community.document_loaders.parsers.grobid.GrobidParser¶
class langchain_community.document_loaders.parsers.grobid.GrobidParser(segment_sentences: bool, grobid_server: str = 'http://localhost:8070/api/processFulltextDocument')[source]¶
Load  article PDF files using Grobid.
Methods
__init__(segment_sentences[, grobid_server])
lazy_parse(blob)
Lazy parsing interface.
parse(blob)
Eagerly parse the blob into a document or documents.
process_xml(file_path, xml_data, ...)
Process the XML file from Grobin.
Parameters
segment_sentences (bool) – 
grobid_server (str) – 
__init__(segment_sentences: bool, grobid_server: str = 'http://localhost:8070/api/processFulltextDocument') → None[source]¶
Parameters
segment_sentences (bool) – 
grobid_server (str) – 
Return type
None
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazy parsing interface.
Subclasses are required to implement this method.
Parameters
blob (Blob) – Blob instance
Returns
Generator of documents
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
process_xml(file_path: str, xml_data: str, segment_sentences: bool) → Iterator[Document][source]¶
Process the XML file from Grobin.
Parameters
file_path (str) – 
xml_data (str) –
langchain_community.document_loaders.blackboard.BlackboardLoader¶
class langchain_community.document_loaders.blackboard.BlackboardLoader(blackboard_course_url: str, bbrouter: str, load_all_recursively: bool = True, basic_auth: Optional[Tuple[str, str]] = None, cookies: Optional[dict] = None, continue_on_failure: bool = False)[source]¶
Load a Blackboard course.
This loader is not compatible with all Blackboard courses. It is only
compatible with courses that use the new Blackboard interface.
To use this loader, you must have the BbRouter cookie. You can get this
cookie by logging into the course and then copying the value of the
BbRouter cookie from the browser’s developer tools.
Example
from langchain_community.document_loaders import BlackboardLoader
loader = BlackboardLoader(
    blackboard_course_url="https://blackboard.example.com/webapps/blackboard/execute/announcement?method=search&context=course_entry&course_id=_123456_1",
    bbrouter="expires:12345...",
)
documents = loader.load()
Initialize with blackboard course url.
The BbRouter cookie is required for most blackboard courses.
Parameters
blackboard_course_url (str) – Blackboard course url.
bbrouter (str) – BbRouter cookie.
load_all_recursively (bool) – If True, load all documents recursively.
basic_auth (Optional[Tuple[str, str]]) – Basic auth credentials.
cookies (Optional[dict]) – Cookies.
continue_on_failure (bool) – whether to continue loading the sitemap if an error
occurs loading a url, emitting a warning instead of raising an
exception. Setting this to True makes the loader more robust, but also
may result in missing data. Default: False
Raises
may result in missing data. Default: False
Raises
ValueError – If blackboard course url is invalid.
Attributes
web_path
Methods
__init__(blackboard_course_url, bbrouter[, ...])
Initialize with blackboard course url.
alazy_load()
A lazy loader for Documents.
aload()
Load text from the urls in web_path async into Documents.
check_bs4()
Check if BeautifulSoup4 is installed.
download(path)
Download a file from an url.
fetch_all(urls)
Fetch all urls concurrently with rate limiting.
lazy_load()
Lazy load text from the url(s) in web_path.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
parse_filename(url)
Parse the filename from an url.
scrape([parser])
Scrape data from webpage and return it in BeautifulSoup format.
scrape_all(urls[, parser])
Fetch all urls, then return soups for all results.
__init__(blackboard_course_url: str, bbrouter: str, load_all_recursively: bool = True, basic_auth: Optional[Tuple[str, str]] = None, cookies: Optional[dict] = None, continue_on_failure: bool = False)[source]¶
Initialize with blackboard course url.
The BbRouter cookie is required for most blackboard courses.
Parameters
blackboard_course_url (str) – Blackboard course url.
bbrouter (str) – BbRouter cookie.
load_all_recursively (bool) – If True, load all documents recursively.
basic_auth (Optional[Tuple[str, str]]) – Basic auth credentials.
cookies (Optional[dict]) – Cookies.
continue_on_failure (bool) – whether to continue loading the sitemap if an error
continue_on_failure (bool) – whether to continue loading the sitemap if an error
occurs loading a url, emitting a warning instead of raising an
exception. Setting this to True makes the loader more robust, but also
may result in missing data. Default: False
Raises
ValueError – If blackboard course url is invalid.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
aload() → List[Document]¶
Load text from the urls in web_path async into Documents.
Return type
List[Document]
check_bs4() → None[source]¶
Check if BeautifulSoup4 is installed.
Raises
ImportError – If BeautifulSoup4 is not installed.
Return type
None
download(path: str) → None[source]¶
Download a file from an url.
Parameters
path (str) – Path to the file.
Return type
None
async fetch_all(urls: List[str]) → Any¶
Fetch all urls concurrently with rate limiting.
Parameters
urls (List[str]) – 
Return type
Any
lazy_load() → Iterator[Document]¶
Lazy load text from the url(s) in web_path.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Returns
List of Documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
parse_filename(url: str) → str[source]¶
Parse the filename from an url.
Parameters
url (str) – Url to parse the filename from.
Returns
The filename.
Return type
str
scrape(parser: Optional[str] = None) → Any¶
Scrape data from webpage and return it in BeautifulSoup format.
Parameters
parser (Optional[str]) – 
Return type
Any
scrape_all(urls: List[str], parser: Optional[str] = None) → List[Any]¶
Fetch all urls, then return soups for all results.
Parameters
urls (List[str]) – 
parser (Optional[str]) – 
Return type
List[Any]
Examples using BlackboardLoader¶
Blackboard