Tony20100/langchain_full huggingface dataset:
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
by_alias (bool) – 
skip_defaults (Optional[bool]) – 
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
encoder (Optional[Callable[[Any], Any]]) – 
models_as_dict (bool) – 
dumps_kwargs (Any) – 
Return type
unicode
lazy_load() → Iterator[Document][source]¶
Load documents lazily. Use this when working at a large scale.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
path (Union[str, Path]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod parse_obj(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
Parameters
obj (Any) – 
Return type
Model
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
b (Union[str, bytes]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
Return type
DictStrAny
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
dumps_kwargs (Any) – 
Return type
unicode
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
Parameters
localns (Any) – 
Return type
None
classmethod validate(value: Any) → Model¶
Parameters
value (Any) – 
Return type
Model
Examples using SharePointLoader¶
Microsoft
Microsoft SharePoint
langchain_community.document_loaders.larksuite.LarkSuiteWikiLoader¶
class langchain_community.document_loaders.larksuite.LarkSuiteWikiLoader(domain: str, access_token: str, wiki_id: str)[source]¶
Load from LarkSuite (FeiShu) wiki.
Initialize with domain, access_token (tenant / user), and wiki_id.
Parameters
domain (str) – The domain to load the LarkSuite.
access_token (str) – The access_token to use.
wiki_id (str) – The wiki_id to load.
Methods
__init__(domain, access_token, wiki_id)
Initialize with domain, access_token (tenant / user), and wiki_id.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load LarkSuite (FeiShu) wiki document.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(domain: str, access_token: str, wiki_id: str)[source]¶
Initialize with domain, access_token (tenant / user), and wiki_id.
Parameters
domain (str) – The domain to load the LarkSuite.
access_token (str) – The access_token to use.
wiki_id (str) – The wiki_id to load.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load LarkSuite (FeiShu) wiki document.
Return type
Iterator[Document]
load() → List[Document]¶
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using LarkSuiteWikiLoader¶
LarkSuite (FeiShu)
langchain_community.document_loaders.bigquery.BigQueryLoader¶
class langchain_community.document_loaders.bigquery.BigQueryLoader(query: str, project: Optional[str] = None, page_content_columns: Optional[List[str]] = None, metadata_columns: Optional[List[str]] = None, credentials: Optional[Credentials] = None)[source]¶
[Deprecated] Load from the Google Cloud Platform BigQuery.
Each document represents one row of the result. The page_content_columns
are written into the page_content of the document. The metadata_columns
are written into the metadata of the document. By default, all columns
are written into the page_content and none into the metadata.
Notes
Deprecated since version 0.0.32.
Initialize BigQuery document loader.
Parameters
query (str) – The query to run in BigQuery.
project (Optional[str]) – Optional. The project to run the query in.
page_content_columns (Optional[List[str]]) – Optional. The columns to write into the page_content
of the document.
metadata_columns (Optional[List[str]]) – Optional. The columns to write into the metadata of the
document.
credentials (Optional[Credentials]) – google.auth.credentials.Credentials, optional
Credentials for accessing Google APIs. Use this parameter to override
default credentials, such as to use Compute Engine
(google.auth.compute_engine.Credentials) or Service Account
(google.oauth2.service_account.Credentials) credentials directly.
Methods
__init__(query[, project, ...])
Initialize BigQuery document loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
langchain_community.document_loaders.rspace.RSpaceLoader¶
class langchain_community.document_loaders.rspace.RSpaceLoader(global_id: str, api_key: Optional[str] = None, url: Optional[str] = None)[source]¶
Load content from RSpace notebooks, folders, documents or PDF Gallery files.
Map RSpace document <-> Langchain Document in 1-1. PDFs are imported using PyPDF.
Requirements are rspace_client (pip install rspace_client) and PyPDF if importingPDF docs (pip install pypdf).
api_key: RSpace API key - can also be supplied as environment variable
‘RSPACE_API_KEY’
url: str
The URL of your RSpace instance - can also be supplied as environment
variable ‘RSPACE_URL’
global_id: str
The global ID of the resource to load,
e.g. ‘SD12344’ (a single document); ‘GL12345’(A PDF file in the gallery);
‘NB4567’ (a notebook); ‘FL12244’ (a folder)
Methods
__init__(global_id[, api_key, url])
api_key: RSpace API key - can also be supplied as environment variable 'RSPACE_API_KEY' url: str The URL of your RSpace instance - can also be supplied as environment variable 'RSPACE_URL' global_id: str  The global ID of the resource to load, e.g. 'SD12344' (a single document); 'GL12345'(A PDF file in the gallery); 'NB4567' (a notebook); 'FL12244' (a folder).
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
validate_environment(values)
Validate that API key and URL exist in environment.
Parameters
global_id (str) – 
api_key (Optional[str]) – 
url (Optional[str]) – 
__init__(global_id: str, api_key: Optional[str] = None, url: Optional[str] = None)[source]¶
api_key: RSpace API key - can also be supplied as environment variable
‘RSPACE_API_KEY’
url: str
The URL of your RSpace instance - can also be supplied as environment
variable ‘RSPACE_URL’
global_id: str
The global ID of the resource to load,
e.g. ‘SD12344’ (a single document); ‘GL12345’(A PDF file in the gallery);
‘NB4567’ (a notebook); ‘FL12244’ (a folder)
Parameters
global_id (str) – 
api_key (Optional[str]) – 
url (Optional[str]) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
classmethod validate_environment(values: Dict) → Dict[source]¶
Validate that API key and URL exist in environment.
Parameters
values (Dict) – 
Return type
Dict
Examples using RSpaceLoader¶
# replace these ids with some from your own research notes.
langchain_community.document_loaders.base_o365.O365BaseLoader¶
class langchain_community.document_loaders.base_o365.O365BaseLoader[source]¶
Bases: BaseLoader, BaseModel
Base class for all loaders that uses O365 Package
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param auth_with_token: bool = False¶
Whether to authenticate with a token or not. Defaults to False.
param chunk_size: Union[int, str] = 5242880¶
Number of bytes to retrieve from each api call to the server. int or ‘auto’.
param recursive: bool = False¶
Should the loader recursively load subfolders?
param settings: _O365Settings [Optional]¶
Settings for the Office365 API client.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
langchain_community.document_loaders.pdf.PyMuPDFLoader¶
class langchain_community.document_loaders.pdf.PyMuPDFLoader(file_path: str, *, headers: Optional[Dict] = None, extract_images: bool = False, **kwargs: Any)[source]¶
Load PDF files using PyMuPDF.
Initialize with a file path.
Attributes
source
Methods
__init__(file_path, *[, headers, extract_images])
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load(**kwargs)
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (str) – 
headers (Optional[Dict]) – 
extract_images (bool) – 
kwargs (Any) – 
__init__(file_path: str, *, headers: Optional[Dict] = None, extract_images: bool = False, **kwargs: Any) → None[source]¶
Initialize with a file path.
Parameters
file_path (str) – 
headers (Optional[Dict]) – 
extract_images (bool) – 
kwargs (Any) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load(**kwargs: Any) → List[Document][source]¶
Load data into Document objects.
Parameters
kwargs (Any) – 
Return type
Load data into Document objects.
Parameters
kwargs (Any) – 
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using PyMuPDFLoader¶
How to load PDFs
langchain_community.document_loaders.email.UnstructuredEmailLoader¶
class langchain_community.document_loaders.email.UnstructuredEmailLoader(file_path: Union[str, Path], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load email files using Unstructured.
Works with both
.eml and .msg files. You can process attachments in addition to the
e-mail message itself by passing process_attachments=True into the
constructor for the loader. By default, attachments will be processed
with the unstructured partition function. If you already know the document
types of the attachments, you can specify another partitioning function
with the attachment partitioner kwarg.
Example
from langchain_community.document_loaders import UnstructuredEmailLoader
loader = UnstructuredEmailLoader(“example_data/fake-email.eml”, mode=”elements”)
loader.load()
Example
from langchain_community.document_loaders import UnstructuredEmailLoader
loader = UnstructuredEmailLoader(“example_data/fake-email-attachment.eml”,
mode=”elements”,
process_attachments=True,
)
loader.load()
Initialize with file path.
Methods
__init__(file_path[, mode])
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (Union[str, Path]) – 
mode (str) – 
unstructured_kwargs (Any) – 
__init__(file_path: Union[str, Path], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Initialize with file path.
Parameters
file_path (Union[str, Path]) – 
mode (str) –
Parameters
file_path (Union[str, Path]) – 
mode (str) – 
unstructured_kwargs (Any) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using UnstructuredEmailLoader¶
Email
Unstructured
langchain_community.document_loaders.bibtex.BibtexLoader¶
class langchain_community.document_loaders.bibtex.BibtexLoader(file_path: str, *, parser: Optional[BibtexparserWrapper] = None, max_docs: Optional[int] = None, max_content_chars: Optional[int] = 4000, load_extra_metadata: bool = False, file_pattern: str = '[^:]+\\.pdf')[source]¶
Load a bibtex file.
Each document represents one entry from the bibtex file.
If a PDF file is present in the file bibtex field, the original PDF
is loaded into the document text. If no such file entry is present,
the abstract field is used instead.
Initialize the BibtexLoader.
Parameters
file_path (str) – Path to the bibtex file.
parser (Optional[BibtexparserWrapper]) – The parser to use. If None, a default parser is used.
max_docs (Optional[int]) – Max number of associated documents to load. Use -1 means
no limit.
max_content_chars (Optional[int]) – Maximum number of characters to load from the PDF.
load_extra_metadata (bool) – Whether to load extra metadata from the PDF.
file_pattern (str) – Regex pattern to match the file name in the bibtex.
Methods
__init__(file_path, *[, parser, max_docs, ...])
Initialize the BibtexLoader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load bibtex file using bibtexparser and get the article texts plus the article metadata.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, *, parser: Optional[BibtexparserWrapper] = None, max_docs: Optional[int] = None, max_content_chars: Optional[int] = 4000, load_extra_metadata: bool = False, file_pattern: str = '[^:]+\\.pdf')[source]¶
Initialize the BibtexLoader.
Parameters
file_path (str) – Path to the bibtex file.
parser (Optional[BibtexparserWrapper]) – The parser to use. If None, a default parser is used.
max_docs (Optional[int]) – Max number of associated documents to load. Use -1 means
no limit.
max_content_chars (Optional[int]) – Maximum number of characters to load from the PDF.
load_extra_metadata (bool) – Whether to load extra metadata from the PDF.
file_pattern (str) – Regex pattern to match the file name in the bibtex.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load bibtex file using bibtexparser and get the article texts plus the
article metadata.
See https://bibtexparser.readthedocs.io/en/master/
Returns
a list of documents with the document.page_content in text format
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
langchain_community.document_loaders.browserbase.BrowserbaseLoader¶
class langchain_community.document_loaders.browserbase.BrowserbaseLoader(urls: Sequence[str], text_content: bool = False, api_key: Optional[str] = None, project_id: Optional[str] = None, session_id: Optional[str] = None, proxy: Optional[bool] = None)[source]¶
Load pre-rendered web pages using a headless browser hosted on Browserbase.
Depends on browserbase package.
Get your API key from https://browserbase.com
Methods
__init__(urls[, text_content, api_key, ...])
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load pages from URLs
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
urls (Sequence[str]) – 
text_content (bool) – 
api_key (Optional[str]) – 
project_id (Optional[str]) – 
session_id (Optional[str]) – 
proxy (Optional[bool]) – 
__init__(urls: Sequence[str], text_content: bool = False, api_key: Optional[str] = None, project_id: Optional[str] = None, session_id: Optional[str] = None, proxy: Optional[bool] = None)[source]¶
Parameters
urls (Sequence[str]) – 
text_content (bool) – 
api_key (Optional[str]) – 
project_id (Optional[str]) – 
session_id (Optional[str]) – 
proxy (Optional[bool]) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load pages from URLs
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using BrowserbaseLoader¶
Browserbase
langchain_community.document_loaders.recursive_url_loader.RecursiveUrlLoader¶
class langchain_community.document_loaders.recursive_url_loader.RecursiveUrlLoader(url: str, max_depth: Optional[int] = 2, use_async: Optional[bool] = None, extractor: Optional[Callable[[str], str]] = None, metadata_extractor: Optional[Union[Callable[[str, str], dict], Callable[[str, str, Union[Response, ClientResponse]], dict]]] = None, exclude_dirs: Optional[Sequence[str]] = (), timeout: Optional[int] = 10, prevent_outside: bool = True, link_regex: Optional[Union[str, Pattern]] = None, headers: Optional[dict] = None, check_response_status: bool = False, continue_on_failure: bool = True, *, base_url: Optional[str] = None, autoset_encoding: bool = True, encoding: Optional[str] = None)[source]¶
Load all child links from a URL page.
Security Note: This loader is a crawler that will start crawlingat a given URL and then expand to crawl child links recursively.
Web crawlers should generally NOT be deployed with network access
to any internal servers.
Control access to who can submit crawling requests and what network access
the crawler has.
While crawling, the crawler may encounter malicious URLs that would lead to a
server-side request forgery (SSRF) attack.
To mitigate risks, the crawler by default will only load URLs from the same
domain as the start URL (controlled via prevent_outside named argument).
This will mitigate the risk of SSRF attacks, but will not eliminate it.
For example, if crawling a host which hosts several sites:
https://some_host/alice_site/
https://some_host/bob_site/
A malicious URL on Alice’s site could cause the crawler to make a malicious
A malicious URL on Alice’s site could cause the crawler to make a malicious
GET request to an endpoint on Bob’s site. Both sites are hosted on the
same host, so such a request would not be prevented by default.
See https://python.langchain.com/docs/security
Initialize with URL to crawl and any subdirectories to exclude.
Parameters
url (str) – The URL to crawl.
max_depth (Optional[int]) – The max depth of the recursive loading.
use_async (Optional[bool]) – Whether to use asynchronous loading.
If True, this function will not be lazy, but it will still work in the
expected way, just not lazy.
extractor (Optional[Callable[[str], str]]) – A function to extract document contents from raw html.
When extract function returns an empty string, the document is
ignored.
metadata_extractor (Optional[_MetadataExtractorType]) – A function to extract metadata from args: raw html, the
source url, and the requests.Response/aiohttp.ClientResponse object
(args in that order).
Default extractor will attempt to use BeautifulSoup4 to extract the
title, description and language of the page.
..code-block:: python
import requests
import aiohttp
def simple_metadata_extractor(raw_html: str, url: str, response: Union[requests.Response, aiohttp.ClientResponse]
) -> dict:content_type = getattr(response, “headers”).get(“Content-Type”, “”)
return {“source”: url, “content_type”: content_type}
exclude_dirs (Optional[Sequence[str]]) – A list of subdirectories to exclude.
timeout (Optional[int]) – The timeout for the requests, in the unit of seconds. If None then
connection will not timeout.
prevent_outside (bool) – If True, prevent loading from urls which are not children
of the root url.