Tony20100/langchain_full huggingface dataset:
A reasonable default function to convert a record into a “metadata” dictionary.
Parameters
row (RowMapping) – 
column_names (Optional[List[str]]) – 
Return type
Dict[str, Any]
static page_content_default_mapper(row: RowMapping, column_names: Optional[List[str]] = None) → str[source]¶
A reasonable default function to convert a record into a “page content” string.
Parameters
row (RowMapping) – 
column_names (Optional[List[str]]) – 
Return type
str
langchain_community.document_loaders.blockchain.BlockchainDocumentLoader¶
class langchain_community.document_loaders.blockchain.BlockchainDocumentLoader(contract_address: str, blockchainType: BlockchainType = BlockchainType.ETH_MAINNET, api_key: str = 'docs-demo', startToken: str = '', get_all_tokens: bool = False, max_execution_time: Optional[int] = None)[source]¶
Load elements from a blockchain smart contract.
The supported blockchains are: Ethereum mainnet, Ethereum Goerli testnet,
Polygon mainnet, and Polygon Mumbai testnet.
If no BlockchainType is specified, the default is Ethereum mainnet.
The Loader uses the Alchemy API to interact with the blockchain.
ALCHEMY_API_KEY environment variable must be set to use this loader.
The API returns 100 NFTs per request and can be paginated using the
startToken parameter.
If get_all_tokens is set to True, the loader will get all tokens
on the contract.  Note that for contracts with a large number of tokens,
this may take a long time (e.g. 10k tokens is 100 requests).
Default value is false for this reason.
The max_execution_time (sec) can be set to limit the execution time
of the loader.
Future versions of this loader can:
Support additional Alchemy APIs (e.g. getTransactions, etc.)
Support additional blockain APIs (e.g. Infura, Opensea, etc.)
Parameters
contract_address (str) – The address of the smart contract.
blockchainType (BlockchainType) – The blockchain type.
api_key (str) – The Alchemy API key.
startToken (str) – The start token for pagination.
get_all_tokens (bool) – Whether to get all tokens on the contract.
max_execution_time (Optional[int]) – The maximum execution time (sec).
Methods
max_execution_time (Optional[int]) – The maximum execution time (sec).
Methods
__init__(contract_address[, blockchainType, ...])
param contract_address
The address of the smart contract.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(contract_address: str, blockchainType: BlockchainType = BlockchainType.ETH_MAINNET, api_key: str = 'docs-demo', startToken: str = '', get_all_tokens: bool = False, max_execution_time: Optional[int] = None)[source]¶
Parameters
contract_address (str) – The address of the smart contract.
blockchainType (BlockchainType) – The blockchain type.
api_key (str) – The Alchemy API key.
startToken (str) – The start token for pagination.
get_all_tokens (bool) – Whether to get all tokens on the contract.
max_execution_time (Optional[int]) – The maximum execution time (sec).
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using BlockchainDocumentLoader¶
Alchemy
Blockchain
langchain_community.document_loaders.blockchain.BlockchainType¶
class langchain_community.document_loaders.blockchain.BlockchainType(value)[source]¶
Enumerator of the supported blockchains.
ETH_MAINNET = 'eth-mainnet'¶
ETH_GOERLI = 'eth-goerli'¶
POLYGON_MAINNET = 'polygon-mainnet'¶
POLYGON_MUMBAI = 'polygon-mumbai'¶
Examples using BlockchainType¶
Alchemy
Blockchain
langchain_community.document_loaders.json_loader.JSONLoader¶
class langchain_community.document_loaders.json_loader.JSONLoader(file_path: Union[str, Path], jq_schema: str, content_key: Optional[str] = None, is_content_key_jq_parsable: Optional[bool] = False, metadata_func: Optional[Callable[[Dict, Dict], Dict]] = None, text_content: bool = True, json_lines: bool = False)[source]¶
Load a JSON file using a jq schema.
Example
[{“text”: …}, {“text”: …}, {“text”: …}] -> schema = .[].text
{“key”: [{“text”: …}, {“text”: …}, {“text”: …}]} -> schema = .key[].text
[“”, “”, “”] -> schema = .[]
Initialize the JSONLoader.
Parameters
file_path (Union[str, Path]) – The path to the JSON or JSON Lines file.
jq_schema (str) – The jq schema to use to extract the data or text from
the JSON.
content_key (str) – The key to use to extract the content from
the JSON if the jq_schema results to a list of objects (dict).
If is_content_key_jq_parsable is True, this has to be a jq compatible
schema. If is_content_key_jq_parsable is False, this should be a simple
string key.
is_content_key_jq_parsable (bool) – A flag to determine if
content_key is parsable by jq or not. If True, content_key is
treated as a jq schema and compiled accordingly. If False or if
content_key is None, content_key is used as a simple string.
Default is False.
metadata_func (Callable[Dict, Dict]) – A function that takes in the JSON
object extracted by the jq_schema and the default metadata and returns
object extracted by the jq_schema and the default metadata and returns
a dict of the updated metadata.
text_content (bool) – Boolean flag to indicate whether the content is in
string format, default to True.
json_lines (bool) – Boolean flag to indicate whether the input is in
JSON Lines format.
Methods
__init__(file_path, jq_schema[, ...])
Initialize the JSONLoader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load and return documents from the JSON file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path], jq_schema: str, content_key: Optional[str] = None, is_content_key_jq_parsable: Optional[bool] = False, metadata_func: Optional[Callable[[Dict, Dict], Dict]] = None, text_content: bool = True, json_lines: bool = False)[source]¶
Initialize the JSONLoader.
Parameters
file_path (Union[str, Path]) – The path to the JSON or JSON Lines file.
jq_schema (str) – The jq schema to use to extract the data or text from
the JSON.
content_key (str) – The key to use to extract the content from
the JSON if the jq_schema results to a list of objects (dict).
If is_content_key_jq_parsable is True, this has to be a jq compatible
schema. If is_content_key_jq_parsable is False, this should be a simple
string key.
is_content_key_jq_parsable (bool) – A flag to determine if
content_key is parsable by jq or not. If True, content_key is
treated as a jq schema and compiled accordingly. If False or if
treated as a jq schema and compiled accordingly. If False or if
content_key is None, content_key is used as a simple string.
Default is False.
metadata_func (Callable[Dict, Dict]) – A function that takes in the JSON
object extracted by the jq_schema and the default metadata and returns
a dict of the updated metadata.
text_content (bool) – Boolean flag to indicate whether the content is in
string format, default to True.
json_lines (bool) – Boolean flag to indicate whether the input is in
JSON Lines format.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load and return documents from the JSON file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using JSONLoader¶
How to load JSON
Timescale Vector (Postgres)
langchain_community.document_loaders.chm.UnstructuredCHMLoader¶
class langchain_community.document_loaders.chm.UnstructuredCHMLoader(file_path: Optional[Union[str, List[str], Path, List[Path]]], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load CHM files using Unstructured.
CHM means Microsoft Compiled HTML Help.
Examples
from langchain_community.document_loaders import UnstructuredCHMLoader
loader = UnstructuredCHMLoader(“example.chm”)
docs = loader.load()
References
https://github.com/dottedmag/pychm
http://www.jedrea.com/chmlib/
Initialize with file path.
Methods
__init__(file_path[, mode])
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (Optional[Union[str, List[str], Path, List[Path]]]) – 
mode (str) – 
unstructured_kwargs (Any) – 
__init__(file_path: Optional[Union[str, List[str], Path, List[Path]]], mode: str = 'single', **unstructured_kwargs: Any)¶
Initialize with file path.
Parameters
file_path (Optional[Union[str, List[str], Path, List[Path]]]) – 
mode (str) – 
unstructured_kwargs (Any) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using UnstructuredCHMLoader¶
Unstructured
langchain_community.document_loaders.markdown.UnstructuredMarkdownLoader¶
class langchain_community.document_loaders.markdown.UnstructuredMarkdownLoader(file_path: Optional[Union[str, List[str], Path, List[Path]]], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load Markdown files using Unstructured.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain_community.document_loaders import UnstructuredMarkdownLoader
loader = UnstructuredMarkdownLoader(“example.md”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/core/partition.html#partition-md
Initialize with file path.
Methods
__init__(file_path[, mode])
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (Optional[Union[str, List[str], Path, List[Path]]]) – 
mode (str) – 
unstructured_kwargs (Any) – 
__init__(file_path: Optional[Union[str, List[str], Path, List[Path]]], mode: str = 'single', **unstructured_kwargs: Any)¶
Initialize with file path.
Parameters
Initialize with file path.
Parameters
file_path (Optional[Union[str, List[str], Path, List[Path]]]) – 
mode (str) – 
unstructured_kwargs (Any) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using UnstructuredMarkdownLoader¶
Apache Doris
How to load Markdown
StarRocks
Unstructured
langchain
langchain_community.document_loaders.lakefs.LakeFSLoader¶
class langchain_community.document_loaders.lakefs.LakeFSLoader(lakefs_access_key: str, lakefs_secret_key: str, lakefs_endpoint: str, repo: Optional[str] = None, ref: Optional[str] = 'main', path: Optional[str] = '')[source]¶
Load from lakeFS.
Parameters
lakefs_access_key (str) – [required] lakeFS server’s access key
lakefs_secret_key (str) – [required] lakeFS server’s secret key
lakefs_endpoint (str) – [required] lakeFS server’s endpoint address,
ex: https://example.my-lakefs.com
repo (str) – [optional, default = ‘’] target repository
ref (str) – [optional, default = ‘main’] target ref (branch name,
tag, or commit ID)
path (str) – [optional, default = ‘’] target path
Attributes
repo
ref
path
Methods
__init__(lakefs_access_key, ...[, repo, ...])
param lakefs_access_key
[required] lakeFS server's access key
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
set_path(path)
set_ref(ref)
set_repo(repo)
__init__(lakefs_access_key: str, lakefs_secret_key: str, lakefs_endpoint: str, repo: Optional[str] = None, ref: Optional[str] = 'main', path: Optional[str] = '')[source]¶
Parameters
Parameters
lakefs_access_key (str) – [required] lakeFS server’s access key
lakefs_secret_key (str) – [required] lakeFS server’s secret key
lakefs_endpoint (str) – [required] lakeFS server’s endpoint address,
ex: https://example.my-lakefs.com
repo (Optional[str]) – [optional, default = ‘’] target repository
ref (Optional[str]) – [optional, default = ‘main’] target ref (branch name,
tag, or commit ID)
path (Optional[str]) – [optional, default = ‘’] target path
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
set_path(path: str) → None[source]¶
Parameters
path (str) – 
Return type
None
set_ref(ref: str) → None[source]¶
Parameters
ref (str) – 
Return type
None
set_repo(repo: str) → None[source]¶
langchain_community.document_loaders.parsers.html.bs4.BS4HTMLParser¶
class langchain_community.document_loaders.parsers.html.bs4.BS4HTMLParser(*, features: str = 'lxml', get_text_separator: str = '', **kwargs: Any)[source]¶
Parse HTML files using Beautiful Soup.
Initialize a bs4 based HTML parser.
Methods
__init__(*[, features, get_text_separator])
Initialize a bs4 based HTML parser.
lazy_parse(blob)
Load HTML document into document objects.
parse(blob)
Eagerly parse the blob into a document or documents.
Parameters
features (str) – 
get_text_separator (str) – 
kwargs (Any) – 
__init__(*, features: str = 'lxml', get_text_separator: str = '', **kwargs: Any) → None[source]¶
Initialize a bs4 based HTML parser.
Parameters
features (str) – 
get_text_separator (str) – 
kwargs (Any) – 
Return type
None
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Load HTML document into document objects.
Parameters
blob (Blob) – 
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
langchain_community.document_loaders.parsers.language.javascript.JavaScriptSegmenter¶
class langchain_community.document_loaders.parsers.language.javascript.JavaScriptSegmenter(code: str)[source]¶
Code segmenter for JavaScript.
Methods
__init__(code)
extract_functions_classes()
is_valid()
simplify_code()
Parameters
code (str) – 
__init__(code: str)[source]¶
Parameters
code (str) – 
extract_functions_classes() → List[str][source]¶
Return type
List[str]
is_valid() → bool[source]¶
Return type
bool
simplify_code() → str[source]¶
Return type
str
langchain_community.document_loaders.obsidian.ObsidianLoader¶
class langchain_community.document_loaders.obsidian.ObsidianLoader(path: Union[str, Path], encoding: str = 'UTF-8', collect_metadata: bool = True)[source]¶
Load Obsidian files from directory.
Initialize with a path.
Parameters
path (Union[str, Path]) – Path to the directory containing the Obsidian files.
encoding (str) – Charset encoding, defaults to “UTF-8”
collect_metadata (bool) – Whether to collect metadata from the front matter.
Defaults to True.
Attributes
DATAVIEW_INLINE_BRACKET_REGEX
DATAVIEW_INLINE_PAREN_REGEX
DATAVIEW_LINE_REGEX
FRONT_MATTER_REGEX
TAG_REGEX
TEMPLATE_VARIABLE_REGEX
Methods
__init__(path[, encoding, collect_metadata])
Initialize with a path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(path: Union[str, Path], encoding: str = 'UTF-8', collect_metadata: bool = True)[source]¶
Initialize with a path.
Parameters
path (Union[str, Path]) – Path to the directory containing the Obsidian files.
encoding (str) – Charset encoding, defaults to “UTF-8”
collect_metadata (bool) – Whether to collect metadata from the front matter.
Defaults to True.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using ObsidianLoader¶
Obsidian
langchain_community.document_loaders.sharepoint.SharePointLoader¶
class langchain_community.document_loaders.sharepoint.SharePointLoader[source]¶
Bases: O365BaseLoader, BaseLoader
Load  from SharePoint.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param auth_with_token: bool = False¶
Whether to authenticate with a token or not. Defaults to False.
param chunk_size: Union[int, str] = 5242880¶
Number of bytes to retrieve from each api call to the server. int or ‘auto’.
param document_library_id: str [Required]¶
The ID of the SharePoint document library to load data from.
param file_id: Optional[str] = None¶
The ID of the file for which we need auth identities
param folder_id: Optional[str] = None¶
The ID of the folder to load data from.
param folder_path: Optional[str] = None¶
The path to the folder to load data from.
param load_auth: Optional[bool] = False¶
Whether to load authorization identities.
param object_ids: Optional[List[str]] = None¶
The IDs of the objects to load data from.
param recursive: bool = False¶
Should the loader recursively load subfolders?
param settings: _O365Settings [Optional]¶
Settings for the Office365 API client.
param site_id: Optional[str] = None¶
The ID of the Sharepoint site of the user where the file is present
param token_path: Path = PosixPath('/home/runner/.credentials/o365_token.txt')¶
The path to the token to make api calls
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
authorized_identities() → List[source]¶
Return type
List
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to include in new model
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to exclude from new model, as with values this takes precedence over include
update (Optional[DictStrAny]) – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
deep (bool) – set to True to make a deep copy of the model
self (Model) – 
Returns
new model instance
Return type
Model