Tony20100/langchain_full huggingface dataset:
langchain_community.document_loaders.mastodon.MastodonTootsLoader¶
class langchain_community.document_loaders.mastodon.MastodonTootsLoader(mastodon_accounts: Sequence[str], number_toots: Optional[int] = 100, exclude_replies: bool = False, access_token: Optional[str] = None, api_base_url: str = 'https://mastodon.social')[source]¶
Load the Mastodon ‘toots’.
Instantiate Mastodon toots loader.
Parameters
mastodon_accounts (Sequence[str]) – The list of Mastodon accounts to query.
number_toots (Optional[int]) – How many toots to pull for each account. Defaults to 100.
exclude_replies (bool) – Whether to exclude reply toots from the load.
Defaults to False.
access_token (Optional[str]) – An access token if toots are loaded as a Mastodon app. Can
also be specified via the environment variables “MASTODON_ACCESS_TOKEN”.
api_base_url (str) – A Mastodon API base URL to talk to, if not using the default.
Defaults to “https://mastodon.social”.
Methods
__init__(mastodon_accounts[, number_toots, ...])
Instantiate Mastodon toots loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load toots into documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(mastodon_accounts: Sequence[str], number_toots: Optional[int] = 100, exclude_replies: bool = False, access_token: Optional[str] = None, api_base_url: str = 'https://mastodon.social')[source]¶
Instantiate Mastodon toots loader.
Parameters
Instantiate Mastodon toots loader.
Parameters
mastodon_accounts (Sequence[str]) – The list of Mastodon accounts to query.
number_toots (Optional[int]) – How many toots to pull for each account. Defaults to 100.
exclude_replies (bool) – Whether to exclude reply toots from the load.
Defaults to False.
access_token (Optional[str]) – An access token if toots are loaded as a Mastodon app. Can
also be specified via the environment variables “MASTODON_ACCESS_TOKEN”.
api_base_url (str) – A Mastodon API base URL to talk to, if not using the default.
Defaults to “https://mastodon.social”.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load toots into documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using MastodonTootsLoader¶
Mastodon
langchain_community.document_loaders.xorbits.XorbitsLoader¶
class langchain_community.document_loaders.xorbits.XorbitsLoader(data_frame: Any, page_content_column: str = 'text')[source]¶
Load Xorbits DataFrame.
Initialize with dataframe object.
Requirements:Must have xorbits installed. You can install with pip install xorbits.
Parameters
data_frame (Any) – Xorbits DataFrame object.
page_content_column (str) – Name of the column containing the page content.
Defaults to “text”.
Methods
__init__(data_frame[, page_content_column])
Initialize with dataframe object.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load records from dataframe.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(data_frame: Any, page_content_column: str = 'text')[source]¶
Initialize with dataframe object.
Requirements:Must have xorbits installed. You can install with pip install xorbits.
Parameters
data_frame (Any) – Xorbits DataFrame object.
page_content_column (str) – Name of the column containing the page content.
Defaults to “text”.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Lazy load records from dataframe.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using XorbitsLoader¶
Xorbits Pandas DataFrame
langchain_community.document_loaders.airbyte.AirbyteSalesforceLoader¶
class langchain_community.document_loaders.airbyte.AirbyteSalesforceLoader(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None)[source]¶
Load from Salesforce using an Airbyte source connector.
Initializes the loader.
Parameters
config (Mapping[str, Any]) – The config to pass to the source connector.
stream_name (str) – The name of the stream to load.
record_handler (Optional[Callable[[Any, Optional[str]], Document]]) – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state (Optional[Any]) – The state to pass to the source connector. Defaults to None.
Attributes
last_state
Methods
__init__(config, stream_name[, ...])
Initializes the loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None) → None[source]¶
Initializes the loader.
Parameters
config (Mapping[str, Any]) – The config to pass to the source connector.
stream_name (str) – The name of the stream to load.
record_handler (Optional[Callable[[Any, Optional[str]], Document]]) – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state (Optional[Any]) – The state to pass to the source connector. Defaults to None.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using AirbyteSalesforceLoader¶
Airbyte Salesforce (Deprecated)
langchain_community.document_loaders.rss.RSSFeedLoader¶
class langchain_community.document_loaders.rss.RSSFeedLoader(urls: Optional[Sequence[str]] = None, opml: Optional[str] = None, continue_on_failure: bool = True, show_progress_bar: bool = False, **newsloader_kwargs: Any)[source]¶
Load news articles from RSS feeds using Unstructured.
Parameters
urls (Optional[Sequence[str]]) – URLs for RSS feeds to load. Each articles in the feed is loaded into its own document.
opml (Optional[str]) – OPML file to load feed urls from. Only one of urls or opml should be provided.  The value
string (can be a URL) – 
string. (or OPML markup contents as byte or) – 
continue_on_failure (bool) – If True, continue loading documents even if
loading fails for a particular URL.
show_progress_bar (bool) – If True, use tqdm to show a loading progress bar. Requires
tqdm to be installed, pip install tqdm.
**newsloader_kwargs (Any) – Any additional named arguments to pass to
NewsURLLoader.
Example
from langchain_community.document_loaders import RSSFeedLoader
loader = RSSFeedLoader(
    urls=["<url-1>", "<url-2>"],
)
docs = loader.load()
The loader uses feedparser to parse RSS feeds.  The feedparser library is not installed by default so you should
install it if using this loader:
https://pythonhosted.org/feedparser/
If you use OPML, you should also install listparser:
https://pythonhosted.org/listparser/
Finally, newspaper is used to process each article:
https://newspaper.readthedocs.io/en/latest/
Initialize with urls or OPML.
Methods
Initialize with urls or OPML.
Methods
__init__([urls, opml, continue_on_failure, ...])
Initialize with urls or OPML.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(urls: Optional[Sequence[str]] = None, opml: Optional[str] = None, continue_on_failure: bool = True, show_progress_bar: bool = False, **newsloader_kwargs: Any) → None[source]¶
Initialize with urls or OPML.
Parameters
urls (Optional[Sequence[str]]) – 
opml (Optional[str]) – 
continue_on_failure (bool) – 
show_progress_bar (bool) – 
newsloader_kwargs (Any) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
langchain_community.document_loaders.fauna.FaunaLoader¶
class langchain_community.document_loaders.fauna.FaunaLoader(query: str, page_content_field: str, secret: str, metadata_fields: Optional[Sequence[str]] = None)[source]¶
Load from FaunaDB.
Parameters
query (str) – 
page_content_field (str) – 
secret (str) – 
metadata_fields (Optional[Sequence[str]]) – 
query¶
The FQL query string to execute.
Type
str
page_content_field¶
The field that contains the content of each page.
Type
str
secret¶
The secret key for authenticating to FaunaDB.
Type
str
metadata_fields¶
Optional list of field names to include in metadata.
Type
Optional[Sequence[str]]
Methods
__init__(query, page_content_field, secret)
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(query: str, page_content_field: str, secret: str, metadata_fields: Optional[Sequence[str]] = None)[source]¶
Parameters
query (str) – 
page_content_field (str) – 
secret (str) – 
metadata_fields (Optional[Sequence[str]]) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using FaunaLoader¶
Fauna
langchain_community.document_loaders.mintbase.MintbaseDocumentLoader¶
class langchain_community.document_loaders.mintbase.MintbaseDocumentLoader(contract_address: str, *, blockchain_type: Literal['mainnet', 'testnet'], api_key: str = '', table: str = '', select: str = '', fields: Optional[List[str]] = None, get_all_tokens: bool = False, max_execution_time: Optional[int] = None)[source]¶
Load elements from a blockchain smart contract.
The supported blockchains are: Near mainnet, Near testnet.
If no BlockchainType is specified, the default is Near mainnet.
The Loader uses the Mintbase API to interact with the blockchain.
MB_API_KEY environment variable must be set to use this loader.
The API returns 100 NFTs per request and can be paginated using the
startToken parameter.
If get_all_tokens is set to True, the loader will get all tokens
on the contract.  Note that for contracts with a large number of tokens,
this may take a long time (e.g. 10k tokens is 100 requests).
Default value is false for this reason.
The max_execution_time (sec) can be set to limit the execution time
of the loader.
Future versions of this loader can:
Support additional Mintbase APIs (e.g. getTokens, etc.)
Example
contractAddress = "nft.yearofchef.near"  # Year of chef contract address
blockchainLoader = MintbaseDocumentLoader(
    contract_address=contractAddress, blockchain_type="mainnet",api_key="omni-site"
)
Parameters
contract_address (str) – The address of the smart contract.
blockchainType – The blockchain type.
api_key (str) – The Mintbase API key.
table (str) – name of the table to query
table (str) – name of the table to query
select (str) – Conditions for querying
fields (Optional[List[str]]) – Information to display after query
get_all_tokens (bool) – Whether to get all tokens on the contract.
max_execution_time (Optional[int]) – The maximum execution time (sec).
blockchain_type (Literal['mainnet', 'testnet']) – 
Methods
__init__(contract_address, *, blockchain_type)
param contract_address
The address of the smart contract.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(contract_address: str, *, blockchain_type: Literal['mainnet', 'testnet'], api_key: str = '', table: str = '', select: str = '', fields: Optional[List[str]] = None, get_all_tokens: bool = False, max_execution_time: Optional[int] = None)[source]¶
Parameters
contract_address (str) – The address of the smart contract.
blockchainType – The blockchain type.
api_key (str) – The Mintbase API key.
table (str) – name of the table to query
select (str) – Conditions for querying
fields (Optional[List[str]]) – Information to display after query
get_all_tokens (bool) – Whether to get all tokens on the contract.
max_execution_time (Optional[int]) – The maximum execution time (sec).
blockchain_type (Literal['mainnet', 'testnet']) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
langchain_community.document_loaders.lakefs.LakeFSClient¶
class langchain_community.document_loaders.lakefs.LakeFSClient(lakefs_access_key: str, lakefs_secret_key: str, lakefs_endpoint: str)[source]¶
Client for lakeFS.
Methods
__init__(lakefs_access_key, ...)
is_presign_supported()
ls_objects(repo, ref, path, presign)
Parameters
lakefs_access_key (str) – 
lakefs_secret_key (str) – 
lakefs_endpoint (str) – 
__init__(lakefs_access_key: str, lakefs_secret_key: str, lakefs_endpoint: str)[source]¶
Parameters
lakefs_access_key (str) – 
lakefs_secret_key (str) – 
lakefs_endpoint (str) – 
is_presign_supported() → bool[source]¶
Return type
bool
ls_objects(repo: str, ref: str, path: str, presign: Optional[bool]) → List[source]¶
Parameters
repo (str) – 
ref (str) – 
path (str) – 
presign (Optional[bool]) – 
Return type
List
langchain_community.document_loaders.gcs_file.GCSFileLoader¶
class langchain_community.document_loaders.gcs_file.GCSFileLoader(project_name: str, bucket: str, blob: str, loader_func: Optional[Callable[[str], BaseLoader]] = None)[source]¶
[Deprecated] Load from GCS file.
Notes
Deprecated since version 0.0.32.
Initialize with bucket and key name.
Parameters
project_name (str) – The name of the project to load
bucket (str) – The name of the GCS bucket.
blob (str) – The name of the GCS blob to load.
loader_func (Optional[Callable[[str], BaseLoader]]) – A loader function that instantiates a loader based on a
file_path argument. If nothing is provided, the
UnstructuredFileLoader is used.
Examples
To use an alternative PDF loader:
>> from from langchain_community.document_loaders import PyPDFLoader
>> loader = GCSFileLoader(…, loader_func=PyPDFLoader)
To use UnstructuredFileLoader with additional arguments:
>> loader = GCSFileLoader(…,
>>      loader_func=lambda x: UnstructuredFileLoader(x, mode=”elements”))
Methods
__init__(project_name, bucket, blob[, ...])
Initialize with bucket and key name.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(project_name: str, bucket: str, blob: str, loader_func: Optional[Callable[[str], BaseLoader]] = None)[source]¶
Initialize with bucket and key name.
Parameters
Initialize with bucket and key name.
Parameters
project_name (str) – The name of the project to load
bucket (str) – The name of the GCS bucket.
blob (str) – The name of the GCS blob to load.
loader_func (Optional[Callable[[str], BaseLoader]]) – A loader function that instantiates a loader based on a
file_path argument. If nothing is provided, the
UnstructuredFileLoader is used.
Examples
To use an alternative PDF loader:
>> from from langchain_community.document_loaders import PyPDFLoader
>> loader = GCSFileLoader(…, loader_func=PyPDFLoader)
To use UnstructuredFileLoader with additional arguments:
>> loader = GCSFileLoader(…,
>>      loader_func=lambda x: UnstructuredFileLoader(x, mode=”elements”))
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
langchain_community.document_loaders.unstructured.validate_unstructured_version¶
langchain_community.document_loaders.unstructured.validate_unstructured_version(min_unstructured_version: str) → None[source]¶
Raise an error if the Unstructured version does not exceed the
specified minimum.
Parameters
min_unstructured_version (str) – 
Return type
None
langchain_community.document_loaders.pubmed.PubMedLoader¶
class langchain_community.document_loaders.pubmed.PubMedLoader(query: str, load_max_docs: Optional[int] = 3)[source]¶
Load from the PubMed biomedical library.
query¶
The query to be passed to the PubMed API.
load_max_docs¶
The maximum number of documents to load.
Initialize the PubMedLoader.
Parameters
query (str) – The query to be passed to the PubMed API.
load_max_docs (Optional[int]) – The maximum number of documents to load.
Defaults to 3.
Methods
__init__(query[, load_max_docs])
Initialize the PubMedLoader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(query: str, load_max_docs: Optional[int] = 3)[source]¶
Initialize the PubMedLoader.
Parameters
query (str) – The query to be passed to the PubMed API.
load_max_docs (Optional[int]) – The maximum number of documents to load.
Defaults to 3.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using PubMedLoader¶
PubMed
langchain_community.document_loaders.pdf.PDFPlumberLoader¶
class langchain_community.document_loaders.pdf.PDFPlumberLoader(file_path: str, text_kwargs: Optional[Mapping[str, Any]] = None, dedupe: bool = False, headers: Optional[Dict] = None, extract_images: bool = False)[source]¶
Load PDF files using pdfplumber.
Initialize with a file path.
Attributes
source
Methods
__init__(file_path[, text_kwargs, dedupe, ...])
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load file.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (str) – 
text_kwargs (Optional[Mapping[str, Any]]) – 
dedupe (bool) – 
headers (Optional[Dict]) – 
extract_images (bool) – 
__init__(file_path: str, text_kwargs: Optional[Mapping[str, Any]] = None, dedupe: bool = False, headers: Optional[Dict] = None, extract_images: bool = False) → None[source]¶
Initialize with a file path.
Parameters
file_path (str) – 
text_kwargs (Optional[Mapping[str, Any]]) – 
dedupe (bool) – 
headers (Optional[Dict]) – 
extract_images (bool) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶