Tony20100/langchain_full huggingface dataset:
langchain_community.document_loaders.max_compute.MaxComputeLoader¶
class langchain_community.document_loaders.max_compute.MaxComputeLoader(query: str, api_wrapper: MaxComputeAPIWrapper, *, page_content_columns: Optional[Sequence[str]] = None, metadata_columns: Optional[Sequence[str]] = None)[source]¶
Load from Alibaba Cloud MaxCompute table.
Initialize Alibaba Cloud MaxCompute document loader.
Parameters
query (str) – SQL query to execute.
api_wrapper (MaxComputeAPIWrapper) – MaxCompute API wrapper.
page_content_columns (Optional[Sequence[str]]) – The columns to write into the page_content of the
Document. If unspecified, all columns will be written to page_content.
metadata_columns (Optional[Sequence[str]]) – The columns to write into the metadata of the Document.
If unspecified, all columns not added to page_content will be written.
Methods
__init__(query, api_wrapper, *[, ...])
Initialize Alibaba Cloud MaxCompute document loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
from_params(query, endpoint, project, *[, ...])
Convenience constructor that builds the MaxCompute API wrapper from
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(query: str, api_wrapper: MaxComputeAPIWrapper, *, page_content_columns: Optional[Sequence[str]] = None, metadata_columns: Optional[Sequence[str]] = None)[source]¶
Initialize Alibaba Cloud MaxCompute document loader.
Parameters
query (str) – SQL query to execute.
api_wrapper (MaxComputeAPIWrapper) – MaxCompute API wrapper.
page_content_columns (Optional[Sequence[str]]) – The columns to write into the page_content of the
Document. If unspecified, all columns will be written to page_content.
metadata_columns (Optional[Sequence[str]]) – The columns to write into the metadata of the Document.
If unspecified, all columns not added to page_content will be written.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
classmethod from_params(query: str, endpoint: str, project: str, *, access_id: Optional[str] = None, secret_access_key: Optional[str] = None, **kwargs: Any) → MaxComputeLoader[source]¶
Convenience constructor that builds the MaxCompute API wrapper fromgiven parameters.
Parameters
query (str) – SQL query to execute.
endpoint (str) – MaxCompute endpoint.
project (str) – A project is a basic organizational unit of MaxCompute, which is
similar to a database.
access_id (Optional[str]) – MaxCompute access ID. Should be passed in directly or set as the
environment variable MAX_COMPUTE_ACCESS_ID.
secret_access_key (Optional[str]) – MaxCompute secret access key. Should be passed in
directly or set as the environment variable
MAX_COMPUTE_SECRET_ACCESS_KEY.
kwargs (Any) – 
Return type
MaxComputeLoader
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using MaxComputeLoader¶
Alibaba Cloud
Alibaba Cloud MaxCompute
langchain_community.document_loaders.cassandra.CassandraLoader¶
class langchain_community.document_loaders.cassandra.CassandraLoader(table: Optional[str] = None, session: Optional[Session] = None, keyspace: Optional[str] = None, query: Union[str, Statement, None] = None, page_content_mapper: Callable[[Any], str] = <class 'str'>, metadata_mapper: Callable[[Any], dict] = <function CassandraLoader.<lambda>>, *, query_parameters: Union[dict, Sequence, None] = None, query_timeout: Optional[float] = <object object>, query_trace: bool = False, query_custom_payload: Optional[dict] = None, query_execution_profile: Any = <object object>, query_paging_state: Any = None, query_host: Optional[Host] = None, query_execute_as: Optional[str] = None)[source]¶
Document Loader for Apache Cassandra.
Parameters
table (Optional[str]) – The table to load the data from.
(do not use together with the query parameter)
session (Optional[Session]) – The cassandra driver session.
If not provided, the cassio resolved session will be used.
keyspace (Optional[str]) – The keyspace of the table.
If not provided, the cassio resolved keyspace will be used.
query (Union[str, Statement, None]) – The query used to load the data.
(do not use together with the table parameter)
page_content_mapper (Callable[[Any], str]) – a function to convert a row to string page content.
Defaults to the str representation of the row.
metadata_mapper (Callable[[Any], dict]) – a function to convert a row to document metadata.
query_parameters (Union[dict, Sequence, None]) – The query parameters used when calling session.execute .
query_timeout (Optional[float]) – The query timeout used when calling session.execute .
query_timeout (Optional[float]) – The query timeout used when calling session.execute .
query_trace (bool) – Whether to use tracing when calling session.execute .
query_custom_payload (Optional[dict]) – The query custom_payload used when calling
session.execute .
query_execution_profile (Any) – The query execution_profile used when calling
session.execute .
query_host (Optional[Host]) – The query host used when calling session.execute .
query_execute_as (Optional[str]) – The query execute_as used when calling session.execute .
query_paging_state (Any) – 
Methods
__init__([table, session, keyspace, query, ...])
Document Loader for Apache Cassandra.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(table: Optional[str] = None, session: Optional[Session] = None, keyspace: Optional[str] = None, query: Union[str, Statement, None] = None, page_content_mapper: Callable[[Any], str] = <class 'str'>, metadata_mapper: Callable[[Any], dict] = <function CassandraLoader.<lambda>>, *, query_parameters: Union[dict, Sequence, None] = None, query_timeout: Optional[float] = <object object>, query_trace: bool = False, query_custom_payload: Optional[dict] = None, query_execution_profile: Any = <object object>, query_paging_state: Any = None, query_host: Optional[Host] = None, query_execute_as: Optional[str] = None) → None[source]¶
Document Loader for Apache Cassandra.
Parameters
table (Optional[str]) – The table to load the data from.
Parameters
table (Optional[str]) – The table to load the data from.
(do not use together with the query parameter)
session (Optional[Session]) – The cassandra driver session.
If not provided, the cassio resolved session will be used.
keyspace (Optional[str]) – The keyspace of the table.
If not provided, the cassio resolved keyspace will be used.
query (Union[str, Statement, None]) – The query used to load the data.
(do not use together with the table parameter)
page_content_mapper (Callable[[Any], str]) – a function to convert a row to string page content.
Defaults to the str representation of the row.
metadata_mapper (Callable[[Any], dict]) – a function to convert a row to document metadata.
query_parameters (Union[dict, Sequence, None]) – The query parameters used when calling session.execute .
query_timeout (Optional[float]) – The query timeout used when calling session.execute .
query_trace (bool) – Whether to use tracing when calling session.execute .
query_custom_payload (Optional[dict]) – The query custom_payload used when calling
session.execute .
query_execution_profile (Any) – The query execution_profile used when calling
session.execute .
query_host (Optional[Host]) – The query host used when calling session.execute .
query_execute_as (Optional[str]) – The query execute_as used when calling session.execute .
query_paging_state (Any) – 
Return type
None
async alazy_load() → AsyncIterator[Document][source]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using CassandraLoader¶
Cassandra
langchain_community.document_loaders.pdf.DocumentIntelligenceLoader¶
class langchain_community.document_loaders.pdf.DocumentIntelligenceLoader(file_path: str, client: Any, model: str = 'prebuilt-document', headers: Optional[Dict] = None)[source]¶
Load a PDF with Azure Document Intelligence
Initialize the object for file processing with Azure Document Intelligence
(formerly Form Recognizer).
This constructor initializes a DocumentIntelligenceParser object to be used
for parsing files using the Azure Document Intelligence API. The load method
generates a Document node including metadata (source blob and page number)
for each page.
Parameters:¶
file_pathstrThe path to the file that needs to be parsed.
client: AnyA DocumentAnalysisClient to perform the analysis of the blob
modelstrThe model name or ID to be used for form recognition in Azure.
Examples:¶
>>> obj = DocumentIntelligenceLoader(
...     file_path="path/to/file",
...     client=client,
...     model="prebuilt-document"
... )
Attributes
source
Methods
__init__(file_path, client[, model, headers])
Initialize the object for file processing with Azure Document Intelligence (formerly Form Recognizer).
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load given path as pages.
load()
Load given path as pages.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, client: Any, model: str = 'prebuilt-document', headers: Optional[Dict] = None) → None[source]¶
Initialize the object for file processing with Azure Document Intelligence
(formerly Form Recognizer).
This constructor initializes a DocumentIntelligenceParser object to be used
This constructor initializes a DocumentIntelligenceParser object to be used
for parsing files using the Azure Document Intelligence API. The load method
generates a Document node including metadata (source blob and page number)
for each page.
Parameters:¶
file_pathstrThe path to the file that needs to be parsed.
client: AnyA DocumentAnalysisClient to perform the analysis of the blob
modelstrThe model name or ID to be used for form recognition in Azure.
Examples:¶
>>> obj = DocumentIntelligenceLoader(
...     file_path="path/to/file",
...     client=client,
...     model="prebuilt-document"
... )
Parameters
file_path (str) – 
client (Any) – 
model (str) – 
headers (Optional[Dict]) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load given path as pages.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load given path as pages.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Parameters
file_path (str) –
langchain_community.document_loaders.onedrive.OneDriveLoader¶
class langchain_community.document_loaders.onedrive.OneDriveLoader[source]¶
Bases: O365BaseLoader
Load from Microsoft OneDrive.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param auth_with_token: bool = False¶
Whether to authenticate with a token or not. Defaults to False.
param chunk_size: Union[int, str] = 5242880¶
Number of bytes to retrieve from each api call to the server. int or ‘auto’.
param drive_id: str [Required]¶
The ID of the OneDrive drive to load data from.
param folder_path: Optional[str] = None¶
The path to the folder to load data from.
param object_ids: Optional[List[str]] = None¶
The IDs of the objects to load data from.
param recursive: bool = False¶
Should the loader recursively load subfolders?
param settings: _O365Settings [Optional]¶
Settings for the Office365 API client.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
Return type
DictStrAny
classmethod from_orm(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
by_alias (bool) – 
skip_defaults (Optional[bool]) – 
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
encoder (Optional[Callable[[Any], Any]]) – 
models_as_dict (bool) – 
dumps_kwargs (Any) – 
Return type
unicode
lazy_load() → Iterator[Document][source]¶
Load documents lazily. Use this when working at a large scale.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
path (Union[str, Path]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod parse_obj(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
b (Union[str, bytes]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
Return type
DictStrAny
ref_template (unicode) – 
Return type
DictStrAny
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
dumps_kwargs (Any) – 
Return type
unicode
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
Parameters
localns (Any) – 
Return type
None
classmethod validate(value: Any) → Model¶
Parameters
value (Any) – 
Return type
Model
Examples using OneDriveLoader¶
Microsoft
Microsoft OneDrive
langchain_community.document_loaders.airtable.AirtableLoader¶
class langchain_community.document_loaders.airtable.AirtableLoader(api_token: str, table_id: str, base_id: str, **kwargs: Any)[source]¶
Load the Airtable tables.
Initialize with API token and the IDs for table and base.
Parameters
api_token (str) – Airtable API token.
table_id (str) – Airtable table ID.
base_id (str) – 
**kwargs (Any) – Additional parameters to pass to Table.all(). Refer to the
pyairtable documentation for available options:
https://pyairtable.readthedocs.io/en/latest/api.html#pyairtable.Table.all
Methods
__init__(api_token, table_id, base_id, **kwargs)
Initialize with API token and the IDs for table and base.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load Documents from table.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(api_token: str, table_id: str, base_id: str, **kwargs: Any) → None[source]¶
Initialize with API token and the IDs for table and base.
Parameters
api_token (str) – Airtable API token.
table_id (str) – Airtable table ID.
base_id (str) – 
**kwargs (Any) – Additional parameters to pass to Table.all(). Refer to the
pyairtable documentation for available options:
https://pyairtable.readthedocs.io/en/latest/api.html#pyairtable.Table.all
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load Documents from table.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using AirtableLoader¶
Airtable
langchain_community.document_loaders.etherscan.EtherscanLoader¶
class langchain_community.document_loaders.etherscan.EtherscanLoader(account_address: str, api_key: str = 'docs-demo', filter: str = 'normal_transaction', page: int = 1, offset: int = 10, start_block: int = 0, end_block: int = 99999999, sort: str = 'desc')[source]¶
Load transactions from Ethereum mainnet.
The Loader use Etherscan API to interact with Ethereum mainnet.
ETHERSCAN_API_KEY environment variable must be set use this loader.
Methods
__init__(account_address[, api_key, filter, ...])
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
getERC1155Tx()
getERC20Tx()
getERC721Tx()
getEthBalance()
getInternalTx()
getNormTx()
lazy_load()
Lazy load Documents from table.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
account_address (str) – 
api_key (str) – 
filter (str) – 
page (int) – 
offset (int) – 
start_block (int) – 
end_block (int) – 
sort (str) – 
__init__(account_address: str, api_key: str = 'docs-demo', filter: str = 'normal_transaction', page: int = 1, offset: int = 10, start_block: int = 0, end_block: int = 99999999, sort: str = 'desc')[source]¶
Parameters
account_address (str) – 
api_key (str) – 
filter (str) –
api_key (str) – 
filter (str) – 
page (int) – 
offset (int) – 
start_block (int) – 
end_block (int) – 
sort (str) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
getERC1155Tx() → List[Document][source]¶
Return type
List[Document]
getERC20Tx() → List[Document][source]¶
Return type
List[Document]
getERC721Tx() → List[Document][source]¶
Return type
List[Document]
getEthBalance() → List[Document][source]¶
Return type
List[Document]
getInternalTx() → List[Document][source]¶
Return type
List[Document]
getNormTx() → List[Document][source]¶
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load Documents from table.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using EtherscanLoader¶
Etherscan
langchain_community.document_loaders.youtube.GoogleApiClient¶
class langchain_community.document_loaders.youtube.GoogleApiClient(credentials_path: Path = PosixPath('/home/runner/.credentials/credentials.json'), service_account_path: Path = PosixPath('/home/runner/.credentials/credentials.json'), token_path: Path = PosixPath('/home/runner/.credentials/token.json'))[source]¶
Generic Google API Client.
To use, you should have the google_auth_oauthlib,youtube_transcript_api,google
python package installed.
As the google api expects credentials you need to set up a google account and
register your Service. “https://developers.google.com/docs/api/quickstart/python”
Example
from langchain_community.document_loaders import GoogleApiClient
google_api_client = GoogleApiClient(
    service_account_path=Path("path_to_your_sec_file.json")
)
Attributes
credentials_path
service_account_path
token_path
Methods
__init__([credentials_path, ...])
validate_channel_or_videoIds_is_set(values)
Validate that either folder_id or document_ids is set, but not both.
Parameters
credentials_path (Path) – 
service_account_path (Path) – 
token_path (Path) – 
__init__(credentials_path: Path = PosixPath('/home/runner/.credentials/credentials.json'), service_account_path: Path = PosixPath('/home/runner/.credentials/credentials.json'), token_path: Path = PosixPath('/home/runner/.credentials/token.json')) → None¶
Parameters
credentials_path (Path) – 
service_account_path (Path) – 
token_path (Path) – 
Return type
None
classmethod validate_channel_or_videoIds_is_set(values: Dict[str, Any]) → Dict[str, Any][source]¶
Validate that either folder_id or document_ids is set, but not both.
Parameters