Tony20100/langchain_full huggingface dataset:
langchain_community.document_loaders.trello.TrelloLoader¶
class langchain_community.document_loaders.trello.TrelloLoader(client: TrelloClient, board_name: str, *, include_card_name: bool = True, include_comments: bool = True, include_checklist: bool = True, card_filter: Literal['closed', 'open', 'all'] = 'all', extra_metadata: Tuple[str, ...] = ('due_date', 'labels', 'list', 'closed'))[source]¶
Load cards from a Trello board.
Initialize Trello loader.
Parameters
client (TrelloClient) – Trello API client.
board_name (str) – The name of the Trello board.
include_card_name (bool) – Whether to include the name of the card in the document.
include_comments (bool) – Whether to include the comments on the card in the
document.
include_checklist (bool) – Whether to include the checklist on the card in the
document.
card_filter (Literal['closed', 'open', 'all']) – Filter on card status. Valid values are “closed”, “open”,
“all”.
extra_metadata (Tuple[str, ...]) – List of additional metadata fields to include as document
metadata.Valid values are “due_date”, “labels”, “list”, “closed”.
Methods
__init__(client, board_name, *[, ...])
Initialize Trello loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
from_credentials(board_name, *[, api_key, token])
Convenience constructor that builds TrelloClient init param for you.
lazy_load()
Loads all cards from the specified Trello board.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(client: TrelloClient, board_name: str, *, include_card_name: bool = True, include_comments: bool = True, include_checklist: bool = True, card_filter: Literal['closed', 'open', 'all'] = 'all', extra_metadata: Tuple[str, ...] = ('due_date', 'labels', 'list', 'closed'))[source]¶
Initialize Trello loader.
Parameters
client (TrelloClient) – Trello API client.
board_name (str) – The name of the Trello board.
include_card_name (bool) – Whether to include the name of the card in the document.
include_comments (bool) – Whether to include the comments on the card in the
document.
include_checklist (bool) – Whether to include the checklist on the card in the
document.
card_filter (Literal['closed', 'open', 'all']) – Filter on card status. Valid values are “closed”, “open”,
“all”.
extra_metadata (Tuple[str, ...]) – List of additional metadata fields to include as document
metadata.Valid values are “due_date”, “labels”, “list”, “closed”.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
classmethod from_credentials(board_name: str, *, api_key: Optional[str] = None, token: Optional[str] = None, **kwargs: Any) → TrelloLoader[source]¶
Convenience constructor that builds TrelloClient init param for you.
Parameters
board_name (str) – The name of the Trello board.
Parameters
board_name (str) – The name of the Trello board.
api_key (Optional[str]) – Trello API key. Can also be specified as environment variable
TRELLO_API_KEY.
token (Optional[str]) – Trello token. Can also be specified as environment variable
TRELLO_TOKEN.
include_card_name – Whether to include the name of the card in the document.
include_comments – Whether to include the comments on the card in the
document.
include_checklist – Whether to include the checklist on the card in the
document.
card_filter – Filter on card status. Valid values are “closed”, “open”,
“all”.
extra_metadata – List of additional metadata fields to include as document
metadata.Valid values are “due_date”, “labels”, “list”, “closed”.
kwargs (Any) – 
Return type
TrelloLoader
lazy_load() → Iterator[Document][source]¶
Loads all cards from the specified Trello board.
You can filter the cards, metadata and text included by using the optional
parameters.
Returns:A list of documents, one for each card in the board.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using TrelloLoader¶
Trello
langchain_community.document_loaders.googledrive.GoogleDriveLoader¶
class langchain_community.document_loaders.googledrive.GoogleDriveLoader[source]¶
Bases: BaseLoader, BaseModel
[Deprecated] Load Google Docs from Google Drive.
Notes
Deprecated since version 0.0.32.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param credentials_path: Path = PosixPath('/home/runner/.credentials/credentials.json')¶
Path to the credentials file.
param document_ids: Optional[List[str]] = None¶
The document ids to load from.
param file_ids: Optional[List[str]] = None¶
The file ids to load from.
param file_loader_cls: Any = None¶
The file loader class to use.
param file_loader_kwargs: Dict[str, Any] = {}¶
The file loader kwargs to use.
param file_types: Optional[Sequence[str]] = None¶
The file types to load. Only applies when folder_id is given.
param folder_id: Optional[str] = None¶
The folder id to load from.
param load_trashed_files: bool = False¶
Whether to load trashed files. Only applies when folder_id is given.
param recursive: bool = False¶
Whether to load recursively. Only applies when folder_id is given.
param service_account_key: Path = PosixPath('/home/runner/.credentials/keys.json')¶
Path to the service account key file.
param token_path: Path = PosixPath('/home/runner/.credentials/token.json')¶
Path to the token file.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to include in new model
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to exclude from new model, as with values this takes precedence over include
update (Optional[DictStrAny]) – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
deep (bool) – set to True to make a deep copy of the model
self (Model) – 
Returns
new model instance
Return type
Model
langchain_community.document_loaders.helpers.detect_file_encodings¶
langchain_community.document_loaders.helpers.detect_file_encodings(file_path: Union[str, Path], timeout: int = 5) → List[FileEncoding][source]¶
Try to detect the file encoding.
Returns a list of FileEncoding tuples with the detected encodings ordered
by confidence.
Parameters
file_path (Union[str, Path]) – The path to the file to detect the encoding for.
timeout (int) – The timeout in seconds for the encoding detection.
Return type
List[FileEncoding]
langchain_community.document_loaders.parsers.language.cobol.CobolSegmenter¶
class langchain_community.document_loaders.parsers.language.cobol.CobolSegmenter(code: str)[source]¶
Code segmenter for COBOL.
Attributes
DIVISION_PATTERN
PARAGRAPH_PATTERN
SECTION_PATTERN
Methods
__init__(code)
extract_functions_classes()
is_valid()
simplify_code()
Parameters
code (str) – 
__init__(code: str)[source]¶
Parameters
code (str) – 
extract_functions_classes() → List[str][source]¶
Return type
List[str]
is_valid() → bool[source]¶
Return type
bool
simplify_code() → str[source]¶
Return type
str
langchain_community.document_loaders.doc_intelligence.AzureAIDocumentIntelligenceLoader¶
class langchain_community.document_loaders.doc_intelligence.AzureAIDocumentIntelligenceLoader(api_endpoint: str, api_key: str, file_path: Optional[str] = None, url_path: Optional[str] = None, api_version: Optional[str] = None, api_model: str = 'prebuilt-layout', mode: str = 'markdown', *, analysis_features: Optional[List[str]] = None)[source]¶
Load a PDF with Azure Document Intelligence.
Initialize the object for file processing with Azure Document Intelligence
(formerly Form Recognizer).
This constructor initializes a AzureAIDocumentIntelligenceParser object to be
used for parsing files using the Azure Document Intelligence API. The load
method generates Documents whose content representations are determined by the
mode parameter.
Parameters:¶
api_endpoint: strThe API endpoint to use for DocumentIntelligenceClient construction.
api_key: strThe API key to use for DocumentIntelligenceClient construction.
file_pathOptional[str]The path to the file that needs to be loaded.
Either file_path or url_path must be specified.
url_pathOptional[str]The URL to the file that needs to be loaded.
Either file_path or url_path must be specified.
api_version: Optional[str]The API version for DocumentIntelligenceClient. Setting None to use
the default value from azure-ai-documentintelligence package.
api_model: strUnique document model name. Default value is “prebuilt-layout”.
Note that overriding this default value may result in unsupported
behavior.
mode: Optional[str]The type of content representation of the generated Documents.
Use either “single”, “page”, or “markdown”. Default value is “markdown”.
analysis_features: Optional[List[str]]List of optional analysis features, each feature should be passed
as a str that conforms to the enum DocumentAnalysisFeature in
as a str that conforms to the enum DocumentAnalysisFeature in
azure-ai-documentintelligence package. Default value is None.
Examples:¶
>>> obj = AzureAIDocumentIntelligenceLoader(
...     file_path="path/to/file",
...     api_endpoint="https://endpoint.azure.com",
...     api_key="APIKEY",
...     api_version="2023-10-31-preview",
...     api_model="prebuilt-layout",
...     mode="markdown"
... )
Methods
__init__(api_endpoint, api_key[, file_path, ...])
Initialize the object for file processing with Azure Document Intelligence (formerly Form Recognizer).
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load given path as pages.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(api_endpoint: str, api_key: str, file_path: Optional[str] = None, url_path: Optional[str] = None, api_version: Optional[str] = None, api_model: str = 'prebuilt-layout', mode: str = 'markdown', *, analysis_features: Optional[List[str]] = None) → None[source]¶
Initialize the object for file processing with Azure Document Intelligence
(formerly Form Recognizer).
This constructor initializes a AzureAIDocumentIntelligenceParser object to be
used for parsing files using the Azure Document Intelligence API. The load
method generates Documents whose content representations are determined by the
mode parameter.
Parameters:¶
api_endpoint: strThe API endpoint to use for DocumentIntelligenceClient construction.
api_key: strThe API key to use for DocumentIntelligenceClient construction.
file_pathOptional[str]The path to the file that needs to be loaded.
file_pathOptional[str]The path to the file that needs to be loaded.
Either file_path or url_path must be specified.
url_pathOptional[str]The URL to the file that needs to be loaded.
Either file_path or url_path must be specified.
api_version: Optional[str]The API version for DocumentIntelligenceClient. Setting None to use
the default value from azure-ai-documentintelligence package.
api_model: strUnique document model name. Default value is “prebuilt-layout”.
Note that overriding this default value may result in unsupported
behavior.
mode: Optional[str]The type of content representation of the generated Documents.
Use either “single”, “page”, or “markdown”. Default value is “markdown”.
analysis_features: Optional[List[str]]List of optional analysis features, each feature should be passed
as a str that conforms to the enum DocumentAnalysisFeature in
azure-ai-documentintelligence package. Default value is None.
Examples:¶
>>> obj = AzureAIDocumentIntelligenceLoader(
...     file_path="path/to/file",
...     api_endpoint="https://endpoint.azure.com",
...     api_key="APIKEY",
...     api_version="2023-10-31-preview",
...     api_model="prebuilt-layout",
...     mode="markdown"
... )
Parameters
api_endpoint (str) – 
api_key (str) – 
file_path (Optional[str]) – 
url_path (Optional[str]) – 
api_version (Optional[str]) – 
api_model (str) – 
mode (str) – 
analysis_features (Optional[List[str]]) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load given path as pages.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Parameters
api_endpoint (str) – 
api_key (str) – 
file_path (Optional[str]) – 
url_path (Optional[str]) – 
api_version (Optional[str]) – 
api_model (str) – 
mode (str) – 
analysis_features (Optional[List[str]]) – 
Examples using AzureAIDocumentIntelligenceLoader¶
Azure AI Document Intelligence
How to load Microsoft Office files
How to load PDFs
Microsoft Excel
Microsoft PowerPoint
Microsoft Word
langchain_community.document_loaders.cube_semantic.CubeSemanticLoader¶
class langchain_community.document_loaders.cube_semantic.CubeSemanticLoader(cube_api_url: str, cube_api_token: str, load_dimension_values: bool = True, dimension_values_limit: int = 10000, dimension_values_max_retries: int = 10, dimension_values_retry_delay: int = 3)[source]¶
Load Cube semantic layer metadata.
Parameters
cube_api_url (str) – REST API endpoint.
Use the REST API of your Cube’s deployment.
Please find out more information here:
https://cube.dev/docs/http-api/rest#configuration-base-path
cube_api_token (str) – Cube API token.
Authentication tokens are generated based on your Cube’s API secret.
Please find out more information here:
https://cube.dev/docs/security#generating-json-web-tokens-jwt
load_dimension_values (bool) – Whether to load dimension values for every string
dimension or not.
dimension_values_limit (int) – Maximum number of dimension values to load.
dimension_values_max_retries (int) – Maximum number of retries to load dimension
values.
dimension_values_retry_delay (int) – Delay between retries to load dimension values.
Methods
__init__(cube_api_url, cube_api_token[, ...])
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Makes a call to Cube's REST API metadata endpoint.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(cube_api_url: str, cube_api_token: str, load_dimension_values: bool = True, dimension_values_limit: int = 10000, dimension_values_max_retries: int = 10, dimension_values_retry_delay: int = 3)[source]¶
Parameters
cube_api_url (str) – 
cube_api_token (str) – 
load_dimension_values (bool) – 
dimension_values_limit (int) – 
dimension_values_max_retries (int) – 
dimension_values_retry_delay (int) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Makes a call to Cube’s REST API metadata endpoint.
Returns
page_content=column_title + column_description
metadata
table_name
column_name
column_data_type
column_member_type
column_title
column_description
column_values
cube_data_obj_type
Return type
A list of documents with attributes
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using CubeSemanticLoader¶
Cube
langchain_community.document_loaders.parsers.pdf.DocumentIntelligenceParser¶
class langchain_community.document_loaders.parsers.pdf.DocumentIntelligenceParser(client: Any, model: str)[source]¶
Loads a PDF with Azure Document Intelligence
(formerly Form Recognizer) and chunks at character level.
Methods
__init__(client, model)
lazy_parse(blob)
Lazily parse the blob.
parse(blob)
Eagerly parse the blob into a document or documents.
Parameters
client (Any) – 
model (str) – 
__init__(client: Any, model: str)[source]¶
Parameters
client (Any) – 
model (str) – 
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazily parse the blob.
Parameters
blob (Blob) – 
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
langchain_community.document_loaders.unstructured.UnstructuredAPIFileIOLoader¶
class langchain_community.document_loaders.unstructured.UnstructuredAPIFileIOLoader(file: Union[IO, Sequence[IO]], mode: str = 'single', url: str = 'https://api.unstructured.io/general/v0/general', api_key: str = '', **unstructured_kwargs: Any)[source]¶
Load files using Unstructured API.
By default, the loader makes a call to the hosted Unstructured API.
If you are running the unstructured API locally, you can change the
API rule by passing in the url parameter when you initialize the loader.
The hosted Unstructured API requires an API key. See
https://www.unstructured.io/api-key/ if you need to generate a key.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain_community.document_loaders import UnstructuredAPIFileLoader
with open(“example.pdf”, “rb”) as f:
loader = UnstructuredFileAPILoader(f, mode=”elements”, strategy=”fast”, api_key=”MY_API_KEY”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition
https://www.unstructured.io/api-key/
https://github.com/Unstructured-IO/unstructured-api
Initialize with file path.
Methods
__init__(file[, mode, url, api_key])
Initialize with file path.
alazy_load()
A lazy loader for Documents.
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file (Union[IO, Sequence[IO]]) – 
mode (str) – 
url (str) – 
api_key (str) – 
unstructured_kwargs (Any) – 
__init__(file: Union[IO, Sequence[IO]], mode: str = 'single', url: str = 'https://api.unstructured.io/general/v0/general', api_key: str = '', **unstructured_kwargs: Any)[source]¶
Initialize with file path.
Parameters
file (Union[IO, Sequence[IO]]) – 
mode (str) – 
url (str) – 
api_key (str) – 
unstructured_kwargs (Any) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
langchain_community.document_loaders.arxiv.ArxivLoader¶
class langchain_community.document_loaders.arxiv.ArxivLoader(query: str, doc_content_chars_max: Optional[int] = None, **kwargs: Any)[source]¶
Load a query result from Arxiv.
The loader converts the original PDF format into the text.
Parameters
ArxivAPIWrapper. (Supports all arguments of) – 
query (str) – 
doc_content_chars_max (Optional[int]) – 
kwargs (Any) – 
Methods
__init__(query[, doc_content_chars_max])
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
get_summaries_as_docs()
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(query: str, doc_content_chars_max: Optional[int] = None, **kwargs: Any)[source]¶
Parameters
query (str) – 
doc_content_chars_max (Optional[int]) – 
kwargs (Any) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
get_summaries_as_docs() → List[Document][source]¶
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶