Tony20100/langchain_full huggingface dataset:
you can pass an already-created ‘astrapy.db.AsyncAstraDB’ instance.
namespace (Optional[str]) – namespace (aka keyspace) where the collection resides.
If not provided, the environment variable ASTRA_DB_KEYSPACE is
inspected. Defaults to the database’s “default namespace”.
filter_criteria (Optional[Dict[str, Any]]) – Criteria to filter documents.
projection (Optional[Dict[str, Any]]) – Specifies the fields to return. If not provided, reads
fall back to the Data API default projection.
find_options (Optional[Dict[str, Any]]) – Additional options for the query.
nb_prefetched (int) – Max number of documents to pre-fetch. Defaults to 1000.
page_content_mapper (Callable[[Dict], str]) – Function applied to collection documents to create
the page_content of the LangChain Document. Defaults to json.dumps.
metadata_mapper (Optional[Callable[[Dict], Dict[str, Any]]]) – 
Return type
None
async alazy_load() → AsyncIterator[Document][source]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
langchain_community.document_loaders.tencent_cos_directory.TencentCOSDirectoryLoader¶
class langchain_community.document_loaders.tencent_cos_directory.TencentCOSDirectoryLoader(conf: Any, bucket: str, prefix: str = '')[source]¶
Load from Tencent Cloud COS directory.
Initialize with COS config, bucket and prefix.
:param conf(CosConfig): COS config.
:param bucket(str): COS bucket.
:param prefix(str): prefix.
Methods
__init__(conf, bucket[, prefix])
Initialize with COS config, bucket and prefix.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
conf (Any) – 
bucket (str) – 
prefix (str) – 
__init__(conf: Any, bucket: str, prefix: str = '')[source]¶
Initialize with COS config, bucket and prefix.
:param conf(CosConfig): COS config.
:param bucket(str): COS bucket.
:param prefix(str): prefix.
Parameters
conf (Any) – 
bucket (str) – 
prefix (str) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using TencentCOSDirectoryLoader¶
Tencent
Tencent COS Directory
langchain_core.document_loaders.base.BaseLoader¶
class langchain_core.document_loaders.base.BaseLoader[source]¶
Interface for Document Loader.
Implementations should implement the lazy-loading method using generators
to avoid loading all Documents into memory at once.
load is provided just for user convenience and should not be overridden.
Methods
__init__()
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__()¶
async alazy_load() → AsyncIterator[Document][source]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document][source]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using BaseLoader¶
How to create a custom Document Loader
How to use the LangChain indexing API
langchain_community.document_loaders.airbyte.AirbyteShopifyLoader¶
class langchain_community.document_loaders.airbyte.AirbyteShopifyLoader(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None)[source]¶
Load from Shopify using an Airbyte source connector.
Initializes the loader.
Parameters
config (Mapping[str, Any]) – The config to pass to the source connector.
stream_name (str) – The name of the stream to load.
record_handler (Optional[Callable[[Any, Optional[str]], Document]]) – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state (Optional[Any]) – The state to pass to the source connector. Defaults to None.
Attributes
last_state
Methods
__init__(config, stream_name[, ...])
Initializes the loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None) → None[source]¶
Initializes the loader.
Parameters
config (Mapping[str, Any]) – The config to pass to the source connector.
stream_name (str) – The name of the stream to load.
record_handler (Optional[Callable[[Any, Optional[str]], Document]]) – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state (Optional[Any]) – The state to pass to the source connector. Defaults to None.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using AirbyteShopifyLoader¶
Airbyte Shopify (Deprecated)
langchain_community.document_loaders.azlyrics.AZLyricsLoader¶
class langchain_community.document_loaders.azlyrics.AZLyricsLoader(web_path: Union[str, Sequence[str]] = '', header_template: Optional[dict] = None, verify_ssl: bool = True, proxies: Optional[dict] = None, continue_on_failure: bool = False, autoset_encoding: bool = True, encoding: Optional[str] = None, web_paths: Sequence[str] = (), requests_per_second: int = 2, default_parser: str = 'html.parser', requests_kwargs: Optional[Dict[str, Any]] = None, raise_for_status: bool = False, bs_get_text_kwargs: Optional[Dict[str, Any]] = None, bs_kwargs: Optional[Dict[str, Any]] = None, session: Optional[Any] = None)[source]¶
Load AZLyrics webpages.
Initialize loader.
Parameters
web_paths (Sequence[str]) – Web paths to load from.
requests_per_second (int) – Max number of concurrent requests to make.
default_parser (str) – Default parser to use for BeautifulSoup.
requests_kwargs (Optional[Dict[str, Any]]) – kwargs for requests
raise_for_status (bool) – Raise an exception if http status code denotes an error.
bs_get_text_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 get_text
bs_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 web page parsing
web_path (Union[str, Sequence[str]]) – 
header_template (Optional[dict]) – 
verify_ssl (bool) – 
proxies (Optional[dict]) – 
continue_on_failure (bool) – 
autoset_encoding (bool) – 
encoding (Optional[str]) – 
session (Any) – 
Attributes
web_path
Methods
session (Any) – 
Attributes
web_path
Methods
__init__([web_path, header_template, ...])
Initialize loader.
alazy_load()
A lazy loader for Documents.
aload()
Load text from the urls in web_path async into Documents.
fetch_all(urls)
Fetch all urls concurrently with rate limiting.
lazy_load()
Lazy load text from the url(s) in web_path.
load()
Load webpages into Documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
scrape([parser])
Scrape data from webpage and return it in BeautifulSoup format.
scrape_all(urls[, parser])
Fetch all urls, then return soups for all results.
__init__(web_path: Union[str, Sequence[str]] = '', header_template: Optional[dict] = None, verify_ssl: bool = True, proxies: Optional[dict] = None, continue_on_failure: bool = False, autoset_encoding: bool = True, encoding: Optional[str] = None, web_paths: Sequence[str] = (), requests_per_second: int = 2, default_parser: str = 'html.parser', requests_kwargs: Optional[Dict[str, Any]] = None, raise_for_status: bool = False, bs_get_text_kwargs: Optional[Dict[str, Any]] = None, bs_kwargs: Optional[Dict[str, Any]] = None, session: Optional[Any] = None) → None¶
Initialize loader.
Parameters
web_paths (Sequence[str]) – Web paths to load from.
requests_per_second (int) – Max number of concurrent requests to make.
default_parser (str) – Default parser to use for BeautifulSoup.
requests_kwargs (Optional[Dict[str, Any]]) – kwargs for requests
raise_for_status (bool) – Raise an exception if http status code denotes an error.
raise_for_status (bool) – Raise an exception if http status code denotes an error.
bs_get_text_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 get_text
bs_kwargs (Optional[Dict[str, Any]]) – kwargs for beatifulsoup4 web page parsing
web_path (Union[str, Sequence[str]]) – 
header_template (Optional[dict]) – 
verify_ssl (bool) – 
proxies (Optional[dict]) – 
continue_on_failure (bool) – 
autoset_encoding (bool) – 
encoding (Optional[str]) – 
session (Optional[Any]) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
aload() → List[Document]¶
Load text from the urls in web_path async into Documents.
Return type
List[Document]
async fetch_all(urls: List[str]) → Any¶
Fetch all urls concurrently with rate limiting.
Parameters
urls (List[str]) – 
Return type
Any
lazy_load() → Iterator[Document]¶
Lazy load text from the url(s) in web_path.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load webpages into Documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Returns
List of Documents.
Return type
List[Document]
scrape(parser: Optional[str] = None) → Any¶
Scrape data from webpage and return it in BeautifulSoup format.
Parameters
parser (Optional[str]) – 
Return type
Any
scrape_all(urls: List[str], parser: Optional[str] = None) → List[Any]¶
Fetch all urls, then return soups for all results.
Parameters
urls (List[str]) – 
parser (Optional[str]) – 
Return type
List[Any]
Examples using AZLyricsLoader¶
AZLyrics
langchain_community.document_loaders.pdf.AmazonTextractPDFLoader¶
class langchain_community.document_loaders.pdf.AmazonTextractPDFLoader(file_path: str, textract_features: Optional[Sequence[str]] = None, client: Optional[Any] = None, credentials_profile_name: Optional[str] = None, region_name: Optional[str] = None, endpoint_url: Optional[str] = None, headers: Optional[Dict] = None, *, linearization_config: Optional[TextLinearizationConfig] = None)[source]¶
Load PDF files from a local file system, HTTP or S3.
To authenticate, the AWS client uses the following methods to
automatically load credentials:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
If a specific credential profile should be used, you must pass
the name of the profile from the ~/.aws/credentials file that is to be used.
Make sure the credentials / roles used have the required policies to
access the Amazon Textract service.
Example
Initialize the loader.
Parameters
file_path (str) – A file, url or s3 path for input file
textract_features (Optional[Sequence[str]]) – Features to be used for extraction, each feature
should be passed as a str that conforms to the enum
Textract_Features, see amazon-textract-caller pkg
client (Optional[Any]) – boto3 textract client (Optional)
credentials_profile_name (Optional[str]) – AWS profile name, if not default (Optional)
region_name (Optional[str]) – AWS region, eg us-east-1 (Optional)
endpoint_url (Optional[str]) – endpoint url for the textract service (Optional)
linearization_config (Optional[TextLinearizationConfig]) – Config to be used for linearization of the output
should be an instance of TextLinearizationConfig from
the textractor pkg
should be an instance of TextLinearizationConfig from
the textractor pkg
headers (Optional[Dict]) – 
Attributes
source
Methods
__init__(file_path[, textract_features, ...])
Initialize the loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load documents
load()
Load given path as pages.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, textract_features: Optional[Sequence[str]] = None, client: Optional[Any] = None, credentials_profile_name: Optional[str] = None, region_name: Optional[str] = None, endpoint_url: Optional[str] = None, headers: Optional[Dict] = None, *, linearization_config: Optional[TextLinearizationConfig] = None) → None[source]¶
Initialize the loader.
Parameters
file_path (str) – A file, url or s3 path for input file
textract_features (Optional[Sequence[str]]) – Features to be used for extraction, each feature
should be passed as a str that conforms to the enum
Textract_Features, see amazon-textract-caller pkg
client (Optional[Any]) – boto3 textract client (Optional)
credentials_profile_name (Optional[str]) – AWS profile name, if not default (Optional)
region_name (Optional[str]) – AWS region, eg us-east-1 (Optional)
endpoint_url (Optional[str]) – endpoint url for the textract service (Optional)
linearization_config (Optional[TextLinearizationConfig]) – Config to be used for linearization of the output
should be an instance of TextLinearizationConfig from
the textractor pkg
headers (Optional[Dict]) – 
Return type
None
the textractor pkg
headers (Optional[Dict]) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load documents
Return type
Iterator[Document]
load() → List[Document][source]¶
Load given path as pages.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using AmazonTextractPDFLoader¶
AWS
Amazon Textract 
How to load PDFs
langchain_community.document_loaders.parsers.audio.YandexSTTParser¶
class langchain_community.document_loaders.parsers.audio.YandexSTTParser(*, api_key: Optional[str] = None, iam_token: Optional[str] = None, model: str = 'general', language: str = 'auto')[source]¶
Transcribe and parse audio files.
Audio transcription is with OpenAI Whisper model.
Initialize the parser.
Parameters
api_key (Optional[str]) – API key for a service account
role. (with the ai.speechkit-stt.user) – 
iam_token (Optional[str]) – IAM token for a service account
role. – 
model (str) – Recognition model name.
Defaults to general.
language (str) – The language in ISO 639-1 format.
Defaults to automatic language recognition.
Either api_key or iam_token must be provided, but not both.
Methods
__init__(*[, api_key, iam_token, model, ...])
Initialize the parser.
lazy_parse(blob)
Lazily parse the blob.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__(*, api_key: Optional[str] = None, iam_token: Optional[str] = None, model: str = 'general', language: str = 'auto')[source]¶
Initialize the parser.
Parameters
api_key (Optional[str]) – API key for a service account
role. (with the ai.speechkit-stt.user) – 
iam_token (Optional[str]) – IAM token for a service account
role. – 
model (str) – Recognition model name.
Defaults to general.
language (str) – The language in ISO 639-1 format.
Defaults to automatic language recognition.
Either api_key or iam_token must be provided, but not both.
Either api_key or iam_token must be provided, but not both.
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazily parse the blob.
Parameters
blob (Blob) – 
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
langchain_community.document_loaders.pdf.OnlinePDFLoader¶
class langchain_community.document_loaders.pdf.OnlinePDFLoader(file_path: Union[str, Path], *, headers: Optional[Dict] = None)[source]¶
Load online PDF.
Initialize with a file path.
Parameters
file_path (Union[str, Path]) – Either a local, S3 or web path to a PDF file.
headers (Optional[Dict]) – Headers to use for GET request to download a file from a web path.
Attributes
source
Methods
__init__(file_path, *[, headers])
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path], *, headers: Optional[Dict] = None)¶
Initialize with a file path.
Parameters
file_path (Union[str, Path]) – Either a local, S3 or web path to a PDF file.
headers (Optional[Dict]) – Headers to use for GET request to download a file from a web path.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using OnlinePDFLoader¶
How to load PDFs
Maritalk
langchain_community.document_loaders.onenote.OneNoteLoader¶
class langchain_community.document_loaders.onenote.OneNoteLoader[source]¶
Bases: BaseLoader, BaseModel
Load pages from OneNote notebooks.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param access_token: str = ''¶
Personal access token
param auth_with_token: bool = False¶
Whether to authenticate with a token or not. Defaults to False.
param authority_url = 'https://login.microsoftonline.com/consumers/'¶
A URL that identifies a token authority
param notebook_name: Optional[str] = None¶
Filter on notebook name
param object_ids: Optional[List[str]] = None¶
The IDs of the objects to load data from.
param onenote_api_base_url: str = 'https://graph.microsoft.com/v1.0/me/onenote'¶
URL of Microsoft Graph API for OneNote
param page_title: Optional[str] = None¶
Filter on section name
param section_name: Optional[str] = None¶
Filter on section name
param settings: _OneNoteGraphSettings [Optional]¶
Settings for the Microsoft Graph API client.
param token_path: FilePath = PosixPath('/home/runner/.credentials/onenote_graph_token.txt')¶
Path to the file where the access token is stored
Constraints
format = file-path
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
dumps_kwargs (Any) – 
Return type
unicode
lazy_load() → Iterator[Document][source]¶
Get pages from OneNote notebooks.
Returns
page_content
metadata
title
Return type
A list of Documents with attributes
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
path (Union[str, Path]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod parse_obj(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
b (Union[str, bytes]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model