Tony20100/langchain_full huggingface dataset:
langchain_community.document_loaders.bilibili.BiliBiliLoader¶
class langchain_community.document_loaders.bilibili.BiliBiliLoader(video_urls: List[str], sessdata: str = '', bili_jct: str = '', buvid3: str = '')[source]¶
Load fetching transcripts from BiliBili videos.
Initialize the loader with BiliBili video URLs and authentication cookies.
if no authentication cookies are provided, the loader can’t get transcripts
and will only fetch videos info.
Parameters
video_urls (List[str]) – List of BiliBili video URLs.
sessdata (str) – SESSDATA cookie value for authentication.
bili_jct (str) – BILI_JCT cookie value for authentication.
buvid3 (str) – BUVI3 cookie value for authentication.
Methods
__init__(video_urls[, sessdata, bili_jct, ...])
Initialize the loader with BiliBili video URLs and authentication cookies.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load and return a list of documents containing video transcripts.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(video_urls: List[str], sessdata: str = '', bili_jct: str = '', buvid3: str = '')[source]¶
Initialize the loader with BiliBili video URLs and authentication cookies.
if no authentication cookies are provided, the loader can’t get transcripts
and will only fetch videos info.
Parameters
video_urls (List[str]) – List of BiliBili video URLs.
sessdata (str) – SESSDATA cookie value for authentication.
sessdata (str) – SESSDATA cookie value for authentication.
bili_jct (str) – BILI_JCT cookie value for authentication.
buvid3 (str) – BUVI3 cookie value for authentication.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load and return a list of documents containing video transcripts.
Returns
List of Document objects transcripts and metadata.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using BiliBiliLoader¶
BiliBili
langchain_community.document_loaders.larksuite.LarkSuiteDocLoader¶
class langchain_community.document_loaders.larksuite.LarkSuiteDocLoader(domain: str, access_token: str, document_id: str)[source]¶
Load from LarkSuite (FeiShu).
Initialize with domain, access_token (tenant / user), and document_id.
Parameters
domain (str) – The domain to load the LarkSuite.
access_token (str) – The access_token to use.
document_id (str) – The document_id to load.
Methods
__init__(domain, access_token, document_id)
Initialize with domain, access_token (tenant / user), and document_id.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load LarkSuite (FeiShu) document.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(domain: str, access_token: str, document_id: str)[source]¶
Initialize with domain, access_token (tenant / user), and document_id.
Parameters
domain (str) – The domain to load the LarkSuite.
access_token (str) – The access_token to use.
document_id (str) – The document_id to load.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load LarkSuite (FeiShu) document.
Return type
Iterator[Document]
load() → List[Document]¶
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using LarkSuiteDocLoader¶
ByteDance
LarkSuite (FeiShu)
langchain_community.document_loaders.vsdx.VsdxLoader¶
class langchain_community.document_loaders.vsdx.VsdxLoader(file_path: Union[str, Path])[source]¶
Initialize with file path.
Methods
__init__(file_path)
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (Union[str, Path]) – 
__init__(file_path: Union[str, Path])[source]¶
Initialize with file path.
Parameters
file_path (Union[str, Path]) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using VsdxLoader¶
Vsdx
langchain_community.document_loaders.reddit.RedditPostsLoader¶
class langchain_community.document_loaders.reddit.RedditPostsLoader(client_id: str, client_secret: str, user_agent: str, search_queries: Sequence[str], mode: str, categories: Sequence[str] = ['new'], number_posts: Optional[int] = 10)[source]¶
Load Reddit posts.
Read posts on a subreddit.
First, you need to go to
https://www.reddit.com/prefs/apps/
and create your application
Initialize with client_id, client_secret, user_agent, search_queries, mode,categories, number_posts.
Example: https://www.reddit.com/r/learnpython/
Parameters
client_id (str) – Reddit client id.
client_secret (str) – Reddit client secret.
user_agent (str) – Reddit user agent.
search_queries (Sequence[str]) – The search queries.
mode (str) – The mode.
categories (Sequence[str]) – The categories. Default: [“new”]
number_posts (Optional[int]) – The number of posts. Default: 10
Methods
__init__(client_id, client_secret, ...[, ...])
Initialize with client_id, client_secret, user_agent, search_queries, mode,
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load reddits.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(client_id: str, client_secret: str, user_agent: str, search_queries: Sequence[str], mode: str, categories: Sequence[str] = ['new'], number_posts: Optional[int] = 10)[source]¶
Initialize with client_id, client_secret, user_agent, search_queries, mode,categories, number_posts.
Example: https://www.reddit.com/r/learnpython/
Parameters
client_id (str) – Reddit client id.
client_secret (str) – Reddit client secret.
user_agent (str) – Reddit user agent.
search_queries (Sequence[str]) – The search queries.
mode (str) – The mode.
categories (Sequence[str]) – The categories. Default: [“new”]
number_posts (Optional[int]) – The number of posts. Default: 10
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load reddits.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using RedditPostsLoader¶
Reddit
langchain_community.document_loaders.notiondb.NotionDBLoader¶
class langchain_community.document_loaders.notiondb.NotionDBLoader(integration_token: str, database_id: str, request_timeout_sec: Optional[int] = 10, *, filter_object: Optional[Dict[str, Any]] = None)[source]¶
Load from Notion DB.
Reads content from pages within a Notion Database.
:param integration_token: Notion integration token.
:type integration_token: str
:param database_id: Notion database id.
:type database_id: str
:param request_timeout_sec: Timeout for Notion requests in seconds.
Defaults to 10.
Parameters
filter_object (Dict[str, Any]) – Filter object used to limit returned
entries based on specified criteria.
E.g.: {
”timestamp”: “last_edited_time”,
“last_edited_time”: {
”on_or_after”: “2024-02-07”
}
} -> will only return entries that were last editedon or after 2024-02-07
Notion docs: https://developers.notion.com/reference/post-database-query-filter
Defaults to None, which will return ALL entries.
integration_token (str) – 
database_id (str) – 
request_timeout_sec (int) – 
Initialize with parameters.
Methods
__init__(integration_token, database_id[, ...])
Initialize with parameters.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load documents from the Notion database.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_page(page_summary)
Read a page.
Load Documents and split into chunks.
load_page(page_summary)
Read a page.
__init__(integration_token: str, database_id: str, request_timeout_sec: Optional[int] = 10, *, filter_object: Optional[Dict[str, Any]] = None) → None[source]¶
Initialize with parameters.
Parameters
integration_token (str) – 
database_id (str) – 
request_timeout_sec (Optional[int]) – 
filter_object (Optional[Dict[str, Any]]) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents from the Notion database.
:returns: List of documents.
:rtype: List[Document]
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
load_page(page_summary: Dict[str, Any]) → Document[source]¶
Read a page.
Parameters
page_summary (Dict[str, Any]) – Page summary from Notion API.
Return type
Document
Examples using NotionDBLoader¶
Notion DB
langchain_community.document_loaders.pdf.BasePDFLoader¶
class langchain_community.document_loaders.pdf.BasePDFLoader(file_path: Union[str, Path], *, headers: Optional[Dict] = None)[source]¶
Base Loader class for PDF files.
If the file is a web path, it will download it to a temporary file, use it, thenclean up the temporary file after completion.
Initialize with a file path.
Parameters
file_path (Union[str, Path]) – Either a local, S3 or web path to a PDF file.
headers (Optional[Dict]) – Headers to use for GET request to download a file from a web path.
Attributes
source
Methods
__init__(file_path, *[, headers])
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path], *, headers: Optional[Dict] = None)[source]¶
Initialize with a file path.
Parameters
file_path (Union[str, Path]) – Either a local, S3 or web path to a PDF file.
headers (Optional[Dict]) – Headers to use for GET request to download a file from a web path.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
langchain_core.document_loaders.blob_loaders.BlobLoader¶
class langchain_core.document_loaders.blob_loaders.BlobLoader[source]¶
Abstract interface for blob loaders implementation.
Implementer should be able to load raw content from a storage system according
to some criteria and return the raw content lazily as a stream of blobs.
Methods
__init__()
yield_blobs()
A lazy loader for raw data represented by LangChain's Blob object.
__init__()¶
abstract yield_blobs() → Iterable[Blob][source]¶
A lazy loader for raw data represented by LangChain’s Blob object.
Returns
A generator over blobs
Return type
Iterable[Blob]
langchain_community.document_loaders.google_speech_to_text.GoogleSpeechToTextLoader¶
class langchain_community.document_loaders.google_speech_to_text.GoogleSpeechToTextLoader(project_id: str, file_path: str, location: str = 'us-central1', recognizer_id: str = '_', config: Optional[RecognitionConfig] = None, config_mask: Optional[FieldMask] = None)[source]¶
[Deprecated] Loader for Google Cloud Speech-to-Text audio transcripts.
It uses the Google Cloud Speech-to-Text API to transcribe audio files
and loads the transcribed text into one or more Documents,
depending on the specified format.
To use, you should have the google-cloud-speech python package installed.
Audio files can be specified via a Google Cloud Storage uri or a local file path.
For a detailed explanation of Google Cloud Speech-to-Text, refer to the product
documentation.
https://cloud.google.com/speech-to-text
Notes
Deprecated since version 0.0.32.
Initializes the GoogleSpeechToTextLoader.
Parameters
project_id (str) – Google Cloud Project ID.
file_path (str) – A Google Cloud Storage URI or a local file path.
location (str) – Speech-to-Text recognizer location.
recognizer_id (str) – Speech-to-Text recognizer id.
config (Optional[RecognitionConfig]) – Recognition options and features.
For more information:
https://cloud.google.com/python/docs/reference/speech/latest/google.cloud.speech_v2.types.RecognitionConfig
config_mask (Optional[FieldMask]) – The list of fields in config that override the values in the
default_recognition_config of the recognizer during this
recognition request.
For more information:
https://cloud.google.com/python/docs/reference/speech/latest/google.cloud.speech_v2.types.RecognizeRequest
Methods
Methods
__init__(project_id, file_path[, location, ...])
Initializes the GoogleSpeechToTextLoader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Transcribes the audio file and loads the transcript into documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(project_id: str, file_path: str, location: str = 'us-central1', recognizer_id: str = '_', config: Optional[RecognitionConfig] = None, config_mask: Optional[FieldMask] = None)[source]¶
Initializes the GoogleSpeechToTextLoader.
Parameters
project_id (str) – Google Cloud Project ID.
file_path (str) – A Google Cloud Storage URI or a local file path.
location (str) – Speech-to-Text recognizer location.
recognizer_id (str) – Speech-to-Text recognizer id.
config (Optional[RecognitionConfig]) – Recognition options and features.
For more information:
https://cloud.google.com/python/docs/reference/speech/latest/google.cloud.speech_v2.types.RecognitionConfig
config_mask (Optional[FieldMask]) – The list of fields in config that override the values in the
default_recognition_config of the recognizer during this
recognition request.
For more information:
https://cloud.google.com/python/docs/reference/speech/latest/google.cloud.speech_v2.types.RecognizeRequest
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Transcribes the audio file and loads the transcript into documents.
It uses the Google Cloud Speech-to-Text API to transcribe the audio file
and blocks until the transcription is finished.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
langchain_community.document_loaders.assemblyai.AssemblyAIAudioTranscriptLoader¶
class langchain_community.document_loaders.assemblyai.AssemblyAIAudioTranscriptLoader(file_path: Union[str, Path], *, transcript_format: TranscriptFormat = TranscriptFormat.TEXT, config: Optional[assemblyai.TranscriptionConfig] = None, api_key: Optional[str] = None)[source]¶
Load AssemblyAI audio transcripts.
It uses the AssemblyAI API to transcribe audio files
and loads the transcribed text into one or more Documents,
depending on the specified format.
To use, you should have the assemblyai python package installed, and the
environment variable ASSEMBLYAI_API_KEY set with your API key.
Alternatively, the API key can also be passed as an argument.
Audio files can be specified via an URL or a local file path.
Initializes the AssemblyAI AudioTranscriptLoader.
Parameters
file_path (Union[str, Path]) – An URL or a local file path.
transcript_format (TranscriptFormat) – Transcript format to use.
See class TranscriptFormat for more info.
config (Optional[assemblyai.TranscriptionConfig]) – Transcription options and features. If None is given,
the Transcriber’s default configuration will be used.
api_key (Optional[str]) – AssemblyAI API key.
Methods
__init__(file_path, *[, transcript_format, ...])
Initializes the AssemblyAI AudioTranscriptLoader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Transcribes the audio file and loads the transcript into documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path], *, transcript_format: TranscriptFormat = TranscriptFormat.TEXT, config: Optional[assemblyai.TranscriptionConfig] = None, api_key: Optional[str] = None)[source]¶
Initializes the AssemblyAI AudioTranscriptLoader.
Parameters
file_path (Union[str, Path]) – An URL or a local file path.
transcript_format (TranscriptFormat) – Transcript format to use.
See class TranscriptFormat for more info.
config (Optional[assemblyai.TranscriptionConfig]) – Transcription options and features. If None is given,
the Transcriber’s default configuration will be used.
api_key (Optional[str]) – AssemblyAI API key.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Transcribes the audio file and loads the transcript into documents.
It uses the AssemblyAI API to transcribe the audio file and blocks until
the transcription is finished.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
langchain_community.document_loaders.word_document.Docx2txtLoader¶
class langchain_community.document_loaders.word_document.Docx2txtLoader(file_path: Union[str, Path])[source]¶
Load DOCX file using docx2txt and chunks at character level.
Defaults to check for local file, but if the file is a web path, it will download it
to a temporary file, and use that, then clean up the temporary file after completion
Initialize with file path.
Methods
__init__(file_path)
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load given path as single page.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (Union[str, Path]) – 
__init__(file_path: Union[str, Path])[source]¶
Initialize with file path.
Parameters
file_path (Union[str, Path]) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load given path as single page.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters