Tony20100/langchain_full huggingface dataset:
cloud (bool, optional) – _description_, defaults to True
number_of_retries (Optional[int], optional) – How many times to retry, defaults to 3
min_retry_seconds (Optional[int], optional) – defaults to 2
max_retry_seconds (Optional[int], optional) – defaults to 10
confluence_kwargs (dict, optional) – additional kwargs to initialize confluence with
space_key (Optional[str], optional) – Space key retrieved from a confluence URL, defaults to None
page_ids (Optional[List[str]], optional) – List of specific page IDs to load, defaults to None
label (Optional[str], optional) – Get all pages with this label, defaults to None
cql (Optional[str], optional) – CQL Expression, defaults to None
include_restricted_content (bool, optional) – defaults to False
include_archived_content (bool, optional) – Whether to include archived content,
defaults to False
include_attachments (bool, optional) – defaults to False
include_comments (bool, optional) – defaults to False
content_format (ContentFormat) – Specify content format, defaults to
ContentFormat.STORAGE, the supported values are:
ContentFormat.EDITOR, ContentFormat.EXPORT_VIEW,
ContentFormat.ANONYMOUS_EXPORT_VIEW,
ContentFormat.STORAGE, and ContentFormat.VIEW.
limit (int, optional) – Maximum number of pages to retrieve per request, defaults to 50
max_pages (int, optional) – Maximum number of pages to retrieve in total, defaults 1000
ocr_languages (str, optional) – The languages to use for the Tesseract agent. To use a
language, you’ll first need to install the appropriate
Tesseract language pack.
keep_markdown_format (bool) – Whether to keep the markdown format, defaults to
False
keep_markdown_format (bool) – Whether to keep the markdown format, defaults to
False
keep_newlines (bool) – Whether to keep the newlines format, defaults to
False
session (Optional[Session]) – 
Raises
ValueError – Errors while validating input
ImportError – Required dependencies not installed.
Methods
__init__(url[, api_key, username, session, ...])
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
is_public_page(page)
Check if a page is publicly accessible.
lazy_load()
A lazy loader for Documents.
load(**kwargs)
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
paginate_request(retrieval_method, **kwargs)
Paginate the various methods to retrieve groups of pages.
process_attachment(page_id[, ocr_languages])
process_doc(link)
process_image(link[, ocr_languages])
process_page(page, include_attachments, ...)
process_pages(pages, ...[, ocr_languages, ...])
Process a list of pages into a list of documents.
process_pdf(link[, ocr_languages])
process_svg(link[, ocr_languages])
process_xls(link)
validate_init_args([url, api_key, username, ...])
Validates proper combinations of init arguments
Validates proper combinations of init arguments
__init__(url: str, api_key: Optional[str] = None, username: Optional[str] = None, session: Optional[Session] = None, oauth2: Optional[dict] = None, token: Optional[str] = None, cloud: Optional[bool] = True, number_of_retries: Optional[int] = 3, min_retry_seconds: Optional[int] = 2, max_retry_seconds: Optional[int] = 10, confluence_kwargs: Optional[dict] = None, *, space_key: Optional[str] = None, page_ids: Optional[List[str]] = None, label: Optional[str] = None, cql: Optional[str] = None, include_restricted_content: bool = False, include_archived_content: bool = False, include_attachments: bool = False, include_comments: bool = False, content_format: ContentFormat = ContentFormat.STORAGE, limit: Optional[int] = 50, max_pages: Optional[int] = 1000, ocr_languages: Optional[str] = None, keep_markdown_format: bool = False, keep_newlines: bool = False)[source]¶
Parameters
url (str) – 
api_key (Optional[str]) – 
username (Optional[str]) – 
session (Optional[Session]) – 
oauth2 (Optional[dict]) – 
token (Optional[str]) – 
cloud (Optional[bool]) – 
number_of_retries (Optional[int]) – 
min_retry_seconds (Optional[int]) – 
max_retry_seconds (Optional[int]) – 
confluence_kwargs (Optional[dict]) – 
space_key (Optional[str]) – 
page_ids (Optional[List[str]]) – 
label (Optional[str]) – 
cql (Optional[str]) – 
include_restricted_content (bool) –
cql (Optional[str]) – 
include_restricted_content (bool) – 
include_archived_content (bool) – 
include_attachments (bool) – 
include_comments (bool) – 
content_format (ContentFormat) – 
limit (Optional[int]) – 
max_pages (Optional[int]) – 
ocr_languages (Optional[str]) – 
keep_markdown_format (bool) – 
keep_newlines (bool) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
is_public_page(page: dict) → bool[source]¶
Check if a page is publicly accessible.
Parameters
page (dict) – 
Return type
bool
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load(**kwargs: Any) → List[Document][source]¶
Load data into Document objects.
Parameters
kwargs (Any) – 
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
paginate_request(retrieval_method: Callable, **kwargs: Any) → List[source]¶
Paginate the various methods to retrieve groups of pages.
Paginate the various methods to retrieve groups of pages.
Unfortunately, due to page size, sometimes the Confluence API
doesn’t match the limit value. If limit is >100 confluence
seems to cap the response to 100. Also, due to the Atlassian Python
package, we don’t get the “next” values from the “_links” key because
they only return the value from the result key. So here, the pagination
starts from 0 and goes until the max_pages, getting the limit number
of pages with each request. We have to manually check if there
are more docs based on the length of the returned list of pages, rather than
just checking for the presence of a next key in the response like this page
would have you do:
https://developer.atlassian.com/server/confluence/pagination-in-the-rest-api/
Parameters
retrieval_method (callable) – Function used to retrieve docs
kwargs (Any) – 
Returns
List of documents
Return type
List
process_attachment(page_id: str, ocr_languages: Optional[str] = None) → List[str][source]¶
Parameters
page_id (str) – 
ocr_languages (Optional[str]) – 
Return type
List[str]
process_doc(link: str) → str[source]¶
Parameters
link (str) – 
Return type
str
process_image(link: str, ocr_languages: Optional[str] = None) → str[source]¶
Parameters
link (str) – 
ocr_languages (Optional[str]) – 
Return type
str
process_page(page: dict, include_attachments: bool, include_comments: bool, content_format: ContentFormat, ocr_languages: Optional[str] = None, keep_markdown_format: Optional[bool] = False, keep_newlines: bool = False) → Document[source]¶
Parameters
Parameters
page (dict) – 
include_attachments (bool) – 
include_comments (bool) – 
content_format (ContentFormat) – 
ocr_languages (Optional[str]) – 
keep_markdown_format (Optional[bool]) – 
keep_newlines (bool) – 
Return type
Document
process_pages(pages: List[dict], include_restricted_content: bool, include_attachments: bool, include_comments: bool, content_format: ContentFormat, ocr_languages: Optional[str] = None, keep_markdown_format: Optional[bool] = False, keep_newlines: bool = False) → Iterator[Document][source]¶
Process a list of pages into a list of documents.
Parameters
pages (List[dict]) – 
include_restricted_content (bool) – 
include_attachments (bool) – 
include_comments (bool) – 
content_format (ContentFormat) – 
ocr_languages (Optional[str]) – 
keep_markdown_format (Optional[bool]) – 
keep_newlines (bool) – 
Return type
Iterator[Document]
process_pdf(link: str, ocr_languages: Optional[str] = None) → str[source]¶
Parameters
link (str) – 
ocr_languages (Optional[str]) – 
Return type
str
process_svg(link: str, ocr_languages: Optional[str] = None) → str[source]¶
Parameters
link (str) – 
ocr_languages (Optional[str]) – 
Return type
str
process_xls(link: str) → str[source]¶
Parameters
link (str) – 
Return type
str
Parameters
link (str) – 
Return type
str
static validate_init_args(url: Optional[str] = None, api_key: Optional[str] = None, username: Optional[str] = None, session: Optional[Session] = None, oauth2: Optional[dict] = None, token: Optional[str] = None) → Optional[List][source]¶
Validates proper combinations of init arguments
Parameters
url (Optional[str]) – 
api_key (Optional[str]) – 
username (Optional[str]) – 
session (Optional[Session]) – 
oauth2 (Optional[dict]) – 
token (Optional[str]) – 
Return type
Optional[List]
Examples using ConfluenceLoader¶
Confluence
langchain_community.document_loaders.airbyte.AirbyteStripeLoader¶
class langchain_community.document_loaders.airbyte.AirbyteStripeLoader(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None)[source]¶
Load from Stripe using an Airbyte source connector.
Initializes the loader.
Parameters
config (Mapping[str, Any]) – The config to pass to the source connector.
stream_name (str) – The name of the stream to load.
record_handler (Optional[Callable[[Any, Optional[str]], Document]]) – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state (Optional[Any]) – The state to pass to the source connector. Defaults to None.
Attributes
last_state
Methods
__init__(config, stream_name[, ...])
Initializes the loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None) → None[source]¶
Initializes the loader.
Parameters
config (Mapping[str, Any]) – The config to pass to the source connector.
stream_name (str) – The name of the stream to load.
record_handler (Optional[Callable[[Any, Optional[str]], Document]]) – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state (Optional[Any]) – The state to pass to the source connector. Defaults to None.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using AirbyteStripeLoader¶
Airbyte Question Answering
Airbyte Stripe (Deprecated)
langchain_community.document_loaders.parsers.language.php.PHPSegmenter¶
class langchain_community.document_loaders.parsers.language.php.PHPSegmenter(code: str)[source]¶
Code segmenter for PHP.
Methods
__init__(code)
extract_functions_classes()
get_chunk_query()
get_language()
get_parser()
is_valid()
make_line_comment(text)
simplify_code()
Parameters
code (str) – 
__init__(code: str)¶
Parameters
code (str) – 
extract_functions_classes() → List[str]¶
Return type
List[str]
get_chunk_query() → str[source]¶
Return type
str
get_language() → Language[source]¶
Return type
Language
get_parser() → Parser¶
Return type
Parser
is_valid() → bool¶
Return type
bool
make_line_comment(text: str) → str[source]¶
Parameters
text (str) – 
Return type
str
simplify_code() → str¶
Return type
str
langchain_community.document_loaders.gutenberg.GutenbergLoader¶
class langchain_community.document_loaders.gutenberg.GutenbergLoader(file_path: str)[source]¶
Load from Gutenberg.org.
Initialize with a file path.
Methods
__init__(file_path)
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load file.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (str) – 
__init__(file_path: str)[source]¶
Initialize with a file path.
Parameters
file_path (str) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load file.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using GutenbergLoader¶
Gutenberg
langchain_community.document_loaders.parsers.audio.FasterWhisperParser¶
class langchain_community.document_loaders.parsers.audio.FasterWhisperParser(*, device: Optional[str] = 'cuda', model_size: Optional[str] = None)[source]¶
Transcribe and parse audio files with faster-whisper.
faster-whisper is a reimplementation of OpenAI’s Whisper model using CTranslate2,
which is up to 4 times faster than openai/whisper for the same accuracy while using
less memory. The efficiency can be further improved with 8-bit quantization on both
CPU and GPU.
It can automatically detect the following 14 languages and transcribe the text
into their respective languages: en, zh, fr, de, ja, ko, ru, es, th, it, pt, vi,
ar, tr.
The gitbub repository for faster-whisper is :
https://github.com/SYSTRAN/faster-whisper
Example: Load a YouTube video and transcribe the video speech into a document.from langchain.document_loaders.generic import GenericLoader
from langchain_community.document_loaders.parsers.audio
    import FasterWhisperParser
from langchain.document_loaders.blob_loaders.youtube_audio
    import YoutubeAudioLoader
url="https://www.youtube.com/watch?v=your_video"
save_dir="your_dir/"
loader = GenericLoader(
    YoutubeAudioLoader([url],save_dir),
    FasterWhisperParser()
)
docs = loader.load()
Initialize the parser.
Parameters
device (Optional[str]) – It can be “cuda” or “cpu” based on the available device.
model_size (Optional[str]) – There are four model sizes to choose from: “base”, “small”,
“medium”, and “large-v3”, based on the available GPU memory.
Methods
“medium”, and “large-v3”, based on the available GPU memory.
Methods
__init__(*[, device, model_size])
Initialize the parser.
lazy_parse(blob)
Lazily parse the blob.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__(*, device: Optional[str] = 'cuda', model_size: Optional[str] = None)[source]¶
Initialize the parser.
Parameters
device (Optional[str]) – It can be “cuda” or “cpu” based on the available device.
model_size (Optional[str]) – There are four model sizes to choose from: “base”, “small”,
“medium”, and “large-v3”, based on the available GPU memory.
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazily parse the blob.
Parameters
blob (Blob) – 
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
langchain_community.document_loaders.parsers.language.go.GoSegmenter¶
class langchain_community.document_loaders.parsers.language.go.GoSegmenter(code: str)[source]¶
Code segmenter for Go.
Methods
__init__(code)
extract_functions_classes()
get_chunk_query()
get_language()
get_parser()
is_valid()
make_line_comment(text)
simplify_code()
Parameters
code (str) – 
__init__(code: str)¶
Parameters
code (str) – 
extract_functions_classes() → List[str]¶
Return type
List[str]
get_chunk_query() → str[source]¶
Return type
str
get_language() → Language[source]¶
Return type
Language
get_parser() → Parser¶
Return type
Parser
is_valid() → bool¶
Return type
bool
make_line_comment(text: str) → str[source]¶
Parameters
text (str) – 
Return type
str
simplify_code() → str¶
Return type
str
langchain_community.document_loaders.datadog_logs.DatadogLogsLoader¶
class langchain_community.document_loaders.datadog_logs.DatadogLogsLoader(query: str, api_key: str, app_key: str, from_time: Optional[int] = None, to_time: Optional[int] = None, limit: int = 100)[source]¶
Load Datadog logs.
Logs are written into the page_content and into the metadata.
Initialize Datadog document loader.
Requirements:
Must have datadog_api_client installed. Install with pip install datadog_api_client.
Parameters
query (str) – The query to run in Datadog.
api_key (str) – The Datadog API key.
app_key (str) – The Datadog APP key.
from_time (Optional[int]) – Optional. The start of the time range to query.
Supports date math and regular timestamps (milliseconds) like ‘1688732708951’
Defaults to 20 minutes ago.
to_time (Optional[int]) – Optional. The end of the time range to query.
Supports date math and regular timestamps (milliseconds) like ‘1688732708951’
Defaults to now.
limit (int) – The maximum number of logs to return.
Defaults to 100.
Methods
__init__(query, api_key, app_key[, ...])
Initialize Datadog document loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Get logs from Datadog.
load_and_split([text_splitter])
Load Documents and split into chunks.
parse_log(log)
Create Document objects from Datadog log items.
parse_log(log)
Create Document objects from Datadog log items.
__init__(query: str, api_key: str, app_key: str, from_time: Optional[int] = None, to_time: Optional[int] = None, limit: int = 100) → None[source]¶
Initialize Datadog document loader.
Requirements:
Must have datadog_api_client installed. Install with pip install datadog_api_client.
Parameters
query (str) – The query to run in Datadog.
api_key (str) – The Datadog API key.
app_key (str) – The Datadog APP key.
from_time (Optional[int]) – Optional. The start of the time range to query.
Supports date math and regular timestamps (milliseconds) like ‘1688732708951’
Defaults to 20 minutes ago.
to_time (Optional[int]) – Optional. The end of the time range to query.
Supports date math and regular timestamps (milliseconds) like ‘1688732708951’
Defaults to now.
limit (int) – The maximum number of logs to return.
Defaults to 100.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Get logs from Datadog.
Returns
A list of Document objects.
page_content
metadata
id
service
status
tags
timestamp
Return type
List[Document]
metadata
id
service
status
tags
timestamp
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
parse_log(log: dict) → Document[source]¶
Create Document objects from Datadog log items.
Parameters
log (dict) – 
Return type
Document
Examples using DatadogLogsLoader¶
Datadog Logs
langchain_community.document_loaders.text.TextLoader¶
class langchain_community.document_loaders.text.TextLoader(file_path: Union[str, Path], encoding: Optional[str] = None, autodetect_encoding: bool = False)[source]¶
Load text file.
Parameters
file_path (Union[str, Path]) – Path to the file to load.
encoding (Optional[str]) – File encoding to use. If None, the file will be loaded
encoding. (with the default system) – 
autodetect_encoding (bool) – Whether to try to autodetect the file encoding
if the specified encoding fails.
Initialize with file path.
Methods
__init__(file_path[, encoding, ...])
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load from file path.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path], encoding: Optional[str] = None, autodetect_encoding: bool = False)[source]¶
Initialize with file path.
Parameters
file_path (Union[str, Path]) – 
encoding (Optional[str]) – 
autodetect_encoding (bool) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load from file path.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using TextLoader¶
Activeloop Deep Lake
Alibaba Cloud OpenSearch
Amazon Document DB
AnalyticDB
Annoy
Atlas
AwaDB
Azure AI Search
Azure Cosmos DB
Bagel
BagelDB
Baidu Cloud ElasticSearch VectorSearch
Baidu VectorDB
Caching
Chat Over Documents with Vectara
China Mobile ECloud ElasticSearch VectorSearch
Chroma
Clarifai
ClickHouse
Cohere reranker
Confident
Couchbase 
Cross Encoder Reranker
DashVector
Databricks Vector Search
DingoDB
DocArray HnswSearch
DocArray InMemorySearch
DuckDB
Elasticsearch
Epsilla
Faiss
Faiss (Async)
FlashRank reranker
Google Memorystore for Redis
Hippo
Hologres
How to create and query vector stores
How to do retrieval with contextual compression
How to load documents from a directory
How to retrieve using multiple vectors per document
How to use a vectorstore as a retriever
How to use the Parent Document Retriever
Intel’s Visual Data Management System (VDMS)
Jaguar Vector Database
JaguarDB Vector Database
Jina Reranker
Kinetica Vectorstore API
Kinetica Vectorstore based Retriever