Tony20100/langchain_full huggingface dataset:
bs_kwargs (Optional[dict]) – Any kwargs to pass to the BeautifulSoup object.
get_text_separator (str) – The separator to use when getting the text
from the soup.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load MHTML document into document objects.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using MHTMLLoader¶
mhtml
langchain_community.document_loaders.azure_blob_storage_container.AzureBlobStorageContainerLoader¶
class langchain_community.document_loaders.azure_blob_storage_container.AzureBlobStorageContainerLoader(conn_str: str, container: str, prefix: str = '')[source]¶
Load from Azure Blob Storage container.
Initialize with connection string, container and blob prefix.
Attributes
conn_str
Connection string for Azure Blob Storage.
container
Container name.
prefix
Prefix for blob names.
Methods
__init__(conn_str, container[, prefix])
Initialize with connection string, container and blob prefix.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
conn_str (str) – 
container (str) – 
prefix (str) – 
__init__(conn_str: str, container: str, prefix: str = '')[source]¶
Initialize with connection string, container and blob prefix.
Parameters
conn_str (str) – 
container (str) – 
prefix (str) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using AzureBlobStorageContainerLoader¶
Azure Blob Storage Container
Microsoft
langchain_community.document_loaders.kinetica_loader.KineticaLoader¶
class langchain_community.document_loaders.kinetica_loader.KineticaLoader(query: str, host: str, username: str, password: str, parameters: Optional[Dict[str, Any]] = None, page_content_columns: Optional[List[str]] = None, metadata_columns: Optional[List[str]] = None)[source]¶
Load from Kinetica API.
Each document represents one row of the result. The page_content_columns
are written into the page_content of the document. The metadata_columns
are written into the metadata of the document. By default, all columns
are written into the page_content and none into the metadata.
Initialize Kinetica document loader.
Parameters
query (str) – The query to run in Kinetica.
parameters (Optional[Dict[str, Any]]) – Optional. Parameters to pass to the query.
page_content_columns (Optional[List[str]]) – Optional. Columns written to Document page_content.
metadata_columns (Optional[List[str]]) – Optional. Columns written to Document metadata.
host (str) – 
username (str) – 
password (str) – 
Methods
__init__(query, host, username, password[, ...])
Initialize Kinetica document loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(query: str, host: str, username: str, password: str, parameters: Optional[Dict[str, Any]] = None, page_content_columns: Optional[List[str]] = None, metadata_columns: Optional[List[str]] = None)[source]¶
Initialize Kinetica document loader.
Parameters
query (str) – The query to run in Kinetica.
parameters (Optional[Dict[str, Any]]) – Optional. Parameters to pass to the query.
page_content_columns (Optional[List[str]]) – Optional. Columns written to Document page_content.
metadata_columns (Optional[List[str]]) – Optional. Columns written to Document metadata.
host (str) – 
username (str) – 
password (str) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using KineticaLoader¶
Kinetica
langchain_community.document_loaders.image.UnstructuredImageLoader¶
class langchain_community.document_loaders.image.UnstructuredImageLoader(file_path: Optional[Union[str, List[str], Path, List[Path]]], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load PNG and JPG files using Unstructured.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain_community.document_loaders import UnstructuredImageLoader
loader = UnstructuredImageLoader(“example.png”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-image
Initialize with file path.
Methods
__init__(file_path[, mode])
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (Optional[Union[str, List[str], Path, List[Path]]]) – 
mode (str) – 
unstructured_kwargs (Any) – 
__init__(file_path: Optional[Union[str, List[str], Path, List[Path]]], mode: str = 'single', **unstructured_kwargs: Any)¶
Initialize with file path.
Parameters
Initialize with file path.
Parameters
file_path (Optional[Union[str, List[str], Path, List[Path]]]) – 
mode (str) – 
unstructured_kwargs (Any) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using UnstructuredImageLoader¶
Images
Unstructured
langchain_community.document_loaders.unstructured.get_elements_from_api¶
langchain_community.document_loaders.unstructured.get_elements_from_api(file_path: Optional[Union[str, List[str], Path, List[Path]]] = None, file: Optional[Union[IO, Sequence[IO]]] = None, api_url: str = 'https://api.unstructured.io/general/v0/general', api_key: str = '', **unstructured_kwargs: Any) → List[source]¶
Retrieve a list of elements from the Unstructured API.
Parameters
file_path (Optional[Union[str, List[str], Path, List[Path]]]) – 
file (Optional[Union[IO, Sequence[IO]]]) – 
api_url (str) – 
api_key (str) – 
unstructured_kwargs (Any) – 
Return type
List
langchain_community.document_loaders.parsers.language.java.JavaSegmenter¶
class langchain_community.document_loaders.parsers.language.java.JavaSegmenter(code: str)[source]¶
Code segmenter for Java.
Methods
__init__(code)
extract_functions_classes()
get_chunk_query()
get_language()
get_parser()
is_valid()
make_line_comment(text)
simplify_code()
Parameters
code (str) – 
__init__(code: str)¶
Parameters
code (str) – 
extract_functions_classes() → List[str]¶
Return type
List[str]
get_chunk_query() → str[source]¶
Return type
str
get_language() → Language[source]¶
Return type
Language
get_parser() → Parser¶
Return type
Parser
is_valid() → bool¶
Return type
bool
make_line_comment(text: str) → str[source]¶
Parameters
text (str) – 
Return type
str
simplify_code() → str¶
Return type
str
langchain_community.document_loaders.parsers.language.kotlin.KotlinSegmenter¶
class langchain_community.document_loaders.parsers.language.kotlin.KotlinSegmenter(code: str)[source]¶
Code segmenter for Kotlin.
Methods
__init__(code)
extract_functions_classes()
get_chunk_query()
get_language()
get_parser()
is_valid()
make_line_comment(text)
simplify_code()
Parameters
code (str) – 
__init__(code: str)¶
Parameters
code (str) – 
extract_functions_classes() → List[str]¶
Return type
List[str]
get_chunk_query() → str[source]¶
Return type
str
get_language() → Language[source]¶
Return type
Language
get_parser() → Parser¶
Return type
Parser
is_valid() → bool¶
Return type
bool
make_line_comment(text: str) → str[source]¶
Parameters
text (str) – 
Return type
str
simplify_code() → str¶
Return type
str
langchain_community.document_loaders.helpers.FileEncoding¶
class langchain_community.document_loaders.helpers.FileEncoding(encoding: Optional[str], confidence: float, language: Optional[str])[source]¶
File encoding as the NamedTuple.
Create new instance of FileEncoding(encoding, confidence, language)
Attributes
confidence
The confidence of the encoding.
encoding
The encoding of the file.
language
The language of the file.
Methods
__init__()
count(value, /)
Return number of occurrences of value.
index(value[, start, stop])
Return first index of value.
Parameters
encoding (Optional[str]) – 
confidence (float) – 
language (Optional[str]) – 
__init__()¶
count(value, /)¶
Return number of occurrences of value.
index(value, start=0, stop=9223372036854775807, /)¶
Return first index of value.
Raises ValueError if the value is not present.
langchain_community.document_loaders.csv_loader.CSVLoader¶
class langchain_community.document_loaders.csv_loader.CSVLoader(file_path: Union[str, Path], source_column: Optional[str] = None, metadata_columns: Sequence[str] = (), csv_args: Optional[Dict] = None, encoding: Optional[str] = None, autodetect_encoding: bool = False)[source]¶
Load a CSV file into a list of Documents.
Each document represents one row of the CSV file. Every row is converted into a
key/value pair and outputted to a new line in the document’s page_content.
The source for each document loaded from csv is set to the value of the
file_path argument for all documents by default.
You can override this by setting the source_column argument to the
name of a column in the CSV file.
The source of each document will then be set to the value of the column
with the name specified in source_column.
Output Example:column1: value1
column2: value2
column3: value3
Parameters
file_path (Union[str, Path]) – The path to the CSV file.
source_column (Optional[str]) – The name of the column in the CSV file to use as the source.
Optional. Defaults to None.
metadata_columns (Sequence[str]) – A sequence of column names to use as metadata. Optional.
csv_args (Optional[Dict]) – A dictionary of arguments to pass to the csv.DictReader.
Optional. Defaults to None.
encoding (Optional[str]) – The encoding of the CSV file. Optional. Defaults to None.
autodetect_encoding (bool) – Whether to try to autodetect the file encoding.
Methods
__init__(file_path[, source_column, ...])
param file_path
The path to the CSV file.
alazy_load()
A lazy loader for Documents.
The path to the CSV file.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path], source_column: Optional[str] = None, metadata_columns: Sequence[str] = (), csv_args: Optional[Dict] = None, encoding: Optional[str] = None, autodetect_encoding: bool = False)[source]¶
Parameters
file_path (Union[str, Path]) – The path to the CSV file.
source_column (Optional[str]) – The name of the column in the CSV file to use as the source.
Optional. Defaults to None.
metadata_columns (Sequence[str]) – A sequence of column names to use as metadata. Optional.
csv_args (Optional[Dict]) – A dictionary of arguments to pass to the csv.DictReader.
Optional. Defaults to None.
encoding (Optional[str]) – The encoding of the CSV file. Optional. Defaults to None.
autodetect_encoding (bool) – Whether to try to autodetect the file encoding.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using CSVLoader¶
Aerospike
CSV
ChatGPT plugin
Conceptual guide
How to load CSVs
Pebblo Safe DocumentLoader
langchain_community.document_loaders.gitbook.GitbookLoader¶
class langchain_community.document_loaders.gitbook.GitbookLoader(web_page: str, load_all_paths: bool = False, base_url: Optional[str] = None, content_selector: str = 'main', continue_on_failure: bool = False)[source]¶
Load GitBook data.
load from either a single page, or
load all (relative) paths in the navbar.
Initialize with web page and whether to load all paths.
Parameters
web_page (str) – The web page to load or the starting point from where
relative paths are discovered.
load_all_paths (bool) – If set to True, all relative paths in the navbar
are loaded instead of only web_page.
base_url (Optional[str]) – If load_all_paths is True, the relative paths are
appended to this base url. Defaults to web_page.
content_selector (str) – The CSS selector for the content to load.
Defaults to “main”.
continue_on_failure (bool) – whether to continue loading the sitemap if an error
occurs loading a url, emitting a warning instead of raising an
exception. Setting this to True makes the loader more robust, but also
may result in missing data. Default: False
Attributes
web_path
Methods
__init__(web_page[, load_all_paths, ...])
Initialize with web page and whether to load all paths.
alazy_load()
A lazy loader for Documents.
aload()
Load text from the urls in web_path async into Documents.
fetch_all(urls)
Fetch all urls concurrently with rate limiting.
lazy_load()
Fetch text from one single GitBook page.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
scrape([parser])
Load Documents and split into chunks.
scrape([parser])
Scrape data from webpage and return it in BeautifulSoup format.
scrape_all(urls[, parser])
Fetch all urls, then return soups for all results.
__init__(web_page: str, load_all_paths: bool = False, base_url: Optional[str] = None, content_selector: str = 'main', continue_on_failure: bool = False)[source]¶
Initialize with web page and whether to load all paths.
Parameters
web_page (str) – The web page to load or the starting point from where
relative paths are discovered.
load_all_paths (bool) – If set to True, all relative paths in the navbar
are loaded instead of only web_page.
base_url (Optional[str]) – If load_all_paths is True, the relative paths are
appended to this base url. Defaults to web_page.
content_selector (str) – The CSS selector for the content to load.
Defaults to “main”.
continue_on_failure (bool) – whether to continue loading the sitemap if an error
occurs loading a url, emitting a warning instead of raising an
exception. Setting this to True makes the loader more robust, but also
may result in missing data. Default: False
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
aload() → List[Document]¶
Load text from the urls in web_path async into Documents.
Return type
List[Document]
async fetch_all(urls: List[str]) → Any¶
Fetch all urls concurrently with rate limiting.
Parameters
urls (List[str]) – 
Return type
Any
lazy_load() → Iterator[Document][source]¶
Fetch text from one single GitBook page.
Return type
Iterator[Document]
Fetch text from one single GitBook page.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
scrape(parser: Optional[str] = None) → Any¶
Scrape data from webpage and return it in BeautifulSoup format.
Parameters
parser (Optional[str]) – 
Return type
Any
scrape_all(urls: List[str], parser: Optional[str] = None) → List[Any]¶
Fetch all urls, then return soups for all results.
Parameters
urls (List[str]) – 
parser (Optional[str]) – 
Return type
List[Any]
Examples using GitbookLoader¶
GitBook
langchain_community.document_loaders.parsers.language.rust.RustSegmenter¶
class langchain_community.document_loaders.parsers.language.rust.RustSegmenter(code: str)[source]¶
Code segmenter for Rust.
Methods
__init__(code)
extract_functions_classes()
get_chunk_query()
get_language()
get_parser()
is_valid()
make_line_comment(text)
simplify_code()
Parameters
code (str) – 
__init__(code: str)¶
Parameters
code (str) – 
extract_functions_classes() → List[str]¶
Return type
List[str]
get_chunk_query() → str[source]¶
Return type
str
get_language() → Language[source]¶
Return type
Language
get_parser() → Parser¶
Return type
Parser
is_valid() → bool¶
Return type
bool
make_line_comment(text: str) → str[source]¶
Parameters
text (str) – 
Return type
str
simplify_code() → str¶
Return type
str
langchain_community.document_loaders.confluence.ConfluenceLoader¶
class langchain_community.document_loaders.confluence.ConfluenceLoader(url: str, api_key: Optional[str] = None, username: Optional[str] = None, session: Optional[Session] = None, oauth2: Optional[dict] = None, token: Optional[str] = None, cloud: Optional[bool] = True, number_of_retries: Optional[int] = 3, min_retry_seconds: Optional[int] = 2, max_retry_seconds: Optional[int] = 10, confluence_kwargs: Optional[dict] = None, *, space_key: Optional[str] = None, page_ids: Optional[List[str]] = None, label: Optional[str] = None, cql: Optional[str] = None, include_restricted_content: bool = False, include_archived_content: bool = False, include_attachments: bool = False, include_comments: bool = False, content_format: ContentFormat = ContentFormat.STORAGE, limit: Optional[int] = 50, max_pages: Optional[int] = 1000, ocr_languages: Optional[str] = None, keep_markdown_format: bool = False, keep_newlines: bool = False)[source]¶
Load Confluence pages.
Port of https://llamahub.ai/l/confluence
This currently supports username/api_key, Oauth2 login or personal access token
authentication.
Specify a list page_ids and/or space_key to load in the corresponding pages into
Document objects, if both are specified the union of both sets will be returned.
You can also specify a boolean include_attachments to include attachments, this
is set to False by default, if set to True all attachments will be downloaded and
ConfluenceReader will extract the text from the attachments and add it to the
Document object. Currently supported attachment types are: PDF, PNG, JPEG/JPG,
Document object. Currently supported attachment types are: PDF, PNG, JPEG/JPG,
SVG, Word and Excel.
Confluence API supports difference format of page content. The storage format is the
raw XML representation for storage. The view format is the HTML representation for
viewing with macros are rendered as though it is viewed by users. You can pass
a enum content_format argument to specify the content format, this is
set to ContentFormat.STORAGE by default, the supported values are:
ContentFormat.EDITOR, ContentFormat.EXPORT_VIEW,
ContentFormat.ANONYMOUS_EXPORT_VIEW, ContentFormat.STORAGE,
and ContentFormat.VIEW.
Hint: space_key and page_id can both be found in the URL of a page in Confluence
- https://yoursite.atlassian.com/wiki/spaces/<space_key>/pages/<page_id>
Example
from langchain_community.document_loaders import ConfluenceLoader
loader = ConfluenceLoader(
    url="https://yoursite.atlassian.com/wiki",
    username="me",
    api_key="12345",
    space_key="SPACE",
    limit=50,
)
documents = loader.load()
# Server on perm
loader = ConfluenceLoader(
    url="https://confluence.yoursite.com/",
    username="me",
    api_key="your_password",
    cloud=False,
    space_key="SPACE",
    limit=50,
)
documents = loader.load()
Parameters
url (str) – _description_
api_key (str, optional) – _description_, defaults to None
username (str, optional) – _description_, defaults to None
oauth2 (dict, optional) – _description_, defaults to {}
token (str, optional) – _description_, defaults to None
cloud (bool, optional) – _description_, defaults to True