Tony20100/langchain_full huggingface dataset:
Initialize with file path.
Parameters
file_path (Optional[Union[str, List[str], Path, List[Path]]]) – 
mode (str) – 
unstructured_kwargs (Any) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using UnstructuredPowerPointLoader¶
Microsoft
Microsoft PowerPoint
Unstructured
langchain_community.document_loaders.duckdb_loader.DuckDBLoader¶
class langchain_community.document_loaders.duckdb_loader.DuckDBLoader(query: str, database: str = ':memory:', read_only: bool = False, config: Optional[Dict[str, str]] = None, page_content_columns: Optional[List[str]] = None, metadata_columns: Optional[List[str]] = None)[source]¶
Load from DuckDB.
Each document represents one row of the result. The page_content_columns
are written into the page_content of the document. The metadata_columns
are written into the metadata of the document. By default, all columns
are written into the page_content and none into the metadata.
Parameters
query (str) – The query to execute.
database (str) – The database to connect to. Defaults to “:memory:”.
read_only (bool) – Whether to open the database in read-only mode.
Defaults to False.
config (Optional[Dict[str, str]]) – A dictionary of configuration options to pass to the database.
Optional.
page_content_columns (Optional[List[str]]) – The columns to write into the page_content
of the document. Optional.
metadata_columns (Optional[List[str]]) – The columns to write into the metadata of the document.
Optional.
Methods
__init__(query[, database, read_only, ...])
param query
The query to execute.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(query: str, database: str = ':memory:', read_only: bool = False, config: Optional[Dict[str, str]] = None, page_content_columns: Optional[List[str]] = None, metadata_columns: Optional[List[str]] = None)[source]¶
Parameters
query (str) – The query to execute.
database (str) – The database to connect to. Defaults to “:memory:”.
read_only (bool) – Whether to open the database in read-only mode.
Defaults to False.
config (Optional[Dict[str, str]]) – A dictionary of configuration options to pass to the database.
Optional.
page_content_columns (Optional[List[str]]) – The columns to write into the page_content
of the document. Optional.
metadata_columns (Optional[List[str]]) – The columns to write into the metadata of the document.
Optional.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
langchain_community.document_loaders.scrapfly.ScrapflyLoader¶
class langchain_community.document_loaders.scrapfly.ScrapflyLoader(urls: List[str], *, api_key: Optional[str] = None, scrape_format: Literal['markdown', 'text'] = 'markdown', scrape_config: Optional[dict] = None, continue_on_failure: bool = True)[source]¶
Turn a url to llm accessible markdown with Scrapfly.io.
For further details, visit: https://scrapfly.io/docs/sdk/python
Initialize client.
Parameters
urls (List[str]) – List of urls to scrape.
api_key (Optional[str]) – The Scrapfly API key. If not specified must have env var
SCRAPFLY_API_KEY set.
scrape_format (Literal['markdown', 'text']) – Scrape result format, one or “markdown” or “text”.
scrape_config (Optional[dict]) – Dictionary of ScrapFly scrape config object.
continue_on_failure (bool) – Whether to continue if scraping a url fails.
Methods
__init__(urls, *[, api_key, scrape_format, ...])
Initialize client.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(urls: List[str], *, api_key: Optional[str] = None, scrape_format: Literal['markdown', 'text'] = 'markdown', scrape_config: Optional[dict] = None, continue_on_failure: bool = True) → None[source]¶
Initialize client.
Parameters
urls (List[str]) – List of urls to scrape.
Initialize client.
Parameters
urls (List[str]) – List of urls to scrape.
api_key (Optional[str]) – The Scrapfly API key. If not specified must have env var
SCRAPFLY_API_KEY set.
scrape_format (Literal['markdown', 'text']) – Scrape result format, one or “markdown” or “text”.
scrape_config (Optional[dict]) – Dictionary of ScrapFly scrape config object.
continue_on_failure (bool) – Whether to continue if scraping a url fails.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using ScrapflyLoader¶
# ScrapFly
langchain_community.document_loaders.airbyte.AirbyteGongLoader¶
class langchain_community.document_loaders.airbyte.AirbyteGongLoader(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None)[source]¶
Load from Gong using an Airbyte source connector.
Initializes the loader.
Parameters
config (Mapping[str, Any]) – The config to pass to the source connector.
stream_name (str) – The name of the stream to load.
record_handler (Optional[Callable[[Any, Optional[str]], Document]]) – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state (Optional[Any]) – The state to pass to the source connector. Defaults to None.
Attributes
last_state
Methods
__init__(config, stream_name[, ...])
Initializes the loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None) → None[source]¶
Initializes the loader.
Parameters
config (Mapping[str, Any]) – The config to pass to the source connector.
stream_name (str) – The name of the stream to load.
record_handler (Optional[Callable[[Any, Optional[str]], Document]]) – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state (Optional[Any]) – The state to pass to the source connector. Defaults to None.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using AirbyteGongLoader¶
Airbyte Gong (Deprecated)
langchain_community.document_loaders.azure_blob_storage_file.AzureBlobStorageFileLoader¶
class langchain_community.document_loaders.azure_blob_storage_file.AzureBlobStorageFileLoader(conn_str: str, container: str, blob_name: str)[source]¶
Load from Azure Blob Storage files.
Initialize with connection string, container and blob name.
Attributes
conn_str
Connection string for Azure Blob Storage.
container
Container name.
blob
Blob name.
Methods
__init__(conn_str, container, blob_name)
Initialize with connection string, container and blob name.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
conn_str (str) – 
container (str) – 
blob_name (str) – 
__init__(conn_str: str, container: str, blob_name: str)[source]¶
Initialize with connection string, container and blob name.
Parameters
conn_str (str) – 
container (str) – 
blob_name (str) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using AzureBlobStorageFileLoader¶
Azure Blob Storage File
Microsoft
langchain_community.document_loaders.discord.DiscordChatLoader¶
class langchain_community.document_loaders.discord.DiscordChatLoader(chat_log: pd.DataFrame, user_id_col: str = 'ID')[source]¶
Load Discord chat logs.
Initialize with a Pandas DataFrame containing chat logs.
Parameters
chat_log (pd.DataFrame) – Pandas DataFrame containing chat logs.
user_id_col (str) – Name of the column containing the user ID. Defaults to “ID”.
Methods
__init__(chat_log[, user_id_col])
Initialize with a Pandas DataFrame containing chat logs.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load all chat messages.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(chat_log: pd.DataFrame, user_id_col: str = 'ID')[source]¶
Initialize with a Pandas DataFrame containing chat logs.
Parameters
chat_log (pd.DataFrame) – Pandas DataFrame containing chat logs.
user_id_col (str) – Name of the column containing the user ID. Defaults to “ID”.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load all chat messages.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
langchain_community.document_loaders.llmsherpa.LLMSherpaFileLoader¶
class langchain_community.document_loaders.llmsherpa.LLMSherpaFileLoader(file_path: Union[str, Path], new_indent_parser: bool = True, apply_ocr: bool = True, strategy: str = 'chunks', llmsherpa_api_url: str = 'https://readers.llmsherpa.com/api/document/developer/parseDocument?renderFormat=all')[source]¶
Load Documents using LLMSherpa.
LLMSherpaFileLoader use LayoutPDFReader, which is part of the LLMSherpa library.
This tool is designed to parse PDFs while preserving their layout information,
which is often lost when using most PDF to text parsers.
Examples
from langchain_community.document_loaders.llmsherpa import LLMSherpaFileLoader
loader = LLMSherpaFileLoader(“example.pdf”,
strategy=”chunks”,
llmsherpa_api_url=”http://localhost:5010/api/parseDocument?renderFormat=all”,
)
docs = loader.load()
Initialize with a file path.
Methods
__init__(file_path[, new_indent_parser, ...])
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (Union[str, Path]) – 
new_indent_parser (bool) – 
apply_ocr (bool) – 
strategy (str) – 
llmsherpa_api_url (str) –
strategy (str) – 
llmsherpa_api_url (str) – 
__init__(file_path: Union[str, Path], new_indent_parser: bool = True, apply_ocr: bool = True, strategy: str = 'chunks', llmsherpa_api_url: str = 'https://readers.llmsherpa.com/api/document/developer/parseDocument?renderFormat=all')[source]¶
Initialize with a file path.
Parameters
file_path (Union[str, Path]) – 
new_indent_parser (bool) – 
apply_ocr (bool) – 
strategy (str) – 
llmsherpa_api_url (str) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using LLMSherpaFileLoader¶
LLM Sherpa
langchain_core.document_loaders.blob_loaders.Blob¶
class langchain_core.document_loaders.blob_loaders.Blob[source]¶
Bases: BaseModel
Blob represents raw data by either reference or value.
Provides an interface to materialize the blob in different representations, and
help to decouple the development of data loaders from the downstream parsing of
the raw data.
Inspired by: https://developer.mozilla.org/en-US/docs/Web/API/Blob
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param data: Optional[Union[bytes, str]] = None¶
Raw data associated with the blob.
param encoding: str = 'utf-8'¶
Encoding to use if decoding the bytes into a string.
Use utf-8 as default encoding, if decoding to string.
param metadata: Dict[str, Any] [Optional]¶
Metadata about the blob (e.g., source)
param mimetype: Optional[str] = None¶
MimeType not to be confused with a file extension.
param path: Optional[Union[str, PurePath]] = None¶
Location where the original content was found.
as_bytes() → bytes[source]¶
Read data as bytes.
Return type
bytes
as_bytes_io() → Generator[Union[BytesIO, BufferedReader], None, None][source]¶
Read data as a byte stream.
Return type
Generator[Union[BytesIO, BufferedReader], None, None]
as_string() → str[source]¶
Read data as a string.
Return type
str
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
by_alias (bool) – 
skip_defaults (Optional[bool]) – 
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
Return type
DictStrAny
classmethod from_data(data: Union[str, bytes], *, encoding: str = 'utf-8', mime_type: Optional[str] = None, path: Optional[str] = None, metadata: Optional[dict] = None) → Blob[source]¶
Initialize the blob from in-memory data.
Parameters
data (Union[str, bytes]) – the in-memory data associated with the blob
encoding (str) – Encoding to use if decoding the bytes into a string
mime_type (Optional[str]) – if provided, will be set as the mime-type of the data
path (Optional[str]) – if provided, will be set as the source from which the data came
metadata (Optional[dict]) – Metadata to associate with the blob
Returns
Blob instance
Return type
Blob
classmethod from_orm(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
classmethod from_path(path: Union[str, PurePath], *, encoding: str = 'utf-8', mime_type: Optional[str] = None, guess_type: bool = True, metadata: Optional[dict] = None) → Blob[source]¶
Load the blob from a path like object.
Parameters
path (Union[str, PurePath]) – path like object to file to be read
encoding (str) – Encoding to use if decoding the bytes into a string
mime_type (Optional[str]) – if provided, will be set as the mime-type of the data
guess_type (bool) – If True, the mimetype will be guessed from the file extension,
if a mime-type was not provided
metadata (Optional[dict]) – Metadata to associate with the blob
Returns
Blob instance
Return type
Blob
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
by_alias (bool) – 
skip_defaults (Optional[bool]) – 
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
encoder (Optional[Callable[[Any], Any]]) – 
models_as_dict (bool) – 
dumps_kwargs (Any) – 
Return type
unicode
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
path (Union[str, Path]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) –
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod parse_obj(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
b (Union[str, bytes]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
Return type
DictStrAny
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
dumps_kwargs (Any) – 
Return type
unicode
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
Parameters
localns (Any) – 
Return type
None
classmethod validate(value: Any) → Model¶
Parameters
value (Any) – 
Return type
Model
property source: Optional[str]¶
The source location of the blob as string if known otherwise none.
If a path is associated with the blob, it will default to the path location.
Unless explicitly set via a metadata field called “source”, in which
case that value will be used instead.
Examples using Blob¶
langchain_community.document_loaders.athena.AthenaLoader¶
class langchain_community.document_loaders.athena.AthenaLoader(query: str, database: str, s3_output_uri: str, profile_name: str, metadata_columns: Optional[List[str]] = None)[source]¶
Load documents from AWS Athena.
Each document represents one row of the result.
- By default, all columns are written into the page_content of the document
and none into the metadata of the document.
- If metadata_columns are provided then these columns are written
into the metadata of the document while the rest of the columns
are written into the page_content of the document.
To authenticate, the AWS client uses this method to automatically load credentials:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
If a specific credential profile should be used, you must pass
the name of the profile from the ~/.aws/credentials file that is to be used.
Make sure the credentials / roles used have the required policies to
access the Amazon Textract service.
Initialize Athena document loader.
Parameters
query (str) – The query to run in Athena.
database (str) – Athena database
s3_output_uri (str) – Athena output path
metadata_columns (Optional[List[str]]) – Optional. Columns written to Document metadata.
profile_name (str) – 
Methods
__init__(query, database, s3_output_uri, ...)
Initialize Athena document loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.