Tony20100/langchain_full huggingface dataset:
langchain_community.document_loaders.s3_file.S3FileLoader¶
class langchain_community.document_loaders.s3_file.S3FileLoader(bucket: str, key: str, *, region_name: Optional[str] = None, api_version: Optional[str] = None, use_ssl: Optional[bool] = True, verify: Union[str, bool, None] = None, endpoint_url: Optional[str] = None, aws_access_key_id: Optional[str] = None, aws_secret_access_key: Optional[str] = None, aws_session_token: Optional[str] = None, boto_config: Optional[botocore.client.Config] = None, mode: str = 'single', post_processors: Optional[List[Callable]] = None, **unstructured_kwargs: Any)[source]¶
Load from Amazon AWS S3 file.
Initialize with bucket and key name.
Parameters
bucket (str) – The name of the S3 bucket.
key (str) – The key of the S3 object.
region_name (Optional[str]) – The name of the region associated with the client.
A client is associated with a single region.
api_version (Optional[str]) – The API version to use.  By default, botocore will
use the latest API version when creating a client.  You only need
to specify this parameter if you want to use a previous API version
of the client.
use_ssl (Optional[bool]) – Whether or not to use SSL.  By default, SSL is used.
Note that not all services support non-ssl connections.
verify (Union[str, bool, None]) – Whether or not to verify SSL certificates.
By default SSL certificates are verified.  You can provide the
following values:
False - do not validate SSL certificates.  SSL will still be
used (unless use_ssl is False), but SSL certificates
will not be verified.
used (unless use_ssl is False), but SSL certificates
will not be verified.
path/to/cert/bundle.pem - A filename of the CA cert bundle to
uses.  You can specify this argument if you want to use a
different CA cert bundle than the one used by botocore.
endpoint_url (Optional[str]) – The complete URL to use for the constructed
client.  Normally, botocore will automatically construct the
appropriate URL to use when communicating with a service.  You can
specify a complete URL (including the “http/https” scheme) to
override this behavior.  If this value is provided, then
use_ssl is ignored.
aws_access_key_id (Optional[str]) – The access key to use when creating
the client.  This is entirely optional, and if not provided,
the credentials configured for the session will automatically
be used.  You only need to provide this argument if you want
to override the credentials used for this specific client.
aws_secret_access_key (Optional[str]) – The secret key to use when creating
the client.  Same semantics as aws_access_key_id above.
aws_session_token (Optional[str]) – The session token to use when creating
the client.  Same semantics as aws_access_key_id above.
boto_config (botocore.client.Config) – Advanced boto3 client configuration options. If a value
is specified in the client config, its value will take precedence
over environment variables and configuration values, but not over
a value passed explicitly to the method. If a default config
object is set on the session, the config object used when creating
the client will be the result of calling merge() on the
default config with the config provided to this call.
mode (str) – Mode in which to read the file. Valid options are: single,
paged and elements.
paged and elements.
post_processors (Optional[List[Callable]]) – Post processing functions to be applied to
extracted elements.
**unstructured_kwargs (Any) – Arbitrary additional kwargs to pass in when
calling partition
Methods
__init__(bucket, key, *[, region_name, ...])
Initialize with bucket and key name.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(bucket: str, key: str, *, region_name: Optional[str] = None, api_version: Optional[str] = None, use_ssl: Optional[bool] = True, verify: Union[str, bool, None] = None, endpoint_url: Optional[str] = None, aws_access_key_id: Optional[str] = None, aws_secret_access_key: Optional[str] = None, aws_session_token: Optional[str] = None, boto_config: Optional[botocore.client.Config] = None, mode: str = 'single', post_processors: Optional[List[Callable]] = None, **unstructured_kwargs: Any)[source]¶
Initialize with bucket and key name.
Parameters
bucket (str) – The name of the S3 bucket.
key (str) – The key of the S3 object.
region_name (Optional[str]) – The name of the region associated with the client.
A client is associated with a single region.
api_version (Optional[str]) – The API version to use.  By default, botocore will
use the latest API version when creating a client.  You only need
to specify this parameter if you want to use a previous API version
of the client.
to specify this parameter if you want to use a previous API version
of the client.
use_ssl (Optional[bool]) – Whether or not to use SSL.  By default, SSL is used.
Note that not all services support non-ssl connections.
verify (Union[str, bool, None]) – Whether or not to verify SSL certificates.
By default SSL certificates are verified.  You can provide the
following values:
False - do not validate SSL certificates.  SSL will still be
used (unless use_ssl is False), but SSL certificates
will not be verified.
path/to/cert/bundle.pem - A filename of the CA cert bundle to
uses.  You can specify this argument if you want to use a
different CA cert bundle than the one used by botocore.
endpoint_url (Optional[str]) – The complete URL to use for the constructed
client.  Normally, botocore will automatically construct the
appropriate URL to use when communicating with a service.  You can
specify a complete URL (including the “http/https” scheme) to
override this behavior.  If this value is provided, then
use_ssl is ignored.
aws_access_key_id (Optional[str]) – The access key to use when creating
the client.  This is entirely optional, and if not provided,
the credentials configured for the session will automatically
be used.  You only need to provide this argument if you want
to override the credentials used for this specific client.
aws_secret_access_key (Optional[str]) – The secret key to use when creating
the client.  Same semantics as aws_access_key_id above.
aws_session_token (Optional[str]) – The session token to use when creating
the client.  Same semantics as aws_access_key_id above.
boto_config (botocore.client.Config) – Advanced boto3 client configuration options. If a value
is specified in the client config, its value will take precedence
over environment variables and configuration values, but not over
a value passed explicitly to the method. If a default config
object is set on the session, the config object used when creating
the client will be the result of calling merge() on the
default config with the config provided to this call.
mode (str) – Mode in which to read the file. Valid options are: single,
paged and elements.
post_processors (Optional[List[Callable]]) – Post processing functions to be applied to
extracted elements.
**unstructured_kwargs (Any) – Arbitrary additional kwargs to pass in when
calling partition
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using S3FileLoader¶
AWS
AWS S3 File
langchain_community.document_loaders.parsers.audio.OpenAIWhisperParserLocal¶
class langchain_community.document_loaders.parsers.audio.OpenAIWhisperParserLocal(device: str = '0', lang_model: Optional[str] = None, batch_size: int = 8, chunk_length: int = 30, forced_decoder_ids: Optional[Tuple[Dict]] = None)[source]¶
Transcribe and parse audio files with OpenAI Whisper model.
Audio transcription with OpenAI Whisper model locally from transformers.
Parameters:
device - device to use
NOTE: By default uses the gpu if available,
if you want to use cpu, please set device = “cpu”
lang_model - whisper model to use, for example “openai/whisper-medium”
forced_decoder_ids - id states for decoder in multilanguage model,
usage example:
from transformers import WhisperProcessor
processor = WhisperProcessor.from_pretrained(“openai/whisper-medium”)
forced_decoder_ids = WhisperProcessor.get_decoder_prompt_ids(language=”french”,
task=”transcribe”)
forced_decoder_ids = WhisperProcessor.get_decoder_prompt_ids(language=”french”,
task=”translate”)
Initialize the parser.
Parameters
device (str) – device to use.
lang_model (Optional[str]) – whisper model to use, for example “openai/whisper-medium”.
Defaults to None.
forced_decoder_ids (Optional[Tuple[Dict]]) – id states for decoder in a multilanguage model.
Defaults to None.
batch_size (int) – batch size used for decoding
Defaults to 8.
chunk_length (int) – chunk length used during inference.
Defaults to 30s.
Methods
__init__([device, lang_model, batch_size, ...])
Initialize the parser.
lazy_parse(blob)
Lazily parse the blob.
parse(blob)
lazy_parse(blob)
Lazily parse the blob.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__(device: str = '0', lang_model: Optional[str] = None, batch_size: int = 8, chunk_length: int = 30, forced_decoder_ids: Optional[Tuple[Dict]] = None)[source]¶
Initialize the parser.
Parameters
device (str) – device to use.
lang_model (Optional[str]) – whisper model to use, for example “openai/whisper-medium”.
Defaults to None.
forced_decoder_ids (Optional[Tuple[Dict]]) – id states for decoder in a multilanguage model.
Defaults to None.
batch_size (int) – batch size used for decoding
Defaults to 8.
chunk_length (int) – chunk length used during inference.
Defaults to 30s.
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazily parse the blob.
Parameters
blob (Blob) – 
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
langchain_community.document_loaders.roam.RoamLoader¶
class langchain_community.document_loaders.roam.RoamLoader(path: Union[str, Path])[source]¶
Load Roam files from a directory.
Initialize with a path.
Methods
__init__(path)
Initialize with a path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
path (Union[str, Path]) – 
__init__(path: Union[str, Path])[source]¶
Initialize with a path.
Parameters
path (Union[str, Path]) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using RoamLoader¶
Roam
langchain_community.document_loaders.chatgpt.ChatGPTLoader¶
class langchain_community.document_loaders.chatgpt.ChatGPTLoader(log_file: str, num_logs: int = -1)[source]¶
Load conversations from exported ChatGPT data.
Initialize a class object.
Parameters
log_file (str) – Path to the log file
num_logs (int) – Number of logs to load. If 0, load all logs.
Methods
__init__(log_file[, num_logs])
Initialize a class object.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(log_file: str, num_logs: int = -1)[source]¶
Initialize a class object.
Parameters
log_file (str) – Path to the log file
num_logs (int) – Number of logs to load. If 0, load all logs.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
langchain_community.document_loaders.brave_search.BraveSearchLoader¶
class langchain_community.document_loaders.brave_search.BraveSearchLoader(query: str, api_key: str, search_kwargs: Optional[dict] = None)[source]¶
Load with Brave Search engine.
Initializes the BraveLoader.
Parameters
query (str) – The query to search for.
api_key (str) – The API key to use.
search_kwargs (Optional[dict]) – The search kwargs to use.
Methods
__init__(query, api_key[, search_kwargs])
Initializes the BraveLoader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(query: str, api_key: str, search_kwargs: Optional[dict] = None)[source]¶
Initializes the BraveLoader.
Parameters
query (str) – The query to search for.
api_key (str) – The API key to use.
search_kwargs (Optional[dict]) – The search kwargs to use.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
langchain_community.document_loaders.blob_loaders.cloud_blob_loader.CloudBlobLoader¶
class langchain_community.document_loaders.blob_loaders.cloud_blob_loader.CloudBlobLoader(url: Union[str, AnyPath], *, glob: str = '**/[!.]*', exclude: Sequence[str] = (), suffixes: Optional[Sequence[str]] = None, show_progress: bool = False)[source]¶
Load blobs from cloud URL or file:.
Example:
loader = CloudBlobLoader("s3://mybucket/id")
for blob in loader.yield_blobs():
    print(blob)
Initialize with a url and how to glob over it.
Use [CloudPathLib](https://cloudpathlib.drivendata.org/).
Parameters
url (Union[str, AnyPath]) – Cloud URL to load from.
Supports s3://, az://, gs://, file:// schemes.
If no scheme is provided, it is assumed to be a local file.
If a path to a file is provided, glob/exclude/suffixes are ignored.
glob (str) – Glob pattern relative to the specified path
by default set to pick up all non-hidden files
exclude (Sequence[str]) – patterns to exclude from results, use glob syntax
suffixes (Optional[Sequence[str]]) – Provide to keep only files with these suffixes
Useful when wanting to keep files with different suffixes
Suffixes must include the dot, e.g. “.txt”
show_progress (bool) – If true, will show a progress bar as the files are loaded.
This forces an iteration through all matching files
to count them prior to loading them.
Examples
Methods
__init__(url, *[, glob, exclude, suffixes, ...])
Initialize with a url and how to glob over it.
count_matching_files()
Count files that match the pattern without loading them.
count_matching_files()
Count files that match the pattern without loading them.
from_path(path, *[, encoding, mime_type, ...])
Load the blob from a path like object.
yield_blobs()
Yield blobs that match the requested pattern.
__init__(url: Union[str, AnyPath], *, glob: str = '**/[!.]*', exclude: Sequence[str] = (), suffixes: Optional[Sequence[str]] = None, show_progress: bool = False) → None[source]¶
Initialize with a url and how to glob over it.
Use [CloudPathLib](https://cloudpathlib.drivendata.org/).
Parameters
url (Union[str, AnyPath]) – Cloud URL to load from.
Supports s3://, az://, gs://, file:// schemes.
If no scheme is provided, it is assumed to be a local file.
If a path to a file is provided, glob/exclude/suffixes are ignored.
glob (str) – Glob pattern relative to the specified path
by default set to pick up all non-hidden files
exclude (Sequence[str]) – patterns to exclude from results, use glob syntax
suffixes (Optional[Sequence[str]]) – Provide to keep only files with these suffixes
Useful when wanting to keep files with different suffixes
Suffixes must include the dot, e.g. “.txt”
show_progress (bool) – If true, will show a progress bar as the files are loaded.
This forces an iteration through all matching files
to count them prior to loading them.
Return type
None
Examples
count_matching_files() → int[source]¶
Count files that match the pattern without loading them.
Return type
int
Count files that match the pattern without loading them.
Return type
int
classmethod from_path(path: AnyPath, *, encoding: str = 'utf-8', mime_type: Optional[str] = None, guess_type: bool = True, metadata: Optional[dict] = None) → Blob[source]¶
Load the blob from a path like object.
Parameters
path (AnyPath) – path like object to file to be read
Supports s3://, az://, gs://, file:// schemes.
If no scheme is provided, it is assumed to be a local file.
encoding (str) – Encoding to use if decoding the bytes into a string
mime_type (Optional[str]) – if provided, will be set as the mime-type of the data
guess_type (bool) – If True, the mimetype will be guessed from the file extension,
if a mime-type was not provided
metadata (Optional[dict]) – Metadata to associate with the blob
Returns
Blob instance
Return type
Blob
yield_blobs() → Iterable[Blob][source]¶
Yield blobs that match the requested pattern.
Return type
Iterable[Blob]
langchain_community.document_loaders.sql_database.SQLDatabaseLoader¶
class langchain_community.document_loaders.sql_database.SQLDatabaseLoader(query: Union[str, Select], db: SQLDatabase, *, parameters: Optional[Dict[str, Any]] = None, page_content_mapper: Optional[Callable[[...], str]] = None, metadata_mapper: Optional[Callable[[...], Dict[str, Any]]] = None, source_columns: Optional[Sequence[str]] = None, include_rownum_into_metadata: bool = False, include_query_into_metadata: bool = False)[source]¶
Load documents by querying database tables supported by SQLAlchemy.
For talking to the database, the document loader uses the SQLDatabase
utility from the LangChain integration toolkit.
Each document represents one row of the result.
Parameters
query (Union[str, Select]) – The query to execute.
db (SQLDatabase) – A LangChain SQLDatabase, wrapping an SQLAlchemy engine.
sqlalchemy_kwargs – More keyword arguments for SQLAlchemy’s create_engine.
parameters (Optional[Dict[str, Any]]) – Optional. Parameters to pass to the query.
page_content_mapper (Optional[Callable[[...], str]]) – Optional. Function to convert a row into a string
to use as the page_content of the document. By default, the loader
serializes the whole row into a string, including all columns.
metadata_mapper (Optional[Callable[[...], Dict[str, Any]]]) – Optional. Function to convert a row into a dictionary
to use as the metadata of the document. By default, no columns are
selected into the metadata dictionary.
source_columns (Optional[Sequence[str]]) – Optional. The names of the columns to use as the source
within the metadata dictionary.
include_rownum_into_metadata (bool) – Optional. Whether to include the row number
into the metadata dictionary. Default: False.
into the metadata dictionary. Default: False.
include_query_into_metadata (bool) – Optional. Whether to include the query
expression into the metadata dictionary. Default: False.
Methods
__init__(query, db, *[, parameters, ...])
param query
The query to execute.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
metadata_default_mapper(row[, column_names])
A reasonable default function to convert a record into a "metadata" dictionary.
page_content_default_mapper(row[, column_names])
A reasonable default function to convert a record into a "page content" string.
__init__(query: Union[str, Select], db: SQLDatabase, *, parameters: Optional[Dict[str, Any]] = None, page_content_mapper: Optional[Callable[[...], str]] = None, metadata_mapper: Optional[Callable[[...], Dict[str, Any]]] = None, source_columns: Optional[Sequence[str]] = None, include_rownum_into_metadata: bool = False, include_query_into_metadata: bool = False)[source]¶
Parameters
query (Union[str, Select]) – The query to execute.
db (SQLDatabase) – A LangChain SQLDatabase, wrapping an SQLAlchemy engine.
sqlalchemy_kwargs – More keyword arguments for SQLAlchemy’s create_engine.
parameters (Optional[Dict[str, Any]]) – Optional. Parameters to pass to the query.
page_content_mapper (Optional[Callable[[...], str]]) – Optional. Function to convert a row into a string
to use as the page_content of the document. By default, the loader
serializes the whole row into a string, including all columns.
serializes the whole row into a string, including all columns.
metadata_mapper (Optional[Callable[[...], Dict[str, Any]]]) – Optional. Function to convert a row into a dictionary
to use as the metadata of the document. By default, no columns are
selected into the metadata dictionary.
source_columns (Optional[Sequence[str]]) – Optional. The names of the columns to use as the source
within the metadata dictionary.
include_rownum_into_metadata (bool) – Optional. Whether to include the row number
into the metadata dictionary. Default: False.
include_query_into_metadata (bool) – Optional. Whether to include the query
expression into the metadata dictionary. Default: False.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
static metadata_default_mapper(row: RowMapping, column_names: Optional[List[str]] = None) → Dict[str, Any][source]¶
A reasonable default function to convert a record into a “metadata” dictionary.
Parameters