Tony20100/langchain_full huggingface dataset:
langchain_community.document_loaders.slack_directory.SlackDirectoryLoader¶
class langchain_community.document_loaders.slack_directory.SlackDirectoryLoader(zip_path: Union[str, Path], workspace_url: Optional[str] = None)[source]¶
Load from a Slack directory dump.
Initialize the SlackDirectoryLoader.
Parameters
zip_path (str) – The path to the Slack directory dump zip file.
workspace_url (Optional[str]) – The Slack workspace URL.
Including the URL will turn
sources into links. Defaults to None.
Methods
__init__(zip_path[, workspace_url])
Initialize the SlackDirectoryLoader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load and return documents from the Slack directory dump.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(zip_path: Union[str, Path], workspace_url: Optional[str] = None)[source]¶
Initialize the SlackDirectoryLoader.
Parameters
zip_path (str) – The path to the Slack directory dump zip file.
workspace_url (Optional[str]) – The Slack workspace URL.
Including the URL will turn
sources into links. Defaults to None.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load and return documents from the Slack directory dump.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using SlackDirectoryLoader¶
Slack
langchain_community.document_loaders.oracleai.OracleTextSplitter¶
class langchain_community.document_loaders.oracleai.OracleTextSplitter(conn: Connection, params: Dict[str, Any], **kwargs: Any)[source]¶
Splitting text using Oracle chunker.
Initialize.
Methods
__init__(conn, params, **kwargs)
Initialize.
atransform_documents(documents, **kwargs)
Asynchronously transform a list of documents.
create_documents(texts[, metadatas])
Create documents from a list of texts.
from_huggingface_tokenizer(tokenizer, **kwargs)
Text splitter that uses HuggingFace tokenizer to count length.
from_tiktoken_encoder([encoding_name, ...])
Text splitter that uses tiktoken encoder to count length.
split_documents(documents)
Split documents.
split_text(text)
Split incoming text and return chunks.
transform_documents(documents, **kwargs)
Transform sequence of documents by splitting them.
Parameters
conn (Connection) – 
params (Dict[str, Any]) – 
kwargs (Any) – 
__init__(conn: Connection, params: Dict[str, Any], **kwargs: Any) → None[source]¶
Initialize.
Parameters
conn (Connection) – 
params (Dict[str, Any]) – 
kwargs (Any) – 
Return type
None
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document]¶
Asynchronously transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
create_documents(texts: List[str], metadatas: Optional[List[dict]] = None) → List[Document]¶
Create documents from a list of texts.
Parameters
texts (List[str]) – 
metadatas (Optional[List[dict]]) – 
Return type
List[Document]
classmethod from_huggingface_tokenizer(tokenizer: Any, **kwargs: Any) → TextSplitter¶
Text splitter that uses HuggingFace tokenizer to count length.
Parameters
tokenizer (Any) – 
kwargs (Any) – 
Return type
TextSplitter
classmethod from_tiktoken_encoder(encoding_name: str = 'gpt2', model_name: Optional[str] = None, allowed_special: Union[Literal['all'], AbstractSet[str]] = {}, disallowed_special: Union[Literal['all'], Collection[str]] = 'all', **kwargs: Any) → TS¶
Text splitter that uses tiktoken encoder to count length.
Parameters
encoding_name (str) – 
model_name (Optional[str]) – 
allowed_special (Union[Literal['all'], ~typing.AbstractSet[str]]) – 
disallowed_special (Union[Literal['all'], ~typing.Collection[str]]) – 
kwargs (Any) – 
Return type
TS
split_documents(documents: Iterable[Document]) → List[Document]¶
Split documents.
Parameters
documents (Iterable[Document]) – 
Return type
List[Document]
split_text(text: str) → List[str][source]¶
Split incoming text and return chunks.
Parameters
text (str) – 
Return type
List[str]
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document]¶
Transform sequence of documents by splitting them.
Parameters
documents (Sequence[Document]) – 
kwargs (Any) – 
Return type
Sequence[Document]
Examples using OracleTextSplitter¶
Oracle AI Vector Search: Document Processing
OracleAI Vector Search
langchain_community.document_loaders.parsers.pdf.PyPDFium2Parser¶
class langchain_community.document_loaders.parsers.pdf.PyPDFium2Parser(extract_images: bool = False)[source]¶
Parse PDF with PyPDFium2.
Initialize the parser.
Methods
__init__([extract_images])
Initialize the parser.
lazy_parse(blob)
Lazily parse the blob.
parse(blob)
Eagerly parse the blob into a document or documents.
Parameters
extract_images (bool) – 
__init__(extract_images: bool = False) → None[source]¶
Initialize the parser.
Parameters
extract_images (bool) – 
Return type
None
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazily parse the blob.
Parameters
blob (Blob) – 
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
langchain_community.document_loaders.snowflake_loader.SnowflakeLoader¶
class langchain_community.document_loaders.snowflake_loader.SnowflakeLoader(query: str, user: str, password: str, account: str, warehouse: str, role: str, database: str, schema: str, parameters: Optional[Dict[str, Any]] = None, page_content_columns: Optional[List[str]] = None, metadata_columns: Optional[List[str]] = None)[source]¶
Load from Snowflake API.
Each document represents one row of the result. The page_content_columns
are written into the page_content of the document. The metadata_columns
are written into the metadata of the document. By default, all columns
are written into the page_content and none into the metadata.
Initialize Snowflake document loader.
Parameters
query (str) – The query to run in Snowflake.
user (str) – Snowflake user.
password (str) – Snowflake password.
account (str) – Snowflake account.
warehouse (str) – Snowflake warehouse.
role (str) – Snowflake role.
database (str) – Snowflake database
schema (str) – Snowflake schema
parameters (Optional[Dict[str, Any]]) – Optional. Parameters to pass to the query.
page_content_columns (Optional[List[str]]) – Optional. Columns written to Document page_content.
metadata_columns (Optional[List[str]]) – Optional. Columns written to Document metadata.
Methods
__init__(query, user, password, account, ...)
Initialize Snowflake document loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(query: str, user: str, password: str, account: str, warehouse: str, role: str, database: str, schema: str, parameters: Optional[Dict[str, Any]] = None, page_content_columns: Optional[List[str]] = None, metadata_columns: Optional[List[str]] = None)[source]¶
Initialize Snowflake document loader.
Parameters
query (str) – The query to run in Snowflake.
user (str) – Snowflake user.
password (str) – Snowflake password.
account (str) – Snowflake account.
warehouse (str) – Snowflake warehouse.
role (str) – Snowflake role.
database (str) – Snowflake database
schema (str) – Snowflake schema
parameters (Optional[Dict[str, Any]]) – Optional. Parameters to pass to the query.
page_content_columns (Optional[List[str]]) – Optional. Columns written to Document page_content.
metadata_columns (Optional[List[str]]) – Optional. Columns written to Document metadata.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
langchain_community.document_loaders.firecrawl.FireCrawlLoader¶
class langchain_community.document_loaders.firecrawl.FireCrawlLoader(url: str, *, api_key: Optional[str] = None, mode: Literal['crawl', 'scrape'] = 'crawl', params: Optional[dict] = None)[source]¶
Load web pages as Documents using FireCrawl.
Must have Python package firecrawl installed and a FireCrawl API key. Seehttps://www.firecrawl.dev/ for more.
Initialize with API key and url.
Parameters
url (str) – The url to be crawled.
api_key (Optional[str]) – The Firecrawl API key. If not specified will be read from env var
FIRECRAWL_API_KEY. Get an API key
mode (Literal['crawl', 'scrape']) – The mode to run the loader in. Default is “crawl”.
Options include “scrape” (single url) and
“crawl” (all accessible sub pages).
params (Optional[dict]) – The parameters to pass to the Firecrawl API.
Examples include crawlerOptions.
For more details, visit: https://github.com/mendableai/firecrawl-py
Methods
__init__(url, *[, api_key, mode, params])
Initialize with API key and url.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(url: str, *, api_key: Optional[str] = None, mode: Literal['crawl', 'scrape'] = 'crawl', params: Optional[dict] = None)[source]¶
Initialize with API key and url.
Parameters
Initialize with API key and url.
Parameters
url (str) – The url to be crawled.
api_key (Optional[str]) – The Firecrawl API key. If not specified will be read from env var
FIRECRAWL_API_KEY. Get an API key
mode (Literal['crawl', 'scrape']) – The mode to run the loader in. Default is “crawl”.
Options include “scrape” (single url) and
“crawl” (all accessible sub pages).
params (Optional[dict]) – The parameters to pass to the Firecrawl API.
Examples include crawlerOptions.
For more details, visit: https://github.com/mendableai/firecrawl-py
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using FireCrawlLoader¶
FireCrawl
langchain_community.document_loaders.twitter.TwitterTweetLoader¶
class langchain_community.document_loaders.twitter.TwitterTweetLoader(auth_handler: Union[OAuthHandler, OAuth2BearerHandler], twitter_users: Sequence[str], number_tweets: Optional[int] = 100)[source]¶
Load Twitter tweets.
Read tweets of the user’s Twitter handle.
First you need to go to
https://developer.twitter.com/en/docs/twitter-api
/getting-started/getting-access-to-the-twitter-api
to get your token. And create a v2 version of the app.
Methods
__init__(auth_handler, twitter_users[, ...])
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
from_bearer_token(oauth2_bearer_token, ...)
Create a TwitterTweetLoader from OAuth2 bearer token.
from_secrets(access_token, ...[, number_tweets])
Create a TwitterTweetLoader from access tokens and secrets.
lazy_load()
A lazy loader for Documents.
load()
Load tweets.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
auth_handler (Union[OAuthHandler, OAuth2BearerHandler]) – 
twitter_users (Sequence[str]) – 
number_tweets (Optional[int]) – 
__init__(auth_handler: Union[OAuthHandler, OAuth2BearerHandler], twitter_users: Sequence[str], number_tweets: Optional[int] = 100)[source]¶
Parameters
auth_handler (Union[OAuthHandler, OAuth2BearerHandler]) – 
twitter_users (Sequence[str]) – 
number_tweets (Optional[int]) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
classmethod from_bearer_token(oauth2_bearer_token: str, twitter_users: Sequence[str], number_tweets: Optional[int] = 100) → TwitterTweetLoader[source]¶
Create a TwitterTweetLoader from OAuth2 bearer token.
Parameters
oauth2_bearer_token (str) – 
twitter_users (Sequence[str]) – 
number_tweets (Optional[int]) – 
Return type
TwitterTweetLoader
classmethod from_secrets(access_token: str, access_token_secret: str, consumer_key: str, consumer_secret: str, twitter_users: Sequence[str], number_tweets: Optional[int] = 100) → TwitterTweetLoader[source]¶
Create a TwitterTweetLoader from access tokens and secrets.
Parameters
access_token (str) – 
access_token_secret (str) – 
consumer_key (str) – 
consumer_secret (str) – 
twitter_users (Sequence[str]) – 
number_tweets (Optional[int]) – 
Return type
TwitterTweetLoader
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load tweets.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using TwitterTweetLoader¶
Twitter
langchain_community.document_loaders.airbyte.AirbyteTypeformLoader¶
class langchain_community.document_loaders.airbyte.AirbyteTypeformLoader(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None)[source]¶
Load from Typeform using an Airbyte source connector.
Initializes the loader.
Parameters
config (Mapping[str, Any]) – The config to pass to the source connector.
stream_name (str) – The name of the stream to load.
record_handler (Optional[Callable[[Any, Optional[str]], Document]]) – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state (Optional[Any]) – The state to pass to the source connector. Defaults to None.
Attributes
last_state
Methods
__init__(config, stream_name[, ...])
Initializes the loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None) → None[source]¶
Initializes the loader.
Parameters
config (Mapping[str, Any]) – The config to pass to the source connector.
stream_name (str) – The name of the stream to load.
record_handler (Optional[Callable[[Any, Optional[str]], Document]]) – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state (Optional[Any]) – The state to pass to the source connector. Defaults to None.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using AirbyteTypeformLoader¶
Airbyte Typeform (Deprecated)
langchain_community.document_loaders.pdf.PyPDFLoader¶
class langchain_community.document_loaders.pdf.PyPDFLoader(file_path: str, password: Optional[Union[str, bytes]] = None, headers: Optional[Dict] = None, extract_images: bool = False)[source]¶
Load PDF using pypdf into list of documents.
Loader chunks by page and stores page numbers in metadata.
Initialize with a file path.
Attributes
source
Methods
__init__(file_path[, password, headers, ...])
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load given path as pages.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (str) – 
password (Optional[Union[str, bytes]]) – 
headers (Optional[Dict]) – 
extract_images (bool) – 
__init__(file_path: str, password: Optional[Union[str, bytes]] = None, headers: Optional[Dict] = None, extract_images: bool = False) → None[source]¶
Initialize with a file path.
Parameters
file_path (str) – 
password (Optional[Union[str, bytes]]) – 
headers (Optional[Dict]) – 
extract_images (bool) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load given path as pages.
Return type
Iterator[Document]
Lazy load given path as pages.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using PyPDFLoader¶
Apache Cassandra
Astra DB
Document Comparison
Google Cloud Storage File
Google Vertex AI Vector Search
How to load PDFs
KDB.AI
Merge Documents Loader
MongoDB Atlas
langchain_community.document_loaders.srt.SRTLoader¶
class langchain_community.document_loaders.srt.SRTLoader(file_path: Union[str, Path])[source]¶
Load .srt (subtitle) files.
Initialize with a file path.
Methods
__init__(file_path)
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load using pysrt file.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (Union[str, Path]) – 
__init__(file_path: Union[str, Path])[source]¶
Initialize with a file path.
Parameters
file_path (Union[str, Path]) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load using pysrt file.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using SRTLoader¶
Subtitle
langchain_community.document_loaders.github.BaseGitHubLoader¶
class langchain_community.document_loaders.github.BaseGitHubLoader[source]¶
Bases: BaseLoader, BaseModel, ABC
Load GitHub repository Issues.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param access_token: str [Required]¶
Personal access token - see https://github.com/settings/tokens?type=beta
param github_api_url: str = 'https://api.github.com'¶
URL of GitHub API
param repo: str [Required]¶
Name of repository
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to include in new model