Tony20100/langchain_full huggingface dataset:
Returns
List of Documents.
Return type
List[Document]
scrape(parser: Optional[str] = None) → Any¶
Scrape data from webpage and return it in BeautifulSoup format.
Parameters
parser (Optional[str]) – 
Return type
Any
scrape_all(urls: List[str], parser: Optional[str] = None) → List[Any]¶
Fetch all urls, then return soups for all results.
Parameters
urls (List[str]) – 
parser (Optional[str]) – 
Return type
List[Any]
Examples using CollegeConfidentialLoader¶
College Confidential
langchain_community.document_loaders.parsers.language.perl.PerlSegmenter¶
class langchain_community.document_loaders.parsers.language.perl.PerlSegmenter(code: str)[source]¶
Code segmenter for Perl.
Methods
__init__(code)
extract_functions_classes()
get_chunk_query()
get_language()
get_parser()
is_valid()
make_line_comment(text)
simplify_code()
Parameters
code (str) – 
__init__(code: str)¶
Parameters
code (str) – 
extract_functions_classes() → List[str]¶
Return type
List[str]
get_chunk_query() → str[source]¶
Return type
str
get_language() → Language[source]¶
Return type
Language
get_parser() → Parser¶
Return type
Parser
is_valid() → bool¶
Return type
bool
make_line_comment(text: str) → str[source]¶
Parameters
text (str) – 
Return type
str
simplify_code() → str¶
Return type
str
langchain_core.document_loaders.base.BaseBlobParser¶
class langchain_core.document_loaders.base.BaseBlobParser[source]¶
Abstract interface for blob parsers.
A blob parser provides a way to parse raw data stored in a blob into one
or more documents.
The parser can be composed with blob loaders, making it easy to reuse
a parser independent of how the blob was originally loaded.
Methods
__init__()
lazy_parse(blob)
Lazy parsing interface.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__()¶
abstract lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazy parsing interface.
Subclasses are required to implement this method.
Parameters
blob (Blob) – Blob instance
Returns
Generator of documents
Return type
Iterator[Document]
parse(blob: Blob) → List[Document][source]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
Examples using BaseBlobParser¶
How to create a custom Document Loader
langchain_community.document_loaders.figma.FigmaFileLoader¶
class langchain_community.document_loaders.figma.FigmaFileLoader(access_token: str, ids: str, key: str)[source]¶
Load Figma file.
Initialize with access token, ids, and key.
Parameters
access_token (str) – The access token for the Figma REST API.
ids (str) – The ids of the Figma file.
key (str) – The key for the Figma file
Methods
__init__(access_token, ids, key)
Initialize with access token, ids, and key.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load file
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(access_token: str, ids: str, key: str)[source]¶
Initialize with access token, ids, and key.
Parameters
access_token (str) – The access token for the Figma REST API.
ids (str) – The ids of the Figma file.
key (str) – The key for the Figma file
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load file
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
langchain_community.document_loaders.parsers.language.c.CSegmenter¶
class langchain_community.document_loaders.parsers.language.c.CSegmenter(code: str)[source]¶
Code segmenter for C.
Methods
__init__(code)
extract_functions_classes()
get_chunk_query()
get_language()
get_parser()
is_valid()
make_line_comment(text)
simplify_code()
Parameters
code (str) – 
__init__(code: str)¶
Parameters
code (str) – 
extract_functions_classes() → List[str]¶
Return type
List[str]
get_chunk_query() → str[source]¶
Return type
str
get_language() → Language[source]¶
Return type
Language
get_parser() → Parser¶
Return type
Parser
is_valid() → bool¶
Return type
bool
make_line_comment(text: str) → str[source]¶
Parameters
text (str) – 
Return type
str
simplify_code() → str¶
Return type
str
langchain_community.document_loaders.astradb.AstraDBLoader¶
class langchain_community.document_loaders.astradb.AstraDBLoader(collection_name: str, *, token: Optional[str] = None, api_endpoint: Optional[str] = None, astra_db_client: Optional[AstraDB] = None, async_astra_db_client: Optional[AsyncAstraDB] = None, namespace: Optional[str] = None, filter_criteria: Optional[Dict[str, Any]] = None, projection: Optional[Dict[str, Any]] = None, find_options: Optional[Dict[str, Any]] = None, nb_prefetched: int = 1000, extraction_function: Callable[[Dict], str] = <function dumps>)[source]¶
[Deprecated]
Notes
Deprecated since version 0.0.29.
Load DataStax Astra DB documents.
Parameters
collection_name (str) – name of the Astra DB collection to use.
token (Optional[str]) – API token for Astra DB usage.
api_endpoint (Optional[str]) – full URL to the API endpoint,
such as https://<DB-ID>-us-east1.apps.astra.datastax.com.
astra_db_client (Optional[AstraDB]) – alternative to token+api_endpoint,
you can pass an already-created ‘astrapy.db.AstraDB’ instance.
async_astra_db_client (Optional[AsyncAstraDB]) – alternative to token+api_endpoint,
you can pass an already-created ‘astrapy.db.AsyncAstraDB’ instance.
namespace (Optional[str]) – namespace (aka keyspace) where the
collection is. Defaults to the database’s “default namespace”.
filter_criteria (Optional[Dict[str, Any]]) – Criteria to filter documents.
projection (Optional[Dict[str, Any]]) – Specifies the fields to return.
projection (Optional[Dict[str, Any]]) – Specifies the fields to return.
find_options (Optional[Dict[str, Any]]) – Additional options for the query.
nb_prefetched (int) – Max number of documents to pre-fetch. Defaults to 1000.
extraction_function (Callable[[Dict], str]) – Function applied to collection documents to create
the page_content of the LangChain Document. Defaults to json.dumps.
Methods
__init__(collection_name, *[, token, ...])
Load DataStax Astra DB documents.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(collection_name: str, *, token: Optional[str] = None, api_endpoint: Optional[str] = None, astra_db_client: Optional[AstraDB] = None, async_astra_db_client: Optional[AsyncAstraDB] = None, namespace: Optional[str] = None, filter_criteria: Optional[Dict[str, Any]] = None, projection: Optional[Dict[str, Any]] = None, find_options: Optional[Dict[str, Any]] = None, nb_prefetched: int = 1000, extraction_function: Callable[[Dict], str] = <function dumps>) → None[source]¶
Load DataStax Astra DB documents.
Parameters
collection_name (str) – name of the Astra DB collection to use.
token (Optional[str]) – API token for Astra DB usage.
api_endpoint (Optional[str]) – full URL to the API endpoint,
such as https://<DB-ID>-us-east1.apps.astra.datastax.com.
such as https://<DB-ID>-us-east1.apps.astra.datastax.com.
astra_db_client (Optional[AstraDB]) – alternative to token+api_endpoint,
you can pass an already-created ‘astrapy.db.AstraDB’ instance.
async_astra_db_client (Optional[AsyncAstraDB]) – alternative to token+api_endpoint,
you can pass an already-created ‘astrapy.db.AsyncAstraDB’ instance.
namespace (Optional[str]) – namespace (aka keyspace) where the
collection is. Defaults to the database’s “default namespace”.
filter_criteria (Optional[Dict[str, Any]]) – Criteria to filter documents.
projection (Optional[Dict[str, Any]]) – Specifies the fields to return.
find_options (Optional[Dict[str, Any]]) – Additional options for the query.
nb_prefetched (int) – Max number of documents to pre-fetch. Defaults to 1000.
extraction_function (Callable[[Dict], str]) – Function applied to collection documents to create
the page_content of the LangChain Document. Defaults to json.dumps.
Return type
None
async alazy_load() → AsyncIterator[Document][source]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
langchain_community.document_loaders.parsers.language.tree_sitter_segmenter.TreeSitterSegmenter¶
class langchain_community.document_loaders.parsers.language.tree_sitter_segmenter.TreeSitterSegmenter(code: str)[source]¶
Abstract class for `CodeSegmenter`s that use the tree-sitter library.
Methods
__init__(code)
extract_functions_classes()
get_chunk_query()
get_language()
get_parser()
is_valid()
make_line_comment(text)
simplify_code()
Parameters
code (str) – 
__init__(code: str)[source]¶
Parameters
code (str) – 
extract_functions_classes() → List[str][source]¶
Return type
List[str]
abstract get_chunk_query() → str[source]¶
Return type
str
abstract get_language() → Language[source]¶
Return type
Language
get_parser() → Parser[source]¶
Return type
Parser
is_valid() → bool[source]¶
Return type
bool
abstract make_line_comment(text: str) → str[source]¶
Parameters
text (str) – 
Return type
str
simplify_code() → str[source]¶
Return type
str
langchain_community.document_loaders.parsers.msword.MsWordParser¶
class langchain_community.document_loaders.parsers.msword.MsWordParser[source]¶
Parse the Microsoft Word documents from a blob.
Methods
__init__()
lazy_parse(blob)
Parse a Microsoft Word document into the Document iterator.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__()¶
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Parse a Microsoft Word document into the Document iterator.
Parameters
blob (Blob) – The blob to parse.
Return type
Iterator[Document]
Returns: An iterator of Documents.
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
langchain_community.document_loaders.github.GitHubIssuesLoader¶
class langchain_community.document_loaders.github.GitHubIssuesLoader[source]¶
Bases: BaseGitHubLoader
Load issues of a GitHub repository.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param access_token: str [Required]¶
Personal access token - see https://github.com/settings/tokens?type=beta
param assignee: Optional[str] = None¶
Filter on assigned user. Pass ‘none’ for no user and ‘*’ for any user.
param creator: Optional[str] = None¶
Filter on the user that created the issue.
param direction: Optional[Literal['asc', 'desc']] = None¶
The direction to sort the results by. Can be one of: ‘asc’, ‘desc’.
param github_api_url: str = 'https://api.github.com'¶
URL of GitHub API
param include_prs: bool = True¶
If True include Pull Requests in results, otherwise ignore them.
param labels: Optional[List[str]] = None¶
Label names to filter one. Example: bug,ui,@high.
param mentioned: Optional[str] = None¶
Filter on a user that’s mentioned in the issue.
param milestone: Optional[Union[int, Literal['*', 'none']]] = None¶
If integer is passed, it should be a milestone’s number field.
If the string ‘*’ is passed, issues with any milestone are accepted.
If the string ‘none’ is passed, issues without milestones are returned.
param page: Optional[int] = None¶
The page number for paginated results.
Defaults to 1 in the GitHub API.
param per_page: Optional[int] = None¶
Number of items per page.
param per_page: Optional[int] = None¶
Number of items per page.
Defaults to 30 in the GitHub API.
param repo: str [Required]¶
Name of repository
param since: Optional[str] = None¶
Only show notifications updated after the given time.
This is a timestamp in ISO 8601 format: YYYY-MM-DDTHH:MM:SSZ.
param sort: Optional[Literal['created', 'updated', 'comments']] = None¶
What to sort results by. Can be one of: ‘created’, ‘updated’, ‘comments’.
Default is ‘created’.
param state: Optional[Literal['open', 'closed', 'all']] = None¶
Filter on issue state. Can be one of: ‘open’, ‘closed’, ‘all’.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Parameters
obj (Any) – 
Return type
Model
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
by_alias (bool) – 
skip_defaults (Optional[bool]) – 
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
encoder (Optional[Callable[[Any], Any]]) – 
models_as_dict (bool) – 
dumps_kwargs (Any) – 
Return type
unicode
lazy_load() → Iterator[Document][source]¶
Get issues of a GitHub repository.
Returns
page_content
metadata
url
title
creator
created_at
last_update_time
closed_time
number of comments
state
labels
assignee
assignees
milestone
locked
number
is_pull_request
Return type
A list of Documents with attributes
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
path (Union[str, Path]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
parse_issue(issue: dict) → Document[source]¶
Create Document objects from a list of GitHub issues.
Parameters
issue (dict) – 
Return type
Document
classmethod parse_obj(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
b (Union[str, bytes]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
Parameters
by_alias (bool) –
Parameters
by_alias (bool) – 
ref_template (unicode) – 
Return type
DictStrAny
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
dumps_kwargs (Any) – 
Return type
unicode
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
Parameters
localns (Any) – 
Return type
None
classmethod validate(value: Any) → Model¶
Parameters
value (Any) – 
Return type
Model
property headers: Dict[str, str]¶
property query_params: str¶
Create query parameters for GitHub API.
property url: str¶
Create URL for GitHub API.
Examples using GitHubIssuesLoader¶
GitHub
langchain_community.document_loaders.readthedocs.ReadTheDocsLoader¶
class langchain_community.document_loaders.readthedocs.ReadTheDocsLoader(path: Union[str, Path], encoding: Optional[str] = None, errors: Optional[str] = None, custom_html_tag: Optional[Tuple[str, dict]] = None, patterns: Sequence[str] = ('*.htm', '*.html'), exclude_links_ratio: float = 1.0, **kwargs: Optional[Any])[source]¶
Load ReadTheDocs documentation directory.
Initialize ReadTheDocsLoader
The loader loops over all files under path and extracts the actual content of
the files by retrieving main html tags. Default main html tags include
<main id=”main-content>, <div role=”main>, and <article role=”main”>. You
can also define your own html tags by passing custom_html_tag, e.g.
(“div”, “class=main”). The loader iterates html tags with the order of
custom html tags (if exists) and default html tags. If any of the tags is not
empty, the loop will break and retrieve the content out of that tag.
Parameters
path (Union[str, Path]) – The location of pulled readthedocs folder.
encoding (Optional[str]) – The encoding with which to open the documents.
errors (Optional[str]) – Specify how encoding and decoding errors are to be handled—this
cannot be used in binary mode.
custom_html_tag (Optional[Tuple[str, dict]]) – Optional custom html tag to retrieve the content from
files.
patterns (Sequence[str]) – The file patterns to load, passed to glob.rglob.
exclude_links_ratio (float) – The ratio of links:content to exclude pages from.
This is to reduce the frequency at which index pages make their
way into retrieved results. Recommended: 0.5
way into retrieved results. Recommended: 0.5
kwargs (Optional[Any]) – named arguments passed to bs4.BeautifulSoup.
Methods
__init__(path[, encoding, errors, ...])
Initialize ReadTheDocsLoader
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(path: Union[str, Path], encoding: Optional[str] = None, errors: Optional[str] = None, custom_html_tag: Optional[Tuple[str, dict]] = None, patterns: Sequence[str] = ('*.htm', '*.html'), exclude_links_ratio: float = 1.0, **kwargs: Optional[Any])[source]¶
Initialize ReadTheDocsLoader
The loader loops over all files under path and extracts the actual content of
the files by retrieving main html tags. Default main html tags include
<main id=”main-content>, <div role=”main>, and <article role=”main”>. You
can also define your own html tags by passing custom_html_tag, e.g.
(“div”, “class=main”). The loader iterates html tags with the order of
custom html tags (if exists) and default html tags. If any of the tags is not
empty, the loop will break and retrieve the content out of that tag.
Parameters
path (Union[str, Path]) – The location of pulled readthedocs folder.
encoding (Optional[str]) – The encoding with which to open the documents.
errors (Optional[str]) – Specify how encoding and decoding errors are to be handled—this
cannot be used in binary mode.
cannot be used in binary mode.
custom_html_tag (Optional[Tuple[str, dict]]) – Optional custom html tag to retrieve the content from
files.
patterns (Sequence[str]) – The file patterns to load, passed to glob.rglob.
exclude_links_ratio (float) – The ratio of links:content to exclude pages from.
This is to reduce the frequency at which index pages make their
way into retrieved results. Recommended: 0.5
kwargs (Optional[Any]) – named arguments passed to bs4.BeautifulSoup.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using ReadTheDocsLoader¶
ReadTheDocs Documentation
langchain_community.document_loaders.sitemap.SitemapLoader¶
class langchain_community.document_loaders.sitemap.SitemapLoader(web_path: str, filter_urls: Optional[List[str]] = None, parsing_function: Optional[Callable] = None, blocksize: Optional[int] = None, blocknum: int = 0, meta_function: Optional[Callable] = None, is_local: bool = False, continue_on_failure: bool = False, restrict_to_same_domain: bool = True, **kwargs: Any)[source]¶
Load a sitemap and its URLs.
Security Note: This loader can be used to load all URLs specified in a sitemap.If a malicious actor gets access to the sitemap, they could force
the server to load URLs from other domains by modifying the sitemap.
This could lead to server-side request forgery (SSRF) attacks; e.g.,
with the attacker forcing the server to load URLs from internal
service endpoints that are not publicly accessible. While the attacker
may not immediately gain access to this data, this data could leak
into downstream systems (e.g., data loader is used to load data for indexing).
This loader is a crawler and web crawlers should generally NOT be deployed
with network access to any internal servers.
Control access to who can submit crawling requests and what network access
the crawler has.
By default, the loader will only load URLs from the same domain as the sitemap
if the site map is not a local file. This can be disabled by setting
restrict_to_same_domain to False (not recommended).
If the site map is a local file, no such risk mitigation is applied by default.
Use the filter URLs argument to limit which URLs can be loaded.
See https://python.langchain.com/docs/security
Initialize with webpage path and optional filter URLs.
Parameters