Tony20100/langchain_full huggingface dataset:
langchain_community.document_loaders.evernote.EverNoteLoader¶
class langchain_community.document_loaders.evernote.EverNoteLoader(file_path: Union[str, Path], load_single_document: bool = True)[source]¶
Load from EverNote.
Loads an EverNote notebook export file e.g. my_notebook.enex into Documents.
Instructions on producing this file can be found at
https://help.evernote.com/hc/en-us/articles/209005557-Export-notes-and-notebooks-as-ENEX-or-HTML
Currently only the plain text in the note is extracted and stored as the contents
of the Document, any non content metadata (e.g. ‘author’, ‘created’, ‘updated’ etc.
but not ‘content-raw’ or ‘resource’) tags on the note will be extracted and stored
as metadata on the Document.
Parameters
file_path (str) – The path to the notebook export with a .enex extension
load_single_document (bool) – Whether or not to concatenate the content of all
notes into a single long Document.
True (If this is set to) – the ‘source’ which contains the file name of the export.
Initialize with file path.
Methods
__init__(file_path[, load_single_document])
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load documents from EverNote export file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path], load_single_document: bool = True)[source]¶
Initialize with file path.
Parameters
file_path (Union[str, Path]) – 
load_single_document (bool) –
file_path (Union[str, Path]) – 
load_single_document (bool) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load documents from EverNote export file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using EverNoteLoader¶
EverNote
langchain_community.document_loaders.parsers.pdf.AmazonTextractPDFParser¶
class langchain_community.document_loaders.parsers.pdf.AmazonTextractPDFParser(textract_features: Optional[Sequence[int]] = None, client: Optional[Any] = None, *, linearization_config: Optional['TextLinearizationConfig'] = None)[source]¶
Send PDF files to Amazon Textract and parse them.
For parsing multi-page PDFs, they have to reside on S3.
The AmazonTextractPDFLoader calls the
[Amazon Textract Service](https://aws.amazon.com/textract/)
to convert PDFs into a Document structure.
Single and multi-page documents are supported with up to 3000 pages
and 512 MB of size.
For the call to be successful an AWS account is required,
similar to the
[AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)
requirements.
Besides the AWS configuration, it is very similar to the other PDF
loaders, while also supporting JPEG, PNG and TIFF and non-native
PDF formats.
`python
from langchain_community.document_loaders import AmazonTextractPDFLoader
loader=AmazonTextractPDFLoader("example_data/alejandro_rosalez_sample-small.jpeg")
documents = loader.load()
`
One feature is the linearization of the output.
When using the features LAYOUT, FORMS or TABLES together with Textract
```python
from langchain_community.document_loaders import AmazonTextractPDFLoader
# you can mix and match each of the features
loader=AmazonTextractPDFLoader(
“example_data/alejandro_rosalez_sample-small.jpeg”,
textract_features=[“TABLES”, “LAYOUT”])
documents = loader.load()
```
it will generate output that formats the text in reading order and
```
it will generate output that formats the text in reading order and
try to output the information in a tabular structure or
output the key/value pairs with a colon (key: value).
This helps most LLMs to achieve better accuracy when
processing these texts.
Initializes the parser.
Parameters
textract_features (Optional[Sequence[int]]) – Features to be used for extraction, each feature
should be passed as an int that conforms to the enum
Textract_Features, see amazon-textract-caller pkg
client (Optional[Any]) – boto3 textract client
linearization_config (Optional['TextLinearizationConfig']) – Config to be used for linearization of the output
should be an instance of TextLinearizationConfig from
the textractor pkg
Methods
__init__([textract_features, client, ...])
Initializes the parser.
lazy_parse(blob)
Iterates over the Blob pages and returns an Iterator with a Document for each page, like the other parsers If multi-page document, blob.path has to be set to the S3 URI and for single page docs the blob.data is taken
parse(blob)
Eagerly parse the blob into a document or documents.
__init__(textract_features: Optional[Sequence[int]] = None, client: Optional[Any] = None, *, linearization_config: Optional['TextLinearizationConfig'] = None) → None[source]¶
Initializes the parser.
Parameters
textract_features (Optional[Sequence[int]]) – Features to be used for extraction, each feature
should be passed as an int that conforms to the enum
Textract_Features, see amazon-textract-caller pkg
client (Optional[Any]) – boto3 textract client
linearization_config (Optional['TextLinearizationConfig']) – Config to be used for linearization of the output
should be an instance of TextLinearizationConfig from
should be an instance of TextLinearizationConfig from
the textractor pkg
Return type
None
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Iterates over the Blob pages and returns an Iterator with a Document
for each page, like the other parsers If multi-page document, blob.path
has to be set to the S3 URI and for single page docs
the blob.data is taken
Parameters
blob (Blob) – 
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
langchain_community.document_loaders.toml.TomlLoader¶
class langchain_community.document_loaders.toml.TomlLoader(source: Union[str, Path])[source]¶
Load TOML files.
It can load a single source file or several files in a single
directory.
Initialize the TomlLoader with a source file or directory.
Methods
__init__(source)
Initialize the TomlLoader with a source file or directory.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazily load the TOML documents from the source file or directory.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
source (Union[str, Path]) – 
__init__(source: Union[str, Path])[source]¶
Initialize the TomlLoader with a source file or directory.
Parameters
source (Union[str, Path]) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazily load the TOML documents from the source file or directory.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
langchain_community.document_loaders.github.GithubFileLoader¶
class langchain_community.document_loaders.github.GithubFileLoader[source]¶
Bases: BaseGitHubLoader, ABC
Load GitHub File
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param access_token: str [Required]¶
Personal access token - see https://github.com/settings/tokens?type=beta
param branch: str = 'main'¶
param file_extension: str = '.md'¶
param file_filter: Optional[Callable[[str], bool]] = None¶
param github_api_url: str = 'https://api.github.com'¶
URL of GitHub API
param repo: str [Required]¶
Name of repository
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to include in new model
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to exclude from new model, as with values this takes precedence over include
update (Optional[DictStrAny]) – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
deep (bool) – set to True to make a deep copy of the model
self (Model) – 
Returns
new model instance
Return type
Model
dict(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False) → DictStrAny¶
Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
by_alias (bool) – 
skip_defaults (Optional[bool]) – 
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
Return type
DictStrAny
classmethod from_orm(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
get_file_content_by_path(path: str) → str[source]¶
Parameters
path (str) – 
Return type
str
Parameters
path (str) – 
Return type
str
get_file_paths() → List[Dict][source]¶
Return type
List[Dict]
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
by_alias (bool) – 
skip_defaults (Optional[bool]) – 
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
encoder (Optional[Callable[[Any], Any]]) – 
models_as_dict (bool) – 
dumps_kwargs (Any) – 
Return type
unicode
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
path (Union[str, Path]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod parse_obj(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
b (Union[str, bytes]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
Return type
DictStrAny
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
dumps_kwargs (Any) – 
Return type
unicode
dumps_kwargs (Any) – 
Return type
unicode
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
Parameters
localns (Any) – 
Return type
None
classmethod validate(value: Any) → Model¶
Parameters
value (Any) – 
Return type
Model
property headers: Dict[str, str]¶
Examples using GithubFileLoader¶
GitHub
langchain_community.document_loaders.html_bs.BSHTMLLoader¶
class langchain_community.document_loaders.html_bs.BSHTMLLoader(file_path: Union[str, Path], open_encoding: Optional[str] = None, bs_kwargs: Optional[dict] = None, get_text_separator: str = '')[source]¶
Load HTML files and parse them with beautiful soup.
initialize with path, and optionally, file encoding to use, and any kwargs
to pass to the BeautifulSoup object.
Parameters
file_path (Union[str, Path]) – The path to the file to load.
open_encoding (Optional[str]) – The encoding to use when opening the file.
bs_kwargs (Optional[dict]) – Any kwargs to pass to the BeautifulSoup object.
get_text_separator (str) – The separator to use when calling get_text on the soup.
Methods
__init__(file_path[, open_encoding, ...])
initialize with path, and optionally, file encoding to use, and any kwargs to pass to the BeautifulSoup object.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load HTML document into document objects.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path], open_encoding: Optional[str] = None, bs_kwargs: Optional[dict] = None, get_text_separator: str = '') → None[source]¶
initialize with path, and optionally, file encoding to use, and any kwargs
to pass to the BeautifulSoup object.
Parameters
file_path (Union[str, Path]) – The path to the file to load.
open_encoding (Optional[str]) – The encoding to use when opening the file.
bs_kwargs (Optional[dict]) – Any kwargs to pass to the BeautifulSoup object.
bs_kwargs (Optional[dict]) – Any kwargs to pass to the BeautifulSoup object.
get_text_separator (str) – The separator to use when calling get_text on the soup.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load HTML document into document objects.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using BSHTMLLoader¶
How to handle long text when doing extraction
How to load HTML
glob="**/*.pdf",
    parser=PyPDFParser()
)
A generic document loader.
Parameters
blob_loader (BlobLoader) – A blob loader which knows how to yield blobs
blob_parser (BaseBlobParser) – A blob parser which knows how to parse blobs into documents
Methods
__init__(blob_loader, blob_parser)
A generic document loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
from_filesystem(path, *[, glob, exclude, ...])
Create a generic document loader using a filesystem blob loader.
get_parser(**kwargs)
Override this method to associate a default parser with the class.
lazy_load()
Load documents lazily.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load all documents and split them into sentences.
__init__(blob_loader: BlobLoader, blob_parser: BaseBlobParser) → None[source]¶
A generic document loader.
Parameters
blob_loader (BlobLoader) – A blob loader which knows how to yield blobs
blob_parser (BaseBlobParser) – A blob parser which knows how to parse blobs into documents
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
classmethod from_filesystem(path: Union[str, Path], *, glob: str = '**/[!.]*', exclude: Sequence[str] = (), suffixes: Optional[Sequence[str]] = None, show_progress: bool = False, parser: Union[Literal['default'], BaseBlobParser] = 'default', parser_kwargs: Optional[dict] = None) → GenericLoader[source]¶
Create a generic document loader using a filesystem blob loader.
Parameters
path (Union[str, Path]) – The path to the directory to load documents from OR the path to a
single file to load. If this is a file, glob, exclude, suffixes
will be ignored.
glob (str) – The glob pattern to use to find documents.
suffixes (Optional[Sequence[str]]) – The suffixes to use to filter documents. If None, all files
matching the glob will be loaded.
exclude (Sequence[str]) – A list of patterns to exclude from the loader.
show_progress (bool) – Whether to show a progress bar or not (requires tqdm).
Proxies to the file system loader.
parser (Union[Literal['default'], ~langchain_core.document_loaders.base.BaseBlobParser]) – A blob parser which knows how to parse blobs into documents,
will instantiate a default parser if not provided.
The default can be overridden by either passing a parser or
setting the class attribute blob_parser (the latter
should be used with inheritance).
parser_kwargs (Optional[dict]) – Keyword arguments to pass to the parser.
Returns
A generic document loader.
Return type
GenericLoader
static get_parser(**kwargs: Any) → BaseBlobParser[source]¶
Override this method to associate a default parser with the class.
Parameters
kwargs (Any) – 
Return type
BaseBlobParser
lazy_load() → Iterator[Document][source]¶
Load documents lazily. Use this when working at a large scale.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document][source]¶
Load all documents and split them into sentences.
Parameters
langchain_community.document_loaders.telegram.TelegramChatApiLoader¶
class langchain_community.document_loaders.telegram.TelegramChatApiLoader(chat_entity: Optional[EntityLike] = None, api_id: Optional[int] = None, api_hash: Optional[str] = None, username: Optional[str] = None, file_path: str = 'telegram_data.json')[source]¶
Load Telegram chat json directory dump.
Initialize with API parameters.
Parameters
chat_entity (Optional[EntityLike]) – The chat entity to fetch data from.
api_id (Optional[int]) – The API ID.
api_hash (Optional[str]) – The API hash.
username (Optional[str]) – The username.
file_path (str) – The file path to save the data to. Defaults to
“telegram_data.json”.
Methods
__init__([chat_entity, api_id, api_hash, ...])
Initialize with API parameters.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
fetch_data_from_telegram()
Fetch data from Telegram API and save it as a JSON file.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(chat_entity: Optional[EntityLike] = None, api_id: Optional[int] = None, api_hash: Optional[str] = None, username: Optional[str] = None, file_path: str = 'telegram_data.json')[source]¶
Initialize with API parameters.
Parameters
chat_entity (Optional[EntityLike]) – The chat entity to fetch data from.
api_id (Optional[int]) – The API ID.
api_hash (Optional[str]) – The API hash.
username (Optional[str]) – The username.
username (Optional[str]) – The username.
file_path (str) – The file path to save the data to. Defaults to
“telegram_data.json”.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
async fetch_data_from_telegram() → None[source]¶
Fetch data from Telegram API and save it as a JSON file.
Return type
None
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using TelegramChatApiLoader¶
Telegram
langchain_community.document_loaders.parsers.language.csharp.CSharpSegmenter¶
class langchain_community.document_loaders.parsers.language.csharp.CSharpSegmenter(code: str)[source]¶
Code segmenter for C#.
Methods
__init__(code)
extract_functions_classes()
get_chunk_query()
get_language()
get_parser()
is_valid()
make_line_comment(text)
simplify_code()
Parameters
code (str) – 
__init__(code: str)¶
Parameters
code (str) – 
extract_functions_classes() → List[str]¶
Return type
List[str]
get_chunk_query() → str[source]¶
Return type
str
get_language() → Language[source]¶
Return type
Language
get_parser() → Parser¶
Return type
Parser
is_valid() → bool¶
Return type
bool
make_line_comment(text: str) → str[source]¶
Parameters
text (str) – 
Return type
str
simplify_code() → str¶
Return type
str