Tony20100/langchain_full huggingface dataset:
langchain_community.document_loaders.polars_dataframe.PolarsDataFrameLoader¶
class langchain_community.document_loaders.polars_dataframe.PolarsDataFrameLoader(data_frame: Any, *, page_content_column: str = 'text')[source]¶
Load Polars DataFrame.
Initialize with dataframe object.
Parameters
data_frame (Any) – Polars DataFrame object.
page_content_column (str) – Name of the column containing the page content.
Defaults to “text”.
Methods
__init__(data_frame, *[, page_content_column])
Initialize with dataframe object.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load records from dataframe.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(data_frame: Any, *, page_content_column: str = 'text')[source]¶
Initialize with dataframe object.
Parameters
data_frame (Any) – Polars DataFrame object.
page_content_column (str) – Name of the column containing the page content.
Defaults to “text”.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load records from dataframe.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using PolarsDataFrameLoader¶
Polars DataFrame
langchain_community.document_loaders.parsers.language.typescript.TypeScriptSegmenter¶
class langchain_community.document_loaders.parsers.language.typescript.TypeScriptSegmenter(code: str)[source]¶
Code segmenter for TypeScript.
Methods
__init__(code)
extract_functions_classes()
get_chunk_query()
get_language()
get_parser()
is_valid()
make_line_comment(text)
simplify_code()
Parameters
code (str) – 
__init__(code: str)¶
Parameters
code (str) – 
extract_functions_classes() → List[str]¶
Return type
List[str]
get_chunk_query() → str[source]¶
Return type
str
get_language() → Language[source]¶
Return type
Language
get_parser() → Parser¶
Return type
Parser
is_valid() → bool¶
Return type
bool
make_line_comment(text: str) → str[source]¶
Parameters
text (str) – 
Return type
str
simplify_code() → str¶
Return type
str
langchain_community.document_loaders.nuclia.NucliaLoader¶
class langchain_community.document_loaders.nuclia.NucliaLoader(path: str, nuclia_tool: NucliaUnderstandingAPI)[source]¶
Load from any file type using Nuclia Understanding API.
Methods
__init__(path, nuclia_tool)
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
path (str) – 
nuclia_tool (NucliaUnderstandingAPI) – 
__init__(path: str, nuclia_tool: NucliaUnderstandingAPI)[source]¶
Parameters
path (str) – 
nuclia_tool (NucliaUnderstandingAPI) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
langchain_community.document_loaders.whatsapp_chat.concatenate_rows¶
langchain_community.document_loaders.whatsapp_chat.concatenate_rows(date: str, sender: str, text: str) → str[source]¶
Combine message information in a readable format ready to be used.
Parameters
date (str) – 
sender (str) – 
text (str) – 
Return type
str
langchain_community.document_loaders.parsers.language.language_parser.LanguageParser¶
class langchain_community.document_loaders.parsers.language.language_parser.LanguageParser(language: Optional[Literal['cpp', 'go', 'java', 'kotlin', 'js', 'ts', 'php', 'proto', 'python', 'rst', 'ruby', 'rust', 'scala', 'swift', 'markdown', 'latex', 'html', 'sol', 'csharp', 'cobol', 'c', 'lua', 'perl']] = None, parser_threshold: int = 0)[source]¶
Parse using the respective programming language syntax.
Each top-level function and class in the code is loaded into separate documents.
Furthermore, an extra document is generated, containing the remaining top-level code
that excludes the already segmented functions and classes.
This approach can potentially improve the accuracy of QA models over source code.
The supported languages for code parsing are:
C: “c” (*)
C++: “cpp” (*)
C#: “csharp” (*)
COBOL: “cobol”
Go: “go” (*)
Java: “java” (*)
JavaScript: “js” (requires package esprima)
Kotlin: “kotlin” (*)
Lua: “lua” (*)
Perl: “perl” (*)
Python: “python”
Ruby: “ruby” (*)
Rust: “rust” (*)
Scala: “scala” (*)
TypeScript: “ts” (*)
Items marked with (*) require the packages tree_sitter and
tree_sitter_languages. It is straightforward to add support for additional
languages using tree_sitter, although this currently requires modifying LangChain.
The language used for parsing can be configured, along with the minimum number of
lines required to activate the splitting based on syntax.
If a language is not explicitly specified, LanguageParser will infer one from
filename extensions, if present.
filename extensions, if present.
Examples
    from langchain_community.document_loaders.generic import GenericLoader
    from langchain_community.document_loaders.parsers import LanguageParser
    loader = GenericLoader.from_filesystem(
        "./code",
        glob="**/*",
        suffixes=[".py", ".js"],
        parser=LanguageParser()
    )
    docs = loader.load()
Example instantiations to manually select the language:
.. code-block:: python
    loader = GenericLoader.from_filesystem(
        "./code",
        glob="**/*",
        suffixes=[".py"],
        parser=LanguageParser(language="python")
    )
Example instantiations to set number of lines threshold:
.. code-block:: python
    loader = GenericLoader.from_filesystem(
        "./code",
        glob="**/*",
        suffixes=[".py"],
        parser=LanguageParser(parser_threshold=200)
    )
Language parser that split code using the respective language syntax.
Parameters
language (Optional[Language]) – If None (default), it will try to infer language from source.
parser_threshold (int) – Minimum lines needed to activate parsing (0 by default).
Methods
__init__([language, parser_threshold])
Language parser that split code using the respective language syntax.
lazy_parse(blob)
Lazy parsing interface.
parse(blob)
Eagerly parse the blob into a document or documents.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__(language: Optional[Literal['cpp', 'go', 'java', 'kotlin', 'js', 'ts', 'php', 'proto', 'python', 'rst', 'ruby', 'rust', 'scala', 'swift', 'markdown', 'latex', 'html', 'sol', 'csharp', 'cobol', 'c', 'lua', 'perl']] = None, parser_threshold: int = 0)[source]¶
Language parser that split code using the respective language syntax.
Parameters
language (Optional[Literal['cpp', 'go', 'java', 'kotlin', 'js', 'ts', 'php', 'proto', 'python', 'rst', 'ruby', 'rust', 'scala', 'swift', 'markdown', 'latex', 'html', 'sol', 'csharp', 'cobol', 'c', 'lua', 'perl']]) – If None (default), it will try to infer language from source.
parser_threshold (int) – Minimum lines needed to activate parsing (0 by default).
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazy parsing interface.
Subclasses are required to implement this method.
Parameters
blob (Blob) – Blob instance
Returns
Generator of documents
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
Examples using LanguageParser¶
Source Code
langchain_community.document_loaders.parsers.generic.MimeTypeBasedParser¶
class langchain_community.document_loaders.parsers.generic.MimeTypeBasedParser(handlers: Mapping[str, BaseBlobParser], *, fallback_parser: Optional[BaseBlobParser] = None)[source]¶
Parser that uses mime-types to parse a blob.
This parser is useful for simple pipelines where the mime-type is sufficient
to determine how to parse a blob.
To use, configure handlers based on mime-types and pass them to the initializer.
Example
from langchain_community.document_loaders.parsers.generic import MimeTypeBasedParser
parser = MimeTypeBasedParser(
handlers={“application/pdf”: …,
},
fallback_parser=…,
)
Define a parser that uses mime-types to determine how to parse a blob.
Parameters
handlers (Mapping[str, BaseBlobParser]) – A mapping from mime-types to functions that take a blob, parse it
and return a document.
fallback_parser (Optional[BaseBlobParser]) – A fallback_parser parser to use if the mime-type is not
found in the handlers. If provided, this parser will be
used to parse blobs with all mime-types not found in
the handlers.
If not provided, a ValueError will be raised if the
mime-type is not found in the handlers.
Methods
__init__(handlers, *[, fallback_parser])
Define a parser that uses mime-types to determine how to parse a blob.
lazy_parse(blob)
Load documents from a blob.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__(handlers: Mapping[str, BaseBlobParser], *, fallback_parser: Optional[BaseBlobParser] = None) → None[source]¶
Define a parser that uses mime-types to determine how to parse a blob.
Parameters
Define a parser that uses mime-types to determine how to parse a blob.
Parameters
handlers (Mapping[str, BaseBlobParser]) – A mapping from mime-types to functions that take a blob, parse it
and return a document.
fallback_parser (Optional[BaseBlobParser]) – A fallback_parser parser to use if the mime-type is not
found in the handlers. If provided, this parser will be
used to parse blobs with all mime-types not found in
the handlers.
If not provided, a ValueError will be raised if the
mime-type is not found in the handlers.
Return type
None
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Load documents from a blob.
Parameters
blob (Blob) – 
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
langchain_community.document_loaders.gcs_directory.GCSDirectoryLoader¶
class langchain_community.document_loaders.gcs_directory.GCSDirectoryLoader(project_name: str, bucket: str, prefix: str = '', loader_func: Optional[Callable[[str], BaseLoader]] = None, continue_on_failure: bool = False)[source]¶
[Deprecated] Load from GCS directory.
Notes
Deprecated since version 0.0.32.
Initialize with bucket and key name.
Parameters
project_name (str) – The ID of the project for the GCS bucket.
bucket (str) – The name of the GCS bucket.
prefix (str) – The prefix of the GCS bucket.
loader_func (Optional[Callable[[str], BaseLoader]]) – A loader function that instantiates a loader based on a
file_path argument. If nothing is provided, the  GCSFileLoader
would use its default loader.
continue_on_failure (bool) – To use try-except block for each file within the GCS
directory. If set to True, then failure to process a file will not
cause an error.
Methods
__init__(project_name, bucket[, prefix, ...])
Initialize with bucket and key name.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(project_name: str, bucket: str, prefix: str = '', loader_func: Optional[Callable[[str], BaseLoader]] = None, continue_on_failure: bool = False)[source]¶
Initialize with bucket and key name.
Parameters
project_name (str) – The ID of the project for the GCS bucket.
Parameters
project_name (str) – The ID of the project for the GCS bucket.
bucket (str) – The name of the GCS bucket.
prefix (str) – The prefix of the GCS bucket.
loader_func (Optional[Callable[[str], BaseLoader]]) – A loader function that instantiates a loader based on a
file_path argument. If nothing is provided, the  GCSFileLoader
would use its default loader.
continue_on_failure (bool) – To use try-except block for each file within the GCS
directory. If set to True, then failure to process a file will not
cause an error.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
langchain_community.document_loaders.parsers.pdf.PyPDFParser¶
class langchain_community.document_loaders.parsers.pdf.PyPDFParser(password: Optional[Union[str, bytes]] = None, extract_images: bool = False)[source]¶
Load PDF using pypdf
Methods
__init__([password, extract_images])
lazy_parse(blob)
Lazily parse the blob.
parse(blob)
Eagerly parse the blob into a document or documents.
Parameters
password (Optional[Union[str, bytes]]) – 
extract_images (bool) – 
__init__(password: Optional[Union[str, bytes]] = None, extract_images: bool = False)[source]¶
Parameters
password (Optional[Union[str, bytes]]) – 
extract_images (bool) – 
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazily parse the blob.
Parameters
blob (Blob) – 
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
langchain_community.document_loaders.concurrent.ConcurrentLoader¶
class langchain_community.document_loaders.concurrent.ConcurrentLoader(blob_loader: BlobLoader, blob_parser: BaseBlobParser, num_workers: int = 4)[source]¶
Load and pars Documents concurrently.
A generic document loader.
Parameters
blob_loader (BlobLoader) – A blob loader which knows how to yield blobs
blob_parser (BaseBlobParser) – A blob parser which knows how to parse blobs into documents
num_workers (int) – 
Methods
__init__(blob_loader, blob_parser[, num_workers])
A generic document loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
from_filesystem(path, *[, glob, exclude, ...])
Create a concurrent generic document loader using a filesystem blob loader.
get_parser(**kwargs)
Override this method to associate a default parser with the class.
lazy_load()
Load documents lazily with concurrent parsing.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load all documents and split them into sentences.
__init__(blob_loader: BlobLoader, blob_parser: BaseBlobParser, num_workers: int = 4) → None[source]¶
A generic document loader.
Parameters
blob_loader (BlobLoader) – A blob loader which knows how to yield blobs
blob_parser (BaseBlobParser) – A blob parser which knows how to parse blobs into documents
num_workers (int) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
Load data into Document objects.
Return type
List[Document]
classmethod from_filesystem(path: Union[str, Path], *, glob: str = '**/[!.]*', exclude: Sequence[str] = (), suffixes: Optional[Sequence[str]] = None, show_progress: bool = False, parser: Union[Literal['default'], BaseBlobParser] = 'default', num_workers: int = 4, parser_kwargs: Optional[dict] = None) → ConcurrentLoader[source]¶
Create a concurrent generic document loader using a filesystem blob loader.
Parameters
path (Union[str, Path]) – The path to the directory to load documents from.
glob (str) – The glob pattern to use to find documents.
suffixes (Optional[Sequence[str]]) – The suffixes to use to filter documents. If None, all files
matching the glob will be loaded.
exclude (Sequence[str]) – A list of patterns to exclude from the loader.
show_progress (bool) – Whether to show a progress bar or not (requires tqdm).
Proxies to the file system loader.
parser (Union[Literal['default'], ~langchain_core.document_loaders.base.BaseBlobParser]) – A blob parser which knows how to parse blobs into documents
num_workers (int) – Max number of concurrent workers to use.
parser_kwargs (Optional[dict]) – Keyword arguments to pass to the parser.
Return type
ConcurrentLoader
static get_parser(**kwargs: Any) → BaseBlobParser¶
Override this method to associate a default parser with the class.
Parameters
kwargs (Any) – 
Return type
BaseBlobParser
lazy_load() → Iterator[Document][source]¶
Load documents lazily with concurrent parsing.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
langchain_community.document_loaders.parsers.pdf.PDFMinerParser¶
class langchain_community.document_loaders.parsers.pdf.PDFMinerParser(extract_images: bool = False, *, concatenate_pages: bool = True)[source]¶
Parse PDF using PDFMiner.
Initialize a parser based on PDFMiner.
Parameters
extract_images (bool) – Whether to extract images from PDF.
concatenate_pages (bool) – If True, concatenate all PDF pages into one a single
document. Otherwise, return one document per page.
Methods
__init__([extract_images, concatenate_pages])
Initialize a parser based on PDFMiner.
lazy_parse(blob)
Lazily parse the blob.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__(extract_images: bool = False, *, concatenate_pages: bool = True)[source]¶
Initialize a parser based on PDFMiner.
Parameters
extract_images (bool) – Whether to extract images from PDF.
concatenate_pages (bool) – If True, concatenate all PDF pages into one a single
document. Otherwise, return one document per page.
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazily parse the blob.
Parameters
blob (Blob) – 
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
langchain_community.document_loaders.dataframe.DataFrameLoader¶
class langchain_community.document_loaders.dataframe.DataFrameLoader(data_frame: Any, page_content_column: str = 'text', engine: Literal['pandas', 'modin'] = 'pandas')[source]¶
Load Pandas DataFrame.
Initialize with dataframe object.
Parameters
data_frame (Any) – Pandas DataFrame object.
page_content_column (str) – Name of the column containing the page content.
Defaults to “text”.
engine (Literal['pandas', 'modin']) – 
Methods
__init__(data_frame[, page_content_column, ...])
Initialize with dataframe object.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load records from dataframe.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(data_frame: Any, page_content_column: str = 'text', engine: Literal['pandas', 'modin'] = 'pandas')[source]¶
Initialize with dataframe object.
Parameters
data_frame (Any) – Pandas DataFrame object.
page_content_column (str) – Name of the column containing the page content.
Defaults to “text”.
engine (Literal['pandas', 'modin']) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Lazy load records from dataframe.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using DataFrameLoader¶
Pandas DataFrame
langchain_community.document_loaders.mhtml.MHTMLLoader¶
class langchain_community.document_loaders.mhtml.MHTMLLoader(file_path: Union[str, Path], open_encoding: Optional[str] = None, bs_kwargs: Optional[dict] = None, get_text_separator: str = '')[source]¶
Parse MHTML files with BeautifulSoup.
initialize with path, and optionally, file encoding to use, and any kwargs
to pass to the BeautifulSoup object.
Parameters
file_path (Union[str, Path]) – Path to file to load.
open_encoding (Optional[str]) – The encoding to use when opening the file.
bs_kwargs (Optional[dict]) – Any kwargs to pass to the BeautifulSoup object.
get_text_separator (str) – The separator to use when getting the text
from the soup.
Methods
__init__(file_path[, open_encoding, ...])
initialize with path, and optionally, file encoding to use, and any kwargs to pass to the BeautifulSoup object.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load MHTML document into document objects.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path], open_encoding: Optional[str] = None, bs_kwargs: Optional[dict] = None, get_text_separator: str = '') → None[source]¶
initialize with path, and optionally, file encoding to use, and any kwargs
to pass to the BeautifulSoup object.
Parameters
file_path (Union[str, Path]) – Path to file to load.
open_encoding (Optional[str]) – The encoding to use when opening the file.
bs_kwargs (Optional[dict]) – Any kwargs to pass to the BeautifulSoup object.