Tony20100/langchain_full huggingface dataset:
Parameters
url (str) – The URL to be processed.
api_key (Optional[str]) – The Spider API key. If not specified, will be read from env
SPIDER_API_KEY. (var) – 
mode (Literal['scrape', 'crawl']) – The mode to run the loader in. Default is “scrape”.
Options include “scrape” (single page) and “crawl” (with deeper
crawling following subpages).
params (Optional[dict]) – Additional parameters for the Spider API.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Load documents based on the specified mode.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using SpiderLoader¶
Spider
langchain_community.document_loaders.notebook.concatenate_cells¶
langchain_community.document_loaders.notebook.concatenate_cells(cell: dict, include_outputs: bool, max_output_length: int, traceback: bool) → str[source]¶
Combine cells information in a readable format ready to be used.
Parameters
cell (dict) – A dictionary
include_outputs (bool) – Whether to include the outputs of the cell.
max_output_length (int) – Maximum length of the output to be displayed.
traceback (bool) – Whether to return a traceback of the error.
Returns
A string with the cell information.
Return type
str
langchain_community.document_loaders.airbyte_json.AirbyteJSONLoader¶
class langchain_community.document_loaders.airbyte_json.AirbyteJSONLoader(file_path: Union[str, Path])[source]¶
Load local Airbyte json files.
Initialize with a file path. This should start with ‘/tmp/airbyte_local/’.
Attributes
file_path
Path to the directory containing the json files.
Methods
__init__(file_path)
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (Union[str, Path]) – 
__init__(file_path: Union[str, Path])[source]¶
Initialize with a file path. This should start with ‘/tmp/airbyte_local/’.
Parameters
file_path (Union[str, Path]) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
langchain_community.document_loaders.telegram.TelegramChatFileLoader¶
class langchain_community.document_loaders.telegram.TelegramChatFileLoader(path: Union[str, Path])[source]¶
Load from Telegram chat dump.
Initialize with a path.
Methods
__init__(path)
Initialize with a path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
path (Union[str, Path]) – 
__init__(path: Union[str, Path])[source]¶
Initialize with a path.
Parameters
path (Union[str, Path]) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using TelegramChatFileLoader¶
Telegram
langchain_community.document_loaders.weather.WeatherDataLoader¶
class langchain_community.document_loaders.weather.WeatherDataLoader(client: OpenWeatherMapAPIWrapper, places: Sequence[str])[source]¶
Load weather data with Open Weather Map API.
Reads the forecast & current weather of any location using OpenWeatherMap’s free
API. Checkout ‘https://openweathermap.org/appid’ for more on how to generate a free
OpenWeatherMap API.
Initialize with parameters.
Methods
__init__(client, places)
Initialize with parameters.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
from_params(places, *[, openweathermap_api_key])
lazy_load()
Lazily load weather data for the given locations.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
client (OpenWeatherMapAPIWrapper) – 
places (Sequence[str]) – 
__init__(client: OpenWeatherMapAPIWrapper, places: Sequence[str]) → None[source]¶
Initialize with parameters.
Parameters
client (OpenWeatherMapAPIWrapper) – 
places (Sequence[str]) – 
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
classmethod from_params(places: Sequence[str], *, openweathermap_api_key: Optional[str] = None) → WeatherDataLoader[source]¶
Parameters
places (Sequence[str]) – 
openweathermap_api_key (Optional[str]) – 
Return type
WeatherDataLoader
lazy_load() → Iterator[Document][source]¶
Return type
WeatherDataLoader
lazy_load() → Iterator[Document][source]¶
Lazily load weather data for the given locations.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using WeatherDataLoader¶
Weather
langchain_community.document_loaders.parsers.vsdx.VsdxParser¶
class langchain_community.document_loaders.parsers.vsdx.VsdxParser[source]¶
Parser for vsdx files.
Methods
__init__()
get_pages_content(zfile, source)
Get the content of the pages of a vsdx file.
get_relationships(page, zfile, filelist, ...)
Get the relationships of a page and the relationships of its relationships, etc.
lazy_parse(blob)
Retrieve the contents of pages from a .vsdx file and insert them into documents, one document per page.
parse(blob)
Parse a vsdx file.
__init__()¶
get_pages_content(zfile: ZipFile, source: str) → List[Tuple[int, str, str]][source]¶
Get the content of the pages of a vsdx file.
zfile¶
The vsdx file under zip format.
Type
zipfile.ZipFile
source¶
The path of the vsdx file.
Type
str
Returns
A list of tuples containing the page number,
the name of the page and the content of the page
for each page of the vsdx file.
Return type
list[tuple[int, str, str]]
Parameters
zfile (ZipFile) – 
source (str) – 
get_relationships(page: str, zfile: ZipFile, filelist: List[str], pagexml_rels: List[dict]) → Set[str][source]¶
Get the relationships of a page and the relationships of its relationships,
etc… recursively.
Pages are based on other pages (ex: background page),
so we need to get all the relationships to get all the content of a single page.
Parameters
page (str) – 
zfile (ZipFile) – 
filelist (List[str]) –
zfile (ZipFile) – 
filelist (List[str]) – 
pagexml_rels (List[dict]) – 
Return type
Set[str]
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Retrieve the contents of pages from a .vsdx file
and insert them into documents, one document per page.
Parameters
blob (Blob) – 
Return type
Iterator[Document]
parse(blob: Blob) → Iterator[Document][source]¶
Parse a vsdx file.
Parameters
blob (Blob) – 
Return type
Iterator[Document]
langchain_community.document_loaders.apify_dataset.ApifyDatasetLoader¶
class langchain_community.document_loaders.apify_dataset.ApifyDatasetLoader[source]¶
Bases: BaseLoader, BaseModel
Load datasets from Apify web scraping, crawling, and data extraction platform.
For details, see https://docs.apify.com/platform/integrations/langchain
Example
from langchain_community.document_loaders import ApifyDatasetLoader
from langchain_core.documents import Document
loader = ApifyDatasetLoader(
    dataset_id="YOUR-DATASET-ID",
    dataset_mapping_function=lambda dataset_item: Document(
        page_content=dataset_item["text"], metadata={"source": dataset_item["url"]}
    ),
)
documents = loader.load()
Initialize the loader with an Apify dataset ID and a mapping function.
Parameters
dataset_id (str) – The ID of the dataset on the Apify platform.
dataset_mapping_function (Callable) – A function that takes a single
dictionary (an Apify dataset item) and converts it to an instance
of the Document class.
param apify_client: Any = None¶
An instance of the ApifyClient class from the apify-client Python package.
param dataset_id: str [Required]¶
The ID of the dataset on the Apify platform.
param dataset_mapping_function: Callable[[Dict], Document] [Required]¶
A custom function that takes a single dictionary (an Apify dataset item)
and converts it to an instance of the Document class.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
dumps_kwargs (Any) – 
Return type
unicode
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load documents.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
path (Union[str, Path]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod parse_obj(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
b (Union[str, bytes]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
Parameters
Parameters
by_alias (bool) – 
ref_template (unicode) – 
Return type
DictStrAny
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
dumps_kwargs (Any) – 
Return type
unicode
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
Parameters
localns (Any) – 
Return type
None
classmethod validate(value: Any) → Model¶
Parameters
value (Any) – 
Return type
Model
Examples using ApifyDatasetLoader¶
Apify
Apify Dataset
langchain_community.document_loaders.oracleai.OracleDocReader¶
class langchain_community.document_loaders.oracleai.OracleDocReader[source]¶
Read a file
Methods
__init__()
generate_object_id([input_string])
read_file(conn, file_path, params)
Read a file using OracleReader :param conn: Oracle Connection, :param file_path: Oracle Directory, :param params: ONNX file name.
__init__()¶
static generate_object_id(input_string: Optional[str] = None) → str[source]¶
Parameters
input_string (Optional[str]) – 
Return type
str
static read_file(conn: Connection, file_path: str, params: dict) → Union[Document, None][source]¶
Read a file using OracleReader
:param conn: Oracle Connection,
:param file_path: Oracle Directory,
:param params: ONNX file name.
Returns
Plain text and metadata as Langchain Document.
Parameters
conn (Connection) – 
file_path (str) – 
params (dict) – 
Return type
Union[Document, None]
langchain_community.document_loaders.mediawikidump.MWDumpLoader¶
class langchain_community.document_loaders.mediawikidump.MWDumpLoader(file_path: Union[str, Path], encoding: Optional[str] = 'utf8', namespaces: Optional[Sequence[int]] = None, skip_redirects: Optional[bool] = False, stop_on_error: Optional[bool] = True)[source]¶
Load MediaWiki dump from an XML file.
Example
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import MWDumpLoader
loader = MWDumpLoader(
    file_path="myWiki.xml",
    encoding="utf8"
)
docs = loader.load()
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000, chunk_overlap=0
)
texts = text_splitter.split_documents(docs)
Parameters
file_path (str) – XML local file path
encoding (str, optional) – Charset encoding, defaults to “utf8”
namespaces (List[int],optional) – The namespace of pages you want to parse.
See https://www.mediawiki.org/wiki/Help:Namespaces#Localisation
for a list of all common namespaces
skip_redirects (bool, optional) – TR=rue to skip pages that redirect to other pages,
False to keep them. False by default
stop_on_error (bool, optional) – False to skip over pages that cause parsing errors,
True to stop. True by default
Methods
__init__(file_path[, encoding, namespaces, ...])
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load from a file path.
load()
Load data into Document objects.
Lazy load from a file path.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path], encoding: Optional[str] = 'utf8', namespaces: Optional[Sequence[int]] = None, skip_redirects: Optional[bool] = False, stop_on_error: Optional[bool] = True)[source]¶
Parameters
file_path (Union[str, Path]) – 
encoding (Optional[str]) – 
namespaces (Optional[Sequence[int]]) – 
skip_redirects (Optional[bool]) – 
stop_on_error (Optional[bool]) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load from a file path.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using MWDumpLoader¶
MediaWiki Dump
MediaWikiDump
langchain_community.document_loaders.org_mode.UnstructuredOrgModeLoader¶
class langchain_community.document_loaders.org_mode.UnstructuredOrgModeLoader(file_path: Union[str, Path], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load Org-Mode files using Unstructured.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain_community.document_loaders import UnstructuredOrgModeLoader
loader = UnstructuredOrgModeLoader(“example.org”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-org
Parameters
file_path (Union[str, Path]) – The path to the file to load.
mode (str) – The mode to load the file from. Default is “single”.
**unstructured_kwargs (Any) – Any additional keyword arguments to pass
to the unstructured.
Methods
__init__(file_path[, mode])
param file_path
The path to the file to load.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Parameters
Parameters
file_path (Union[str, Path]) – The path to the file to load.
mode (str) – The mode to load the file from. Default is “single”.
**unstructured_kwargs (Any) – Any additional keyword arguments to pass
to the unstructured.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using UnstructuredOrgModeLoader¶
Org-mode
Unstructured
langchain_community.document_loaders.parsers.pdf.PyMuPDFParser¶
class langchain_community.document_loaders.parsers.pdf.PyMuPDFParser(text_kwargs: Optional[Mapping[str, Any]] = None, extract_images: bool = False)[source]¶
Parse PDF using PyMuPDF.
Initialize the parser.
Parameters
text_kwargs (Optional[Mapping[str, Any]]) – Keyword arguments to pass to fitz.Page.get_text().
extract_images (bool) – 
Methods
__init__([text_kwargs, extract_images])
Initialize the parser.
lazy_parse(blob)
Lazily parse the blob.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__(text_kwargs: Optional[Mapping[str, Any]] = None, extract_images: bool = False) → None[source]¶
Initialize the parser.
Parameters
text_kwargs (Optional[Mapping[str, Any]]) – Keyword arguments to pass to fitz.Page.get_text().
extract_images (bool) – 
Return type
None
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazily parse the blob.
Parameters
blob (Blob) – 
Return type
Iterator[Document]
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob (Blob) – Blob instance
Returns
List of documents
Return type
List[Document]
langchain_community.document_loaders.acreom.AcreomLoader¶
class langchain_community.document_loaders.acreom.AcreomLoader(path: Union[str, Path], encoding: str = 'UTF-8', collect_metadata: bool = True)[source]¶
Load acreom vault from a directory.
Initialize the loader.
Attributes
FRONT_MATTER_REGEX
Regex to match front matter metadata in markdown files.
file_path
Path to the directory containing the markdown files.
encoding
Encoding to use when reading the files.
collect_metadata
Whether to collect metadata from the front matter.
Methods
__init__(path[, encoding, collect_metadata])
Initialize the loader.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
path (Union[str, Path]) – 
encoding (str) – 
collect_metadata (bool) – 
__init__(path: Union[str, Path], encoding: str = 'UTF-8', collect_metadata: bool = True)[source]¶
Initialize the loader.
Parameters
path (Union[str, Path]) – 
encoding (str) – 
collect_metadata (bool) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using AcreomLoader¶
Acreom
acreom
langchain_community.document_loaders.tensorflow_datasets.TensorflowDatasetLoader¶
class langchain_community.document_loaders.tensorflow_datasets.TensorflowDatasetLoader(dataset_name: str, split_name: str, load_max_docs: Optional[int] = 100, sample_to_document_function: Optional[Callable[[Dict], Document]] = None)[source]¶
Load from TensorFlow Dataset.
dataset_name¶
the name of the dataset to load
split_name¶
the name of the split to load.
load_max_docs¶
a limit to the number of loaded documents. Defaults to 100.
sample_to_document_function¶
a function that converts a dataset sample
into a Document
Example
from langchain_community.document_loaders import TensorflowDatasetLoader
def mlqaen_example_to_document(example: dict) -> Document:
    return Document(
        page_content=decode_to_str(example["context"]),
        metadata={
            "id": decode_to_str(example["id"]),
            "title": decode_to_str(example["title"]),
            "question": decode_to_str(example["question"]),
            "answer": decode_to_str(example["answers"]["text"][0]),
        },
    )
tsds_client = TensorflowDatasetLoader(
        dataset_name="mlqa/en",
        split_name="test",
        load_max_docs=100,
        sample_to_document_function=mlqaen_example_to_document,
    )
Initialize the TensorflowDatasetLoader.
Parameters
dataset_name (str) – the name of the dataset to load
split_name (str) – the name of the split to load.
load_max_docs (Optional[int]) – a limit to the number of loaded documents. Defaults to 100.
sample_to_document_function (Optional[Callable[[Dict], Document]]) – a function that converts a dataset sample
into a Document.
Attributes
load_max_docs