Tony20100/langchain_full huggingface dataset:
Parameters
obj (Any) – 
Return type
Model
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
by_alias (bool) – 
skip_defaults (Optional[bool]) – 
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
encoder (Optional[Callable[[Any], Any]]) – 
models_as_dict (bool) – 
dumps_kwargs (Any) – 
Return type
unicode
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
langchain_community.document_loaders.rocksetdb.RocksetLoader¶
class langchain_community.document_loaders.rocksetdb.RocksetLoader(client: ~typing.Any, query: ~typing.Any, content_keys: ~typing.List[str], metadata_keys: ~typing.Optional[~typing.List[str]] = None, content_columns_joiner: ~typing.Callable[[~typing.List[~typing.Tuple[str, ~typing.Any]]], str] = <function default_joiner>)[source]¶
Load from a Rockset database.
To use, you should have the rockset python package installed.
Example
# This code will load 3 records from the "langchain_demo"
# collection as Documents, with the `text` column used as
# the content
from langchain_community.document_loaders import RocksetLoader
from rockset import RocksetClient, Regions, models
loader = RocksetLoader(
    RocksetClient(Regions.usw2a1, "<api key>"),
    models.QueryRequestSql(
        query="select * from langchain_demo limit 3"
    ),
    ["text"]
)
)
Initialize with Rockset client.
Parameters
client (Any) – Rockset client object.
query (Any) – Rockset query object.
content_keys (List[str]) – The collection columns to be written into the page_content
of the Documents.
metadata_keys (Optional[List[str]]) – The collection columns to be written into the metadata of
the Documents. By default, this is all the keys in the document.
content_columns_joiner (Callable[[List[Tuple[str, Any]]], str]) – Method that joins content_keys and its values into a
string. It’s method that takes in a List[Tuple[str, Any]]],
representing a list of tuples of (column name, column value).
representing a list of tuples of (column name, column value).
By default, this is a method that joins each column value with a new
line. This method is only relevant if there are multiple content_keys.
Methods
__init__(client, query, content_keys[, ...])
Initialize with Rockset client.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(client: ~typing.Any, query: ~typing.Any, content_keys: ~typing.List[str], metadata_keys: ~typing.Optional[~typing.List[str]] = None, content_columns_joiner: ~typing.Callable[[~typing.List[~typing.Tuple[str, ~typing.Any]]], str] = <function default_joiner>)[source]¶
Initialize with Rockset client.
Parameters
client (Any) – Rockset client object.
query (Any) – Rockset query object.
content_keys (List[str]) – The collection columns to be written into the page_content
of the Documents.
metadata_keys (Optional[List[str]]) – The collection columns to be written into the metadata of
the Documents. By default, this is all the keys in the document.
content_columns_joiner (Callable[[List[Tuple[str, Any]]], str]) – Method that joins content_keys and its values into a
string. It’s method that takes in a List[Tuple[str, Any]]],
representing a list of tuples of (column name, column value).
By default, this is a method that joins each column value with a new
line. This method is only relevant if there are multiple content_keys.
line. This method is only relevant if there are multiple content_keys.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using RocksetLoader¶
Rockset
langchain_community.document_loaders.pdf.MathpixPDFLoader¶
class langchain_community.document_loaders.pdf.MathpixPDFLoader(file_path: str, processed_file_format: str = 'md', max_wait_time_seconds: int = 500, should_clean_pdf: bool = False, extra_request_data: Optional[Dict[str, Any]] = None, **kwargs: Any)[source]¶
Load PDF files using Mathpix service.
Initialize with a file path.
Parameters
file_path (str) – a file for loading.
processed_file_format (str) – a format of the processed file. Default is “md”.
max_wait_time_seconds (int) – a maximum time to wait for the response from
the server. Default is 500.
should_clean_pdf (bool) – a flag to clean the PDF file. Default is False.
extra_request_data (Optional[Dict[str, Any]]) – Additional request data.
**kwargs (Any) – additional keyword arguments.
Attributes
data
source
url
Methods
__init__(file_path[, processed_file_format, ...])
Initialize with a file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
clean_pdf(contents)
Clean the PDF file.
get_processed_pdf(pdf_id)
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
send_pdf()
wait_for_processing(pdf_id)
Wait for processing to complete.
__init__(file_path: str, processed_file_format: str = 'md', max_wait_time_seconds: int = 500, should_clean_pdf: bool = False, extra_request_data: Optional[Dict[str, Any]] = None, **kwargs: Any) → None[source]¶
Initialize with a file path.
Initialize with a file path.
Parameters
file_path (str) – a file for loading.
processed_file_format (str) – a format of the processed file. Default is “md”.
max_wait_time_seconds (int) – a maximum time to wait for the response from
the server. Default is 500.
should_clean_pdf (bool) – a flag to clean the PDF file. Default is False.
extra_request_data (Optional[Dict[str, Any]]) – Additional request data.
**kwargs (Any) – additional keyword arguments.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
clean_pdf(contents: str) → str[source]¶
Clean the PDF file.
Parameters
contents (str) – a PDF file contents.
Return type
str
Returns:
get_processed_pdf(pdf_id: str) → str[source]¶
Parameters
pdf_id (str) – 
Return type
str
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
langchain_community.document_loaders.url_playwright.UnstructuredHtmlEvaluator¶
class langchain_community.document_loaders.url_playwright.UnstructuredHtmlEvaluator(remove_selectors: Optional[List[str]] = None)[source]¶
Evaluate the page HTML content using the unstructured library.
Initialize UnstructuredHtmlEvaluator.
Methods
__init__([remove_selectors])
Initialize UnstructuredHtmlEvaluator.
evaluate(page, browser, response)
Synchronously process the HTML content of the page.
evaluate_async(page, browser, response)
Asynchronously process the HTML content of the page.
Parameters
remove_selectors (Optional[List[str]]) – 
__init__(remove_selectors: Optional[List[str]] = None)[source]¶
Initialize UnstructuredHtmlEvaluator.
Parameters
remove_selectors (Optional[List[str]]) – 
evaluate(page: Page, browser: Browser, response: Response) → str[source]¶
Synchronously process the HTML content of the page.
Parameters
page (Page) – 
browser (Browser) – 
response (Response) – 
Return type
str
async evaluate_async(page: AsyncPage, browser: AsyncBrowser, response: AsyncResponse) → str[source]¶
Asynchronously process the HTML content of the page.
Parameters
page (AsyncPage) – 
browser (AsyncBrowser) – 
response (AsyncResponse) – 
Return type
str
langchain_community.document_loaders.ifixit.IFixitLoader¶
class langchain_community.document_loaders.ifixit.IFixitLoader(web_path: str)[source]¶
Load iFixit repair guides, device wikis and answers.
iFixit is the largest, open repair community on the web. The site contains nearly
100k repair manuals, 200k Questions & Answers on 42k devices, and all the data is
licensed under CC-BY.
This loader will allow you to download the text of a repair guide, text of Q&A’s
and wikis from devices on iFixit using their open APIs and web scraping.
Initialize with a web path.
Methods
__init__(web_path)
Initialize with a web path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_device([url_override, include_guides])
Loads a device
load_guide([url_override])
Load a guide
load_questions_and_answers([url_override])
Load a list of questions and answers.
load_suggestions([query, doc_type])
Load suggestions.
Parameters
web_path (str) – 
__init__(web_path: str)[source]¶
Initialize with a web path.
Parameters
web_path (str) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
load_device(url_override: Optional[str] = None, include_guides: bool = True) → List[Document][source]¶
Loads a device
Parameters
url_override (Optional[str]) – A URL to override the default URL.
include_guides (bool) – Whether to include guides linked to from the device.
Defaults to True.
Return type
List[Document]
Returns:
load_guide(url_override: Optional[str] = None) → List[Document][source]¶
Load a guide
Parameters
url_override (Optional[str]) – A URL to override the default URL.
Return type
List[Document]
Returns: List[Document]
load_questions_and_answers(url_override: Optional[str] = None) → List[Document][source]¶
Load a list of questions and answers.
Parameters
url_override (Optional[str]) – A URL to override the default URL.
Return type
List[Document]
Returns: List[Document]
static load_suggestions(query: str = '', doc_type: str = 'all') → List[Document][source]¶
Load suggestions.
Parameters
query (str) – A query string
langchain_community.document_loaders.odt.UnstructuredODTLoader¶
class langchain_community.document_loaders.odt.UnstructuredODTLoader(file_path: Union[str, Path], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load OpenOffice ODT files using Unstructured.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain_community.document_loaders import UnstructuredODTLoader
loader = UnstructuredODTLoader(“example.odt”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-odt
Parameters
file_path (Union[str, Path]) – The path to the file to load.
mode (str) – The mode to use when loading the file. Can be one of “single”,
“multi”, or “all”. Default is “single”.
**unstructured_kwargs (Any) – Any kwargs to pass to the unstructured.
Methods
__init__(file_path[, mode])
param file_path
The path to the file to load.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Parameters
file_path (Union[str, Path]) – The path to the file to load.
mode (str) – The mode to use when loading the file. Can be one of “single”,
“multi”, or “all”. Default is “single”.
**unstructured_kwargs (Any) – Any kwargs to pass to the unstructured.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using UnstructuredODTLoader¶
Open Document Format (ODT)
Unstructured
langchain_community.document_loaders.spreedly.SpreedlyLoader¶
class langchain_community.document_loaders.spreedly.SpreedlyLoader(access_token: str, resource: str)[source]¶
Load from Spreedly API.
Initialize with an access token and a resource.
Parameters
access_token (str) – The access token.
resource (str) – The resource.
Methods
__init__(access_token, resource)
Initialize with an access token and a resource.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(access_token: str, resource: str) → None[source]¶
Initialize with an access token and a resource.
Parameters
access_token (str) – The access token.
resource (str) – The resource.
Return type
None
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
langchain_community.document_loaders.word_document.UnstructuredWordDocumentLoader¶
class langchain_community.document_loaders.word_document.UnstructuredWordDocumentLoader(file_path: Optional[Union[str, List[str], Path, List[Path]]], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load Microsoft Word file using Unstructured.
Works with both .docx and .doc files.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain_community.document_loaders import UnstructuredWordDocumentLoader
loader = UnstructuredWordDocumentLoader(“example.docx”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-docx
Initialize with file path.
Methods
__init__(file_path[, mode])
Initialize with file path.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Load file.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
Parameters
file_path (Optional[Union[str, List[str], Path, List[Path]]]) – 
mode (str) – 
unstructured_kwargs (Any) – 
__init__(file_path: Optional[Union[str, List[str], Path, List[Path]]], mode: str = 'single', **unstructured_kwargs: Any)¶
Initialize with file path.
Parameters
file_path (Optional[Union[str, List[str], Path, List[Path]]]) – 
mode (str) – 
unstructured_kwargs (Any) – 
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document]¶
Load file.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using UnstructuredWordDocumentLoader¶
Microsoft
Microsoft Word
Unstructured
langchain_community.document_loaders.pyspark_dataframe.PySparkDataFrameLoader¶
class langchain_community.document_loaders.pyspark_dataframe.PySparkDataFrameLoader(spark_session: Optional[SparkSession] = None, df: Optional[Any] = None, page_content_column: str = 'text', fraction_of_memory: float = 0.1)[source]¶
Load PySpark DataFrames.
Initialize with a Spark DataFrame object.
Parameters
spark_session (Optional[SparkSession]) – The SparkSession object.
df (Optional[Any]) – The Spark DataFrame object.
page_content_column (str) – The name of the column containing the page content.
Defaults to “text”.
fraction_of_memory (float) – The fraction of memory to use. Defaults to 0.1.
Methods
__init__([spark_session, df, ...])
Initialize with a Spark DataFrame object.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
get_num_rows()
Gets the number of "feasible" rows for the DataFrame
lazy_load()
A lazy loader for document content.
load()
Load from the dataframe.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(spark_session: Optional[SparkSession] = None, df: Optional[Any] = None, page_content_column: str = 'text', fraction_of_memory: float = 0.1)[source]¶
Initialize with a Spark DataFrame object.
Parameters
spark_session (Optional[SparkSession]) – The SparkSession object.
df (Optional[Any]) – The Spark DataFrame object.
page_content_column (str) – The name of the column containing the page content.
Defaults to “text”.
fraction_of_memory (float) – The fraction of memory to use. Defaults to 0.1.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
get_num_rows() → Tuple[int, int][source]¶
Gets the number of “feasible” rows for the DataFrame
Return type
Tuple[int, int]
lazy_load() → Iterator[Document][source]¶
A lazy loader for document content.
Return type
Iterator[Document]
load() → List[Document][source]¶
Load from the dataframe.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Return type
List[Document]
Examples using PySparkDataFrameLoader¶
PySpark
langchain_community.document_loaders.dataframe.BaseDataFrameLoader¶
class langchain_community.document_loaders.dataframe.BaseDataFrameLoader(data_frame: Any, *, page_content_column: str = 'text')[source]¶
Initialize with dataframe object.
Parameters
data_frame (Any) – DataFrame object.
page_content_column (str) – Name of the column containing the page content.
Defaults to “text”.
Methods
__init__(data_frame, *[, page_content_column])
Initialize with dataframe object.
alazy_load()
A lazy loader for Documents.
aload()
Load data into Document objects.
lazy_load()
Lazy load records from dataframe.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(data_frame: Any, *, page_content_column: str = 'text')[source]¶
Initialize with dataframe object.
Parameters
data_frame (Any) – DataFrame object.
page_content_column (str) – Name of the column containing the page content.
Defaults to “text”.
async alazy_load() → AsyncIterator[Document]¶
A lazy loader for Documents.
Return type
AsyncIterator[Document]
async aload() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
lazy_load() → Iterator[Document][source]¶
Lazy load records from dataframe.
Return type
Iterator[Document]
load() → List[Document]¶
Load data into Document objects.
Return type
List[Document]
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
Parameters