Tony20100/langchain_full huggingface dataset:
Asynchronously transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Translate text documents using Google Translate.
Parameters
source_language_code – ISO 639 language code of the input document.
target_language_code – ISO 639 language code of the output document.
For supported languages, refer to:
https://cloud.google.com/translate/docs/languages
mime_type – (Optional) Media Type of input text.
Options: text/plain, text/html
documents (Sequence[Document]) – 
kwargs (Any) – 
Return type
Sequence[Document]
"required": ["movie_title", "critic", "tone"]
}
# Must be an OpenAI model that supports functions
llm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613")
document_transformer = create_metadata_tagger(schema, llm)
original_documents = [
    Document(page_content="Review of The Bee Movie
By Roger Ebert
This is the greatest movie ever made. 4 out of 5 stars.”),Document(page_content=”Review of The Godfather
By Anonymous
This movie was super boring. 1 out of 5 stars.”, metadata={“reliable”: False}),]
enhanced_documents = document_transformer.transform_documents(original_documents)
Parameters
metadata_schema (Union[Dict[str, Any], Type[BaseModel]]) – 
llm (BaseLanguageModel) – 
prompt (Optional[ChatPromptTemplate]) – 
tagging_chain_kwargs (Optional[Dict]) – 
Return type
OpenAIMetadataTagger
Examples using create_metadata_tagger¶
OpenAI metadata tagger
langchain_community.document_transformers.doctran_text_translate.DoctranTextTranslator¶
class langchain_community.document_transformers.doctran_text_translate.DoctranTextTranslator(openai_api_key: Optional[str] = None, language: str = 'english', openai_api_model: Optional[str] = None)[source]¶
Translate text documents using doctran.
Parameters
openai_api_key (Optional[str]) – OpenAI API key. Can also be specified via environment variable
OPENAI_API_KEY. – 
language (str) – The language to translate to.
openai_api_model (Optional[str]) – 
Example
from langchain_community.document_transformers import DoctranTextTranslator
# Pass in openai_api_key or set env var OPENAI_API_KEY
qa_translator = DoctranTextTranslator(language=”spanish”)
translated_document = await qa_translator.atransform_documents(documents)
Methods
__init__([openai_api_key, language, ...])
atransform_documents(documents, **kwargs)
Asynchronously transform a list of documents.
transform_documents(documents, **kwargs)
Translates text documents using doctran.
__init__(openai_api_key: Optional[str] = None, language: str = 'english', openai_api_model: Optional[str] = None) → None[source]¶
Parameters
openai_api_key (Optional[str]) – 
language (str) – 
openai_api_model (Optional[str]) – 
Return type
None
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Asynchronously transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Translates text documents using doctran.
Parameters
documents (Sequence[Document]) – 
kwargs (Any) – 
Return type
Sequence[Document]
Examples using DoctranTextTranslator¶
Doctran: language translation
langchain_community.document_transformers.doctran_text_qa.DoctranQATransformer¶
class langchain_community.document_transformers.doctran_text_qa.DoctranQATransformer(openai_api_key: Optional[str] = None, openai_api_model: Optional[str] = None)[source]¶
Extract QA from text documents using doctran.
Parameters
openai_api_key (Optional[str]) – OpenAI API key. Can also be specified via environment variable
OPENAI_API_KEY.
openai_api_model (Optional[str]) – 
Example
from langchain_community.document_transformers import DoctranQATransformer
# Pass in openai_api_key or set env var OPENAI_API_KEY
qa_transformer = DoctranQATransformer()
transformed_document = await qa_transformer.atransform_documents(documents)
Methods
__init__([openai_api_key, openai_api_model])
atransform_documents(documents, **kwargs)
Asynchronously transform a list of documents.
transform_documents(documents, **kwargs)
Extracts QA from text documents using doctran.
__init__(openai_api_key: Optional[str] = None, openai_api_model: Optional[str] = None) → None[source]¶
Parameters
openai_api_key (Optional[str]) – 
openai_api_model (Optional[str]) – 
Return type
None
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Asynchronously transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
Returns
A list of transformed Documents.
Return type
Sequence[Document]
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Extracts QA from text documents using doctran.
Parameters
documents (Sequence[Document]) – 
kwargs (Any) – 
Return type
Sequence[Document]
Examples using DoctranQATransformer¶
Doctran: interrogate documents
langchain_community.document_transformers.embeddings_redundant_filter.EmbeddingsRedundantFilter¶
class langchain_community.document_transformers.embeddings_redundant_filter.EmbeddingsRedundantFilter[source]¶
Bases: BaseDocumentTransformer, BaseModel
Filter that drops redundant documents by comparing their embeddings.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param embeddings: Embeddings [Required]¶
Embeddings to use for embedding document contents.
param similarity_fn: Callable = <function cosine_similarity>¶
Similarity function for comparing documents. Function expected to take as input
two matrices (List[List[float]]) and return a matrix of scores where higher values
indicate greater similarity.
param similarity_threshold: float = 0.95¶
Threshold for determining when two documents are similar enough
to be considered redundant.
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document]¶
Asynchronously transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
values (Any) – 
Return type
Model
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to include in new model
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to exclude from new model, as with values this takes precedence over include
update (Optional[DictStrAny]) – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
deep (bool) – set to True to make a deep copy of the model
self (Model) – 
Returns
new model instance
Return type
Model
dict(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False) → DictStrAny¶
Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
by_alias (bool) – 
skip_defaults (Optional[bool]) – 
exclude_unset (bool) – 
exclude_defaults (bool) –
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
Return type
DictStrAny
classmethod from_orm(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
by_alias (bool) – 
skip_defaults (Optional[bool]) – 
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
encoder (Optional[Callable[[Any], Any]]) – 
models_as_dict (bool) – 
dumps_kwargs (Any) – 
Return type
unicode
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
path (Union[str, Path]) – 
content_type (unicode) – 
encoding (unicode) –
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod parse_obj(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
b (Union[str, bytes]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
Return type
DictStrAny
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
dumps_kwargs (Any) – 
Return type
unicode
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Filter down documents.
Parameters
documents (Sequence[Document]) – 
kwargs (Any) – 
Return type
Sequence[Document]
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
Parameters
localns (Any) – 
Return type
None
classmethod validate(value: Any) → Model¶
Parameters
value (Any) – 
Return type
Model
langchain_community.document_transformers.beautiful_soup_transformer.BeautifulSoupTransformer¶
class langchain_community.document_transformers.beautiful_soup_transformer.BeautifulSoupTransformer[source]¶
Transform HTML content by extracting specific tags and removing unwanted ones.
Example
from langchain_community.document_transformers import BeautifulSoupTransformer
bs4_transformer = BeautifulSoupTransformer()
docs_transformed = bs4_transformer.transform_documents(docs)
Initialize the transformer.
This checks if the BeautifulSoup4 package is installed.
If not, it raises an ImportError.
Methods
__init__()
Initialize the transformer.
atransform_documents(documents, **kwargs)
Asynchronously transform a list of documents.
extract_tags(html_content, tags, *[, ...])
Extract specific tags from a given HTML content.
remove_unnecessary_lines(content)
Clean up the content by removing unnecessary lines.
remove_unwanted_classnames(html_content, ...)
Remove unwanted classname from a given HTML content.
remove_unwanted_tags(html_content, unwanted_tags)
Remove unwanted tags from a given HTML content.
transform_documents(documents[, ...])
Transform a list of Document objects by cleaning their HTML content.
__init__() → None[source]¶
Initialize the transformer.
This checks if the BeautifulSoup4 package is installed.
If not, it raises an ImportError.
Return type
None
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Asynchronously transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
static extract_tags(html_content: str, tags: Union[List[str], Tuple[str, ...]], *, remove_comments: bool = False) → str[source]¶
Extract specific tags from a given HTML content.
Parameters
html_content (str) – The original HTML content string.
tags (Union[List[str], Tuple[str, ...]]) – A list of tags to be extracted from the HTML.
remove_comments (bool) – 
Returns
A string combining the content of the extracted tags.
Return type
str
static remove_unnecessary_lines(content: str) → str[source]¶
Clean up the content by removing unnecessary lines.
Parameters
content (str) – A string, which may contain unnecessary lines or spaces.
Returns
A cleaned string with unnecessary lines removed.
Return type
str
static remove_unwanted_classnames(html_content: str, unwanted_classnames: Union[List[str], Tuple[str, ...]]) → str[source]¶
Remove unwanted classname from a given HTML content.
Parameters
html_content (str) – The original HTML content string.
unwanted_classnames (Union[List[str], Tuple[str, ...]]) – A list of classnames to be removed from the HTML.
Returns
A cleaned HTML string with unwanted classnames removed.
Return type
str
static remove_unwanted_tags(html_content: str, unwanted_tags: Union[List[str], Tuple[str, ...]]) → str[source]¶
Remove unwanted tags from a given HTML content.
Parameters
html_content (str) – The original HTML content string.
unwanted_tags (Union[List[str], Tuple[str, ...]]) – A list of tags to be removed from the HTML.
Returns
A cleaned HTML string with unwanted tags removed.
Return type
str
Returns
A cleaned HTML string with unwanted tags removed.
Return type
str
transform_documents(documents: Sequence[Document], unwanted_tags: Union[List[str], Tuple[str, ...]] = ('script', 'style'), tags_to_extract: Union[List[str], Tuple[str, ...]] = ('p', 'li', 'div', 'a'), remove_lines: bool = True, *, unwanted_classnames: Union[Tuple[str, ...], List[str]] = (), remove_comments: bool = False, **kwargs: Any) → Sequence[Document][source]¶
Transform a list of Document objects by cleaning their HTML content.
Parameters
documents (Sequence[Document]) – A sequence of Document objects containing HTML content.
unwanted_tags (Union[List[str], Tuple[str, ...]]) – A list of tags to be removed from the HTML.
tags_to_extract (Union[List[str], Tuple[str, ...]]) – A list of tags whose content will be extracted.
remove_lines (bool) – If set to True, unnecessary lines will be removed.
unwanted_classnames (Union[Tuple[str, ...], List[str]]) – A list of class names to be removed from the HTML
remove_comments (bool) – If set to True, comments will be removed.
kwargs (Any) – 
Returns
A sequence of Document objects with transformed content.
Return type
Sequence[Document]
Examples using BeautifulSoupTransformer¶
Beautiful Soup
langchain_community.document_transformers.nuclia_text_transform.NucliaTextTransformer¶
class langchain_community.document_transformers.nuclia_text_transform.NucliaTextTransformer(nua: NucliaUnderstandingAPI)[source]¶
Nuclia Text Transformer.
The Nuclia Understanding API splits into paragraphs and sentences,
identifies entities, provides a summary of the text and generates
embeddings for all sentences.
Methods
__init__(nua)
atransform_documents(documents, **kwargs)
Asynchronously transform a list of documents.
transform_documents(documents, **kwargs)
Transform a list of documents.
Parameters
nua (NucliaUnderstandingAPI) – 
__init__(nua: NucliaUnderstandingAPI)[source]¶
Parameters
nua (NucliaUnderstandingAPI) – 
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Asynchronously transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
Examples using NucliaTextTransformer¶
Nuclia
Raises ValidationError if the input data cannot be parsed to form a valid model.
param tagging_chain: Any = None¶
The chain used to extract metadata from each document.
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Asynchronously transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to include in new model
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to exclude from new model, as with values this takes precedence over include
update (Optional[DictStrAny]) – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
Parameters
by_alias (bool) – 
ref_template (unicode) – 
Return type
DictStrAny
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
dumps_kwargs (Any) – 
Return type
unicode
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Automatically extract and populate metadata
for each document according to the provided schema.
Parameters
documents (Sequence[Document]) – 
kwargs (Any) – 
Return type
Sequence[Document]
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
Parameters
localns (Any) – 
Return type
None
classmethod validate(value: Any) → Model¶
Parameters
value (Any) – 
Return type
Model
langchain_community.document_transformers.html2text.Html2TextTransformer¶
class langchain_community.document_transformers.html2text.Html2TextTransformer(ignore_links: bool = True, ignore_images: bool = True)[source]¶
Replace occurrences of a particular search pattern with a replacement string
Parameters
ignore_links (bool) – Whether links should be ignored; defaults to True.
ignore_images (bool) – Whether images should be ignored; defaults to True.
Example
Methods
__init__([ignore_links, ignore_images])
atransform_documents(documents, **kwargs)
Asynchronously transform a list of documents.
transform_documents(documents, **kwargs)
Transform a list of documents.
__init__(ignore_links: bool = True, ignore_images: bool = True) → None[source]¶
Parameters
ignore_links (bool) – 
ignore_images (bool) – 
Return type
None
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Asynchronously transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
Examples using Html2TextTransformer¶
Activeloop Deep Memory
Async Chromium
HTML to text
langchain_community.document_transformers.markdownify.MarkdownifyTransformer¶
class langchain_community.document_transformers.markdownify.MarkdownifyTransformer(strip: Optional[Union[str, List[str]]] = None, convert: Optional[Union[str, List[str]]] = None, autolinks: bool = True, heading_style: str = 'ATX', **kwargs: Any)[source]¶
Converts HTML documents to Markdown format with customizable options for handling
links, images, other tags and heading styles using the markdownify library.
Parameters
strip (Optional[Union[str, List[str]]]) – A list of tags to strip. This option can’t be used with the convert option.
convert (Optional[Union[str, List[str]]]) – A list of tags to convert. This option can’t be used with the strip option.
autolinks (bool) – A boolean indicating whether the “automatic link” style should be used when a a tag’s contents match its href. Defaults to True.
heading_style (str) – Defines how headings should be converted. Accepted values are ATX, ATX_CLOSED, SETEXT, and UNDERLINED (which is an alias for SETEXT). Defaults to ATX.
**kwargs (Any) – Additional options to pass to markdownify.
Example
More configuration options can be found at the markdownify GitHub page:
https://github.com/matthewwithanm/python-markdownify
Methods
__init__([strip, convert, autolinks, ...])
atransform_documents(documents, **kwargs)
Asynchronously transform a list of documents.
transform_documents(documents, **kwargs)
Transform a list of documents.
transform_documents(documents, **kwargs)
Transform a list of documents.
__init__(strip: Optional[Union[str, List[str]]] = None, convert: Optional[Union[str, List[str]]] = None, autolinks: bool = True, heading_style: str = 'ATX', **kwargs: Any) → None[source]¶
Parameters
strip (Optional[Union[str, List[str]]]) – 
convert (Optional[Union[str, List[str]]]) – 
autolinks (bool) – 
heading_style (str) – 
kwargs (Any) – 
Return type
None
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Asynchronously transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
Examples using MarkdownifyTransformer¶
Markdownify
langchain_community.document_transformers.embeddings_redundant_filter.get_stateful_documents¶
langchain_community.document_transformers.embeddings_redundant_filter.get_stateful_documents(documents: Sequence[Document]) → Sequence[_DocumentWithState][source]¶
Convert a list of documents to a list of documents with state.
Parameters
documents (Sequence[Document]) – The documents to convert.
Returns
A list of documents with state.
Return type
Sequence[_DocumentWithState]