Document Transformers
Transform documents
langchain.document_transformers.get_stateful_documents(documents)[source]
Convert a list of documents to a list of documents with state.
Parameters
documents (Sequence[langchain.schema.Document]) – The documents to convert.
Returns
A list of documents with state.
Return type
Sequence[langchain.document_transformers._DocumentWithState]
class langchain.document_transformers.EmbeddingsRedundantFilter(*, embeddings, similarity_fn=<function cosine_similarity>, similarity_threshold=0.95)[source]
Bases: langchain.schema.BaseDocumentTransformer, pydantic.main.BaseModel
Filter that drops redundant documents by comparing their embeddings.
Parameters
embeddings (langchain.embeddings.base.Embeddings) – 
similarity_fn (Callable) – 
similarity_threshold (float) – 
Return type
None
attribute embeddings: langchain.embeddings.base.Embeddings [Required]
Embeddings to use for embedding document contents.
attribute similarity_fn: Callable = <function cosine_similarity>
Similarity function for comparing documents. Function expected to take as input
two matrices (List[List[float]]) and return a matrix of scores where higher values
indicate greater similarity.
attribute similarity_threshold: float = 0.95
Threshold for determining when two documents are similar enough
to be considered redundant.
async atransform_documents(documents, **kwargs)[source]
Asynchronously transform a list of documents.
Parameters
documents (Sequence[langchain.schema.Document]) – 
kwargs (Any) – 
Return type
Sequence[langchain.schema.Document]
transform_documents(documents, **kwargs)[source]
Filter down documents.
Parameters
documents (Sequence[langchain.schema.Document]) – 
kwargs (Any) – 
Return type
Sequence[langchain.schema.Document]
Text Splitters
Functionality for splitting text.
class langchain.text_splitter.TextSplitter(chunk_size=4000, chunk_overlap=200, length_function=<built-in function len>, keep_separator=False, add_start_index=False)[source]
Bases: langchain.schema.BaseDocumentTransformer, abc.ABC
Interface for splitting text into chunks.
Parameters
chunk_size (int) – 
chunk_overlap (int) – 
length_function (Callable[[str], int]) – 
keep_separator (bool) – 
add_start_index (bool) – 
Return type
None
abstract split_text(text)[source]
Split text into multiple components.
Parameters
text (str) – 
Return type
List[str]
create_documents(texts, metadatas=None)[source]
Create documents from a list of texts.
Parameters
texts (List[str]) – 
metadatas (Optional[List[dict]]) – 
Return type
List[langchain.schema.Document]
split_documents(documents)[source]
Split documents.
Parameters
documents (Iterable[langchain.schema.Document]) – 
Return type
List[langchain.schema.Document]
classmethod from_huggingface_tokenizer(tokenizer, **kwargs)[source]
Text splitter that uses HuggingFace tokenizer to count length.
Parameters
tokenizer (Any) – 
kwargs (Any) – 
Return type
langchain.text_splitter.TextSplitter
classmethod from_tiktoken_encoder(encoding_name='gpt2', model_name=None, allowed_special={}, disallowed_special='all', **kwargs)[source]
Text splitter that uses tiktoken encoder to count length.
Parameters
encoding_name (str) – 
model_name (Optional[str]) – 
allowed_special (Union[Literal['all'], typing.AbstractSet[str]]) – 
disallowed_special (Union[Literal['all'], typing.Collection[str]]) – 
kwargs (Any) – 
Return type
langchain.text_splitter.TS
transform_documents(documents, **kwargs)[source]
Transform sequence of documents by splitting them.
Parameters
documents (Sequence[langchain.schema.Document]) – 
kwargs (Any) – 
Return type
Sequence[langchain.schema.Document]
async atransform_documents(documents, **kwargs)[source]
Asynchronously transform a sequence of documents by splitting them.
Parameters
documents (Sequence[langchain.schema.Document]) – 
kwargs (Any) – 
Return type
Sequence[langchain.schema.Document]
class langchain.text_splitter.CharacterTextSplitter(separator='\n\n', **kwargs)[source]
Bases: langchain.text_splitter.TextSplitter
Implementation of splitting text that looks at characters.
Parameters
separator (str) – 
kwargs (Any) – 
Return type
None
split_text(text)[source]
Split incoming text and return chunks.
Parameters
text (str) – 
Return type
List[str]
class langchain.text_splitter.LineType[source]
Bases: TypedDict
Line type as typed dict.
metadata: Dict[str, str]
content: str
class langchain.text_splitter.HeaderType[source]
Bases: TypedDict
Header type as typed dict.
level: int
name: str
data: str
class langchain.text_splitter.MarkdownHeaderTextSplitter(headers_to_split_on, return_each_line=False)[source]
Bases: object
Implementation of splitting markdown files based on specified headers.
Parameters
headers_to_split_on (List[Tuple[str, str]]) – 
return_each_line (bool) – 
aggregate_lines_to_chunks(lines)[source]
Combine lines with common metadata into chunks
:param lines: Line of text / associated header metadata
Parameters
lines (List[langchain.text_splitter.LineType]) – 
Return type
List[langchain.schema.Document]
split_text(text)[source]
Split markdown file
:param text: Markdown file
Parameters
text (str) – 
Return type
List[langchain.schema.Document]
class langchain.text_splitter.Tokenizer(chunk_overlap: 'int', tokens_per_chunk: 'int', decode: 'Callable[[list[int]], str]', encode: 'Callable[[str], List[int]]')[source]
Bases: object
Parameters
chunk_overlap (int) – 
tokens_per_chunk (int) – 
decode (Callable[[list[int]], str]) – 
encode (Callable[[str], List[int]]) – 
Return type
None
chunk_overlap: int
tokens_per_chunk: int
decode: Callable[[list[int]], str]
encode: Callable[[str], List[int]]
langchain.text_splitter.split_text_on_tokens(*, text, tokenizer)[source]
Split incoming text and return chunks.
Parameters
text (str) – 
tokenizer (langchain.text_splitter.Tokenizer) – 
Return type
List[str]
class langchain.text_splitter.TokenTextSplitter(encoding_name='gpt2', model_name=None, allowed_special={}, disallowed_special='all', **kwargs)[source]
Bases: langchain.text_splitter.TextSplitter
Implementation of splitting text that looks at tokens.
Parameters
encoding_name (str) – 
model_name (Optional[str]) – 
allowed_special (Union[Literal['all'], AbstractSet[str]]) – 
disallowed_special (Union[Literal['all'], Collection[str]]) – 
kwargs (Any) – 
Return type
None
split_text(text)[source]
Split text into multiple components.
Parameters
text (str) – 
Return type
List[str]
class langchain.text_splitter.SentenceTransformersTokenTextSplitter(chunk_overlap=50, model_name='sentence-transformers/all-mpnet-base-v2', tokens_per_chunk=None, **kwargs)[source]
Bases: langchain.text_splitter.TextSplitter
Implementation of splitting text that looks at tokens.
Parameters
chunk_overlap (int) – 
model_name (str) – 
tokens_per_chunk (Optional[int]) – 
kwargs (Any) – 
Return type
None
split_text(text)[source]
Split text into multiple components.
Parameters
text (str) – 
Return type
List[str]
count_tokens(*, text)[source]
Parameters
text (str) – 
Return type
int
class langchain.text_splitter.Language(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]
Bases: str, enum.Enum
CPP = 'cpp'
GO = 'go'
JAVA = 'java'
JS = 'js'
PHP = 'php'
PROTO = 'proto'
PYTHON = 'python'
RST = 'rst'
RUBY = 'ruby'
RUST = 'rust'
SCALA = 'scala'
SWIFT = 'swift'
MARKDOWN = 'markdown'
LATEX = 'latex'
HTML = 'html'
SOL = 'sol'
class langchain.text_splitter.RecursiveCharacterTextSplitter(separators=None, keep_separator=True, **kwargs)[source]
Bases: langchain.text_splitter.TextSplitter
Implementation of splitting text that looks at characters.
Recursively tries to split by different characters to find one
that works.
Parameters
separators (Optional[List[str]]) – 
keep_separator (bool) – 
kwargs (Any) – 
Return type
None
split_text(text)[source]
Split text into multiple components.
Parameters
text (str) – 
Return type
List[str]
classmethod from_language(language, **kwargs)[source]
Parameters
language (langchain.text_splitter.Language) – 
kwargs (Any) – 
Return type
langchain.text_splitter.RecursiveCharacterTextSplitter
static get_separators_for_language(language)[source]
Parameters
language (langchain.text_splitter.Language) – 
Return type
List[str]
class langchain.text_splitter.NLTKTextSplitter(separator='\n\n', **kwargs)[source]
Bases: langchain.text_splitter.TextSplitter
Implementation of splitting text that looks at sentences using NLTK.
Parameters
separator (str) – 
kwargs (Any) – 
Return type
None
split_text(text)[source]
Split incoming text and return chunks.
Parameters
text (str) – 
Return type
List[str]
class langchain.text_splitter.SpacyTextSplitter(separator='\n\n', pipeline='en_core_web_sm', **kwargs)[source]
Bases: langchain.text_splitter.TextSplitter
Implementation of splitting text that looks at sentences using Spacy.
Parameters
separator (str) – 
pipeline (str) – 
kwargs (Any) – 
Return type
None
split_text(text)[source]
Split incoming text and return chunks.
Parameters
text (str) – 
Return type
List[str]
class langchain.text_splitter.PythonCodeTextSplitter(**kwargs)[source]
Bases: langchain.text_splitter.RecursiveCharacterTextSplitter
Attempts to split the text along Python syntax.
Parameters
kwargs (Any) – 
Return type
None
class langchain.text_splitter.MarkdownTextSplitter(**kwargs)[source]
Bases: langchain.text_splitter.RecursiveCharacterTextSplitter
Attempts to split the text along Markdown-formatted headings.
Parameters
kwargs (Any) – 
Return type
None
class langchain.text_splitter.LatexTextSplitter(**kwargs)[source]
Bases: langchain.text_splitter.RecursiveCharacterTextSplitter
Attempts to split the text along Latex-formatted layout elements.
Parameters
kwargs (Any) – 
Return type
None