Tony20100/langchain_full huggingface dataset:
Parameters
path (Union[str, Path]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod parse_obj(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
b (Union[str, bytes]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
Return type
DictStrAny
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
dumps_kwargs (Any) – 
Return type
unicode
select_examples(input_variables: Dict[str, str]) → List[dict][source]¶
Select which examples to use based on the input lengths.
Parameters
input_variables (Dict[str, str]) – 
Return type
List[dict]
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
Parameters
localns (Any) – 
Return type
None
classmethod validate(value: Any) → Model¶
Parameters
value (Any) –
langchain_core.chat_history.InMemoryChatMessageHistory¶
class langchain_core.chat_history.InMemoryChatMessageHistory[source]¶
Bases: BaseChatMessageHistory, BaseModel
In memory implementation of chat message history.
Stores messages in an in memory list.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param messages: List[BaseMessage] [Optional]¶
A property or attribute that returns a list of messages.
In general, getting the messages may involve IO to the underlying
persistence layer, so this operation is expected to incur some
latency.
async aadd_messages(messages: Sequence[BaseMessage]) → None[source]¶
Add messages to the store
Parameters
messages (Sequence[BaseMessage]) – 
Return type
None
async aclear() → None[source]¶
Remove all messages from the store
Return type
None
add_ai_message(message: Union[AIMessage, str]) → None¶
Convenience method for adding an AI message string to the store.
Please note that this is a convenience method. Code should favor the bulk
add_messages interface instead to save on round-trips to the underlying
persistence layer.
This method may be deprecated in a future release.
Parameters
message (Union[AIMessage, str]) – The AI message to add.
Return type
None
add_message(message: BaseMessage) → None[source]¶
Add a self-created message to the store
Parameters
message (BaseMessage) – 
Return type
None
add_messages(messages: Sequence[BaseMessage]) → None¶
Add a list of messages.
Implementations should over-ride this method to handle bulk addition of messages
in an efficient manner to avoid unnecessary round-trips to the underlying store.
Parameters
in an efficient manner to avoid unnecessary round-trips to the underlying store.
Parameters
messages (Sequence[BaseMessage]) – A list of BaseMessage objects to store.
Return type
None
add_user_message(message: Union[HumanMessage, str]) → None¶
Convenience method for adding a human message string to the store.
Please note that this is a convenience method. Code should favor the
bulk add_messages interface instead to save on round-trips to the underlying
persistence layer.
This method may be deprecated in a future release.
Parameters
message (Union[HumanMessage, str]) – The human message to add
Return type
None
async aget_messages() → List[BaseMessage][source]¶
Async version of getting messages.
Can over-ride this method to provide an efficient async implementation.
In general, fetching messages may involve IO to the underlying
persistence layer.
Return type
List[BaseMessage]
clear() → None[source]¶
Remove all messages from the store
Return type
None
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
langchain_elasticsearch.chat_history.ElasticsearchChatMessageHistory¶
class langchain_elasticsearch.chat_history.ElasticsearchChatMessageHistory(index: str, session_id: str, *, es_connection: Optional[Elasticsearch] = None, es_url: Optional[str] = None, es_cloud_id: Optional[str] = None, es_user: Optional[str] = None, es_api_key: Optional[str] = None, es_password: Optional[str] = None, esnsure_ascii: Optional[bool] = True)[source]¶
Chat message history that stores history in Elasticsearch.
Parameters
es_url (Optional[str]) – URL of the Elasticsearch instance to connect to.
es_cloud_id (Optional[str]) – Cloud ID of the Elasticsearch instance to connect to.
es_user (Optional[str]) – Username to use when connecting to Elasticsearch.
es_password (Optional[str]) – Password to use when connecting to Elasticsearch.
es_api_key (Optional[str]) – API key to use when connecting to Elasticsearch.
es_connection (Optional[Elasticsearch]) – Optional pre-existing Elasticsearch connection.
esnsure_ascii (Optional[bool]) – Used to escape ASCII symbols in json.dumps. Defaults to True.
index (str) – Name of the index to use.
session_id (str) – Arbitrary key that is used to store the messages
of a single chat session.
Attributes
messages
Retrieve the messages from Elasticsearch
Methods
__init__(index, session_id, *[, ...])
aadd_messages(messages)
Add a list of messages.
aclear()
Remove all messages from the store
add_ai_message(message)
Convenience method for adding an AI message string to the store.
add_message(message)
Add a message to the chat session in Elasticsearch
add_messages(messages)
Add a list of messages.
add_user_message(message)
add_messages(messages)
Add a list of messages.
add_user_message(message)
Convenience method for adding a human message string to the store.
aget_messages()
Async version of getting messages.
clear()
Clear session memory in Elasticsearch
__init__(index: str, session_id: str, *, es_connection: Optional[Elasticsearch] = None, es_url: Optional[str] = None, es_cloud_id: Optional[str] = None, es_user: Optional[str] = None, es_api_key: Optional[str] = None, es_password: Optional[str] = None, esnsure_ascii: Optional[bool] = True)[source]¶
Parameters
index (str) – 
session_id (str) – 
es_connection (Optional[Elasticsearch]) – 
es_url (Optional[str]) – 
es_cloud_id (Optional[str]) – 
es_user (Optional[str]) – 
es_api_key (Optional[str]) – 
es_password (Optional[str]) – 
esnsure_ascii (Optional[bool]) – 
async aadd_messages(messages: Sequence[BaseMessage]) → None¶
Add a list of messages.
Parameters
messages (Sequence[BaseMessage]) – A list of BaseMessage objects to store.
Return type
None
async aclear() → None¶
Remove all messages from the store
Return type
None
add_ai_message(message: Union[AIMessage, str]) → None¶
Convenience method for adding an AI message string to the store.
Please note that this is a convenience method. Code should favor the bulk
add_messages interface instead to save on round-trips to the underlying
persistence layer.
This method may be deprecated in a future release.
Parameters
message (Union[AIMessage, str]) – The AI message to add.
Return type
None
Return type
None
add_message(message: BaseMessage) → None[source]¶
Add a message to the chat session in Elasticsearch
Parameters
message (BaseMessage) – 
Return type
None
add_messages(messages: Sequence[BaseMessage]) → None¶
Add a list of messages.
Implementations should over-ride this method to handle bulk addition of messages
in an efficient manner to avoid unnecessary round-trips to the underlying store.
Parameters
messages (Sequence[BaseMessage]) – A list of BaseMessage objects to store.
Return type
None
add_user_message(message: Union[HumanMessage, str]) → None¶
Convenience method for adding a human message string to the store.
Please note that this is a convenience method. Code should favor the
bulk add_messages interface instead to save on round-trips to the underlying
persistence layer.
This method may be deprecated in a future release.
Parameters
message (Union[HumanMessage, str]) – The human message to add
Return type
None
async aget_messages() → List[BaseMessage]¶
Async version of getting messages.
Can over-ride this method to provide an efficient async implementation.
In general, fetching messages may involve IO to the underlying
persistence layer.
Return type
List[BaseMessage]
clear() → None[source]¶
Clear session memory in Elasticsearch
Return type
None
Examples using ElasticsearchChatMessageHistory¶
Elasticsearch
langchain_core.chat_history.BaseChatMessageHistory¶
class langchain_core.chat_history.BaseChatMessageHistory[source]¶
Abstract base class for storing chat message history.
Implementations guidelines:
Implementations are expected to over-ride all or some of the following methods:
add_messages: sync variant for bulk addition of messages
aadd_messages: async variant for bulk addition of messages
messages: sync variant for getting messages
aget_messages: async variant for getting messages
clear: sync variant for clearing messages
aclear: async variant for clearing messages
add_messages contains a default implementation that calls add_message
for each message in the sequence. This is provided for backwards compatibility
with existing implementations which only had add_message.
Async variants all have default implementations that call the sync variants.
Implementers can choose to over-ride the async implementations to provide
truly async implementations.
Usage guidelines:
When used for updating history, users should favor usage of add_messages
over add_message or other variants like add_user_message and add_ai_message
to avoid unnecessary round-trips to the underlying persistence layer.
Example: Shows a default implementation.
class FileChatMessageHistory(BaseChatMessageHistory):
    storage_path:  str
    session_id: str
   @property
   def messages(self):
       with open(os.path.join(storage_path, session_id), 'r:utf-8') as f:
           messages = json.loads(f.read())
        return messages_from_dict(messages)
   def add_messages(self, messages: Sequence[BaseMessage]) -> None:
       all_messages = list(self.messages) # Existing messages
       all_messages.extend(messages) # Add new messages
       serialized = [message_to_dict(message) for message in all_messages]
       # Can be further optimized by only writing new messages
       # using append mode.
# Can be further optimized by only writing new messages
       # using append mode.
       with open(os.path.join(storage_path, session_id), 'w') as f:
           json.dump(f, messages)
   def clear(self):
       with open(os.path.join(storage_path, session_id), 'w') as f:
           f.write("[]")
Attributes
messages
A property or attribute that returns a list of messages.
Methods
__init__()
aadd_messages(messages)
Add a list of messages.
aclear()
Remove all messages from the store
add_ai_message(message)
Convenience method for adding an AI message string to the store.
add_message(message)
Add a Message object to the store.
add_messages(messages)
Add a list of messages.
add_user_message(message)
Convenience method for adding a human message string to the store.
aget_messages()
Async version of getting messages.
clear()
Remove all messages from the store
__init__()¶
async aadd_messages(messages: Sequence[BaseMessage]) → None[source]¶
Add a list of messages.
Parameters
messages (Sequence[BaseMessage]) – A list of BaseMessage objects to store.
Return type
None
async aclear() → None[source]¶
Remove all messages from the store
Return type
None
add_ai_message(message: Union[AIMessage, str]) → None[source]¶
Convenience method for adding an AI message string to the store.
Please note that this is a convenience method. Code should favor the bulk
add_messages interface instead to save on round-trips to the underlying
persistence layer.
This method may be deprecated in a future release.
Parameters
message (Union[AIMessage, str]) – The AI message to add.
Return type
None
Return type
None
add_message(message: BaseMessage) → None[source]¶
Add a Message object to the store.
Parameters
message (BaseMessage) – A BaseMessage object to store.
Return type
None
add_messages(messages: Sequence[BaseMessage]) → None[source]¶
Add a list of messages.
Implementations should over-ride this method to handle bulk addition of messages
in an efficient manner to avoid unnecessary round-trips to the underlying store.
Parameters
messages (Sequence[BaseMessage]) – A list of BaseMessage objects to store.
Return type
None
add_user_message(message: Union[HumanMessage, str]) → None[source]¶
Convenience method for adding a human message string to the store.
Please note that this is a convenience method. Code should favor the
bulk add_messages interface instead to save on round-trips to the underlying
persistence layer.
This method may be deprecated in a future release.
Parameters
message (Union[HumanMessage, str]) – The human message to add
Return type
None
async aget_messages() → List[BaseMessage][source]¶
Async version of getting messages.
Can over-ride this method to provide an efficient async implementation.
In general, fetching messages may involve IO to the underlying
persistence layer.
Return type
List[BaseMessage]
abstract clear() → None[source]¶
Remove all messages from the store
Return type
None
Examples using BaseChatMessageHistory¶
Build a Chatbot
Build an Agent
Conversational RAG
How to add chat history
langchain_text_splitters.base.Tokenizer¶
class langchain_text_splitters.base.Tokenizer(chunk_overlap: int, tokens_per_chunk: int, decode: Callable[[List[int]], str], encode: Callable[[str], List[int]])[source]¶
Tokenizer data class.
Attributes
chunk_overlap
Overlap in tokens between chunks
tokens_per_chunk
Maximum number of tokens per chunk
decode
Function to decode a list of token ids to a string
encode
Function to encode a string to a list of token ids
Methods
__init__(chunk_overlap, tokens_per_chunk, ...)
Parameters
chunk_overlap (int) – 
tokens_per_chunk (int) – 
decode (Callable[[List[int]], str]) – 
encode (Callable[[str], List[int]]) – 
__init__(chunk_overlap: int, tokens_per_chunk: int, decode: Callable[[List[int]], str], encode: Callable[[str], List[int]]) → None¶
Parameters
chunk_overlap (int) – 
tokens_per_chunk (int) – 
decode (Callable[[List[int]], str]) – 
encode (Callable[[str], List[int]]) – 
Return type
None
langchain_text_splitters.base.TokenTextSplitter¶
class langchain_text_splitters.base.TokenTextSplitter(encoding_name: str = 'gpt2', model_name: Optional[str] = None, allowed_special: Union[Literal['all'], AbstractSet[str]] = {}, disallowed_special: Union[Literal['all'], Collection[str]] = 'all', **kwargs: Any)[source]¶
Splitting text to tokens using model tokenizer.
Create a new TextSplitter.
Methods
__init__([encoding_name, model_name, ...])
Create a new TextSplitter.
atransform_documents(documents, **kwargs)
Asynchronously transform a list of documents.
create_documents(texts[, metadatas])
Create documents from a list of texts.
from_huggingface_tokenizer(tokenizer, **kwargs)
Text splitter that uses HuggingFace tokenizer to count length.
from_tiktoken_encoder([encoding_name, ...])
Text splitter that uses tiktoken encoder to count length.
split_documents(documents)
Split documents.
split_text(text)
Split text into multiple components.
transform_documents(documents, **kwargs)
Transform sequence of documents by splitting them.
Parameters
encoding_name (str) – 
model_name (Optional[str]) – 
allowed_special (Union[Literal['all'], AbstractSet[str]]) – 
disallowed_special (Union[Literal['all'], Collection[str]]) – 
kwargs (Any) – 
__init__(encoding_name: str = 'gpt2', model_name: Optional[str] = None, allowed_special: Union[Literal['all'], AbstractSet[str]] = {}, disallowed_special: Union[Literal['all'], Collection[str]] = 'all', **kwargs: Any) → None[source]¶
Create a new TextSplitter.
Parameters
Create a new TextSplitter.
Parameters
encoding_name (str) – 
model_name (Optional[str]) – 
allowed_special (Union[Literal['all'], ~typing.AbstractSet[str]]) – 
disallowed_special (Union[Literal['all'], ~typing.Collection[str]]) – 
kwargs (Any) – 
Return type
None
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document]¶
Asynchronously transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
create_documents(texts: List[str], metadatas: Optional[List[dict]] = None) → List[Document]¶
Create documents from a list of texts.
Parameters
texts (List[str]) – 
metadatas (Optional[List[dict]]) – 
Return type
List[Document]
classmethod from_huggingface_tokenizer(tokenizer: Any, **kwargs: Any) → TextSplitter¶
Text splitter that uses HuggingFace tokenizer to count length.
Parameters
tokenizer (Any) – 
kwargs (Any) – 
Return type
TextSplitter
classmethod from_tiktoken_encoder(encoding_name: str = 'gpt2', model_name: Optional[str] = None, allowed_special: Union[Literal['all'], AbstractSet[str]] = {}, disallowed_special: Union[Literal['all'], Collection[str]] = 'all', **kwargs: Any) → TS¶
Text splitter that uses tiktoken encoder to count length.
Parameters
encoding_name (str) – 
model_name (Optional[str]) – 
allowed_special (Union[Literal['all'], ~typing.AbstractSet[str]]) –
allowed_special (Union[Literal['all'], ~typing.AbstractSet[str]]) – 
disallowed_special (Union[Literal['all'], ~typing.Collection[str]]) – 
kwargs (Any) – 
Return type
TS
split_documents(documents: Iterable[Document]) → List[Document]¶
Split documents.
Parameters
documents (Iterable[Document]) – 
Return type
List[Document]
split_text(text: str) → List[str][source]¶
Split text into multiple components.
Parameters
text (str) – 
Return type
List[str]
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document]¶
Transform sequence of documents by splitting them.
Parameters
documents (Sequence[Document]) – 
kwargs (Any) – 
Return type
Sequence[Document]
Examples using TokenTextSplitter¶
Apache Doris
Azure AI Search
How to handle long text when doing extraction
How to split text by tokens 
StarRocks
langchain_text_splitters.base.Language¶
class langchain_text_splitters.base.Language(value)[source]¶
Enum of the programming languages.
CPP = 'cpp'¶
GO = 'go'¶
JAVA = 'java'¶
KOTLIN = 'kotlin'¶
JS = 'js'¶
TS = 'ts'¶
PHP = 'php'¶
PROTO = 'proto'¶
PYTHON = 'python'¶
RST = 'rst'¶
RUBY = 'ruby'¶
RUST = 'rust'¶
SCALA = 'scala'¶
SWIFT = 'swift'¶
MARKDOWN = 'markdown'¶
LATEX = 'latex'¶
HTML = 'html'¶
SOL = 'sol'¶
CSHARP = 'csharp'¶
COBOL = 'cobol'¶
C = 'c'¶
LUA = 'lua'¶
PERL = 'perl'¶
HASKELL = 'haskell'¶
Examples using Language¶
How to split code
Source Code
langchain_text_splitters.base.TextSplitter¶
class langchain_text_splitters.base.TextSplitter(chunk_size: int = 4000, chunk_overlap: int = 200, length_function: ~typing.Callable[[str], int] = <built-in function len>, keep_separator: ~typing.Union[bool, ~typing.Literal['start', 'end']] = False, add_start_index: bool = False, strip_whitespace: bool = True)[source]¶
Interface for splitting text into chunks.
Create a new TextSplitter.
Parameters
chunk_size (int) – Maximum size of chunks to return
chunk_overlap (int) – Overlap in characters between chunks
length_function (Callable[[str], int]) – Function that measures the length of given chunks
keep_separator (Union[bool, Literal['start', 'end']]) – Whether to keep the separator and where to place it
in each corresponding chunk (True=’start’)
add_start_index (bool) – If True, includes chunk’s start index in metadata
strip_whitespace (bool) – If True, strips whitespace from the start and end of
every document
Methods
__init__([chunk_size, chunk_overlap, ...])
Create a new TextSplitter.
atransform_documents(documents, **kwargs)
Asynchronously transform a list of documents.
create_documents(texts[, metadatas])
Create documents from a list of texts.
from_huggingface_tokenizer(tokenizer, **kwargs)
Text splitter that uses HuggingFace tokenizer to count length.
from_tiktoken_encoder([encoding_name, ...])
Text splitter that uses tiktoken encoder to count length.
split_documents(documents)
Split documents.
split_text(text)
Split text into multiple components.
transform_documents(documents, **kwargs)
Transform sequence of documents by splitting them.
transform_documents(documents, **kwargs)
Transform sequence of documents by splitting them.
__init__(chunk_size: int = 4000, chunk_overlap: int = 200, length_function: ~typing.Callable[[str], int] = <built-in function len>, keep_separator: ~typing.Union[bool, ~typing.Literal['start', 'end']] = False, add_start_index: bool = False, strip_whitespace: bool = True) → None[source]¶
Create a new TextSplitter.
Parameters
chunk_size (int) – Maximum size of chunks to return
chunk_overlap (int) – Overlap in characters between chunks
length_function (Callable[[str], int]) – Function that measures the length of given chunks
keep_separator (Union[bool, Literal['start', 'end']]) – Whether to keep the separator and where to place it
in each corresponding chunk (True=’start’)
add_start_index (bool) – If True, includes chunk’s start index in metadata
strip_whitespace (bool) – If True, strips whitespace from the start and end of
every document
Return type
None
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document]¶
Asynchronously transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
create_documents(texts: List[str], metadatas: Optional[List[dict]] = None) → List[Document][source]¶
Create documents from a list of texts.
Parameters
texts (List[str]) – 
metadatas (Optional[List[dict]]) – 
Return type
List[Document]
metadatas (Optional[List[dict]]) – 
Return type
List[Document]
classmethod from_huggingface_tokenizer(tokenizer: Any, **kwargs: Any) → TextSplitter[source]¶
Text splitter that uses HuggingFace tokenizer to count length.
Parameters
tokenizer (Any) – 
kwargs (Any) – 
Return type
TextSplitter
classmethod from_tiktoken_encoder(encoding_name: str = 'gpt2', model_name: Optional[str] = None, allowed_special: Union[Literal['all'], AbstractSet[str]] = {}, disallowed_special: Union[Literal['all'], Collection[str]] = 'all', **kwargs: Any) → TS[source]¶
Text splitter that uses tiktoken encoder to count length.
Parameters
encoding_name (str) – 
model_name (Optional[str]) – 
allowed_special (Union[Literal['all'], ~typing.AbstractSet[str]]) – 
disallowed_special (Union[Literal['all'], ~typing.Collection[str]]) – 
kwargs (Any) – 
Return type
TS
split_documents(documents: Iterable[Document]) → List[Document][source]¶
Split documents.
Parameters
documents (Iterable[Document]) – 
Return type
List[Document]
abstract split_text(text: str) → List[str][source]¶
Split text into multiple components.
Parameters
text (str) – 
Return type
List[str]
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Transform sequence of documents by splitting them.
Parameters
documents (Sequence[Document]) – 
kwargs (Any) – 
Return type
Sequence[Document]