Tony20100/langchain_full huggingface dataset:
langchain_community.chat_loaders.facebook_messenger.SingleFileFacebookMessengerChatLoader¶
class langchain_community.chat_loaders.facebook_messenger.SingleFileFacebookMessengerChatLoader(path: Union[Path, str])[source]¶
Load Facebook Messenger chat data from a single file.
Parameters
path (Union[Path, str]) – The path to the chat file.
path¶
The path to the chat file.
Type
Path
Methods
__init__(path)
lazy_load()
Lazy loads the chat data from the file.
load()
Eagerly load the chat sessions into memory.
__init__(path: Union[Path, str]) → None[source]¶
Parameters
path (Union[Path, str]) – 
Return type
None
lazy_load() → Iterator[ChatSession][source]¶
Lazy loads the chat data from the file.
Yields
ChatSession – A chat session containing the loaded messages.
Return type
Iterator[ChatSession]
load() → List[ChatSession]¶
Eagerly load the chat sessions into memory.
Return type
List[ChatSession]
Examples using SingleFileFacebookMessengerChatLoader¶
Facebook - Meta
Facebook Messenger
langchain_community.chat_loaders.utils.merge_chat_runs_in_session¶
langchain_community.chat_loaders.utils.merge_chat_runs_in_session(chat_session: ChatSession, delimiter: str = '\n\n') → ChatSession[source]¶
Merge chat runs together in a chat session.
A chat run is a sequence of messages from the same sender.
Parameters
chat_session (ChatSession) – A chat session.
delimiter (str) – 
Returns
A chat session with merged chat runs.
Return type
ChatSession
langchain_community.chat_loaders.utils.map_ai_messages_in_session¶
langchain_community.chat_loaders.utils.map_ai_messages_in_session(chat_sessions: ChatSession, sender: str) → ChatSession[source]¶
Convert messages from the specified ‘sender’ to AI messages.
This is useful for fine-tuning the AI to adapt to your voice.
Parameters
chat_sessions (ChatSession) – 
sender (str) – 
Return type
ChatSession
langchain_community.chat_loaders.telegram.TelegramChatLoader¶
class langchain_community.chat_loaders.telegram.TelegramChatLoader(path: Union[str, Path])[source]¶
Load telegram conversations to LangChain chat messages.
To export, use the Telegram Desktop app from
https://desktop.telegram.org/, select a conversation, click the three dots
in the top right corner, and select “Export chat history”. Then select
“Machine-readable JSON” (preferred) to export. Note: the ‘lite’ versions of
the desktop app (like “Telegram for MacOS”) do not support exporting chat
history.
Initialize the TelegramChatLoader.
Parameters
path (Union[str, Path]) – Path to the exported Telegram chat zip,
directory, json, or HTML file.
Methods
__init__(path)
Initialize the TelegramChatLoader.
lazy_load()
Lazy load the messages from the chat file and yield them in as chat sessions.
load()
Eagerly load the chat sessions into memory.
__init__(path: Union[str, Path])[source]¶
Initialize the TelegramChatLoader.
Parameters
path (Union[str, Path]) – Path to the exported Telegram chat zip,
directory, json, or HTML file.
lazy_load() → Iterator[ChatSession][source]¶
Lazy load the messages from the chat file and yield them
in as chat sessions.
Yields
ChatSession – The loaded chat session.
Return type
Iterator[ChatSession]
load() → List[ChatSession]¶
Eagerly load the chat sessions into memory.
Return type
List[ChatSession]
Examples using TelegramChatLoader¶
Telegram
langchain_experimental.data_anonymizer.deanonymizer_mapping.create_anonymizer_mapping¶
langchain_experimental.data_anonymizer.deanonymizer_mapping.create_anonymizer_mapping(original_text: str, analyzer_results: List[RecognizerResult], anonymizer_results: EngineResult, is_reversed: bool = False) → Dict[str, Dict[str, str]][source]¶
Create or update the mapping used to anonymize and/ordeanonymize a text.
This method exploits the results returned by the
analysis and anonymization processes.
If is_reversed is True, it constructs a mapping from each original
entity to its anonymized value.
If is_reversed is False, it constructs a mapping from each
anonymized entity back to its original text value.
If there are multiple entities of the same type, the mapping will
include a count to differentiate them. For example, if there are
two names in the input text, the mapping will include NAME_1 and NAME_2.
Example of mapping:
{
“PERSON”: {“<original>”: “<anonymized>”,
“John Doe”: “Slim Shady”
},
“PHONE_NUMBER”: {
“111-111-1111”: “555-555-5555”
}
Parameters
original_text (str) – 
analyzer_results (List[RecognizerResult]) – 
anonymizer_results (EngineResult) – 
is_reversed (bool) – 
Return type
Dict[str, Dict[str, str]]
langchain_experimental.data_anonymizer.faker_presidio_mapping.get_pseudoanonymizer_mapping¶
langchain_experimental.data_anonymizer.faker_presidio_mapping.get_pseudoanonymizer_mapping(seed: Optional[int] = None) → Dict[str, Callable][source]¶
Get a mapping of entities to pseudo anonymize them.
Parameters
seed (Optional[int]) – 
Return type
Dict[str, Callable]
langchain_experimental.data_anonymizer.presidio.PresidioAnonymizer¶
class langchain_experimental.data_anonymizer.presidio.PresidioAnonymizer(analyzed_fields: Optional[List[str]] = None, operators: Optional[Dict[str, OperatorConfig]] = None, languages_config: Optional[Dict] = None, add_default_faker_operators: bool = True, faker_seed: Optional[int] = None)[source]¶
Anonymizer using Microsoft Presidio.
Parameters
analyzed_fields (Optional[List[str]]) – List of fields to detect and then anonymize.
Defaults to all entities supported by Microsoft Presidio.
operators (Optional[Dict[str, OperatorConfig]]) – Operators to use for anonymization.
Operators allow for custom anonymization of detected PII.
Learn more:
https://microsoft.github.io/presidio/tutorial/10_simple_anonymization/
languages_config (Optional[Dict]) – Configuration for the NLP engine.
First language in the list will be used as the main language
in self.anonymize(…) when no language is specified.
Learn more:
https://microsoft.github.io/presidio/analyzer/customizing_nlp_models/
faker_seed (Optional[int]) – Seed used to initialize faker.
Defaults to None, in which case faker will be seeded randomly
and provide random values.
add_default_faker_operators (bool) – 
Methods
__init__([analyzed_fields, operators, ...])
param analyzed_fields
List of fields to detect and then anonymize.
add_operators(operators)
Add operators to the anonymizer
add_recognizer(recognizer)
Add a recognizer to the analyzer
anonymize(text[, language, allow_list])
Anonymize text.
anonymize(text[, language, allow_list])
Anonymize text.
__init__(analyzed_fields: Optional[List[str]] = None, operators: Optional[Dict[str, OperatorConfig]] = None, languages_config: Optional[Dict] = None, add_default_faker_operators: bool = True, faker_seed: Optional[int] = None)¶
Parameters
analyzed_fields (Optional[List[str]]) – List of fields to detect and then anonymize.
Defaults to all entities supported by Microsoft Presidio.
operators (Optional[Dict[str, OperatorConfig]]) – Operators to use for anonymization.
Operators allow for custom anonymization of detected PII.
Learn more:
https://microsoft.github.io/presidio/tutorial/10_simple_anonymization/
languages_config (Optional[Dict]) – Configuration for the NLP engine.
First language in the list will be used as the main language
in self.anonymize(…) when no language is specified.
Learn more:
https://microsoft.github.io/presidio/analyzer/customizing_nlp_models/
faker_seed (Optional[int]) – Seed used to initialize faker.
Defaults to None, in which case faker will be seeded randomly
and provide random values.
add_default_faker_operators (bool) – 
add_operators(operators: Dict[str, OperatorConfig]) → None¶
Add operators to the anonymizer
Parameters
operators (Dict[str, OperatorConfig]) – Operators to add to the anonymizer.
Return type
None
add_recognizer(recognizer: EntityRecognizer) → None¶
Add a recognizer to the analyzer
Parameters
recognizer (EntityRecognizer) – Recognizer to add to the analyzer.
Return type
None
anonymize(text: str, language: Optional[str] = None, allow_list: Optional[List[str]] = None) → str¶
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.case_insensitive_matching_strategy¶
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.case_insensitive_matching_strategy(text: str, deanonymizer_mapping: Dict[str, Dict[str, str]]) → str[source]¶
Case insensitive matching strategy for deanonymization.
It replaces all the anonymized entities with the original onesirrespective of their letter case.
Parameters
text (str) – text to deanonymize
deanonymizer_mapping (Dict[str, Dict[str, str]]) – mapping between anonymized entities and original ones
Return type
str
Examples of matching:keanu reeves -> Keanu Reeves
JOHN F. KENNEDY -> John F. Kennedy
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.combined_exact_fuzzy_matching_strategy¶
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.combined_exact_fuzzy_matching_strategy(text: str, deanonymizer_mapping: Dict[str, Dict[str, str]], max_l_dist: int = 3) → str[source]¶
Combined exact and fuzzy matching strategy for deanonymization.
It is a RECOMMENDED STRATEGY.
Parameters
text (str) – text to deanonymize
deanonymizer_mapping (Dict[str, Dict[str, str]]) – mapping between anonymized entities and original ones
max_l_dist (int) – maximum Levenshtein distance between the anonymized entity and the
text segment to consider it a match
Return type
str
Examples of matching:Kaenu Reves -> Keanu Reeves
John F. Kennedy -> John Kennedy
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.exact_matching_strategy¶
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.exact_matching_strategy(text: str, deanonymizer_mapping: Dict[str, Dict[str, str]]) → str[source]¶
Exact matching strategy for deanonymization.
It replaces all the anonymized entities with the original ones.
Parameters
text (str) – text to deanonymize
deanonymizer_mapping (Dict[str, Dict[str, str]]) – mapping between anonymized entities and original ones
Return type
str
langchain_experimental.data_anonymizer.deanonymizer_mapping.DeanonymizerMapping¶
class langchain_experimental.data_anonymizer.deanonymizer_mapping.DeanonymizerMapping(mapping: ~typing.Dict[str, ~typing.Dict[str, str]] = <factory>)[source]¶
Deanonymizer mapping.
Attributes
data
Return the deanonymizer mapping.
mapping
Methods
__init__([mapping])
update(new_mapping)
Update the deanonymizer mapping with new values.
Parameters
mapping (Dict[str, Dict[str, str]]) – 
__init__(mapping: ~typing.Dict[str, ~typing.Dict[str, str]] = <factory>) → None¶
Parameters
mapping (Dict[str, Dict[str, str]]) – 
Return type
None
update(new_mapping: Dict[str, Dict[str, str]]) → None[source]¶
Update the deanonymizer mapping with new values.
Duplicated values will not be added
If there are multiple entities of the same type, the mapping will
include a count to differentiate them. For example, if there are
two names in the input text, the mapping will include NAME_1 and NAME_2.
Parameters
new_mapping (Dict[str, Dict[str, str]]) – 
Return type
None
langchain_experimental.data_anonymizer.base.ReversibleAnonymizerBase¶
class langchain_experimental.data_anonymizer.base.ReversibleAnonymizerBase[source]¶
Base abstract class for reversible anonymizers.
Methods
__init__()
anonymize(text[, language, allow_list])
Anonymize text.
deanonymize(text_to_deanonymize[, ...])
Deanonymize text
reset_deanonymizer_mapping()
Abstract method to reset deanonymizer mapping
__init__()¶
anonymize(text: str, language: Optional[str] = None, allow_list: Optional[List[str]] = None) → str¶
Anonymize text.
Parameters
text (str) – 
language (Optional[str]) – 
allow_list (Optional[List[str]]) – 
Return type
str
deanonymize(text_to_deanonymize: str, deanonymizer_matching_strategy: ~typing.Callable[[str, ~typing.Dict[str, ~typing.Dict[str, str]]], str] = <function exact_matching_strategy>) → str[source]¶
Deanonymize text
Parameters
text_to_deanonymize (str) – 
deanonymizer_matching_strategy (Callable[[str, Dict[str, Dict[str, str]]], str]) – 
Return type
str
abstract reset_deanonymizer_mapping() → None[source]¶
Abstract method to reset deanonymizer mapping
Return type
None
langchain_experimental.data_anonymizer.presidio.PresidioReversibleAnonymizer¶
class langchain_experimental.data_anonymizer.presidio.PresidioReversibleAnonymizer(analyzed_fields: Optional[List[str]] = None, operators: Optional[Dict[str, OperatorConfig]] = None, languages_config: Optional[Dict] = None, add_default_faker_operators: bool = True, faker_seed: Optional[int] = None)[source]¶
Reversible Anonymizer using Microsoft Presidio.
Parameters
analyzed_fields (Optional[List[str]]) – List of fields to detect and then anonymize.
Defaults to all entities supported by Microsoft Presidio.
operators (Optional[Dict[str, OperatorConfig]]) – Operators to use for anonymization.
Operators allow for custom anonymization of detected PII.
Learn more:
https://microsoft.github.io/presidio/tutorial/10_simple_anonymization/
languages_config (Optional[Dict]) – Configuration for the NLP engine.
First language in the list will be used as the main language
in self.anonymize(…) when no language is specified.
Learn more:
https://microsoft.github.io/presidio/analyzer/customizing_nlp_models/
faker_seed (Optional[int]) – Seed used to initialize faker.
Defaults to None, in which case faker will be seeded randomly
and provide random values.
add_default_faker_operators (bool) – 
Attributes
anonymizer_mapping
Return the anonymizer mapping This is just the reverse version of the deanonymizer mapping.
deanonymizer_mapping
Return the deanonymizer mapping
Methods
__init__([analyzed_fields, operators, ...])
param analyzed_fields
List of fields to detect and then anonymize.
add_operators(operators)
Add operators to the anonymizer
add_operators(operators)
Add operators to the anonymizer
add_recognizer(recognizer)
Add a recognizer to the analyzer
anonymize(text[, language, allow_list])
Anonymize text.
deanonymize(text_to_deanonymize[, ...])
Deanonymize text
load_deanonymizer_mapping(file_path)
Load the deanonymizer mapping from a JSON or YAML file.
reset_deanonymizer_mapping()
Reset the deanonymizer mapping
save_deanonymizer_mapping(file_path)
Save the deanonymizer mapping to a JSON or YAML file.
__init__(analyzed_fields: Optional[List[str]] = None, operators: Optional[Dict[str, OperatorConfig]] = None, languages_config: Optional[Dict] = None, add_default_faker_operators: bool = True, faker_seed: Optional[int] = None)[source]¶
Parameters
analyzed_fields (Optional[List[str]]) – List of fields to detect and then anonymize.
Defaults to all entities supported by Microsoft Presidio.
operators (Optional[Dict[str, OperatorConfig]]) – Operators to use for anonymization.
Operators allow for custom anonymization of detected PII.
Learn more:
https://microsoft.github.io/presidio/tutorial/10_simple_anonymization/
languages_config (Optional[Dict]) – Configuration for the NLP engine.
First language in the list will be used as the main language
in self.anonymize(…) when no language is specified.
Learn more:
https://microsoft.github.io/presidio/analyzer/customizing_nlp_models/
faker_seed (Optional[int]) – Seed used to initialize faker.
Defaults to None, in which case faker will be seeded randomly
and provide random values.
add_default_faker_operators (bool) –
and provide random values.
add_default_faker_operators (bool) – 
add_operators(operators: Dict[str, OperatorConfig]) → None¶
Add operators to the anonymizer
Parameters
operators (Dict[str, OperatorConfig]) – Operators to add to the anonymizer.
Return type
None
add_recognizer(recognizer: EntityRecognizer) → None¶
Add a recognizer to the analyzer
Parameters
recognizer (EntityRecognizer) – Recognizer to add to the analyzer.
Return type
None
anonymize(text: str, language: Optional[str] = None, allow_list: Optional[List[str]] = None) → str¶
Anonymize text.
Parameters
text (str) – 
language (Optional[str]) – 
allow_list (Optional[List[str]]) – 
Return type
str
deanonymize(text_to_deanonymize: str, deanonymizer_matching_strategy: ~typing.Callable[[str, ~typing.Dict[str, ~typing.Dict[str, str]]], str] = <function exact_matching_strategy>) → str¶
Deanonymize text
Parameters
text_to_deanonymize (str) – 
deanonymizer_matching_strategy (Callable[[str, Dict[str, Dict[str, str]]], str]) – 
Return type
str
load_deanonymizer_mapping(file_path: Union[Path, str]) → None[source]¶
Load the deanonymizer mapping from a JSON or YAML file.
Parameters
file_path (Union[Path, str]) – Path to file to load the mapping from.
Return type
None
Example:
.. code-block:: python
anonymizer.load_deanonymizer_mapping(file_path=”path/mapping.json”)
reset_deanonymizer_mapping() → None[source]¶
Reset the deanonymizer mapping
Return type
None
Reset the deanonymizer mapping
Return type
None
save_deanonymizer_mapping(file_path: Union[Path, str]) → None[source]¶
Save the deanonymizer mapping to a JSON or YAML file.
Parameters
file_path (Union[Path, str]) – Path to file to save the mapping to.
Return type
None
Example:
.. code-block:: python
anonymizer.save_deanonymizer_mapping(file_path=”path/mapping.json”)
Examples using PresidioReversibleAnonymizer¶
Microsoft
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.ngram_fuzzy_matching_strategy¶
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.ngram_fuzzy_matching_strategy(text: str, deanonymizer_mapping: Dict[str, Dict[str, str]], fuzzy_threshold: int = 85, use_variable_length: bool = True) → str[source]¶
N-gram fuzzy matching strategy for deanonymization.
It replaces all the anonymized entities with the original ones.
It uses fuzzy matching to find the position of the anonymized entity in the text.
It generates n-grams of the same length as the anonymized entity from the text and
uses fuzzy matching to find the position of the anonymized entity in the text.
Parameters
text (str) – text to deanonymize
deanonymizer_mapping (Dict[str, Dict[str, str]]) – mapping between anonymized entities and original ones
fuzzy_threshold (int) – fuzzy matching threshold
use_variable_length (bool) – whether to use (n-1, n, n+1)-grams or just n-grams
Return type
str
langchain_experimental.data_anonymizer.base.AnonymizerBase¶
class langchain_experimental.data_anonymizer.base.AnonymizerBase[source]¶
Base abstract class for anonymizers.
It is public and non-virtual because it allowswrapping the behavior for all methods in a base class.
Methods
__init__()
anonymize(text[, language, allow_list])
Anonymize text.
__init__()¶
anonymize(text: str, language: Optional[str] = None, allow_list: Optional[List[str]] = None) → str[source]¶
Anonymize text.
Parameters
text (str) – 
language (Optional[str]) – 
allow_list (Optional[List[str]]) – 
Return type
str
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.fuzzy_matching_strategy¶
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.fuzzy_matching_strategy(text: str, deanonymizer_mapping: Dict[str, Dict[str, str]], max_l_dist: int = 3) → str[source]¶
Fuzzy matching strategy for deanonymization.
It uses fuzzy matching to find the position of the anonymized entity in the text.
It replaces all the anonymized entities with the original ones.
Parameters
text (str) – text to deanonymize
deanonymizer_mapping (Dict[str, Dict[str, str]]) – mapping between anonymized entities and original ones
max_l_dist (int) – maximum Levenshtein distance between the anonymized entity and the
text segment to consider it a match
Return type
str
Examples of matching:Kaenu Reves -> Keanu Reeves
John F. Kennedy -> John Kennedy
langchain_experimental.data_anonymizer.presidio.PresidioAnonymizerBase¶
class langchain_experimental.data_anonymizer.presidio.PresidioAnonymizerBase(analyzed_fields: Optional[List[str]] = None, operators: Optional[Dict[str, OperatorConfig]] = None, languages_config: Optional[Dict] = None, add_default_faker_operators: bool = True, faker_seed: Optional[int] = None)[source]¶
Base Anonymizer using Microsoft Presidio.
See more: https://microsoft.github.io/presidio/
Parameters
analyzed_fields (Optional[List[str]]) – List of fields to detect and then anonymize.
Defaults to all entities supported by Microsoft Presidio.
operators (Optional[Dict[str, OperatorConfig]]) – Operators to use for anonymization.
Operators allow for custom anonymization of detected PII.
Learn more:
https://microsoft.github.io/presidio/tutorial/10_simple_anonymization/
languages_config (Optional[Dict]) – Configuration for the NLP engine.
First language in the list will be used as the main language
in self.anonymize(…) when no language is specified.
Learn more:
https://microsoft.github.io/presidio/analyzer/customizing_nlp_models/
faker_seed (Optional[int]) – Seed used to initialize faker.
Defaults to None, in which case faker will be seeded randomly
and provide random values.
add_default_faker_operators (bool) – 
Methods
__init__([analyzed_fields, operators, ...])
param analyzed_fields
List of fields to detect and then anonymize.
add_operators(operators)
Add operators to the anonymizer
add_recognizer(recognizer)
Add a recognizer to the analyzer
anonymize(text[, language, allow_list])
Add a recognizer to the analyzer
anonymize(text[, language, allow_list])
Anonymize text.
__init__(analyzed_fields: Optional[List[str]] = None, operators: Optional[Dict[str, OperatorConfig]] = None, languages_config: Optional[Dict] = None, add_default_faker_operators: bool = True, faker_seed: Optional[int] = None)[source]¶
Parameters
analyzed_fields (Optional[List[str]]) – List of fields to detect and then anonymize.
Defaults to all entities supported by Microsoft Presidio.
operators (Optional[Dict[str, OperatorConfig]]) – Operators to use for anonymization.
Operators allow for custom anonymization of detected PII.
Learn more:
https://microsoft.github.io/presidio/tutorial/10_simple_anonymization/
languages_config (Optional[Dict]) – Configuration for the NLP engine.
First language in the list will be used as the main language
in self.anonymize(…) when no language is specified.
Learn more:
https://microsoft.github.io/presidio/analyzer/customizing_nlp_models/
faker_seed (Optional[int]) – Seed used to initialize faker.
Defaults to None, in which case faker will be seeded randomly
and provide random values.
add_default_faker_operators (bool) – 
add_operators(operators: Dict[str, OperatorConfig]) → None[source]¶
Add operators to the anonymizer
Parameters
operators (Dict[str, OperatorConfig]) – Operators to add to the anonymizer.
Return type
None
add_recognizer(recognizer: EntityRecognizer) → None[source]¶
Add a recognizer to the analyzer
Parameters
recognizer (EntityRecognizer) – Recognizer to add to the analyzer.
Return type
None
langchain_google_community.docai.DocAIParser¶
class langchain_google_community.docai.DocAIParser(*, client: Optional[DocumentProcessorServiceClient] = None, location: Optional[str] = None, gcs_output_path: Optional[str] = None, processor_name: Optional[str] = None)[source]¶
Google Cloud Document AI parser.
For a detailed explanation of Document AI, refer to the product documentation.
https://cloud.google.com/document-ai/docs/overview
Initializes the parser.
Parameters
client (Optional[DocumentProcessorServiceClient]) – a DocumentProcessorServiceClient to use
location (Optional[str]) – a Google Cloud location where a Document AI processor is located
gcs_output_path (Optional[str]) – a path on Google Cloud Storage to store parsing results
processor_name (Optional[str]) – full resource name of a Document AI processor or processor
version
You should provide either a client or location (and then a clientwould be instantiated).
Methods
__init__(*[, client, location, ...])
Initializes the parser.
batch_parse(blobs[, gcs_output_path, ...])
Parses a list of blobs lazily.
docai_parse(blobs, *[, gcs_output_path, ...])
Runs Google Document AI PDF Batch Processing on a list of blobs.
get_results(operations)
is_running(operations)
lazy_parse(blob)
Parses a blob lazily.
online_process(blob[, ...])
Parses a blob lazily using online processing.
operations_from_names(operation_names)
Initializes Long-Running Operations from their names.
parse(blob)
Eagerly parse the blob into a document or documents.
parse_from_results(results)