Tony20100/langchain_full huggingface dataset:
langchain_community.document_transformers.long_context_reorder.LongContextReorder¶
class langchain_community.document_transformers.long_context_reorder.LongContextReorder[source]¶
Bases: BaseDocumentTransformer, BaseModel
Reorder long context.
Lost in the middle:
Performance degrades when models must access relevant information
in the middle of long contexts.
See: https://arxiv.org/abs//2307.03172
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Asynchronously transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to include in new model
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to exclude from new model, as with values this takes precedence over include
update (Optional[DictStrAny]) – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
deep (bool) – set to True to make a deep copy of the model
self (Model) – 
Returns
new model instance
Return type
Model
dict(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False) → DictStrAny¶
Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
by_alias (bool) – 
skip_defaults (Optional[bool]) – 
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
Return type
DictStrAny
classmethod from_orm(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
Parameters
obj (Any) – 
Return type
Model
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
b (Union[str, bytes]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
Return type
DictStrAny
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
dumps_kwargs (Any) – 
Return type
unicode
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Reorders documents.
Parameters
documents (Sequence[Document]) – 
kwargs (Any) – 
Return type
Sequence[Document]
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
Parameters
localns (Any) – 
Return type
None
classmethod validate(value: Any) → Model¶
Parameters
value (Any) – 
Return type
Model
Examples using LongContextReorder¶
How to reorder retrieved results to mitigate the “lost in the middle” effect
LOTR (Merger Retriever)
langchain_community.document_transformers.doctran_text_extract.DoctranPropertyExtractor¶
class langchain_community.document_transformers.doctran_text_extract.DoctranPropertyExtractor(properties: List[dict], openai_api_key: Optional[str] = None, openai_api_model: Optional[str] = None)[source]¶
Extract properties from text documents using doctran.
Parameters
properties (List[dict]) – A list of the properties to extract.
openai_api_key (Optional[str]) – OpenAI API key. Can also be specified via environment variable
OPENAI_API_KEY.
openai_api_model (Optional[str]) – 
Example
from langchain_community.document_transformers import DoctranPropertyExtractor
properties = [
    {
        "name": "category",
        "description": "What type of email this is.",
        "type": "string",
        "enum": ["update", "action_item", "customer_feedback", "announcement", "other"],
        "required": True,
    },
    {
        "name": "mentions",
        "description": "A list of all people mentioned in this email.",
        "type": "array",
        "items": {
            "name": "full_name",
            "description": "The full name of the person mentioned.",
            "type": "string",
        },
        "required": True,
    },
    {
        "name": "eli5",
        "description": "Explain this email to me like I'm 5 years old.",
        "type": "string",
        "required": True,
    },
]
# Pass in openai_api_key or set env var OPENAI_API_KEY
property_extractor = DoctranPropertyExtractor(properties)
transformed_document = await qa_transformer.atransform_documents(documents)
Methods
transformed_document = await qa_transformer.atransform_documents(documents)
Methods
__init__(properties[, openai_api_key, ...])
atransform_documents(documents, **kwargs)
Extracts properties from text documents using doctran.
transform_documents(documents, **kwargs)
Extracts properties from text documents using doctran.
__init__(properties: List[dict], openai_api_key: Optional[str] = None, openai_api_model: Optional[str] = None) → None[source]¶
Parameters
properties (List[dict]) – 
openai_api_key (Optional[str]) – 
openai_api_model (Optional[str]) – 
Return type
None
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Extracts properties from text documents using doctran.
Parameters
documents (Sequence[Document]) – 
kwargs (Any) – 
Return type
Sequence[Document]
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Extracts properties from text documents using doctran.
Parameters
documents (Sequence[Document]) – 
kwargs (Any) – 
Return type
Sequence[Document]
Examples using DoctranPropertyExtractor¶
Doctran: extract properties
langchain_community.document_transformers.beautiful_soup_transformer.get_navigable_strings¶
langchain_community.document_transformers.beautiful_soup_transformer.get_navigable_strings(element: Any, *, remove_comments: bool = False) → Iterator[str][source]¶
Get all navigable strings from a BeautifulSoup element.
Parameters
element (Any) – A BeautifulSoup element.
remove_comments (bool) – 
Returns
A generator of strings.
Return type
Iterator[str]
langchain_community.document_transformers.embeddings_redundant_filter.EmbeddingsClusteringFilter¶
class langchain_community.document_transformers.embeddings_redundant_filter.EmbeddingsClusteringFilter[source]¶
Bases: BaseDocumentTransformer, BaseModel
Perform K-means clustering on document vectors.
Returns an arbitrary number of documents closest to center.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param embeddings: Embeddings [Required]¶
Embeddings to use for embedding document contents.
param num_closest: int = 1¶
The number of closest vectors to return for each cluster center.
param num_clusters: int = 5¶
Number of clusters. Groups of documents with similar meaning.
param random_state: int = 42¶
Controls the random number generator used to initialize the cluster centroids.
If you set the random_state parameter to None, the KMeans algorithm will use a
random number generator that is seeded with the current time. This means
that the results of the KMeans algorithm will be different each time you
run it.
param remove_duplicates: bool = False¶
By default duplicated results are skipped and replaced by the next closest
vector in the cluster. If remove_duplicates is true no replacement will be done:
This could dramatically reduce results when there is a lot of overlap between
clusters.
param sorted: bool = False¶
By default results are re-ordered “grouping” them by cluster, if sorted is true
result will be ordered by the original position from the retriever
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document]¶
Asynchronously transform a list of documents.
Parameters
documents (Sequence[Document]) – A sequence of Documents to be transformed.
kwargs (Any) – 
Returns
kwargs (Any) – 
Returns
A list of transformed Documents.
Return type
Sequence[Document]
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to include in new model
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to exclude from new model, as with values this takes precedence over include
update (Optional[DictStrAny]) – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
deep (bool) – set to True to make a deep copy of the model
self (Model) – 
Returns
new model instance
Return type
Model
ref_template (unicode) – 
Return type
DictStrAny
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
dumps_kwargs (Any) – 
Return type
unicode
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Filter down documents.
Parameters
documents (Sequence[Document]) – 
kwargs (Any) – 
Return type
Sequence[Document]
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
Parameters
localns (Any) – 
Return type
None
classmethod validate(value: Any) → Model¶
Parameters
value (Any) – 
Return type
Model
Examples using EmbeddingsClusteringFilter¶
LOTR (Merger Retriever)
langchain_experimental.generative_agents.memory.GenerativeAgentMemory¶
class langchain_experimental.generative_agents.memory.GenerativeAgentMemory[source]¶
Bases: BaseMemory
Memory for the generative agent.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param add_memory_key: str = 'add_memory'¶
param aggregate_importance: float = 0.0¶
Track the sum of the ‘importance’ of recent memories.
Triggers reflection when it reaches reflection_threshold.
param current_plan: List[str] = []¶
The current plan of the agent.
param importance_weight: float = 0.15¶
How much weight to assign the memory importance.
param llm: BaseLanguageModel [Required]¶
The core language model.
param max_tokens_limit: int = 1200¶
param memory_retriever: TimeWeightedVectorStoreRetriever [Required]¶
The retriever to fetch related memories.
param most_recent_memories_key: str = 'most_recent_memories'¶
param most_recent_memories_token_key: str = 'recent_memories_token'¶
param now_key: str = 'now'¶
param queries_key: str = 'queries'¶
param reflecting: bool = False¶
param reflection_threshold: Optional[float] = None¶
When aggregate_importance exceeds reflection_threshold, stop to reflect.
param relevant_memories_key: str = 'relevant_memories'¶
param relevant_memories_simple_key: str = 'relevant_memories_simple'¶
param verbose: bool = False¶
async aclear() → None¶
Clear memory contents.
Return type
None
add_memories(memory_content: str, now: Optional[datetime] = None) → List[str][source]¶
Add an observations or memories to the agent’s memory.
Parameters
memory_content (str) – 
now (Optional[datetime]) – 
Return type
List[str]
add_memory(memory_content: str, now: Optional[datetime] = None) → List[str][source]¶
Add an observation or memory to the agent’s memory.
Parameters
memory_content (str) – 
now (Optional[datetime]) – 
Return type
List[str]
async aload_memory_variables(inputs: Dict[str, Any]) → Dict[str, Any]¶
Return key-value pairs given the text input to the chain.
Parameters
inputs (Dict[str, Any]) – 
Return type
Dict[str, Any]
async asave_context(inputs: Dict[str, Any], outputs: Dict[str, str]) → None¶
Save the context of this chain run to memory.
Parameters
inputs (Dict[str, Any]) – 
outputs (Dict[str, str]) – 
Return type
None
chain(prompt: PromptTemplate) → LLMChain[source]¶
Parameters
prompt (PromptTemplate) – 
Return type
LLMChain
clear() → None[source]¶
Clear memory contents.
Return type
None
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
Return type
DictStrAny
fetch_memories(observation: str, now: Optional[datetime] = None) → List[Document][source]¶
Fetch related memories.
Parameters
observation (str) – 
now (Optional[datetime]) – 
Return type
List[Document]
format_memories_detail(relevant_memories: List[Document]) → str[source]¶
Parameters
relevant_memories (List[Document]) – 
Return type
str
format_memories_simple(relevant_memories: List[Document]) → str[source]¶
Parameters
relevant_memories (List[Document]) – 
Return type
str
classmethod from_orm(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
classmethod get_lc_namespace() → List[str]¶
Get the namespace of the langchain object.
For example, if the class is langchain.llms.openai.OpenAI, then the
namespace is [“langchain”, “llms”, “openai”]
Return type
List[str]
classmethod is_lc_serializable() → bool¶
Is this class serializable?
Return type
bool
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
by_alias (bool) – 
skip_defaults (Optional[bool]) – 
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
encoder (Optional[Callable[[Any], Any]]) – 
models_as_dict (bool) – 
dumps_kwargs (Any) – 
Return type
unicode
classmethod lc_id() → List[str]¶
A unique identifier for this class for serialization purposes.
The unique identifier is a list of strings that describes the path
to the object.
Return type
List[str]
load_memory_variables(inputs: Dict[str, Any]) → Dict[str, str][source]¶
Return key-value pairs given the text input to the chain.
Parameters
inputs (Dict[str, Any]) – 
Return type
Dict[str, str]
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
path (Union[str, Path]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
classmethod parse_obj(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
Parameters
obj (Any) – 
Return type
Model
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
Parameters
b (Union[str, bytes]) – 
content_type (unicode) – 
encoding (unicode) – 
proto (Protocol) – 
allow_pickle (bool) – 
Return type
Model
pause_to_reflect(now: Optional[datetime] = None) → List[str][source]¶
Reflect on recent observations and generate ‘insights’.
Parameters
now (Optional[datetime]) – 
Return type
List[str]
save_context(inputs: Dict[str, Any], outputs: Dict[str, Any]) → None[source]¶
Save the context of this model run to memory.
Parameters
inputs (Dict[str, Any]) – 
outputs (Dict[str, Any]) – 
Return type
None
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
Return type
DictStrAny
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
dumps_kwargs (Any) – 
Return type
unicode
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
Return type
Union[SerializedConstructor, SerializedNotImplemented]
to_json_not_implemented() → SerializedNotImplemented¶
Return type
SerializedNotImplemented
classmethod update_forward_refs(**localns: Any) → None¶
SerializedNotImplemented
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
Parameters
localns (Any) – 
Return type
None
classmethod validate(value: Any) → Model¶
Parameters
value (Any) – 
Return type
Model
property lc_attributes: Dict¶
List of attribute names that should be included in the serialized kwargs.
These attributes must be accepted by the constructor.
property lc_secrets: Dict[str, str]¶
A map of constructor argument names to secret ids.
For example,{“openai_api_key”: “OPENAI_API_KEY”}
property memory_variables: List[str]¶
Input keys this memory class will load dynamically.
langchain_experimental.generative_agents.generative_agent.GenerativeAgent¶
class langchain_experimental.generative_agents.generative_agent.GenerativeAgent[source]¶
Bases: BaseModel
Agent as a character with memory and innate characteristics.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param age: Optional[int] = None¶
The optional age of the character.
param daily_summaries: List[str] [Optional]¶
Summary of the events in the plan that the agent took.
param last_refreshed: datetime [Optional]¶
The last time the character’s summary was regenerated.
param llm: BaseLanguageModel [Required]¶
The underlying language model.
param memory: GenerativeAgentMemory [Required]¶
The memory object that combines relevance, recency, and ‘importance’.
param name: str [Required]¶
The character’s name.
param status: str [Required]¶
The traits of the character you wish not to change.
param summary: str = ''¶
Stateful self-summary generated via reflection on the character’s memory.
param summary_refresh_seconds: int = 3600¶
How frequently to re-generate the summary.
param traits: str = 'N/A'¶
Permanent traits to ascribe to the character.
param verbose: bool = False¶
chain(prompt: PromptTemplate) → LLMChain[source]¶
Create a chain with the same settings as the agent.
Parameters
prompt (PromptTemplate) – 
Return type
LLMChain
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
Parameters
_fields_set (Optional[SetStr]) – 
values (Any) – 
Return type
Model
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to include in new model
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – fields to exclude from new model, as with values this takes precedence over include
update (Optional[DictStrAny]) – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
deep (bool) – set to True to make a deep copy of the model
self (Model) – 
Returns
new model instance
Return type
Model
dict(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False) → DictStrAny¶
Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
Parameters
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) –
include (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
exclude (Optional[Union[AbstractSetIntStr, MappingIntStrAny]]) – 
by_alias (bool) – 
skip_defaults (Optional[bool]) – 
exclude_unset (bool) – 
exclude_defaults (bool) – 
exclude_none (bool) – 
Return type
DictStrAny
classmethod from_orm(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
generate_dialogue_response(observation: str, now: Optional[datetime] = None) → Tuple[bool, str][source]¶
React to a given observation.
Parameters
observation (str) – 
now (Optional[datetime]) – 
Return type
Tuple[bool, str]
generate_reaction(observation: str, now: Optional[datetime] = None) → Tuple[bool, str][source]¶
React to a given observation.
Parameters
observation (str) – 
now (Optional[datetime]) – 
Return type
Tuple[bool, str]
get_full_header(force_refresh: bool = False, now: Optional[datetime] = None) → str[source]¶
Return a full header of the agent’s status, summary, and current time.
Parameters
force_refresh (bool) – 
now (Optional[datetime]) – 
Return type
str
get_summary(force_refresh: bool = False, now: Optional[datetime] = None) → str[source]¶
Return a descriptive summary of the agent.
Parameters
force_refresh (bool) – 
now (Optional[datetime]) – 
Return type
str