Tony20100/langchain_full huggingface dataset:
to False.
run_name (Optional[str]) – 
Returns
A dict of named outputs. Should contain all outputs specified inChain.output_keys.
Return type
Dict[str, Any]
Notes
Deprecated since version langchain==0.1.0: Use ainvoke instead.
async aevaluate_agent_trajectory(*, prediction: str, agent_trajectory: Sequence[Tuple[AgentAction, str]], input: str, reference: Optional[str] = None, **kwargs: Any) → dict¶
Asynchronously evaluate a trajectory.
Parameters
prediction (str) – The final predicted response.
agent_trajectory (List[Tuple[AgentAction, str]]) – The intermediate steps forming the agent trajectory.
input (str) – The input to the agent.
reference (Optional[str]) – The reference answer.
kwargs (Any) – 
Returns
The evaluation result.
Return type
dict
async ainvoke(input: Dict[str, Any], config: Optional[RunnableConfig] = None, **kwargs: Any) → Dict[str, Any]¶
Default implementation of ainvoke, calls invoke from a thread.
The default implementation allows usage of async code even if
the runnable did not implement a native async version of invoke.
Subclasses should override this method if they can run asynchronously.
Parameters
input (Dict[str, Any]) – 
config (Optional[RunnableConfig]) – 
kwargs (Any) – 
Return type
Dict[str, Any]
apply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
[Deprecated] Call the chain on all inputs in the list.
Notes
Deprecated since version langchain==0.1.0: Use batch instead.
Parameters
input_list (List[Dict[str, Any]]) –
update (Optional[DictStrAny]) – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
deep (bool) – set to True to make a deep copy of the model
self (Model) – 
Returns
new model instance
Return type
Model
dict(**kwargs: Any) → Dict¶
Dictionary representation of chain.
Expects Chain._chain_type property to be implemented and for memory to benull.
Parameters
**kwargs (Any) – Keyword arguments passed to default pydantic.BaseModel.dict
method.
Returns
A dictionary representation of the chain.
Return type
Dict
Example
chain.dict(exclude_unset=True)
# -> {"_type": "foo", "verbose": False, ...}
evaluate_agent_trajectory(*, prediction: str, agent_trajectory: Sequence[Tuple[AgentAction, str]], input: str, reference: Optional[str] = None, **kwargs: Any) → dict¶
Evaluate a trajectory.
Parameters
prediction (str) – The final predicted response.
agent_trajectory (List[Tuple[AgentAction, str]]) – The intermediate steps forming the agent trajectory.
input (str) – The input to the agent.
reference (Optional[str]) – The reference answer.
kwargs (Any) – 
Returns
The evaluation result.
Return type
dict
classmethod from_llm(llm: BaseLanguageModel, agent_tools: Optional[Sequence[BaseTool]] = None, output_parser: Optional[TrajectoryOutputParser] = None, **kwargs: Any) → TrajectoryEvalChain[source]¶
Create a TrajectoryEvalChain object from a language model chain.
Parameters
llm (BaseChatModel) – The language model chain.
agent_tools (Optional[Sequence[BaseTool]]) – A list of tools
available to the agent.
available to the agent.
output_parser (Optional[TrajectoryOutputParser]) – The output parser
used to parse the chain output into a score.
kwargs (Any) – 
Returns
The TrajectoryEvalChain object.
Return type
TrajectoryEvalChain
classmethod from_orm(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
static get_agent_trajectory(steps: Union[str, Sequence[Tuple[AgentAction, str]]]) → str[source]¶
Get the agent trajectory as a formatted string.
Parameters
steps (Union[str, List[Tuple[AgentAction, str]]]) – The agent trajectory.
Returns
The formatted agent trajectory.
Return type
str
get_graph(config: Optional[RunnableConfig] = None) → Graph¶
Return a graph representation of this runnable.
Parameters
config (Optional[RunnableConfig]) – 
Return type
Graph
get_input_schema(config: Optional[RunnableConfig] = None) → Type[BaseModel]¶
Get a pydantic model that can be used to validate input to the runnable.
Runnables that leverage the configurable_fields and configurable_alternatives
methods will have a dynamic input schema that depends on which
configuration the runnable is invoked with.
This method allows to get an input schema for a specific configuration.
Parameters
config (Optional[RunnableConfig]) – A config to use when generating the schema.
Returns
A pydantic model that can be used to validate input.
Return type
Type[BaseModel]
classmethod get_lc_namespace() → List[str]¶
Get the namespace of the langchain object.
For example, if the class is langchain.llms.openai.OpenAI, then the
namespace is [“langchain”, “llms”, “openai”]
Return type
List[str]
def add_one(x: int) -> int:
    return x + 1
def mul_two(x: int) -> int:
    return x * 2
runnable_1 = RunnableLambda(add_one)
runnable_2 = RunnableLambda(mul_two)
sequence = runnable_1.pipe(runnable_2)
# Or equivalently:
# sequence = runnable_1 | runnable_2
# sequence = RunnableSequence(first=runnable_1, last=runnable_2)
sequence.invoke(1)
await sequence.ainvoke(1)
# -> 4
sequence.batch([1, 2, 3])
await sequence.abatch([1, 2, 3])
# -> [4, 6, 8]
Parameters
others (Union[Runnable[Any, Other], Callable[[Any], Other]]) – 
name (Optional[str]) – 
Return type
RunnableSerializable[Input, Other]
prep_inputs(inputs: Union[Dict[str, Any], Any]) → Dict[str, str][source]¶
Validate and prep inputs.
Parameters
inputs (Union[Dict[str, Any], Any]) – 
Return type
Dict[str, str]
prep_outputs(inputs: Dict[str, str], outputs: Dict[str, str], return_only_outputs: bool = False) → Dict[str, str]¶
Validate and prepare chain outputs, and save info about this run to memory.
Parameters
inputs (Dict[str, str]) – Dictionary of chain inputs, including any inputs added by chain
memory.
outputs (Dict[str, str]) – Dictionary of initial chain outputs.
return_only_outputs (bool) – Whether to only return the chain outputs. If False,
inputs are also added to the final outputs.
Returns
A dict of the final chain outputs.
Return type
Dict[str, str]
Returns
A dict of the final chain outputs.
Return type
Dict[str, str]
run(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶
[Deprecated] Convenience method for executing chain.
The main difference between this method and Chain.__call__ is that this
method expects inputs to be passed directly in as positional arguments or
keyword arguments, whereas Chain.__call__ expects a single input dictionary
with all the inputs
Parameters
*args (Any) – If the chain expects a single input, it can be passed in as the
sole positional argument.
callbacks (Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]]) – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags (Optional[List[str]]) – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.
**kwargs (Any) – If the chain expects multiple inputs, they can be passed in
directly as keyword arguments.
metadata (Optional[Dict[str, Any]]) – 
**kwargs – 
Returns
The chain output.
Return type
Any
Example
# Suppose we have a single-input chain that takes a 'question' string:
chain.run("What's the temperature in Boise, Idaho?")
# -> "The temperature in Boise is..."
# Suppose we have a multi-input chain that takes a 'question' string
# and 'context' string:
# and 'context' string:
question = "What's the temperature in Boise, Idaho?"
context = "Weather report for Boise, Idaho on 07/03/23..."
chain.run(question=question, context=context)
# -> "The temperature in Boise is..."
Notes
Deprecated since version langchain==0.1.0: Use invoke instead.
save(file_path: Union[Path, str]) → None¶
Save the chain.
Expects Chain._chain_type property to be implemented and for memory to benull.
Parameters
file_path (Union[Path, str]) – Path to file to save the chain to.
Return type
None
Example
chain.save(file_path="path/chain.yaml")
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
Return type
DictStrAny
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
Parameters
by_alias (bool) – 
ref_template (unicode) – 
dumps_kwargs (Any) – 
Return type
unicode
stream(input: Input, config: Optional[RunnableConfig] = None, **kwargs: Optional[Any]) → Iterator[Output]¶
Default implementation of stream, which calls invoke.
Subclasses should override this method if they support streaming output.
Parameters
input (Input) – 
config (Optional[RunnableConfig]) – 
kwargs (Optional[Any]) – 
Return type
Iterator[Output]
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
Serialize the runnable to JSON.
Return type
Union[SerializedConstructor, SerializedNotImplemented]
on_end (Optional[Union[Callable[[Run], None], Callable[[Run, RunnableConfig], None]]]) – 
on_error (Optional[Union[Callable[[Run], None], Callable[[Run, RunnableConfig], None]]]) – 
Return type
Runnable[Input, Output]
with_retry(*, retry_if_exception_type: ~typing.Tuple[~typing.Type[BaseException], ...] = (<class 'Exception'>,), wait_exponential_jitter: bool = True, stop_after_attempt: int = 3) → Runnable[Input, Output]¶
Create a new Runnable that retries the original runnable on exceptions.
Example:
from langchain_core.runnables import RunnableLambda
count = 0
def _lambda(x: int) -> None:
    global count
    count = count + 1
    if x == 1:
        raise ValueError("x is 1")
    else:
         pass
runnable = RunnableLambda(_lambda)
try:
    runnable.with_retry(
        stop_after_attempt=2,
        retry_if_exception_type=(ValueError,),
    ).invoke(1)
except ValueError:
    pass
assert (count == 2)
Parameters
retry_if_exception_type (Tuple[Type[BaseException], ...]) – A tuple of exception types to retry on
wait_exponential_jitter (bool) – Whether to add jitter to the wait time
between retries
stop_after_attempt (int) – The maximum number of attempts to make before giving up
Returns
A new Runnable that retries the original runnable on exceptions.
Return type
Runnable[Input, Output]
with_types(*, input_type: Optional[Type[Input]] = None, output_type: Optional[Type[Output]] = None) → Runnable[Input, Output]¶
Bind input and output types to a Runnable, returning a new Runnable.
Bind input and output types to a Runnable, returning a new Runnable.
Parameters
input_type (Optional[Type[Input]]) – 
output_type (Optional[Type[Output]]) – 
Return type
Runnable[Input, Output]
property InputType: Type[Input]¶
The type of input this runnable accepts specified as a type annotation.
property OutputType: Type[Output]¶
The type of output this runnable produces specified as a type annotation.
property config_specs: List[ConfigurableFieldSpec]¶
List configurable fields for this runnable.
property input_keys: List[str]¶
Get the input keys for the chain.
Returns
The input keys.
Return type
List[str]
property input_schema: Type[BaseModel]¶
The type of input this runnable accepts specified as a pydantic model.
property lc_attributes: Dict¶
List of attribute names that should be included in the serialized kwargs.
These attributes must be accepted by the constructor.
property lc_secrets: Dict[str, str]¶
A map of constructor argument names to secret ids.
For example,{“openai_api_key”: “OPENAI_API_KEY”}
name: Optional[str] = None¶
The name of the runnable. Used for debugging and tracing.
property output_keys: List[str]¶
Get the output keys for the chain.
Returns
The output keys.
Return type
List[str]
property output_schema: Type[BaseModel]¶
The type of output this runnable produces specified as a pydantic model.
property requires_input: bool¶
Whether this evaluator requires an input string.
property requires_reference: bool¶
Whether this evaluator requires a reference label.
langchain.evaluation.regex_match.base.RegexMatchStringEvaluator¶
class langchain.evaluation.regex_match.base.RegexMatchStringEvaluator(*, flags: int = 0, **kwargs: Any)[source]¶
Compute a regex match between the prediction and the reference.
Examples
>>> evaluator = RegexMatchStringEvaluator(flags=re.IGNORECASE)
>>> evaluator.evaluate_strings(
        prediction="Mindy is the CTO",
        reference="^mindy.*cto$",
    )  # This will return {'score': 1.0} due to the IGNORECASE flag
>>> evaluator = RegexMatchStringEvaluator()
>>> evaluator.evaluate_strings(
        prediction="Mindy is the CTO",
        reference="^Mike.*CEO$",
    )  # This will return {'score': 0.0}
>>> evaluator.evaluate_strings(
        prediction="Mindy is the CTO",
        reference="^Mike.*CEO$|^Mindy.*CTO$",
    )  # This will return {'score': 1.0} as the prediction matches the second pattern in the union
Attributes
evaluation_name
Get the evaluation name.
input_keys
Get the input keys.
requires_input
This evaluator does not require input.
requires_reference
This evaluator requires a reference.
Methods
__init__(*[, flags])
aevaluate_strings(*, prediction[, ...])
Asynchronously evaluate Chain or LLM output, based on optional input and label.
evaluate_strings(*, prediction[, reference, ...])
Evaluate Chain or LLM output, based on optional input and label.
Parameters
flags (int) – 
kwargs (Any) – 
__init__(*, flags: int = 0, **kwargs: Any)[source]¶
Parameters
flags (int) – 
kwargs (Any) –
Parameters
flags (int) – 
kwargs (Any) – 
async aevaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict¶
Asynchronously evaluate Chain or LLM output, based on optional input and label.
Parameters
prediction (str) – The LLM or chain prediction to evaluate.
reference (Optional[str], optional) – The reference label to evaluate against.
input (Optional[str], optional) – The input to consider during evaluation.
**kwargs – Additional keyword arguments, including callbacks, tags, etc.
Returns
The evaluation results containing the score or value.
Return type
dict
evaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict¶
Evaluate Chain or LLM output, based on optional input and label.
Parameters
prediction (str) – The LLM or chain prediction to evaluate.
reference (Optional[str], optional) – The reference label to evaluate against.
input (Optional[str], optional) – The input to consider during evaluation.
**kwargs – Additional keyword arguments, including callbacks, tags, etc.
Returns
The evaluation results containing the score or value.
Return type
dict
langchain.evaluation.criteria.eval_chain.LabeledCriteriaEvalChain¶
class langchain.evaluation.criteria.eval_chain.LabeledCriteriaEvalChain[source]¶
Bases: CriteriaEvalChain
Criteria evaluation chain that requires references.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param callback_manager: Optional[BaseCallbackManager] = None¶
[DEPRECATED] Use callbacks instead.
param callbacks: Callbacks = None¶
Optional list of callback handlers (or callback manager). Defaults to None.
Callback handlers are called throughout the lifecycle of a call to a chain,
starting with on_chain_start, ending with on_chain_end or on_chain_error.
Each custom chain can optionally call additional callback methods, see Callback docs
for full details.
param criterion_name: str [Required]¶
The name of the criterion being evaluated.
param llm: Union[Runnable[LanguageModelInput, str], Runnable[LanguageModelInput, BaseMessage]] [Required]¶
Language model to call.
param llm_kwargs: dict [Optional]¶
param memory: Optional[BaseMemory] = None¶
Optional memory object. Defaults to None.
Memory is a class that gets called at the start
and at the end of every chain. At the start, memory loads variables and passes
them along in the chain. At the end, it saves any returned variables.
There are many different types of memory - please see memory docs
for the full catalog.
param metadata: Optional[Dict[str, Any]] = None¶
Optional metadata associated with the chain. Defaults to None.
This metadata will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
You can use these to eg identify a specific instance of a chain with its use case.
param output_parser: BaseOutputParser [Optional]¶
The parser to use to map the output to a structured result.
param prompt: BasePromptTemplate [Required]¶
Prompt object to use.
param return_final_only: bool = True¶
Whether to return only the final parsed result. Defaults to True.
If false, will return a bunch of extra information about the generation.
param tags: Optional[List[str]] = None¶
Optional list of tags associated with the chain. Defaults to None.
These tags will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param verbose: bool [Optional]¶
Whether or not run in verbose mode. In verbose mode, some intermediate logs
will be printed to the console. Defaults to the global verbose value,
accessible via langchain.globals.get_verbose().
__call__(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, run_name: Optional[str] = None, include_run_info: bool = False) → Dict[str, Any]¶
[Deprecated] Execute the chain.
Parameters
inputs (Union[Dict[str, Any], Any]) – Dictionary of inputs, or single input if chain expects
only one param. Should contain all inputs specified in
Chain.input_keys except for inputs that will be set by the chain’s
memory.
return_only_outputs (bool) – Whether to return only outputs in the
response. If True, only new keys generated by this chain will be
# -> {"_type": "foo", "verbose": False, ...}
evaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict¶
Evaluate Chain or LLM output, based on optional input and label.
Parameters
prediction (str) – The LLM or chain prediction to evaluate.
reference (Optional[str], optional) – The reference label to evaluate against.
input (Optional[str], optional) – The input to consider during evaluation.
**kwargs – Additional keyword arguments, including callbacks, tags, etc.
Returns
The evaluation results containing the score or value.
Return type
dict
classmethod from_llm(llm: BaseLanguageModel, criteria: Optional[Union[Mapping[str, str], Criteria, ConstitutionalPrinciple]] = None, *, prompt: Optional[BasePromptTemplate] = None, **kwargs: Any) → CriteriaEvalChain[source]¶
Create a LabeledCriteriaEvalChain instance from an llm and criteria.
Parameters
llm (BaseLanguageModel) – The language model to use for evaluation.
criteria (CRITERIA_TYPE - default=None for "helpfulness") – 
The criteria to evaluate the runs against. It can be:
a mapping of a criterion name to its description
a single criterion name present in one of the default criteria
a single ConstitutionalPrinciple instance
prompt (Optional[BasePromptTemplate], default=None) – The prompt template to use for generating prompts. If not provided,
a default prompt will be used.
**kwargs (Any) – Additional keyword arguments to pass to the LLMChain
constructor.
Returns
An instance of the LabeledCriteriaEvalChain class.
Return type
LabeledCriteriaEvalChain
Examples
>>> from langchain_openai import OpenAI
>>> from langchain.evaluation.criteria import LabeledCriteriaEvalChain
Returns
A dict of the final chain outputs.
Return type
Dict[str, str]
prep_prompts(input_list: List[Dict[str, Any]], run_manager: Optional[CallbackManagerForChainRun] = None) → Tuple[List[PromptValue], Optional[List[str]]]¶
Prepare prompts from inputs.
Parameters
input_list (List[Dict[str, Any]]) – 
run_manager (Optional[CallbackManagerForChainRun]) – 
Return type
Tuple[List[PromptValue], Optional[List[str]]]
classmethod resolve_criteria(criteria: Optional[Union[Mapping[str, str], Criteria, ConstitutionalPrinciple, str]]) → Dict[str, str]¶
Resolve the criteria to evaluate.
Parameters
criteria (CRITERIA_TYPE) – 
The criteria to evaluate the runs against. It can be:
a mapping of a criterion name to its description
a single criterion name present in one of the default criteria
a single ConstitutionalPrinciple instance
Returns
A dictionary mapping criterion names to descriptions.
Return type
Dict[str, str]
Examples
>>> criterion = "relevance"
>>> CriteriaEvalChain.resolve_criteria(criteria)
{'relevance': 'Is the submission referring to a real quote from the text?'}
run(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶
[Deprecated] Convenience method for executing chain.
The main difference between this method and Chain.__call__ is that this
method expects inputs to be passed directly in as positional arguments or
keyword arguments, whereas Chain.__call__ expects a single input dictionary
with all the inputs
Parameters
*args (Any) – If the chain expects a single input, it can be passed in as the
sole positional argument.
langchain.evaluation.string_distance.base.StringDistance¶
class langchain.evaluation.string_distance.base.StringDistance(value)[source]¶
Distance metric to use.
DAMERAU_LEVENSHTEIN¶
The Damerau-Levenshtein distance.
LEVENSHTEIN¶
The Levenshtein distance.
JARO¶
The Jaro distance.
JARO_WINKLER¶
The Jaro-Winkler distance.
HAMMING¶
The Hamming distance.
INDEL¶
The Indel distance.
DAMERAU_LEVENSHTEIN = 'damerau_levenshtein'¶
LEVENSHTEIN = 'levenshtein'¶
JARO = 'jaro'¶
JARO_WINKLER = 'jaro_winkler'¶
HAMMING = 'hamming'¶
INDEL = 'indel'¶
langchain.evaluation.embedding_distance.base.EmbeddingDistanceEvalChain¶
class langchain.evaluation.embedding_distance.base.EmbeddingDistanceEvalChain[source]¶
Bases: _EmbeddingDistanceChainMixin, StringEvaluator
Use embedding distances to score semantic difference between
a prediction and reference.
Examples
>>> chain = EmbeddingDistanceEvalChain()
>>> result = chain.evaluate_strings(prediction="Hello", reference="Hi")
>>> print(result)
{'score': 0.5}
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param callback_manager: Optional[BaseCallbackManager] = None¶
[DEPRECATED] Use callbacks instead.
param callbacks: Callbacks = None¶
Optional list of callback handlers (or callback manager). Defaults to None.
Callback handlers are called throughout the lifecycle of a call to a chain,
starting with on_chain_start, ending with on_chain_end or on_chain_error.
Each custom chain can optionally call additional callback methods, see Callback docs
for full details.
param distance_metric: EmbeddingDistance = EmbeddingDistance.COSINE¶
param embeddings: Embeddings [Optional]¶
param memory: Optional[BaseMemory] = None¶
Optional memory object. Defaults to None.
Memory is a class that gets called at the start
and at the end of every chain. At the start, memory loads variables and passes
them along in the chain. At the end, it saves any returned variables.
There are many different types of memory - please see memory docs
for the full catalog.
param metadata: Optional[Dict[str, Any]] = None¶
Optional metadata associated with the chain. Defaults to None.
This metadata will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.