Tony20100/langchain_full huggingface dataset:
This metadata will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param normalize_by: Optional[float] = None¶
The value to normalize the score by, if specified.
param output_parser: BaseOutputParser [Optional]¶
Output parser to use.
Defaults to one that takes the most likely string but does not change it
otherwise.
param prompt: BasePromptTemplate [Required]¶
Prompt object to use.
param return_final_only: bool = True¶
Whether to return only the final parsed result. Defaults to True.
If false, will return a bunch of extra information about the generation.
param tags: Optional[List[str]] = None¶
Optional list of tags associated with the chain. Defaults to None.
These tags will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param verbose: bool [Optional]¶
Whether or not run in verbose mode. In verbose mode, some intermediate logs
will be printed to the console. Defaults to the global verbose value,
accessible via langchain.globals.get_verbose().
__call__(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, run_name: Optional[str] = None, include_run_info: bool = False) → Dict[str, Any]¶
[Deprecated] Execute the chain.
Parameters
inputs (Union[Dict[str, Any], Any]) – Dictionary of inputs, or single input if chain expects
# -> {"_type": "foo", "verbose": False, ...}
evaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict¶
Evaluate Chain or LLM output, based on optional input and label.
Parameters
prediction (str) – The LLM or chain prediction to evaluate.
reference (Optional[str], optional) – The reference label to evaluate against.
input (Optional[str], optional) – The input to consider during evaluation.
**kwargs – Additional keyword arguments, including callbacks, tags, etc.
Returns
The evaluation results containing the score or value.
Return type
dict
classmethod from_llm(llm: BaseLanguageModel, *, prompt: Optional[PromptTemplate] = None, criteria: Optional[Union[Mapping[str, str], Criteria, ConstitutionalPrinciple, str]] = None, normalize_by: Optional[float] = None, **kwargs: Any) → LabeledScoreStringEvalChain[source]¶
Initialize the LabeledScoreStringEvalChain from an LLM.
Parameters
llm (BaseLanguageModel) – The LLM to use.
prompt (PromptTemplate, optional) – The prompt to use.
criteria (Union[CRITERIA_TYPE, str], optional) – The criteria to use.
normalize_by (float, optional) – The value to normalize the score by.
**kwargs (Any) – Additional keyword arguments.
Returns
The initialized LabeledScoreStringEvalChain.
Return type
LabeledScoreStringEvalChain
Raises
ValueError – If the input variables are not as expected.
classmethod from_orm(obj: Any) → Model¶
Parameters
obj (Any) – 
Return type
Model
classmethod from_string(llm: BaseLanguageModel, template: str) → LLMChain¶
classmethod from_string(llm: BaseLanguageModel, template: str) → LLMChain¶
Create LLMChain from LLM and template.
Parameters
llm (BaseLanguageModel) – 
template (str) – 
Return type
LLMChain
generate(input_list: List[Dict[str, Any]], run_manager: Optional[CallbackManagerForChainRun] = None) → LLMResult¶
Generate LLM result from inputs.
Parameters
input_list (List[Dict[str, Any]]) – 
run_manager (Optional[CallbackManagerForChainRun]) – 
Return type
LLMResult
get_graph(config: Optional[RunnableConfig] = None) → Graph¶
Return a graph representation of this runnable.
Parameters
config (Optional[RunnableConfig]) – 
Return type
Graph
get_input_schema(config: Optional[RunnableConfig] = None) → Type[BaseModel]¶
Get a pydantic model that can be used to validate input to the runnable.
Runnables that leverage the configurable_fields and configurable_alternatives
methods will have a dynamic input schema that depends on which
configuration the runnable is invoked with.
This method allows to get an input schema for a specific configuration.
Parameters
config (Optional[RunnableConfig]) – A config to use when generating the schema.
Returns
A pydantic model that can be used to validate input.
Return type
Type[BaseModel]
classmethod get_lc_namespace() → List[str]¶
Get the namespace of the langchain object.
For example, if the class is langchain.llms.openai.OpenAI, then the
namespace is [“langchain”, “llms”, “openai”]
Return type
List[str]
get_name(suffix: Optional[str] = None, *, name: Optional[str] = None) → str¶
Get the name of the runnable.
Get the name of the runnable.
Parameters
suffix (Optional[str]) – 
name (Optional[str]) – 
Return type
str
get_output_schema(config: Optional[RunnableConfig] = None) → Type[BaseModel]¶
Get a pydantic model that can be used to validate output to the runnable.
Runnables that leverage the configurable_fields and configurable_alternatives
methods will have a dynamic output schema that depends on which
configuration the runnable is invoked with.
This method allows to get an output schema for a specific configuration.
Parameters
config (Optional[RunnableConfig]) – A config to use when generating the schema.
Returns
A pydantic model that can be used to validate output.
Return type
Type[BaseModel]
get_prompts(config: Optional[RunnableConfig] = None) → List[BasePromptTemplate]¶
Parameters
config (Optional[RunnableConfig]) – 
Return type
List[BasePromptTemplate]
invoke(input: Dict[str, Any], config: Optional[RunnableConfig] = None, **kwargs: Any) → Dict[str, Any]¶
Transform a single input into an output. Override to implement.
Parameters
input (Dict[str, Any]) – The input to the runnable.
config (Optional[RunnableConfig]) – A config to use when invoking the runnable.
The config supports standard keys like ‘tags’, ‘metadata’ for tracing
purposes, ‘max_concurrency’ for controlling how much work to do
in parallel, and other keys. Please refer to the RunnableConfig
for more details.
kwargs (Any) – 
Returns
The output of the runnable.
Return type
Dict[str, Any]
classmethod is_lc_serializable() → bool¶
Is this class serializable?
Return type
bool
output_type (Optional[Type[Output]]) – 
Return type
Runnable[Input, Output]
property InputType: Type[Input]¶
The type of input this runnable accepts specified as a type annotation.
property OutputType: Type[Output]¶
The type of output this runnable produces specified as a type annotation.
property config_specs: List[ConfigurableFieldSpec]¶
List configurable fields for this runnable.
property evaluation_name: str¶
Get the name of the evaluation.
Returns
The name of the evaluation.
Return type
str
property input_schema: Type[BaseModel]¶
The type of input this runnable accepts specified as a pydantic model.
property lc_attributes: Dict¶
List of attribute names that should be included in the serialized kwargs.
These attributes must be accepted by the constructor.
property lc_secrets: Dict[str, str]¶
A map of constructor argument names to secret ids.
For example,{“openai_api_key”: “OPENAI_API_KEY”}
name: Optional[str] = None¶
The name of the runnable. Used for debugging and tracing.
property output_schema: Type[BaseModel]¶
The type of output this runnable produces specified as a pydantic model.
property requires_input: bool¶
Return whether the chain requires an input.
Returns
True if the chain requires an input, False otherwise.
Return type
bool
property requires_reference: bool¶
Return whether the chain requires a reference.
Returns
True if the chain requires a reference, False otherwise.
Return type
bool
langchain.evaluation.schema.EvaluatorType¶
class langchain.evaluation.schema.EvaluatorType(value)[source]¶
The types of the evaluators.
QA = 'qa'¶
Question answering evaluator, which grades answers to questions
directly using an LLM.
COT_QA = 'cot_qa'¶
Chain of thought question answering evaluator, which grades
answers to questions using
chain of thought ‘reasoning’.
CONTEXT_QA = 'context_qa'¶
Question answering evaluator that incorporates ‘context’ in the response.
PAIRWISE_STRING = 'pairwise_string'¶
The pairwise string evaluator, which predicts the preferred prediction from
between two models.
SCORE_STRING = 'score_string'¶
The scored string evaluator, which gives a score between 1 and 10
to a prediction.
LABELED_PAIRWISE_STRING = 'labeled_pairwise_string'¶
The labeled pairwise string evaluator, which predicts the preferred prediction
from between two models based on a ground truth reference label.
LABELED_SCORE_STRING = 'labeled_score_string'¶
The labeled scored string evaluator, which gives a score between 1 and 10
to a prediction based on a ground truth reference label.
AGENT_TRAJECTORY = 'trajectory'¶
The agent trajectory evaluator, which grades the agent’s intermediate steps.
CRITERIA = 'criteria'¶
The criteria evaluator, which evaluates a model based on a
custom set of criteria without any reference labels.
LABELED_CRITERIA = 'labeled_criteria'¶
The labeled criteria evaluator, which evaluates a model based on a
custom set of criteria, with a reference label.
STRING_DISTANCE = 'string_distance'¶
Compare predictions to a reference answer using string edit distances.
EXACT_MATCH = 'exact_match'¶
Compare predictions to a reference answer using exact matching.
Compare predictions to a reference answer using exact matching.
REGEX_MATCH = 'regex_match'¶
Compare predictions to a reference answer using regular expressions.
PAIRWISE_STRING_DISTANCE = 'pairwise_string_distance'¶
Compare predictions based on string edit distances.
EMBEDDING_DISTANCE = 'embedding_distance'¶
Compare a prediction to a reference label using embedding distance.
PAIRWISE_EMBEDDING_DISTANCE = 'pairwise_embedding_distance'¶
Compare two predictions using embedding distance.
JSON_VALIDITY = 'json_validity'¶
Check if a prediction is valid JSON.
JSON_EQUALITY = 'json_equality'¶
Check if a prediction is equal to a reference JSON.
JSON_EDIT_DISTANCE = 'json_edit_distance'¶
Compute the edit distance between two JSON strings after canonicalization.
JSON_SCHEMA_VALIDATION = 'json_schema_validation'¶
Check if a prediction is valid JSON according to a JSON schema.
langchain.evaluation.schema.AgentTrajectoryEvaluator¶
class langchain.evaluation.schema.AgentTrajectoryEvaluator[source]¶
Interface for evaluating agent trajectories.
Attributes
requires_input
Whether this evaluator requires an input string.
requires_reference
Whether this evaluator requires a reference label.
Methods
__init__()
aevaluate_agent_trajectory(*, prediction, ...)
Asynchronously evaluate a trajectory.
evaluate_agent_trajectory(*, prediction, ...)
Evaluate a trajectory.
__init__()¶
async aevaluate_agent_trajectory(*, prediction: str, agent_trajectory: Sequence[Tuple[AgentAction, str]], input: str, reference: Optional[str] = None, **kwargs: Any) → dict[source]¶
Asynchronously evaluate a trajectory.
Parameters
prediction (str) – The final predicted response.
agent_trajectory (List[Tuple[AgentAction, str]]) – The intermediate steps forming the agent trajectory.
input (str) – The input to the agent.
reference (Optional[str]) – The reference answer.
kwargs (Any) – 
Returns
The evaluation result.
Return type
dict
evaluate_agent_trajectory(*, prediction: str, agent_trajectory: Sequence[Tuple[AgentAction, str]], input: str, reference: Optional[str] = None, **kwargs: Any) → dict[source]¶
Evaluate a trajectory.
Parameters
prediction (str) – The final predicted response.
agent_trajectory (List[Tuple[AgentAction, str]]) – The intermediate steps forming the agent trajectory.
input (str) – The input to the agent.
reference (Optional[str]) – The reference answer.
kwargs (Any) – 
Returns
The evaluation result.
Return type
dict
langchain.evaluation.criteria.eval_chain.CriteriaEvalChain¶
class langchain.evaluation.criteria.eval_chain.CriteriaEvalChain[source]¶
Bases: StringEvaluator, LLMEvalChain, LLMChain
LLM Chain for evaluating runs against criteria.
Parameters
llm (BaseLanguageModel) – The language model to use for evaluation.
criteria (Union[Mapping[str, str]]) – The criteria or rubric to evaluate the runs against. It can be a mapping of
criterion name to its description, or a single criterion name.
prompt (Optional[BasePromptTemplate], default=None) – The prompt template to use for generating prompts. If not provided, a
default prompt template will be used based on the value of
requires_reference.
requires_reference (bool, default=False) – Whether the evaluation requires a reference text. If True, the
PROMPT_WITH_REFERENCES template will be used, which includes the
reference labels in the prompt. Otherwise, the PROMPT template will be
used, which is a reference-free prompt.
**kwargs (Any) – Additional keyword arguments to pass to the LLMChain constructor.
Returns
An instance of the CriteriaEvalChain class.
Return type
CriteriaEvalChain
Examples
>>> from langchain_anthropic import ChatAnthropic
>>> from langchain.evaluation.criteria import CriteriaEvalChain
>>> llm = ChatAnthropic(temperature=0)
>>> criteria = {"my-custom-criterion": "Is the submission the most amazing ever?"}
>>> evaluator = CriteriaEvalChain.from_llm(llm=llm, criteria=criteria)
>>> evaluator.evaluate_strings(prediction="Imagine an ice cream flavor for the color aquamarine", input="Tell me an idea")
{
{
    'reasoning': 'Here is my step-by-step reasoning for the given criteria:\n\nThe criterion is: "Is the submission the most amazing ever?" This is a subjective criterion and open to interpretation. The submission suggests an aquamarine-colored ice cream flavor which is creative but may or may not be considered the most amazing idea ever conceived. There are many possible amazing ideas and this one ice cream flavor suggestion may or may not rise to that level for every person. \n\nN',
    'value': 'N',
    'score': 0,
}
>>> from langchain_openai import ChatOpenAI
>>> from langchain.evaluation.criteria import LabeledCriteriaEvalChain
>>> llm = ChatOpenAI(model="gpt-4", temperature=0)
>>> criteria = "correctness"
>>> evaluator = LabeledCriteriaEvalChain.from_llm(
...     llm=llm,
...     criteria=criteria,
... )
>>> evaluator.evaluate_strings(
...   prediction="The answer is 4",
...   input="How many apples are there?",
...   reference="There are 3 apples",
...   )
{
    'score': 0,
    'reasoning': 'The criterion for this task is the correctness of the submission. The submission states that there are 4 apples, but the reference indicates that there are actually 3 apples. Therefore, the submission is not correct, accurate, or factual according to the given criterion.\n\nN',
    'value': 'N',
}
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param callback_manager: Optional[BaseCallbackManager] = None¶
[DEPRECATED] Use callbacks instead.
param callbacks: Callbacks = None¶
[DEPRECATED] Use callbacks instead.
param callbacks: Callbacks = None¶
Optional list of callback handlers (or callback manager). Defaults to None.
Callback handlers are called throughout the lifecycle of a call to a chain,
starting with on_chain_start, ending with on_chain_end or on_chain_error.
Each custom chain can optionally call additional callback methods, see Callback docs
for full details.
param criterion_name: str [Required]¶
The name of the criterion being evaluated.
param llm: Union[Runnable[LanguageModelInput, str], Runnable[LanguageModelInput, BaseMessage]] [Required]¶
Language model to call.
param llm_kwargs: dict [Optional]¶
param memory: Optional[BaseMemory] = None¶
Optional memory object. Defaults to None.
Memory is a class that gets called at the start
and at the end of every chain. At the start, memory loads variables and passes
them along in the chain. At the end, it saves any returned variables.
There are many different types of memory - please see memory docs
for the full catalog.
param metadata: Optional[Dict[str, Any]] = None¶
Optional metadata associated with the chain. Defaults to None.
This metadata will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param output_parser: BaseOutputParser [Optional]¶
The parser to use to map the output to a structured result.
param prompt: BasePromptTemplate [Required]¶
Prompt object to use.
param return_final_only: bool = True¶
Whether to return only the final parsed result. Defaults to True.
If false, will return a bunch of extra information about the generation.
param tags: Optional[List[str]] = None¶
param tags: Optional[List[str]] = None¶
Optional list of tags associated with the chain. Defaults to None.
These tags will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param verbose: bool [Optional]¶
Whether or not run in verbose mode. In verbose mode, some intermediate logs
will be printed to the console. Defaults to the global verbose value,
accessible via langchain.globals.get_verbose().
__call__(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, run_name: Optional[str] = None, include_run_info: bool = False) → Dict[str, Any]¶
[Deprecated] Execute the chain.
Parameters
inputs (Union[Dict[str, Any], Any]) – Dictionary of inputs, or single input if chain expects
only one param. Should contain all inputs specified in
Chain.input_keys except for inputs that will be set by the chain’s
memory.
return_only_outputs (bool) – Whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks (Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]]) – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
these runtime callbacks will propagate to calls to other objects.
tags (Optional[List[str]]) – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.
metadata (Optional[Dict[str, Any]]) – Optional metadata associated with the chain. Defaults to None
include_run_info (bool) – Whether to include run info in the response. Defaults
to False.
run_name (Optional[str]) – 
Returns
A dict of named outputs. Should contain all outputs specified inChain.output_keys.
Return type
Dict[str, Any]
Notes
Deprecated since version langchain==0.1.0: Use invoke instead.
async aapply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Utilize the LLM generate method for speed gains.
Parameters
input_list (List[Dict[str, Any]]) – 
callbacks (Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]]) – 
Return type
List[Dict[str, str]]
async aapply_and_parse(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → Sequence[Union[str, List[str], Dict[str, str]]]¶
Call apply and then parse the results.
Parameters
input_list (List[Dict[str, Any]]) – 
callbacks (Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]]) – 
Return type
Sequence[Union[str, List[str], Dict[str, str]]]
Return type
Sequence[Union[str, List[str], Dict[str, str]]]
async abatch(inputs: List[Input], config: Optional[Union[RunnableConfig, List[RunnableConfig]]] = None, *, return_exceptions: bool = False, **kwargs: Optional[Any]) → List[Output]¶
Default implementation runs ainvoke in parallel using asyncio.gather.
The default implementation of batch works well for IO bound runnables.
Subclasses should override this method if they can batch more efficiently;
e.g., if the underlying runnable uses an API which supports a batch mode.
Parameters
inputs (List[Input]) – 
config (Optional[Union[RunnableConfig, List[RunnableConfig]]]) – 
return_exceptions (bool) – 
kwargs (Optional[Any]) – 
Return type
List[Output]
async abatch_as_completed(inputs: Sequence[Input], config: Optional[Union[RunnableConfig, Sequence[RunnableConfig]]] = None, *, return_exceptions: bool = False, **kwargs: Optional[Any]) → AsyncIterator[Tuple[int, Union[Output, Exception]]]¶
Run ainvoke in parallel on a list of inputs,
yielding results as they complete.
Parameters
inputs (Sequence[Input]) – 
config (Optional[Union[RunnableConfig, Sequence[RunnableConfig]]]) – 
return_exceptions (bool) – 
kwargs (Optional[Any]) – 
Return type
AsyncIterator[Tuple[int, Union[Output, Exception]]]
Returns
A dict of named outputs. Should contain all outputs specified inChain.output_keys.
Return type
Dict[str, Any]
Notes
Deprecated since version langchain==0.1.0: Use ainvoke instead.
async aevaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict¶
Asynchronously evaluate Chain or LLM output, based on optional input and label.
Parameters
prediction (str) – The LLM or chain prediction to evaluate.
reference (Optional[str], optional) – The reference label to evaluate against.
input (Optional[str], optional) – The input to consider during evaluation.
**kwargs – Additional keyword arguments, including callbacks, tags, etc.
Returns
The evaluation results containing the score or value.
Return type
dict
async agenerate(input_list: List[Dict[str, Any]], run_manager: Optional[AsyncCallbackManagerForChainRun] = None) → LLMResult¶
Generate LLM result from inputs.
Parameters
input_list (List[Dict[str, Any]]) – 
run_manager (Optional[AsyncCallbackManagerForChainRun]) – 
Return type
LLMResult
async ainvoke(input: Dict[str, Any], config: Optional[RunnableConfig] = None, **kwargs: Any) → Dict[str, Any]¶
Default implementation of ainvoke, calls invoke from a thread.
The default implementation allows usage of async code even if
the runnable did not implement a native async version of invoke.
Subclasses should override this method if they can run asynchronously.
Parameters
input (Dict[str, Any]) – 
config (Optional[RunnableConfig]) – 
kwargs (Any) – 
Return type
Dict[str, Any]
# -> {"_type": "foo", "verbose": False, ...}
evaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict¶
Evaluate Chain or LLM output, based on optional input and label.
Parameters
prediction (str) – The LLM or chain prediction to evaluate.
reference (Optional[str], optional) – The reference label to evaluate against.
input (Optional[str], optional) – The input to consider during evaluation.
**kwargs – Additional keyword arguments, including callbacks, tags, etc.
Returns
The evaluation results containing the score or value.
Return type
dict
classmethod from_llm(llm: BaseLanguageModel, criteria: Optional[Union[Mapping[str, str], Criteria, ConstitutionalPrinciple]] = None, *, prompt: Optional[BasePromptTemplate] = None, **kwargs: Any) → CriteriaEvalChain[source]¶
Create a CriteriaEvalChain instance from an llm and criteria.
Parameters
llm (BaseLanguageModel) – The language model to use for evaluation.
criteria (CRITERIA_TYPE - default=None for "helpfulness") – 
The criteria to evaluate the runs against. It can be:
a mapping of a criterion name to its description
a single criterion name present in one of the default criteria
a single ConstitutionalPrinciple instance
prompt (Optional[BasePromptTemplate], default=None) – The prompt template to use for generating prompts. If not provided,
a default prompt template will be used.
**kwargs (Any) – Additional keyword arguments to pass to the LLMChain
constructor.
Returns
An instance of the CriteriaEvalChain class.
Return type
CriteriaEvalChain
Examples
>>> from langchain_openai import OpenAI
>>> from langchain.evaluation.criteria import LabeledCriteriaEvalChain