Package: lemon-explain
LEMON: Explainable Entity MatchingLEMON is an explainability method that addresses the unique challenges of explaining entity matching models.Installationpipinstalllemon-explainorpipinstalllemon-explain[storage]# Save and load explanationspipinstalllemon-explain[matchers]# To run matchers in lemon.utilspipinstalllemon-explain[all]# All dependenciesUsageimportlemon# You need a matcher that follows this api:defpredict_proba(records_a,records_b,record_id_pairs):...# predict probabilities / confidence scoresreturnprobaexp=lemon.explain(records_a,records_b,record_id_pairs,predict_proba)# exp can be visualized in a Jupyter notebook or saved to a json fileexp.save("explanation.json")See the example notebookDocumentationlemon.explain()lemon.explain(
    records_a: pd.DataFrame,
    records_b: pd.DataFrame,
    record_id_pairs: pd.DataFrame,
    predict_proba: Callable,
    *,
    num_features: int = 5,
    dual_explanation: bool = True,
    estimate_potential: bool = True,
    granularity: str = "counterfactual",
    num_samples: int = None,
    token_representation: str = "record-bow",
    token_patterns: Union[str, List[str], Dict] = "[^ ]+",
    explain_attrs: bool = False,
    attribution_method: str = "lime",
    show_progress: bool = True,
    random_state: Union[int, np.random.Generator, None] = 0,
    return_dict: bool = None,
) -> Union[MatchingAttributionExplanation, Dict[any, MatchingAttributionExplanation]]:Parametersrecords_a: pd.DataFrameRecords from data source a.records_b: pd.DataFrameRecords from data source b.record_id_pairs: pd.DataFrameWhich record pairs to explain.
Must be a pd.DataFrame with columns"a.rid"and"b.rid"that reference the index ofrecords_aandrecords_brespectively.predict_proba: CallableMatcher function that predicts the probability of match.
Must accept three arguments:records_a,records_b, andrecord_id_pairs.
Should return array-like (list, np.ndarray, pd.Series, ...) of floats between 0 and 1 - the predicted probability that a record pair is a match - for all record pairs described byrecord_id_pairsin the same order.num_features: int, default = 5The number of features to select for the explanation.dual_explanation: bool, default = TrueWhether to use dual explanations or not.estimate_potential: bool, default = TrueWhether to estimate potential or not.granularity: {"tokens", "attributes", "counterfactual"}, default = "counterfactual"The granularity of the explanation.
For more info on"counterfactual"granularity see our paper.num_samples: int, default = NoneThe number of neighborhood samples to use.
If None a heuristic will automatically pick the number of samples.token_representation: {"independent", "shared-bow", "record-bow"}, default = "record-bow"Which token representation to use.independent: All tokens are unique.shared-bow: Bag-of-words representation shared across both recordsrecord-bow: Bag-of-words representation per individual recordtoken_patterns: str, List[str], or Dict, default ="[^ ]+"Regex patterns for valid tokens in strings.
A single string will be interpreted as a regex pattern and all strings will be tokenized into non-overlapping matches of this pattern.
You can specify a list of patterns to tokenize into non-overlapping matches of any pattern.
For fine-grained control of how different parts of records are tokenized you can provide a dictionary with keys on the format("a" or "b", attribute_name, "attr" or "val")and values that are lists of token regex patterns.explain_attrs: bool, default = FalseWhether to explain attribution names or not.
If True,predict_probashould accept the keyword argumentattr_strings- a list that specifies what strings to use as attributes for each prediction.
Each list element is on the format {("a" or "b", record_column_name): attr_string}.attribution_method: {"lime", "shap"}, default = FalseWhich underlying method to use contribution estimation.
Note that in order to use shapestimate_potentialmust be False and the shap package must be installed.show_progress: bool, default = TrueWhether to show progress or not. This is passed topredict_probaif it accepts this parameter.return_dict: bool, default = NoneIf True a dictionary of explanations will be returned where the keys are labels from the index ofrecord_id_pairs.
If False a single explanation will be returned (an exception is raised iflen(record_id_pairs) > 1).
If None it will return a single explanation iflen(record_id_pairs)and a dictionary otherwise.Returnslemon.MatchingAttributionExplanationisntance or anDict[any, lemon.MatchingAttributionExplanation],
depending on the input to thereturn_dictparameter.lemon.MatchingAttributionExplanationAttributesrecord_pair: pd.DataFramestring_representation: Dict[Tuple, Union[None, str, TokenizedString]],attributions: List[Attribution],prediction_score: floatdual: boolmetadata: Dict[str, any]Methodssave(path: str = None) -> Optional[Dict]Save the explanation to a json file.
If path is not specified a json-serializable dictionary will be returned.
Requires pyarrow to be installed (pip install lemon-explain[storage]).static load(path: Union[str, Dict]) -> MatchingAttributionExplanationLoad an explanation from a json file.
Instead of a path, one can instead provide a json-serializable dictionary.
Requires pyarrow to be installed (pip install lemon-explain[storage]).lemon.AttributionAttributesweight: floatpotential: Optional[float]positions: List[Union[Tuple[str, str, str, Optional[int]]]]name: Optional[str]