Package: metric-eval
MetricEvalMetricEval: A framework that conceptualizes and operationalizes key desiderata of metric evaluation, in terms of reliability and validity. Please seeEvaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theoryfor more details.SummaryIn thisGithub repo, you will find the implementation of our framework, metric-eval, a python pkg for evaluation metrics analysis.@article{xiao2023evaluating,
  title={Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory},
  author={Xiao, Ziang and Zhang, Susu and Lai, Vivian and Liao, Q Vera},
  journal={arXiv preprint arXiv:2305.14889},
  year={2023}
}metirc-evalQuick StartInstall from PyPIpipinstallmetric-evalInstall from RepoClone the repository:gitclonegit@github.com:isle-dev/MetricEval.gitcdmetirc-evalInstall Dependencies:condacreate--namemetirc-evalpython=3.10
condaactivatemetirc-eval
pipinstall-rrequirements.txtUsagePlease refer to metric_eval/example.py for the detailed usage of metirc-eval, seeGithub repo. To interprete the evalaution results, seepaper.Import Moduleimportmetric_evalLoad DataExample data could be found in metric_eval/data/* , seeGithub repo. The data is a csv file with the following format:-test_id:arraycontainingidsoftestexamples-model_id:characterarraycontainingidsofmodels-additionalcolumnscontainingscoresoneachmetric(metricnameascolumnname)data=pd.read_csv("data/metric_scores_long.csv")data_2nd_run=pd.read_csv("data/metric_scores_long_2nd_run.csv")Metric StabilityThe function compares the average metric scores for models between the two data sets (data1 and data2). It calculates the Pearson correlation coefficient between the average metric scores from the first run (data) and the second run (data_2nd_run) for each metric.rel_cor=metric_eval.metric_stability(data,data_2nd_run)print(rel_cor)Metric ConsistencyMetric Consistency describes how the metric score fluctuates within a benchmark dataset, i.e., across data points. Metric consistency estimates (alphas) and the standard error of measurement (sems) of each metric given N randomly samples. Defult N (-1) refers all avaliable samples in the dataset.alphas,sems=metric_eval.metric_consistency(data,N=-1)print(alphas)print(sems)MTMM TableThe MTMM table presents a way to scrutinize whether observed metric scores act in concert with theory on what they intend to measure, when two or more constructs are measured using two or more methods. By convention, an MTMM table reports the pairwise correlations of the observed metric scores across raters and traits on the off-diagonals and the reliability coefficients of each score on the diagonals.metric_names=data.columns[2:14].tolist()trait=['COH','CON','FLU','REL']*3method=['Expert_1']*4+['Expert_2']*4+['Expert_3']*4# Create the MTMM_design DataFrameMTMM_design=pd.DataFrame({'trait':trait,'method':method,'metric':metric_names})MTMM_result=metric_eval.MTMM(data,MTMM_design,method='pearson')print(MTMM_result)Metric Concurrent ValidityThe function computes the concurrent validity for each criterion variable by calculating the Kendall's Tau correlation coefficient between the criterion variable and each metric.criterion=['Expert.1.COH','Expert.1.CON','Expert.1.FLU','Expert.1.REL']concurrent_val_table=metric_eval.concurrent_validity(data,criterion)metric_eval.print_concurrent_validity_table(concurrent_val_table)Get InvolvedWe welcome contributions from the community! Please create a GitHub issue if you have any questions, suggestions, requests or bug-reports. If you would like to contribute to the codebase, please create a pull request.ContactIf you have any questions, please contactZiang Xiaoat ziang dot xiao at jhu dot edu orSusu Zhangat szhan105 at illinois dot edu.