Package: scorer
AuthorsProjectBuild StatusLicenseCode QualityCoverageN. CurtiScorerLinux/MacOS:Windows:Codacy:Codebeat:Machine Learning ScoresOptimization and extension of thePyCMlibrary with c++ support and network dependencies.Getting StartedPrerequisitesUsageLicenseContributionAuthorsAcknowledgmentsCitationGetting StartedTo build the project you can use theMakefileor theCMake(recommended).In the first case you can see the whole set of available rules just typingmakeormake exampleto build theexamplescript in c++ ormake pythonicto build the Cython version of the code.The builds scripts (build.shandbuild.ps1) allow an automatic build of the full library in Windows and Linux/MacOS systems using CMake.gitclonehttps://github.com/Nico-Curti/scorer.gitcdscorer
./build.shFor a manual installation of the Python version you can also run thesetupscript aspython-mpipinstall-rrequirements.txt
pythonsetup.pybuild_ext--inplaceor for developerspython-mpipinstall-rrequirements.txt
pythonsetup.pydevelop--userIf you are working withoutrootprivileges we suggest to use theShutscripts to easily install all the required dependencies.PrerequisitesTo build the c++ version of the code at least c++14 is required.
C++ supported compilers:Python version supported :The OMP version of the code is automatically available if OMP is installed and in this way all the metric computations are performed in parallel according to the generated dependency graph.The full list of prerequisites of Python version is listed inrequirements.Note: if you are interested on the visualization of the dependency graph and some other utilities to manage the evaluated metrics you can find all the scripts in theutilsfolder.
In this case you need to install also the following Python packages:networkx>=2.2
pandas>=0.23.3
matplotlib>=2.2.2
mpld3>=0.3UsageYou can use the libraries either inside your c++ project with a simple include of thescorer.hheader file or inside your Python application, where the scorer class is wrapped to give a simple dictionary in return.If you use the c++ versionPAY ATTENTIONto the parallel environment (open the parallel region before the run of scores computation as shown in the following example code).#include<array>#include<scorer.h>intmain(){constintn_labels=12;std::array<int,n_labels>y_true={{2,0,2,2,0,1,1,2,2,0,1,2}};std::array<int,n_labels>y_pred={{0,0,2,1,0,2,1,0,2,0,2,2}};scorerscore;#ifdef _OPENMP#pragma omp parallel shared (score){#endifscore.compute_score(y_true.data(),y_pred.data(),n_labels,n_labels);#ifdef _OPENMP}#endifscore.print();return0;}The same code can be rewritten in Python (also with different labels dtype) as#import numpy as npfromscorerimportScorery_true=['a','b','a','a','b','c','c','a','a','b','c','a']# np.array([2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2], dtype=np.int32)y_pred=['b','b','a','c','b','a','c','b','a','b','a','a']# np.array([0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2], dtype=np.int32)scorer=Scorer()scorer.evaluate(y_true,y_pred)print(scorer)The output is the same in both the executions and it should be something like this$python./scorer/example/main.py
Classes:0,1,2ConfusionMatrix:3.00.00.00.01.02.02.01.03.0

ClassStatistics:

TP(Truepositive/hit)3.0001.0003.000
FN(Falsenegative/miss/type2error)0.0002.0003.000
FP(Falsepositive/type1error/falsealarm)2.0001.0002.000
TN(Truenegative/correctrejection)7.0008.0004.000
POP(Population)12.00012.00012.000
P(Conditionpositiveorsupport)3.0003.0006.000
N(Conditionnegative)9.0009.0006.000
TOP(Testoutcomepositive)5.0002.0005.000
TON(Testoutcomenegative)7.00010.0007.000
TPR(Sensitivity/recall/hitrate/truepositiverate)1.0000.3330.500
TNR(Specificityortruenegativerate)0.7780.8890.667
PPV(Precisionorpositivepredictivevalue)0.6000.5000.600
NPV(Negativepredictivevalue)1.0000.8000.571
FNR(Missrateorfalsenegativerate)0.0000.6670.500
FPR(Fall-outorfalsepositiverate)0.2220.1110.333
FDR(Falsediscoveryrate)0.4000.5000.400
FOR(Falseomissionrate)0.0000.2000.429
ACC(Accuracy)0.8330.7500.583
F1(F1score-harmonicmeanofprecisionandsensitivity)0.7500.4000.545
F0.5(F0.5score)0.6520.4550.577
F2(F2score)0.8820.3570.517
MCC(Matthewscorrelationcoefficient)0.6830.2580.169
BM(Informednessorbookmakerinformedness)0.7780.2220.167
MK(Markedness)0.6000.3000.171
PLR(Positivelikelihoodratio)4.5003.0001.500
NLR(Negativelikelihoodratio)0.0000.7500.750
DOR(Diagnosticoddsratio)inf4.0002.000
PRE(Prevalence)0.2500.2500.500
G(G-measuregeometricmeanofprecisionandsensitivity)0.7750.4080.548
RACC(Randomaccuracy)0.1040.0420.208
ERR(Errorrate)0.1670.2500.417
RACCU(Randomaccuracyunbiased)0.1110.0430.210
J(Jaccardindex)0.6000.2500.375
IS(Informationscore)1.2631.0000.263
CEN(Confusionentropy)0.2500.4970.604
MCEN(Modifiedconfusionentropy)0.2640.5000.688
AUC(AreaUndertheROCcurve)0.8890.6110.583
dInd(Distanceindex)0.2220.6760.601
sInd(Similarityindex)0.8430.5220.575
DP(Discriminantpower)inf0.3320.166
Y(Youdenindex)0.7780.2220.167
PLRI(Positivelikelihoodratiointerpretation)PoorPoorPoor
NLRI(Negativelikelihoodratiointerpretation)GoodNegligibleNegligible
DPI(Discriminantpowerinterpretation)NonePoorPoor
AUCI(AUCvalueinterpretation)VeryGoodFairPoor
GI(Giniindex)0.7780.2220.167
LS(Liftscore)2.4002.0001.200
AM(Differencebetweenautomaticandmanualclassification)2.000-1.000-1.000
OP(Optimizedprecision)0.7080.2950.440
IBA(Indexofbalancedaccuracy)0.9510.1320.278
GM(G-meangeometricmeanofspecificityandsensitivity)0.8820.5440.577
Q(YuleQ-coefficientofcolligation)nan0.6000.333
AGM(Adjustedgeometricmean)0.8370.6920.607
MCCI(Matthewscorrelationcoefficientinterpretation)ModerateNegligibleNegligible
AGF(AdjustedF-score)0.9140.5400.552
OC(Overlapcoefficient)1.0000.5000.600
OOC(Otsuka-Ochiaicoefficient)0.7750.4080.548
AUPR(AreaunderthePRcurve)0.8000.4170.550
BCD(Bray-Curtisdissimilarity)0.0830.0420.042
ICSI(Individualclassificationsuccessindex)0.600-0.1670.100

OverallStatistics:

OverallACC0.583
OverallRACCU0.365
OverallRACC0.354
Kappa0.355
ScottPI0.344
GetAC10.389
BennettS0.375
KappaStandardError0.220
KappaUnbiased0.344
KappaNoPrevalence0.167
Kappa95%CI(0.7867531180381775,-0.0770757719874382)StandardError0.14295%CI(0.8622781038284302,0.30438855290412903)Chi-Squared6.600
Phi-Squared0.550
CramerV0.524
ResponseEntropy1.483
ReferenceEntropy1.500
CrossEntropy1.594
JointEntropy2.459
ConditionalEntropy0.959
MutualInformation0.524
KLDivergence0.094
LambdaB0.429
LambdaA0.167
Chi-SquaredDF4.000
OverallJ(1.225000023841858,0.40833333134651184)Hammingloss0.417
Zero-oneLoss5.000
NIR0.500
P-value0.387
OverallCEN0.464
OverallMCEN0.519
OverallMCC0.367
RR4.000
CBA0.478
AUNU0.694
AUNP0.667
RCI0.349
PearsonC0.596
F1Micro0.583
PPVMicro0.583
TPRMicro0.583
SOA1(Landis&Koch)Fair
SOA2(Fleiss)Poor
SOA3(Altman)Fair
SOA4(Cicchetti)Poor
SOA5(Cramer)RelativelyStrong
SOA6(Matthews)Weak
ACCMacro0.722
F1Macro0.565
TPRMacro0.611
PPVMacro0.567
CSI0.178If you are working with non-integer labels the Scorer object provides a usefull_label2numbers(encodein C++) function.
Its usage is mandatory inC++since the function signature requiresonlyinteger (int32) values, while thePythonversion automatically encode/decode the labels according to requirements.In theutilsfolder some utility scripts are reported.
Themake_script.pyallows to write the complete parallel version of the scorer class in c++.
In this way if you add a new operator to the library you can just run this code to update the scorer class parallelization strategies (computed independency_graphs.pyscript).Withview_stats.pyyou can see the complete graph of computed statistics with an HTML support for a more pleasant vision (e.g.graph)NOTE: usingview_stats.pyyou can find the errorObjectoftypendarrayisnotJSONserializableI suggest to use the following command to overcome thismpld3issue:python-mpipinstall--user"git+https://github.com/javadba/mpld3@display_fix"A full list of informations about the score functions and their meaning can be found in the amazingdocumentationof the original PyCM project.LicenseTheScorerpackage is licensed under the MIT "Expat" License.ContributionAny contribution is more than welcome :heart:. Just fill anissueor apull requestand we will check ASAP!Seeherefor further informations about how to contribute with this project.AuthorsNico Curtigit,uniboSee also the list ofcontributorswho participated to this project.AcknowledgmentsThanks goes to all contributors of this project:Daniele Dall'OlioCitationIf you have foundScorerhelpful in your research, please consider citing the project@misc{Scorer,
  author ={Nico Curti},
  title ={Scorer},
  year ={2019},
  publisher ={GitHub},
  howpublished ={\url{https://github.com/Nico-Curti/scorer}},}