Package: yard
Author:Tamás NepuszThis is yet another Python package for drawing ROC curves. It also
lets you draw precision-recall, accumulation and concentrated ROC
(CROC) curves, sensitivity-specificity plots, F-score curves and calculate
the AUC (area under curve) statistics.  The significance of differences
between AUC scores can also be tested using paired permutation tests.Where to getyardyardhas two homes at the moment:ThePython package index. This page hosts the most recent stable
version ofyard. Sinceyardis under heavy development at the
moment, you might not get all the latest and greatest features ofyard, but you will most likely find a version here that should
not collapse even under exceptional circumstances.A page onGitHub. On this page you can follow the development ofyardas closely as possible; you can get the most recent
development version, file bug reports, or even fork the project
to start adding your own features.RequirementsYou will need the following tools to runyard:Python 3.7or later.Matplotlib, which is responsible for plotting the curves. If
you don’t haveMatplotlib, you can export the points of the
curves and then use an external plotting tool such asGNUPlotto plot them later.NumPyis an optional dependency; some functions will be
slightly faster if you haveNumPy, butyardshould work
fine without it as well.InstallationThe simplest way to installyardis by usingpip:$ pip install yardThis goes to thePython package indexpage, fetches the most recent
stable version and installs it, creating two scripts in your path:yard-aucfor AUC score calculation,yard-plotfor plotting andyard-significancefor significance testing.If you want the bleeding edge version, you should go to theGitHubpage, download a ZIP or .tar.gz file, extract it to some directory
and then run the following command:$ python setup.py installRunningyardyardworks with simple tabular flat files, and assumes that the first
row in each file is a header. Each row contains data related to a particular
test example. By default, the first column contains theexpectedoutcome
of a binary classifier for a given test example (i.e. whether the example is
positive or negative), while the remaining columns contain the output of
the probabilistic classifiers being tested on the test dataset. The
expected outcome must be positive for positive examples and zero or negative
for negative examples - this means that you may use either zeros and ones
or -1 and 1 for negative and positive test examples, respectively. The
outcomes of the classifiers may be in any range, but they are most frequently
in the interval [0; 1]. The following snippet shows an example input file:output  Method1 Method2 Method3
-1      0.2     0.3     0.6
-1      0.4     0.15    0.1
+1      0.7     0.2     0.9
+1      0.3     0.85    1.0Columns must be separated by tabs per default, but this can be overridden
with the-foption on the command line. The actual columns being used
can also be overridden using-c; for instance, if you have the expected
outcome in column 4 and the actual outcomes in columns 1-3, you may use-c4,1-3to specify that.Some usage examples are presented here; for more details, typeyard-plot--helporyard-significance--help.To show a ROC curve for an arbitrary number of classifiers where the expected
and actual outcomes are defined ininput_data.txt:$ yard-plot input_data.txtIf the actual outcomes are in columns 3-5, the expected outcome is in
column 6 and the columns are separated by semicolons:$ yard-plot -f ';' -c 6,3-5 input_data.txtTo plot precision-recall curves instead of ROC curves and also show the
AUC statistics:$ yard-plot -t pr --show-auc input_data.txtSupported curve types are:rocfor ROC curves (default),prfor
precision-recall curves,crocfor CROC curves,acfor accumulation
curves,sespefor sensitivity-specificity plots,fscorefor
F-score curves.To use a logarithmic X axis for the ROC curve and use the standard input
instead of a file:$ yard-plot -l xThe omission of an input filename instructsyard-plotto use the standard
input. You may have also used-in place of the filename to specify that.To save a ROC curve into a PDF file:$ yard-plot -o roc_curve.pdf input_data.txtYou may specify other formats as long as they are supported by Matplotlib:$ yard-plot -o roc_curve.ps input_data.txt
$ yard-plot -o roc_curve.png input_data.txtThe PDF backend also supports multiple plots in separate pages:$ yard-plot -t pr -t roc -t croc -o curves.pdf input_data.txtThe figure size, the DPI ratio and the font size can also be adjusted:$ yard-plot -o roc_curve.pdf --font-size 8 -s '8cm x 6cm' input_data.txtTo calculate the AUC statistics for multiple curves without plotting them:$ yard-auc -t pr -t roc input_data.txtTo test whether the ROC curves of multiple classifiers are significantly
different:$ yard-significance input_data.txtQuestions, commentsIf you have a question or comment aboutyardor you think you have
found a bug, feel free tocontact me.Acknowledgments and referencesThe inclusion of CROC curves and the statistical significance testing
was inspired by the following publication (which also provides more
details on what CROC curves are and why they are more useful than ROC
curves in many cases):A CROC Stronger than ROC: Measuring, Visualizing and Optimizing
Early Retrieval.
S. Joshua Swamidass, Chloe-Agathe Azencott, Kenny Daily and Pierre Baldi.Bioinformatics, April 2010, doi:10.1093/bioinformatics/btq140