Package: trialML
trialML: Preparing a machine learning model for a statistical trialtrialMLis apythonpackage designed to help researchers and practioners prepare their machine learning models for a statistical trial to establish a lower-bound on model performance. Specifically, this package helps to calibrating the operating threshold of a binary classifier and carry out a power analysis. A more formal description of these techniques can be found in the correspondingarXiv paper.FeaturesThe main modules fromtrialMLcan be called in with one line of code:from trialML import trial, power. Their key methods are outlined below, and are described with more detail by the docstrings (e.g.help('trialML.trial.classification')).trial.classification(gamma, m, alpha): determine optimal threshold and calculate power of future trialstatistic(y, s, threshold, pval=..): return the performance measure for a given threshold (and possibly p-value from null hypothesis)learn_threshold(y, s, method='..'): calibrate the opearting threshold to obtain at leastgamma1-alpha% of the time.calculate_power(spread, n_trial, threshold): estimate power for a given trial sample size and null hypothesis margin (spread). Threshold can be provided to estimate percent of samples that are class-specific.power.twosided_classification(m1, m2, alpha): estimate performance measure and power range (confidence interval) for two performance measures:m1andm2.set_threshold(y, s, gamma1): Set the threshold to get a performance level ofgamma1for the first performance measurem1.statistic_CI(y, s, threshold): Get the (1-alpha) confidence interval for the empirical values ofm1andm2.statistic_pval(y, s, gamma0): Get the p-value on trial data for a given null hypothesis.How to useThe code block below shows how to calibrate a classifier for toy example of a classifier trained on random data. For more detailed examples wiht real data, please see thetutorialsfolder.# Load modulesimportnumpyasnpfromsklearn.linear_modelimportLogisticRegressionfromtrialML.trialimportclassification## (1) Train a model and obtain scores on a test setnp.random.seed(1)n,p=150,10k1,k2=50,100X,y=np.random.randn(n,p),np.random.binomial(1,0.5,n)X_train,y_train,X_test,y_test=X[:k1],y[:k1],X[k1:k2],y[k1:k2]mdl=LogisticRegression(penalty='none',solver='lbfgs')mdl.fit(X=X_train,y=y_train)# test set scoress_test=mdl.predict_proba(X_test)[:,1]s_test=np.log(s_test/(1-s_test))# logit transform## (2) Calibrate operating threshold to achieve 50% sensitivity, 95% of the timegamma=0.5# performance measure targetalpha=0.05# type-I error rate for threshold selectionm='sensitivity'# currently supports sensitivity/specificity/precision# Set up statistical toolcalibration=classification(gamma=gamma,alpha=alpha,m=m)# Learn thresholdcalibration.learn_threshold(y=y_test,s=s_test,method='percentile',n_bs=1000,seed=1)# Observe test-set performancegamma_hat_test=calibration.statistic(y=y_test,s=s_test,threshold=calibration.threshold_hat)print('Empirical sensitivity on test-set:%0.1f%%'%(100*gamma_hat_test))## (3) Estimate power for trial dataX_trial,y_trial=X[k1:],y[k1:]n_trial=len(X_trial)gamma0=0.45spread=gamma-gamma0calibration.calculate_power(spread,n_trial,threshold=calibration.threshold_hat)print('Expected trial power for a%0.1f%%margin is at least%0.1f%%'%(100*spread,100*calibration.power_hat))## (4) Run trials_trial=mdl.predict_proba(X_trial)[:,1]s_trial=np.log(s_trial/(1-s_trial))# logit transformgamma_trial,pval_trial=calibration.statistic(y=y_trial,s=s_trial,gamma0=gamma0,threshold=calibration.threshold_hat)print('Trial sensitivity:%0.1f%%, trial null-hypothesis:%0.1f%%, trial p-value:%0.5f'%(100*gamma_trial,100*gamma0,pval_trial))How to installtrialMLis available onPyPIcan be installed in one line:pip install trialML.