Package: learning-curves
learning-curvesLearning-curves is Python module that extendssklearn's learning curve feature. It will help you visualizing the learning curve of your models.Learning curves give an opportunity to diagnose bias and variance in supervised learning models, but also to visualize how training set size influence the performance of the models (more informationshere).Such plots help you answer the following questions:Do I have enough data?What would be the best accuracy I would have if I had more data?Can I train my model with less data?Is my training set biased?Learning-curves will also help you fitting the learning curve to extrapolate and find the saturation value of the curve.Installation$ pip install learning-curvesTo create learning curve plots, first import the module withimport learning_curves.UsageIt is as simple as:lc = LearningCurve()
lc.get_lc(estimator, X, Y)Whereestimatorimplementsfit(X,Y)andpredict(X,Y).Output:On this example the green curve suggests that adding more data to the training set is likely to improve a bit the model accuracy. The green curve also shows a saturation near 0.96. We can easily fit a function to this curve:lc.plot(predictor="best")Output:Here we used a predefined function,pow, to fit the green curve. The R2 score is very close to 1, meaning that the fit is optimal. We can therefore use this curve to extrapolate the evolution of the accuracy with the training set size.This also tells us how many data we should use to train our model to maximize performances and accuracy.Add custom functions to fit the learning curveSuch function are calledPredictor. You can create aPredictorlike this:predictor = Predictor("myPredictor", lambda x,a,b : a*x + b, [1,0])Here we created a Predictor called "myPredictor" with the functiony(x) = a*x + b.
Because internally SciPyoptimize.curve_fitis called, a first guess of the parametersaandbare required. Here we gave them respective value 1 and 0.
You can then add thePredictorto theLearningCurveobject in two different ways:Pass thePredictorto theLearningCurveconstructor:lc = LearningCurve([predictor])Register thePredictorinside the predictors of theLearningCurveobject:lc.predictors.append(predictor)By default, 4Predictorsare instantiated:self.predictors = [
    Predictor("pow",        lambda x, a, b, c, d    : a - (b*x+d)**c,                [1, 1.7, -.5, 1e-3]),
    Predictor("pow_log",    lambda x, a, b, c, m, n : a - b*x**c + m*np.log(x**n),   [1, 1.7, -.5, 1e-3, 1e-3], True),
    Predictor("pow_log_2",  lambda x, a, b, c       : a / (1 + (x/np.exp(b))**c),    [1, 1.7, -.5]),
    Predictor("inv_log",    lambda x, a, b          : a - b/np.log(x),               [1, 1.6])
]Some predictors perform better (R2 score is closer to 1) than others, depending on the dataset, the model and the value to be preditected.Find the best PredictorTo find the Predictor that will fit best your learning curve, we can callget_predictorfunction:lc.get_predictor("best")Output:(pow [params:[   0.9588563    11.74747659   -0.36232639 -236.46115903]][score:0.9997458683912492])Plot the PredictorsYou can plot anyPredictors fitted function with theplotfunction:lc.plot(predictor="all")Output:Save and load LearningCurve instancesBecausePredictorcontains lambda functions, you can not simply save aLearningCurveinstance. One possibility is to only save the data points of the curve insidelc.recorder["data"]and retrieve then later on. But then the custom predictors are not saved. Therefore it is recommended to use thesaveandloadmethods:lc.save("path/to/save.pkl")
lc = LearningCurve.load("path/to/save.pkl")This internally uses thedilllibrary to save theLearningCurveinstance with all thePredictors.Find the best training set sizelearning-curveswill help you finding the best training set size by extrapolation of the best fitted curve:lc.plot(predictor="all", saturation="best")Output:The horizontal red line shows the saturation of the curve. The intersection of the two blue lines shows the best accuracy we can get, given a certainthreshold(see below).To retrieve the value of the best training set size:lc.threshold(predictor="best", saturation="best")Output:(0.9589, 31668, 0.9493)This tells us that the saturation value (the maximum accuracy we can get from this model without changing any other parameter) is0.9589. This value corresponds to an infinite number of samples in our training set! But with a threshold of0.99(this parameter can be changed withthreshold=x), we can have an accuracy0.9493if our training set contains31668samples.Note: The saturation value is always thesecond parameterof the function. Therefore, if you create your ownPredictor, place the saturation factor in second position (called a in the predefinedPredictors). If the function of your customPredictoris diverging, then no saturation value can be retrieven. In that case, passdiverging=Trueto the constructor of thePredictor. The saturation value will then be calculated considering themax_scalingparameter of thethreshold_custfunction (see documentation for details). You should set this parameter to the maximum number of sample you can add to your training set.DocumentationSome functions have theirfunction_name_custequivalent. Calling the function without the_custsuffix will internally call the function with the_custsuffix with default parameters (such as the data points of the learning curves). Thanks tokwargs, you can pass exactly the same parameters to both functions.Function/ClassParametersTypeDefaultDescriptionPredictor.__init__Instantiate aPredictorobject.namestrRequiredName of thePredictorfuncLambdaRequiredLambda function used for fitting of the learning curveguessListRequiredStarting parameters used for fitting the curvedivergingBoolFalseIf the function is diverging, set diverging to True. If the function is converging, then the first parameter of the function has to be the convergence value.LC.__init__Instantiate aLearningCurveobject.predictorsListemptyPredictors to add to theLearningCurveobjectscoringCallabler2_scoreScoring function used to evaluate the fits of the learning curveLC.get_lcCompute and plot the learning curveestimatorObjectRequiredModel (any object implementingfit(X,Y)andpredict(X,Y)methods)XarrayRequiredX numpy array used for predictionYarrayRequiredY numpy array used for predictionLC.trainCompute the learning curve of an estimator over a dataset. Returns an object that can then be passed to plot_lc functionXarrayRequiredX numpy array used for predictionYarrayRequiredY numpy array used for predictiontrain_sizesListPredefinedList of training size used for calculating the learning curve. Can be a  list of floats between 0 and 1 (assumed to be percentages)  or a list of integers (assumed to be number of values)test_sizeint/float0.2percentage / value of the test set sizen_splitsint3Number of splits used for cross validationverboseint1The higher, the more verbosen_jobsint-1Number of workers. -1 sets to maximum possible. See sklearn.LC.get_predictorGet the first predictor with matching {name}. Returns None if no predictor matches.predstr, List(str), Predictor, List(Predictor)RequiredName of the predictor(s). Can be "all" or "best" or even Predictor(s).LC.fit_allFit a curve with all the Predictors and retrieve score if y_pred is finite. Returns an array of predictors with the updated params and score.xArrayRequired1D array (list) representing the training sizesyArrayRequired1D array (list) representing the scoresLC.fit_all_custSame asfit_allx,yArrayRequiredSeefit_allpredictorsList(Predictors)RequiredThe predictors to use for the fitting.LC.fitFit a curve with a predictor and retrieve score (default:R2) if y_pred is finite. Returns the predictor with the updated params and score.predictorPredictorRequiredThe predictor to use for fitting the learning curvexArrayRequired1D array (list) representing the training sizesyArrayRequired1D array (list) representing the scoresLC.thresholdFind the training set size providing the highest accuracy up to a predefined threshold.  P(x) = y and for x -> inf, y -> saturation value.   This method approximates x_thresh such as P(x_thresh) = threshold * saturation value. Returns (saturation value, x_thresh, y_thresh)Pstr, List(str), Predictor, list(Predictor"best"The predictor to use for the calculation of the saturation value.kwargsdictEmtpySeeLC.threshold_custfor optional parameters.LC.threshold_custSeethresholdPstr, List(str), Predictor, list(Predictor"best"The predictor to use for the calculation of the saturation value.xarrayRequiredX values (training set sizes)thresholdfloat [0.0, 1.0]0.99Percentage of the saturation value to use for the calculus of the best training set size.max_scalingint3Order of magnitude added to the order of magnitude of the maximum train set size.  IfPredictoris diverging, the total order of magnitude is used for the calculation of the saturation value. Generally, a value of3is enough. A value bigger than5may lead toMemoryException.forceBoolFalseSet toTruenot to raise aValueErrorifmax_scalingis > 5LC.get_scaleReturns the order of magnitude of the mean of an arrayvalarrayRequiredLC.best_predictorReturns the best predictor of theLearningCurvedata for the test score learning curvekwargsdictEmptySeeLC.best_predictor_custfor optional parameters.LC.best_predictor_custSeebest_predictorpredictorsList(Predictors)RequiredPredictors to considerxArrayRequired1D array (list) representing the training sizesyArrayRequired1D array (list) representing the scoresLC.best_predictor_custFind the best predictor for a custom learning curvexArrayRequired1D array (list) representing the training sizesyArrayRequired1D array (list) representing the scoresfitBoolTruePerform a fit of thePredictors before classifyingLC.plotPlot the training and test learning curve of theLearningCurvedata, and optionally a fitted functionpredictorstr, List(str), Predictor, List(Predictor)NonePredictors to use for plotting the fitted curve. Can also be "all" and "best".kwargsdictNoneSeeLC.plot_custfor optional parametersLC.plot_custPlot any training and test learning curve, and optionally a fitted function.train_sizesarrayRequiredData points of the learning curve. The output ofLC.traincan be used as parameters of this functiontrain_scores_meanarrayRequiredSeetrain_sizesparametertrain_scores_stdarrayRequiredSeetrain_sizesparametertest_scores_meanarrayRequiredSeetrain_sizesparametertest_scores_stdarrayRequiredSeetrain_sizesparameterpredictorarrayRequiredSeeLC.plotylim2-upleNoneLimits of the y axis of the plotfigsize2-upleNoneSize of the figuretitlestrNoneTitle of the plotsaturationstr, List(str), Predictor, List(Predictor)NonePredictors to consider for displaying the saturation on the plot.kwargsdictEmptySeeplot_saturationfor optional parametersLC.plot_fitted_curveAdd to a matplotlib figure a fitted curveaxaxeRequiredFigure where the curve will be printedPPredictorRequiredPredictorto use for the computing of the curvexarrayRequired1D array (list) representing the training sizesscoresBoolTrueShow the score of thePredictorsLC.saveSave theLearningCurveobject in disk usingdillpathPath/strlc_data.pklPath to the file where the save will be doneLC.loadLoad aLearningCurveobject from disk.pathPath/strlc_data.pklPath to the file where the save is locatedLC.plot_saturationAdd saturation lines to a plot.axmatplotlib axRequiredfigure to usePPredictorRequiredPredictorto consideralphafloat1alpha applied to the lineslwfloat1.3matplotlib lw parameter applied to the lines.LC.get_unique_listReturn a list of unique predictors.predictorsList(Predictor)RequiredList ofPredictors to consider.