Package: math-stats-ml
Utility functions for"Mathematical Statistics with a View Toward Machine Learning", by John MyersA Python package for all utility functions used in the book and programming assignments. To install, do the usual:pip install math_stats_mlAll other materials are containedhere.Table of contents:gdsubmodule: Gradient descent utilitiesGD_outputclass: Container class for output of algorithmsGDfunction: Gradient descentSGDfunction: Stochastic gradient descentplot_gdfunction: plot the output of gradient descentplot_sgdfunction: plot the output of stochastic gradient descentgdsubmodule: Gradient descent utilitiesContains all utilities for the gradient descent algorithms used inthe book.GD_outputclass: Container class for output of algorithmsclassGD_outputA class holding the outputs of both the gradient descent (GD) and stochastic gradient descent (SGD) algorithms. All attributes below are optional and default toNone.AttributesNameTypeDescriptionparametersdictA dictionary containing the parameters of the objective function passed to eitherGDorSGD. Each value in the dictionary is a tensor whose zero-th dimension indexes the number of gradient steps.per_step_objectivestorch.TensorA tensor containing the running objective values, per gradient step.per_epoch_objectivestorch.TensorA tensor containing the running mean objective values, per epoch.epoch_step_numstorch.TensorA tensor containing the number of each gradient step on which an epoch begins/ends.grad_stepsiterAn iterable ranging from $0$ to one less than the total number of gradient steps. (This is convenient for plotting purposes.)lrfloatLearning rate.num_stepsintNumber of gradient steps to run the gradient descent (GD) algorithm.decay_ratefloatLearning rate decay.batch_sizeintMini-batch size for the stochastic gradient descent (SGD) algorithm.num_epochsintNumber of epochs for the stochastic gradient descent (SGD) algorithm.max_stepsintMaximum number of gradient steps after which we terminate the stochastic gradient descent (SGD) algorithm.type_flagstrEitherNoneorgd. In the latter case, indicates whether theGD_outputobject was obtained from the gradient descent (GD) algorithm.GDfunction: Gradient descentGD(J,init_parameters,lr,num_steps,decay_rate=0)Implementation of gradient descent. The notation below is intended to match the notation in the description inthe book.OutputThe output type is an object of theGD_outputclass.ParametersNameTypeDescriptionJfunctionObjective function to be minimized. The parameters of the function are either a single tensor or a dictionary of tensors (in the case that the parameters fall into natural groups, e.g., weights and biases).init_parameterstorch.TensorordictInitial parameters.lrfloatLearning rate, corresponding to $\alpha$ in the book.num_stepsintThe number of gradient steps after which the algorithm should halt, corresponding to $N$ in the book.decay_ratefloatLearning rate decay, corresponding to $\beta$ in the book. Defaults to0.SGDfunction: Stochastic gradient descentSGD(L,init_parameters,X,lr,batch_size,num_epochs,y=None,decay_rate=0,max_steps=-1,shuffle=True,random_state=None)Implementation of stochastic gradient descent. The notation and terminology below is intended to matchthe book.OutputThe output type is an object of theGD_outputclass.ParametersNameTypeDescriptionLfunctionLoss function for the algorithm. The call signature of the function is of the formL(parameters, x)orL(parameters, x, y), wherexis a feature vector andyis an (optional) ground truth label of a single instance in the dataset, andparametersis either a single parameter tensor or a dictionary of parameter tensors (in the case that the parameters fall into natural groups, e.g., weights and biases). We assume thatLis "vectorized," so that it may accept a design matrixXin place ofxand an entire vector of ground truth labels fory.init_parameterstorch.TensorordictInitial parameters.Xtorch.TensorDesign matrix. The rows are the feature vectors that are fed into the loss functionL.lrfloatLearning rate, corresponding to $\alpha$ in the book.batch_sizeintMini-batch size, corresponding to $k$ in the book.num_epochsintThe number of epochs after which the algorithm should halt, corresponding to $N$ in the book.ytorch.TensorVector of ground truth labels for the data in the design matrixX. Optional, defaults toNone.decay_ratefloatLearning rate decay, corresponding to $\beta$ in the book. Defaults to0.max_stepsintMaximum number of gradient steps after which the algorithm should halt. Defaults to-1, in which case the algorithm will complete allnum_epochsmany epochs.shuffleboolDetermines whether to shuffle the dataset before looping through an epoch. Defaults toTrue.random_stateintIf notNoneandshuffle=True, random seed to be passed totorch.manual_seed. Defaults toNone.plot_gdfunction: plot the output of gradient descentplot_gd(gd_output,w=5,h=4,plot_title=True,parameter_title=True,show_xlabel=True,xlabel='gradient steps',show_ylabel=True,ylabel='surprisal',alpha=1,color=None,ax=None)Later...plot_sgdfunction: plot the output of stochastic gradient descentplot_sgd(sgd_output,w=5,h=4,plot_title=True,parameter_title=True,show_step=True,show_epoch=True,show_xlabel=True,xlabel='gradient steps',show_ylabel=True,ylabel='cross entropy',legend=True,per_step_alpha=0.25,per_step_color=None,per_step_label='cross entropy per step',per_epoch_color=None,per_epoch_label='mean cross entropy per epoch',s=10,ax=None)Later...