Package: weightwatcher
WeightWatcher(WW) is an open-source, diagnostic tool for analyzing Deep Neural Networks (DNN), without needing access to training or even test data.  It is based on theoretical research into Why Deep Learning Works, based on our Theory of Heavy-Tailed Self-Regularization (HT-SR).  It uses ideas from Random Matrix Theory (RMT), Statistical Mechanics, and Strongly Correlated Systems.It can be used to:analyze pre/trained pyTorch, Keras, DNN models (Conv2D and Dense layers)monitor models, and the model layers, to see if they are over-trained or over-parameterizedpredict test accuracies across different models, with or without training datadetect potential problems when compressing or fine-tuning pretrained modelslayer warning labels: over-trained; under-trainedQuick LinksPlease seeour latest talk from the Sillicon Valley ACM meetupJoin theDiscord ServerFor a deeper dive into the theory, seeour latest talk at ENSand some of the most recent Podcasts:Practical AIThe Prompt DeskMore details and demos can be found on theCalculated Content BlogAnd in the notebooks provided in theexamplesdirectoryInstallation:  Version 0.7.5pipinstallweightwatcherif this fails tryCurrent TestPyPI  Version 0.7.5python3-mpipinstall--index-urlhttps://test.pypi.org/simple/--extra-index-urlhttps://pypi.org/simpleweightwatcherUsageimportweightwatcheraswwimporttorchvision.modelsasmodelsmodel=models.vgg19_bn(pretrained=True)watcher=ww.WeightWatcher(model=model)details=watcher.analyze()summary=watcher.get_summary(details)It is as easy to run and generates a pandas dataframe with details (and plots) for each layerandsummarydictionary of generalization metrics{'log_norm':2.11,'alpha':3.06,'alpha_weighted':2.78,'log_alpha_norm':3.21,'log_spectral_norm':0.89,'stable_rank':20.90,'mp_softrank':0.52}Advanced UsageThewatcherobject has several functions and analysis features described belowNotice the min_evals setting:  the power law fits need at least 50 eigenvalues to make sense
but the describe and other methods do notwatcher.analyze(model=None,layers=[],min_evals=50,max_evals=None,plot=True,randomize=True,mp_fit=True,pool=True,savefig=True):...watcher.describe(self,model=None,layers=[],min_evals=0,max_evals=None,plot=True,randomize=True,mp_fit=True,pool=True):...watcher.get_details()watcher.get_summary(details)orget_summary()watcher.get_ESD()...watcher.distances(model_1,model_2)PEFT / LORA models  (experimental)To analyze an PEFT / LORA fine-tuned model, specify the peft option.peft = True:  Forms the BA low rank matric and analyzes the delta layers, with 'lora_BA" tag in namedetails = watcher.analyze(peft='peft_only')peft = 'with_base':  Analyes the base_model, the delta, and the combined layer weight matrices.details = watcher.analyze(peft=True)The base_model and fine-tuned model must have the same layer names.  And weightwatcher will ignore layers that do not share the same name.
Also,at this point, biases are not considered.  Finally, both models should be stored in the same format (i.e safetensors)Note: If you want to select by layer_ids, you must first run describe(peft=False), and then selectboththe lora_A and lora_B layersUsage: Base ModelPloting and Fitting the Empirical Spectral Density (ESD)WW creates plots for each layer weight matrix to observe how well the power law fits workdetails=watcher.analyze(plot=True)For each layer, WeightWatcher plots the ESD--a histogram of the eigenvalues of the layer correlation matrixX=WTW.  It then fits the tail of ESD to a (Truncated) Power Law, and plots these fits on different axes. The summary metrics (above) characterize the Shape and Scale of each ESD.  Here's an example:Generally speaking, the ESDs in the best layers, in the best DNNs can be fit to a Power Law (PL), with PL exponentsalphacloser to2.0.
Visually, the ESD looks like a straight line on a log-log plot (above left).Generalization MetricsThe goal of the WeightWatcher project is find generalization metrics that most accurately reflect observed test accuracies, across many different models and architectures, for pre-trained models and models undergoing training.Our HTSR theorysays that well trained, well correlated layers should be signficantly different from the MP (Marchenko-Pastur) random bulk, and specifically to be heavy tailed. There are different layer metrics in WeightWatcher for this, including:rand_distance: the  distance in distribution from the randomized layeralpha: the slope of the tail of the ESD, on a log-log scalealpha-hatoralpha_weighted: a scale-adjusted form ofalpha(similar to the alpha-shatten-Norm)stable_rank: a norm-adjusted measure of the scale of the ESDnum_spikes: the number of spikes outside the MP bulk regionmax_rand_eval: scale of the random noise etcAll of these attempt to measure how on-random and/or non-heavy-tailed the layer ESDs are.Scale Metricslog Frobenius norm :log_spectral_norm:stable_rank:mp_softrank:Shape Metricsalpha:Power Law (PL) exponent(Truncated) PL quality of fitD:(the Kolmogorov Smirnov Distance metric)(advanced usage)TPL : (alpha and Lambda) Truncated Power Law FitE_TPL : (alpha and Lambda) Extended Truncated Power Law FitScale-adjusted Shape Metricsalpha_weighted:log_alpha_norm: (Shatten norm):Direct Correlation MetricsThe random distance metric is a new, non-parameteric approach that appears to work well in early testing.See this recent blog postrand_distance:Distance of layer ESD from the ideal RMT MP ESDThere re also related metrics, including the new'ww_maxdist''ww_softrank'Misc DetailsN, M:  Matrix or Tensor Slice Dimensionsnum_spikes:  number of spikes outside the bulk region of the ESD, when fit to an MP distributionnum_rand_spikes:  number of Correlation Trapsmax_rand_eval: scale of the random noise in the layerSummary Statistics:The layer metrics are averaged in thesummarystatistics:Get the average metrics, as asummary(dict), from the given (or current)detailsdataframedetails=watcher.analyze(model=model)summary=watcher.get_summary(model)or justsummary=watcher.get_summary()The summary statistics can be used to gauge the test error of a series of pre/trained models, without needing access to training or test data.averagealphacan be used to compare one or more DNN models with different hyperparemeter settingsθ, when depth is not a driving factor (i.e transformer models)averagelog_spectral_normis useful to compare models of different depthsLat a coarse grain levelaveragealpha_weightedandlog_alpha_normare suitable for DNNs of differing hyperparemetersθand depthsLsimultaneously. (i.e CV models like VGG and ResNet)Predicting the Generalization ErrorWeightWatcher (WW) can be used to compare the test error for a series of models, trained on the similar dataset, but with different hyperparametersθ, or even different but related architectures.Our Theory of HT-SR predicts that models with smaller PL exponentsalpha, on average, correspond to models that generalize better.Here is an example of thealpha_weightedcapacity metric for all the current pretrained VGG models.Notice: wedid not peekat the ImageNet test data to build this plot.This can be reproduced with the Examples Notebooks forVGGand also forResNetDetecting signs of Over-Fitting and Under-FittingWeightWatcher can help you detect the signatures of over-fitting and under-fitting in specific layers of a pre/trained Deep Neural Networks.WeightWatcher will analyze your model, layer-by-layer, and show you where these kind of problems may be lurking.Correlation TrapsTherandomizeoption lets you compare the ESD of the layer weight matrix (W) to the ESD of its randomized form.
This is good way to visualize the correlations in the true ESD, and detect signatures of over- and under-fittingdetails=watcher.analyze(randomize=True,plot=True)Fig (a) is well trained; Fig (b) may be over-fit.That orange spike on the far right is the tell-tale clue; it's caled aCorrelation Trap.ACorrelation Trapis characterized by Fig (b); here the actual (green) and random (red) ESDs look almost identical, except for a small shelf of correlation (just right of 0). And random (red) ESD, the largest eigenvalue (orange) is far to the right of and seperated from the bulk of the ESD.When layers look like Figure (b) above, then they have not been trained properly because they look almost random, with only a little bit of information present. And the information the layer learned may even be spurious.Moreover, the metricnum_rand_spikes(in thedetailsdataframe) contains the number of spikes (or traps) that appear in the layer.TheSVDSharpnesstransform can be used to remove Correlation Traps during training (after each epoch) or after training usingsharpemed_model=watcher.SVDSharpness(model=...)Sharpening a model is similar to clipping the layer weight matrices, but uses Random Matrix Theory to do this in a more principle way than simple clipping.Early StoppingNote:This is experimental but we have seen some success hereThe WeightWatcheralphametric may be used to detect when to apply early stopping.  When the averagealpha(summary statistic) drops below2.0, this indicates that the model may be over-trained and early stopping is necesary.Below is an example of this, showing training loss and test lost curves for a small Transformer model, trained from scratch, along with the averagealphasummary statistic.We can see that as the training and test losses decrease, so doesalpha. But when the test loss saturates and then starts to increase,alphadrops below2.0.Note:this only work for very well trained models, where the optimalalpha=2.0is obtainedAdditional FeaturesThere are many advanced features, described belowFilteringfilter by layer typesww.LAYER_TYPE.CONV2D|ww.LAYER_TYPE.CONV2D|ww.LAYER_TYPE.DENSEasdetails=watcher.analyze(layers=[ww.LAYER_TYPE.CONV2D])filter by layer ID or namedetails=watcher.analyze(layers=[20])Calculationsminimum, maximum number of eigenvalues of the layer weight matrixSets the minimum and maximum size of the weight matrices analyzed.
Setting max is useful for a quick debugging.details=watcher.analyze(min_evals=50,max_evals=500)specify the Power Law fitting proceedureTo replicate results using TPL or E_TPL fits, use:details=watcher.analyze(fit='PL'|'TPL'|'E_TPL')Thedetailsdataframe will now contain two quality metrics, and for each layer:alpha: basically (but not exactly) the same PL exponent as before, useful foralpha > 2.0Lambda: a new metric, now useful when the (TPL)alpha < 2.0(The TPL fits correct a problem we have had when the PL fits over-estimatealphafor TPL layers)As with thealphametric, smallerLambdaimplies better generalization.VisualizationSave all model figuresSaves the layer ESD plots for each layerwatcher.analyze(savefig=True,savefig='/plot_save_directory')generating 4 files per layerww.layer#.esd1.png
ww.layer#.esd2.png
ww.layer#.esd3.png
ww.layer#.esd4.pngNote:additional plots will be saved whenrandomizeoption is usedfit ESDs to a Marchenko-Pastur (MP) distrbutionThemp_fitoption tells WW to fit each layer ESD as a Random Matrix as a Marchenko-Pastur (MP) distribution, as described in our papers on HT-SR.details=watcher.analyze(mp_fit=True,plot=True)and reports thenum_spikes,mp_sigma,andmp_sofrankAlso works for randomized ESD and reportsrand_num_spikes,rand_mp_sigma,andrand_mp_sofrankfetch the ESD for a specific layer, for visualization or additional analysiswatcher.analyze()esd=watcher.get_ESD()Model Analysisdescribe a modelDescribe a model and report thedetailsdataframe, without analyzing itdetails=watcher.describe(model=model)comparing two modelsThe new distances method reports the distances between two models, such as the norm between the initial weight matrices and the final, trained weight matricesdetails=watcher.distances(initial_model,trained_model)Compatabilitycompatability with version 0.2.xThe new 0.4.x version of WeightWatcher treats each layer as a single, unified set of eigenvalues.
In contrast, the 0.2.x versions split the Conv2D layers into n slices, one for each receptive field.
Thepool=Falseoption provides results which are back-compatable with the 0.2.x version of WeightWatcher,
(which used to be calledww2x=True) with details provide for each slice for each layer.
Otherwise, the eigenvalues from each slice of th3 Conv2D layer are pooled into one ESD.details=watcher.analyze(pool=False)RequirementsPython 3.7+Frameworks supportedTensorflow 2.x / KerasPyTorch 1.xHuggingFaceNote:  the current version requires both tensorflow and torch; if there is demand, this will be updates to make installation easier.Layers supportedDense / Linear / Fully Connected (and Conv1D)Conv2DTips for First Time UsersOn using WeighWtatcher for the first time.  I recommend selecting at least one trained model, and running `weightwatcher` with all analyze options enabled, including the plots.  From this, look for:if the layers ESDs are well formed and heavy tailedif any layers are nearly random, indicating they are not well trainedif all the power law a fits appear reasonable, andxminis small enough that the fit captures a reasonable section of the ESD tailMoreover, the Power Laws and alpha fit only work well when the ESDs are both heavy tailedandcan be easily fit to a single power law.
Occasionally the power law and/or alpha fits don't work.  This happens whenthe ESD is random (not heavy tailed),alpha > 8.0the ESD is multimodal (rare, but does occur)the ESD is heavy tailed, but not well described by a single power law.  In these cases, sometimesalphaonly fits the thevery lastpart of the tail, and istoolarge. This is easily seen on the Lin-Lin plotsIn any of these cases, I usually throw away results wherealpha > 8.0because they are spurious. If you suspect your layers are undertrained, you have to look both atalphaand a plot of the ESD itself (to see if it is heavy tailed or just random-like).How to ReleasePublishing to the PyPI repository:# 1. Check in the latest code with the correct revision number (__version__ in __init__.py)viweightwatcher/__init__.py# Increse release number, remove -dev to revision numbergitcommit# 2. Check out latest version from the repo in a fresh directorycd~/temp/
gitclonehttps://github.com/CalculatedContent/WeightWatchercdWeightWatcher/# 3. Use the latest version of the toolspython-mpipinstall--upgradesetuptoolswheeltwine# 4. Create the packagepythonsetup.pysdistbdist_wheel# 5. Test the packagetwinecheckdist/*# 7. Upload the package to TestPyPI firsttwineupload--repositorytestpypidist/*# 8. Test the TestPyPI installpython3-mpipinstall--index-urlhttps://test.pypi.org/simple/weightwatcher
...# 9. Upload to actual PyPItwineuploaddist/*# 10. Tag/Release in github by creating a new release (https://github.com/CalculatedContent/WeightWatcher/releases/new)LicenseApache License 2.0Academic Presentations and Media AppearancesThis tool is based on state-of-the-art research done in collaboration with UC Berkeley:WeightWatcher has been featured in top journals like JMLR and Nature:#### Latest papers and talks[SETOL: A Semi-Empirical Theory of (Deep) Learning] (in progress)Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metricsEvaluating natural language processing models with robust generalization metrics that do not need access to any training or testing data(Nature paper) Predicting trends in the quality of state-of-the-art neural networks without access to training or testing dataRepo for Nature paper(JMLR in press) Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for LearningTraditional and Heavy Tailed Self Regularization in Neural Network ModelsNotebook for above 2 papers (https://github.com/CalculatedContent/ImplicitSelfRegularization)ICML 2019 Theoretical Physics Workshop PaperHeavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural NetworksNotebook for paper (https://github.com/CalculatedContent/PredictingTestAccuracies)Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behaviorand has been presented at Stanford, UC Berkeley, KDD, etc:NERSC Summer 2018UC Berkeley/ICSI 12/13/2018Institute for Pure & Applied Mathematics (IPAM)Physics Informed Machine LearningTalk at Stanford ICME 2020Talk at UCL (UK) 2022KDD2019 WorkshopKDD 2019 Workshop: Statistical Mechanics Methods for Discovering
Knowledge from Production-Scale Neural NetworksKDD 2019 Workshop: SlidesWeightWatcher has also been featured at local meetups and many popular podcastsPopular Popdcasts and BlogsThis Week in MLData Science at Home PodcastAggregate Intellect VLogRebellion Research VLogRebellion Research BLogLightOn AI MeetupThe Sillicon Valley ACM meetupApplied AI CommunityPractical AILatest Results2021 Short PresentationsMLC Research Jam  March 2021PyTorch2021 Poster  April 2021Recent talk(s) by Mike Mahoney, UC BerekelyIARAI, the Institute for Advanced Research in Artificial IntelligenceExperimental / Most Recent version    (not ready yet)You may install the latest / Trunk from testpypipython3 -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple weightwatcherThe testpypi version usually has the most recent updates, including experimental methods and bug fixes.
But pypi has changed the way it handles testpypi requiring non-testpypi dependencies.
e.g., torch and tensorflow fail on testpypiIf you have them installed already in your env, you're fine.
Otherwise, you need to install them firstContributorsCharles H Martin, PhDCalculation ConsultingSerena PengChristopher HinrichsConsulting PracticeCalculation Consulting homepageCalculated Content Blog