Package: alethiometer
zero-cost-proxiesIndependent ZC proxies only for testing on it.Modified and simplified fromforesight repo, fix some bugs in model output, remove some unwanted code snippets.Supported zc-metrics are:=========================================================
= grad_norm,                                            =
=-------------------------------------------------------=
= grasp,                                                =
=-------------------------------------------------------=
= snip,                                                 =
=-------------------------------------------------------=
= synflow,                                              =
=-------------------------------------------------------=
= nwot, (NASWOT)                                        =
=       [nwot, nwot_Kmats]                              =
=-------------------------------------------------------=
= lnwot, (Layerwise NASWOT)                             =
=       [lnwot, lnwot_Kmats]                            =
=-------------------------------------------------------=
= nwot_relu, (original RELU based NASWOT metric)        =
=       [nwot_relu, nwot_relu_Kmats]                    =
=-------------------------------------------------------=
= zen,                                                  =
=      Your network need have attribute fn:             =
=         `forward_before_global_avg_pool(inputs)`      =
=      to calculate zenas score                         =
=      (see sample code in tests/test_zc.py)            =
=-------------------------------------------------------=
= tenas,                                                =
=      must work in `gpu` env,                          =
=      might encouter bug on `cpu`.                     =
=      also contains metrics:                           =
= ntk,                                                  =
= lrn,                                                  = 
=-------------------------------------------------------=
= zico, not work in torch-cpu, I will check it later.   =
=     zico must use at least two batches of data,       =
=     in order to calculate cross-batch (non-zero) std  =
=-------------------------------------------------------=
= tcet,                                                 =
= snr-synflow,                                          =
= snr-snip,                                             =
=========================================================0. How to install.First create conda env with python version >= 3.6, this repo has been completely tested on python 3.9.condainstallpytorch==1.13.1torchvision==0.14.1torchaudio==0.13.1pytorch-cuda=11.6-cpytorch-cnvidiaInstall torch, torchvision, cudatoolkit.Testedon:pytorch==1.13.1(py3.9_cuda11.6_cudnn8.3.2_0)python==3.9.16
cuda11.6torchvision==0.14.1(py39_cu116)torchaudio==0.13.1(py39_cu116)this repo is perfectly compatible with current mainstream zc testing framework, including zennas, naslib, nb201 related repos, nb101, nb1shot1, blox, etc.If you still cannot use this repo, try to contact me, or try to setup some mainstream nas testing benchmarks, then most problems would be solved.Finally, if all the previous basic enviroment requirements are met, then try this lib with just one single command.pip install -e .
# running this command under the root directory where the setup.py locates in.check installation success.cdtests/
pythontest_zc.py1. TestsImageNet16-120 cannot be automatically downloaded. Using script underscripts/download_data.shto download:sourcescripts/download_data.shnb201ImageNet16-120# do not use `bash`, use `source` instead2. VersionsV1.1.2Fix bug intenas, add net instance deep copy to avoid weight changes.V1.1.1Fix warnings intenas, now using new torch api to calc eigenvalue.Fix bug intcet, add net instance deep copy to avoid weight changes, add manually designedtcetcopy process, remove bn in synflow, add bn in snip.V1.1.0Addtcetmetric, which calculates TCET score.
Addsnrmetrics, which calculates SNR family scores.V1.0.10addzicometric, which calculates ZICO score.V1.0.9fix readme format, no code change.V1.0.8fix bug innwot_relufor wrong for/backward fn register,fix bug inzenfor missed necessary attribute check, add test sample forzenmetric,fix bug inzenfor return value have not .item() attribute,addtenasmetric, which calculates TE-NAS score. (tenas,ntk,lrn)V1.0.7addzenmetric, which calculates ZenNAS score.V1.0.6add originalnaswotimplements based on RELU, can be calculated using metircnwot_relu, also fix potential oom bug, and more reliable GPU memory cache removal code snippets.V1.0.5addnaswot, lnwotinto matsV1.0.4fix bugs in calculation, add more test codes.V1.0.3add shortcuts to import directly from package root directory.3. Quick Bug Fixif you encouther this error:RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'Traceback(mostrecentcalllast):
File"/home/u2280887/GitHub/zero-cost-proxies/tests/test_zc.py",line87,in<module>test_zc_proxies()File"/home/u2280887/GitHub/zero-cost-proxies/tests/test_zc.py",line49,intest_zc_proxiesresults=calc_zc_metrics(metrics=mts,model=net,train_queue=train_loader,device=device,aggregate=True)File"/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zc_proxy.py",line115,incalc_zc_metricsmt_vals=calc_vals(net_orig=model,trainloader=train_queue,device=device,metric_names=metrics,loss_fn=loss_fn)File"/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zc_proxy.py",line101,incalc_valsraisee
File"/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zc_proxy.py",line73,incalc_valsval=M.calc_metric(mt_name,net_orig,device,inputs,targets,loss_fn=loss_fn,split_data=ds)File"/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zero_cost_metrics/__init__.py",line42,incalc_metricreturn_metric_impls[name](net,device,*args,**kwargs)File"/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zero_cost_metrics/__init__.py",line24,inmetric_implret=func(net,*args,**kwargs,**impl_args)File"/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zero_cost_metrics/tenas.py",line316,incompute_TENAS_scoreRN=compute_RN_score(net,inputs,targets,split_data,loss_fn,num_batch)File"/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zero_cost_metrics/tenas.py",line201,incompute_RN_scorenum_linear_regions=float(lrc_model.forward_batch_sample()[0])File"/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zero_cost_metrics/tenas.py",line170,inforward_batch_samplereturn[LRCount.getLinearReginCount()forLRCountinself.LRCounts]File"/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zero_cost_metrics/tenas.py",line170,in<listcomp>return[LRCount.getLinearReginCount()forLRCountinself.LRCounts]File"/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zero_cost_metrics/tenas.py",line93,ingetLinearReginCountself.calc_LR()File"/home/u2280887/miniconda3/envs/zc-alth/lib/python3.9/site-packages/torch/autograd/grad_mode.py",line27,indecorate_contextreturnfunc(*args,**kwargs)File"/home/u2280887/GitHub/zero-cost-proxies/alethiometer/zero_cost_metrics/tenas.py",line62,incalc_LRres=torch.matmul(self.activations.half(),(1-self.activations).T.half())RuntimeError:"addmm_impl_cpu_"notimplementedfor'Half'please check your lib installation, we need gpu support fortorch.half(), please check your cuda version and pytorch version, and reinstall pytorch with cuda support. It seem current cpu version of pytorch does not supporttorch.half(), even if we are using float32 not float16.....