Package: difftopk
Differentiable Top-k Classification LearningOfficial implementation for our ICML 2022 Paper "Differentiable Top-k Classification Learning".Thedifftopklibrary provides different differentiable sorting and ranking methods as well as a wrapper for using them
in aTopKCrossEntropyLoss.difftopkbuilds on PyTorch.Paper @ArXiv,
Video @Youtube.💻 Installationdifftopkcan be installed via pip from PyPI withpipinstalldifftopkSparse ComputationFor the functionality of evaluating the differentiable topk operators in a sparse way, the packagetorch-sparsehas to be installed.
This can be done, e.g., viapipinstalltorch-scattertorch-sparse-fhttps://data.pyg.org/whl/torch-1.13.0+cpu.htmlFor more information on how to installtorch-sparse, seehere.Example for Full Installation from Scratch and with all Dependencies(click to expand)Depending on your system, the following commands will have to be adjusted.virtualenv-ppython3.env_topk
..env_topk/bin/activate
pipinstallboto3numpyrequestsscikit-learntqdmpipinstalltorch==1.13.0+cu116torchvision==0.14.0+cu116-fhttps://download.pytorch.org/whl/torch_stable.html
pipinstalldiffsort# optional for torch-sparseFORCE_CUDA=1pipinstall--no-cache-dirtorch-scattertorch-sparse-fhttps://data.pyg.org/whl/torch-1.13.0+cu116.html

pipinstall.👩‍💻 DocumentationThedifftopklibrary provides of differentiable sorting and ranking methods as well as a wrapper for using them in aTopKCrossEntropyLoss. The differentiable sorting and ranking methods included are:Variants of Differentiable Sorting NetworksbitonicBitonic Differentiable Sorting Networks (sparse)bitonic__non_sparseBitonic Differentiable Sorting Networkssplitter_selectionDifferentiable Splitter Selection Networks (sparse)odd_evenOdd-Even Differentiable Sorting NetworksneuralsortNeuralSortsoftsortSoftSortFurthermore, this library also includes the smooth top-k loss from Lapin et al. (SmoothTopKLossandSmoothHardTopKLoss.)TopKCrossEntropyLossIn the center of the library lies thedifftopk.TopKCrossEntropyLoss, which may be used as a drop-in replacement fortorch.nn.CrossEntropyLoss. The signature ofTopKCrossEntropyLossis defined as follows:loss_fn=difftopk.TopKCrossEntropyLoss(diffsort_method='odd_even',# the sorting / ranking method as discussed aboveinverse_temperature=2,# the inverse temperature / steepnessp_k=[.5,0.,0.,0.,.5],# the distribution P_Kn=1000,# number of classesm=16,# the number m of scores to be sorted (can be smaller than n to make it efficient)distribution='cauchy',# the distribution used for differentiable sorting networksart_lambda=None,# the lambda for the ART used if `distribution='logistic_phi'`device='cpu',# the device to compute the loss ontop1_mode='sm'# makes training more stable and is the default value)It can be used asloss_fn(outputs, labels).DiffTopkNetdifftopk.DiffTopkNetfollows the signature ofdiffsort.DiffSortNetfrom thediffsortpackage.
However, instead of returning the full differentiable permutation matrices of sizenxn, it returns differentiable top-k attribution matrices of sizenxk.
More specifically, given an input of shapebxn, the module returns a tuple ofNoneand a Tensor of shapebxnxk.
(It returns a tuple to maintain the signature ofDiffSortNet.)sorter=difftopk.DiffTopkNet(sorting_network_type='bitonic',size=16,# Number of inputsk=5,# ksparse=True,# whether to use a sparse implementationdevice='cpu',# the devicesteepness=10.0,# the inverse temperatureart_lambda=0.25,# the lambda for the ART used if `distribution='logistic_phi'`distribution='cauchy'# the distribution used for the differentiable relaxation)# Usage example for difftopk on a random inputx=torch.randperm(16).unsqueeze(0).float()*100.print(x@sorter(x)[1][0])NeuralSort/SoftSortsorter=difftopk.NeuralSort(tau=2.,# A temperature parameter)sorter=difftopk.SoftSort(tau=2.)🧪 Experiments🧫 ImageNet Fine-TuningWe provide pre-computed embeddings for the ImageNet data set. ⚠️ These embedding files are very large (>10 GB each.)
Feel free to also use the embeddings for other fine-tuning experiments.# Resnext101 WSL ImageNet-1K (~11GB each)wgethttps://nyc3.digitaloceanspaces.com/publicdata1/ImageNet_embeddings_labels_train_test_IGAM_Resnext101_32x48d_320.p
wgethttps://nyc3.digitaloceanspaces.com/publicdata1/ImageNet_embeddings_labels_train_test_IGAM_Resnext101_32x32d_320.p
wgethttps://nyc3.digitaloceanspaces.com/publicdata1/ImageNet_embeddings_labels_train_test_IGAM_Resnext101_32x16d_320.p
wgethttps://nyc3.digitaloceanspaces.com/publicdata1/ImageNet_embeddings_labels_train_test_IGAM_Resnext101_32x8d_320.p# Resnext101 WSL ImageNet-21K-P (~50GB)wgethttps://publicdata1.nyc3.digitaloceanspaces.com/ImageNet21K-P_embeddings_labels_train_test_IGAM_Resnext101_32x48d_224_float16.p# Noisy Student EfficientNet-L2 ImageNet-1K (~14GB)wgethttps://publicdata1.nyc3.digitaloceanspaces.com/ImageNet_embeddings_labels_train_test_tf_efficientnet_l2_ns_timm_transform_800_float16.pThe following are the hyperparameter combinations for reproducing the tables in the paper.
The DiffSortNet entries in Table 5 can be reproduced usingpythonexperiments/train_imagenet.py-d./ImageNet_embeddings_labels_train_test_IGAM_Resnext101_32x48d_320.p--nloglr4.5\--p_k.2.2.2.2.2--m16--methodbitonic--distributionlogistic_phi--inverse_temperature1.--art_lambda.5pythonexperiments/train_imagenet.py-d./ImageNet_embeddings_labels_train_test_IGAM_Resnext101_32x48d_320.p--nloglr4.5\--p_k.2.2.2.2.2--m16--methodsplitter_selection--distributionlogistic_phi--inverse_temperature1.--art_lambda.5

pythonexperiments/train_imagenet.py-d./ImageNet_embeddings_labels_train_test_tf_efficientnet_l2_ns_timm_transform_800_float16.p--nloglr4.5\--p_k.25.0.0.0.75--m16--methodbitonic--distributionlogistic--inverse_temperature.5pythonexperiments/train_imagenet.py-d./ImageNet_embeddings_labels_train_test_tf_efficientnet_l2_ns_timm_transform_800_float16.p--nloglr4.5\--p_k.25.0.0.0.75--m16--methodsplitter_selection--distributionlogistic--inverse_temperature.5and, for the remaining methods and tables, the hyperparameters are defined in the following:(click to expand)# Tables 2+3 (1K):pythonexperiments/train_imagenet.py-d./ImageNet_embeddings_labels_train_test_IGAM_Resnext101_32x48d_320.p--m16--nloglr4.5# Tables 2+3 (21K):pythonexperiments/train_imagenet.py-d./ImageNet21K-P_embeddings_labels_train_test_IGAM_Resnext101_32x48d_224_float16.p--m50--nloglr4.--n_epochs40# combined with one of each of the following--methodsoftmax_cross_entropy
--methodbitonic--distributionlogistic_phi--inverse_temperature1.--art_lambda.5
--methodsplitter_selection--distributionlogistic_phi--inverse_temperature1.--art_lambda.5
--methodneuralsort--inverse_temperature0.5
--methodsoftsort--inverse_temperature0.5
--methodsmooth_hard_topk--inverse_temperature1.--p_k1.0.0.0.0.
--p_k0.0.0.0.1.
--p_k.50.0.0..5
--p_k.250.0.0..75
--p_k.10.0.0..9
--p_k.2.2.2.2.2🎆 CIFAR-100 Training from ScratchIn addition to ImageNet fine-tuning, we can also train a ResNet18 on CIFAR-100 from scratch.(click to expand)# Tables 1+4:pythonexperiments/train_cifar100.py

--methodsoftmax_cross_entropy
--methodbitonic--distributionlogistic_phi--inverse_temperature.5--art_lambda.5
--methodsplitter_selection--distributionlogistic_phi--inverse_temperature.5--art_lambda.5
--methodneuralsort--inverse_temperature0.0625
--methodsoftsort--inverse_temperature0.0625
--methodsmooth_hard_topk--inverse_temperature1.

--p_k1.0.0.0.0.
--p_k0.0.0.0.1.
--p_k.50.0.0..5
--p_k.250.0.0..75
--p_k.10.0.0..9
--p_k.2.2.2.2.2# Examples:pythonexperiments/train_cifar100.py--methodsoftmax_cross_entropy--p_k1.0.0.0.0.
pythonexperiments/train_cifar100.py--methodsplitter_selection--distributionlogistic_phi--inverse_temperature.5--art_lambda.5--p_k.2.2.2.2.2📖 Citing@inproceedings{petersen2022difftopk,title={{Differentiable Top-k Classification Learning}},author={Petersen, Felix and Kuehne, Hilde and Borgelt, Christian and Deussen, Oliver},booktitle={International Conference on Machine Learning (ICML)},year={2022}}📜 Licensedifftopkis released under the MIT license. SeeLICENSEfor additional details about it.