Package: torchdistill
torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillationtorchdistill(formerlykdkit) offers various state-of-the-art knowledge distillation methods
and enables you to design (new) experiments simply by editing a declarative yaml config file instead of Python code.
Even when you need to extract intermediate representations in teacher/student models,
you willNOTneed to reimplement the models, that often change the interface of the forward, but instead
specify the module path(s) in the yaml file. Refer tothese papersfor more details.In addition to knowledge distillation, this framework helps you design and perform general deep learning experiments
(WITHOUT coding) for reproducible deep learning studies. i.e., it enables you to train models without teachers
simply by excluding teacher entries from a declarative yaml config file.
You can find such examples below and inconfigs/sample/.When you refer totorchdistillin your paper, please citethese papersinstead of this GitHub repository.If you usetorchdistillas part of your work, your citation is appreciated and motivates me to maintain and upgrade this framework!Forward hook managerUsingForwardHookManager, you can extract intermediate representations in model without modifying the interface of its forward function.This example notebookwill give you a better idea of the usage such as knowledge distillation and analysis of intermediate representations.1 experiment → 1 declarative PyYAML config fileIntorchdistill, many components and PyTorch modules are abstracted e.g., models, datasets, optimizers, losses,
and more! You can define them in a declarative PyYAML config file so that can be seen as a summary of your experiment,
and in many cases, you willNOT need to write Python code at all.
Take a look at some configurations available inconfigs/.
You'll see what modules are abstracted and how they are defined in a declarative PyYAML config file to design an experiment.If you want to use your own modules (models, loss functions, datasets, etc) with this framework,
you can do so without editing code in the local packagetorchdistill/.Seethe official documentationandDiscussionsfor more details.Top-1 validation accuracy for ILSVRC 2012 (ImageNet)T: ResNet-34*PretrainedKDATFTCRDTf-KDSSKDL2PAD-L2KRS: ResNet-1869.76*71.3770.9071.5670.9370.5270.0971.0871.7171.64Original workN/AN/A70.7071.43**71.1770.4271.6270.9071.7171.61* The pretrained ResNet-34 and ResNet-18 are provided by torchvision.** FT is assessed with ILSVRC 2015 in the original work.For the 2nd row (S: ResNet-18), most of the results are reported inthis paper,
and their checkpoints (trained weights), configuration and log files areavailable,
and the configurations reuse the hyperparameters such as number of epochs used in the original work except for KD.ExamplesExecutable code can be found inexamples/such asImage classification: ImageNet (ILSVRC 2012), CIFAR-10, CIFAR-100, etcObject detection: COCO 2017, etcSemantic segmentation: COCO 2017, PASCAL VOC, etcText classification: GLUE, etcFor CIFAR-10 and CIFAR-100, some models are reimplemented and available as pretrained models intorchdistill.
More details can be foundhere.Some Transformer models fine-tuned bytorchdistillfor GLUE tasks are available atHugging Face Model Hub.
Sample GLUE benchmark results and details can be foundhere.Google Colab ExamplesThe following examples are available indemo/.
Note that these examples are for Google Colab users. Usually,examples/would be a better reference
if you have your own GPU(s).CIFAR-10 and CIFAR-100Training without teacher modelsKnowledge distillationGLUEFine-tuning without teacher modelsKnowledge distillationThese examples write out test prediction files for you to see the test performance atthe GLUE leaderboard system.PyTorch HubIf you find models onPyTorch Hubor GitHub repositories supporting PyTorch Hub,
you can import them as teacher/student models simply by editing a declarative yaml config file.e.g., If you use a pretrained ResNeSt-50 available inhuggingface/pytorch-image-models(akatimm) as a teacher model for ImageNet dataset, you can import the model via PyTorch Hub with the following entry
in your declarative yaml config file.models:teacher_model:name:'resnest50d'repo_or_dir:'huggingface/pytorch-image-models'kwargs:num_classes:1000pretrained:TrueHow to setupPython >= 3.8pipenv (optional)Install by pip/pipenvpip3 install torchdistill
# or use pipenv
pipenv install torchdistillInstall from this repository (not recommended)git clone https://github.com/yoshitomo-matsubara/torchdistill.git
cd torchdistill/
pip3 install -e .
# or use pipenv
pipenv install "-e ."Issues / Questions / RequestsFeel free to create an issue if you find a bug.If you have either a question or feature request, start a new discussionhere.
Please make sure the issue/question/request has not been addressed yet by searching through the issues and discussions.CitationIf you usetorchdistillin your research, please cite the following papers:[Paper] [Preprint]@inproceedings{matsubara2021torchdistill,title={{torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation}},author={Matsubara, Yoshitomo},booktitle={International Workshop on Reproducible Research in Pattern Recognition},pages={24--44},year={2021},organization={Springer}}[OpenReview] [Preprint]@article{matsubara2023torchdistill,title={{torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP}},author={Matsubara, Yoshitomo},journal={arXiv preprint arXiv:2310.17644},year={2023}}AcknowledgmentsThis project has been supported by Travis CI's OSS credits andJetBrain's Free License Programs (Open Source)since November 2021 and June 2022, respectively.References:mag:pytorch/vision/references/classification/:mag:pytorch/vision/references/detection/:mag:pytorch/vision/references/segmentation/:mag:huggingface/transformers/examples/pytorch/text-classification:mag:Geoffrey Hinton, Oriol Vinyals and Jeff Dean."Distilling the Knowledge in a Neural Network"(Deep Learning and Representation Learning Workshop: NeurIPS 2014):mag:Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta and Yoshua Bengio."FitNets: Hints for Thin Deep Nets"(ICLR 2015):mag:Junho Yim, Donggyu Joo, Jihoon Bae and Junmo Kim."A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning"(CVPR 2017):mag:Sergey Zagoruyko and Nikos Komodakis."Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer"(ICLR 2017):mag:Nikolaos Passalis and Anastasios Tefas."Learning Deep Representations with Probabilistic Knowledge Transfer"(ECCV 2018):mag:Jangho Kim, Seonguk Park and Nojun Kwak."Paraphrasing Complex Network: Network Compression via Factor Transfer"(NeurIPS 2018):mag:Byeongho Heo, Minsik Lee, Sangdoo Yun and Jin Young Choi."Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons"(AAAI 2019):mag:Tong He, Chunhua Shen, Zhi Tian, Dong Gong, Changming Sun, Youliang Yan."Knowledge Adaptation for Efficient Semantic Segmentation"(CVPR 2019):mag:Wonpyo Park, Dongju Kim, Yan Lu and Minsu Cho."Relational Knowledge Distillation"(CVPR 2019):mag:Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence and Zhenwen Dai."Variational Information Distillation for Knowledge Transfer"(CVPR 2019):mag:Yoshitomo Matsubara, Sabur Baidya, Davide Callegaro, Marco Levorato and Sameer Singh."Distilled Split Deep Neural Networks for Edge-Assisted Real-Time Systems"(Workshop on Hot Topics in Video Analytics and Intelligent Edges: MobiCom 2019):mag:Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou and Zhaoning Zhang."Correlation Congruence for Knowledge Distillation"(ICCV 2019):mag:Frederick Tung and Greg Mori."Similarity-Preserving Knowledge Distillation"(ICCV 2019):mag:Yonglong Tian, Dilip Krishnan and Phillip Isola."Contrastive Representation Distillation"(ICLR 2020):mag:Yoshitomo Matsubara and Marco Levorato."Neural Compression and Filtering for Edge-assisted Real-time Object Detection in Challenged Networks"(ICPR 2020):mag:Li Yuan, Francis E.H.Tay, Guilin Li, Tao Wang and Jiashi Feng."Revisiting Knowledge Distillation via Label Smoothing Regularization"(CVPR 2020):mag:Guodong Xu, Ziwei Liu, Xiaoxiao Li and Chen Change Loy."Knowledge Distillation Meets Self-Supervision"(ECCV 2020):mag:Youcai Zhang, Zhonghao Lan, Yuchen Dai, Fangao Zeng, Yan Bai, Jie Chang and Yichen Wei."Prime-Aware Adaptive Distillation"(ECCV 2020):mag:Pengguang Chen, Shu Liu, Hengshuang Zhao, Jiaya Jia."Distilling Knowledge via Knowledge Review"(CVPR 2021)