Package: canary-sefi
Canary SEFI中文版DocumentPlease refer to our 📖document 👉👉Canary Document👈👈 for using Canary.The document you are currently reading is an early version, which is incomplete and is being edited by the author and will be updated continuously. We will provide the complete document as soon as possible, as well as the English version of the document. If you have questions, please contactjiazheng.sun@bit.edu.cnfor more information.IntroductionSEFI is a framework for evaluating the robustness of deep learning-based image recognition models.It uses selected attack methods to generate adversarial samples based on selected models and uses these adversarial examples to attack any model you want. In the process, it collects data including multiple evaluation metrics to assess AI model robustness and attack method effectiveness while trying to find the best defense solution.It also provides a toolkit containing multiple models, SOTA attack methods and defense methods, and allows users to make additional integrations themselves.SEFI was created and is maintained by researchers at BIT.FunctionsWhat models do we support？We have supported 15 models on 4 datasets, of which all 15 models for the ImageNet dataset are available. We hope that you will participate in improving our model library and share your own model structure and weighting information to help more people.We have built a public repository of model weights(SEFI-LW) located:https://github.com/NeoSunJZ/Canary_SEFI_Lib_WeightModelsSubstructureDatasetFully AvailableSourceSupport GCAM?Weights AvailableTop-1 AccLeNetV5N/AFashion-MNISTCome SoonCNN-for-Fashion-MNISTPlanned✔SEFI-LWAlexNetN/AFashion-MNISTCome SoonCNN-for-Fashion-MNISTPlanned✔SEFI-LW92.19%AlexNetN/AImageNet✔Torchvision✔✔OfficialVGGvgg16_bnImageNet✔Torchvision✔✔OfficialVGGvgg16_bnCIFAR-10✔PyTorch CIFAR10✔✔SEFI-LWVGGvgg16_bnCIFAR-100✔PyTorch CIFAR ModelsPlanned✔SEFI-LWGoogLeNetN/AImageNet✔Torchvision✔✔OfficialGoogLeNetN/ACIFAR-10✔PyTorch CIFAR10✔✔SEFI-LWInceptionV3N/AImageNet✔Torchvision✔✔OfficialInceptionV3N/ACIFAR-10✔PyTorch CIFAR10✔✔SEFI-LWResNetresnet50ImageNet✔Torchvision✔✔OfficialResNetresnet50CIFAR-10✔PyTorch CIFAR10✔✔SEFI-LWResNetresnet56CIFAR-100Come SoonPyTorch CIFAR ModelsPlanned✔SEFI-LW72.63%ResNetresnet19lightFashion-MNISTCome SoonCNN-for-Fashion-MNISTPlanned✔SEFI-LWDenseNetdensenet161*ImageNet✔Torchvision✔✔OfficialDenseNetdensenet161CIFAR-10✔PyTorch CIFAR10✔✔SEFI-LWSqueezeNetsqueezenet1_1ImageNet✔Torchvision✔✔OfficialMobileNetV3v3_largeImageNet✔Torchvision✔✔OfficialMobileNetV2N/AImageNet✔Torchvision✔✔OfficialMobileNetV2N/ACIFAR-10✔PyTorch CIFAR10Planned✔SEFI-LWMobileNetV2v2_x1_0CIFAR-100✔PyTorch CIFAR ModelsPlanned✔SEFI-LW73.61%ShuffleNetV2v2_x2_0ImageNet✔Torchvision✔✔OfficialShuffleNetV2v2_x2_0CIFAR-10Come SoonPyTorch CIFAR ModelsN/AWaiting UploadShuffleNetV2v2_x2_0CIFAR-100✔PyTorch CIFAR ModelsPlanned✔SEFI-LWMNASNetmnasnet1_3ImageNet✔Torchvision✔✔OfficialEfficientNetV2v2_sImageNet✔Torchvision✔✔OfficialVisionTransformervit_b_32ImageNet✔Torchvision✔✔OfficialRegNety_8gfImageNet✔Torchvision✔✔OfficialSwinTransformerswin_sImageNet✔Torchvision✔✔OfficialConvNextconvnext_baseImageNet✔Torchvision✔✔OfficialWideResNetwideresnet34_10CIFAR-10Come SoonPyTorch CIFAR10✔Waiting UploadWhat attack methods do we support？We support 22 common attack methods, including:Attack MethodsMethod TypeAttack ApproachNot Support ModelsProvide default parameters?FGSMWhite-BoxGradientNoneYes, applicable to ImageNetJSMAWhite-BoxGradientNoneYes, applicable to ImageNetDeepFoolWhite-BoxGradientNoneYes, applicable to ImageNetI-FGSM (BIM)White-BoxGradientNoneYes, applicable to ImageNetC&W AttackWhite-BoxGradientNoneYes, applicable to ImageNetProjectedGradient Descent(PGD)White-BoxGradientNoneYes, applicable to ImageNetMI-FGSM (MIM)Transferable Black-boxTransfer, GradientNoneYes, applicable to ImageNetSI-FGSM (SIM)Transferable Black-boxTransfer, GradientNoneYes, applicable to ImageNetNI-FGSM (NIM)Transferable Black-boxTransfer, GradientNoneYes, applicable to ImageNetVMI-FGSM (VMIM)Transferable Black-boxTransfer, GradientNoneYes, applicable to ImageNetElastic-Net Attack (EAD)White-BoxGradientNoneYes, applicable to ImageNetSSAHWhite-BoxGradientInceptionV3、SwinT、ViTYes, applicable to ImageNetOne-pixel Attack (OPA)Black-BoxQuery, ScoreNot testedNoLocal Search Attack (LSA)Black-BoxQuery, ScoreNoneYes, applicable to ImageNetBoundary Attack (BA)Black-BoxQuery, DecisionNoneYes, applicable to ImageNetSpatial Attack (SA)Black-BoxQueryNoneYes, applicable to ImageNetHop Skip Jump Attack (HSJA)Black-BoxQuery, DecisionNoneYes, applicable to ImageNetGen Attack (GA)Black-BoxQuery, ScoreNoneYes, applicable to ImageNetSPSABlack-BoxQuery, ScoreNoneYes, applicable to ImageNetZeroth-OrderOptimization(ZOO)Black-BoxQuery, ScoreNot testedNoAdvGanBlack-BoxQuery, ScoreNoneYes, applicable to ImageNetTREMBABlack-BoxQuery, ScoreGoogLeNet、EfficientNetV2Yes, applicable to ImageNetWe are looking for more good and classic attack methods to add to our library, if you are the author of a method, feel free to contribute your method. Some of the methods that are not fully tested may not be shown in the above table, but it may appear in the code earlier, if it fails to appear in the above list it means it may not be stable or have stable support for the time being.What defense methods do we support? (Experimental)The entire defense module is currently experimental, which means they may not be stable.We support 8 common defense methods, including:Defense MethodsMethod TypeNot Support ModelsProvide default parameters?NATAdversarial TrainingMartAdversarial TrainingNaturalAdversarial TrainingTradesAdversarial TrainingJpegImage ProcessingQuantizeImage ProcessingTVMImage ProcessingQuiltingImage ProcessingWe are looking for more good and classic defense methods to add to our library, if you are the author of a method, feel free to contribute your method. Some of the methods that are not fully tested may not be shown in the above table, but it may appear in the code earlier, if it fails to appear in the above list it means it may not be stable or have stable support for the time being.What data do we support collecting？We support the full collection of the following four types of metrics. Please refer to our Paper section 3.1 or the user manual for the specific meaning of the metrics.Model capability measurement metricsClean Example Accuracy (Clear Accuracy, CA)Clean example F1 score (Clear F1, CF)Clear Confidence (CC)Attack effectiveness measurement metricsMisclassification Ratio (MR) /  Targeted Attack Success (TAS)Adversarial Example Confidence Change (ACC): Average Increase in Adversarial-class Confidence (AIAC) / Average Reduction in True-class Confidence (ARTC)Average Class Activation Mapping Change (ACAMC)Observable Transfer Rate (OTR)Cost of attack measurement metricsCalculation Time Cost (CTC)Query Number Cost (QNC)Average Norm Distortion(AND): Average Maximum Distortion (AMD) / Average Euclidean Distortion (AED) / Average Pixel Change Ratio (APCR)Average Euclidean Distortion in Frequency Domain (AED-FD)Average Metrics Similarity (AMS): Average Deep Metrics Similarity (ADMS) / Average Low-level Metrics Similarity (ALMS)Effectiveness of defense measurement metricsModel Capability Variance (MCV): Accuracy Variance (AV) / F1-Score Variance (FV) / Mean Confidence Variance (CV)Rectify/Sacrifice Ratio (RR/SR)Attack Capability Variance (ACV): MR Variance (MRV) / AND Variance (ANDV) / AMS Variance (AMSV)Average Adversarial Confidence Change (AACC):Average Reduction in Adversarial-class Confidence (ARAC) / Average Increase in True-class Confidence (AITC)Maintainers@NeoSunJz.ContributorsOur main contributors are：孙家正（Jiazheng Sun）、Li Chen、Chenxiao Xia、Da Zhang、 Rong Huang、Zhi Qu、Wenqi XiongWe are particularly grateful for：Jun Zheng 、Yu’an TanCiteWe sincerely hope that Canary can be helpful to you, and we also welcome you to cite our articles when using Canary to complete your research work:@Article{electronics12173665,
  AUTHOR = {Sun, Jiazheng and Chen, Li and Xia, Chenxiao and Zhang, Da and Huang, Rong and Qiu, Zhi and Xiong, Wenqi and Zheng, Jun and Tan, Yu-An},
  TITLE = {CANARY: An Adversarial Robustness Evaluation Platform for Deep Learning Models on Image Classification},
  JOURNAL = {Electronics},
  VOLUME = {12},
  YEAR = {2023},
  NUMBER = {17},
  ARTICLE-NUMBER = {3665},
  URL = {https://www.mdpi.com/2079-9292/12/17/3665},
  ISSN = {2079-9292},
  DOI = {10.3390/electronics12173665}
}LicenseApache 2.0© Beijing Institute of Technology (BIT)