Package: imagenet-w
A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others (CVPR 2023)Zhiheng Li,Ivan Evtimov,Albert Gordo,Caner Hazirbas,Tal Hassner,Cristian Canton Ferrer,Chenliang Xu,Mark Ibrahim[Preprint]TL; DR: Our benchmark results on UrbanCars and ImageNet reveal the overlooked Whac-A-Mole dilemma in shortcut mitigation, i.e., mitigating one shortcut amplifies the reliance on other shortcuts.ImageNet-WWe discover the new watermark shortcut in ImageNet. and create ImageNet-W test set to study (1) state-of-the-art vision models' reliance on the watermark shortcut; (2) the reliance on multiple shortcuts on ImageNet when using ImageNet-W along with other out-of-distribution variants of ImageNet (e.g., ImageNet-R).Use ImageNet-WInstall ImageNet-W package:pipinstallgit+https://github.com/facebookresearch/Whac-A-Mole.gitUseAddWatermarktransform for ImageNet:fromimagenet_wimportAddWatermarkfromtorchvision.datasetsimportImageNetresize_size=256crop_size=224test_transform=transforms.Compose([transforms.Resize(resize_size),transforms.CenterCrop(crop_size),transforms.ToTensor(),AddWatermark(crop_size),# insert AddWatermark before normalizenormalize,])imagenet_w=ImageNet(root,split="val",transform=test_transform)Requirementspipinstall-rrequirements.txtUrbanCars ExperimentsWe construct UrbanCars dataset, a new dataset with multiple shortcuts (i.e., background and co-occurring object), facilitating the study of multi-shortcut learning under the controlled setting.Generate UrbanCars Datasetbashscripts/prepare_dataset_models/create_urbancars.shTrain Shortcut Mitigation Methods on UrbanCarsUse shell scripts inscripts/train_urbancarsto run each method, e.g.,:bashscripts/train_urbancars/$METHOD.shwhere$METHODshould be replaced by method names listed inscripts/train_urbancars.ImageNet ExperimentsPrepare ImageNet and its out-of-distribution variantsSeeprepare_ImageNet.mdPrepare models for evaluating shortcut relianceSeeprepare_checkpoints_for_eval.mdEvaluate state-of-the-art vision models' watermark shortcut reliancePYTHONPATH=.:$PYTHONPATHpythoneval_shortcuts/eval_watermark_shortcut.pyEvaluate reliance on multiple shortcutsPYTHONPATH=.:$PYTHONPATHpythoneval_shortcuts/eval_multiple_shortcuts.pyTrainingWe uselast layer retrainingfor ImageNet experiments.PYTHONPATH=.:$PYTHONPATHpythonimagenet_trainers/launcher.py--method${METHOD}--amp--feature_extractorresnet50_erm--lr${LR}[--wandb][--slurm_partition${SLURM_PARTITION}][--slurm_job_name${METHOD}_imagenet]${METHOD}is the method name (check options for--methodbyPYTHONPATH=.:$PYTHONPATH python imagenet_trainers/launcher.py --help).[Optional] Use--slurm_partition ${SLURM_PARTITION}and--slurm_job_name ${METHOD}_imagenetwhen training on aSlurmcluster, where we usesubmititto submit jobs.[Optional] Turn on--wandbto usewandbfor logging.--feature_extractor resnet50_ermmeans using ResNet-50 trained with ERM to as the feature extractor. For our porposed Last Layer Ensemble (LLE) method, we also usevit-b_mae-ft,vit-l_mae-ft,vit-h_mae-ft(ViT architecture with finetunedMAE) , andvit-b_swag-ft(ViT-B architecture with finetunedSWAG).${LR}is the learning rate. We tune learning rates based on IN-1k top-1 accuracy. The learning rates after tuning and the checkpoints are shown in the following table:methodarchitectureIN-1kIN-W GapCarton GapSIN GapIN-R GapIN-9 GapLRdownloadERMResNet-5076.39-25.40+30-69.43-56.22-5.191e-3modelMixupResNet-5076.17-24.87+34-68.18-55.79-5.601e-4modelCutMixResNet-5075.90-25.78+32-69.31-56.36-5.651e-4modelCutoutResNet-5076.40-25.11+32-69.39-55.93-5.351e-3modelAugMixResNet-5076.23-23.41+38-68.51-54.91-5.851e-4modelSDResNet-5076.39-26.03+30-69.42-56.36-5.331e-3modelWTM AugResNet-5076.32-5.78+14-69.31-56.22-5.341e-3modelTXT AugResNet-5075.94-25.93+36-63.99-53.24-5.661e-4modelBG AugResNet-5076.03-25.01+36-68.41-54.51-4.671e-4modelLfFResNet-5076.35-26.19+36-69.34-56.02-5.611e-4modelJTTResNet-5076.33-26.40+32-69.48-56.30-5.551e-2modelEIILResNet-5071.51-33.17+24-65.93-61.09-6.271e-4modelDebiANResNet-5076.33-26.40+36-69.37-56.29-5.531e-4modelLLE (ours)ResNet-5076.25-6.18+10-61.00-54.89-3.821e-3modelMAE +LLE (ours)ViT-B83.68-2.48+6-58.78-44.96-3.701e-3modelMAE +LLE (ours)ViT-L85.84-1.74+12-56.32-34.64-2.771e-3modelMAE +LLE (ours)ViT-H86.84-1.11+28-55.69-30.95-2.351e-3modelSWAG +LLE (ours)ViT-B85.37-2.50+8-60.92-28.37-3.191e-4modelIn our proposed Last Layer Ensemble (LLE) method, we also use edge detection for data augmentation, i.e., Edge Aug. The details of how to generate edge detection data on ImageNet and the checkpoints are inEdge_Aug.md.EvaluationTo evaluate the trained models, download the checkpoint from the table above and use its file path as${PATH_TO_CHECKPOINT}:PYTHONPATH=.:$PYTHONPATHpythonimagenet_trainers/launcher.py--method${METHOD}--amp--feature_extractorresnet50_erm[--wandb][--slurm_partition${SLURM_PARTITION}][--slurm_job_name${METHOD}_imagenet]--evaluate--resume${PATH_TO_CHECKPOINT}CitationIf you use UrbanCars dataset or ImageNet-W dataset, or compare with our proposed Last Layer Ensemble (LLE) method, please cite our paper:@article{li_2023_whac_a_mole,title={A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others},author={Zhiheng Li and Ivan Evtimov and Albert Gordo and Caner Hazirbas and Tal Hassner and Cristian Canton Ferrer and Chenliang Xu and Mark Ibrahim},booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},month={June},year={2023},url={https://arxiv.org/abs/2212.04825}}LicenseSeeLICENSEfor details.AttributionThe Whack-A-Mole icon is created by Flat Icons - Flaticon