Package: sdenet
SDE Net (Keras)This repo contains the code for the paper:Lingkai Kong, Jimeng Sun and Chao Zhang, SDE-Net: Equipping Deep Neural Network with Uncertainty Estimates, ICML2020.[paper] [video]Package installationvirtualenv-ppython3venv&&sourcevenv/bin/activate# optional but recommended.pipinstall-rrequirements.txt&&pipinstall-e.# install the package.Training & EvaluationSupported datasets are: MNIST, SVHN, CIFAR10, CIFAR100. Supported models are RESNET and SDENET.Look at the bash run scripts at the root of the repository to get started for training and evaluation.Comparison between official Pytorch implementation and KerasThis comparison is just the result of one run. No runs were handpicked. Overall it's very similar.Except probably SDENET on SVHN (95% vs 94%).PytorchMNIST RESNET
_________________________________

Final Accuracy: 9945/10000 (99.45%)

generate log  from out-of-distribution data
calculate metrics for OOD
OOD  Performance of Baseline detector
TNR at TPR 95%:            88.783%
AUROC:                     95.939%
Detection acc:             92.169%
AUPR In:                   86.441%
AUPR Out:                  98.434%

calculate metrics for mis
mis  Performance of Baseline detector
TNR at TPR 95%:            89.791%
AUROC:                     97.510%
Detection acc:             93.041%
AUPR In:                   99.985%
AUPR Out:                  34.000%


MNIST SDENET
_________________________________

Final Accuracy: 9927/10000 (99.27%)

generate log  from out-of-distribution data
calculate metrics for OOD
OOD  Performance of Baseline detector
TNR at TPR 95%:            99.372%
AUROC:                     99.804%
Detection acc:             98.692%
AUPR In:                   99.483%
AUPR Out:                  99.887%
calculate metrics for mis
mis  Performance of Baseline detector
TNR at TPR 95%:            92.544%
AUROC:                     97.525%
Detection acc:             94.485%
AUPR In:                   99.979%
AUPR Out:                  41.739%


SVHN RESNET
_________________________________

Final Accuracy: 24609/25856 (95.18%)

generate log  from out-of-distribution data
calculate metrics for OOD
OOD  Performance of Baseline detector
TNR at TPR 95%:            66.552%
AUROC:                     94.421%
Detection acc:             90.136%
AUPR In:                   97.639%
AUPR Out:                  84.998%
calculate metrics for mis
mis  Performance of Baseline detector
TNR at TPR 95%:            64.376%
AUROC:                     90.458%
Detection acc:             85.371%
AUPR In:                   99.301%
AUPR Out:                  44.899%


SVHN SDENET
_________________________________

Final Accuracy: 24588/25856 (95.10%)

generate log  from out-of-distribution data
calculate metrics for OOD
OOD  Performance of Baseline detector
TNR at TPR 95%:            65.215%
AUROC:                     94.308%
Detection acc:             89.746%
AUPR In:                   97.694%
AUPR Out:                  84.017%
calculate metrics for mis
mis  Performance of Baseline detector
TNR at TPR 95%:            67.831%
AUROC:                     91.267%
Detection acc:             86.501%
AUPR In:                   99.270%
AUPR Out:                  48.871%KerasMNIST RESNET
_________________________________

 Final Accuracy: 9944/10000 (99.44%)

generate log  from out-of-distribution data
calculate metrics for OOD
OOD  Performance of Baseline detector
TNR at TPR 95%:            93.162%
AUROC:                     97.946%
Detection acc:             94.250%
AUPR In:                   94.842%
AUPR Out:                  99.215%
calculate metrics for mis
mis  Performance of Baseline detector
TNR at TPR 95%:            96.997%
AUROC:                     98.863%
Detection acc:             96.697%
AUPR In:                   99.994%
AUPR Out:                  26.744%

MNIST SDENET
_________________________________

Final Accuracy: 9934/10000 (99.34%)

generate log  from out-of-distribution data
calculate metrics for OOD
OOD  Performance of Baseline detector
TNR at TPR 95%:            98.425%
AUROC:                     99.567%
Detection acc:             97.804%
AUPR In:                   98.613%
AUPR Out:                  99.872%
calculate metrics for mis
mis  Performance of Baseline detector
TNR at TPR 95%:            95.515%
AUROC:                     98.763%
Detection acc:             95.825%
AUPR In:                   99.992%
AUPR Out:                  32.524%

SVHN RESNET
_________________________________

 Final Accuracy: 24487/25856 (94.71%)

generate log  from out-of-distribution data
calculate metrics for OOD
OOD  Performance of Baseline detector
TNR at TPR 95%:            56.648%
AUROC:                     93.602%
Detection acc:             87.504%
AUPR In:                   97.627%
AUPR Out:                  81.664%
calculate metrics for mis
mis  Performance of Baseline detector
TNR at TPR 95%:            63.765%
AUROC:                     91.843%
Detection acc:             85.721%
AUPR In:                   99.386%
AUPR Out:                  46.231%

SVHN SDENET
_________________________________

Final Accuracy: 24261/25856 (93.83%)

generate log  from out-of-distribution data
calculate metrics for OOD
OOD  Performance of Baseline detector
TNR at TPR 95%:            55.568%
AUROC:                     93.406%
Detection acc:             87.942%
AUPR In:                   97.538%
AUPR Out:                  82.881%
calculate metrics for mis
mis  Performance of Baseline detector
TNR at TPR 95%:            61.798%
AUROC:                     89.845%
Detection acc:             85.974%
AUPR In:                   99.183%
AUPR Out:                  47.664%