VisionAD, a software package of performant anomaly detection algorithms, and Proportion Localised, an interpretable metric
Authors: Alexander D. J. Taylor
Phillip Tregidgo
Jonathan James Morrison
Neill D. F. Campbell
Venue: Accepted by TMLR ()

Abstract: We release VisionAD, an anomaly detection library in the domain of images. The library forms the largest and most performant collection of such algorithms to date. Each algorithm is written through a standardised API, for ease of use. The library has a focus on fair benchmarking intended to mitigate the issue of cherry-picked results. It enables rapid experimentation and straightforward integration of new algorithms. In addition, we propose a new metric, Proportion Localised (PL). This reports the proportion of anomalies that are sufficiently localised via classifying each discrete anomaly as localised or not. The metric is far more intuitive as it has a real physical relation, meaning it is attractive to industry-based professionals. We also release the VisionADIndustrial (VADI) benchmark, a thorough benchmarking of the top anomaly detection algorithms. This benchmark calculates the mean across the pooled classes of the MVTec and VisA datasets. We are committed to hosting an updated version of this leaderboard online, and encourage researchers to add, tweak and improve algorithms to climb this leaderboard. VisionAD code is found at https://github.com/alext1995/VisionAD, and Proportion Localised code is found at https://github.com/alext1995/proportion_localised.

Published in Transactions on Machine Learning Research (06/2024)
VisionAD, software package of performant anomaly detec- tion algorithms, and Proportion Localised, an interpretable metric
Alexander D.J. Taylor
adjt200bath.com ac. uk
ART-AI
Department of Computer Science University of Bath
Jonathan James Morrison
Defence Operalions
Rolls-Royce
Phillip Tregidgo
Defence Operations
Rolls- Rolls-Royce
Neill D.F. Campbell
nc537
ART-AI
Department of Computer Science
University of Bath
Reviewed on OpenReview: //tps://ogg optrreviewwreew... net/ forum? id= o5kYH7bNNe3
Abstract
We release VisionAD, an anomaly detection library in the domain of images. The library forms the largest and most performant collection of such algorithms date. Each algo- rithm is written through standardised API, for case of fair benchmarking intended to miligate the issue of cherry-picked results. It enables rapid experimentation and straightforward integration of new algorithms. In addition, we pro- pose a new metric, Proportion Localised (PL). This reports the proportion of anomalies that are sufficiently localised via classifying each discrete anomaly as localised or not. The metric is far more intuitive as has real physical relation, meaning is attractive to industry-basee based professionals. We also release the DIndustallll (VADI) benchmark, thorough benchnarking of the top anomaly detection algorithms. This benchmark cal- culates the mean across the pooled classes of the MVTec and VisA datasets. We are com- mitted to hosting updated version of this leaderboard online, and encourage rescarchers to add, tweak and improve algorithms to climb this leaderboard. VisionAD code is found and Proportion Localised code found at http:://ggghbbb..coa//tttttttt// locallised
The library has focus
at http://ghhtat.coooooooottt//////
1 Introduction
In the real world many classification problems do not contain sufficient data of various classes. Therefore the supervised training of deep learning models presents a challenge. Due to this, the field of anomaly detection has arisen. Anomaly detection refers the process of training model using only nominal samples, where the model intended to differentiate nominal and anomalous samples during inference. The de facto dataset in the field of visual anomaly detection is MVTec, from Bergmann al. (2019) MVTec contains 5000

---

Published in Transactions on Machine Learning Research (06/2024)
images across 15 classes of various industrial and household objects. The field of anomaly detection currently
enjoys growing attention, with number of algorithms recently released Yu et al. 2021 Roth et al. 2022
Gudovskiy et al., 2022 Tsai et al. 2022: Wan et al. 2022b a Cao et al., 2023 Lee et al.
2022 Kim et al.
2022 Bac et al., 2023 Deng & Li 2022 Rudolph et al., 2022). The VisA datasct, by
Zou ct al. (2022)
contains 12 classes of household and industrial objects, and more challenging than the MVTee dataset.
It has not experienced the same attention as MVTec, but we propose that it has equal value due to its challenging nature and varied classes.
The PapersWithhode (PWC) leaderboard, hosted by MetaAl 2022 can be used to judge the performance of these algorithms for the MVTec dataset. However, the results from the publications on the leaderboard are self-reported, and the leaderboard is editable by anyone. This means that the positions on the leaderboard may not be representative of the relative performance of the algorithms. researchers wish to experiment with different algorithms, they currently need to download many different GitHub repositories, which con- tain different data-loading code, metric systems, styles and environments. Despite the best intentions of researchers make their code friendly to use, this overhead is significant.
The dominant metrics in the field of anomaly detection have a number of disadvantages, Image-wise AUC (I-AUC) works at the image level, and does not mcasure localisation of anomalies. Pixel-wise AUC (P-AUC) and Area Under Per-Region Overlap (AUPRO) contain noise that stems from ambignous pixel labelling, produce values in a small range, and lack a physical relation.
In light of the above issues, we make the following contributions:
1. We release a comprchensive library of anomaly detection algorithms, VisionAD. All algorithms are standardised through an API. The algorithms share the same data-loading and evaluation code, meaning that fair benchmarking is possible. The library has a focus on ease of use, with a simple system for researchers to implement their own algorithms.
We release a new metric, Proportion Localised (PL), for anomaly detection. This classifies each anomaly as discretely localised or not, and reports the proportion of the anomalies 'localised'. This is interpretable, and negates the noise associated with the labelling of ambiguous pixels. A standalone Python "PTP" package for this metric is published.
3. We undertake a benchmarking of the currently implemented algorithms on the MVTec and VisA datasets. We refer to this as the VisisnnDinduusriall benchmark. An online leaderboard will be updated as and when new algorithms are released.
1.1 Comparison of VisionAD and Anomalib
The most similar library currently available is Anomalib, from Akcay et al. (2022). As of writing, Anomalib contains 14 anomaly detection algorithms. The library contains shared code for metric and data-handling. Algorithms are also available to export in OpenVINO and ONNX, and the library comes with a number of tools such as Lightning, Gradio inference, hyperparameter sweeping, and integrated web Uls. However, these extensive features of Anomalib may result in costly implementation of algorithms, as algorithms must comply with all of these features. This may also impede rapid experimentation.
VisionAD is designed allow rescarchers to add their own algorithms very casily if they wish. This is intended to encourage VisionAD to grow fast. The user-friendly focus of VisionAD is designed to enable quick experimentation. For instance, researcher can load the provided IPython file and immediately start the modelling process, as data-loading and evaluation are handled. Implementation of the code into the library can be achieved by copying the experimental code, as the IPython file ensures the API structure is followed.
The algorithms of Anomalib are overall less performant and older than those of VisionAD, according to the PWC leaderboard (MetaAl 2022). VisionAD also contains more algorithms, meaning it is the largest library of its kind, and it is hoped will grow further new publications are released.
2

---

Published in Transactions on Machine Learning Research (06/2024)
The authors do not intend to discredit Anomalib, which has enjoyed lot of community attention and is positive addition to the field. The authors propose that VisionAD can provide value to the community by overcoming the shortfalls of Anomalib. We propose that VisionAD and Anomalib have different strengths and complement each other. Anomalib has rich features for analysis and deployment, whilst VisionAD provides lightweight alternative with a focus on rapid development and benchmarking.
1.2 Algorithms
Many different classes of algorithms exist in the anomaly detection space. Transfer learning makes use of a backbone network pre-trained on ImageNet (Deng et al. 2009), and is found in almost all recent anomaly detection algorithms. The most performant algorithms tend to make use of one or more of the concepts described below. Note the descriptions below are not exhaustive, and we refer the reader to review works by Pang ct al. (2020) and Thudumu et al. (2020) for more information.
Distillation algorithms make use of the concept that during inference, the difference between the predictions of teacher model and student model should highlight anomalous regions. Some algorithms of this class are CDO CCao et al. 2023). RSTDN (Yamada et al. 2022). AST (Rudolph et al. 2022), and EfficientAD
[atzzer et al. )024
Normalising flows map data of an arbitrary distribution to a normal distribution whilst conserving volume, using learnable bi-directional transformations. Usually, features created via a pre-trained model are mapped via normalising flows a normal distribution. In the case of anomalous inputs, the normalising flow function should fail to create a normal distribution, the function has not seen the anomalous data before. Examples of algorithuns that fall into this category are Fastflow (Yu et al. 2021 CFlow (Gudovskiy el. al.
2022 MSFlow (Zhou et al. 2024). and Fastflow+Alluub (Kim et al. 2022
Memory bank algorithms extract and process features from the training data, then filter or reduce these Teatures before saving them inlo a memory bank. During inference, Lest. features are processed and compared the features in the memory bank via some distance or hearest-neighhour measure. A larger distance implies that & feature belongs an anomalous region. Algorithms of this class are PatchCore (Roth al. 2022 PNI [Bae et al. 2023), MemSeg Yang et al. 2022). and CFA (Lee al. 2022).
Synthetic anomalies can be used to either train a model using traditional supervision or to refine the outputs using supervision. These anomalies can be entirely synthetic using a mathematical formula such as Perlin noise, or can be constructed using examples from other datasets. Algorithms which make use of this concept are PNT (Bae et al. 2023). MemSeg (Yang et al. 2022). CDO (Cao et al. 2023 and SimpleNet (Liu et al. 2023b).
Generative algorithms learn to recreate the input, with the intention that during inference, the model will fail to recreate anomalous region, therefore highlighting its existence. An algorithm which makes use of this concept is PPDM (Liu et al. 2023a
Other algorithms use novel concepts altogether. For example PFM Wan et al. 2022a and PEFM Waa et al. 2022b) train different models to recreate the same output when given an image. The difference in output should indicate anomalous regions.
2 VisionAD
We release VisionAD, a library comprising of many of the best anomaly delection algorithms, all written such that they communicate with the library using a standardised AP1. Figure [igure[[showe shows the architecture, The wrapper responsible for loading the configuration file, calling the data-loading code, calling the algorithm through the various standardised API calls, evaluating the performance using the specified metrics, and logging the results. A user guide can be found in Appendix A.
3

---

Published in Transactions on Machine Learning Research (06/2024)
2.1 Implemented algorithms
The algorithms currently implemented are: FastFlow Yu al. SimpleNet (Liu et al. 2023b). PPDM (Liu al. 2023a MSFlow
2021):
PatchCore Roth al. 2022)
(Zhou et al. 2024 PFM (Wan et al.
2022a). PEFM M (Wan et al., 2022b
CFlow (Gudovskiy et al. 2022 MSPBA (Tai et al. 2022), Memneg
Yang et al. 2022), FastFlow AAt UB Kim et al.) (022), CDO (Cao et al. 2023), CFA (Lee et al. 2022)
Reverse Distillation Deng & Li 2022), AD (Batzner et al. 2024), and AST (Rudolph et al. 2022)
As of March 2024, these comprise the top 15 algorithms with code available on the following columns of the
MVTec leaderboard, I-AUC, P-AUC, and AUPRO. We choose not to implement algorithms from scratch, due the risk of introducing errors and unfair representation. AltUB is the one exception this rule. It implemented entircly from scratch due to its simplicity.
Algorithms were reformatied from their existing structure to fit the standardised structure necessary for VisionAD. Utmost care was taken to ensure the algorithm formatting was done without errors. Im- plementations were based on official code where pos- sible, and unofficial code where necessary. MSPBA, Fastflow and MemSeg were based on unofficial code. We give due credit to the authors of these unofi-- cial repositories in the code base. Whilst we cannot guarantee implementations are perfect representa- tions, algorithms were tested against the bottle and carpet class MVTec to ensure the same results were achieved by the repository and reformatted code.
Datai Date londing parsing ougriaditton
files
Config detesslfs) modiels) typerparmmettrr maining parametere
Wrapper folcing Algoditss troughh dianstrdlocd API
Evaluzaion HAUC PLAUG
Algo
Inz
train, wwt__/_____eee sepchh
pro_evel precessing
gro ovelevel procossing
evel dotallader save_model load, load_model model
evel, ditalladerr
The authors are committed to adding more algo- rithms as they are published, and encourage re- searchers to integrate their own algorithms via pull requests.
Kavn, model
laad, model
Figure 1: Flow diagram showing the architecture of the VisionAD library.
Certain aspects of the code inherit the licences of the previous code. Therefore care must be taken regarding of the library. Due to this, the authors recommend academic and non-commercial use.
2.2 Features and
Adding algorithms intended be easy as possible, as the rescarcher needs to implement methods of Python class for full integration with the library. VisionAD contains efficient implementations of the standard metrics (I-AUC, P-AUC, AUPRO), alongside the introduced PL, which discussed below. Implementing metric is also simple. Due to the ease of writing complex configurations in Python, Python configuration files control the behavior of each run. Only minimal command line arguments are used: the path to the config file, the GPU device, and an optional run description. For development purposes, an IPython notebook is provided with an algorithm that makes random predictions, allowing researchers to experiment in an interactive environment. more detailed description of the above features can found in Appendix A, and guide getting started quickly is available in the readme.md file.
3 Proportion Localised (PL) metric
We propose a new anomaly delection metric, Proportion Localised (PL). The motivation of PL is to provide a metric that summarises the performance of an algorithm by evaluating whether it sufficiently highlights the approximate arcas of the anomalies, as opposed whether the algorithm preciscly sogments each pixel. We propose that this is more suitable for industrial applications which an algorithm functions correctly if it. 'locates' the anomaly, as opposed to whether it replicates the exact shape the pixel level. We also
4

---

Published in Transactions on Machine Learning Research (06/2024)
state the desire for metric which reports number with real physical relation, so that it can be readily understood by non-machine learning professionals.
In summary, for each anomaly, PL calculates an IoU score between each prediction heatmap and the rotated bounding box for the anomaly. To obtain the binary prediction heatmap, the contimuous algorithm output needs to be put through a prediction threshold. Here a sweep done across many prediction thresholds. For each resultant binary prediction heatmap, the proportion of anomalies with a prediction and ground truth IoU greater than the IoU limit is calculated, and the highest value amongst this array is reported as the final value. We use 0.3 as the IoU limit (we explain this choice below). Therefore PL reports the proportion of anomalies which are sufficiently found at the best threshold. The calculation is described in detail in Section 3.4 Looping through different prediction thresholds and taking the best score means the best prediction threshold is used. This means that like P-AUC and AUPRO, the algorithm is independent of the prediction threshold.
The minimum area rotated bounding box is drawn around each anomaly, and these bounding boxes act as the ground truth. This is intended to remove the noise of pixel by pixel matching. Due to this, we deliberately state that the metric evaluates anomaly localisation, as opposed to anomaly detection (image- wise), or anomaly segmentation (pixel-wise). Many experts can not agree on the classification of bordering pixels of anomalies, and due to this, we believe that an algorithm should be judged via whether it sufficiently highlights the region of the anomaly, as opposed to classifying each pixel exactly. An algorithm cannot classify every pixel correctly due to this ambiguity, and when evaluated using the traditional P-AUC and AUPRO, cannot achieve perfection. Furthermore, we emphasise that for many industrial applications, an anomaly detection algorithm functions as required if it highlights the approximate location and size of the anomaly. Unlike AUC and AUPRO, PL can report perfection if the algorithm produces an IoU greater than the required IoU limit (0.3) for every anomaly.
3.1 Weaknesses of current metrics
Table summarises the drawbacks of the current metrics and the advantages of PT. I-AUC fails to properly measure images with more than one anomaly, and fails to measure any degree of anomaly localisation or segmentation. The pixel-wise metries, P-AUC and AUPRO, amplify any ambiguity in the labelling of the pixels. Experts disagree on the individual classificatinn of certain pixels, so therefore it is an impossible task for anomaly detection algorithm to achieve perfect scores these metries. In addition to this, anomaly detection algorithms tend to output prediction heatinas at a lower resolution than the input image, meaning it is impossible for most algorithms to extract the exact shape of an irregular anomaly. This all contributes to noise in the output of these metrics. We argue that anomaly detection algorithms function as intended if they highlight the approximate area of the anomaly.
The pixel-wise metrics also suffer from vastly optimistic scores due to the large imbalance of anomalous and regular pix-
P-AUC/AUPRO
PL advantages Less effected by pixel-level noise through BB use
drawbacks
els. The P-AUC metric has a low range of output. On an imbalanced dataset, a bad score may be 0.9 and a good score may be 0.99. This undesirable, it is preferable if a bad score is and a good score is 1. The sloU metric (Chan et al. (022]) is innovative, and recognises the draw- backs of P-AUC and AUPRO by func- tioning at the anomaly level. However it does have a number of downsides. It is slow to run, due to the looping where the prodictions have to be scparated into non-overlapping regions for many different thresholds. In addition, the multiple counting of false positive prediction area for different anomalies in a single image is over-penalisation.
Pixel-level matching contains ambiguity and noise
Reports an abstract number (no physical relation)
Reports number with real physical relation
Small output range, usually predicts between 0.8 and 1.0
Larger output range, can predict between 0.0 and 1.0
Lack of undenstandability for non-ML professionala
Readily understandabll by nou-ML professionals
Table 1: Drawbacks of P-AUC and AUPRO, and advantages of PL.
5

---

Published in Transactions on Machine Learning Research (06/2024)
Finally, none of these metrics readily understood by users without machine learning expertise. They do not report number with a physical relation.
3.2 Strengths of the introduced PL
PL directly reports the proportion of anoma- lies localised. Engineers, inspectors and doc- tors are likely interested in what anomalies have been 'localised' (found) by an algorithm, where localised is a binary descriptor, and is true if the given algorithm has highlighted enough of the anomaly, and false if not.
0.8
Metric score
0.6
A major advantage of PI. is that it is read- ily understandable by non-ML professionals. One could understand that PLs 0.1, 0.5, and 0.99 mean that 10%, 50,, and 99% of anoma- lies are sufficiently located. We propose that these numbers could be used to make in- formed decision in an industrial and medical setting. We emphasise that the friendliness of this metric to industry professions is a major advantage.
0.4
Pixelwise_AUU (0.93-1)
0.2
Pixelwise AUPRO (0.72-0.99)
PL (0.29-0.97)
mean (ordered via)
0.0
Ordered algorithmiclass outputs
Figure 2: Comparing the behaviour of PT. to the traditional
metrics, (5th 95th percentile)
PL uses rotated bounding box labels, which negates the issuc of noise in pixel-wise match- ing, algorithms are not judged whether than locate the exact pixels. Instead they are judged on whether they sufficiently locate the bounding box area of the anomaly. This allows algorithms to output heatmaps at a lower resolution to the image without penalisation (many algorithms do this). It also removes the requirement that algorithms must replicate the exact shape of an anomaly. This brings anomaly detection accordance with the object detection field, which uses IoU to mcasure overlap to discretely classify whether an object is found or not (mAP, AP familics) [Lin et al. 2014 Redmon al., 2015; Girshick et al. 2014
Finally PL uses larger range of outputs in comparison to the other common metrics. Behaviour of the PL metric is compared to P-AUC and AUPRO in Figure
3.3 Behaviour of PL
Figure shows the behaviour of the PT metric compared to the P-AUC and P-AUPRO metric. Each vertical set three points represents the P-AUG, P-AUPRO, and PL outputs of a specific algorithm and dataset class combination. These are ordered based on their mean value across these metries. The classes used are from MVTec and VisA. The relative value of each metric of importance here, not the performance of the algorithms. Firstly note that P-AUC predicts in much smaller range of values, with most predictions greater than 0.95. AUPRO produces greater range of values, but the values still generally between 0.8 and 0.95 (with small number of algorithm class combinations greater than this). Meanwhile PT. predicts across a much greater range (0.1-0.98), and has much more linear behaviour.
3.4 Calculation of PL
Algorithm [ outlines the calculation. For each anomaly ground truth and corresponding prediction, the algorithm loops through 25 lincarly spaced prediction heatmap thresholds, and creates 2d array of shape anomalees,,s, 25). For each threshold, the proportion of anomalies with an IoU greater than the IoU limit (0.3) is calculated. The final score is the maximum of these.

---

Published in Transactions on Machine Learning Research (06/2024)
Algorithm 1: Calculation of Proportion Localised. Pred set and GT_sst set are the groups of prediction heatmaps and ground truth images respectively. Details of the sub functions can be found in Appendix
C
function Calculate (pred set, GT set)
2
IoUs 4-
for prediction heatmap, GT image in pred set, GT set do BBs ExcraccostatoBBuuninggreeee (GT (mmgem
BBs + Scale8Bs (BBs)
BBs + MergeDverlaappingBee (BBs)
filters CreatelllarresttaFFltteee (BBa)
for single_BB, filter in BBa, filters do
comb.filter +- LogicalOr (single _BB _BB, filter)
10
pred filtered comb filter prediction heatmap loUs.per.threshold e-
11
for threshold in thresholds do binary-pred pree_fffltereddd threshold
13
14
IoU + CalaaoU pred, single BB)
Iolsapper.Chreshold append (IoU)
16
IoUs appenddiola..prr.threeoooo per_theesoold)
(IoUs
IaUs array (IoUs)
ret +- IoUs > IoU_limit 18
19
ret + MeanAxis0 (ret)
ret Max(ret)
return ret
There are a few nuances which make the process work. Firstly, if labels are pixel-wise, they need to be converted to freely-rotated bounding boxes. This is easily achieved by creating the minimum area bounding box over every non-overlapping ground truth region. The bounding boxes are freely rotated to best match the shapes of the anomalies, and to ensure the metric independent of image rotation.
It was found that during inference very small anomalies cause activity on the heatmap of an anomaly label. This is because most anomaly detection al- gorithms output prediction heatmaps at resolutions lower than the input image, e.g. 32 x 32 in comparison to 256 x 256.
much larger than the
2
If an anomaly is very thin, is near-impossible for the al- gorithm to achieve the desirable ToU, even if the algorithm prediction is sufficiently centered on the anomaly. We do not believe this should be penalised, in this case the algorithm has still located the anomaly. Therefore in the PL calculation, the bounding boxes are enforced to have minimum width and height in relation to the respective image dimension. A value of 1/ (12.5%) was found to work well. This allows coarse prediction heatmaps
Figure 3: Demonstration of splitting of im- ages with more than one anomaly. Columns left to right: input image, raw ground truth, su subsequent splits (blue bounding box mask, groy ignored arca).
judged fairly. This minimum dimen- sion enforcement of 1/8th of the image dimension part of the introduced PL metric. The scaling is shown graphically via the difference between the first and second rows of the examples in Figure This scaling results in small boost to the PL scores, which we believe is justified, as it prevents penalising relatively big predictions centered on small anomalies.
The metrics handle an arbitrary number of anomalies one image by splitting the prediction heatmap at the boundaries between the centers of each anomaly (cach pixel of the heatmap designated to its nearest anomaly). For given anomaly x, every pixel which is not closest to the anomaly center point, or inside the bounding box, is zeroed before the ToU calculation. The process is shown graphically in Figure 3 via the

---

Published in Transactions on Machine Learning Research (06/2024)
3rd 4th and 5th rows. In the subsequent steps, each anomaly and heatmap region is handled independently, meaning the metric functions at the anomaly level, as opposed to the image pixel level
With the process explained above, a group of small and close anomalies would result a large number overlapping bound- ing boxes, which undesirable. If there significant overlap in the scaled bounding boxes, the anomalies should be merged and treated as single anomaly. Bounding boxes which overlap by more than 1/3rd with another are merged. The underlying pixel-wise ground truths are merged, and new rotated bound- ing box is drawn and scaled. This process is shown graphically in Figure 4 and explained in more detail in Appendix C.
BB:
PL requires the choice of an IoU limit. Only anomalics with prediction and ground truth loU greater than this limit are classified as located. We use and recommend an IoU limit of 0.3, although a researcher can use the value they see the most fit for their problem. We justify the choice of 0.3 in two ways. Firstly, we take precedence from the AUPRO calcula- tion, from Bergmann et al. (2019), which calculates the area under the curve up to a false positive rate of 0.3. Secondly, we use Figure 3 to undertake a visual analysis of predictions and bounding box caleulations across the MVTec and VisA datasets. We wish to choose loU limit which excludes un- satisfactory overlap between the prediction and label, whilst including satisfactory overlap. For simplicity, we wish to use a value which is a multiple of 0.05. We believe 0.3 is the best value which separates salisfactory and con-satisfactory over- laps. Part way of the third row of Figure and onwards shows satisfactory overlap between the prediction and bounding box, whilst the images before it do not. The value of 0.3 may be perceived as low, but this allows some inevitable mismatch be- tween the bounding box target and the prediction. Note that some predictions produce what we perceive to be good overlap, but still have IoU values between 0.4 and 0.6. Whilst the published PL uses 0.3 as the limit, the researcher can choose any value, and this can be represented by PLxx, iee. PL50 for 0.5 limit, and PI.25 for 0.25 limit.
12 DDs
EBc
+
Figure 4: Demonstration of merging over- lapping anomalics. Columns left right: input image, raw ground truth, bounding boxes around non-overlapping regions (no merging), bounding boxes after merging.
The metric does not require the of any regular testing images. The reader may assume that the PL metric does not penalise false positive predictions. PL does penolise false positive predictions. As the anomalous images still contain a significant amount of non-anomalous regions, if an algorithm were to predict non- anomalous regions as anomalous, the loU of the closest anomaly would be reduced such that it would not meet the loU limit, and would not be classified as localised by PL, therefore reducing the final score.
A Python package to calculate PL also relcased allow rescarchers to quickly integrate these metrics into their code. The metric is coded efficiently. It takes approxinately the same amount of time to calculate PL it takes to calculate P-AUC AUPRO.
4 VisionADIndustrral (VADI) benchmark
4.1 Results
benchmark. For the respective benchmark, this is the mean value of the pooled class scores the MVTec and VisA datasets. Effectively we combine the datasets. The scores for each respective class in both datascts are added and then we divide by the total number of classes (27). This ensures each class is weighted equally. We do not caleulate the individual mean of the MVTec and VisA datasets, and take the mean of the means, because this would result in the Vis dataset classes having
We introduce the
8

---

Published in Transactions on Machine Learning Research (06/2024)
loU: 0.000
IoU: 0.000
loU: 0.000
IoU: 0.000
loU; 0.001
loU: 0.017
loU: 0.030
IoU: 0.061
lol: 0.092
loU: 0.119
loU: 0.1.36
laU: 0.149
laU: 0.159
lol: 0.170
lol: 0.191
loU: 0.211
lol:: 0.245
loU: 0.270
loU: 0.289
loU: 0.307
loU:
loU: 0.331
lol: 0.339
loU: 0.344
loU: 0.354
loU: 0.359
loU: 0.364
lol: 0.374
0.381
loU: 0.387
lol: 0.404
IoU: 0.416
loU: 0.425
loU: 0,433
loU: 0.447
loU: 0.454
loU: 0.459
loU: 0.471
lol: 0.476
loU: 0.482
IoU: 0.491
loU: 0.496
loU: 0.505
loU: 0.508
loU: 0.514
loU: 0.518
loU: 0.521
loU: 0.529
loU: 0.538
loU: 0.541
loU: 0.548
loU: 0.557
loU: 0.564
loU: 0.571
loU: 0.577
loU: 0.579
loU: 0.585
loU: 0.595
laU: 0.601
loU: 0.610
loU: 0.617
IoU: 0.625
loU: 0.634
loU: 0.641
loU: 0.645
loU: 0.654
laU: 0.659
loU: 0.666
loU: 0.677
loU: 0.687
Figure 5: Demonstration of ordered IoU values between prediction (red) and bounding box label (bluc). Ran- dom images from various MVTeec and VisA classes, with predictions created using the PatchCore algorithm [Roth al. 2022). We recommend an loU limit of 0.3, as propose that from this point onwards (the third row), the prediction and bounding box have satisfactory overlap. Note the gray area means ignored pixels, due to there being other closer anomalies in the given image (see above for explanation).

---

Published in Transactions on Machine Learning Research (06/2024)
1.2
0.00
0.97
0.97
0.96
0.55
0.88
10.5
1095
0.95
1.2
0.00
0.90
00.983
0599
0.77
0.970
0,970
H4 (1944
40
1.0
1.0
0.8
0.8
1XUC
PAUC
0.6
0.2
0.2
0.0
0.0
podm
revense distilition
efficirtad
patchcore-
mallow
pim
clow
simplanet
pelm
fastllow
mstlow
revense distiiition
patchcore.
podm-
cdo
odhm
resditio sto0
pmn
efficerta
mamseg
fasifbou
(a) Algorithm rankings based the image-wise AUC
(b) Algorithm rankings based the pixel-wise AUC
metric.
metric.
12
1.2
1.-
977
0.534
0.a
9.87
888
0.66
0.883
0.888
0.837
0.852
0.
1.0-
0.75g
0.7
0.33
0.736
0.76
0.72
0.708
0.69
0.689
0. 0.68
0.674
1.674
0.666
0.8
0.8
AUFRO
0.387
0.6
5 0.6
0.4
0.4
Rovers disilattoon
spom
cd0
pfm
fastlow
patchooee
pafm
reathow 33.00
cl
cllow
matlow
simpleret
efficiensied
memseg
patcheore
reverse_se distilaation
fasttlow
spdi-
fastllow alt
cl
simplennt
pfm
<do
choo
msflow
afficiantad
(c) Algorithm rankings based on the AUPRO metric.
(d) Algorithm rankings based on the PL metric.
Figure 6: We present the VADI results. This is the mean of the pooled classes of the MVTee and VisA datasets (each class of each dataset weighted equally). As measure of variance, we also show the standard deviation of the result across each class as error bars.
greater weighting due the lower number of classes in VisA (12) compared MVTec (15). We propose that the VADI benchmark produces the most comprehensive and fair set of results currently available for industrial anomaly detection.
We trialled the top 15 anomaly detection algorithms which were selected by the criteria described in Sec- tion 2.1 Each algorithm was given a training time of two hours (evaluation and metric calculation is not counted). There were no extra image augmentations carried out training (we leave a similar run with maximal training augmentations for future work). Each algorithm was run using the default paramoters from the publication. The hyperparameters were kept. the same across each class (this may disadvantage algorithms which change hyperparameters based class category, e.g. texture object). We do this because we believe that amending hyperparameters for a certain class makes use of forbidden test set infor- mation. When an industry professional is presented with a new problem, they would not have the luxury of amending the parameters based on the test set results (unless a large number of labelled anomalies are available).
Figure 0 shows the VADI scores for each algorithm and metric. Tables presenting the class breakdown of these results across all algorithms shown in Appendix B.
10

---

Published in Transactions on Machine Learning Research (06/2024)
The VADI benchmark finds that PPDM, Fastflow, Reverse Distillation, and PatchCore are the best anomaly detection algorithms for I-AUC, P-AUC, AUPRO, and PI, respectively. Due to the advantages of PT. mentioned above, we use PL version as final ranking. Therefore we find PatchCore to be the best algorithm currently available. We note that according to the PWC publicly available leaderboard (MetaAl 2022). PatchCore is in the 11th position for I-AUC, and 23dd position for P-AUC, with no result for AUPRO (April 2024). We use this example to show the importance of not relying on self-reported results to judge algorithms.
The error bars of Figure G show the standard deviation taken across the classes. The standard deviation bars of P-AUC show small output of this metric as the bars are very small. The error bars show that PL has the greatest range of output, highlighting an advantage of PL.
A final takeaway from Figure p is that the performance of the algorithms are relatively close to each other. There is not one dominant algorithm in the field of anomaly dctection. This implics that all these families of algorithms warrant further research.
5 Conclusion and Future Work
We release the VisionAD anomaly detection library intended for fair benchmarking and algorithm develop- ment. We also release a novel metric Proportion Localised, intended to give researchers more intuitive metric by classifying each anomaly as localised not. We relcase the Vision.ADIndustraal (VADI) bench- mark, ranking the top 15 algorithms available on the tradit and introduced metric(s) via the performance on the MVTec and VisA datasets. We find that PatchCore scores the highest for the PL metric, and therefore we find that PatchCore is the current best algorithm for industrial anomaly detection.
A limitation of VisionAD that many algorithms are formatted to work in one system. This can mean certain algorithms are written less naturally than they may be written when standing alone. Formatting different algorithms to fit into one API pattern is a necessary compromise to achieve the benefits VisionAD offers. A limitation of the VADI benchmark is the lack of hyperparameter sweeping and training augmentations (although we used the recommended hyperparameters for each algorithm). We leave longer experiments, training augmentations, and hyperparameter tuning to future iterations of the VisionAD leaderboard.
Future work includes adding more algorithms to VisionAD. Features could be added such as memory logging and a results visualisation dashboard. Vi is backed by a rescarch group committed to improving and maintaining the library.
Funded by Rolls-Royer Defence; supported by UK Research and Innovation (UKRI), grant reference number EP /S00033437/1.
11

---

Published in Transactions on Machine Learning Research (06/2024)
References
Samet Akcay, Dick Ameln, Ashwin Vaidya, Barath Lakshmanan, Nilesh Ahuja, and Utku Genc. Anomalib: A deep learning library for anomaly detection. In 2022 IEEE International Conference on Image Processing (ICIP), pp. 1706-1706777 1710. IEEE, 2022.
Jaehyeok Bae, Jae- Han Lee, and Seyun Kim. Pni Industrial anomaly detection using position and neigh- borhood information. arXiv preprint 10.48550/ar /ar/500/////... 2221111//1 2023.
Kilian Batzner, Lars Heckler, and Rebecca König. Elficientad: Accurate visual anomaly detection millisecond-level latencies. In 2024 IEEE/CVE Winter Conference Applications of Computer Vision (WACV), pp. 127-137, 2024. doi: 10.1109/ WWNAAVV7701.202..000
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad a comprehensive real- world dataset for unsupervised anomaly detection. In 2019 IEEE/CVF Conference on Computer Vision and Pallern Recogrition (CVPR), 06 2019. doi: 111100//CVR..229.00095
Gary Bradski and Adrian Kaehler. Openev: Open source computer vision library, 2020. URL https: //opencv.org org/ [Online; accessed 2024-04-18).
Yunkang Cao, Xiaohao Xu, Zhaoge Liu, and Weiming Shen. Collaborative discrepancy optimization for reliable image anomaly localization. IEEE Transactions on Industrial Informatics, pp. 1 10, 2023. doi:
Robin Chan, Krzysztof Lis, Svenja Uhlemeyer, Hermann Blum, Sina Honari, Roland Siegwart, Pascal Fua, Mathieu Salzmann, and Matthias Rottmann. Segmentmmeifyoucan:: A bench- mark for anomaly segmentation. In J. Vanschoren and S. Yeung (ods.), Proceedings of the Newral Information Processing Systems Track Dalasels and Benchmarks, volume 1. Curran, 2021. URL hatt::///aataattt--oomooorrrrppp proceedn neurips wwtpcpprppaaaaaeeeeeee www.ffllleaapaeee patps///ppp/////////// pdf
Hanqiu Deng and Xingyu Li. Anomaly detection via reverse distillation from one-class embodding. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9727-9736, 2022. doi:
Jia Deng, Wei Dong, Richard Socher, Ti-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 580-587. IEFE, 2014.
Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizing flows. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1819- 1828, 2022. doi: 10.1109, 1199WWCC514588... 2022000188.
Yeongmin Kim, Huiwon Jang, Dongkeon Lee, and Ho-Jin Choi. Altub: Alternating training method update base distribution of normalizing flow for anomaly detection. ArXiv, abs/2210.149133, 2022. URL
https://api. semanticacholar oamratttieachollar.ooo//Cooolllo : 2533157302
Sungwook Lee, Seunghyun Lee, and Byung Cheol Song. Cla: Coupled-hypeespheeee feature adaptation for target- anomaly localization. IEEE Access, 10:78446-784555 2022. doi: 10.1109/ACCESS. 2022.3193699.
Tsung-Y Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014. URL h /attps://api. semanticscholar....... htt.a/acceeaaaaa.ooooooooaoaaaaa
12

---

Published in Transactions on Machine Learning Research (06/2024)
W. Liu, H. Chang, B. Ma, S. Shan, and X. Chen. IEFE/CVP Conference on Compuler Vision and Pallern Recognition (CVPR), pp- 12147-12156, Los Alamitos, CA, USA, jun 2023a. IEEE Computer Society, doi:
anomaly detection. In 2023
200.:27722.222.1116600 URL
/ttps://do..eeeccoptteeeee.
org/10...0....o 1000/////00/// 2023. 011699
Zhikang Liu, Yiming Zhou, Yuansheng Xu, and Zilei Wang. Simplenet: A simple network for image anomaly detection and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), pp- 20402-2011 20402, June 2023b.
MetaAI. Papers with code mvtec ad benchmark (anomaly detection). https:///ppprree.thcccd. camf//oot//aammlllyydddettrorrrr
2022. URL http:://papersrttuooee..c.oo// aaoomaly-dettection-------eee avteec--d OOuline; accessed 2024-04-18).
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton van den Hengel. Deep learning for anomaly detection: A review. CoRR, abs/2007.025000 2020. URL https://arxii: httpaaaaavv.rrrrarrr/aaaeaaa org/abs/2007 00t.//0000000000000
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. CoRR,
2015. URL
02640
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. To- wards total recall in industrial anomaly detection. In 2022 IEEE/C Conference on Computer Vision doi: D0.1109/CVPR52668..222
and Pallern Recognition (CVPR), pp. 14298 14308, 2022.
Marco Rudolph, Tom Wehrbein, Bodo Rosenhahm, and Bastian Wandt. Asymmetric student-teacher networks for industrial anomaly detection. Xixiv preprint (0.48550/ARX/// www...22............ 2022. doi: 10.48550/ARB ARXIV 2210.07829. URL https: wwt./rgg///////eeee/o/eeeeeeeeee http//bb/2211.... 07829.
Srikanth Thudumu, Philip Branch, Jiong Jin, and Jugdutt Singh. A comprehensive survey of anomaly detection techniques for high dimensional big data. Journal of Big Data 7, 07 2020. doi: 10.1186/
>=10537-020-00320xx.
Chin-Chia Tsai, Tsung- Hsuan Wu, and Shang-Hong Lai. Multi-scale patch-based representation learning for image anomaly detection and segmentation. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 3065-3073, 2022. doi: 10.1109/WACV51
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, et al. Scipy: Open source scientific tools for Python, 2020. URL htttps://ww.sccpp/ org/ [Ouline; accessed 2024-04-18).
Qian Wan, Liang Gao, Xinyu Li, and Long Wen. Unsupervised image anomaly detection and segmentation based on pre-trained feature mapping. IEEE Transactions on Industrial Informatics, 2022a. doi: 10.1109/ TIL.2022.3182388
Qian Wan, Cao YunKang, Liang Gao, Shen Weiming, and Xinyu Li. Position encoding enhanced feature mapping for image anomaly detection. In 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE), 2022b.
Shinji Yamada, Satoshi Kamiya, and Kazuhiro Hotta. Reconstructod studentt and discriminative networks for anomaly detection, 2022. URL
method Minghui Yang. Peng Wu, Jing Liu, and Hui Feng. Memseg: A semi-supervised method for image surface
defect detection using differences and commonalities, 2022. URL http:::s////aiiv.rrrtt http:/ivv.rr//ab//////oa habbs/2205.000 00908
Jiawei Yu, Ye Zheng, Xiang Wang, Wei Li, Yushuang Wu, Rui Zhao, and Liwei Wu. Fastflow: Unsupervised
anomaly detection and localization via 2d normalizing flows. CoRR, abs/2111.07677,, 2021. URL https:
//axxiv.org/ abs//bss...... 07677
Yixuan Zhou, Xing Xu, Jingkuan Song, Fumin Shen, and Heng Tao Shen. Msflow: Multiscale flow-based framework for unsupervised anomaly detection. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-14, 2024. doi: (0.1109/TNNISS.0233.3333
13

---

Published in Transactions on Machine Learning Research (06/2024)
Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. Spot-the-diffrence self- supervised pre-training for anomaly detection and segmentation. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022,
pp. 392-408, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-20056-4
14

---

Published in Transactions on Machine Learning Research (06/2024)
Appendix A
Adding algorithms
The ease of adding algorithin is intended to be one of the major strengths of VisionAD, The process is described below.
Each algorithm is built from a class called { {algo_nnnnl name) Wrapper, which inhorits the ModelWrapper class. Using the provided template, a researcher only needs implement six methods for the algorithu to work. Fach method discussed below:
Initialisation: init (sclf)
Initialise the model and optimisers.
Training: train one eseocc(sslll) Train the model using the self.dataloader train (automatically attached via the wrapper).
Pre-eval processing: prev reve_ processing(self)
Optionally undertake any necessary pre-processing before evaluation, using access to self.dataloader train. Left 'pass' in most algorithms. This cannot be done at the start of eval outputs dataloader, eval_outar ttttputs_datllaaeer dataloader may be called disk, where the train generator would not be available.
new class instance in the case of loading the model from
Evaluation: eval _oal_outuust_. ceiiputs_ddaalladeresslll generator, len generator) This method called twice, once with a generator of regular test images, and once with a genera- tor of anomalous test images, To avoid data leakage, this method does not have access to ground truths paths. The method must iterate over the passed generator, saving to memory an anomaly map and score for each image. Returns tuple of two dictionaries: dictionary of anomaly maps: map {' f'anomaly map_ method a': torch.tensor/np.aare image scores torch. wewp.tteno///.rrroeo of shape (no. images]]) Notably, the system allows any number of different methods for creating image scores and heatmaps for a given algorithm. For instance, the researcher may wish to trial reducing a feature channel via mean, standard deviation, and max, without wishing to rewrite three algorithms. The metric code will provide results for each key and tensor passed through.
of shape (no. thoiim height, width)), and dictionary
Saving: save Lave_model(set)) location)
Saves the parameters of the model to given location.
Loading: load mseel location)
Load the parameters of the model from given location.
A trivial example of a random classifier included to help rescarchers with the small but necessary template code.
Metrics
VisionAD has comprehensive list of metrics, including the standard I-AUC, P-AUC, and AUPRO, alongside the introduced PL. However, researchers can add metrics as they desire. A researcher only needs to implement a function which accepts an array of targets, predictions, labels and image scores, and returns a single value.
Run configuration files
Config files are Python files. Each file contains a model list variable, which is list of dictionaries each containing the information to run model (algo_ name, model parameters, epoch, n epochs opochs_test).. Each file also contains a dataset list, which contains strings which match the dataset keys (discussed below). The wrapper will run each model on each dataset in a given config file.
The choice of Python config file over Yaml done because gives the rescarcher the expressiveness of Python when creating experiments. For instance, each algorithm contains default set of model parameterr, and these be imported into different config files whenever this algorithm is (with changes made when
15

---

Published in Transactions on Machine Learning Research (06/2024)
necessary). Looping also allow one parameter to be changed whilst other are kept the same via copying the model dictionary inside model list X times, but changing the target hyperparameter and description each time. Many other similar tricks are available which save the researcher the painful process of copy and pasting parameters.
Basic and complex examples configuration files are given to allow researchers to get started quickly.
Data configuration files
The dataset file datasets/ cooniiuureeddaaaeeeeee dataset contains a variable called datasets, which is a dictionary of dataset keys, where the item of each key a dictionary with various dataset information (training image path, testing image path.... ..). These should be self explanatory. These dataset keys should match the strings in the run configuration files. To add dataset user needs to add dataset key and fill the necessary paths.
Command line use
The wrapper is designed to be as easy as possible run, with minimal command line arguments. The call python3 run.py config configs/mi figgs//ym_//////mmmmm//// /my ezperiment-py device 'cuda:0" is sufficient to load config file and overwrite the gpu device. The config file contains everything else needed.
IPython experimentation
Enabling easy experimentation is a goal of VisionAD. Early stage algorithm development is often done IPython environment. The necessary template code can be loaded into TPython environment such Jupyter notebook. The researcher experiment with the algorithm class whilst the data loading and evaluation code is handled. A starter notebook is provided to demonstrate this using the trivial random classifier.
Other features
As mentioned above, the library does not give algorithms access to the ground truths during training, to ensure data Icakage is not possible. The wrapper also contains number of other bug checking features such as ensuring the outpuls of the algorithms are the right dimensions. The wrapper also measures total training time, training time per image, total inference time, and inference time per image. The wrapper allows all metrics be logged to Weights and Biases (Wandb) if desired, and code also provided to pull these results from Wandb and parse them into the tables shown in this publication.
Some algorithms require synthetic anomalies. To facilitate this, we allow callback to be ran on the training data before it is sent through the dataloader. The training dataloader outputs three items, the training image, the result of this callback, and the image file name. The default of the callback result is to return 0, which would be passed to the algorithm and ignored. However the case that synthetic anomalies are added to the training images, this callback could return the corresponding mask, which the algorithm can use.
All VisionAD algorithms work Windows and Linux, CPU and GPU. Currently all algorithms fit RTX 3090 24Gb GPU.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Appendix B
Image-wise AUC MVTec
Table
we show the results using the imago-wisc AUC metric on the MVTec dataset.
efficientad
reverse distillation
fastflow
msflow
ppdm
fastflow altub
patchcore
bottle
0.999
1
cable
0.979
0.982
0.986
0.997
0.979
0.98
0.998
capsule
0.987
0.984
0.995
0.991
0.982
0.99
0.991
carpet
0.987
1
0.988
0.987
0.997
0.989
0.978
grid
0.998
0.994
0.99
0.927
0.994
hazelnut
1
1
1
leather
1
1
1
metal nut.
1
0.999
0.997
pill
0.996
0.974
0.979
0.962
0.968
0.965
0.975
screw
0.945
0.986
0.945
0.926
0.982
0.923
0.974
tile
1
toothbrush
1
1
0.981
1
0.975
0.914
transistor
1
0.977
0.998
0.999
0.995
wood
0.998
0.996
0.998
1
1
0.993
zipper
0.997
0.982
0.996
0.997
0.992
0.993
0.995
mean
0.992
0.992
0.991
0.99
0.988
0.988
0.987
cfa
pfm
pefm
cflow
simplenet
memseg
edo
bottle
0.998
0.995
0.987
cable
0.998
0.996
0.996
0.978
0.993
0.957
0.908
0.957
capsule
0.958
0.958
0.945
0.963
0.99
0.97
0.83
0.895
carpet
0.984
0.99
0.996
0.999
0.976
0.927
0.999
0.994
grid
0.96
0.991
0.992
0.953
0.728
0.938
0.952
0.759
hazelnut
0.998
1
0.998
1
0.992
0.991
0.959
leather
1
1
1
1
1
metal nut
0.999
1
0.997
0.999
1
0.992
0.965
pill
0.986
0.976
0.987
0.948
0.974
0.91
0.918
0.877
screw
0.868
0.962
0.915
0.934
0.885
0.997
0.999
0.735
tile
1
0.997
0.997
0.999
1
1
1
0.996
toothbrush
1
0.886
0.892
0.967
0.906
1
0.853
0.961
transistor
0.999
0.978
0.996
0.904
1
0.969
0.921
0.976
wood
0.988
0.996
0.997
0.992
0.995
0.999
0.992
0.971
zipper
0.985
0.969
0.97
0.975
0.996
0.778
0.971
0.98
mean
0.981
0.98
0.979
0.973
0.963
0.962
0.955
0.934
Table 2: VisionAD results on the MVToc dataset based on the imago-wise AUC metric.
17

---

Published in Transactions on Machine Learning Research (06/2024)
Pixel-wise AUC
MVTec
Table
we show the results using the pixel-wise AUC metric the MVTec dataset.
cfa
fastflow
msflow
patchcore
pefm
reverse distillation
cflow
bottle
0.99
0.989
0.985
0.988
0.985
0.987
0.987
cable
0.989
0.977
0.986
0.986
0.98
0.979
0.975
capsule
0.99
0.991
0.989
0.991
0.983
0.987
0.99
carpet
0.985
0.989
0.99
0.99
0.99
0.992
0.991
grid
0.978
0.993
0.981
0.986
0.988
0.991
0.974
hazelmut
0.987
0.983
0.982
0.989
0.991
0.991
0.989
leather
0.995
0.996
0.996
0.995
0.994
0.995
0.996
metal nut
0.985
0.988
0.987
0.99
0.974
0.976
0.985
pill
0.985
0.978
0.988
0.981
0.979
0.984
0.989
screw
0.986
0.989
0.983
0.994
0.991
0.997
0.988
tile
0.982
0.968
0.981
0.972
0.962
0.964
0.967
toothbrush
0.992
0.983
0.982
0.989
0.988
0.991
0.984
transistor
0.984
0.98
0.983
0.937
0.973
0.936
0.948
wood
0.962
0.969
0.952
0.959
0.96
0.959
0.953
zipper
0.99
0.986
0.985
0.99
0.984
0.987
0.979
mean
0.985
0.984
0.983
0.982
0.981
0.981
0.98
cdo
ppdm
fastflow altub
efficientad
simplenet
pfm
ast.
memseg
bottle
0.991
0.989
0.98
0.957
0.981
0.984
0.928
0.958
cable
0.97
0.981
0.964
0.984
0.975
0.969
0.961
0.794
capsule
0.98
0.982
0.987
0.985
0.99
0.985
0.97
0.926
carpet.
0.992
0.992
0.985
0.964
0.984
0.989
0.984
0.93
grid
0.986
0.916
0.99
0.976
0.902
0.988
0.97
0.895
hazelnut
0.992
0.991
0.974
0.978
0.982
0.991
0.981
0.826
leather
0.996
0.993
0.995
0.991
0.994
0.994
0.985
0.988
metal nut
0.985
0.98
0.978
0.967
0.988
0.975
0.959
0.771
pill
0.983
0.983
0.974
0.982
0.986
0.974
0.953
0.887
screw
0.985
0.995
0.979
0.987
0.99
0.99
0.983
0.885
tile 0.974
0.962
0.95
0.979
0.965
0.963
0.937
0.968
toothbrush
0.988
0.992
0.98
0.98
0.985
0.987
0.984
0.965
transistor
0.911
0.965
0.974
0.985
0.976
0.891
0.927
0.706
wood
0.973
0.956
0.948
0.953
0.942
0.961
0.923
0.909
zipper
0.982
0.977
0.979
0.961
0.984
0.983
0.967
0.891
mean
0.979
0.977
0.976
0.975
0.975
0.975
0.961
0.887
Table 3: VisionAD results on the MVTec dataset based on the pixel-wise AUC metric.
18

---

Published in Transactions on Machine Learning Research (06/2024)
AUPRO MVTec
Table
we show the results using the AUPRO metric on the MVTec dataset.
reverse distillation
pefm
cfa
cdo
fastflow
patchcore
pfm
bottle
0.949
0.941
0.941
0.951
0.924
0.938
0.934
cable
0.913
0.91
0.909
0.89
0.84
0.914
0.893
capsule
0.923
0.909
0.936
0.909
0.938
0.937
0.922
carpet
0.957
0.957
0.939
0.962
0.946
0.943
0.95
grid
0.955
0.953
0.897
0.95
0.956
0.92
0.951
hazelnut
0.95
0.955
0.926
0.949
0.941
0.943
0.952
leather
0.979
0.983
0.972
0.984
0.988
0.962
0.985
metal nut
0.941
0.937
0.923
0.942
0.912
0.94
0.933
pill
0.967
0.961
0.961
0.964
0.94
0.956
0.96
screw
0.978
0.94
0.922
0.92
0.932
0.966
0.942
tile
0.846
0.841
0.901
0.893
0.875
0.861
0.845
toothbrush
0.912
0.878
0.905
0.873
0.879
0.875
0.878
transistor
0.805
0.843
0.884
0.768
0.869
0.808
0.714
wood
0.909
0.924
0.892
0.932
0.933
0.883
0.906
zipper
0.946
0.937
0.951
0.931
0.933
0.954
0.937
mean
0.929
0.925
0.924
0.921
0.92
0.92
0.913
ppdm
cflow
msflow
fastflow altub
simplenet
elficientad
ast.
memseg
bottle
0.952
0.903
0.899
0.881
0.871
0.782
0.814
0.786
cable
0.903
0.877
0.866
0.813
0.872
0.841
0.822
0.425
capsule
0.889
0.918
0.906
0.923
0.916
0.916
0.841
0.594
carpet.
0.952
0.95
0.932
0.936
0.901
0.836
0.935
0.704
grid
0.7
0.895
0.897
0.937
0.709
0.901
0.848
0.43
hazelnut
0.945
0.94
0.917
0.931
0.91
0.905
0.876
0.652
leather
0.972
0.984
0.977
0.985
0.955
0.94
0.954
0.786
metal nut
0.942
0.883
0.863
0.89
0.894
0.778
0.857
0.569
pill
0.953
0.946
0.935
0.907
0.933
0.889
0.802
0.197
screw
0.968
0.938
0.894
0.84
0.941
0.934
0.902
0.197
tile
0.829
0.837
0.88
0.831
0.822
0.879
0.79
0.734
toothbrush
0.913
0.856
0.812
0.795
0.826
0.831
0.824
0.524
transistor
0.822
0.775
0.864
0.784
0.824
0.899
0.748
0.401
wood
0.873
0.898
0.872
0.917
0.814
0.803
0.847
0.686
zipper
0.925
0.889
0.928
0.901
0.922
0.813
0.865
0.574
mean
0.903
0.899
0.896
0.885
0.874
0.863
0.848
0.551
Table 4: VisionAD results on the MVTec dataset based on the AUPRO metric.
19

---

Published in Transactions on Machine Learning Research (06/2024)
PL MVTec
Table
we show the results using the introduced PL metric on the MVTec dataset.
reverse distillation
patchcore
fastflow
fastflow altub
cfa
pefm
cdo
bottle
0.971
0.971
0.956
0.965
0.956
0.971
0.985
cable
0.904
0.936
0.8
0.805
0.912
0.928
0.832
capsule
0.763
0.842
0.798
0.806
0.754
0.57
0.544
carpet
0.914
0.903
0.914
0.928
0.882
0.925
0.946
grid
0.897
0.868
0.934
0.931
0.846
0.669
0.86
hazelnut
0.958
0.938
0.917
0.901
0.906
0.948
0.948
leather
0.979
0.895
1
1.02
0.905
0.989
metal nut
0.992
0.975
0.967
0.938
0.942
0.975
0.967
pill
0.754
0.856
0.814
0.79
0.868
0.868
0.832
screw
0.722
0.746
0.635
0.634
0.516
0.452
0.5
tile
0.966
0.966
0.966
0.946
0.977
0.966
0.966
toothbrush
0.846
0.827
0.788
0.798
0.865
0.808
0.788
transistor
0.837
0.744
0.93
0.932
0.907
0.884
0.651
wood
0.916
0.877
0.961
0.971
0.91
0.923
0.942
zipper
0.926
0.926
0.869
0.854
0.898
0.858
0.875
mean
0.89
0.885
0.883
0.881
0.87
0.849
0.842
pfin
eflow
msflow
ppdm
simplenet
efficientad
ast.
memseg
bottle
0.971
0.926
0.941
0.985
0.926
0.882
0.662
0.787
cable
0.864
0.76
0.88
0.912
0.88
0.712
0.758
0.378
capsule
0.702
0.544
0.667
0.623
0.807
0.798
0.395
0.235
carpet
0.892
0.914
0.914
0.935
0.849
0.882
0.903
0.22
grid
0.691
0.64
0.787
0.272
0.485
0.904
0.679
0.168
hazelnut
0.958
0.914
0.812
0.938
0.875
0.896
0.884
0.353
leather
0.989
1.05
0.968
0.947
0.811
0.884
0.937
0.51
metal nut
0.983
0.946
0.942
0.975
0.942
0.867
0.899
0.381
pill
0.85
0.763
0.731
0.844
0.772
0.653
0.461
0.159
screw
0.389
0.587
0.452
0.603
0.54
0.484
0.37
0.00719
tile
0.966
0.941
0.966
0.966
0.943
0.898
0.942
0.598
toothbrush
0.808
0.899
0.692
0.865
0.788
0.769
0.692
0.42
transistor
0.698
0.777
0.93
0.744
0.86
0.535
0.721
0.295
wood
0.877
0.92
0.91
0.903
0.819
0.774
0.865
0.236
zipper
0.903
0.853
0.835
0.778
0.812
0.608
0.67
0.167
mean
0.836
0.829
0.828
0.819
0.807
0.77
0.723
0.328
Table 5: VisionAD results on the MVTec dataset based on the introduced PL metric.
20

---

Published in Transactions on Machine Learning Research (06/2024)
Image-wise AUC
VisA
Table we show the results using the image-wise AUC metric on the VisA dataset.
ppdm
reverse distillation
efficientad
simplenet
patchcore
cflow
pfm
pipe fryum
0.994
0.993
0.721
0.99
0.996
0.95
0.996
pcb4
0.996
0.999
0.979
0.981
0.994
0.975
0.977
peb3
0.949
0.952
0.923
0.936
0.968
0.712
0.959
pcb2
0.982
0.94
0.956
0.959
0.956
0.82
0.929
pebl
0.977
0.969
0.977
0.979
0.98
1.01
0.921
macaroni2
0.917
0.85
0.897
0.759
0.719
0.805
0.799
macaronil
0.982
0.968
0.972
0.935
0.93
0.929
0.907
fryum
0.947
0.914
0.978
0.934
0.932
0.988
0.972
chewinggum
0.983
0.99
0.988
0.978
0.979
1.06
0.99
cashew
0.966
0.964
0.964
0.912
0.968
0.967
0.903
capsules
0.849
0.903
0.917
0.863
0.751
0.831
0.766
candle
0.959
0.925
0.971
0.973
0.958
1.01
0.905
mean
0.958
0.947
0.937
0.933
0.928
0.921
0.919
cdo
msflow
cfa
pefin
memseg
fastflow altub
fastflow
ast
pipe fryum
0.96
0.972
0.976
0.992
0.837
0.678
0.683
0.689
pcb4
0.953
0.988
0.99
0.979
0.934
0.977
0.968
0.96
pcb3
0.849
0.909
0.91
0.923
0.839
0.805
0.825
0.774
pcb2
0.95
0.955
0.872
0.902
0.874
0.88
0.865
0.89
pebl
0.938
0.959
0.994
0.907
0.887
0.884
0.885
0.735
macaroni2
0.807
0.683
0.719
0.727
0.709
0.629
0.621
0.584
macaronil
0.93
0.884
0.864
0.875
0.819
0.724
0.725
0.581
fryum
0.936
0.949
0.918
0.925
0.871
0.936
0.92
0.614
chewinggum
0.975
0.986
0.998
0.971
0.934
0.985
0.854
cashew
0.975
0.904
0.919
0.896
0.852
0.773
0.776
0.745
capsules
0.809
0.74
0.715
0.767
0.732
0.569
0.571
0.795
candle
0.926
0.975
0.958
0.914
0.893
0.891
0.913
0.864
mean
0.917
0.909
0.903
0.898
0.848
0.812
0.811
0.757
Table 6: VisionAD results on the Visa dataset based on the image-wise AUC metric.
21

---

Published in Transactions on Machine Learning Research (06/2024)
Pixel-wise AUC
VisA
Table
we show the results using the pixel-wise AUC metric on the VisA dataset.
ppdm
reverse distillation
fastflow
msflow
edo
pfm
patcheore
pipe fryum
0.992
0.99
0.99
0.993
0.99
0.99
0.992
pcb4
0.988
0.985
0.983
0.98
0.986
0.968
0.977
peb3
0.992
0.992
0.989
0.978
0.987
0.99
0.99
peb2
0.99
0.987
0.984
0.981
0.981
0.987
0.988
pebl
0.998
0.997
0.995
0.996
0.995
0.996
0.997
macaroni2
0.996
0.994
0.984
0.98
0.992
0.991
0.969
macaronil
0.997
0.996
0.994
0.99
0.991
0.992
0.991
fryum
0.97
0.964
0.947
0.961
0.938
0.957
0.952
chewinggum
0.985
0.984
0.988
0.983
0.987
0.993
0.986
cashew
0.978
0.961
0.984
0.992
0.982
0.981
0.987
capsules
0.994
0.993
0.987
0.987
0.987
0.975
0.978
candle
0.975
0.987
0.985
0.987
0.988
0.98
0.992
0.988
0.986
0.984
0.984
0.984
0.983
0.983
simplenet
fastflow altub
pefm
efficientad
cfa
cflow
ast
memsse
pipe fryum
0.991
0.988
0.992
0.98
0.993
0.951
0.984
0.834
pcb4
0.977
0.972
0.978
0.987
0.909
0.983
0.985
0.824
peb3
0.984
0.996
0.989
0.986
0.987
0.979
0.962
0.833
peb2
0.982
0.972
0.986
0.965
0.984
0.975
0.981
0.829
pebl
0.997
0.996
0.995
0.993
0.992
0.947
0.876
0.831
macaroni2
0.96
0.96
0.98
0.959
0.938
0.932
0.906
0.817
macaronil
0.992
0.994
0.988
0.977
0.994
0.946
0.903
0.829
fryum
0.944
0.948
0.963
0.91
0.848
0.902
0.903
0.791
chewinggum
0.986
1.01
0.98
0.985
0.988
0.945
0.941
0.829
cashew
0.988
0.98
0.961
0.954
0.988
0.929
0.835
0.814
capsules
0.986
0.979
0.964
0.971
0.977
0.943
0.988
0.827
candle
0.985
0.98
0.98
0.956
0.966
0.942
0.984
0.826
mean
0.981
0.981
0.98
0.969
0.964
0.948
0.937
0.824
Table 7: VisionAD results on the VisA dataset based on the pixel-wise AUC metric.
22

---

Published in Transactions on Machine Learning Research (06/2024)
AUPRO VisA
Table
we show the results using the AUPRO metric on the VisA dataset.
ppdm
reverse distillation
pfm
cdo
fastflow
patchcore
fastflow altub
pipe fryum
0.931
0.927
0.906
0.9
0.855
0.921
0.843
pcb4
0.847
0.811
0.713
0.83
0.758
0.734
0.769
peb3
0.86
0.861
0.852
0.8
0.808
0.817
0.808
pcb2
0.864
0.846
0.823
0.791
0.795
0.852
0.774
pebl
0.931
0.899
0.883
0.88
0.849
0.873
0.849
macaroni2
0.956
0.93
0.895
0.924
0.884
0.823
0.867
macaronil
0.942
0.921
0.901
0.884
0.927
0.899
0.909
fryum
0.823
0.833
0.833
0.788
0.715
0.787
0.721
chewinggum
0.767
0.798
0.842
0.749
0.847
0.789
0.85
cashew
0.912
0.898
0.894
0.891
0.866
0.888
0,862
capsules
0.877
0.912
0.806
0.846
0.899
0.78
0.904
candle
0.887
0.911
0.903
0.925
0.923
0.931
0.929
0.883
0.879
0.854
0.851
0.844
0.841
0.84
simplenet
efficientad
cflow
pefm
msflow
cfa
ast
memseg
pipe fryum
0.845
0.882
0.863
0.897
0.85
0.875
0.898
0.762
pcb4
0.734
0.86
0.796
0.754
0.733
0.65
0.841
0.666
pcb3
0.701
0.784
0.714
0.826
0.646
0.828
0.517
0.665
pcb2
0.716
0.805
0.734
0.816
0.736
0.806
0.785
0.685
pebl
0.887
0.848
0.808
0.853
0.824
0.774
0.443
0.714
macaroni2
0.892
0.804
0.839
0.794
0.929
0.771
0.736
0.741
macaronil
0.934
0.869
0.87
0.865
0.913
0.931
0.712
0.768
fryum
0.747
0.726
0.721
0.764
0.645
0.525
0.739
0.637
chewinggum
0.812
0.764
0.765
0.651
0.788
0.832
0.708
0.675
cashew
0.845
0.877
0,821
0.764
0.781
0.738
0.752
0.725
capsules
0.82
0.782
0.798
0.664
0.791
0.682
0.884
0.705
candle
0.908
0.84
0.882
0.921
0.914
0.887
0.893
0.779
mean
0.82
0.82
0.801
0.797
0.796
0.775
0.742
0.71
Table 8: VisionAD results on the VisA dataset based on the AUPRO metric.
23

---

Published in Transactions on Machine Learning Research (06/2024)
PL VisA
Table 9
we show the results using the introduced PL metric on the VisA dataset.
ppdm
patchcore
simplenet
reverse distillation
fastflow
fastflow altub
efficientad
pipe fryum
0.806
0.823
0.774
0.798
0.774
0.759
0.774
pcb4
0.696
0.675
0.613
0.639
0.629
0.62
0.706
peb3
0.592
0.424
0.584
0.504
0.505
0.568
peb2
0.606
0.642
0.489
0.496
0.482
0.474
0,482
pebl
0.74
0.679
0.672
0.672
0.603
0.602
0.496
macaroni2
0.689
0.301
0.544
0.398
0.398
0.406
0.369
macaronil
0.627
0.464
0.709
0.409
0.627
0.62
0.455
fryum
0.588
0.534
0.573
0.519
0,481
0.473
0.389
chowinggum
0.687
0.699
0.675
0.711
0.747
0.762
0.735
cashew
0.664
0.664
0.557
0.733
0.534
0.52
0.656
capsules
0.429
0.371
0.371
0.41
0.371
0.369
0.4
candle
0.453
0.648
0.617
0.445
0.547
0.542
0.414
0.631
0.592
0.585
0.568
0.558
0.554
0.537
cfa
pfm
cflow
cdo
msflow
memseg
pefm
ast
pipe fryum
0.79
0.839
0.8
0.806
0.653
0.681
0.831
0.56
pcb4
0.722
0.531
0.536
0.696
0.541
0.564
0.562
0.662
peb3
0.496
0.512
0.304
0.472
0.288
0.402
0.432
0.171
pcb2
0.489
0.474
0.35
0.438
0.394
0.435
0.511
0.504
pebl
0.58
0.611
0.589
0.618
0.527
0.501
0.534
0.12
macaroni2
0.282
0.233
0.373
0.388
0.485
0.318
0.126
0.0577
macaronil
0.464
0.345
0.454
0.282
0.482
0.386
0.2
0.036
fryum
0.45
0.504
0.477
0.275
0.435
0.406
0.489
0.265
chewinggum
0.711
0.675
0.676
0.633
0.675
0.575
0.554
0.226
cashew
0.534
0.626
0.567
0,504
0.443
0.483
0.496
0.221
capsules
0.324
0.314
0.394
0.333
0.371
0.335
0.295
0.454
candle
0.555
0.461
0.52
0.508
0.469
0.443
0.438
0.368
mean
0.533
0.51
0.503
0.496
0.48
0.461
0.456
0.304
Table 9: VisionAD results on the VisA dataset based on the introduced PL, metric.
24

---

Published in Transactions on Machine Learning Research (06/2024)
For the interested reader show the per class results and per dataset mean for each algorithm in Tables 2 9 These are what we believe to be the most comprehensive and fair set of results currently available for the MVTee and VisA datasets.
25

---

Published in Transactions on Machine Learning Research (06/2024)
Appendix C
Here we provide details of the subfunctions of PL introduced in Section 3.4
ExtradRotateddBunddnggooxxs
Here we use the ménArraaRect function from the OpenCV library (Brddski & Kaehler 2020). This creates rotated bounding box around every non-overlapping group of pixels.
ScaleBBs
For every bounding box with a width and height of less than 1/8th of the image width and height, the width and/or height is set to 1/8th, This achieved easily because the bounding boxes are parameterised via center, width, height, and rotation.
MergeOverlappinggBBS
This is the most complicated the sub functions. Each non-overlapping ground truth is compared with one another, and the overlap between the respective bounding boxes is calculatod. Groups of bounding boxes are extracted, where for each group; one of the bounding boxes meets the overlap limit with another in the group. For each group, the underlying pixels used to create each respective bounding box are merged into one image, and new rotated bounding box is created around the pixels in this image. Each group now reprosents one image and bounding box. The process is ran once again on the new images and bounding boxes, We find this process sufficiently merges close anomalies. We use an overlap limit of 0.33.
CreateNearestBBFitter
For group of bounding box, wish to split image into set of masks, where cach mask corresponds to bounding box, and each mask highlights the pixels which are closest to its respective bounding box. This can be achieved by using the adist function from the SciPy package (Virtanen et al. 2020). The distance between each pixel and bounding box center is calculated for each mask, and an arymin function ran on the mask to get the one hot encoded final values. These filters are used to split the image into regions which correspond each bounding box.
26


Reviews:
[{"review_id": "oHoUbVfn5H", "reviewer": "Reviewer_rj9M", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. As the ML research progresses rapidly, it is good to have a platform to compare results fairly. \n2. The proposed software architecture is easy to integrate any new AD algorithms.\n\nWeaknesses:\n\n1. As this task is closer to semantics segmentation or instance segmentation, the Dice score should be compared.\n2. The difference from the normal metrics used in object detection is unclear; moreover, in object detection, an average score across different IoU is reported.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new software package for the vision anomaly detection algorithms and hosts a leaderboard such that the researchers could fairly compare among different algorithms; moreover, this paper proposes a new metric, which provides more straight-forward interpretation.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Weaknesses\n\nHere are other comments\n\n1. In section 1.1, when comparing to Anomalib, the authors claim the visionAD is more light-weight to use but it is unclear to me, if VisionAD become larger and larger, wouldn't it have the similar issue? Will it become another anamalib, which is comprehensive but more efforts? Can the authors elaborate more?\n\n2. how flexible of the proposed VisionAD to adopt algorithms added into Anomalib? So, the researchers do not need to duplicate their work to different frameworks.\n\n3. As the proposed PL needs the bounding box label; what if the users incorporate that into P-AUC and AUPRO calculation? Does that solve the weakness the authors mentioned?\n\n4. Caption for figure 3, there is no color red used in the figure.\n\n5. In figure 5, under different metrics, the rank of algorithms are different; how should the users to really compare the algorithms?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "K6OIlETUtb", "reviewer": "Reviewer_Q98x", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The paper is well-motivated, and the library seems valuable for benchmarking purposes.\n2. The library contains many anomaly detection algorithms.\n3. From the visualization results, the performance of the proposed metric seems reasonable.\n\n## Weaknesses\n1. Having two different contributions simultaneously is good, but I don’t see enough support for each contribution. See **Requested Changes** for more details. To be honest, I don’t see the reason the authors present both VisionAD and PL with a short submission limit. I’d suggest adding more experimental support for each and submitting them as long submissions. \n2. The paper contains several minor problems, e.g., citation style, typos, and grammar.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces the implementation of a new anomaly detection library. Specifically, the authors implement VisionAD, a new lightweight library for performance benchmarking. Then, the author proposes a new metric, Proportion Localized (PL), for measuring the performance of an anomaly detection algorithm.\n\nThe authors briefly describe the library's motivation and components. The library contains 15 different anomaly algorithms and several performance metrics to evaluate anomaly detection algorithms. This library would help develop a new anomaly detection method by efficiently comparing an algorithm to other existing algorithms.\n\nThen, in Section 3, the author proposes a new performance metric included in the library. The proposed metric (PL) can overcome several disadvantages of the existing performance metrics: PL captures anomaly localization, has no noise from pixel-wise matching, and has clear physical meaning. With an experiment, PL’s performance measurement is compared to two other existing metrics. Section 4 covers more details about PL metric implementation with its detection result.\n\nSection 5 presents the benchmark result for the anomaly detection algorithms in VisionAD on the MVTec dataset. In Section 6, the authors conclude the paper with discussions about limitations and potential future works.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. This paper should present more benchmarks with different configurations as a library paper. This will better demonstrate the library's usefulness.\n   * Did the author try any dataset other than the MVTec dataset? Having similar results on a similar (but different) dataset would show the library's consistency in benchmarking. \n   * Why did the authors keep the hyperparameters the same across each class? The library can easily support different hyperparameters, right?\n   * How would this library support benchmarking those algorithms that change hyperparameters? \n2. More quantitative experiments should be performed on the anomaly detection performance metric. \n   * Figure 2 provides a good picture of PL's behavior, but we don’t know whether the observation is generalizable. An additional table showing the metric scores on different sets of data would be better.\n   * Figure 4 can be a good way to show the authors’ choice of IoU limit, but it is better to provide a quantitative number that measures such a “satisfactory overlap” rather than spending a whole page with images.\n3. Minor changes\n   * I don’t think that the citation style follows the format in the [author guidelines](https://jmlr.org/tmlr/author-guide.html). Please use the [style file](https://github.com/JmlrOrg/tmlr-style-file/archive/refs/heads/main.zip) provided in the author guidelines.\n   * Please proofread the paper once again and fix all the spelling and grammar errors.", "broader_impact_concerns": "I don’t see a particular concern about the broader impact of this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PKOc5IwNei", "reviewer": "Reviewer_xc77", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Pros**\n\n**Ease of Use and Integration:** VisionAD's design allows for the straightforward addition of new algorithms, encouraging rapid experimentation.\n**Standardization:** The uniform API across algorithms aids in fair comparison and benchmarking.\n**New Metric (PL):**  Offers an intuitive metric that addresses the limitations of existing anomaly detection metrics.\n\n**Cons**\n\nI have several concerns about the proposed metric, specifically the PL metric, which seems inadequate for the diverse conditions encountered in anomaly detection. Firstly, in situations where the ground truth is very thin or U-shaped, resembling a 45-degree angle—essentially, when the pixel-wise ground truth forms the diagonal of your bounding box—a perfect segmentation might not be recognized due to an Intersection over Union (IoU) lower than 0.3 with the bounding box. Secondly, in cases where the ground truth consists of small, non-continuous dots that a prediction fully covers, the PL metric inaccurately scores a 1 because it segments the prediction into many parts. This scenario indicates the model predicts a single anomaly, yet the PL score suggests perfect performance.\n\nConducting more experiments with the proposed metric on real-world datasets could be helpful in these issues.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose VisionAD, a comprehensive library for anomaly detection in images designed for easy integration, experimentation, and fair benchmarking. It proposes standard API across all included algorithms to facilitate comparison, which improves the ease of integrating new algorithms.\nThey also introduce the Proportion Localised (PL) metric as a more intuitive and industrially relevant metric. This metric focuses on the proportion of anomalies that are adequately localized rather than pixel-perfect detection and aims to provide a more physically meaningful number for evaluation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address my concerns mentioned above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new software package for the vision anomaly detection algorithms and hosts a leaderboard such that the researchers could fairly compare among different algorithms; moreover, this paper proposes a new metric, which provides more straight-forward interpretation.

The paper is well-motivated, and the library seems valuable for benchmarking purposes. The library contains many anomaly detection algorithms. From the visualization results, the performance of the proposed metric seems reasonable. However, the majority part of the work is about the software, and the paper does not provide well-supported claims and evidence for each contribution, and I believe the authors should provide more results. Therefore, based on three qualified reviews, this paper can be accepted with minor revision and the authors are encouraged to merge the comments into their update versions.
Decision: Accept (poster)