FOCT: Few-shot Industrial Anomaly Detection with Foreground-aware Online Conditional Transport

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 05:15:57

Abstract: Few-Shot Industrial Anomaly Detection (FS-IAD) has drawn great attention most recently since data efficiency and the ability of designing algorithms for fast migration across products become the main concerns. The difficulty of memory-based IAD in low-data regimes primarily lies in inefficient measurement between the memory bank and query images. We address such pivot issues from a new perspective of optimal matching between features of image regions. Taking the unbalanced nature of patch-wise industrial image features into consideration, we adopt Conditional Transport (CT) as a metric to compute the structural distance between representations of the memory bank and query images to determine feature relevance. The CT generates the optimal matching flows between unbalanced structural elements that achieve the minimum matching cost, which can be directly used for IAD since it well reflects the differences of query images compared with the normal memory. Realizing the fact that query images usually come one-by-one or batch-by-batch, we further propose an Online Conditional Transport (OCT) by making full use of current and historical query images for IAD via simultaneously calibrating the memory bank and matching features between the calibrated memory and the current query features. Go one step further, for sparse foreground products, we employ a predominant segment model to implement Foreground-aware OCT (FOCT) to improve the effectiveness and efficiency of OCT by forcing the model to pay more attention to diverse targets rather than redundant background when calibrating the memory bank. FOCT can improve the diversity of calibrated memory during the whole IAD process, which is critical for robust FS-IAD in practice. Besides, FOCT is flexible since it can be friendly plugged and played with any pre-trained backbones, such as WRN, and any pre-trained segment models, such as SAM. The effectiveness of our model is demonstrated across diverse datasets, including benchmarks of MVTec and MPDD, achieving SOTA performance.

Main paper: 
 introduction: 
 The fragmented nature of industrial anomalies, such as subtle bruises and obvious breakages with various appearances and scales [24, 28, 40] , raises difficulties in detecting and classifying industrial anomalies in the fully-supervised manner as primal researches do [9, 11] . Therefore, unsupervised Industrial Anomaly Detection (IAD) methods have been developed most recently, which efficiently use the model trained with only normal industrial images of each product to detect anomalous industrial images and localize the corresponding anomalous regions precisely [3, 4, 17, 23] .Researches on unsupervised IAD can be divided into two categories, namely reconstruction-based models [25, 26, 42] and memorybased models [10, 27, 40] . Reconstruction-based models aim to learn continuous or discrete feature representations of normal images utilizing Deep Generative Models (DGMs) such as Variational Au-toEncoding (VAE) [20, 42] and diffusion models [13, 25] through a cumbersome training process. Memory-based models usually extend DGMs with a memory bank to record the normal features given normal training images, manifesting in various forms, including CNN-based multi-scale features [10, 28] and Transformer-based learnable query embeddings [40] . At test time, feature matching between test images and memory is preferred. However, both lines of work heavily rely on massive normal training images, either for DGMs optimization or memory learning, and fail to generalize across products quickly in low-data regime.Inspired by how human beings detect anomalies, Few-Shot Learning (FSL) [35, 36] is introduced to IAD for learning a common model shared among multiple products and also generalizable to novel products where limited normal training images are provided, for example, 1 or 2 shot per product. This new paradigm is also known as Few-Shot IAD (FS-IAD) [8, 14, 15, 31] , which can be divided into Inductive FS-IAD (IFS-IAD) models [14, 15, 31] and Transductive FS-IAD (TFS-IAD) [8] models. The former leverages statistics of limited support (training) images to conduct IAD, while the latter uses statistics of both support and query (test) images for more generalizable IAD. TFS-IAD and IFS-IAD models are separately developed to extend the generalization capability, mainly for reconstruction-based and memory-based models of the previous studies. Additionally, meta-learning based IFS-IAD models [14] have also been proposed to realize fast generalization to new products.Despite their promising results, we observe that redundant backgrounds and large appearance variations of support images may drive the image-level embeddings from the normal pattern far apart in a given metric space at test time. Although the issue could be alleviated by Deep Neural Networks (DNNs) under data-hungry supervised training, it is almost inevitably amplified in FS-IAD and thus negatively impacts anomaly detection and localization. Moreover, employing heuristic and linear metrics for anomaly detection [8, 31] Then we design a novel online learning pipeline for memory calibration and anomaly detection, simultaneously, which are implemented using the optimal matching flows between the memory and query images.provide discriminative and transferable information across normal and anomalous patterns. It should be critical for IAD in the fewshot scenario. Therefore, a desirable metric-based algorithm should have the ability to leverage the local discriminative representations for feature matching and minimize the impact caused by the irrelevant regions for robustly distinguishing anomaly images and their corresponding regions. Intuitively, we hope to observe that an anomalous region should be most irrelevant to the space spanned by normal features. Furthermore, how to discover the characteristics of IAD itself and absorb them into a metric-based algorithm design for improving generalization is another crucial problem to be contemplated. We aim to solve the pivot issues mentioned above under the line of memory-based TFS-IAD, which was rarely studied before. First of all, considering the fact that query features exhibit unbalanced distribution along the memory bank, as detailed in the methodology later, we adopt a theoretically guaranteed non-linear measurement function called Conditional Transport (CT) to compare the building blocks of two complex structured representations. CT [43] is a function for computing distance between structural representations, which was originally proposed for GAN-based DGM optimization, and can be friendly cascaded with DNNs for feature matching, such as WRN [41] in this work. Given the distance between all element pairs, CT has the formulation of the transport problem [12, 37] and the optimal matching flows between two structures can be achieved by minimizing the matching cost via Stochastic Gradient Descent (SGD). Secondly, previous researches ignore the fact that industrial images to be detected are usually coming online, which provides a hotbed for enhancing the capability of CT function with both current and historical query images. Therefore, we initialize a memory bank with support features and propose an online learning pipeline to calibrate the memory with statistics of online query images, the anomaly detection is thereby implemented by measuring the distance between the calibrated memory and query features with the CT function. We name this enhanced model Online CT (OCT). Thirdly, we find that distinguishing products and their backgrounds in the query images is necessary to realize robust FS-IAD for sparse foreground products. On the one hand, redundant backgrounds are not conducive to OCT, which may introduce overwhelming invalid calculations. On the other hand, a well-calibrated memory with a proper foreground and background ratio performs steadily in FS-IAD. We use a predominant segment model, such as SAM [21] in this work to implement Foreground-aware OCT (FOCT) to improve the effectiveness and efficiency of OCT by forcing the model to pay more attention to diverse targets rather than redundant backgrounds. We sum up our proposed framework in Fig. 1 . Our work contributes in the following ways:• We propose a novel memory-based TFS-IAD framework that primarily adopts a non-linear and theoretical guaranteed CT function between the memory bank and query images for precise feature matching. • We make full use of the characteristics of IAD and design a new pipeline called FOCT to enhance the generalization capability of the aforementioned CT function. • Our proposed model achieves SOTA anomaly detection and localization performances in the few-shot regime under various datasets, including MVTec [1] and MPDD [16] and fruitful settings, including 1/2/4-shot. 
related work 2.1 industrial anomaly detection: 
 IAD involves handling industrial training datasets that exclusively consist of normal data, presenting challenges due to the varying subtleties of defects. In this field, the prevailing unsupervised techniques are predominantly reconstruction-based methods [10, 38, 40] and memory-based models [5, 7, 28] . Reconstruction-based methods are trained exclusively with normal data on the premise that anomalies will yield significantly higher reconstruction errors. Nevertheless, this assumption does not invariably apply, occasionally leading these approaches to encounter the identical shortcut issue, where they inadvertently learn to reconstruct anomalies, thus reducing their discriminative efficacy. Recently, memory-based models have achieved promising performances in anomaly detection and localization, leveraging pre-trained features stored in memory banks containing various feature hierarchies, such as PaDiM [7] , SPADE [5] , Patchcore [28] , thereby significantly enhancing the detection capabilities. Furthermore, various sophisticated adaptations have been implemented, including normalizing flows [19, 29] and student-teacher knowledge distillation [2, 30] integrated with the features obtained from pre-trained networks, to effectively manage the discrepancy in distribution between the pre-training natural image datasets and the distinct nuances present in industrial imagery. This paper distinguishes from the previous studies by concentrating on few-shot anomaly detection, where only a limited number of normal images are accessible.FS-IAD has emerged as a compelling new area of research, aiming to identify anomalies with only a few support samples from target categories. FS-IAD methodologies can generally be categorized into two types: Inductive FS-IAD (IFS-IAD) models [14, 15, 31] and Transductive FS-IAD (TFS-IAD) models [8] . The majority of existing research has focused on inductive FS-IAD approaches, which utilize statistical information from a small set of support (training) images, without incorporating query (test) images to perform anomaly detection. Within this realm, generative models have gained popularity recently, including the hierarchical generative model TDG [33] , and the normalizing flow-based model DiffNet [29] , which both sketch out data distributions from the limited support samples. Furthermore, the recently proposed Opt-PatchCore [31] introduces an effective data augmentation method that substantially enhances feature diversity, thus boosting the generative capabilities of the model's memory bank. Concurrently, a feature-augmentation strategy embodied by GraphCore [39] incorporates visual isometric invariant features into the memory-based anomaly detection framework, thereby improving its ability to distinguish anomalies. In contrast, TFS-IAD models enhance anomaly detection's generalizability by utilizing statistical data from both support and query images. An innovative approach is demonstrated by Fastrecon [8] , which employs a regression technique with distribution regularization. This technique achieves the optimal transformation from support to query features, ensuring the reconstruction closely resembles the query sample while preserving the characteristics of normal samples. However, the reliance on heuristic and linear metrics for anomaly detection can compromise image structure integrity and overlook the importance of local features. In this work, we introduce a novel memory-based TFS-IAD framework that leverages Conditional Transport (CT) as a metric to calculate the structural distance between the memory bank and the query images, thereby enhancing the generalizability of FS-IAD. 
preliminary: 
 In this section, we introduce the preliminaries on the task formulation of FS-IAD and the theory of CT function to prepare for subsequently presenting our proposed model, FOCT.We formally define the one-class industrial anomaly detection task following the standard 𝑛 way 𝑘 shot few-shot learning setting. In each category, there are 𝑘 normal support images 𝒙 𝒙 q t being whether normal or anomalous and its historical query image series 𝒙 q 1:t-1 of the corresponding category as 𝒙 s 1:k do. The model predicts whether or not the query image 𝒙 q t is anomalous at the pixel and image level based on the query image series 𝒙 q 1:t . It is worth mentioning that previous methods only leverage statistics of the current query image 𝒙 q t to make predictions while they always ignore the statistics of the historical query image series 𝒙 q 1:t-1 , which may yield some suboptimal results as revealed by our work.Conditional Transport (CT) is proposed to measure the distance between two sets of weighted objects or distributions using the non-linear mapping function such as DNN, which is built upon the basic distance between individual objects. Specifically, given two sets of discrete sampled objects 𝒚 1:𝑛 and 𝒛 1:𝑚 from distributions 𝑝 (𝒚) and 𝑝 (𝒛), where 𝑝 (𝒚) = 𝑚 𝑖=1 𝑎 𝑖 𝛿 𝒚 𝑖 and 𝑝 (𝒛) = 𝑛 𝑗=1 𝑏 𝑗 𝛿 𝒛 𝑗 . 𝒂 ∈ Δ 𝑚 and 𝒃 ∈ Δ 𝑛 separately denote the probability simplex of R 𝑚 and R 𝑛 . The CT distance is defined as:L 𝝓,𝜌 = min 𝝓 ∑︁ 𝒚 ∈𝑝 (𝒚 ) ∑︁ 𝒛 ∈𝑝 (𝒛 ) 𝜌 C 𝝓 (𝒚 → 𝒛) + (1 -𝜌)C 𝝓 (𝒛 → 𝒚) (1)where C 𝝓 (𝒚 → 𝒛) and C 𝝓 (𝒛 → 𝒚) separately denote the forward and backward CT distances, also known as navigators in the scope of CT theory, 𝝓 is optimizable parameters, and 𝜌 is a trade-off factor controlling strengths between the forward and backward navigators. To better understand how CT works when minimizing Eq. 1, we take the forward navigator as an example as follows:EQUATIONwhere 𝑐 ( The pipeline of episodic memory calibration consists of memory augmentation and memory compression. We use the optimal transport matrix 𝝅 * derived from the optimal matching flows to select the Top-K query features for augmenting the episodic memory. We employ the coreset technique to downsample the episodic memory for compression, as shown in (c).… … … …   * , , , i j i j i j CT S Q c     Minimize CT distance * m n R    Optimal Transporttransporting a random source point to one of the 𝑚 randomly instantiated "anchors" of the target distribution. By minimizing the forward CT of Eq. 2, an unbiased statistical estimation of the two distributions arrives. Hence, it is reasonable to measure the relevance of one distribution against another. 
methodology: 
 In this section, we outline the details of our proposed FOCT model from the following perspectives: We first use pre-trained WRN [41] and SAM [21] to extract foreground-aware support features preparing for constructing the semantic memory (Sec. 4.1); Then, we calibrate the episodic memory with the current query image using CT function (Sec. 4.2); Finally, we implement anomaly detection by measuring the distance of the calibrated memory and the current query image (Sec. 4.3). Moreover, we conduct analysis on the properties and complexity of our proposed model (Sec. 4.4).In practice, we may encounter the problem of anomaly detection for sparse foreground products. For the sake of algorithm effectiveness and computation efficiency, we propose to use pre-trained WRN and SAM to extract foreground and background features of support samples for constructing the semantic memory, as shown in Fig. 2 . Specifically, following the existing studies [7, 31] , we firstly use WRN parameterized by 𝜽 WRN to extract the multi-scale feature maps of support images as 𝒇 1:k = 𝑔 𝜽 WRN (𝒙 1:k ). Concurrently, we employ SAM with parameters 𝜽 SAM to calculate masks of the corresponding support images and downsample them to the scale of support feature maps by 𝒎 1:k = ds(g 𝜽 SAM (𝒙 1:k )), where ds(•) is downsampling operation, 𝒇 1:k ∈ R k×h×w×c and 𝒎 1:k ∈ R k×h×w . Foreground and background features of support images can then be separately derived by:EQUATION) where fla(•) is a flatten operation. N f and N b are the number of feature vectors for the foreground and background, which are usually huge. Hence, we compress the size of foreground and background features according to different downsampling ratios 𝛼 and 𝛽 to reduce redundancy. Finally, the semantic memory can be represented by concatenation asM s = [𝒇 fore 1:𝛼N f , 𝒇 back 1:𝛽N b ] ∈ R n×c .Aware of the fact that query images are always coming online, we design an episodic memory calibration that is complementary to its semantic peer to better capture statistics of the current and historical query images. The episodic memory calibration is composed of memory augmentation and memory compression, the pipeline of which is shown in Fig. 3 .Memory augmentation aims to gather statistics of historical query images 𝒙 q 1:t-1 in the old episodic memory M e t-1 with the features of the current query image 𝒙 q t to form the augmented memory M e t , for the beginning we have M e 0 = ∅. As shown in Fig. 3 (b), memory bank is defined as the concatenation of semantic and episodic memories and denoted byM t-1 = [M s , M e t-1]. We use the same WRN g 𝜽 WRN as in extracting features of semantic memory do for obtaining features of the current query image as 𝒇 q 1:m = fla(g 𝜽 WRN (𝒙 q t )), where we replace the subscript t with q for simplicity. In order to obtain the new episodic memory augmented with statistics of the current query image, we need to solve the optimal matching flows between M s and 𝒇 q 1:m . According to the analysis of CT theory in the preliminaries, we regard M s and 𝒇 q 1:m as two sets sampled from two distributions and formulate the distance with CT function as: For efficiently optimizing Eq. 4, we separately define the forward and backward transport matrices 𝜋 𝑛 and 𝜋 𝑚 as follows:EQUATIONEQUATIONwhere 𝝅 𝑛 ∈ R 𝑚×𝑛 and 𝝅 𝑚 ∈ R 𝑛×𝑚 . By minimizing Eq. 4 w.r.t. 𝝓 via SGD algorithms such as Adam [18] , Optimal flows between the semantic memory and current query features arrive.Once the optimal forward transport matrix 𝝅 * 𝑛 is obtained, the memory augmentation with Top-K selection can be implemented:EQUATIONwhere ΔM e t ∈ R K×c . It is worth noting that we conduct argmax(•) operation for each column since we want to select query features most relevant to normal features. Finally, the new episodic memory is refreshed by M e t = [M e t-1 , ΔM e t ]. The augmented memory bank can now be expressed by M t = [M s , M e t ] ∈ R (n+L t ) ×c , where L t is the length of episodic memory at the 𝑡-th query image.In practice, the memory bank can not be infinite, suppose the capacity of the episodic memory is 𝐿. As the number of online query images increases, the episodic memory goes to full when L t ≥ L. Therefore, we design another mechanism called memory compression for efficient data management of episodic memory. The objective is to identify a subset M e t ⊂ M e t that the subset should enable the most accurate and expedited approximation of problem solutions over M e t , aligning closely with those computed over the entire M e t . In our focus on anomaly detection, the upcoming subsection employs nearest neighbor computation. To guarantee comparable coverage of M e t to the original episodic memory M e t , we adopt a min-max facility location coreset selection [34] , addressing the inherent NP-hard problem: where M e, * t ∈ R 𝜆L×c , 𝜆 ∈ {0, 1} is compressive ratio. To solve the NP-hard problem, we employ an iterative greedy approximation [32] . To enhance the efficiency of memory compression, we use the Johnson-Lindenstrauss theorem proposed by [6] to reduce the After parameters 𝝓 * of the CT function are optimized, the anomaly score map of the current query image can be expressed by:EQUATIONFor image-level anomaly detection, we use the maximum distance score 𝑠 * among all the pixels 𝒔 ∈ R m to represent𝑠 * = max j=[1,m] 𝒔 j .For pixel-level localization, we first upscale the score map with bilinear interpolation to match the original input resolution. Then, we smooth the score map with a Gaussian kernel width 4.4.4.1 Property of FOCT on adaptive prior. One of the important and appealing properties our proposed FOCT enjoys is that the prior distribution of query features can be adaptively acquired through the CT function optimization, which fits the practical scenario where the prior distribution of query features is usually unknown in advance. According to the forward transport matrix in Eq. 5, we have 𝑛 𝑖=1 𝜋 𝑛 (M s i |𝒇 q j , 𝝓 * ) = 1 while 𝑚 𝑗=1 𝜋 𝑛 (M s i |𝒇 q j , 𝝓 * ) = C i , usually we have C i ≠ 1 and its real value is adaptive with query features and the procedure of CT function optimization.Another practical and interesting property of our model is that it is computationally efficient, especially for sparse foreground products, which is usually encountered in real-world applications. As shown in Fig. 5 , compared with its variant OCT without foreground-aware mechanism, FOCT shows more reasonable attention allocation in processing the foreground and background.We implement 𝝓 using a ReLU-activated MLP. Taking a two-layer MLP with dimensions [𝑑 1 , 𝑑 2 , 𝑑 3 ] for example. The forward and backward computational complexities per iteration take O (NMd 2 (d 1 + d 3 )), where 𝑁 and 𝑀 separately refer to the number of features of the two sets. To reduce the complexity, we can further control the size of MLP or the size of the two sets in achieving a trade-off between accuracy and real-time efficiency.  
experiments 5.1 experimental setups: 
 Dataset: We conduct experiments on the MVTec [1] and MPDD [16] datasets. MVTec dataset comprises 5,354 images across 15 categories, with 3,629 defect-free and 1,725 defective images. Each category exhibits an average of five distinct defect types, with resolutions ranging from 700×700 to 1,024×1,024. MPDD dataset contains 6 classes of metal products. The images are captured from various distances and spatial angles amidst non-uniform backgrounds, which poses a significant challenge for the FS-IAD task. MPDD consists of 888 normal training images and 458 test images, which separately contain 176 normal images and 282 abnormal images. The image resolution is fixed as 1,024 × 1,024. Both datasets provide pixel-level ground truth annotations for defective regions.Competing Methods: We compare our proposed FOCT against the most recently proposed SOTA full-data IAD and FS-IAD methods, including PaDiM [7] , RegAD [14] , PatchCore [28] , and FastRecon [8] .For a fair comparison, we keep the query images fixed and conduct 10 random support image splittings under 1/2/4 shot scenarios.Then we use the official codes of PaDiM, RegAD, and PatchCore to implement experiments. We report the results of FastRecon with our own reproduction since the official code is not available yet. Evaluation Protocols: To quantify the model performance of image-level anomaly detection, we employ the Area Under the Receiver Operator Curve (AUROC) and the F1-score at the optimal threshold (F1-max) as our evaluation metrics, being consistent with the previous works [15] . Furthermore, we utilize pixel-level AUROC and F1-max to measure the defect localization performance. All the results of image-level and pixel-level AUROC and F1-max for competing methods and our model are averaged on the 10 splittings. Implementation Details: In our experiment, we employ a WRN-50 [41] pre-trained on ImageNet dataset [22] as the feature extractor. Image features are obtained from the intermediate layers following [28] . Both support and query images are reshaped to 336×336 in the MVTec and MPDD datasets. To initialize the semantic memory, foreground, and background subsampling ratios are 𝛼 = 10% and 𝛽 = 7.5%, respectively. Meanwhile, the episodic memory has a capacity of 𝐿 = 100 feature vectors with a compression ratio 𝜆 = 25%. We select the top 10 foreground and top 40 background feature vectors with the maximum transport probabilities. We update parameters of the CT function using Adam [18] with a learning rate of 0.0001 for 100 epochs, with the trade-off factor of CT 𝜌 = 0.5. All the experiments are conducted on one NVIDIA GTX 3090 GPU. We will release the source code upon acceptance.Few-shot Anomaly Detection. We compare the few-shot anomaly detection performance of our proposed FOCT with recently proposed competitive baselines and report image-level results with black in Table 1 . Our model significantly outperforms the previous methods on both MVTec and MPDD datasets under image-level AUROC and image-level F1-max metrics. Compared with the results of PaDiM and PatchCore, taking 1-shot for example, AUROC and F1-max of our model separately exceed the two baselines more than 4% and 3%, which reveals the fact that transductive inference in few-shot scenario benefits image-level anomaly detection a lot since more statistics on query images are considered. When it comes to results between FastRecon and our FOCT, also considering the 1-shot case, we observe that our model outperforms FastRecon more than 2% in AUROC. We attribute this to the fact that our model simultaneously employs powerful non-linear measurement functions and leverages statistics of query images more thoroughly. It is worth noting that products' sparsity in MPDD is more obvious than MVTec, and our model achieves more gain on MPDD against MVTec. It verifies the effectiveness of our proposed foreground-aware semantic memory to some extent. Specifically, our FOCT exceeds the second-best model by more than 6% and 5% in image-level AUROC and image-level F1-max under 1-shot setting. Qualitative results are reported in Fig. 6 . Few-shot Anomaly Localization. We report the few-shot anomaly localization results of our model compared with other competitive baselines using pixel-level AUROC and pixel-level F1-max with gray in Table 1 . Our model outperforms the competitors on the MVTec dataset across the two metrics under various few-shot scenarios including 1/2/4 shot. Our model achieves competitive pixel-level AUROC and exceeds more than 2% on average using pixel-level F1-max against other competitors. An interesting observation is that our model does not achieve SOTA performance in pixel-level AUROC, which we attempt to answer hereafter. We find that the ground truth masks of MPDD datasets are sometimes larger than the actual anomaly areas, examples can be found in Fig. 6 . However, anomaly score maps derived from our model have low uncertainty. Therefore, the predicted mask after binarizing the score map may have some relatively concentrated anomalous areas. When calculating the metric of pixel-level AUROC, although our model precisely localizes the anomalous areas, the IOU between the human-annotated ground truth mask and the concentrated predicted mask remains low, which negatively impacts the recall value of our model and leads to lower pixel-level AUROC. Besides, we provide performance comparisons per product and report the results in Table 2 for more detailed comprehension.In our ablation study, we develop three variants of our proposed FOCT called vanilla, CT, and OCT to evaluate the effectiveness of different components of our model. Compared with FOCT, OCT removes the foreground-aware mechanism in semantic memory and only uses downsampling to construct the semantic memory. Compared with OCT, CT only leverages statistics of the current query image and ignores statistics of historical query images. Compared with CT, vanilla uses statistics of the current query image via the linear measurement function introduced in FastRecon [8] .The results are reported in Table 3 and Fig. 7 , respectively.Affect of non-linear measurement function. Comparing the results between vanilla and CT in Table 3 , we observe that CT significantly outperforms vanilla in almost all metrics. It approves the superiority and effectiveness of non-linear measurement powered by the CT function against its linear measurement peer.Affect of statistics of historical query images. Comparing the results between CT and OCT, OCT achieves consistent improvement in general. It reveals that statistics of historical query images matter in anomaly detection when the normal training images are extremely limited. We also find that the metric of AUROC drops a little in the OCT variant, and we attribute this to that OCT augments too many redundant backgrounds into the memory bank, thus restricting anomaly detection performance in low-data regime.Affect of foreground-aware semantic memory. According to the results between OCT and FOCT, there exhibits some significant improvements in image-level metrics of AUROC and F1-max, demonstrating the necessity and effectiveness of foreground-aware semantic memory, qualitative visualizations can be found in Fig. 5 .In this part, we mainly focus on analyzing the robustness of our model against various hyper-parameters, including foreground and background subsampling ratios 𝛼 and 𝛽, foreground and background selection numbers n and m, memory capacity L, memory compression ratio 𝜆, and trade-off factor 𝜌 of CT function. Results are separately reported in Fig. 8 and Fig. 9 .Affect of memory initialization. We regard 𝛼, 𝛽, n, and m mentioned above as hyper-parameters of memory initialization. As we can see an initialized memory bank with a proper foreground and background ratio is crucial for generalized anomaly detection in the low-data regime. We attribute this to the fact that the representativeness of the initial memory is related to such a ratio.Affect of memory capacity. We observe that the oversized memory capacity with large L is unfriendly with boosting performance. We believe that unexpected interference would be introduced with the increase in memory capacity, which should have negative impacts on anomaly detection.Affect of memory compression ratio. As we can see, generally speaking, the low or high compression ratio 𝜆 hurts AUROCs. On one hand, the low 𝜆 may sacrifice the diversity of the memory bank. On the other hand, the high 𝜆 may not fully utilize the statistics of the current query images.Affect of trade-off factor in CT function. Our model achieves robust AUROCs versus various 𝜌 values, demonstrating the flexibility of our model when learning the CT function. Besides, the best AU-ROCs arrive at 𝜌 = 0.5, which implies that both the forward and backward CTs are important for improving performance. 
conclusion: 
 In this work, we propose a novel memory-based TFS-IAD framework, called FOCT, which has rarely been studied before. It consists of three components, including foreground-aware semantic construction, episodic memory calibration, and anomaly detection in the few-shot scenario. The episodic memory calibration and anomaly detection are both powered by the non-linear measurement CT function, compared with its linear counterparts without considering foreground sparsity and statistics of online query images, FOCT shows impressive generalization capability across various benchmark datasets under fruitful few-shot scenarios of 1/2/4 shot.  
 
 Appendix:  


Decision: This submission presents a novel approach to few-shot industrial anomaly detection (FS-IAD) by introducing a framework that leverages Conditional Transport (CT) and dynamic memory bank updates using SAM for foreground segmentation. The proposed method is evaluated on various FS-IAD benchmarks, demonstrating its potential effectiveness.

Pros:
- The use of Conditional Transport as a metric for determining feature relevance is novel in anomaly detection.

- The approach of dynamically updating the memory bank for foreground segmentation is interesting.

Cons:

- The lack of competing methods was mentioned by reviewers. During the rebuttal, authors presented additional comparisons with GraphCore and WinCLIP, which eased the concerns.

- The random selection of support samples can lead to significant variations in results under low-data conditions. During rebuttal, authors included additional results demonstrating the stability of resulting by taking the average and standard deviation over multiple runs.

Overall, this paper presents a promising approach to few-shot industrial anomaly detection with some innovative ideas. However, there are several areas that require further clarification and additional experiments to strengthen the claims. Given the current ratings (2 weak accepts, 1 borderline accept, and 1 borderline reject), I would recommend to accept this submission.

Reviews: review: paper_summary: paper_summary: This research paper presents a framework for addressing Few-Shot Industrial Anomaly Detection (FS-IAD), which is critical in scenarios where data efficiency and the ability to quickly adapt across different products are prioritized. The framework utilizes pre-trained models for initial image segmentation and employs Optimal Matching Flows, Conditional Transport (CT), and Online Conditional Transport (OCT) to dynamically calibrate and match memory banks with query images in real-time. This approach is particularly tailored for scenarios where query images vary significantly and are processed either individually or in batches. The model also introduces Foreground-aware OCT (FOCT) to enhance focus on diverse targets over redundant backgrounds during memory calibration, which is beneficial for sparse foreground settings.
strength_weakness: strength_weakness: 1.Enhanced Data Efficiency: The model is designed to work effectively with very few samples per category, making it ideal for applications where data is scarce or expensive to obtain.
2.Adaptability: The framework supports rapid adaptation to new product lines or changes in production scenarios without requiring extensive retraining, thanks to its Few-Shot learning approach.
3.Dynamic Calibration: By using OCT, the model continually updates its memory bank with new query images, allowing for real-time anomaly detection and improved responsiveness to new abnormalities.
4.Focus on Relevant Features: FOCT improves the model’s efficiency by emphasizing significant foreground elements, which is crucial for detecting anomalies in environments with sparse foreground data.
5.State-of-the-Art Performance: Demonstrated effectiveness across diverse datasets, achieving state-of-the-art results, which indicates its robustness and reliability in industrial settings.
limitations: limitations: 1.The framework's performance heavily relies on the quality of the pre-trained segmentation and backbone models, which might limit its effectiveness if not properly tuned or if unsuitable models are used.
2. The use of advanced techniques like CT and OCT may introduce complexity in implementation and require significant computational resources, especially in real-time scenarios.
3.The introduction of pre-trained models and SAM etc. has increased the complexity and computational requirements, and the authors should probably give more relevant analyses.
4.For addressing few-shot problems, it is beneficial for researchers to validate their algorithms using more robust and challenging datasets. An exemplary dataset for such evaluations is the Real-iad: A real-world multi-view dataset designed specifically for benchmarking versatile industrial anomaly detection. This dataset provides a practical and comprehensive testing ground to assess the effectiveness and robustness of new algorithms under real-world conditions.
paper_summary: paper_summary: To address the matching issue between queries and supports in Few-Shot Industrial Anomaly Detection, the authors proposed using Conditional Transport (CT) as a metric to compute the structural distance between two sets of representations to determine feature relevance. Additionally, they dynamically update the memory bank by utilizing SAM for foreground segmentation. Extensive experiments on various FS-IAD benchmarks demonstrate the performance of the proposed method.
strength_weakness: strength_weakness: 1. The feature visualizations of the experiments are reasonable.
2. Conditional Transport is used for the first time in anomaly detection.
limitations: limitations: 1. The author mentioned that FOCT is flexible and a Plug-and-Play Module, but there is no additional experimental evidence to prove its effectiveness in different scenarios.
2. The online method is used for updating the memory bank. However, the results in Table 3 show little improvement.
3. The random selection of support samples may lead to significant variations in the results under the low-data regime. It is recommended to select multiple sets of support samples and calculate the mean and standard deviation.
4. What does the red box in Figure 5 mean?
5. How is old episodic memory acquired, and is it the same feature format as semantic memory?
6. For certain objects like Carpet, Leather, Tile, and Wood, how to utilize SAM for foreground and background separation. Lacks some specific examples for demonstration
paper_summary: paper_summary: This paper proposes a novel memory-based TFS-IAD framework that primarily adopts a non-linear and theoretical guaranteed CT function between the memory bank and query images for 
precise feature matching. This framework can be plugged and played with any pre-trained backbones, such as WRN and SAM. The effectiveness is demonstrated on MVTec-AD and MPDD.
strength_weakness: strength_weakness: This paper focuses on online/transductive anomaly detection, which is a promising and practical perspective.
limitations: limitations: 1. The writing and organization are somewhat confusing and can be largely polished.
2. Some contents are missing/unclear, e.g., what the \delta in line 325-326 means? And for lines 628-629: “For a fair comparison, we keep the query images fixed and conduct 10 random support image splittings under 1/2/4 shot scenarios.” Can you provide a clearer explanation?
3. I also question the process of “episodic memory calibration”. In my opinion, the anomaly percentage of the query set is unknown, so how are the two hyper-parameters 10 and 40 in the “top 10 foreground and top 40 background feature” determined? And their ablation studies are missing.
4. To date, the results are not convincing. A series of important baselines are missing, e.g., GraphCore, WinCLIP, etc. Additionally, for the screw category of the MVTec-AD dataset, the AUC result does not show obvious improvement in the TFS setting. Some analysis is necessary.
5. To my understanding, the F1-max only depends on the anomaly score. So why this evaluation is missing in PaDim on MPDD and RegAD on MVTec-AD and MPDD?
6. For IAD, the efficiency of inference time also matters but the authors do not mention that.
paper_summary: paper_summary: This paper propose a memory-based transductive few-shot industrial anomaly detection framework. This framework adopts a non-linear conditional transport (CT) function between the memory bank and query images for precise feature matching. A pipeline called FOCT is designed to enhance the generalization capability of the aforementioned CT function.
strength_weakness: strength_weakness: 1. This paper propose a memory-based TFS-IAD framework that primarily adopts a non linear and theoretical guaranteed CT function between the memory bank and query images for precise feature matching;
2. A pipeline called FOCT is designed to enhance the generalization capability of the aforementioned CT function;
3. Compared with other competing methods, the performance of the proposed methods has improvement.
limitations: limitations: 1. The standard deviation of the results of multiple experiments is not reported;
2. The number of competing methods is suggested to increase, and currently there are only four competing methods;
3. The review about AD should be added in the related work section;
4. Insufficient description of foreground-aware semantic memory methods. In section 4.1, please explain why it is necessary to downsample the foreground and background with two different proportions of α and β. The previous content seems to mention only that the foreground and background have huge sizes.