# Pixel-Level Analysis for Enhancing Threat Detection in Large-Scale X-ray Security Images
## 3 Dataset  
SIXray [3] is the largest publicly available benchmark dataset for X-ray security images comprised of more than 1 million images, wherein more than 8000 of which are labeled for containing at least one of the following objects: gun, knife, wrench, pliers, and scissors. The dataset is further split into subsets that correspond to the increasing ratio of imbalance between positive samples and negative samples: SIXray10, SIXray100, and SIXray1000. Even so, the dataset only consists of image-level and object-level annotations. For the task of object separation, pixel-level annotations are required. Figure 3 shows three different versions of ground truth masks for the tasks of semantic segmentation, instance segmentation, and object separation. In semantic and instance segmentation, occlusion restricts each pixel to a single semantic group and object instance, respectively, whereas object separation allows pixels to be classified under multiple object instances that describe the area of overlap between objects. Thus, one of the defining attributes of the object separation task is the ground truth labels used in supervised training.  
Noisy or corrupted labels are not uncommon in the field of machine learning. It is reported that real-world datasets include 8% to 38.5% of corrupted labels [22]. As a result, an entire field of methods for training models robust to these noisy labels has emerged, which is beyond the scope of this paper. We found that the SIXray dataset contains only about 1% mislabeled negative samples, as illustrated in Figure 6. Still, including mislabeled negative samples in evaluating algorithms is counterintuitive to the goal of security applications. In practice, we want these samples to be predicted as positives, but during evaluation, they are considered otherwise. Furthermore, using ranking metrics such as classification m_AP_ adds further confusion since the predictions are sorted in descending order on the basis of the confidence scores. The best threat detection algorithms will predict these mislabeled negative samples as positive with very high confidence scores, which will cause a significant deterioration in the performance metric. Hence, in our evaluations, we removed the mislabeled negative samples from the test datasets to reveal the true strength of the algorithms. The proportion of mislabeled negatives is small enough to not affect the ratio of imbalance in the subsets. For comparison with previous works, we did not remove the mislabeled negative samples in the dataset we used for training our models, and we reevaluated previous works using the test datasets without the mislabeled data. Additionally, we have provided the list of the mislabeled negative samples as well as the pixel-level annotations of the data subset ([[https://github.com/jodumagpi/Xray-ObjSep-v1](https://github.com/jodumagpi/Xray-ObjSep-v1)]([https://github.com/jodumagpi/Xray-ObjSep-v1](https://github.com/jodumagpi/Xray-ObjSep-v1)), accessed on 8 October 2021).  
Figure 5: Mislabeled negative samples. Images are incorrectly labeled negatives despite containing (**a**) knife, (**b**) scissors, (**c**) scissors, (**d**) pliers, (**e**) wrench, and (**f**) wrench.