Semantic-Syntactic Discrepancy in Images (SSDI): Learning Meaning and Order of Features from Natural Images
Authors: Chun Tao
Timur Ibrayev
Kaushik Roy
Venue: Accepted by TMLR ()

Abstract: Despite considerable progress in image classification tasks, classification models seem unaffected by the images that significantly deviate from those that appear natural to human eyes. Specifically, while human perception can easily identify abnormal appearances or compositions in images, classification models overlook any alterations in the arrangement of object parts as long as they are present in any order, even if unnatural. Hence, this work exposes the vulnerability of having semantic and syntactic discrepancy in images (SSDI) in the form of corruptions that remove or shuffle image patches or present images in the form of puzzles. To address this vulnerability, we propose the concept of "image grammar", comprising "image semantics" and "image syntax". Image semantics pertains to the interpretation of parts or patches within an image, whereas image syntax refers to the arrangement of these parts to form a coherent object. We present a semi-supervised two-stage method for learning the image grammar of visual elements and environments solely from natural images. While the first stage learns the semantic meaning of individual object parts, the second stage learns how their relative arrangement constitutes an entire object. The efficacy of the proposed approach is then demonstrated by achieving SSDI detection rates ranging from 70% to 90% on corruptions generated from CelebA and SUN-RGBD datasets. Code is publicly available at: https://github.com/ChunTao1999/SSDI/.

Published in Transactions on Machine Learning Research (04/2025)
Semantic-Syntactic Discrepancy in Images (SSDI): Learning Meaning and Order of Features from Natural Images
Chun Tao* Department Electrical and Computer Engineering Purduc University
tao888purdue.cm edu
Timur Ibrayev*
tibreyeuSparruue.cmm edu
Department of Electrionl and Computer Engineering Purdue Universily
Kaushik Roy
kaushik Baaisks--puuuu..... edu
Department of Electrical and Computer Engineering Purdue University
Reviewed on OpenReview: https: // opter/vveerrvvee....m net/ forum? id= BotbGorZK2
Abstract
Despite considerable progress in image classification tasks, classification models seem unaf- fected by the images that significantly deviate from those that appear natural human eyes. Specifically, while human perception can easily identify abnormal appearances or composi- tions in images, classification models overlook any alterations in the arrangement, of object parts as long as they are present in any order, even if unnatural. Hence, this work exposes the vulncrability of having semantic and syntactic discrepancy in images (SSDI) in the form of corruptions that remove or shuflle image patches or present images in the form of puzzles. To address this vulnerability, we propose the concept of "image grammar", comprising "im- age semantics" and "image syntax". Image semantics pertains to the interpretation of parts or patches within an image, whereas image syntax refers the arrangement of these parts to form coherent object. We present semi-supervised two-stage method for learning the image grammar of visual elements and enviromments solely from natural images. While the first stage learns the semantic meaning of individual object parts, the second stage learns how their relative arrangement constitutes an entire object. The efficacy of the proposed approach is then demonstrated by achicving SSDI detection rates ranging from 70% to 90% on corruptions generated from CelebA and SUN-RGBD datasets. Code is publicly available at: /attps://github. com
1 Introduction
The task of image classification has significantly evolved with the advancements in deep neural networks (DNNs), the point of achieving performance comparable to humans. For such tasks, the state-of-the art (SoTA) models are based on convolutional neural networks (CNNs) (Krizhevsky et al. 2012
Simonyan
& Zisserman 2015 He et al. 2016) and vision transformers (ViTs) (Dosovitskiy et al. 2021 Liu ct al, 2021). The fundamental idea. behind these models was to develop the ability to identily image by understanding (and recognizing) its
object in an
(semantics). For example, when an image is classified as a "dog", it is because contains attributes of dogs, such as "fur", "dog tail", "dog paws". Indeed, such models learn the distribution of images belonging to different object classes by extracting
"These authors contributed equally and share first authorship.

---

Published in Transactions on Machine Learning Research (04/2025)
High
The degree sample appearing "ratural" to human eves
Low
ViT- ViT-B/16,224 /1,,24
CIE UNC CORRORR INCORRUPPEED
NATION
VIT
VIT
VIT
PAAAAA PAN
Beagle
42.3%
Bengle
54.6%
Beagle
47 77.0%%
Walker Hound
30.3 19.7%
PART
Waller
English
English Fachcund
11.4%
English Foshound
7.3%
Tertier
0.3%
Toy forrier
2.4%
Toy Terrier
0.4%
Bluttck
0.2%
Google Gemini
Prompt: Desembe this image.
Prommpt:
What type of d'dug is this?
Gemini: The image you me shows golden retriever dog standing in field at sunset dog is facing
Geminn: dog
image is most likelly a Golden Retriever. Golden Retrievers medium-stiee sizee, long-haatred dogs with gentle and friendly personality. They popular fumily pets working dogs, such as guide dogs for the blind and soarch and dogs.
has its mouth open as if it panting, The für the dog's back ilkuminsted the golden light the setting sun. The field in the backgromd green there fow trees the distance.
Figure 1: Examples illustrating susceptibility of various models (ViT-B/16,224 and Google Gemini Ad- vanced) to images with unnatural appearance. The outputs of both models seem to be completely unaffected by any of the abnormalities present in the input images when performing the given tasks.
low- and high-level semantic information from images and encoding them in hidden features of stacked layers (Hua et al. 2018: Orlego et al. 2021). The learning processes are designed to mainly rely on image- level labels: the global information about the object type depicted in the entire image. Hence, to learn object feature//attributee /attributes present the images labeled as the corresponding class. In other words, since there is no explicit information provided for each feature like "fur", "paws", "dog tail", the models must implicitly learn these attributes characteristic traits of dog by observing large variety of "dog" images. However, this learning process has underlying assumption that causes vulnerability in the pipeline of classification models.
classes, the classification models have to implicitly learn class features/
The vulnerability arises from assuming that input images always accurately depict the object in its natural appearance and composition. As a result, classification models do not explicitly learn how the constituent features of an object class are arranged natural images. For instance, they do not ensure that the "dog tail" is at the posterior end of the dog, or that the "paws" are in the lower part of the dog's body. In other words, the question being answered is What constituent class features and not *Does this image contain meaningful object of any observed class?". This vulnerability appears the discrepancy between human's 's ability to instinctively recognize natural images and classification model's tendency to be easily fooled into making high-confidence predictions for unnatural images. Figure [ illustrates how a vision transformer model (ViT-B/16,224) (Wu et al.] 2020) and a large multimodal model (Google Gemini Advanced) (Team al. 2023) are susceptible soen that the VIT model consistently predicts various dog broeds as the top-5 prodictions, disregarding the appearance of images that increasingly diverge from the clean natural image. Similarly, the Gemini model ignores the unnaturalness of the dog displayed in the image when asked to describe the image identify the type of the dog. We use the term Semantic-Syntactio Discrepancy in Images (SSDI) to refer to this vulnerability.
present in the image?",
this type of vulnerability.
can be
To address this vulnerability to SSDI corruptions in the visual appearance input samples, we propose the concept of "image grammar". The concopt akin to grammar in language (Gunter al. 1997 von Stechow 2019), and comprises both semantic meaning ("mmage semantics") and a syntactical structure ("image syntax"). "Image semantics" pertains to the existence and semantic significance of individual
2

---

Published in Transactions on Machine Learning Research (04/2025)
features defining an object, while "image syntaz" concerns the spatial arrangement and correct placement of features to depict an object as it would naturally appear in the real world.
Consequently, we propose a semi-supervised two-stage deep learning frumework that successively learns both "image semantics" and " "image syntar" mentioned above. This framework combines a deep chustering method with a bi-directional LSTM. The deep clustering method treats the image as a set of semantic features, while the bi-directional LSTM ensures that the features are learned in relation to their spatial neighbors, enforcing the model's understanding of individual features and their natural arrangement and composition. This approach aims to train a classification model to not only predict object classes but also assess the degree of naturalness in the appearance of an image. Furthermore, the framework is specifically designed to learn the concept of "image grammar" in a strict setting that assumes SSDI corruptions are neither generated nor used during the training phase, Such a setting is motivated by the idea that, similar to human capabilities, the meaning and the order of features should be inherently learned together solely from natural images.
2
Aomantic-Syntactic Discrepancy in Images (SSDI)
2.1 Problem Definition and SSDI Types
Semantic-syntactic discrepancy (SSDI) occurs in images that have visually identifiable semantic features but look unnatural to humans in their appearance and composition due to incorrect arrangement or missing some of those features. Figure illustrates examples that expose the described vulnerability of classification models. As humans, we can readily recognize that the shown images aim to represent dogs due to the presence of recognizable dog features. We can also perceive that the images in the three rightmost columns ((c)-(e) are more likely to be unnatural or altered in appearance (compared to images in columns (a) and (b)). We can make such distinction even, for example, with "puzzle" image with rearranged patches that we have not encountered before. In contrast, a classification model trained on large-scale datasct like ImageNet (Deng et al. 2009) consistently assigns output labels corresponding to various dog breeds for the images in columns (b) (e) solely based on the presence of identifying semantic features (the semantics). disregarding any abnormalitios the overall appearance (the syntax).
In this work, we explore three types of corruptions causing SSDI. The first corruption is patch shuffing, where a subset of image patches is randomly swapped. The second corruption is patch blackenóng, where subset of image patches is blackened out. The third corruption is puzzle solving, where a subset of image patches is swapped, similarly to the patch shuffling. However, unlike patch shuffling, in puzzle solving, all possible permutations are generated for the specified number of image patches allowed for corruption. Puzzle solving is then assessed based on the capability to determine the original image out of all its variations. These corruptions set the basic framework for image manipulations inducing semantic- systatiiccsttttiic discrepancy in images: rearranging object parts but keeping them visible enough to identify the original object. While they might not define the exhaustive list of all such transformations, they serve as simple exemplar to illustrate the strong effects of feature presence on the decision-making of various models regardless of how natural composition of object parts changes within the image.
2.2 Pervasiveness of SSDI in the Existing Methods
As a part of this work, the initial set of experiments is to evaluate the performance of the various image processing methods against samples with semantic-sntactic discrepancies. Two types of methods were considered: relying only on image processing as well as on the combination of images and language (also known lasion-language models, VLMs). These experiments have three major goals. The first goal is to validate the presence of the vulnerability by illustrating the cases when patch shuffling and patch blackening have minimal impact on model performances despile significantly impactful changes to the image syntax. The second goal is to show when such manipulations become SSDI corruptions. In particular, through the combination of quantitative results and qualitative asscssment of the corresponding samples, we will illustrate the range of corrupted image patches (in terms of the size and their count) that makes the corrupt image fall under the definition of SSDI given in Section 2.1 The third goal is to demonstrate that even
3

---

Published in Transactions on Machine Learning Research (04/2025)
The degree of a sample appearing "natural" to human eyes
High
Low
Clean and Uncorrupted Images
Corrupted Images crafted by:
Adversarial Images
Part
Patch Permutations:
Blackout
Mild
Strong
(a)
(b)
(c)
(d)
(e)
Low
The probability of features, understandable to human eyes, to be correctly inferred by the classification models
High
Figure 2: Examples explaining the vulnerability of classification models to semantic-syntactic discrepancies in images (SSDI). While all images appear to display different dogs, the visual appearance and composition of the images in columns (c) (e) deviate from clean and uncorrupted images shown in column (b). This discrepancy, however, is not caught by the current classification models, which solely focus on the presence of features. This can be contrasted with adversarial examples (a), which target the decision-making of models while preserving the natural visual appearance. Note that adversarial examples are not considered in this study but are used to paint complete picture.
multimodal models are vulnerable in their own ways to SSDI-type corruptions by not explicitly reflecting any abnormalities unless explicitly requested by the user.
2.2.1 Purely Image-based Methods
Table and Table illustrate the classification accuracy of ViT-B-16 model on ImageNet2012 val samples under patch shuffling and patch blackening, respectively. Various patch sizes and numbers of patches were used to generate corrupted samples. With the original images being of size 224 x 224 pixels, it is possible to divide each image into patches of different sizes as represented by column headings. Consequently, there are different numbers of patches that are available for the corresponding SSDT type as represented by row headings. The selection of which patches are corrupted happens random. Hence, order account for randomness, each cell represents the average accuracy of experiments with different random seeding, For example, shown in Table 2 the model accuracy drops from 81.07% base accuracy to average of 69.16% when 2 out of 4 patches of size 112 x 112 were removed from every sample in ImageNet2012 val set. Appendix C contains tables illustrating performance results along with standard deviations for ViT-B-16 and ResNet-50 models, whose behavior matches that of the ViT model.
Based on the quantitative results of classification accuracy alone, the following observations can be made. First, for patch sizes equal to and smaller than 14x14 pixels (left halves of the tables), the model performance degrades drastically greater number of patches are corrupted. For patch blackening, the degradation
4

---

Published in Transactions on Machine Learning Research (04/2025)
Table 1: Performance of ViT-B-16 model on ImageNet2012 with "patch shuffling" SSDI corruptions.
Patch Size
12564 putches
3136 patches
1024 patches
781 potches
256 putches
196 putches of
61 patches of
40 patches of
16 putches of
patches
x
4
x
14
16xx 16
28 x 28
52 32
112 x 112
S1.07
31.07
S1.07
81.07
81.07
S1.07
81.07
SL07
81:07
$1.07
80.50
80.49
85
30.71
79.90
70.06
76.11
60.36
70:84
79.70
Number of Corrupted Patches
73.73
06.25
51
196
74.64
1024
65.63
28.08
0.41
2046
10.00
0.65
31.6
33.80
0.24
4136
21.45
8132
0.82
12544
0.29
Correspoading exammpees are shown in Figure
High
The degree of a sample appearing "natural" to human eyes
Low
Patch
Shuffling
SSDI
Patch Blackening SSDI
Original image of 224x224 pixels size
196 of 12544 corrupt patches of 2x2 size
64 784 corrupt patches 8x8 size
32 out 196 corrupt patches of 16x16 size
16 out of 49 corrupt patches of 32x32 size
out corrupt patches of 112x112 size
Figure Examples of patch manipulations applied to randomly picked ImageNee20122 val set samples.
reaches random guessing (0.10%) since such image manipulations result in trivial processing of black images. Second, for bigger patch sizes (above 16 16, right halves of the tables), the degradation is much more graceful. With the exception of random guessing in the case of patch blackening, modifications made to the samples are disregarded by the model in about half the cases. Finally, it can be observed that the accuracy remains relatively high compared to the base accuracy of 81.07% for a large portion of the table (green and yellow colored regions). While this behavior might have been favorable purely in terms of detecting object fcatures, also highlights that the model is largely "unbothered" by image manipulations.
Figure demonstrates some random samples with different degrees of patch shuffling and patch blackening manipulations. Qualitative examination of these examples suggests explanations for the above quantitative obsorvations. Image manipulations with small patch sizes (2 2 and 8 x 8) are much less impactful in terms of both being perceptible to human eyes and affecting object parts as a whole. Instead, such changes would probably be perceived as noise both by humans and vision models, with its severity increasing with the number of corrupted image patches. Such cases would lic outside the definition of semantic-syntactio discrepancy, the noise affects semantics more than syntax by "destroying perpoiiiinnrrringg"..g /erasing" the object features, instead of changing their natural arrangement.
5

---

Published in Transactions on Machine Learning Research (04/2025)
Table 2: Performance of ViT-B-16 model on ImageNet2012 with "patch blackening" SSDI corruptions.
Patch Size
12564 putches
3136 patches
1024 patches
781 potches
256 putches
196 putches of
61 patches of
40 patches of
16 patches of
patches
x
4
x
8
14
x6 16
28 x 28
52 52
112 112
S1.07
31.07
S1.07
81.07
81.07
S1.07
81.07
S1077
81.07
$1.07
.8
05
80.03
80.95
80.53
77-49
2
00:00
06
50.97
80.52
80:62
70.55
69.16'
40.77
79.70
Number of Corrupted Patches
16
40.01
79.75
76.30
70.94
78.61*
256
74.24
H1
784
72.40
56.67
0.78
1024
70.23
46.25
0.10
2046
61.70
4.80
81.80
52.08
0.10
4136
46.19
8192
9.88
12544
0,10
Corresptading cxnamples are shown in Figure
3
On the other hand, the manipulations with the large patch sizes (16 x 16 and above) do not necessarily target the fine-gr ahn-grraullrrty of individual object features. Instead, they affect object features as a whole, such as obscuring "eyes" and "ears", or swapping "body parts". This explains why the accuracy results show change at the level of manipulations affecting up to half of the image. We acknowledge that the degree of naturalness is subjective measure, similar to estimating saliency and attention in scenes, as it varies from person-to-person or swapping 2 out of 4 patches of size 112 x 112 is more conspicuous to human eyes than that at 64 out of 784 patches of size 8 x 8, both of which result in the same accuracy performance of the model. As a result, this reflects on the definition of SSDT corruptions. Specifically, for the given ImageNet2012 samples of size 224 224 pixels, patch shuffling and blackening the patch sizes equal and larger than 16 16 pixels cause images to appear unnatural while the underlying object parts remain visually identifiable.
However, we also highlight that the amount of changes induced by removing
2.2.2 Image and Language-based Methods
Table iillstrates the experimental results conducted on Meta's LLaMA large-scale vision-language model. Notably, LLaMA version 3.2 with 11 billion parameters available HuggingFace hub was utilized. The experiment was conducted as follows. A subset of 3000 images was randomly selected from ImageNet2012 val set, such that there are 3 samples from each one of the thousand classes. Samples with original size 224 x 224 pixels were corrupted based on one of the ten configurations shown in the left column. Then, these samples were processed by the multimodal model along with one of the three language prompts. These language prompts are:
Prompt 1: What is the class of the object shown in the given image?
Prompt 2: Describe the given image.
Prompt 3: Does the given image oppear natural or unnatural?
Each of these prompts provides a different degree of guidance in terms of revealing the possibility of abnor- malities being present in the given images. Prompt 1 only requests the class of an object, mimicking the bchavior of the classifier. Prompt 2 requests a general description an image. Prompt 3 implics that there are possible deviations present in the image. Please note that in all three cases the model is given complete freedom in its responses, such that extra conditions are set (e.g., respond with single sentence).

---

Published in Transactions on Machine Learning Research (04/2025)
Table 3: Performance of Meta's LLaMA model on 3k samples of ImageNet2012 with various corruptions and with different language-based prompts. Each percentage value represents the percentage of LLaMA's s responses that explicitly reflected that given samples looked "unnatural, corrupted, or lampered with". Please refer to the experiment details in Section 2.2.2
SSDI corruption
Prompt given to Meta's LLaMA model
(patch size, total of patches, # of corrupted patches)
What is the class of the object
Docs the given image
Describe the given image.
shown in the given image? !
appear natural or unnatura??
Patch Shuffling
112 x 112, 4,
0.03%
0.67%
93.23%
32 x 32, 49,
4.60%
46.70%
90.67%
16 16, 196, 32
4.77%
35.60%
98.60%
8 x 8, 784, 64
4.53%
30.93%
74.37%
2 x 2, 12544, 196
0.27%
8.03%
97.20%
Patch Blackening
112 x 112, 4,
1.83%
38.40%
51.53%
32 x 32, 49, 16
2.50%
44.03%
85.47%
16 x 16, 196, 32
1.43%
31.37%
77.23%
8 8, 784, 64
0.07%
31.00%
82.60%
2 x 2, 12544, 196
0.00%
33.03%
89.03%
After all pairs corrupt images and prompts are processed by the model and its responses are collected, we utilized OpenAl's ChatGPT to evaluate the output responses. Specifically, ChatGPT evaluated Meta's LLaMA responses with the following evaluation prompt: "Does the given response mention anylhing that indicates that the given image is unnatural, corrupted,
tampered with? Please answer with yes no " Based on such evaluation setup, Table 3 shows the percentage of samples for which the VLM indicated that the given images were corrupted in some way.
Based on these results of the vision-language model, the following observations can be made. When asked directly whether there are any abnormalities with the given images (with prompt 3), the detection rate of the VLM model is steadily high. This means that prior needs to be set in order for the VLM actively search for SSDI corruptions. On the other hand, interestingly, the detection rate is substantially lower in the other two cases. In case of prompt it is indeed possible to argue that the model was not supposed to provide any extra observations. However, prompt 2 is the most generic, arguably serving as the go-to option for general applications. Under all generated corruptions, the VLM mentions their possibility in the images in fewer than half of the samples. As a result, it can be said that semantic-syntactic discrepancy remains a relevant issue for vision-language models, which manifests itself when no prior precautions or warnings are supplied through the correct prompting.
2.2.3 Puzzle Solving as a Proxy Task
Part of the problem with SSDI corruptions is that the current classification models do not have explicit way of estimating the degree of naturalness of input images. Hence, conducted experiment in which puzzle solving SSDI corruption serves as proxy for the corruption detection task. Particularly, as was described in Section 2.1 for the puzzle solving SSDI lype we generate all the possible permutations of an image given the number of patches into which it is broken down. For ImageNet2012 val samples of original size 224 x 224 pixels, we only considered puzzle solving when images are divided into equally sized square patches of 112 x 112 pixels. As a result, for every image we generated 24 variations of the image resulting from all 4! permutations of image patches (including the original image).
7

---

Published in Transactions on Machine Learning Research (04/2025)
Table 4: Performance of different models on ImageNet2012 samples with "puzzle solving" SSDI corrup- tions, illustrating the capabilities of each model at (1) prodicting the class of an object shown in the given image and (2) determining the most natural arrangement of an image (i.e. the original image) out of all possibilities. Please refer to the experiment details in Section 2.2.3
Prediction Accuracy
Correct Class of an Object
Correct "Puzzle" Arrangement of an Image
NesNet-50
78.04%
0.12% (59/50k)
ViT-B-16
78.98%
78% (6390/50k)
Meta's LLaMA
N/A
4.00% (120/3k)
The models are then assessed by their ability to determine the original image with the natural arrangement of object parts out of all variations. For classification models (ResNet-50 and ViT-B-16), the prediction of the correct puzzle relied on the output (softmax) confidence scores. Specifically, all puzzles (including the original image) were processed by these models. For cach image, the puzzle with the highest output confidence was chosen as the model's prediction for the image with the correct arrangement. Furthermore, the output class of the predicted puzzle also served as the model's prediction for the class of an object shown in the image. For the VLM model (Meta's LLaMA), all puzzles (including the original image) were given in a single request along with the following prompt: "which one of these images appears be the most natural? Give only the numeric value of the inder of the image.". To account for the cases when the model response includes anything more than the index, the responses were post-processed to verify and, if necessary, to clean the putp to refer to one of the puzzle indices. The class labels were not requested as the language models would predict classes with open vocabulary.
Table shows the performance of different models at correctly determining the image with the original natural arrangement of features/ caauuurrs/ojeer /object parts. It. can be seen that all models lack the capability to distinguish the original image from all of the possible patch shuffling variations. The highest accuracy solving the puzzle was achieved with the vision transformer, which might be due to their inherent training procedures relying on processing images a set of patches with positional encoding. The most interesting observation, however, is that despite the models failing to predict the image with the natural arrangement, there is almost no drop in the accuracy of object class predictions with respect to the base accuracies. Since detecting the presence of object features is enough to make the classification prediction, the significant difference between class and puzzle predictions demonstrates the corresponding gap between their comprehension of image semantics and image syntax. The main concerning implication is that the high classification accuracy means the models internally generate and rely on a similar set of features when processing the correct image and the image with any unnatrall appearance.
2.3
Importance and Application
The ability to recognize and classify objects is foundational task in computer vision. However, current classification models primarily focus on the presence of features rather than their spatial arrangement, making them vulnerable to SSDI corruptions. Addressing SSDI is important in safety-critical and real-world applications where models must not only identify objects but also evaluate their structural validity.
While our study uses controlled SSDI corruptions (e.g., patch shuffling, missing parts), similar issues arise naturally in many real-world scenarios, such as facial recognition security, medical imaging, and robotics. For example, face recognition systems often misidentify individuals when their features are partially occluded or misaligned (c.g., wearing masks, disguises, or under poor lighting). Consequently, if a model solely relics on feature presence, it may wrongly authenticate or misclassify identities. A similar example can be drawn in medical scenarios, where diagnostics rely on anatomical structures. Artifacts, occlusions, or unnatural feature arrangements can lead to false diagnoses if the model lacks awareness of natural structures. Such scenarios are especially prominent today, as generative models create increasingly realistic images, but sometimes fail to preserve natural image syntax.
8

---

Published in Transactions on Machine Learning Research (04/2025)
As DNN models move beyond simple classification tasks toward visual reasoning and multimodal learning, addressing SSDI is crucial. Our experimental results show that even advanced vision-language models struggle to detect SSDI inconsistencies, failing to flag unnatural images unless explicitly prompted. This underscores the broader need to develop a learning approach that captures both semantics and syntactic structures in the visual domain.
SSDI vulnerability represents a distinct failure mode that is not captured by existing robustness paradigms, such as common corruptions, adversarial attacks, and out-of-distribution (OoD) shifts. Unlike common corruptions, SSDI does not degrade images randomly but instead disrupts the spatial integrity of object features. A shuffled or misarranged object may retain all its semantic components, yet fail to represent a meaningful whole. Unlike adversarial attacks, SSDI does not involve imperceptible pixel-level perturbations. Instead, SSDI creates visible, unnatural compositions that models assign high-confidence predictions to, exposing their reliance on feature presence over structural coherence. Unlike OoD shifts, SSDI does not involve a semantic shift to new domain. Instead, SSDI occurs within the same category as the training data but in incorrect arrangement. that does not appear in natural images.
2.4 Learning Setting
The contrast between human perception and classification models in these examples underscores three crucial properties ncoded to address this vulnerability: (1) classification models need to learn to estimate the degree to which an image depicts natural occurrence in the real world, (2) based on this degree, they need to be able to distinguish between natural and unnatural images, and (3) preferably, they should be capable of learning this based solely on natural images.
This work proposes a way to enable classification models with properties (1) and (2) in a learning set set ting motivated by property (3). In particular, the method describes how the architecture of a classification model and its training can be modified to allow it to distinguish between natural and unnatural images by only observing natural unaltered images during training. This setting is based on the combination of two reasons. First, humans can instinctively recognize unnaturalness of different kinds even when they never came across such examples. Both the meaning and the order of features are learned together without any need to see images with some parts removed or shuffled. Second, it is challenging to consider and train on all the possible ways samples with semantic-synlactic discrepancy can be crafted (including, but not limited to corruptions described in this work). For example, if an image is divided into n X n patches, assuming that only the original image depicts a meaningful object, the number of unnatural samples becomes (n²!-1) Not only does this become intractable for values of n > 3, but also has no direct way of ensuring that all of these samples do not depict meaningful objects. Hence, it is more intuitive to teach "What the natural arrangement. of features should look like!" rather than " "How natural and unnatural images differ?".
3 Related Works
To the best of the authors' knowledge, the problem of SSDI vulnerability is not yet well-established. This makes it difficult to contextualize it as standalone research direction. Nevertheless, the issue of SSDI can be considered in the context of two categories of works: from the perspective of learning the compositionality of visual representations and from the perspective of vulnerabilities in computer vision.
3.1
Compositionality of Image Representations
Learning image representations creates basis for visual tasks, such image classification (He et al, 2016 and object detection (He et al. 2017 Lin et al.] 2017). Although various supervised (Krizhevsky' ct al. 2012) and
(He et al. 2022) methods have been proposed to learn the underlying distribution of image feat reliance purely on the presence of features has recently become more prominent (Thrush et al., 2022). In the work of Yukssegonull et al. (2023), the authors discuss and analyze the effects of naive reliance on contrastive pre-training and accuracy metrics. They conclude that large vision and language models, which currently serve as the basis for a wide range of applications, disregard composition, i.e., the underlying structure, of both visual and language inputs. Their findings support. the ideas and
9

---

Published in Transactions on Machine Learning Research (04/2025)
highlight the relevance of SSDI vulnerability discussed in this paper. The work of Qin et al. (022 proposed a solution for ViT-based models (Wu et al., 2020) through patch-based negative augmentation. From this
perspective, our work serves as an alternative approach, which proposes a way of learning compositionality along with the semantics of features without the need for generation and reliance on negative samples (as described in the Learning Setting subsection of Section 2). Even though our experiments focus on CNN- basod models, is possible extend the method to vision transformers by considering them as the feature extractor model.
3.2
Vulnerability of Computer Vision
The SSDI vulnerability, where differences in the visual appearance of an image are noticcable to humans, stands in contrast. to adversarial examples (Szegedy al. 2013 Carlini & Wagner 2017 Madry et al.,
2017 Wang & He 2021 Gubri et al.
2022 Zhang et al.) 2023), where the adversarial image and the clean
image appear visually similar to humans. Adversarial examples have been extensively studied in the works of Madry et al. (2017): Shah et al.
(2023); Mo et al. (2022); Andriushchenko & Flammarion (2020), which explore methods to prevent detrimental changes in the output predictions of classífication models. However, these approaches do not apply to SSDI, as they do not detect abnormalities in appearance. Moving away from the broader concept of adversarial examples, [Hendrycks & Dietterich (2018) rntuced a benchmark, ImageNet-C, C, for common corruptions and perturbations. While ImageNet-C C addresses changes in the natural appcarance of images through noisc color manipulations, it does not address cases where corruption stems from the rearrangement of natural feature order without affecting classifier predictions.
4 Methodology
In this section, we outline our proposed classification framework, which detects SSDI using semi-supervised. two-stage approach. The first stage, which we refer to as part semantics, focuses on learning " "image semantics". The idea is to create a pixel-wise segmentation map of the imput image that produces a (limited) set of clusters, each representing meaningful object parts (object semantics). The second stage then utilizes the part semantics to learn how their relative arrangement constitutes an entire object, i.e., the "image syntax". The idca is for the model to learn the expected arrangement and composition based on part semantics obtainable from natural images. During inference, the deviation from the expected arrangement and the composition learned from natural images serves as the metric to detect SSDI.
Stage 1: Learning Part Semantics As mentioned, the first stage focuses on learning the meaningful attributes of an object- - its part semantics. To learn this, the input pixels are segmented (grouped) into a set of classes that represent individual object parts instead of the objects themsolves. Althongh fully unsupervised (no labels) or weakly supervised (only object class labels) approaches are more appealing, our observations showed that these methods did not produce semantic maps with the desired level of detail and accuracy. These approaches only allow the separation of the entire object from the background (e.g., the entire face), but not its parts (e.g., nose, cyes, mouth). This is because there is greater variation in color between the object and the background than among the parts within the object itself. Hence, this stage was implemented in semi-supervise approach by using only a fraction of the ground-truth semantic segmentation maps with the deep clustering (Caron et al. 2018 Cho et al. 2021
The training procedure shown in Figure 4 Consider a set of input images -,,i=1,... N. Let a subset of them have pixel-Ievel annotations map: = I,..., M,M & N, where p stands for the p-th pixel. We assume that we have C' semantic classes in the pre-processed ground-truth (GT) annotations and V(i,p), mip E [0, 10.1....,C C 1]. The feature extractor has an embedding function fe, which produces pixel-level feature vectors Zip = fol Zip E Rd.
First, we fine-tune the DNN feature extractor under semi-supervision. Pixel-level embedding features are fed to a linear classifier Iws w E RdxC so that the feature dimensions are projected onto the number of semantic classes C. (Note: once the fcature extractor has been fino-tuned, the classifier g can be discarded.) For all images 2, that have GT masks, the resulting pixel-level prediction becomes gu (2ip). The fine-tuning is then realized by minimizing the cross-entropy loss between pixel-level prediction g (Zip) and GT segmentation
10

---

Published in Transactions on Machine Learning Research (04/2025)
Input: Aligned image (reszed]
Stage Part Semuntic clustering
Celeb/Ha Mask (7 semantic dasses)
Multiscall feature maps
Piaz hhvl-level features
Resnet-18 Model
FPN Model
FC classifier
DNN: detector
Color transform 1
Pisel-evel features p
(brightness, contrast, saturation, etc.)
ccentrood * -features
Get trarfoom
Ficus (11 (00)
K-means Clustering
Latas (p (1
DNN: Mask detector
Color transform
Geometric transform
(12)
Late ( w.ll))
Lousettnne 2
Cetue (1..) 11100)
DeepCluster (PICIE) Training
DeepCluster Testing
semanticcs clusters
Output: part semantic masks
A
Placi level features Pi
DNN: Mask detector
Fainwise distance
Figure 4: The first stage of training SSDI detection pipeline: learning part semantics of objects.
mask mip: as shown in Equation
ming,w Lor(a mip) where LCr - log exp(g=(Sip) [m,,]]
(1)
Amo exp(gw [[])
As second step, deep clustering technique is used to expand upon semi-supervised knowledge. We used PICIE, deep clustering technique :Cho et al., 2021) in our approach. The core idea of PICIE is perform K-means clustering while allowing the model to account for inductive biases in the form of photometric invariance and geometric equivariance. In particular, photometric invariance accounts for the color variation (e.g., color jitter), whereas geometric equivariance accounts for the size and orientation variations (e.g.. random crop) of semantic features in the given image. As shown in Figure 4 this is implemented by sampling each input into two transformation streams. In the first (upper) stream, an image F; is transformed according to photometric transformation p(1), processed by the model, and the resulting features transformed according geometric transformation G(1) Contrary, in the second (bottom) stream, an image 24 is transformed according to photometric transformation p(2), followed by geometric transformation G(2), and finally processed by the model. Geometric transforms (()) and (( share the properties, such the crop size, to ensure the resulting features are of the same dimensions. As result, pixel-level features before K-means for two streams can be formulated as defined by Equations 2
2/p c(' (fo(P!
(1)
(2)
(2) (P!
(3)
ip
Given these pixel-level features, PiCIE optimizes the set of losses for K-means clustering as well as for induc- tive biases. Specifically, the K-means clustering is achieved by minimizing the loss defined by Equation
L LKmeans = - Σ
1 11.12
4
11

---

Published in Transactions on Machine Learning Research (04/2025)
where 2ipk indicates that the distance of each pixel-level feature is minimized with respect to their corre- sponding closest cluster centroid pk-
The loss used to train inductive biases comprises two parts. The first part trains the model to extract pixel- level features that are consistently assigned to the same cluster centroid when a specific set of transformations is applied. In other words, if the same set of photometric (P) and geometric (G) transformations is used, this component of the loss ensures that the resulting features are as close as possible the corresponding cluster centroid representing those features. Since this component solely considers the pixel-level features from cach individual stream, it is denoted Lwithin. However, this loss alone does not guarantee that the model will ignore variations when a different set of transformations is applied to the same image (i.c., it docs not ensure invariance to photometric transformations or equivariance to geometric transformations). Therefore, the second part trains the model so that pixel-level features from two streams are assigned to the same cluster centroid even when different sets of transformations are applied. Because this component compares features between the two streams, is denoted as Leross. The total loss to minimize is Liotal Lusthin + Leross:
described by Equations
6 and 7
Cwithin = Σ E, Coo (1) Mpp. () + LDo (2ip s Dip 1) Mip
(2)
(2) (2)
(5)
Lerose =
Σn 1= E, CDO (1) Vip p)= + LDC Sip
Vip p)
(1)
(6)
exp( d(zip. Hy))
where
Cpc( v. = - log Ex=1 exp((d((2a) Pk))
(7)
where K is the total (pre-sclected) number of clusters, is the matrix of cluster centroids, y is the centroid index in j that is closest to Zip; and -( is the cosine distance. As shown in Figure | the result of the first training stage is a DNN that can produce semantic masks for object parts, i.e. part semantics. The key is the desired gramularity, such that the resulting part semantics are consistent for any given portion/patch of an image.
Stage 2: Learning Image Syntax The second stage is the core of training the pipeline to detect SSDI. The idea is to utilize the relation between neighboring part semantics. For example, if we know that some portion of an image contains a human nose, the expectation is to have two eyes and a mouth depicted above and below it, respectively.
The first step in achieving this is to obtain different patches from within the input image. The works of FALcon [Ibrayev et al., 2023) and GFNet (Wang et al., (020) explore processing imput in a patch-wise wise soquential manner with only the class label given as the supervision. In this work, we consider a simple and generic approach to dividing images, i... the model on its own does not make any special decisions on how the patches are obtained. For the two datasets considered in this work, CelebA and SUN-RGBD, the patches are obtained as five crops around the center of the image (Figure and as a zig-zag pattern from top to bottom, from left to right (Figure g respectively.
The second step is to traverse through the set of image patches, extract their corresponding part semantics, and learn the relation between the part semantics of neighboring patches. Suppose for image 260,
(6) for t t= 1,..., N, we extract a total of G patches and obtain their corresponding part semantics m,
using the DNN trained the first stage. The image syntax depends on the presence of the object parts with respect to each other rather than the exact part semantics (0) Hence, each of them is converted into a part semantics vector st which is the ratio of pixels belonging to each class in the part semantics.
Figure 5 how the image synlax is learned based on part semantics vectors using bidirectional long short- term memory (bi-LSTM). The bi-LSTM is used because both directions in the traversal of part semantics are valid relations to be learned. If it is parameterized by weights W and biases b, then based on the inputs (6) m, and s as well as hidden states h at the processing step t, the bi-LSTM makes two predictions: (i.for) Lw.b (m("), (6) n, ((.foe)) for the next semantics # and pt (f.back) = Lw. (m((). ((()), .n. 1(f.back)) for the Pi
(1)
(6) previous semantics 1- Across the set of training images x, the learning goal of bi-LSTM is to capture the transition patterns between image part semantics. This is achieved by optimizing the mean squared error
12

---

Published in Transactions on Machine Learning Research (04/2025)
Part semantic mask (pradicted) m9g
semantics
Stage 2: Image Syntax
Beckground 6.011
beciground:
0.507
Seckground: 0.043
Racigraand:
halr
0,28
pait
0.215
Hel:
Haln:
Harn
Fonehaed:
0.60
Eyes
427
Nose:
Mouth
Nosc:
Mouth
0074
0/423
Moath
Mose:
0.811
Mouth
0.735
0052
Nack
Nock:
Mock
Mede
encoder
combine
combine
combine
combine
concat
(Ap, Fg)
bi-LSTM
bi-LSTM
bi-LSTM
bi-LSTM
bi-LSTM
(Ag.
Losterst
forward
Lossyss
fonwand
Lostyes
Jammar
LosssgeE
Losssess
Loss
Lossagst
Lossuse
5,
p tachward
hockwardd
Ps
Figure 5: The second stage of training SSDI detection pipeline: learning image syntax.
Part semantic mask (Averaged over trainset)
Grammar Validation Methods
Bi-LSTM + next semantics
11 Savg.1
22 Savg,2
lppt ir
-
2 + ss
i.back back
1|
epred
Pt
S
t=2
t=1
5 Savg.5
2
Bi-LSTM + avg. semantics
epred
lpt
iffor
for
21
+
Ipt
Lback
back saug.t
|
t=2
t=1
3
mloU w./ avg. semantics
Mavg, Savg mavg.3, Savg.3
eprd
mloU (m), mavg,t)
Figure 6: Three methods of quantifying the semantic-syntactic discrepancy in images, SSDI.
loss formulated by Equation
EMSElP (0) (0) EM" (En (f.for) 1
G-1 (t,back) Ilp
b/ 2(0012)
(8)
Inference: Detecting SSDI The detection images with semantic-syntactic discrepancy (SSDI) is achieved by processing images similar patch-wise traversing manner and verifying whether the arrange- ment of object parts matches the expected arrangements learned from natural images. Figure G illustrates three methods of formally quantifying this process. The first method detects SSDI based on the relation of part semantics present in the image itself. Similar to the training process, for every image patch based its part semantics, the bi-LSTM predicts the part semantics of its neighboring patches. The error is then computed as the average difference between the predicted part semantics and the part semantics estimated by the DNN.
The other two methods detect SSDI based on the memorization of the average part semantics of every pixel. Specifically, each pixel assigned the most frequently predicted semantic class over the training set. For a test image, the error is then computed the average difference between the predicted part semantios and
13

---

Published in Transactions on Machine Learning Research (04/2025)
Correct image
Patches
Patch masks
Corrupted image
Patches
Patch masks
Figure 7: Image patches and the traversal sequences of processing image syntax of CelebA.
Pred error (lorward)
compped
Pred error (backward)
correct corrupted
correct comuupted
correct comrupted
error (both)
0.5-
95
1.0
0.4-
0.0
Count
0.3
33
0.6
Patch iteration
Patch iteration
Patch iteration
Total residual
Figure 8: From left to right: the forward, the backward, and the total bi-LSTM prediction errors as well as the histogram of total errors over test samples CelebA dataset.
the part semantics memorized based on the average semantics of training images. In the third validation method, the mean intersection union (mIoU) is used instead of the difference, which is common segmentation metric (Garcia-Garcia et al. 2017).
In all three cases, we obtain a quantitative measure of estimating the discrepancy in the semantic-syntactic relation in the form of computed errors. To use it as the detection during inference, we need to separate the error values for images with a natural and unnatural appearance. This achieved by generating images with semantic-syttactic discrepancy and estimating thresholds for each method based on the validation set images. Specifically,
plot histograms of corrupted and original images from the validation set based their total prediction errors. The threshold is then chosen as the value that maximizes the separation between the two distributions. An example of this can be seen in Figure g where the green dashed verlical line marks the selected threshold.
5 Results
Generation of SSDI and Performance Evaluation this work, we consider three types of corruption causing semantic-syntactic discrepancy images, as described in Section 2 For cach considered datasot, SSDI corruptions are generated using half of the test images chosen random. All corruptions are considered separately, i.e., the paper does not consider simultaneous mixture of different. SSDI corruptions.
As described in Section both natural and corrupted images with SSDI are processed by the framework in patch-wise traversal manner, and the (residual) errors in predicting forward and backward part semantics by the bi-LSTM are computed. The threshold determined based on the validation set used to distinguish between natural and corrupted images. We evaluate the performance based on two metrics: (1) the detection accuracy (i.e., binary classification accuracy) and (2) the detection rate (i.e., recall) described by Equation
TruePositive (TP) + Tegetieaaivee (TN) Det. Acc. FalseNegative (FN) TP + FalsePositive (FP) + TN and Det. Rate FN TP
TP
CelebA The first dataset is the aligned-and-croppee CelebA Liu al. 2015 dataset with 202,599
RGB images resized to 256 256. The DNN feature extractor consists of Feature Pyramid Network
14

---

Published in Transactions on Machine Learning Research (04/2025)
Table The performance of the proposed approach on detecting SSDI corruptions on CelebA.
Dataset
Part
Grammar
Shuffle
Shuffle
Black
Puzzles
Puzzles
Semantics
Validation
Model
Method
20x20
30x30
20x20
20x20
30x30
CelebA
ResNet-18-FPN (ours)
Bi-LSTM + next semantics
65.66
76.22
70.91
86.74
92.51
(68.67)
(79.72)
(67.04)
CelebA
ResNet-18FFPN (ours)
BI-LSM + avg. semanties
61.53
68.89
65.84
85.62
91.15
(71.72)
(80.48)
(72.38)
CelehA
ResNet-18-FPN (ours)
mlaU w./ avg. semantics
60.04
69.59
60.90
99.25
99.58
(56.16)
(59.60)
(61.87)
CelebA
SemanticGAN
BiLSTM + next semantics
61.35
70.20
68.98
82.37
88.09
Li el ml.
2021
(69.87)
(73.68)
(71.04)
CelebA
SemantieGAN
Bi-LSTM + avg. semantics
58.26
63.57
61. 21
81.23
86.79
Li el al.
2021
(67.91)
(71.49)
(65.78)
CelehA
SemantieGAN
mloU w./ avg. semantics
56.76
62.75
62.13
98.60
99.12
Li ct al.
2021
(58.34)
(64.225)
(61.87)
(FPN) (Lin et al. 2017 with an ImageNet pre-trained ResNet- 18 (He et al. 2016) as the backbone to ex- tract 128 x 64 x 64 downsampled pixel-level features. The FPN is fine-tuned with 30,000 CelebAHQ (Lee ct al. 2020) face part segmentation masks (about 15% CelebA). We perform mini-batch K-means clus- tering every 20 batches before a single centroid update. The number of clusters is set to 20. Part semantics masks were restricted to have categories, i.e. every object is limited being described by 7 object parts (including background) from selecting the top 7 clusters. For image syntax lcarning, 1-layer bi-LSTM model is trained. Input and hidden vectors are 135-dimensional (128-d encoded mask concalenated with 7-d semantics), and the projected outputs are 7-d. Figure 7 shows the natural and unnatural images during the test, along with the patches used in the traversal sequence and their corresponding predicted part semantics.
SSDI Detection Performance on CelebA Figure shows the forward, backward, and the sum of both residual errors of the bi-LSTM across iterations of processing image patches of both natural and corrupted images of the CclebA datasct. The solid lines mcan values, whereas the shaded region covers between the 25% and 75% percentiles. It can be seen that there is separation between natural (blue) and corrupted SSDI (red) samples. The histogram shows the distribution of test samples based on the total residual error, along with the threshold (green dashed line) used to distinguish samples from natural and corrupted.
Table illustrates the performance of the proposed framework on the CelebA dataset for various grammar validation methods. Different columns represent corruption types, with the number and the size of the affected image patches. For example, Shuffle 2 30 x 30 means that 2 image patches of size 30 30 pixels are swapped to generate the patch shuffling corruption, whereas Puzzles 4 20 x 20 means that the frame- work processes batch generated from all possible permutations (including that of the original image) of 4 randomly selected patches of size 20 x 20 pixels. Both the detection accuracy and the detection rate results are reported, with the latter presented parentheses.
When ResNet-18 and FPN are used as the models for extraction of part semantics, it can be seen that using the memorized average part semantics performs the best among the three grammar validation methods. This can be explained by the fact that in the CelebA dataset, the object is the human face with a very well-defined structure. Hence, it is more effective to learn not only the expected relation between the neighboring object parts (i.c., the image syntax) but also what those object parts are expected to be (i.c., the memorized average part semantics). The puzzle variation of SSDI corruption poses an interesting problem, since a large number of possible rearrangements which they can be distinguished based on the residual error. Interestingly, the framework achieves above 99% performance on puzzles, highlighting that the proposed approach indeed learned the correct composition of object parts.
object parts observed at the same time significantly reduces the margin at
15

---

Published in Transactions on Machine Learning Research (04/2025)
Correct RGB depth
Image
Patch masks
Corrupted RGB depth
Image mask
Patch masks
Figure 9: Image patches of size 160 and the sequences of processing image synlax of SUN-RGBD.
Pred error (forward)
Pred error (backward)
1.8
correct
0.--
0.-
Pred error (both)
144
cornupted
Count
0.0
0.0
10.
08
0.4
0.4
02-
correct
corrupted
comupted
024
OONNaCT com-plad
12
10 11
Patch iteration
Patch iteration
Patch iteration
Total residual
Figure 10: From left to right: the forward, the backward, and the total bi-LSTM prediction errors as well as the histogram of total errors over test samples SUN-RGBD datasot.
The second dataset is the SUN-RGBD (Song et al. 2015) dataset with 10,335 room layout RGB and depth images. The DNN feature extractor consists of a pre-trained Residual Encoder-Decodes (RedNet) (Jiang al. 2018), which used retrieve part semantics masks from 640 480 sized images. The object parts were restricted to 13 based on the obtained semantic masks. For syntax learning, use the same bi-LSTM model as in CelebA. As shown in Figure g we define zig-zag traversal rules on sequences of patches of sizes 160 160 and 80 x 80 pixels. The residual prediction errors and the histogram for test set images are shown in Figure 10
SUN-RGBD
Table shows detection accuracy and detection rates on the SUN-RGBD dataset. Unlike the performance on the object-centric CelebA dataset, is interesting to notice that the first grammar validation method is the most optimal for other corruptions on the scenc-centric SUN-RGBD datasct. This can be explained by the more diverse nature of images in SUN-RGBD: different objects can be arranged in multiple different ways with respect to the entire scene and other objects. Hence, as scenes might change substantially, the method of using the stored average part semantics might be less useful than relying solely the relations between object parts local to individual images.
Effect of Segmentation Granularity On both datasets, we also consider more specialized alternatives for the segmentation masks, which are expected to produce more accurate and finer part semantics. The methods of SemanticGAN (Li et al., 2027) and Dformer (Yin al. 2023) were considered as part semantics models for the CelebA and SUN-RGBI datasets, respectively. From test results presented in Table 5 and Table 6 it can be seen that finer semantic masks produced by more specialized segmentation methods do not al aaws lead to higher SSDI detection rates, especially an object-centric dataset like CelebA. This can probably be explained by the fact that it is casier to capture the natural arrangement of object parts if their part semantics coarser. The relationship between semantic granularity and corruption detection deserves further investigation.
6 Limitations
While this work proposes a pioneering approach to tackle SSDI corruptions, there are a few limitations. First, the semi-supervised approach requires some fraction of semantic segmentation masks obtain semantic
separation of parts within the object, rather than objects from the background. possible solution is consider fully self-supervise segmentation methods, like DINO (Caron et al. 2021 Second, the current
16

---

Published in Transactions on Machine Learning Research (04/2025)
Table 6:
The performance of the proposed approach on detecting SSDI corruptions on SUN-RGBD.
Datasct
Part
Grammar
Shuffle
Shuffle
Black
Puzzles
Puzzles
Semantics
Validation
4
16
Model
Method
160x160
80x80
160x160
160x160
80x80
SUN-RGBD
ResNet-50++ Ennnoder-
Bi-LSTM + next se-
60.57
76.57
66.55
72.89
91.17
(13-cls)
Decoder (ours)
mantics
(72.04)
(73.37)
(67.21)
SUN-RGBD
ResNet-50+ Encoder-
RosNet-50-Eucoder
Bi-LSTM + avg. se-
54.97
61.52
58.46
62.47
75.48
(13-cls))
Decoder (ours)
mantics
(65.36)
(64.09)
(59.65)
SUN-RGBD
ResNet-50+ Encoder-
mloU w./ avg. seman-
57.98
58.98
56.32
97.09
98.20
(13-cls.)
Decoder (ours)
ties
(68.04)
(62.24)
(57.19)
SUN-RGBD
Dformer-S
Bi-LSTM + next se-
62.16
74.23
68.86
76.61
92.84
(13-cla)
Yin sl.
2023
mantics
(73.47)
(72.80)
(71.23)
SUN-RGHD
Dforme-88
Bi-LSTM + avg. se-
53.77
62.45
61.98
61.97
74.78
(13-cls.)
Yin et al.
2023
mantics
(67.23)
(69.09)
(62.45)
SUN-RGBD
Diormer-S
mloU w./ avg. tics
53.24
60.46
59.79
98.13
98.62
(13-cls))
Yin al.
2023
(63.38)
(66.21)
(61.80)
image syntax learning relies on the extraction and traversal of patches with a fixed pattern, which might limit its applicability to a variety of datasets. This shortcoming can overcome by incorporating modules that learn "where to look" along with the proposed syntax learning approach, which do not have to rely fixed number and order of patches. Active vision me(hods, such as Saccader (Elsayed et al. 2019). GFNet (Wang et al. 2020), or FALcon (Ibrayev et al., 2023), provide appropriate frameworks that enable computer vision pipelines with the capability to process inputs through series of glimpses. Since such active vision approaches rely on learning inherent clues between the sequence of glimpses, they might find interesting synergy with syntax learning, which naturally assumes the presence of such clues.
7 Conclusion
Motivated to bridge the gap between human and machine perception related to unnatural images, we in- troduce a novel deep learning framework for addressing semantic-syutactic discrepancy in images (SSDI). Aligned with the concept. of language grammar, the framework learns both the meaning and the natural ar- rangement of object parts using deep clustering and a bi-directional LSTM. The effectiveness of the approach is shown through its capability of solving puzzles and achieving detection rates of 70 -0090% on CelebA and 60 80% SUN-RGBD. The pioneering aspect of the problem suggests broader impact in areas requiring reliable analysis of image content.
Broader Impact Statement
Machine learning systems and, particularly, computer vision techniques are having increased influence everyday tasks that benefit from automation. These tasks range from those with minimal impact on human life, such as facial recognition systems on mobile phones, to highly significant ones, such as obstacle detection in autonomous vehicles. A huge amount of trust is being placed the learning frameworks that are used for performing these tasks. These frameworks are often trained on large datasets curated by retrieving images using machine learning recognition techniques. If the images gathered are unnatural (do not follow the syntax of real-world images), the learning of the downstream learning frameworks is also tainted. Hence, it is important for these learning frameworks to (a) be trained on data that accurately represents real- world sccnarios and (b) learn the syntax and semantics of the data accurately. This work addresses both these concerns by highlighting (a) the SSDI vulnerability in datasets and the importance of learning "image grammar" consisting "image syntax" and "image semantics" as well as (b) proposing a two-stage weakly supervised framework that attempts minimize the possibility of the malicious use of these corruptions by learning the mentioned "image grammar". This paves the pathway for future researchers to implement computer vision techniques that operate holistically learn both the syntax and semantics.
17

---

Published in Transactions on Machine Learning Research (04/2025)
However, in many examples in the history of digital systems, the exposure of vulnerabilities may lead to path for malicious entities to exploit those vulnerabilities in existing systems. For example, when the vulnerability of computer vision techniques to additive imperceptible noise was exposed, many adversarial techniques were developed to trick cxisting machine learning frameworks. Consequently, there is a potential for developing altacks or malicious techniques that exploit the SSDI vulnerability purposes. Moreover, the situation worsened by the fact that such vulnerabilities are currently not de- tectable purely by machine learning, but would require a human expert in the loop for security. This further underscores the need for systems to wholly learn the "image syntax" along with the "image grammar".
images for nelarious
Acknowledgments
This work was supported part by, the Center for the Co-Design of Cognitive Systems (CoCoSys), a DARPA-sponsored JUMP 2.0 center, the Semiconductor Research Corporation (SRC), and the National Science Foundation.
References
Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. In NeurIPS, 2020.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp. 39-57. Icee, 2017.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference Compuler Vision (ECCV), September 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021.
Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan. Picie: Unsupervised semantic seg- mentation using invariance and equivariance in clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16794-168044 June 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, Icce, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Gamaleldin Flsayed, Simon Kornblith, and Quoc V Le. Saccader: Improving accuracy of hard attention models for vision. Advances in Neural Information Processing Systems, 32, 2019.
Alberto Carcia-Garcia, Sergio Osto-Escolano, Sergiu Oprea, Victor Villena-Marting and Jose Garcia- Rodriguez. A review on deep learning techniques applied to semantic segmentation. arXiv preprinl arXiv:1704.0688 2017.
Martin Gubri, Maxime Cordy, Mike Papadakis, Yves Le Traon, and Konshik Sen. Lgv: Boosting adversarial example transferability from large geometric vicinity. In ECCV, 2022.
Thomas C. Gunter, Laurie A. Stowe, and Gusbertus Mulder. When syntax meets semantics. Psy-
chophysiology, 34(6):660-676, 1997. doi: https://do.orr//10111///
111.:////11111000011191199777179 198...997...0.214442 http://///////////../iiii.. 1469- 1997 19w...1..99.......... tb02142.x
URL
Ankur Handa, Viorica Päträucean, Simon Stent, and Roberto Cipolla. Scenenet: an annotated model
generator for indoor scene understanding. In ICRA, 2016.
18

---

Published in Transactions on Machine Learning Research (04/2025)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, June 2016. doi: N0.1109/CVPR.20100
URL http:t::///eeeepporreeeeeee ieee. /ttp./e////dccceee//////00055550
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on compuler vision, pp. 2961 2969, 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked auloencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 16000-16009, 2022.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2018
Yuansheng Hua, Lichao Mou, and Xiao Xiang Zhu. Lahnet: A convolutional neural network fusing low- and high-level features for aerial scene classification. In IGARSS 2018 2018 IEEE International Geoscience and Remote Sensing Symposium, pp. 4731, 2018. doi: TTA.///.........STTT....
Timur Ibrayev, Manish Nagaraj, Amitangshu Mukherjee, and Kaushik Roy. Exploring fovcation and saccade for improved weakly-supervised localization In NeuRIPS 2023 Workshop on Gaze Meels ML, 2023. URL https: https:://opeereviee wttp://wwwwweeeeeffrreeeeeee http/ffrruum11dqffffs1133..
Jindong Jiang, Lunan Zheng, Fei Luo, and Zhijun Zhang. Rednet: Residual encoder-decoder network for indoor rgb-d semantic segmentation. arXiv preprint XXiv:1806.01054 2018
Alexander Kirillov, Ross B. Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, pp. 6399 6408. Computer Vision Foundation / IEEE, 2019. URL http://dblp uni-trier.deee de/ ht//conf/cvpp/ccpr200// html#KirllovGHHD
Alex
Imagenet classifica- 4824 4824- wwwp:mmgeee--lllsssaattit.o wtt.aatooo...ttddddeeoooooooaaaa www.:attttoonnaeeaaaeeeeettttooo
Krizhevsky, Lyy Sutskever, and Geoffrey E Hinton. tion with deep convolutional neural networks.
In Advances in neural informa-
tion
processing systems, pp. 1097-1105, 2012.
URL http::/paperrs.....///////// witrssnips..o wwwpsnic...cccppprr...eepepee cc/paper/
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Tsung-Yi Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. In CVPR, pp. 936 944. IEEE Computer Society, 2017. ISBN 978- 1-5386-0457-1. URL http:://bbly wttpllp.uiirrrtt uni uni-trier de/ cotp//on/eeee cwtpf//vvp/c/pp21111111 htm1#LinDCHHB11
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. 2021 IEEE/CVF International Con- ference on Computer Vision (ICCV), pp. 9992-10002, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint ar Xiv: 176. 060606 2017.
Yichuan Mo, Dongxian Wu, Yifei Wang, Yiwen Guo, and Yisen Wang. When adversarial training meets vision transformers: Recipes from training to architecture. In Advances in Neural Information Processing Systems, 2022.
19

---

Published in Transactions on Machine Learning Research (04/2025)
Diego Ortego, Eric Arazo, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Towards robust learning with different label noise distributions. In 2020 25th International Conference on Pollern Recognation (ICPR), pp. 7020 7027, 2021. doi: 0tA.0///CCR888002220.9922222
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http::/paperr: http://pppprrs...uiisessse neurips. cctppppreccppppr//// 9015 bigh
lecepp--ernninn--- wibrargg-bbbr.r..a pdf
Yao Qin, Chiyuan Zhang, Ting Chen, Balaji Lakshminarayanan, Alex Beutel, and Xuezhi Wang. Under- standing and improving robustness of vision transformers through patch- based negative augmentation. Advances in Neural Information Processing Systems, 35:16276 16289, 2022.
Muhammad A Shah, Aqsa Kashaf, and Bhiksha Raj. Training on foveated images improves robustness to adversarial attacks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. Yoshua Bengio and Yann LeCun (eds.), ICLR, 2015.
ic1r2015. 1smoyy110aa
Shuran Song, Samuel P. Lichtenberg. and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding bench- mark suite. In CVPR, pp. 567 576. IEEE Computer Sociely, 2015. ISBN 978-1-4673-6966--. URL http://dblp.. mnirrrier... ww/dd////ddd/eeeeeee colb//con//cc/ http://oo///pp//pp/////ee httppr2015...111m5ogg...
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312. 6199, 2013.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Baptiste Alayrac, Jiahui Yu, Radu Sori- cut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. Xiv preprint ar 21v: 1...11111111 11805, 2023.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Putlern Recognition, pp. 5238 5248, 2022.
Arnim von Stechow. Syntax and semantics: an overview. Semantics--nlerffcee,, pp. 169, 2019.
Xiaosen Wang and Kun He. Enhancing the transferability of adversarial attacks through variance tuning. In CVPR, 2021.
Yulin Wang. Kangchen Lv, Rui Huang, Shiji Song, Le Yang, and Gao Huang. Glance and focus: dynamic approach to reducing spatial redundancy in image classification. In Advances in Neural Information Processing Systems, 2020.
Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision, 2020.
Bowen Yin, Xuying Zhang, Zhongyu Li, Li Liu, Ming-Ming Cheng, and Qibin Hou. Dformer: Rethinking rgbd representation learning for semantic segmentation. ar preprint arXiv:2309.0966 2023.
Mert Yuksekgomul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations, 2023. https:://peenrrrienn http://pwwooooooeeeeeo//oooo/aaa
Jianping Zhang, Yizhan Huang, Weibin Wu, and Michael R Lyu. Transferable adversarial attacks on vision transformers with token gradient regularization. In CVPR, 2023.
20

---

Published in Transactions on Machine Learning Research (04/2025)
A Experimental details
The entire framework was implemented using the PyTorch framework (Paskke et al. 2019 The models were trained and evaluated using NVIDIA GeForce GTX 2080 Ti GPU cards.
A.1 Datasets
CelebA (Liu et al. 2015) contains 202,599 face images. We used the aligued-and-cropee version where faces are localized. Images are divided into train set of size 162,770, a validation set of size 19,867, and a test set of size 19,962. First, the FPN network finetuned on 30,000 CelebAHQ (Lee et al. 2020 images selected from CelebA. CelebAHQ images and pixel-wise labels are center-cropped (size=160) and resized 256 256. After fine-tuning, PICIE deep clustering technique (Cho et al., 2021) is trained on train images rosized to 256 256 and validated on the validation set. 64 x 64 patches and are upsampled using bilinear interpolation to 256 x 256. The 7 semantic classes and their names are illustrated on the left part of Figure II During the test, the corruptions are generated around 5 facial part landmark locations (left eye, right eye, nose, left mouth, right mouth) using half of the randomly selected test images.
cropped from segmentation
Label
Class name
Label
Class name
Label
Class name
0
Background
Bed
8
Picture
1
Hair
2
Books
9
Sofa
5
2
Forehead
3
Ceiling
10
Table
3
Eyes
4
Chair
11
TV
4
Nose
5
Floor
12
Walls
2
5
Mouth
6
Furniture
13
Window
6
Neck
7
Objects
Figure 11: Segmentation Classes. Left: CelebA; Right: SUN-RGBD
SUN-RGBD (Song et al. 2015 contains 10335 RGB images and corresponding depth images, split into a train set of size 4785, a validation set of size 1000, and a test set of size 5050. RGB and depth images are resized to 640 x 480 and normalized. The pre-trained RedNet (Jiang et al. 2018) also produces 640 x 480 part semantie masks. From each segmentation, 160 x 160 and 80 The segmentation contains scmantic classos, but we merge them into 13 classes with the same mapping used in the work of Handa et al. (2016). The 13 semantic classes are shown on the right part of Figure 11 During the test, add corruptions to image patches using half of the randomly selected test images.
patch semantic masks are cropped.
A.2 Architectures
On CelebA Figure illustrates the feature pyramid network (FPN) Lin et al. 2017) that was used for part semantics clustering and segmentation of CelebA (Liu al. 2015) images. Specifically, Panoptic FPN proposed in the work of Kirillov al. (2019) was used in the combination with ResNet-18 (He et all, 2016) the backbone feature extractor. While the backbone encodes multi-scale feature representation in top-down fashion, FPN model utilizes
convolutional (Conv) layers in the bottom-up manner to project pixel-wise features to 128-dimensional space. The projected features of different spatial sizes are upsampled using bilinear interpolation to 1/4 height and width of the original image and then element-wiss summed. The pixel-wise representations thus have shape 128 x h w (128 x 64 x 64 for input images of shape 256 256). The extracted pixel-wise features are used for the deep clustering (Cho al.) 20021). ) We record the resulting clustering centroids final Conv layer to project these features to the dimension of the number of semantic classes represent part semantics.
21

---

Published in Transactions on Machine Learning Research (04/2025)
Image: 3x256x256
Conv1
Output: 7x64x64
Maxpool1
Conv
1x1
Layer1
Outputt
Conv 1x1
2x
Layer2 (w./ downsample)
Output2
Conv 1x1
2x
Layer3 (w./ downsample)
Output3
Conv 1x1
Layer4 (w./ downsample)
2x
Conv 1x1
Output4
Figure 12: DNN architectures: Feature Pyramid Network (FPN) with ResNet-18 as the backbone
Figure 13 shows the structure of the bi-directional Long Short- Memory (bi-LSTM). We use it for image syntax learning. For each patch in the traversal sequence of length 5, the predicted mask is first flattened before being fed to fully-connected (FC) encoder layer. The encoder projects the semantic mask to a 128-d vector, which concatenated with the 7-d semantics vector. The resulting 135-d vector contains both spatial and semantic information about each patch's part semantics, and it is used as the input 2 LSTM layers in forward and backward directions. After projection layers, the outputs include 7-d forward prediction of the next part semantics semantics in the sequence and 7-d backward prediction of the previous part semantics. For the first and last patches, only forward and backward predictions are made, respectively.
patch mask:
th patch semantics: 1x7
1x64x64
flatten
mask_ proj (4096, 128)
concatenation
forward
forward
LSTM
pred bockward
out proj (135, 7)
out proj (135. 7)
pred fowardd
H backward
LSTM
backwaadd [+]
Figure 13: DNN architectures: bi-directional Long Short-Term Memory (bi-LSTM)
22

---

Published in Transactions on Machine Learning Research (04/2025)
For segmentation of SUN-RGBD (Song et al 2015 images, we use Residual Encoder- Decoder Network (RedNet) (Jiang et al. 2018) with ResNet-50 as the backbone. It is a state-of-the-art room scene parsing architecture that achieved 47.8 8% mloU accuracy on 37-class SUN-RGBD. Similar to FPN, the RedNet extracts multi-scale feature maps from the input image with its encoder residual layers. However, each projected feature map is skip-connected with the output of upsampling residual units in a decoder (Jiang ctt al., 2018) before being summed with the next level in the feature pyramid. An additional depth branch is fused with the RGB branch. We performed 13-class (merged from 37-class) semantic segmentation with trained RedNet. The RedNet produces the same-resolution (640 480) segmentation from the original image. Patch semantic masks of sizes 160 x 160 and 80 x 80 are cropped and used for preparing patch semantic vectors. At each stage of patch sequences, part semantics masks are flattened and projected to 128-d vectors before concatenating with a 13-d part semantics vector. The resulting 141-d vectors are passed to LSTM blocks for forward and backward predictions (both 13-d).
On SUN-RGBD
A.3 Hyperparameters
2009 Cele- On CelebA During fine-tuning of the FPN network pre-trained on ImageNet (Deng et al.
bAHQ [Lee et al. 2020) masks are pre-processed to contain only 7 coarse semantic classes. Adam opti- mizer is used with start Ir = le-t, We trained for num epochs = 20 with a constant learning rale and batch _sice_sse_____ee siie__train_____eeeeeaiiiiaaaeea 128. The fine-tuned model is used for deep clustering PICIE (Cho et al. (021)) training with batch size_train train = 256. We adopted over-clustering and sct the number of clusters, K train 20, as that results in better clusters compared to KM_nnum num = 10 and KM_ num 30. Mini-batch K-means clustering is performed. K KM- iinit= 20 is the number of batches we collect before the first K-means clus- tering. KM. num = 20 is the interval of batches between consecutive clustering. KM iter = 100 is the number of K-means clustering iterations before convergence. After each clustering, the total PICIE loss is back-propegated to optimize FPN model parameters. We use Adom optimizer with start Ir = 1e-1. De- spite the complexity of the total PICIE training loss, the training converges fast in num_epochss = 10 with a constant learning rate. To get semantic segmentation masks, we initiate a x 1 Conv layer, the weights of which are loaded as the trained centroid matrix, and append it to the trained FPN network. We post-process pixel-wise class assignments to merge semantically-sinii Further, we tune the FPN network on 64 x 64 patches, using crops on the saved face part semantics as the supervision. The part semantics mask detector is trained with batch. size Irain = 128, slart ly = le-t for a total of num tou__epochs = 20, We chose the num _erpessepss____ = 20 model over the num _ere_s_eesss___ = 40 model because the smooth boundarics in the semantic masks help with image syntax learning.
clusters and obtain 7-class face part semantics.
After we obtain 7-class part semantics masks for each patch in the traversal sequence, feed the concatena- tion of encoded mask (128-d) and semantics vector (7-d) into a bi-LSTM model. The parameters for bi-LSTM are: input_ input_sizee__ = 135, hidden size = 135, num num_loyers = 1, bidirectional = True, proj_size = 7. We use Adam optimizer with slart Ir = 1 and num epochs = 40, with the learning rale changed to 1e-5 after 20 epochs.
On SUN-RGBD On SUN-RGBD, the pre-trained RedNet network (Jiang et al., 2018 produces 37 seg- mentation classes. We merge these classes into 13-class as shown in Figure 11 When the patches are shuffled, the pre-trained network is able to produce precise part semantics, so that a part mask detector is not needed as in CelebA. We start with image syntax training with two configurations, patch_siaa size 160 and patch_siss size- = 80. In both configurations, the part semantics masks are encoded as 128-d vectors, while the part semantics vector is 13-d. Thus, the bi-LSTM model has the configuration with input_size size = 141, hidden size- = 141, num layers = 1, bidirectional = True, proj size = 13. We use Adam optimizer with start_ Ir = le for a total of num epochs = 40. A multi-step learning rate scheduler is used with gamma 0.8 and milestones = [5, 10, 15, 20, 25, 35).
23

---

Published in Transactions on Machine Learning Research (04/2025)
B Generation of SSDI corruptions
One of the main contributions of this work is exposing the vulnerability of classification models to semantic- syntactic discrepancy in images (SSDI). We define three types of corruptions cansing semantic-syntace discrepancy in images and provide the algorithmic methodology of generating these SSDI corruptions. Fur- thermore, the supplementary material contains the code to generate the corresponding SSDI corruptions.
Algorithu illustrates the algorithm used to generate the first type of SSDI corruption called a patch shuffling. This type of corruption is generated by swapping the positions of the subset of image patches. The performance of the SSDI detection approach is evaluated based its capability to detect whether the given imput is natural image or not. Algorithm 2 illustrates the algorithın used lo generale the second type of SSDI corruption called patch blackeming. This type of corruption is generated by blackening out a subset of image patches. Similarly to the patch shuffling, the performance is evaluated based on the detection capability of natural and corrupt images. Algorithm 3 illustrates the algorithm used to generate the third type of SSDI corruption called a puzzle solving. Similarly the patch shuffling, this type of corruption permutes a subset of image patches, However, unlike patch shuffling, in puzzle solving, we generate all possible permutations for the specified number of image patches allowed for corruption. Then, all possible patch permutations and the original natural image are fed to the framework, which has to predict the image with the natural arrangement of image patches/object parts. For each considered dataset, SSDI corruptions are generated using half of the test set images chosen at random. All corruptions are considered separately, i.e the paper does not consider a simultancous mixture of different SSDI corruptions. Figure [4] illustrates examples of all SSDI corruptions considered in this work on CelebA and SUN-RGBD test images.
correct
shuffle patches
blackout patches
Puzzle solving: real, fakes
CelebA
Image size (256, 256)
Patch (30,
correct
shuffle 12 patches
blackout patches
Puzzle solving: 1 real, fakes
SUN-RGBD
Image size = (640, 480)
Patch size (160, 160)
correct
shuffie 48 patches
blackout 24 patches
Puzzle solving: fakes
SUN-RGBD
Image size (640, 480)
Patch size = (80, 80)
Figure 14: Demo types of SSDI corruptions.
24

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm Patch shuffling
def ShuffcsSSD[ num patch ps):
result +
for it, X in enumerate((ensors) do
patches < mn.F. Funffolldd, ps, ps, 0)
5: p patchessshapef-1]
6:
if ===-0 then
7:
indices < sample(range((), num namppatch)
orig + tensor(range(p))
perm tensor(range(p)
10: for j in range(num patchlaahhhhhhhh do
11:
perm[indices|ji] origfindices((] Llyooom__paatetteee__ 12: end for
13: end if
14: patches t concat(patch]: perm]for patch in patches)
15:
X wm..F...lllll.. poattfo,, X.shape[-2:].pe ps, 0)
16:
(esult.append
17: end for
18: return result
Algorithm 2 Patch blackout
1: def (lackoulSSDDeennoors, num patch, ps):
2: result + []
for it. X in cnumeratc(tensors) do patches un.FFunfold(X, X, ps, ps, 0)
p t- pmtches.shapef11]
if it= 0 then
7: indices + sample(range(p), num_patch)
8: end if
for idr, X in enumerate(patches) do
10: patches[idr[[;- indices) +0
11: end for
12: X nem....ool(paaccees X.shapel-2.l.ps, ps,
13: (x)ul))append(X)
14: end for
15: return result
25

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm 3 Create puzzles
def (ensoos, wumpperri ps):
result
for it, X in enumerate((ensors)) do patches < mn.F. FunFfolflld ps, ps, 0)
5: p patches.shapef-1
6: if ===00 then
7: perms +0
for perm id in range(reum perm) do
permsappend(randpenm. end for end if
10:
11:
12:
reS 4- X.clone()
13:
for perm in perms do
14:
new prt_patcess + concat (patchf; perm]for patch in patches)
15:
new x nn. ww...fooofff...... fol..ofcccsscees X.shape|-2:]p ps, 0)
16:
res f concat(res,, new_))
17: end for
18:
result.append(
19: end for
20: return result
26

---

Published in Transactions on Machine Learning Research (04/2025)
C
Performance results of classification models against patch corruptions
and 18 Illustrate classification accuracy results achieved with different classification models against patch shulling and patch blackening corruptions generated from ImageNe(2012 val set samples of original size 224 x 224 pixels.
Figures 15
16
Performance
ImogeNeta012 "patch shuffting correptions
Clen accurocy: 61 07% originglimoge siee 224x224)
Patch Sice
112<112
50:56
patches
32x52
patches
28k28
136 patches
16x16
patches
14x14
784 patches of Bul
1024 patches
7x7
4xd
2x2
2
77:88 0.06
79.54 0.008
00.50:0..4 10.0
80.49 0.00
80.87:0.033
10.00
80:9410.00 0.05
$0.80 10.09
80.9410022
80.99
80.54...4
80.65::000
15
70.39 0.1.4
75.88 10.08
75.52. 0.08
79.75:00.6
78.85
80.19
78.84
80.366:0006
80.63
10.11
78.04:0.31
7..210.00
79.32
78,70 10.08
79.73..000
80.40 10.0010..2
02.27-0005
55.20 0.16
75.89
72.2
77.42 10.07
70.31+0.00
86.11 +0.02
Comupted
0.07
73.00 0..1
67.72.0..1
77.31 00000
70.71:..00
78.81 1.000
52.59
35.77.0.:17
76.64.0.07
196
36.98::0.06
8.16
64.2810.13
60.94
74.64:..13
17.99:0..6
512
4.86100.00
8.1110..05
6L.7110.13
73.4410.05
764
0.09::0.02
0.59
45.19:0.11
69.27
28.3810.155
2048
0.65:0.22
43.99:20.14
3136
0.24
11.19
8150
0.92 0.002
12544
0.29 0.29
Figure 15: Performance results of ViT-B-16 model against patch shuffling of the different degrees. Each cell represents the mean and standard deviation of classification accuracy for 5 runs with different random seeding.
Performance of VIT model on Imopelet2012 with "potch NocRoning SSDI compptions
Gleon accurocy: 21.0%% originatimoge siter 224x2244
patches of
112x112
16 patches of
5b:5
49 patches of
32xDD
patches
28x38
196 patches
16x16
256 patthes of
14x14
784 patches of Bxl
1004 patches of
7x7
3136 patches
12544 patches
2x2
77,49 0..06
10.05
80.78:00.3
$0.89.1000 0.05
80.35:10.03
80.99.0.08
90.98.0.02
80.95 10.02
80.561.0:3
81.0010.01
08.1610.09
10.00
80.6210006
80.8410.02
80.8210.07
80.5710.02
80.86
80.9010003
80.9910.08
0.000
00.10
79.75 10.08
30.26 0.77
80.05::0.04
80.0710.06
80.88:0.03
30.77:10.6
80.35
86.9616.02
16
0.30+
25.19
76.50:.0.0
29.75
79.4
80.39:0.05
80.01 t0.06
80.41:0.02
80.61
e
78.60
71.11
79.54:0000
49
0..0.0..00
43.36 0.12
77.27
74.09:0.11
78.89
78.20 10.10
79.512.006
80.24
Cornupted
178
61.75 0.12
46.11 0.20
74.50
70.14+0.09
70.00
75.30
255
0.10
+..11
61.68 +0.07
70.24+0.10
77.90-0.01
22,00
80.39:0.10
75.12
1091
0.10+0.00
46.28*00.1
70.22
2048
4.89
63.78
52.99
4056
45.29
9.88 0.0.
Figure 16: Performance results of ViT-B-16 model against patch blackening the different degrees. Each cell represents the mean and standard deviation classification accuracy for 5 runs with different random soeding.
27

---

Published in Transactions on Machine Learning Research (04/2025)
Performanee
cormpptio
(lean accorocy: 80.89, original imngee aite 226:236)
Potch Size
patches
patches
784 patches Bd
1004 patches
132x312
56:56
33x/2
28c38
16x16
14x14
7x7
4e4
2x2
77.76 = 0.04
79.275
85.21 50.05
80.38 0.02
00.67:0.04
80.71:1..22
80.75:0.04
50.75 20.05
00.51::0.04
80.03
00.04
77.571..008
79.50:0.00
78.8910077
80.53:0.07
80.5610.05
80.7010.03
50.08:0.09
80.74:.002
86.77
16
71.3810.00
70.45 20.05
76.11:0000
79.3910.88
79.72.0.09
06:0510.05
$0.1810.07
80.29:0.03
80.42.0.06
32
64.00
07.32:0.33
78.06:0.06
79.13:
79.20
79.55:0006
75.81 t:0.8
1.,06
77.9620.07
Number of Comupted
64
$8.73
09.97:0..9
7..99:0.11
76.70
76.79:0.10
77.64:..09
7..33
196
19.23
19.16
57.7220.16
39.92
67:9720155
7..3
512
2.34+
6.00+0.06
28.19+0.13
784
0.671...2
0.62:0.02
18.75
48.04
2048
0.4110.01
26.55
3136
0.271081
0.009
8152
0.37
12546
0.22:0001
Figure 17: Performance results of ResNet-50 model against patch shuffling of the different degrees. Each cell represents the mean and standard deviation of classification accuracy for 5 runs with different random sceding.
Performace Rester50 modil on Imogeter2202 with "patch blackming SSell comuptions Glian accurocy: 80.8%% originalimage siber 224x220)
4 gatches of
16 16patches.d of 59:36
49 patches of
patches of
136 patches
patchhs of
1034 patches of
5136 patches
12564 patches
112x112
32x32
28x28
16x16
14x14
7B4 patches of BxB
7x7
77.57=0.09
80 0.00
80.54:0.05
80.71:...3
80.77
80.7910.11
80.8050.02
80.81 +0.00
80.52:00:1
80.81 0.0.1
10.13
0.16410.10
80.52:0.04
80.7310.04
80.7500.44
80.7010.02
50.7710.04
80.501002
80.81 1000
16
0.10=0.00
77.50.0.13
78.47:0.11
80.13+0.05
80.28+0.05
80.38::.05
$0.05:40.4
80.32:0044
80.36
03.8610.16
79.3210.30
79.9910.05
79.8810.04
79.2910.05
79.761004
78.11
Number of Comupped
64
0,10
76.87
77.55.0.12
78.5450.15
40.06
78.19:0006
08.00
03.77
7188 10.05
75.17
70.0910.08
75.09:0.10
76.23 0.100
256
0.30
63.8740.11
+0.07
69.34:0006
72.75
0.15
28.56:0.15
54.30:012
66.56 0.13
3024
0.10:00.00
30.5610.16
$7.61
2038
0.22
53 13.10:0.12
4086
38.45.0.14 0.14
8102
5.45
Figure 18: Performance results of ResNet-50 model against patch blackening of the different degrees. Each cell represents the mean and standard deviation classification accuracy for 5 runs with different random seeding.
28

---

Published in Transactions on Machine Learning Research (04/2025)
D Extra quantitative results
In Table and Table 10 we add more results for the detection task of SSDI corruptions, on CclebA (7 classes)
and SUN-RGBD (13 classes), respectively.
CclebA The segmentation model is ResNet-18 with FPN. The input images have sizes 256 x 256. All results are reported using the first grammar validation method, which relies on the combination of using bi-LSTM and comparing the predicted part semantics to the estimated part semantics within the image itself (i.e. without the memorized average part semantics). From Table 7 we observe that the proposed approach exhibits reliable performance when the size of the corrupted patch exceeds 30 30: >80% on patch shuffing SSDI, 92.51% on puzzle solving SSDI with 4 permutations, ~80% on patch blackout SSDI.
The segmentation model is ResNet-50 with encoder-decoder (RedNet). The input images have sizes 640 x 480. All results are reported using the first grammar validation method, which relies on the combination using bi-LSTM and comparing the predicted part semantics to the estimated part semantics within the image itself (i.e. without the memorized average part semantics). Two segmentation granularities are used, 13-class coarse segmentation merged from 37-classes, and the original 37-class fine segmentation. From Table 10 we observe SSDI detection performance for the two levels of granularity. For patch shuffling SSDI and puzzle solving SSDI, using coarser 13-class semantic masks boosts syntax sequence reasoning and improves grammar validation performance. This is due to the better sequential reasoning capability that the bi-LSTM learns from coarse patch masks. Meanwhile, in the task of patch blackout, using finer 37-class semantic masks has an odge. The blackened patch semantics casier to discern when the segmentation is finer, leading lo a longer synlax sequence and higher variations in the predicted semantic values.
SUN-RGBD
Table 7: SSDI detection performance on CelebA with 256x256 sized images.
SSDI Corruption: patch shuffling
Test Accuracy (%)
Patch size: 10
Patch size: 20
Patch size: 30
Patch size: 40
Patch size:50
shuffle 2 patches
53.86 (75.58)
65.66 (68.67)
76.22
(79.72)
83.89 (86.33)
88.22 (93.04)
shuffle 3 patches
55.67 (75.16)
73.16 (78.97)
85.68 (91.08)
90.96 (95.22)
93.05 (97.42)
shuffle 4 patches
57.33 (74.73)
78.40 (82.29)
89.79 79.79 (94.57)
93.00 (97.34)
94.34 (98.07)
shuffle 5 patches
59.13 (70.93)
82.88 (88.15)
91.72 (96.06)
93.93 (97.25)
94.69 (97.39)
SSDI Corruption: puzzle solving
Test Accuracy (%)
Patch size: 10
Patch size: 20
Patch size: 30
Patch size: 40
Patch size:50
real, 3 fake
57.84
86.74
92.51 66.15
93.97
94.81
real, 119 fake (all perm)
8.39
37.43
80.37
87.23
SSDI Corruption: patch blackening
Test Accuracy (%)
Patch size: 10
Patch size: 20
Patch size: 30
Patch size: 40
Patch size:50
blacken patch
54.73 (55.36)
70.91
(67.04)
81.25
(81.08)
87.37
(88.04)
91.25 (95.12)
blacken 2 patches
60.43 (63.78)
83.83 (85.22)
91.77 (93.43)
95.41 (97.41)
95.78 (97.58)
blacken patches
65.79 (68.94)
90.60 (93.83)
96.48 (98.49)
97.83 (98.57)
98.50 (99.14)
blacken patches
71.85 (71.34)
94.56 (97.14)
98.08 (98.88)
98.86 (99.36)
99.05 (99.49)
blacken 5 patches
77.32 (85.28)
97.22 (98.20)
98.87 (99.48)
98.95 (99.29)
99.13 (99.64)
29

---

Published in Transactions on Machine Learning Research (04/2025)
Table 8: 13 segmentation classes
SSDI Corruption: patch shuffling
Test Accuracy (%)
Patch size: 160
Test Accuracy (%)
Patch size: 80
shuffle 4 patches
60.57 (72.04)
shuffle 16 patches
76.57 (73.37)
shuffle 8 patches
69.31 (72.59)
shuffle 32 patches
86.63 (82.38)
shuffle 12 patches
73.47 (77.50)
shuffle 48 patches
87.09 (82.97)
SSDI Corruption: puzzle solving
Test Accuracy (%)
Patch size: 160
Patch Size: 80
1 real, 3 fake
72.89
91.17
1 real, 99 fake
28.95
69.67
SSDI Corruption: patch blackening
Test Accuracy (%)
Patch size: 160
Test Accuracy (%)
Patch size: 80
blacken 4 patches
66.55 (67.21)
blacken 16 patches
67.43 (65.53)
blacken 8 patches
66.59 (67.17)
blacken 32 patches
75.68 (70.22)
blacken 12 patches
66.75 (67.49)
blacken 48 patches
73.66 (74.81)
Table 37 segmentation classes
SSDI Corruption: patch shuffling
Test Accuracy (%)
Patch size: 160
Test Accuracy (%)
Patch size: 80
shuffle 4 patches
62.32 (74.57)
shuffle 16 patches
67.74 (74.77)
shuffic 8 patches
71.53 (76.99)
shufflc 32 patches
78.04 (76.20)
shuffle 12 patches
74.93 (77.43)
shuffle 48 patches
79.84 (78.69)
SSDI Corruption: puzzle solving
Test Accuracy (%)
Patch size: 160
Patch Size: 80
1 real, 3 fake
76.10
81.23
1 real, 99 fake
33.85
55.99
SSDI Corruption: patch blackening
Test Accuracy (%)
Patch size: 160
Test Accuracy (%)
Patch size: 80
blacken 4 patches
61.58 (71.37)
blacken 16 patches
61.41 (71.25)
blacken 8 patches
67.53 (72.20)
blacken 32 patches
69.07 (71.80)
blacken 12 patches
66.18 (76.00)
blacken 48 patches
64.59 (72.24)
Table 10: SSDI detection performance on SUN-RGBD with 640x480 sized images.
30

---

Published in Transactions on Machine Learning Research (04/2025)
E Extra qualitative results
E.1 Semantic clustering and segmentation
Figure 19 and Figure 20 show selected part semantics obtained on CelebA face images and SUN-RGED room scene images, respectively. All samples are from the train set. The displayed part semantics masks are generated by trained ResNet-18 and trained ResNet--50 encoder-decoder, respectively.
Figure 19: 7-class semantic segmentation on CelebA
Figure 20: 13-class semantic sogmentation on SUN-RGBD
E.2 Syntactic sequences of part semantics
Figure 21 shows the pixel-level part semantics masks and the distribution of part semantics in cach mask, averaged over correct samples in the entire CelebA test set. During training, we optimized MSE loss bet ween bi-LSTM predictions and the actual semantics inside each patch, in order to model the mean of the part semantics distribution each patch itcration. Hence, here we show the average traversal patterns learned. It can be seen that face-part semantics transitions and the face syntax in the CelebA dataset are captured by the trained bi-LSTM model. For example, in the first 2 patch iterations, the "forehead" semantics is
31

---

Published in Transactions on Machine Learning Research (04/2025)
dominant. In the third and fourth patches, "mouth" semantics is dominant. While for the last patch, "eyes" semantics is dominant. These part semantics along with transitions of those semantios is key to identifying the presence of SSDI.
Background
Hair
Averaged patch mask transitions
Forehead
Eyes
0.6
Nose
Mouth
0.5
Semantics ratio
0.4
0.3
0.2
0.1
0.0
patch
patch 2
patch
patch
patch
Patch number
Figure 21: Left: Averaged patch mask transitions over CelebA test set; Right: Averaged patch semantics over ColebA test set
Figure 22 shows the pixel-level part semantics masks and the distribution of part semantics in each mask, averaged over correct samples in the entire SUN-RGBD test set. Here, each of the 13 semantic classes is either a stuff (e.g,, ceiling, floor, wall, etc.) or a thing (e-g., books, char, table, etc.). Within the traversal soquence, the first few patches at the top of the image have "wall" as the dominant part semantics, while the last few patches at the bottom have "floor" the dominant part semanties. Semantic classes belonging to the thing category reside in the middle patches, such "chair" and "table". This reveals that our proposed framework is capable of learning the syntax of the scene-centric images containing stuff and thing layouts even from diverse SUN-RGBD dataset with a wide variety of different room scenes.
Averaged patch mask transitions
0.6
Calling
Flcor
Semantics ratio
0.3
0.2
0.1
0.0
2
4
5
7
10
11
12
Patch number
Figure 22: Left: Averaged patch mask transitions over SUN-RGBD test set; Right: Averaged patch semantics over SUN-RGBD test set
32


Reviews:
[{"review_id": "9rmcYYkfX8", "reviewer": "Reviewer_np8H", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. This paper is generally well-written and easy-to-follow.\n1. The proposed framework seems to be extensible, e.g., other backbones.\n---\n### Weaknesses\n1. **SSDI is ill-defined and not well-motivated.** I think SSDI is designed for unnaturalness detection, but \"unnaturalness\" focused in this paper is unclear. This paper also does not describe a specific application where SSDI can be utilized. I believe that the definition and motivation of SSDI are too vague. In addition, I am wondering what is the advantage of the concept of SSDI over existing unnaturalness detection concepts, including out-of-distribution/anomaly detection and adversarial defense.\n1. **The scope of \"unnaturalness\" covered in this work is very limited.** This paper has mainly focused on \"patch shuffling\" for creating unnatural images and evaluating models. I think the approach is not enough to detect various plausible unnatural cases. Additionally, the proposed framework seems to able to detect only patch-shuffled or -masked images, so I am not convinced that the framework is widely applicable.\n1. **Experiments are limited to SSDI detection evaluation on two datasets.** I think the importance of SSDI detection in real-world applications/scenarios should be demonstrated in the experiment section, but this paper only focuses on evaluating SSDI detection performance on few datasets. I am not convinced that the SSDI detection performance is a meaningful metric for detecting unnaturalness in various real-world applications such as anomaly detection, out-of-distribution detection, etc. In addition, the generalization ability (for unseen datasets, unseen corruptions, etc) should be tested.\n1. **Existing pretrained models have not been utilized enough.** In recent years, there are many pre-trained models, especially for dense prediction tasks like object detection or segmentation, for example, Grounding DINO (Liu et al., 2023) and Segment Anything Model (Kirillov et al., 2023). I think, in this problem, it is very crucial to utilize various pretrained models because it is already shown that they can be generalized to various image domains and datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Deep learning models often struggle to recognize unnatural objects that are easily detected by human perception. So, this work highlights the vulnerability of classification models to semantic and syntactic discrepancy in images (SSDI), where object parts are shuffled or removed. To address this, this paper proposes a semi-supervised framework that first (1) extracts patch/pixel-level representations based on deep clustering method, and then (2) summarizes them based on spatial relationships using bi-directional LSTM for detecting unnatural corruptions. In experiments, this paper demonstrates that the proposed framework achieves high SSDI detection performance on CelebA and SUN-RGBD datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The aforementioned weaknesses.\n1. Minor clarification\n    - In Eq (2), what is $z_{ipk}$?\n    - What is the difference between $z^{(1)}$ and $z^{(2)}$?\n    - How to \"features of different images should map to the different clusters\" using Eq (4)? The objective seems to assign $z_{ip}^{(1)} $  to the cluster $y_{ip}^{(2)}$. How does this assign \"features of different images\" to the \"different clusters\"?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "00AELnc3IP", "reviewer": "Reviewer_f9ix", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nCompelling problem identification and exposition of problem (SSDI vulnerability of vision models)\n\nProposal of a semi-supervised solution, which proposes to leverage bi-directional LSTM architecture that enables additionally grasp expected syntax, in order to identify if images are syntactically out of distribution come inference time\n\nEvaluation on two diverse real world datasets\n\nThorough evaluation of classification methods (3 variations of dataset corruptions, 3 types of error estimation, and two evaluation criteria of accuracy and recall). \n\nWeaknesses\n\nThe 5 patch based method for training the bi-directional LSTM does not seem robust. Use of 5 patches in a very  particular region is quite a fragile methodological proposal. This choice may be quite sensitive to the size of relevant image features, and may need to be tuned for object types of various scales and location placement. A more robust proposal, perhaps with e.g. a greater number / coverage of randomly selected patches, might be more compelling and less tailored to the demonstrative dataset.\n\nLack of benchmarking results against the highlighted model at beginning of problem statement (i.e. Gemini)\n\nThe proposed method does not consistently outperform the baselines (SemanticGAN and Dformer)\n\nMinor\n\nThe method for determining threshold value is not explicitly noted (assume it is done using ROC curve, but could not find the sentence in the paper.) If it is not already present somewhere, please include an additional one line note.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work identifies a vulnerability in common image classification models. Namely, such models will mistakenly classify an object as belonging to a particular category, even if the respective object has been broken up and reshuffled in the image to form an unnatural arrangement. The reason such models still confidently classify the reorganized image as belonging to the original category, is that they are trained to make predictions based on constituent features, irrespective of the order in which such features are present (essentially, the models have no understanding of “image syntax” only “image semantics”.) The authors term this vulnerability Semantic Syntactic Discrepancy in Images (SSDI).\n\nTo resolve the SSDI issue in available vision models, the authors leverage an architecture meant to explicitly encourage learning of expected “image syntax” (or expected order of image parts) such that out of order images can be more readily identified. Image grammar is learned in a two stage method;  the first stage is a DNN feature extractor that learns “image semantics” and the second stage is a bi-directional LSTM that learns “image syntax” (the order in which constituent image parts appear heuristically). \n\nThe authors propose to evaluate a model for its ability to handle SSDI appropriately, by creating a dataset of both naturally occurring and unnatural images (e.g. shuffled patches). In order to enable the model to infer whether an image is syntactically in distribution or out of grammatically reasonable distribution, the authors establish 3 methods of quantifying how likely an image is to belong to a natural set of images. The methods include 1) computing the average difference between predicted part semantics from the syntax aware bi-directional LSTM vs. the part semantics predicted by the first stage DNN, 2) computing the average difference between predicted part semantics and average per pixel semantics from the training dataset (like taking the difference between prediction and average expected value) and 3) the same as 2 but using mIoU (mean intersection over union). For a set of corrupted and uncorrupted images, the error values are determined, and an error-value threshold is set with which to classify images as corrupted or uncorrupted during inference.\n\nThe authors create 3 types of corruptions, and apply their method to two datasets CelebA and SUN-RGBD. They evaluate the accuracy and recall values, for identifying corrupted vs uncorrupted images in the test datasets, and benchmark against SemanticGAN. The authors find their method able to recall SSDI for 70-90% of corrupted images on these two datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please benchmark against Gemini for detecting SSDI. This might require a bit of creativity (e.g. determining what prompt to use for detecting SSDI). As long as the method is well described, that should be sufficient. \n\nDiscuss the limitations of the patching method, and how a more robust patching system could be implemented (which would be more robust to scales, types of images, where parts are located, etc.)\n\nDiscuss why the proposed method may not be outperforming the two proposed baselines. \n\nMinor\n\nAdd the line about how the threshold is calculated", "broader_impact_concerns": "No concerns, helpful statement as to the potential impact of not correcting SSDI (which the paper highlights as a currently unresolved issue).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GYXhqfminX", "reviewer": "Reviewer_ftNh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper is easy to read.\n- The finding of SSDI vulnerability is both novel and intriguing.\n\n**Weaknesses**\n- *Evaluation of Existing Classification Models*: The paper lacks a comprehensive analysis of the extent to which current classification models suffer from various types of corruption, particularly those linked to SSDI. Quantifying the performance degradation of existing models across different corruption types—such as patch shuffling, blackening, or puzzle-like transformations—would provide a more apparent context for the significance of SSDI vulnerabilities. Additionally, such analysis could help identify the most critical corruption types to address and establish whether the proposed method offers a tangible improvement over current approaches. Without this evaluation, the impact and necessity of the proposed solution remain unclear.\n- *Persuasiveness of SSDI Vulnerability*: While SSDI vulnerability is an interesting concept, the argument for its significance is not entirely convincing. Classification models often assume that input images are curated and verified. The frequency and relevance of corrupted data, as simulated by experiments involving patch shuffling or puzzle solving, remain unclear. Furthermore, accurate classification is often crucial in some instances despite corruptions such as patch blackening. These types of corruption appear somewhat artificial and may not represent common real-world scenarios. For instance, it is arguably more critical for models to maintain classification accuracy even under common corruptions rather than addressing artificially generated SSDI cases. Thus, the real-world importance of addressing SSDI vulnerabilities is insufficiently demonstrated.\n- *Baseline Comparisons*: Although SSDI vulnerability is not yet well-established, including baseline comparisons would strengthen the validity of the proposed method. For instance, the authors could employ VLM/LLM to detect corrupted images by identifying vulnerabilities or transformations rather than solely classifying the object. Measuring VLM/LLM performance in explaining and detecting transformations, alongside evaluating their failure rates and error patterns, would provide a stronger foundation for comparison.\n- *Scalability*: The proposed method's scalability to more complex datasets raises concerns. As datasets increase in complexity—such as moving from CelebA to SUN-RGBD or ImageNet—the distinction between corrupted and original images diminishes. CelebA, with its relatively simple structure (e.g., centered faces), is less challenging than datasets involving intricate and diverse scenes. This calls into question the feasibility of applying the current approach to more complex datasets without significant adaptations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper highlights a new limitation of existing classification models: they tend to overlook changes in the arrangement of object parts, provided that all parts are present, even in an unnatural random order. To figure out this overlooked behavior of the existing classification model, this paper introduces the concept of Semantic and Syntactic Discrepancies in Images (SSDI), framing it as a vulnerability to corruptions that involve removing or shuffling image patches or presenting images in the form of puzzles. To address this vulnerability, the authors propose a novel method to detect SSDI by analyzing the arrangement of image components using a semi-supervised two-stage approach. The efficacy of the proposed method is validated through evaluations of two benchmark datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe the weaknesses related to the Persuasiveness of SSDI Vulnerability, Evaluation of Existing Classification Models, and Baseline Comparisons are critical for recommending acceptance, as they address the problem's significance and the proposed method's effectiveness. Without addressing these points, it becomes challenging to justify the need to solve this problem and identify effective strategies for tackling it. The other weaknesses, such as scalability, would enhance the work but are not essential for acceptance.", "broader_impact_concerns": "To address ethical considerations, it is recommended that CelebA corruptions be visualized using synthetic or generated images instead of real images from the CelebA dataset. This approach mitigates privacy concerns and ensures the ethical use of data, particularly when dealing with sensitive facial imagery.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper highlights a vulnerability in image classifiers, termed Semantic and Syntactic Discrepancy in Images (SSDI), where models fail to detect unnatural arrangements of object parts, such as patch shuffling or masking. To address this, the authors propose a two-stage framework that captures both semantic content and syntactic structure to detect SSDI corruptions.

All reviewers found the problem formulation interesting and the proposed method clear and well-evaluated on diverse datasets. The authors also successfully addressed the reviewers’ concerns in their revision, leading to unanimous agreement on acceptance. Therefore, the AE recommends acceptance.
Decision: Accept (poster)