On the Influence of Shape, Texture and Color for Learning Semantic Segmentation

## Abstract

Abstract In recent years, a body of works has emerged, studying shape and texture biases of off-the-shelf pre-trained deep neural networks (DNN) for image classification. These works study how much a trained DNN relies on image cues, predominantly shape and texture. In this work, we switch the perspective, posing the following questions: What can a DNN learn from each of the image cues, i.e., shape, texture and color, respectively? How much does each cue influence the learning success? And what are the synergy effects between different cues? Studying these questions sheds light upon cue influences on learning and thus the learning capabilities of DNNs.
We study these questions on semantic segmentation which allows us to address our questions on pixel level.
To conduct this study, we develop a generic procedure to decompose a given dataset into multiple ones, each of them only containing either a single cue or a chosen mixture. This framework is then applied to two real-world datasets, Cityscapes and PASCAL Context, and a synthetic data set based on the CARLA simulator. We learn the given semantic segmentation task from these cue datasets, creating cue experts. Early fusion of cues is performed by constructing appropriate datasets. This is complemented by a late fusion of experts which allows us to study cue influence location-dependent on pixel level. Our study on three datasets reveals that neither texture nor shape clearly dominate the learning success, however a combination of shape and color but without texture achieves surprisingly strong results. Our findings hold for convolutional and transformer backbones. In particular, qualitatively there is almost no difference in how both of the architecture types extract information from the different cues.

## I Introduction

Visual perception relies on visual stimuli, so-called visual cues, providing information about the perceived scene. Visual environments offer multiple cues about a scene, and observers need to assess the reliability of each cue to integrate the information effectively [jacobs2002Whatdetermines]. To recognize and distinguish objects, for example, their shape and texture provide complementary cues. In cognitive science and psychology the influence of different cues and their combinations for human perception is addressed in various studies [bankieris2017Sensorycuecombination, jacobs2002Whatdetermines, michel2008Learningoptimal].
Given the widespread use of deep neural networks (DNNs) for automatically extracting semantic information from scenes, it is important to investigate 1) how these models process and rely on different types of cue information, as well as 2) what they are able to learn when only having access to specific cues.
With respect to the first question several hypotheses about dominating cue exploitation (cue biases) of trained convolutional neural networks (CNNs) were formulated and supported by experimental results for image classification tasks [geirhos2018ImageNettrainedCNNs, islam2021ShapeTexture, tuli2021AreConvolutional].
Early in the evolution of CNNs a shape bias was hypothesized, stating that representations of CNN outputs seem to relate to human perceptual shape judgement [kubilius2016DeepNeural].
Even though this suggests that CNNs tend to base their prediction on shape information, this is only valid on a local perspective [baker2018Deepconvolutional] and not an intrinsic property [hosseini2018AssessingShape].
On the contrary, multiple studies were performed on ImageNet-trained CNNs [deng2009ImageNetLargeScale], indicating that those CNNs have a bias towards texture [geirhos2018ImageNettrainedCNNs, baker2018Deepconvolutional, brendel2018ApproximatingCNNs].
To reveal biases in trained DNNs, cue conflicts are often generated [geirhos2018ImageNettrainedCNNs, gavrikov2024AreVision]. To this end, a style transfer with a texture image of one class, e.g. showing an animal’s fur or skin, is applied to an ImageNet image representing a different class.

In summary, previous studies 1) mostly study the biases of trained DNNs w.r.t. to image cues and their influence on the networks robustness [geirhos2018ImageNettrainedCNNs, kamann2020IncreasingRobustness, naseer2021Intriguingproperties, qiu2024ShapebiasedCNNs], 2) often rely on style transfer or similar image manipulations to test for biases [li2020ShapeTextureDebiased, islam2021ShapeTexture, dai2022Rethinkingimage], and 3) largely focus on image classification DNNs [brendel2018ApproximatingCNNs, hermann2020OriginsPrevalence, tuli2021AreConvolutional].

In this work, we present a study that is novel in all three aspects. 1) We switch the perspective, studying how much influence different image cues (and arbitrary combinations of those) have on the learning success of DNNs. 2) We do not rely on style transfer but rather utilize and develop a set of methods, combining them into a generic procedure to derive any desired cue combination of shape, texture and color from a given dataset, cf. fig. 1. 3) We lift our study to the task of semantic segmentation. This opens up paths to completely new studies as it allows for the decomposition of datasets into arbitrary combinations of cues and enables more fine-grained analyses such as image-location-dependent cue influences.

Figure: Figure 1: A sample of cues and cue combinations extracted from the Cityscapes dataset, based on which cue expert models are trained.
Refer to caption: cue_overview_landscape.pdf

Our study setup serves as basis to train expert networks exclusively relying on a specific cue or cue combination. We perform an in-depth behavioral analysis of CNNs and transformers on three different semantic segmentation datasets, namely Cityscapes [cordts2016CityscapesDataset], PASCAL Context [mottaghi_cvpr14] and a synthetic one recorded with the CARLA driving simulator [dosovitskiy2017CARLAOpen].
Our study brings the different cue and cue combination influences on DNN learning into an order. It turns out that neither texture nor shape clearly dominate in terms of learning success. However, a combination of shape and color achieves surprisingly strong results. These findings hold for CNNs and transformers.
In particular, qualitatively there is almost no difference in how both architecture types extract information from the different cues, i.e., the choice of backbone has almost no impact on the order of the cue influences. Additionally, by a pixel-wise late fusion of cue (combination) experts, we study the role of each pixel for cue influences, showing that small objects and pixels at object borders are dominantly better predicted by shape experts.
Our contributions are summarized as follows:

- •
We provide a generic procedure to derive cue combination datasets from a given semantic segmentation dataset. In particular, we provide a method to derive texture-only datasets.
- •
Our general setup allows to study disentangled image cues, down to the detail degree of brightness-only. We perform an in-depth analysis with up to $14$ learned cue combination experts per dataset and several late fusion models, contributing the first cue influence study in semantic segmentation. Additionally, we include transformers that are so far underresearched in the broader context of cue influences.
- •
While previous studies report strong biases of ImageNet-trained DNNs towards texture under style transfer, our study reveals that for real-world data shape and texture are equally important cues for successful learning. The role of the cues varies across classes and image location, but not w.r.t. choosing between CNNs and transformers.

Our code including the data generation procedure will be publicly available soon.

## II Related Work

Cue biases and cue decompositions are pivotal concepts in understanding how neural networks interpret visual information. We start by reviewing existing approaches of cues bias analyses in classification, followed by methods to decompose or manipulate data to allow for investigations on different cues. We conclude by addressing the underexplored area of texture and shape biases in semantic segmentation.

#### Shape and Texture Biases in Image Classification

DNNs learn unintended cues that help to solve the given task but limit their generalization capability [geirhos2020unintended]. In [geirhos2018ImageNettrainedCNNs] it was measured whether the shape or the texture cue dominates the decision process of an ImageNet-trained classification CNN by inferring images with conflicting cues (e.g. shape of a cat combined with the skin of an elephant), revealing that ImageNet pre-training leads to a texture bias. Technically, [geirhos2018ImageNettrainedCNNs] stylize images via style transfer to create cue conflict images. This was also adopted by [li2020ShapeTextureDebiased] and [islam2021ShapeTexture]. In the latter work, the bias is computed on a per-neuron level.
[hermann2020OriginsPrevalence] showed that the selection of data and the nature of the task influence the cue biases learned by a CNN.
Experiments in [naseer2021Intriguingproperties, tripathi2023EdgesShapes, tuli2021AreConvolutional, geirhos2021Partialsuccess] demonstrate that transformers exhibit a shape bias in classification tasks, which is attributed to their content-dependent receptive field [naseer2021Intriguingproperties].
Our focus differs in two key aspects: 1) we study cue influences on learning success rather than biases, 2) we consider the task of semantic segmentation instead of image classification.

#### Data Manipulation for Bias Investigation

To examine the influence of cues, datasets are manipulated by either artificially combining cues to induce conflicts [baker2018Deepconvolutional, geirhos2018ImageNettrainedCNNs, tripathi2023EdgesShapes, theodoridis2022TrappedTexture] or by selectively removing specific cues from the data [dai2022Rethinkingimage, zhang2024Convolutionalneural].
To remove all but the shape cue, edge maps [baker2018Deepconvolutional, mummadi2020Doesenhanced, tripathi2023EdgesShapes], contour maps [baker2018Deepconvolutional], silhouettes [baker2018Deepconvolutional] or texture reduction methods [dai2022Rethinkingimage, heinert2024Reducingtexture] are used. Patch shuffling has been proposed to remove the shape cue but preserve the texture [brendel2018ApproximatingCNNs, luo2020DefectiveConvolutional, dai2022Rethinkingimage].
In contrast to classification, data preparation is more complex for semantic segmentation as multiple objects and classes are present in the input which differ in their cues. In particular semantic segmentation on texture-only data is challenging [cote2023Semanticsegmentation].
Removing shape by dividing and shuffling an image as used by [dai2022Rethinkingimage] for classification compromises semantic integrity of the image and disrupts the segmentation task. Therefore, we propose an alternative method for extracting texture from the dataset, which is sufficiently flexible to generate new segmentation tasks using in-domain textures from the original dataset.

#### Shape and Texture Biases in Image Segmentation

Up to now, a limited number of works studied shape and texture biases beyond image classification.
In [li2020ShapeTextureDebiased], for image classification, images are stylized using a second image from the same dataset. For semantic segmentation, only a specific object rather than the full image is used as texture source to perform style transfer.
The stylized data is added to the training data to debias the CNN and increase its robustness.
Similarly, [theodoridis2022TrappedTexture] stylize data to analyze the robustness of instance segmentation networks under randomized texture. Additionally, an object-centric and a background-centric stylization are used for this study.
In [zhang2024Convolutionalneural], the change in shape bias of semantic segmentation networks is studied under varying cues in data. The experiments on datasets with a limited number of images and classes reveal that CNNs prioritize non-shape cues if multiple cues are present.
[kamann2020IncreasingRobustness] propose to colorize images with respect to the class IDs to reduce the influence of texture during training to prevent texture bias and thereby improving robustness of the trained model.
In [heinert2024Reducingtexture], an anisotropic diffusion image processing method is used for removing texture from images. Based on that the texture bias is studied and also reduced.
All works mentioned in this paragraph study or reduce biases of DNNs in image segmentation. To the best of our knowledge, the present work is the first one to switch the perspective and study the influence of cues and cue combinations on learning success in semantic segmentation DNNs. This is done on complex datasets with at least $15$ classes, by which we obtain different insights.

## III Cue Decomposition and Cue Expert Fusion

In this section we introduce the different methods that extract image cues and cue combinations from a given base dataset (original dataset with all cues), from which we train cue and cue combination experts.

#### Color (C) Cue Extraction

Most cue extractions presented in this work modify the base dataset. In order to isolate the color information, we do not modify the base dataset but constrain the cue expert, i.e., the neural network, to process a single pixel’s color values via $(1\times 1)$-convolutions. This prevents the model from learning spatial patterns such as shapes or textures. Furthermore, we decompose color into two components: its gray value (V) and its chromatic value (HS) by switching to the HSV color space and discarding the value channel. Gray values are obtained by averaging or maximizing RGB channels to extract the degree of darkness or lightness of a given color. In what follows, we use the shorthand C=V+HS to refer to the respective cues.

#### Texture (T) Cue Extraction

The texture dataset is constructed in three main steps: 1) texture is extracted by cropping object patches using semantic segmentation masks; 2) therefrom, for each class a mosaic image is created by stitching image patches of that class in an overlapping manner; and 3) a new semantic segmentation task is constructed via Voronoi diagrams where each cell is randomly assigned a class and filled with a cutout of the mosaic image corresponding to that class. Figure 2 illustrates this process.

Figure: Figure 2: Extraction process of the texture (T) cue. It consists of the three main steps: class-wise patch extraction, class-wise mosaic image construction and segmentation dataset creation based on Voronoi diagrams.
Refer to caption: texture-dataset_scheme_reduced_thick.pdf

#### Shape (S) Cue Extraction

The shape of an object can be described in multiple ways [feldman2024Visualperception], defined by the object with unrecognizable/removed texture or by the object defining edges. To this end, we consider two shape extraction methods: Holistically-nested edge detection (HED) [xie2015Holisticallynestededge] and a variation of Edge Enhancing Diffusion (EED) [heinert2024Reducingtexture]. HED approximately extracts the object defining edges based on a fully convolutional network which has learned to predict a dense edge map through end-to-end training and nested multiscale feature learning. HED approximates the S cue. In contrast, EED diminishes texture through diffusion along small color gradients in a given image. The diffusion process follows a partial-differential-equation formulation. As it preserves color, it extracts the cues S+V+HS, and analogously to the treatment of the C cue provides the cues S+V as well.

#### Texture-free Image Data from CARLA ( S rmv subscript S rmv \text{S}_{\text{rmv}} S start_POSTSUBSCRIPT rmv end_POSTSUBSCRIPT )

In the case of simulated data where access to the rendering engine is granted, a nearly texture-free environment can be generated.
The open-source simulator CARLA [dosovitskiy2017CARLAOpen] was employed to generate a virtual environment devoid of texture.
We replaced all objects’ textures by a gray checkerboard, a default texture pattern in CARLA. In addition, weather conditions were set to ’clear noon’ to obtain a uniform appearance of the sky. Strongly subsampled video sequences were recorded from an ego perspective in gray scale. This procedure provides an additional way to obtain the cues S+V in CARLA.

#### Remaining Cue Combinations and Summary

The remaining cue combinations are obtained as follows: S+T+V+HS, S+T+HS and S+T+V are obtained by treating an original image analogously to the color cue extraction, i.e., by transforming it into HSV color space and projecting accordingly. Analogously, we can process texture images to obtain T+HS as well as T+V.
To represent the complete absence of cues, i.e. no information is given, we consider the performance of randomly initialized DNNs.
An overview of all cue expert (datasets) including additional shorthands, encoding the method used and the cue extracted, are summarized in table I. Note that, in what follows we rely heavily on the shorthands. For additional technical details and exemplary images of the cue extraction methods, we refer the reader to sections A-A and A-B.

**TABLE I: Overview of cues and cue extraction methods. The included cues are S = shape, T = texture, V = gray component of the color and HS = hue and saturation component of the color. Orig is replaced by a shorthand of the respective base dataset.**
| shorthand | included cues | description |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| all cues | S | + | T | + | V | + | HS | original images / all cues |
| $\text{Orig}_{\text{HS}}$ | S | + | T |  |  | + | HS | all but gray cues |
| $\text{Orig}_{\text{V}}$ | S | + | T | + | V |  |  | all but chromatic cues |
| $\text{T}_{\text{RGB}}$ |  |  | T | + | V | + | HS | texture with color; shape removed |
| $\text{T}_{\text{HS}}$ |  |  | T |  |  | + | HS | texture with hue and saturation only |
| $\text{T}_{\text{V}}$ |  |  | T | + | V |  |  | texture with grayness only; |
| $\text{S}_{\text{EED-RGB}}$ | S |  |  | + | V | + | HS | shape with color via smoothing texture by edge enhancing diffusion |
| $\text{S}_{\text{EED-HS}}$ | S |  |  |  |  | + | HS | shape with hue and saturation by edge enhancing diffusion |
| $\text{S}_{\text{EED-V}}$ | S |  |  | + | V |  |  | shape with grayness by grayscaled edge enhancing diffusion results |
| $\text{S}_{\text{HED}}$ | S |  |  |  |  |  |  | shape only via contour map by holistically-nested edge detection |
| $\text{S}_{\text{rmv}}$ | S |  |  | + | V |  |  | shape with grayness via texture removal in the CARLA simulation |
| RGB |  |  |  |  | V | + | HS | complete color component of an RGB image, pixel-wise |
| HS |  |  |  |  |  |  | HS | hue and saturation of an RGB image, pixel-wise |
| V |  |  |  |  | V |  |  | gray component of an RGB image, pixel-wise |
| no info |  |  |  |  |  |  |  | no information about the data is given representing the absence of all cues |

#### Late Fusion of Cue Experts

Fusing the cue experts’ softmax activations serves as an assessment of the reliability of the different cues to combine the information of multiple cues effectively.

## IV Experiments

To analyze the cue influence in learning semantic segmentation tasks, we train all cue and cue combination experts on three different base datasets introduced below and ranging from real-world street scenes over synthetic street scenes to diverse in- and outdoor scenes.
We perform an in-depth analysis and comparison of several cue (combination) experts across these three datasets, across different evaluation granularities ranging from dataset-level over class-level to pixel-level evaluations. This is complemented by a comparison of CNN and transformer results. Additional experimental studies and qualitative examples are provided in section A-C.

### IV-A Base Datasets & Networks

#### Cityscapes

Cityscapes [cordts2016CityscapesDataset] is an automotive dataset which consists of high-resolution images of street scenes from $50$ (mostly German) cities with pixel annotations for $30$ classes, respectively. There are $2,\!975$ fine labeled training and $500$ validation images. As common practice we use only the $19$ common classes and the validation data for testing. For model selection purposes we created another validation set from a subset of the $20,\!000$ coarsely annotated images of Cityscapes by pseudo-labelling that subset using an off-the-shelf DeepLabV3+ [chen2018EncoderDecoderAtrous].

#### CARLA

We generated a dataset using the open-source simulator CARLA [dosovitskiy2017CARLAOpen], version 0.9.14,
containing multiple enumerated maps of which we used the towns $1$–$5$ and $7$ of similar visual detail level for data generation.
We record one frame per second from an ego perspective while driving with autopilot through a chosen city, accumulating $5,\!000$ frames per city in total.
For comparability to Cityscapes, we reduced the set of considered classes to $15$:
road, sidewalk, building, wall, pole,
traffic lights, traffic sign, vegetation, terrain, sky, person, car, truck, bus, guard rail.
The classes ‘bicycle’, ‘rider’ and ‘motorcycle’ are excluded for technical reasons because these actors were problematic for the autopilot mode in CARLA.
To ensure a location-wise disjoint training and test split, we use town $1$ and $5$ for testing only, whereas $2,3,4$ and $7$ are used for training. This results in $20,\!000$ training images and $10,\!000$ test images. For model selection purposes, we randomly split of 10% of the training images for validation.

#### PASCAL Context

As a third dataset we analyze cue influences on the challenging PASCAL Context dataset [mottaghi_cvpr14]. It provides annotations for the whole images of PASCAL VOC 2010 [Everingham2010]
in semantic segmentation style. The images are photographs from the flickr(^1^11[https://www.flickr.com](https://www.flickr.com)) photo-sharing website recorded and uploaded by consumers covering diverse indoor and outdoor scenes [Everingham2010]. We consider $33$ labels that are contextual categories from [mottaghi_cvpr14]. In our experiments we use $4,\!996$ training images, $2,\!042$ validation images and $3,\!062$ test images where the training images of [mottaghi_cvpr14] are split into training and validation images and the original validation images are utilized as test images.

#### Neural Networks and Training Details

For better comparison and to ensure that only the intended cues are learned, all neural networks are trained from scratch multiple times with different random seeds. For all cue (combination) expert models, except for the color cue ones, we employ DeepLabV3 with a ResNet18 backbone [chen2017RethinkingAtrous, he2016DeepResidual] ($15.9$ million learnable parameters) and a SegFormer model with B1 backbone [xie2021SegFormerSimple] ($13.7$ million learnable parameters). Both models have similar learning capacity and achieve similar mean IoU (mIoU) [jaccard1912DistributionFlora] segmentation accuracy. For the color experts, the receptive field is constrained to one pixel, achieved through a fully convolutional neural network with two to three ($1\times 1$)-convolutions (tuned to achieve maximal mIoU). We trained the CNN models with the Adam optimizer for $200$ epochs and a poly-linear learning rate decay with an initial learning rate of $5\cdot 10^{-4}$. The transformers were trained with the default optimizer and inference settings as specified in the MMSegmentation library [mmsegmentationcontributors2020OpenMMLabsemantic] except that we increased the number of gradient update steps to $170,\!000$ iterations.
Note that, to not mix cues unexpectedly, we restricted data augmentation to cropping and horizontal flipping and refrained from augmentations like color jittering.
For the late-fusion approach, we use an even smaller version of DeepLabV3 by limiting the backbone to two instead of four blocks to prevent overfitting to the simpler task.
The weights of the late fusion models are initialized randomly within a uniform distribution of $[-10^{-3},10^{-3}]$.

### IV-B Numerical Results

#### Cue Influence on Mean Segmentation Performance

In this section, we investigate the accuracies of the different cue (combination) experts in terms of mIoU when evaluated on the test base images across the three different base datasets. To ensure input compatibility, we evaluate experts with reduced channels like the V expert on the corresponding $\text{Orig}_{\text{V}}$ data. Comparing the performance drop (gap) between the reduced model and the full model demonstrates the significance of the removed cues in terms of their impact on the original semantic segmentation task. The corresponding results are presented in tables II and III. Firstly, we note a certain consistency in the presented mIoU results across base datasets. C experts are mostly dominated by T experts as well as S experts, and those are in turn dominated by S+T experts. The S+T+V expert is the only one getting close to the ‘expert’ trained on all cues. The order of the specific experts varies slightly across datasets, however not drastically. Also across all three datasets the RGB version of EED providing the cues S+C achieves surprisingly high mIoU values, indicating that in presence of color and shape, a DNN can learn well without having access to texture. However, this mIoU value is dominated by S+T+V as well as S+T+HS. Note that a combination of S+T only is impossible since T cannot exist without some kind of brightness or color cue, i.e., V and/or HS.

**TABLE II: Cue influence in terms of mIoU performance drop on Cityscapes for DeepLabV3 with ResNet backbone and SegFormer. Cue description follows the listing in table I. MIoU gaps to maximal performance are stated in percent points (pp.). The abbreviation “City” refers to Cityscapes.**
|  |  |  | Color | CNN mIoU | CNN gap | transformer mIoU | transformer gap | change in rank |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | S | T | V | HS | (%, $\uparrow$) | (pp., $\downarrow$) | (%, $\uparrow$) | (pp., $\downarrow$) | w.r.t. CNN |
| no info |  |  |  |  | $\,\;0.25\pm 0.35$ | $64.97$ | $\;\,0.33\pm 0.47$ | $66.02$ | $\to$ |
| V |  |  | ✓ |  | $\,\;6.39\pm 0.04$ | $58.83$ |  |  |  |
| HS |  |  |  | ✓ | $\,\;9.33\pm 0.18$ | $55.89$ |  |  |  |
| RGB |  |  | ✓ | ✓ | $11.31\pm 0.52$ | $53.91$ |  |  |  |
| $\text{S}_{\text{HED}}$ | ✓ |  |  |  | $13.38\pm 2.00$ | $51.84$ | $11.31\pm 1.95$ | $55.05$ | $\to$ |
| $\text{T}_{\text{V}}$ |  | ✓ | ✓ |  | $17.85\pm 1.30$ | $47.37$ | $29.02\pm 0.31$ | $37.33$ | $\to$ |
| $\text{S}_{\text{EED-HS}}$ | ✓ |  |  | ✓ | $19.48\pm 3.19$ | $45.74$ | $30.93\pm 0.64$ | $35.42$ | $\searrow_{1}$ |
| $\text{T}_{\text{RGB}}$ |  | ✓ | ✓ | ✓ | $20.10\pm 0.98$ | $45.12$ | $31.88\pm 0.38$ | $34.47$ | $\searrow_{1}$ |
| $\text{T}_{\text{HS}}$ |  | ✓ |  | ✓ | $20.63\pm 1.41$ | $44.59$ | $29.49\pm 0.50$ | $36.86$ | $\nearrow^{2}$ |
| $\text{S}_{\text{EED-V}}$ | ✓ |  | ✓ |  | $27.86\pm 3.17$ | $37.36$ | $39.01\pm 0.75$ | $27.34$ | $\to$ |
| $\text{S}_{\text{EED-RGB}}$ | ✓ |  | ✓ | ✓ | $42.22\pm 2.13$ | $23.00$ | $50.48\pm 0.55$ | $15.87$ | $\to$ |
| $\text{City}_{\text{HS}}$ | ✓ | ✓ |  | ✓ | $59.89\pm 0.74$ | $\;\,5.33$ | $58.79\pm 0.40$ | $\;\,7.56$ | $\to$ |
| $\text{City}_{\text{V}}$ | ✓ | ✓ | ✓ |  | $64.21\pm 0.60$ | $\;\,1.01$ | $64.47\pm 0.21$ | $\;\,1.88$ | $\to$ |
| all cues | ✓ | ✓ | ✓ | ✓ | $65.22\pm 0.47$ | $\;\,0.00$ | $66.35\pm 0.29$ | $\;\,0.00$ | $\to$ |

**TABLE III: Cue influence measured in terms of mIoU performance on the synthetic CARLA dataset and PASCAL Context (short: PASCAL). MIoU gaps to maximal performance are stated absolute in percent points (pp.).**
| CARLA | PASCAL Context |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | mIoU | gap | rank change w.r.t. |  | mIoU | gap | rank change w.r.t. |
|  | ($\%,\uparrow$) | (pp., $\downarrow$) | Cityscapes CNN |  | ($\%,\uparrow$) | (pp., $\downarrow$) | Cityscapes CNN |
| no info | $\,\;0.38\pm 0.44$ | $\,75.27$ | $\to$ | no info | $\;\,0.11\pm 0.11$ | $45.34$ | $\to$ |
| V | $\,\;6.01\pm 0.10$ | $\,69.64$ | $\to$ | V | $\;\,2.27\pm 0.04$ | $43.18$ | $\to$ |
| $\text{S}_{\text{HED}}$ | $11.35\pm 0.65$ | $\,64.30$ | $\nearrow^{2}$ | HS | $\;\,3.35\pm 0.10$ | $42.10$ | $\to$ |
| HS | $14.70\pm 0.18$ | $\,60.95$ | $\searrow_{1}$ | $\text{S}_{\text{HED}}$ | $\;\,4.71\pm 1.15$ | $40.74$ | $\nearrow^{1}$ |
| RGB | $15.60\pm 0.56$ | $\,60.05$ | $\searrow_{1}$ | RGB | $\;\,4.91\pm 0.05$ | $40.54$ | $\searrow_{1}$ |
| $\text{S}_{\text{textureless}}$ | $26.96\pm 2.00$ | $\,48.69$ |  |  |  |  |  |
| $\text{S}_{\text{EED-V}}$ | $37.10\pm 2.13$ | $\,38.55$ | $\nearrow^{4}$ | $\text{T}_{\text{HS}}$ | $11.39\pm 0.36$ | $34.06$ | $\nearrow^{3}$ |
| $\text{S}_{\text{EED-HS}}$ | $44.97\pm 0.93$ | $\,30.68$ | $\to$ | $\text{T}_{\text{RGB}}$ | $17.75\pm 0.82$ | $27.70$ | $\nearrow^{1}$ |
| $\text{T}_{\text{V}}$ | $46.07\pm 3.34$ | $\,29.58$ | $\searrow_{2}$ | $\text{S}_{\text{EED-HS}}$ | $17.80\pm 1.67$ | $27.65$ | $\searrow_{1}$ |
| $\text{T}_{\text{HS}}$ | $52.38\pm 1.65$ | $\,23.27$ | $\to$ | $\text{T}_{\text{V}}$ | $18.43\pm 0.47$ | $27.02$ | $\searrow_{3}$ |
| $\text{T}_{\text{RGB}}$ | $56.51\pm 1.75$ | $\,19.14$ | $\searrow_{2}$ | $\text{S}_{\text{EED-V}}$ | $25.80\pm 2.40$ | $19.65$ | $\to$ |
| $\text{S}_{\text{EED-RGB}}$ | $61.46\pm 1.03$ | $\,14.18$ | $\to$ | $\text{S}_{\text{EED-RGB}}$ | $31.32\pm 0.82$ | $14.13$ | $\to$ |
| $\text{CARLA}_{\text{HS}}$ | $71.02\pm 0.93$ | $\;\;\,4.63$ | $\to$ | $\text{PASCAL}_{\text{HS}}$ | $36.10\pm 0.34$ | $\;\,9.35$ | $\to$ |
| all cues | $75.65\pm 1.80$ | $\;\;\,0.00$ | $\nearrow^{1}$ | $\text{PASCAL}_{\text{V}}$ | $45.39\pm 0.71$ | $\;\,0.06$ | $\to$ |
| $\text{CARLA}_{\text{V}}$ | $75.73\pm 1.07$ | $-0.08$ | $\searrow_{1}$ | all cues | $45.45\pm 0.18$ | $\;\,0.00$ | $\to$ |

An additional surprise might be that HED, representing the S cue, reaches only very low mIoU, showing consistently weak performance across all datasets. At the first glance, this is in contrast to human vision since an HED image seems almost enough for a human to estimate the class of each segment in an HED image, cf. fig. 1. It should be noted that each expert is trained on its specific cue and then tested on an original input image from the given base dataset.
When alternatively applying a given experts cue extraction technique as pre-processing to real-world images, like Cityscapes or PASCAL Context, HED (with HED pre-processing) surpasses EED (with EED-pre-processing) distinctly and achieves $55.80\%\pm 0.59$ percent points (pp.) mIoU compared to $48.47\%\pm 0.45$ pp. by EED on Cityscapes. On CARLA, EED and HED are on par, reaching an mIoU of $65.93\%\pm 0.72$ pp. and $63.33\%\pm 1.11$ pp. respectively.

Figure: Figure 3: Class-specific cue influence for CNN based $\text{S}_{\text{EED-RGB}}$ and $\text{T}_{\text{RGB}}$ on Cityscapes.
Refer to caption: x1.png

#### Cue Influence on Different Semantic Classes

Figure: Figure 4: Comparison of the prediction of the two experts $\text{S}_{\text{EED-RGB}}$ (left) and $\text{T}_{\text{RGB}}$ (mid) for Cityscapes, CARLA and PASCAL Context. As a reference the ground truth is displayed in the third column (right).
Refer to caption: x2.png

Figure 3 provides an evaluation for Cityscapes, comparing a shape expert based on colored EED images providing the cues S+C and another expert trained on colored Voronoi cells of class-specific stitched patches, providing the queues T+C. This evaluation is broken down into IoU values over the $19$ Cityscapes classes. In a visual inspection, we found that an IoU of at least $20\%$ for a given class starts to support the claim that the chosen expert extracts information for the given class.
Hence, we see in fig. 3 that the S+C expert can deal with most of the classes while the T+C expert specializes to classes that usually cover large areas of the image, like vegetation, road and building, see also figs. 4 and 15 for visual examples.
This size dependence is also studied in fig. 6 for CARLA as base dataset for the classes road and person, where the former contains large ground truth segments and the latter small ones. The quality of detection of a ground truth segment is measured in terms of a segment-wise recall, which is the fraction of pixels in the ground truth segment that are covered by a prediction of the same class.
It can be seen that for the large road segments, the texture expert achieves a high segment-wise recall, indeed finding most of the road segments, while the shape expert has only a low segment-wise recall for those large segments.

#### Cue Influence Dependent on Location in an Image

Figure: texture expert (T+C) prediction
Refer to caption: extracted/5938599/figures/fusion/texture_shape_visuals/CS/frankfurt_000001_072295_leftImg8bit_texture_pred.png

In this paragraph, we provide a detailed comparison of the same two experts from the previous paragraph, the shape expert based on EED having access to the cues S and C, and the texture expert based on the Voronoi images having access to the cues T and C. Here, cues are considered based on their location in an image, and we provide results for all three datasets.

We already noted in the previously presented class-specific study that the T+C expert focuses on classes covering larger areas of an image. To study this effect in-depth, we trained a late fusion that processes the softmax outputs of both experts T+C and S+C, learning a pixel-wise weighting of both expert’s outputs. A visual example for Cityscapes is provided in fig. 5. In general, we observed that the shape influences the fusion in regions containing boundaries of class-segments while priority is given to the texture inside larger segments. As can also be seen in fig. 5, the texture experts often have difficulties to accurately segment the boundaries of areas corresponding to a semantic class. We measured this effect quantitatively and provide results in table IV. The results reveal that the shape experts clearly outperform the texture expert on segment boundaries in terms of accuracy averaged over all boundary pixels.
Herein, a pixel is considered as part of a segment boundary, if its neighborhood of four pixels distance (in Manhattan metric) contains a pixel of a different class according to the ground truth. On the segments’ interior, on Cityscapes and PASCAL Context, the shape expert is on average still more useful than the texture expert. This is the other way round in CARLA. This can be explained by the observations that Cityscapes is in general relatively poor in texture and the textures in both real-world datasets are not very discriminatory. However, in the driving simulator CARLA, the limited number of different textures rendered onto objects is highly discriminatory and increases the texture expert’s performance. Quantitative and additional qualitative results for the late fusion of cue experts are provided in the appendix.

Figure: Figure 6: Coverage of experts over the classes ’road’ (left) and ’person’ (right) on the CARLA dataset. The recall on the y-axis is defined by the fraction of pixels in a ground truth segment covered by a prediction of the same class.
Refer to caption: extracted/5938599/figures/scatterplots/road_dist_anisodiff_rgb.png

**TABLE IV: Comparison of the pixel accuracy for $\text{S}_{\text{{EED-RGB}}}$ and $\text{T}_{\text{RGB}}$ with respect to segment boundary pixels and segment interior pixels.**
| pixel-averaged | Cityscapes | CARLA | PASCAL Context |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| accuracy (%, $\uparrow$) | $\text{S}_{\text{EED-RGB}}$ | $\text{T}_{\text{RGB}}$ | $\text{S}_{\text{EED-RGB}}$ | $\text{T}_{\text{RGB}}$ | $\text{S}_{\text{EED-RGB}}$ | $\text{T}_{\text{RGB}}$ |
| segment interior | $88.59$ | $75.10$ | $82.63$ | $89.83$ | $56.48$ | $39.96$ |
| segment boundary | $56.49$ | $37.16$ | $70.44$ | $47.94$ | $38.83$ | $25.99$ |
| overall | $86.17$ | $72.24$ | $81.84$ | $87.09$ | $55.15$ | $38.91$ |

#### Cue Influence in Different Architectures

In addition to the previously discussed CNN-based cue experts, we also studied the cue extraction capabilities of a segmentation transformer, namely SegFormer-B1 [xie2021SegFormerSimple]. The results on Cityscapes are provided in table II.
Although the CNN and transformer mIoU is almost on par when all cues are present, we observe a distinct increase in mIoU for the individual cue experts when using a transformer instead of a CNN. This holds for all T experts and S experts and their combinations with V and HS with one exception which is the HED cue extraction. We expect that the HED mIoU suffers from the strong domain shift between HED images and original images, to which the HED expert is applied in table II. Similarly, SegFormer achieves an mIoU of $54.81\%\pm 0.36$ pp. when applying HED as pre-processing.
Nonetheless, qualitatively, i.e., in terms of the rankings of the different cue experts, we do not observe any serious differences between CNNs and transformers. These findings generalize to the class level where we observe an increase in performance in nearly all classes independent of the expert, but qualitatively the influence of the cues does not change.

## V Conclusion and Outlook

In this paper, we provided the first study on what can be learned by semantic segmentation DNNs from different image cues. Here, as opposed to image classification, studying cue influence is much more intricate and yields more specific and more fine-grained results. We introduced a generic procedure to extract cue-specific datasets from a given semantic segmentation dataset and studied several cue (combination) expert models, CNNs and transformers, across three different datasets and different evaluation granularities.
We compared different cue (combination) experts in terms of mIoU on the whole dataset, in terms of semantic classes and in terms of image-location dependence. Except for grayscale images, there is no reduced cue combination that achieves a performance close to all cues.
Cues influence the learning of DNNs over image locations, indeed shape often matters more in the vicinity of semantic class boundaries.
Shape cues are important for all classes, texture cues mostly for classes where the corresponding segment covers large regions of the images in the dataset. We expect that our generic cue extraction procedure (or at least parts of it) can also be utilized for studying biases in pre-trained off-the-shelf DNNs in cue conflict schemes, typical for that field of research.

## Acknowledgments

This work is supported by the German Federal Ministry for Economic Affairs and Climate Action within the project “KI Delta Learning”, grant no. 19A19013Q. We thank the consortium for the successful cooperation. A. M., N. G., E. H. and M. R. acknowledge support through the junior research group project “UnrEAL” by the German Federal Ministry of Education and Research (BMBF), grant no. 01IS22069.
Moreover, the authors gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for funding this project by providing computing time through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS at [julichsupercomputingcentre2019JUWELSModular] (JSC).

## Appendix A Appendix

In the appendix, we provide additional results as well as technical details of the procedure.

### A-A Cue Decomposition

In fig. 7 we show exemplary images of cues extracted from the Cityscapes dataset.
For all but the images containing the T cue, the cues can be extracted solely based on the base image (all cues). In contrast, the texture image is generated based on multiple images of the base dataset.

Figure: Figure 7: Overview of the cue decomposition of a Cityscapes image (all cues) into texture, shape, hue and saturation (HS) and gray components. For the color cues the RGB, HS and V value distribution of the image are scattered for visualization purpose.
Refer to caption: x12.png

### A-B Technical and Implementation Details

#### Texture Data Generation Details

A detailed scheme of the texture extraction procedure is given in fig. 8.
The class-wise texture extraction is realized by masking all segments of one class and isolating each segment with the help of the border following algorithm proposed by [suzuki1985Topologicalstructural]. We discard segments with less than $36$ pixels. The remaining segments are used to mask the original image and cut out the image patch of the enclosing bounding box of the segment. To enlarge the resulting patch pool, we apply horizontal flipping, random center crop and shift-scale-rotation augmentation. To mitigate the class imbalance in terms of pixel counts, we add more transformed patches for underrepresented classes than for classes which cover large areas of an image.
It is essential to ensure that the transformations do not alter the texture.
Once all the texture patches within the underlying dataset have been extracted, they are randomly composed into mosaic images. We iteratively fill images of the same size as images from the base dataset with overlapping texture patches until we obtain completely filled mosaic images. To further reduce an unintended dependency on the shape cue we fill the mosaics in the original segmentation mask but assign each pixel to the same class (contour filled texture images). This implies that the segment boundaries are not discriminatory anymore. Based on a pool of contour filled texture images of different classes we can create a surrogate segmentation task by filling the segments of arbitrary segmentation masks with the generated texture images. We choose Voronoi diagrams [torquato2002CellRandomField] as surrogate segmentation task and fill each cell randomly but uniformly with respect to the class with crops of the pool of contour filled texture images. To generate
an entire dataset, we create and fill as many Voronoi diagrams as the number of images in the base dataset.

Figure: Figure 8: Detailed scheme of the texture cue extraction process. It consists of the three main steps: class-wise patch extraction, class-wise mosaic image construction and segmentation dataset creation based on Voronoi diagrams.
Refer to caption: texture-dataset_scheme_v4.pdf

#### HED Implementation Details

To generate object defining edge maps we use the implementation and pre-trained model of [harary2022UnsupervisedDomain] which bases on the PyTorch re-implementation of the original method by [niklaus2018ReimplementationHED].

#### EED Data Generation Details

EED is an anisotropic diffusion technique based on Partial Differential Equations (PDEs) [perona1994anisotropic, weickert1998anisotropic]. Starting with the original image as the initial value, it utilizes a spatially dependent variation of the Laplace operator to propagate color information along edges but not across them, thus leading to texture being largely removed from images and higher level features being largely preserved.
For the production of the EED data [weickert1998anisotropic] we have used a variation proposed in [heinert2024Reducingtexture] that avoids circular artifacts and preserves shapes particularly well by applying spatial orientation smoothing as proposed for Coherence Enhancing Diffusion in [weickert1999coherence]. The PDE is solved using explicit Euler, channel coupling from [weickert2006tensor, p. 321] and the discretization described in [weickert20132, p. 380-391].

The diffusion parameters are the same for all three data sets, Cityscapes, CARLA and PASCAL Context:

- •
Contrast parameter $\lambda=1/15$.
- •
Gaussian blurring kernel size $k=5$ and standard deviation $\sigma=\sqrt{5}$.
- •
Time step length $\tau=0.2$ and artificial spatial distance $h=1$.
- •
Number of time steps $N_{EED}=8192$.
- •
Discretization parameters $\alpha=0.49$ and $\beta=0$.

For a visual example see fig. 7. The used implementation can be found in [https://github.com/eheinert/reducing-texture-bias-of-dnns-via-eed/](https://github.com/eheinert/reducing-texture-bias-of-dnns-via-eed/).

#### Carla Data Generation Details

Although CARLA claims to support online texture switching as of version 0.9.14, this feature is limited to specific instances and found to be inadequate for the purposes of this study. As a consequence, we manually modified each material instance, replacing surface textures with a basic default texture pattern (gray checkerboard). Since the sky is not a meshed object, it was not possible to manipulate its texture. Instead, we set the weather conditions to clear noon to achieve a uniform texture. The dataset is recorded by a vehicle in CARLA with an RGB camera and semantic segmentation sensor in the ego perspective driving in autopilot mode as described in section IV-A. In a post-processing step the images are gray scaled to remove the sky color. The basic components of the procedure are visualized in fig. 9.

Figure: (a) basic texture
Refer to caption: extracted/5938599/figures/textureless/master_material_crop_heller.png

#### Model Implementation Details

For the CNN-based models we adapt the DeepLabV3 model from torchvision(^2^22[https://github.com/pytorch/vision/blob/main/torchvision/models/segmentation/deeplabv3.py](https://github.com/pytorch/vision/blob/main/torchvision/models/segmentation/deeplabv3.py)) to our needs. For the transformer models we use the MMSegmentation framework [mmsegmentationcontributors2020OpenMMLabsemantic].

### A-C Additional Numerical Experiments

#### Comparison of Cue Influence Without Domain Shift

Except for the texture extraction procedure all cue extraction methods can be applied as an online transformation during inference. This allows us to compare the cue influence without domain shift for the shape experts. We observe that the S cue based on object contours ($\text{S}_{\text{HED}}$) mostly outperforms other experts trained on shape cues, also when the latter receive additional color cue information, see table V.

**TABLE V: Cue influence in terms of mIoU performance for the CNN shape experts when evaluated in-domain, i.e., the validation dataset is pre-processed with the same cue extraction method as the training dataset. The table is sorted according to the performances reported in table II. The best in-domain performance is highlighted in bold and the second best is underlined.**
|  | Cityscapes in-domain | CARLA in-domain | PASCAL Context in-domain |
| --- | --- | --- | --- |
|  | mIoU (%, $\uparrow$) | mIoU (%, $\uparrow$) | mIoU (%, $\uparrow$) |
| $\text{S}_{\text{HED}}$ | $\textbf{55.80}\pm 0.59$ | $\underline{63.33}\pm 1.11$ | $\textbf{37.64}\pm 0.14$ |
| $\text{S}_{\text{EED-HS}}$ | $45.89\pm 0.51$ | $61.67\pm 1.90$ | $29.13\pm 1.14$ |
| $\text{S}_{\text{EED-V}}$ | $45.02\pm 0.51$ | $60.73\pm 2.70$ | $33.05\pm 0.37$ |
| $\text{S}_{\text{EED-RGB}}$ | $\underline{48.47}\pm 0.45$ | $\textbf{65.93}\pm 0.72$ | $\underline{35.35}\pm 0.76$ |
| $\text{S}_{\text{rmv}}$ | - | $62.03\pm 1.51$ | - |

#### Cue Influence on Different Semantic Classes

Figures 10, 11 and 12 show that the results reported on Cityscapes generalize to the other base datasets as well as to the transformer model. However, for the CARLA base dataset we observe that the texture is more discriminatory compared to the real-world datasets (cf. column $4$ ‘rank change’ in table III) and the performance of the $\text{T}_{\text{RGB}}$ expert on classes like road, sidewalk and building surpasses the performance of the comparable shape expert $\text{S}_{\text{EED-RGB}}$. In fig. 15 we provide additional qualitative results for this finding.

Figure: Figure 10: Class-specific cue influence for the shape and texture cue, both with RGB color cue, on CARLA.
Refer to caption: x25.png

Figure: Figure 11: Class-specific cue influence for the shape and texture, both cue with RGB color cue, on PASCAL Context.
Refer to caption: x26.png

Figure: Figure 12: Class-specific cue influence for the transformer based shape and texture cue, both with RGB color cue, on Cityscapes.
Refer to caption: x27.png

#### Cue Influence Dependent on Location in an Image

Furthermore, our analysis on pixel level reveals for the CARLA base dataset, that the S+C and T+C cues are complementary.
Fusing $\text{S}_{\text{EED-RGB}}$ and $\text{T}_{\text{RGB}}$ improves the overall scene understanding by a notable margin. The fusion model achieves a performance of $78.10\%$ mIoU on the CARLA test set which is about $19$ pp. superior to the test set performance of $\text{S}_{\text{EED-RGB}}$ and more than $22$ pp. superior to when relying only on $\text{T}_{\text{RGB}}$ (cf. fig. 10). In CARLA the described dominated shape influence on segment boundaries (cf. table IV) is more pronounced and thus exploited by the fusion model, see fig. 13. Additional qualitative results of the fusion prediction and the corresponding pixel-wise weighting of the expert influence is depicted in fig. 14.

Figure: prediction of texture expert
Refer to caption: extracted/5938599/figures/fusion/texture_shape_visuals/carla_fusion/Town05_00024742_textureRgb_pred.png

Figure: Figure 14: Fusion of the softmax prediction of the two experts $\text{S}_{\text{EED-RGB}}$ and $\text{T}_{\text{RGB}}$ on CARLA (see fig. 15 for the single expert prediction). The heatmap shows how much influence the expert’s prediction has in the fusion model. Dark green pixels mean the fusion model bases its prediction on the shape expert, while light purple pixels mean it is based on the texture expert.
Refer to caption: Town01_fusion_pred-und-heatmap_PuBuGn.pdf

A qualitative overview of the predictions of all cue experts is shown in figs. 16, 17 and 18 for Cityscapes, CARLA and PASCAL Context, respectively.
We observed for the street scene base datasets that already a non-negligible amount of information, like the position of a car, can be learned solely based on the color value of a pixel (cf. figs. 16 and 17 last row).

Figure: Figure 15: Comparison of the prediction of the two experts $\text{S}_{\text{EED-RGB}}$ (left) and $\text{T}_{\text{RGB}}$ (mid) for Cityscapes, CARLA and PASCAL Context. As a reference the ground truth is displayed in the third column (right).
Refer to caption: x29.png

Figure: (a) image
Refer to caption: extracted/5938599/figures/city_pred_experts/munster_000045_000019_leftImg8bit.png

Figure: (a) image
Refer to caption: extracted/5938599/figures/carla_pred_expert/Town01_000048_image.png

Figure: Figure 18: Overview of the predictions of all cues for a PASCAL Context image.
Refer to caption: x41.jpg

#### Cue Influence in Different Architectures.

The transformer experiments conducted on the PASCAL Context dataset demonstrate results comparable to those obtained on the Cityscapes dataset (cf. table II).
These findings suggest that the influence of cues remains largely consistent between CNN and transformer architectures, as the order of cues has not experienced significant changes (see last column of table VI for reference).

**TABLE VI: Cue influence in terms of mIoU performance drop on PASCAL Context for DeepLabV3 with ResNet backbone and SegFormer. Cue description follows the listing in table I. MIoU gaps to maximal performance are stated in percent points (pp.). The abbreviation “PASCAL” refers to PASCAL Context.**
|  |  |  | Color | CNN mIoU | CNN gap | transformer mIoU | transformer gap | change in rank |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | S | T | V | HS | (%, $\uparrow$) | (pp., $\downarrow$) | (%, $\uparrow$) | (pp., $\downarrow$) | w.r.t. CNN |
| no info |  |  |  |  | $\,\;0.11\pm 0.11$ | $45.34$ | $\,\;0.45\pm 0.34$ | $43.14$ | $\to$ |
| V |  |  | ✓ |  | $\,\;2.27\pm 0.04$ | $43.18$ |  |  |  |
| HS |  |  |  | ✓ | $\,\;3.35\pm 0.10$ | $42.10$ |  |  |  |
| $\text{S}_{\text{HED}}$ | ✓ |  |  |  | $\;\,4.71\pm 1.15$ | $40.74$ | $\;\,5.04\pm 0.47$ | $38.55$ | $\searrow^{1}$ |
| RGB |  |  | ✓ | ✓ | $\;\,4.91\pm 0.05$ | $40.54$ |  |  |  |
| $\text{T}_{\text{HS}}$ |  | ✓ |  | ✓ | $11.39\pm 0.36$ | $34.06$ | $13.70\pm 0.14$ | $29.89$ | $\to$ |
| $\text{T}_{\text{RGB}}$ |  | ✓ | ✓ | ✓ | $17.75\pm 0.82$ | $27.70$ | $19.99\pm 0.21$ | $23.60$ | $\to$ |
| $\text{S}_{\text{EED-HS}}$ | ✓ |  |  | ✓ | $17.80\pm 1.67$ | $27.65$ | $21.15\pm 0.49$ | $22.44$ | $\searrow^{1}$ |
| $\text{T}_{\text{V}}$ |  | ✓ | ✓ |  | $18.43\pm 0.47$ | $27.02$ | $20.27\pm 0.23$ | $23.32$ | $\nearrow_{1}$ |
| $\text{S}_{\text{EED-V}}$ | ✓ |  | ✓ |  | $25.80\pm 2.40$ | $19.65$ | $22.78\pm 0.65$ | $20.81$ | $\to$ |
| $\text{S}_{\text{EED-RGB}}$ | ✓ |  | ✓ | ✓ | $31.32\pm 0.82$ | $14.13$ | $32.09\pm 0.75$ | $11.50$ | $\to$ |
| $\text{PASCAL}_{\text{HS}}$ | ✓ | ✓ |  | ✓ | $36.10\pm 0.34$ | $\;\,9.35$ | $35.03\pm 0.16$ | $\;\,8.56$ | $\to$ |
| $\text{PASCAL}_{\text{V}}$ | ✓ | ✓ | ✓ |  | $45.39\pm 0.71$ | $\;\,0.06$ | $43.22\pm 0.04$ | $\,\;0.37$ | $\to$ |
| all cues | ✓ | ✓ | ✓ | ✓ | $45.45\pm 0.18$ | $\;\,0.00$ | $43.59\pm 0.36$ | $\;\,0.00$ | $\to$ |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Using Voronoi diagrams, assigned randomly to semantic classes, raises questions about whether the generated "texture" dataset truly reflects the original distribution of classes. How about assigning semantic classes to Voronoi diagrams by their frequency in the original dataset?

The reliance on an external edge detector for shape cues introduces external biases. For example, if the edge detectors are trained on semantic boundaries, using them will introduce additional semantic information. How about using low-level filter-based edge detectors?

When shape and color are combined (no texture), the input image becomes piecewise constant. Since the segmentation maps are also piecewise constant, this operation simplifies the mapping between input and output. The authors present this result as surprising, yet it is somewhat intuitive given the characteristics of segmentation tasks.

mIoU may not be a good metric to fully capture the performance across classes with different frequencies. A more robust analysis would include additional metrics such as frequency-weighted IoU and pixel accuracy.

Figures 10 and 11 are interesting and important. I suggest moving them to the main paper.
soundness: 3
strengths: The paper is easy to read, with a significant level of detail dedicated to the experiments and results. The study is comprehensive in its scope, providing extensive combinations of cue-based datasets and evaluating them across different model architectures. 

By isolating and recombining shape, color, and texture cues, the paper sheds light on how each of these elements contributes to segmentation performance. This could be beneficial for understanding data domain gaps and predicting failure modes of segmentation models.
summary: The paper presents an analysis of the influence of shape, texture, and color on semantic segmentation performance, proposing a methodology that leverages augmented datasets to isolate these specific ‘cues’ and assess their individual contributions. To achieve this, the paper uses HSV channels to represent colors, edge detection to represent shapes, and ground-truth mask cropping to isolate textures. Through extensive combinations of these cues, the authors evaluate the performance of semantic segmentation using both real-world datasets and the CARLA simulator, applying this methodology to both convolutional and transformer-based architectures.
weaknesses: The study’s approach, while straightforward, lacks sufficient depth in its experimental design and interpretation. The experiments primarily provide surface-level insights without delving into a more profound analysis of underlying factors.

Most of the findings and corresponding discussion, for example, that shape cues correspond heavily to semantic boundaries and that textures play a significant role in broader regions—are valid but not particularly surprising and interesting to computer vision researchers. Most of the results presented in the paper fall into this category. The insights gained from this study feel somewhat limited, leaving the reader with few novel takeaways. As such, it may fall marginally below the acceptance threshold in its current form.

There are several technical and methodological concerns, please see Questions.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. After we have studied the impact of different visual cues on the segmentation effect of the model, can we use certain deep learning operations to specifically improve the effect of the model?
2. In Table 3, we find that the ranking changes of the CARLA dataset are larger than those of the Cityscapes dataset. Why do visual cues show such differences on different datasets?
soundness: 3
strengths: 1. This paper comprehensively analyzes the impact of shape, texture, gray and their combination on semantic segmentation tasks, and provides a method to derive a texture-only dataset. This paper compares the different effects of these image cues on CNN and Transformer.
2. The structure of the paper is relatively clear and the introduction of the methods is relatively detailed.
3. Studying the impact of different image cues on semantic segmentation is very meaningful for designing networks and contributes to the deep learning community.
summary: This work studies the impact of shape and texture on training DNN. They develop a generic procedure to decompose a given dataset into multiple ones, each of them only containing either a single cue or a chosen mixture. they develop a generic procedure to decompose a given dataset into multiple ones, each of them only containing either a single cue or a chosen mixture. The study on three datasets reveals that neither texture nor shape clearly dominates the learning success, however, a combination of shape and color but without texture achieves surprisingly strong results. The findings hold for convolutional and transformer backbones. In particular, qualitatively there is almost no difference in how both of the architecture types extract information from the different cues.
weaknesses: 1. This paper shows the effects of different visual cues on semantic segmentation, but lacks a detailed analysis of why these effects occur, and explores which modules and operations are introduced into the model to reduce these effects.
2. In Table 3, we find that the rank change of the CARLA dataset is large relative to the Cityscapes dataset. Please provide more explanations why different performances are shown on different datasets.
3. Figure 6 is not clear enough.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Do you expect similar conclusions for panoptic segmentation as well? This might open some new questions. For example, if the same clues have equal effect on the performance in thing and stuff classes? Mask transformers [1] disentangle the panoptic inference into mask localization and classification. It would be interesting to see which clues are more important for localization, and which ones for classification. What are your thoughts about this and the future work?

In table 2, the convolutional model and the transformer achieve similar performance when considering regular images with all cues. However, the transformer consistently outperforms the convolutional model by a large margin on datasets which consider only subsets of clues. What is the reason behind that?
soundness: 3
strengths: Previous shape and textures bias studies rarely considered dense prediction tasks, so this looks like a valuable contribution.

The proposed study is the first that investigates the effect of individual cues (and different combinations of cues) on the training process of the semantic segmentation models.

The paper consolidates a method for extraction of different image cues from natural images. This enables transformation of the original datasets into variants that contain a single image cue or selected combination of cues.
summary: The paper investigates the importance of different image cues (e.g. color, shape, texture) for successful training of deep semantic segmentation models. Different from previous work, this study does not focus on the analysis of conventionally pretrained models. Instead, it is based on "expert" models trained on custom datasets containing only one or a combination of selected cues. To enable this type of analysis, the paper proposes several approaches to transform original image datasets into versions that contain only specific cues. The analysis includes two real image datasets: Pascal Context and Cityscapes, as well as a single synthetic dataset based on CARLA which provides more control over the image generation process. Additionally, the study explores considers convolutional as well as transformer based architectures.
weaknesses: Presentation quality could be improved. 
I suggest placing the tables and figures right after being referenced in the text. 

I found the Texture Cue Extraction paragraph confusing. The main manuscript should include more details and be more descriptive. I am not sure how the Voronoi diagrams are created and do they depend on the content of the corresponding image. Are class frequencies and distributions preserved in this texture cue dataset? 

The last paragraph in 4.1 should also describe the evaluation protocol. What kind of input each expert model considers during the evaluation? This is partly addressed at the end of the first paragraph in 4.2. However, I think it deserves a separate paragraph to clarify the edge cases, so that the reader has no uncertanties when considering the numerical results. I would also consider including the EED and HED results with processing in the corresponding table. 

I am not entirely convinced with the conclusions of the discussion about the influence of the size on the performance of texture and shape experts. The texture expert might specialize in these classes not because the size of their segments, but just because they are more frequent, or their texture is more unique and therefore easier to discriminate. I am not sure. It obviously depends on the properties of the training dataset. I think this requires further considerations.

I feel like the paper lacks some practical takeaways or ideas for future work that would use the findings of this study for improving the performance or robustness of deep models for semantic segmentation.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. When exacting the cue colors, is 1x1 convolution used for all layers or only for the first? In the former case, it is really hard to understand whether the network has enough capacity to learn a difficult task such as semantic segmentation, while the latter case would invalidate the experiment as the following layers would have a larger receptive field.

2. The second part of the sentence at L193: “As it preserves color, it extracts the cues S+V+HS, and analogously to the treatment of the C cue provides the cues S+V as well.” is not clear to me.

3. About weaknesses 3, I wonder whether it is possible to test each cue expert under the scenario it has been trained on, and not on the original images. Maybe the authors could clarify this point and motivate this choice.
soundness: 2
strengths: 1. In my opinion, this paper addresses a very interesting problem, namely, which are the cues to learn from. A thorough analysis could be beneficial for other problems such as explainability, Domain Adaptation, and  Domain Generalization.

2. The paper is well-written and clear, and I particularly appreciate Table 1 summarising all the possible settings.

3. A good number of experiments across different datasets and architectures have been conducted.
summary: In this work, the authors try to answer an important question, which is what can DNNs learn from image cues such as shape, texture, and color. They argue that this is the opposite of what is typically done in previous literature, as previous works rather focus on understanding how much a trained neural network relies on different cues. In short, the aim of this paper is to understand what are the important cues for successful trainings.
To answer this question, the authors propose to decompose a given dataset into multiple different sub-datasets, with each one including one or more cues to learn from. They set up different experiments (cue-experts) from different combinations of cues in the context of semantic segmentation and using both CNNs and Transformers.
The outcome of these experiments is that there is no clear winner between relying on shapes rather than texture is essential for good performance, but rather a combination of shape and colors.
weaknesses: 1. My main concern about this paper is that the results are unclear. The rank change column is particularly helpful in understanding the current scenario, and all datasets clearly have different cue orders when taken individually across datasets. I understand that the authors claim that they do not drastically differ, but still, they are not consistent in drawing clear conclusions. 

2. The claimed results are not surprising, and I do not see a clear impact or the direction of this paper. For example, it is obvious to me that cues such as texture is overall better than color since texture also contains color information, as also stated by the authors. Furthermore, is not surprising that using all cues is better.

3. One final major concern is about the validity of some experiments. In particular, on how evaluation is done for the color cue expert. To my understanding, in such cases, all convolutions have been replaced with 1x1 convolution kernel. These produce very poor results, but we do not know whether this is because of a model that does not have enough capacity or the color is not a useful cue.
Furthermore, I am not sure that it is a valid method to train a model one cue and then test it on the original images. In some cases, this could make sense, but in other cases, the distribution shift between source and target images could be so high that the model is not able to perform well.

Meta-review: This paper studies the importance of different image cues like color, shape, texture, etc., for the learning of deep semantic segmentation models. The manuscript was reviewed by four experts in the field. The recommendations are (2 x "5: marginally below the acceptance threshold", 2 x "6: marginally above the acceptance threshold"). The reviewers raised many concerns regarding the paper, e.g., unclear motivation and statement without in-depth analysis, unconvincing experimental evaluation results, etc. Considering the reviewers' concerns, we regret that the paper cannot be recommended for acceptance at this time. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.

Decision: Reject