Simplicity in Complexity

Authors: tingke  shen
surabhi s nath
aenne  brielmann
peter  dayan
Venue: ICLR_cc_2024_Workshop_Re-Align
Published: 2024-02-01 19:01:49

Abstract: The complexity of visual stimuli plays an important role in many cognitive phenomena, including attention, engagement, memorability, time perception and aesthetic evaluation. Despite its importance, complexity is poorly understood and ironically, previous models of image complexity have been quite complex. There have been many attempts to find handcrafted features that explain complexity, but these features are usually dataset specific, and hence fail to generalise. On the other hand, more recent work has employed deep neural networks to predict complexity, but these models remain difficult to interpret, and do not guide a theoretical understanding of the problem. Here we propose to model complexity using segment-based representations of images. We use state-of-the-art segmentation models, SAM and FC-CLIP, to quantify the number of segments at multiple granularities, and the number of classes in an image respectively. We find that complexity is well-explained by a simple linear model with these two features across six diverse image-sets of naturalistic scene and art images. This suggests that the complexity of images can be surprisingly simple.

Main paper: 
 introduction: 
 The subjective complexity of sensory stimuli plays an important role in many cognitive phenomena, including attention, engagement, memorability, time perception or aesthetic evaluation Kyle-Davidson & Evans (2023) ; Sun & Firestone (2021) ; Van Geert & Wagemans (2020) ; Palumbo et al. (2014) , and is relevant to a wide range of real-world applications such as advertising, web design, and computer graphics Pieters et al. (2010) ; Wu et al. (2016) ; Reinecke et al. (2013) ; King et al. (2020) ; Ramanarayanan et al. (2008) . It is therefore important to understand the factors and mechanisms underlying the perception of complexity. Most empirical and theoretical work concerns artificial or naturalistic images Chikhman et al. (2012) ; Gartus & Leder (2017) ; Nath et al. (2023) ; Machado et al. (2015) ; Nagle & Lavie (2020) ; Guo et al. (2023) ; the latter are the focus of our work.There is by now a range of datasets containing human ratings of the complexity of various subcategories of naturalistic images-we consider RSIVL (RSIVL-RS1) Corchs et al. (2016) , VISC (VISC-C) Kyle-Davidson et al. (2023) , Savoias Saraee et al. (2020) and IC9600 Feng et al. (2022) . Duly, there has then been a number of attempts to predict these ratings, and thereby understand the computations concerned. Note, though, that these methods have hitherto largely been applied on their own, separate, datasets, rather than being directly compared. The methods fall into two broad categories: using either simple (often linear) combinations of handcrafted image features, or modern convolutional neural networks (CNNs) as predictors or feature extractors. We advocate a middle ground, revealing an unexpected degree of simplicity in modelling complexity.For the first category of methods, several qualitative and quantitative image features have been proposed and shown to predict complexity. These include the number and variety of elements, colour, edge density, file size, Fourier slope, HOG and information-theoretic measures such as entropy and information gain Van Geert & Wagemans (2020) . Corchs et al. compiled 11 measures based on spatial, frequency and color properties which were combined linearly to fit perceived complexity ratings on the RSIVL dataset. They found the number of regions, frequency factor and number of colours received the largest weights Corchs et al. (2016) .Equally, Kyle-Davidson et al. proposed measures of clutter (see also Olivia et al. (2004) ; Rosenholtz et al. (2007); Fan et al. (2017) ), entropy and patch-wise symmetry as determinants of complexity, showing good performance on the VISC dataset. The advantage of hand-crafted features is that they are largely interpretable. However, they are often dataset-specific, possibly due to the difficulty of evaluating such rather subjectively-defined measures in general. Perhaps as a result, a large number of these potentially noisy features seem to be required to predict subjective complexity well.More recently, it has become popular to exploit the computational capabilities of deep neural networks to extract relevant image features. Analysis on Savoias dataset, comprising of 1400 images across 7 categories showed that activations from intermediate layers of a CNN pretrained on object or scene recognition correlated best with human complexity ratings Saraee et al. (2020) . These authors also compared unsupervised and supervised methods, suggesting that supervision can improve prediction.Feng and colleagues built further on this work, first by introducing a large-scale visual complexity dataset comprising on 9600 images across 8 semantic categories, and then providing a CNN-based method predicting scores and activation maps Feng et al. (2022) . This model achieved high test performance, outperforming previous methods.However, although such CNN-based models perform well, and can even generalise competently to unseen images, they are hard to interpret (as activation maps do not convey much information, and can also be unreliable Bilodeau et al. (2024) ) and do not guide a theoretical understanding of the problem.Here, we benefit from both categories of methods. We use modern foundation models Bommasani et al. (2021) to evaluate particular hand-crafted features in a way that generalizes across many classes of images. We then combine these features linearly to predict complexity.To choose hand-crafted features, we start from the observation that features that fragment images in meaningful ways tend to estimate complexity relatively well (for example, clutter in Kyle-Davidson et al. (2023) or the number of regions in Corchs et al. (2016) ). We therefore leverage the capabilities of state of the art (SOTA) image segmentation models to extract relevant segments from the image at multiple spatial granularities. Such models are the closest existing approximations to how humans represent scenes for two main reasons: first, the models are trained using a vast amount of annotations from several humans and hence reflect relevant inductive biases, and second, the architecture of CNNs and transformers are inspired by the human visual processing systems and generalize surprisingly well to unseen images. With the help of such models, we obtain semantically consistent segments at different spatial granularities relevant to perceptual image processing Epstein & Baker (2019) . We then derive from them the core components of perceived complexity.With improved quality of feature extraction and evaluation, we make the central observation that only few features are necessary to predict complexity well, justifying the claim that complexity can be surprisingly simple. 
method: 
 We develop a parsimonious model of the perceived complexity of naturalistic images using two types of segmented features, namely the number of segments, and the number of named classes, extracted from SOTA segmentation models. Our method is described in Figure 1 . We also use an additional measure called patch-symmetry (borrowed from Kyle-Davidson et al. ( 2023)) to address a main failure mode of our model. 2020), containing 1400 images across 7 categories, and IC9600, containing 9600 images across 8 categories Feng et al. (2022) . We use the mean subjective complexity per image across raters (there were between 10 to 26 raters per image across datasets) as ground truth. We restrict to scenes and art image categories and omit advertisement (Savoias and IC9600) and visualisation (Savoias) categories since they contain substantial amounts of text. We combine similar image categories within a dataset to generate 8 image-sets for analysis: (1) RSIVL, containing all RSIVL-RS1 images; (2) Savoias Scenes (Sav. Scenes), comprising of Savoias scene and object categories; (3) IC9600 Scenes (IC9. Scenes), comprising of IC9600 scene, object, person, transportation and architecture categories; (4) Savoias Art (Sav. Art); (5) Savoias Suprematism (Sav. Suprematism); (6) IC9600 Paintings (IC9. Paintings); (7) VISC, containing all VISC-C images, and lastly (8) Savoias Interior Design (Sav. Int), which is considered separately as it contains software-generated 3D-rendered images.We extracted segments in images using the SOTA Segment Anything Model (SAM) Kirillov et al. (2023) . SAM detects blobs of segments in an image at different scales. SAM was trained on the largest public segmentation dataset to date, is capable of zero-shot generalization, and achieves SOTA performance. Based on pilot studies, we set the spatial granularity parameter points-perside to 64. This allowed the network to find finer segments, and correlated well with ground truth complexity. We set all other parameters of SAM to their default values and evaluated the total number of detected segments per image (num seg).We found the nameable class instances in an image using FC-CLIP Yu et al. (2023) . FC-CLIP is an open-vocabulary panoptic segmentation algorithm that can find multiple instances of each class, and achieves SOTA performance Yu et al. (2023) . We use panoptic semantic segmentation to predict classes because multi-scale methods like Semantic SAM Chen et al. (2023) produced many false positives. We set all parameters of FC-CLIP to default and evaluated the number of detected classes (including repeated classes) per image (num class). Intuitively, FC-CLIP finds the most salient, lower granularity semantic classes in the image while SAM finds sub-components of these classes at higher granularities. As a result, num seg is larger than num class for all images.We estimate subjective complexity using multiple linear regression. A preliminary examination showed that subjective complexity scales roughly linearly with √ num seg and √ num class, hence, we apply a square-root transformation to our features. We used the statsmodels OLS function in Python to fit multiple linear regression on each image-set. We perform 3-fold crossvalidation M times, where M is larger for smaller image-sets, and report the average Spearman correlation over all test sets. We compare our models to six baselines from previous work. These baselines include three handcrafted feature-based baselines-two from Corchs et al. ( 2016  
excellent performance on natural scenes and art: 
 Table 1 shows the performance of our models and baselines for 6 image-sets. We see that our linear model with √ num seg and √ num class attains a Spearman correlation between 0.73 to 0.89 with human complexity judgments across natural scenes and art image-sets. Notably, our model performs better than all handcrafted feature baselines, the transfer-learning neural network method from Saraee et al. (2020) and the supervised neural network from Kyle-Davidson et al. (2023) .It performs similarly to the supervised neural network from Feng et al. (2022) . The exceptions are the test datasets from the same paper and Savoias Art. This neural network directly learns a high-dimensional mapping from image to complexity, thereby discovering features that best predict complexity in a supervised way. We show that in many cases, this high-dimensional relationship can be distilled down to simply the number of segments and named instances in the image. Hence, we provide evidence that complexity is computable from segmentation features, rather than requiring features that are explicitly optimized for complexity.We also compared the full model with versions restricted to just one of the √ num seg or √ num class terms. We see that both terms contribute to the variance explained for all datasets except Savoias Suprematism, where √ num class fails to explain additional variance on top of √ num seg. This is because Savoias Suprematism contains abstract art images with geometric shapes, and FC-CLIP fails to find appropriate nameable classes as in its training set. However, for Savoias Suprematism, the model with only √ num seg already explains high variance and achieves performance superior to all other models, suggesting that the number of segments at multiple granularities drives perceived complexity in images composed of geometrical shapes that lack overt semantics (at least for artnovice raters).Figure 2 shows the images with the highest and lowest predicted complexity from each of the 6 image-sets in Figure 3 shows the mean and standard deviation of the ground truth subjective complexity for images for each bin of √ num seg and √ num class on an example image-set, IC9600 Scenes. In general, and as expected, mean ground truth complexity increases with increasing √ num class and increasing √ num seg (also well-matched to predictions) showing that a complex image is one with both a large number of segments and classes. The standard deviation of the subjective complexity, which contributes markedly to the prediction error, is particularly high in the bins with the greatest and least √ num class. This suggests that FC-CLIP might over-or under-predict classes. The largest discrepancies lie in the bin with the highest √ num seg and the lowest √ num class. Here, FC-CLIP often fails to find any nameable segments at all. The statistics of segments and classes at multiple granularities explain most of the variance in the datasets we tested. However, the structure in the image, i.e., the spatial and functional relationships between elements is also known to be an important contributor to complexity (Chipman (1977) ; Ichikawa (1985) , Gestalt theory of perception). Indeed, we find that segment statistics alone are not enough to adequately explain complexity judgments in two other image-sets: VISC and Savoias Interior Design. Figure 4 shows an example from each dataset with the highest prediction errors. In each case, our model over-predicts complexity because SAM finds too many segments without accounting for the fact that many segments are arranged in a spatial pattern (books in the top row and windows in the bottom row). Further, num class does not contribute to reducing complexity in such cases, either because it is also high (since the uniqueness of classes is not accounted for), or because the weight of the num class term is learned to be low (for example in VISC) We see that both of these images have high patch-symmetry (defined in Kyle-Davidson et al. ( 2023)). Figure 5 illustrates a significant, positive correlation between patch-symmetry and model error (prediction minus ground truth), showing that our model tends to over-predict when the image is more spatially symmetric, i.e. has more spatial structure. Table 2 shows that when patch symmetry is added as a feature to the regression, our model improves in Spearman correlation by atleast 0.12, and becomes competitive with most baselines. Int. Model overpredicts when patchsymmetry is high. Linear regression reveals a significant Pearson correlation of 0.51.Sav.Int.Corchs  
discussion: 
 We presented a linear model of complexity using two features extracted using SOTA segmentation neural networks: num seg and num class. Our model outperforms most baselines achieving a Spearman correlation between 0.73 to 0.89 with subjective complexity ratings across six tested image-sets of naturalistic scenes and art. As a result, our model provides a simple explanation of perceived complexity that generalizes across multiple domains and image types. Our results suggest that segment-based representations are good proxies for the cognitive processes underlying human judgments of complexity, a result that could be extended to attention, memorability, aesthetic evaluation, etc.Our model performs better than all handcrafted feature-based baselines. A possible reason for this is that num seg is a significant improvement over previously suggested features for image fragmentation (such as "number of regions" from Corchs et al. ( 2016)) that approximate the segments in an image. Further, to our best knowledge, we are the first to exploit named-segments corresponding to semantic classes to predict complexity, whose count provides an estimate of the number of lower granularity segments in an image.Importantly, the segments and classes are both computed by neural networks trained on large datasets of human annotations. Therefore, the predictions are likely to be semantically meaningful, reflecting not only pixel information but also the annotator's prior experiences with the contents of the images in the training set. The annotations and hence segments also encompass multiple levels of spatial and semantic granularity, capturing contributions to complexity across scales. This is in contrast to past works that have tried to approximate spatial granularity and semantic variety using only sliding windows or pyramid scaling of filters Corchs et al. ( 2016 Our model performance was generally comparable to the supervised neural network of Feng et al. (2022) , which was trained on a large dataset to directly predict complexity. The difference in performance can be explained by the fact that the neural network potentially utilizes many more than two features and conditionally chooses those features based on the context and distribution of images. However, we have shown that only two features can explain complexity equally well on multiple datasets and domains, elucidating a simpler view of complexity.In addition, unlike the CNN models, our model is highly interpretable. As we demonstrate in Figure 4 , we can attribute predictions or diagnose failure cases by visually inspecting the outputs of SAM and FC-CLIP. Also, the contributions of the segments and classes to the complexity score can be clearly elucidated (as the square root of their counts).However, our model has limitations. The accuracy of our model depends on the accuracy of the segments and classes predicted by SAM and FC-CLIP. Currently, SAM is incapable of detecting thin, "one-dimensional" patterns. FC-CLIP sometimes misses salient classes or repeated classes (failing to predict any classes for some images outside its training distribution, e.g. images in Sav. Suprematism) and doesn't predict nested classes at multiple granularities (e.g. both the "house" and its "window"). As the SOTA segmentation models improve, we expect the performance and interpretability of our model to also increase further.We also addressed the inability of num seg and num class to account for structure in an image which reduces perceived complexity. We saw that adding patch-symmetry to the regression led to competitive performance on VISC and Sav. Int image-sets. However, as part of future works, we aim to build a more parsimonious model using a segment-based feature of structure. For example, scene-graphs Chang et al. (2021) or generative programs Sablé-Meyer et al. ( 2022) can be used to organize the named entities detected by FC-CLIP by their spatial and semantic relationships, and image complexity can be derived from the complexity of these representations, for example as their compressibility Mahon & Lukasiewicz (2023) ; Dehaene et al. (2022) ; Karjus et al. (2023) .We modeled subjective complexity ratings which was the mean rating across multiple raters. However, complexity judgments are known to vary across both individuals or groups (age, cultures, etc.) Gartus & Leder (2017) . For the art datasets, the complexity ratings were given by art-novices and would likely differ significantly from ratings of art-experts Pihko et al. (2011); Bimler et al. (2019) . These individual differences could be caused by differences in the segments people perceive (the regions of an image they consider to be part of the same segment). For example, different individuals may segment at different granularities. Individual differences can also be caused by the mapping from the perceived segments to complexity. Explicitly accounting for individual variability using subject-specific data (for example by fine-tuning the regression or segmentation models) will be an important part of future work.In conclusion, we develop a parsimonious and interpretable account of human perceived complexity in naturalistic images using segmentation-based methods, showing that complexity can be surprisingly simple given the right image representations. 
 
 Appendix:  



Reviews: review: title: title: Ignored complexity of model, simplicity in insight
main_review: main_review: The authors critique the inadequacy and dataset-specific nature of handcrafted features, as well as the opacity of deep neural network models that have been previously used to gauge complexity in pursuit of developing a simpler model. They propose, what they describe as a a simpler approach, using segment-based representations through advanced segmentation models—SAM and FC-CLIP—to evaluate image complexity. By quantifying the number of image segments and classes, they demonstrate that a simple linear model can effectively capture complexity across varied sets of natural and artistic images, challenging the notion that understanding image complexity requires complex models.

This is an interesting attempt to operationalize "visual complexity". The author's approach is fairly simple, but uses very complex segmentation algorithms. Saying that the new model is simple (given they are derived from SOA SAM and FC-CLIP, which are highly complex) is rather misleading. I am guessing that simpler segmentation models do not do as well? This would be a direction to explore further. 

The central insight that I drew from this work is that whatever human participants mean by complexity can be approximated by the ~number of objects (and segments) present in an image. 

Another interesting direction could be the development of better evaluation benchmarks of visual complexity (decorrelated from number of objects) leveraging the failure models of the current models. 

I do appreciate the authors delving deeper into their method and identifying failure models (Figure 4).
strength_weakness: strength_weakness: Rating wise, I think this a good paper for the meeting. But it does not present a particularly novel approach to warrant a talk? I did enjoy reading the study.
title: title: Review: Simplicity in Complexity
main_review: main_review: The authors of this paper introduce a novel method for assessing human-perceived complexity in naturalistic images. This approach employs an image segmentation model, specifically SAM, and an open-vocabulary object detection model, FC-CLIP. The authors demonstrate that a linear combination of the square root of detected segments and the square root of detected classes can serve as a measure for image complexity. The authors show that this approach performs exceptionally well, even achieving top performance on several datasets. A key feature of the proposed approach is its interpretability, as the estimated score can always be traced back to the number of found segments or number of classes.

 - *Clarity*: The paper is well-structured and easy to comprehend. The authors provide a comprehensive overview of related works, and the description of the methods is lucid.
 - *Correctness*: The paper’s claims are substantiated by evidence from several experiments and comparisons.
 - *Novelty*: The proposed method exhibits novelty.
 - *Interest to the Community*: The proposed approach demonstrates that human-perceived complexity can be estimated using image segmentation and object detection models. The authors also address the role of symmetry and structure in human complexity perception. Although this work is somewhat outside the general scope of representation alignment, I believe it will be of interest to the community.

A few comments:
1. It would be beneficial to report the impact of the points-per-side parameter in the SAM model, as it limits the number of potential segments. Given that this might be crucial for image complexity estimation, it would be interesting to observe the changes in the performance of the proposed approach when varying this parameter.
2. For the estimation of the number of classes, are only unique classes counted, or are classes for all detected objects included?
3. From the writing, it remains somewhat unclear whether the linear regression model is retrained on each dataset or is general across all datasets. Additionally, I believe it is important to state the linear coefficients to illustrate which term is more significant to the complexity estimation.
strength_weakness: strength_weakness: Discussed in review.
title: title: Interesting finding suggests simple explanation for visual complexity, more connection to the workshop theme might be needed
main_review: main_review: Summary:
This paper investigates an important question, namely the factors and mechanisms underlying subjective complexity of sensory stimuli. The paper offers a very simple explanation to model visual complexity, namely segment-based representations of images. They show complexity of visual stimuli (human ratings) can be well predicted by a simple linear model using information such as number of classes and number of segments extracted from segment-based representations using segmentation models such as SAM and FC-CLIP. A method falls in between using combinations of handcrafted image features and using uninterpretable feature maps out of convnets.

I thought the finding is novel in that it suggests segment-based representations might be good proxies for the cognitive processes underlying human judgments of complexity. However, what prevents me recommending the paper in its current form from acceptance at ReAlign is its lack of clear engagement to the central theme of the workshop, namely the alignment between artificial and biological systems. I believe the finding itself is of relevance as it helps to understand what sort of representations could be at play under perception of complexity in the human brain, I believe more discussion is needed as to what implications are there in terms of understanding alignment.
I find some of the motivations and justifications for the choice of the SOTA segmentation models to understand biological representations a little confusing and probably need more clarification. For instance, I am not sure if SOTA segmentation models like SAM can be categorized as “the closest existing approximations to humans represent scenes” for using CNNs/Transformers underneath. Yes CNNs are shown to correspond to human visual systems but transformers are not and certainly not inspired by visual processing. More importantly, SAM underneath is a mixture of ViT, CLIP and transformer decoders which is a beast at creating good image segments, but it feels a bit of a stretch to me that it is justified to say SAM aligns with how humans perceive visual scenes and create segments. In fact, doesn’t it suggest the opposite that so long as you have a powerful segmentation model to produce good segments, they can be linearly combined to predict complexity ratings? We probably need evidence (not suggesting it should be done in the current work!) supporting that segments like these are indeed present in biological representations in the brain. But I don’t want to take the credit away from this work. It has certainly offered an interesting start in this direction.

I find the writing of the paper could be improved especially for readers who aren’t neck deep into the exact tasks, models and datasets presented in this work. High-level overviews of the baseline models for instance could be expanded a little more to give readers more intuitions as to what the authors are comparing against. At times I find the results are not clearly connected to figures. For instance, “We show that in many cases, this high-dimensional relationship can be distilled down …” -- where is the figure supporting this? Though the result as stated itself is interesting.
strength_weakness: strength_weakness: N/A