Learning from Pattern Completion: Self-supervised Controllable Generation
Authors: Zhiqiang Chen
Guofan Fan
Jinying Gao
Lei Ma
Bo Lei
Tiejun Huang
Shan Yu
Venue: NeurIPS 2024 poster (2024)
Keywords: Self-supervised Controllable Generation，Modularization，Pattern Completion，Associative Generation
Primary area: generative_models
TL;DR: Inspired by the neural mechanisms that may contribute to brain’s associative power, specifically the cortical modularization and hippocampal pattern completion, here we propose a self-supervised controllable generation (SCG) framework.

Abstract: The human brain exhibits a strong ability to spontaneously associate different visual attributes of the same or similar visual scene, such as associating sketches and graffiti with real-world visual objects, usually without supervising information. In contrast, in the field of artificial intelligence, controllable generation methods like ControlNet heavily rely on annotated training datasets such as depth maps, semantic segmentation maps, and poses, which limits the method’s scalability. Inspired by the neural mechanisms that may contribute to the brain’s associative power, specifically the cortical modularization and hippocampal pattern completion, here we propose a self-supervised controllable generation (SCG) framework. Firstly, we introduce an equivariance constraint to promote inter-module independence and intra-module correlation in a modular autoencoder network, thereby achieving functional specialization. Subsequently, based on these specialized modules, we employ a self-supervised pattern completion approach for controllable generation training. Experimental results demonstrate that the proposed modular autoencoder effectively achieves functional specialization, including the modular processing of color, brightness, and edge detection, and exhibits brain-like features including orientation selectivity, color antagonism, and center-surround receptive fields. Through self-supervised training, associative generation capabilities spontaneously emerge in SCG, demonstrating excellent zero-shot generalization ability to various tasks such as superresolution, dehaze and associative or conditional generation on painting, sketches, and ancient graffiti. Compared to the previous representative method ControlNet, our proposed approach not only demonstrates superior robustness in more challenging high-noise scenarios but also possesses more promising scalability potential due to its self-supervised manner. Codes are released on Github and Gitee.

Learning from Pattern Completion: Self-supervised Controllable Generation
Zhiqiang Chen 1.2%; Guofan Fan"*, F Jinying Gao Lei Ma¹, Bo Lei',
Tiejun Huang!,, and Shan Yu
Beijing Academy of Artificial Intelligence
Institute of Automation, Chinese Academy of Science 'Xi' an Jiaotong University * University of Chinese Academy of Science Peeking University
Abstract
The human brain exhibits strong ability spontaneously associate different visual attributes of the same or similar visual scene, such as associating sketches and graffiti with real-world visual objects, usually without supervising information. In contrast, in the field of artificial intelligence, controllable generation methods like Net heavily rely annotated training datasets such as depth maps, semantic segmentation maps, and poses, which limits the method's scalability. Inspired by the neural mechanisms that may contribute to the brain's 's associative power, specifically the cortical modularization and hippocampal pattern comple- tion, here we propose self-supervised controllable generation (SCG) framework. Firstly, we introduce an equivariance constraint to promote module indepen- dence and intra-module correlation in a modular autoencoder network, thereby achieving functional specialization. Subsequently, based on these specialized mod- ules, we employ self-supervised pattern completion approach for controllable generation training. Experimental results demonstrate that the proposed modular autoencoder effectively achieves functional specialization, including the modular processing of color, brightness, and edge detection, and exhibits brain-like features including orientation selectivity, color antagonism, and center-surround receptive fields. Through self-supervised training, associative generation capabilities sponta- neously emerge SCG, demonstrating excellent zero-shot generalization ability various tasks such superresolution, dehaze and associative conditional generation on painting, sketches, and ancient graffiti. Compared to the previous representative method ControlNet, our proposed approach not only demonstrates superior robustness more challenging high-noise scenarios but also possesses more promising scalability potential due its self-supervised manner. Codes are released on Github and Gitee)
Introduction
In contrast to artificial intelligence, the human brain exhibits remarkable characteristic: the sponta-
neous emergence of associative generation 17 23 18] such as associating real- world visual scenes through pictures, sketches, graffiti, and (Figure Ta). With the development of controllable
"Equal Contribution.
i Work done during Internship at Beijing Academy of Artificial Intelligence. Corresponse author. X chenzhigiang @ mails.ucas weraaaac.cc..o ac.,s.hhnnnnnnnnn.....a...aaaae, @ nlpr.ia.ac.em.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Brain Model of Pattern Completion
Complete input
Incomplete input
modullas
Pattem Completion in Human
Completion
Hipposampus
Saff-supervise--trainedd Functional Column
Pattern Completion Module
Modality A
Modality B
Learning Module
Modality C
Figure 1: Framework of SCG. SCG has two components. One is to promote the network to spontaneously specialize different functional modules through our designed modular equivariance constraint; The other is to perform self-supervised controllable generation through pattern completion.
generation technology 59 54]. many excellent works have recently been able to achieve similar
associative generation or conditional generation capabilities, such as ControlNet(55), However, existing controllable generation models often require supervised training to achieve this function, such as edge maps (53,, semantic segmentation maps (32,, depth maps (31, and pose (50).
The brain's 's remarkable ability to generate associations emerges spontaneously and likely stems from two key mechanisms. One the brain's 's modularity in terms structure and function enabling to decouple and separate external stimuli into distinct patterns. This modularity is essential for a range of subsequent cognitive functions, including associative generation. Although previous works such as group convolution (57 333, capsule networks 43 21], and mixture of experts models [60] 42) have designed modular grouping structures, and often show good advantages in model parameter efficiency and computational efficiency, they cannot spontaneously specialize reliable functional modules. Another is the pattern completion ability hippocampus/455 (Figure [[]), such associating the visual signal of pepper with its taste, associating sketch with real visual scene, etc. If we regard edge maps, semantic segmentation maps, depth maps, etc. as general modalities or patterns, then existing controllable generation works in artificial intelligence can also be regarded as kind of pattern complction pr Due to the lack of the brain's 's ability to spontancously and reliably specialize functional modules [10, controllable generation often adopts supervised approach. It brings two problems: One is the need for manually defined supervised conditions; The other the need for large amount of annotated data. This supervised approach may limit the scalability of associative generation, which is one of the most important capabilities in the era of large models [3 39 19]. Therefore, it is meaningful enable the network spontaneously specialize functional modules and perform pattern completion for controllable generation in a sule--suee manner
To mimick functional modularity 35 5 of brain, we incorporate an equivariance constraint 4
12
into the training process of an unsupervised modular autoencoder to enhance the correlation within module and the independence between the modules, thus promoting the functional specialization.

---

Experiments on multiple datasets including MNIST and ImageNet demonstrate that the proposed equivariance constraint can enable each module to form closed and complete independent sub- manifold, resulting in significant and reliable functional specialization similar to biological systems, such as orientation, brightness, and color. As shown in Figure D.. with the spontaneously special- ized functional modules as conditions, we train self-supervised controllable generation (SCG) adapter based on a pretrained diffusion model to achieve pattern completion and generate images. Experimental results on the COCO dataset show that SCG achieves better structural similarity and semantic similarity compared to ControlNet. At the same time, SCG also has excellent zero-shot generalization in associative generation tasks ranging from oil paintings, ink paintings, sketches, and more challenging ancient graffiti. The generated images both have similar contents and structures to the original images and rich and vivid generated details. Especially in the subjective evaluation of associative generation of oil paintings and ancient graffiti, compared to previous excellent work Nen SCG has higher similar winning rate in fidelity and significantly higher winning rate in aesthetics.
Related Works
Controllable Generation:
Controllable generative models enhance the generation process by in- corporating control conditions(59) (4), leading to images that more closely align with desired outcomes. Compared to using text as control conditions [2] 25 28, 26 49 37). image-based con- trol conditions 7 36 55 58 34] afford finer-grained controllability. which often need annotated paired images for supervised training. While some methods leverage structural similarity for addi- tional constraint 2 61, enabling style transfer[41] while maintaining structural consistency, these approaches typically rely on GANs [66 rather than stronger diffusion models 22
Modular Neural Networks:
The most direct approach to modular network design is structural modularity, achieved by techniques such as dividing convolutional kernels into distinct groups within
each layer 57
33 56 45 or designing specialized blocks [7 48 Capsule networks 21 40 27
and Mixture of Experts (MoE) [60] 42) both adopt dynamic routing mechanism between diverse
blocks. These methods often improve parameter and computational efficiency but struggle to achieve reliable and stable functional specialization. Group equivariance [2] 44 [22 51 24 9 offers novel approach for functional specialization. By imposing equivariance constraints, the network automatically learns crientation-selective features akin to simple cells in the visual cortex, resulting in clear functional specialization [15]. However, this approach is limited to translational equivariance.
3 Self-supervised Controllable Generation
The whole framework of our Self- self-supervised Controllable Generation (SCG) is shown in Figure De. This framework comprises two components: a modular autoencoder and a conditional generator for pattern completion. We first train special autoencoder to encode the input image into disentangled feature spaces or modalities called Modular Autoencoder. Then we use part of the modules as control conditions to complete the missing information and reconstruct the input image. This self- supervised approach does not require mamually designed feature extractors or manual annotations, making it casier to train and more scalable compared to supervised approaches. Notable, given the extensive research on conditional generation, we leverage the existing, mature ControlNet(55) for pattern completion part. Our core contribution lies in designing modular autoencoder based on proposed equivariance constraint, successfully enabling the network to spontaneously develop relatively independent and highly complementary modular features. These features are crucial for subsequent conditional generation.
The main principle of the modular autoencoder is to enhance the indepen- dence between modules and the correlation within modules. Inspired by the primary visual cortex of biological systems, we design a modular equivariance constraint to promote functional specialization. Specifically, for a typical autoencoder, can be formally represented as:
Modular Autoencoder:
E(I) f. D(f) =
where I is the input image, f is the latent representation, E and D are the encoder and decoder, respectively. For modular autoencoder, we define the independence between modules as:
f = (1)) ( (".,,.. ((k-")),, (0)
= (1).
(2)
3

---

M(D)
M01
Lequ
Laymn
f
M
Mr (6mm)
X
X
f
(K-1)
G W
La())
n (
W
G W
(2-
Figure 2: Detail architecture of proposed Modular Autoencoder. The latent space is divided into several modules. We use a prediction matrix M (6) to build relationship on latent space between input image pairs. (+) is the learnable codebooks for each modules.
can be divided into several modulars a general transformation determined by a low dimension parameter 5. Ps a prediction function in the latent space with parameter d. It means that each module independently predict its state by the specific change 6, which indicates that each module is equivariant on the corresponding change group. For the correlation within module, we define
f where f()) is the ith modular. Ls is
P(.)) (f(())
where <(0 and fm denote one-hot representations with I in the mth and nth dimensions, respectively, For any two dimensions f/ and f(() in f()),, there always exist corresponding prediction parameters 5% such that above equation holds.
(i))
In practical implementation, use simple translation and rotation transformations the change Lg, and learnable linear transformation matrix ()) as the prediction function.
Equivariance constraint:
As shown in Fig. 2 we utilize convolution operation as the encoder E with kernel W. We train the autoencoder on image pairs and I', where I' (I)... The image encoding process is denoted as f = I * W and I' = I' W, where W is learnable convolutional kernels, denotes convolution, and f and I'' represent the extracted feature maps. The decoding process is f @ and I' @ W. where W shares kernels used in the encoding process, and denotes deconvolution. The objective of the autoencoder is to make the reconstructed image as similar as possible to the original image. So we design the reconstruction loss as follows:
I Trecon |[ʃ & W 7112 + II W 1'112
Then, we construct equivariant loss drive modules in the networks to be more independent. We divide the convolutional kemnels W into k groups: W = (w) W(1) ...., W(k-1)), and the
corresponding feature maps / are also divided into h groups: = [f(0), r(i) (k-1)). Within each module, we apply an equivariance constraint such that
Legu Σ - (6) iE (0, 1, 10.,,...,A -1
(5)
where (8) is a learnable prediction matrix determined by 8. Specifically, 60 is a 3D matrix codebook indexed by (linear interpolation), where å is composed of two translation parameters fx

---

where is the module length, and Mi (6) is the row and column of the prediction matrix, which indicates the relationship between mthh and nth dimensions module f() corresponding to d. s is the convolution stride. 8(1)" is the best transformation parameter from nth dimension to mith dimension Self-generated visual functional column
Brain-like functional specialization
Figure 3: Feature Visualization of modular autoencoder. Each panel shows all features learned by an individual model with multiple modules (one module each row). We trained modular autoencoder with translationrrotatio equivariance constraint a)MNIST and b)mmageNet, respectively. c) On ImageNet, we also train an autoencoder with an additional translation equivariance constraint besides the vanslation-rotatio equivariance constraint each module. d) We visualize reconstructed images by features of each module in
and by and one rotation parameter To. Next, we construct a symmetry loss to enhance the relationship within each module:
(O/T
8()) Σ
8 ES e C-M (O)/T * 8,,, n E {0, (0.,,...,, 1}, 11
6,6' = (tz: ty. ro); to E 0.5ss 0.5s). [0, (T))
(6)
Laym ΣΣΣ (1 M (5 iE {0, 1, 1,..., 1}
calculated by softmax function with temperature T. p(t) and f((²) are two one-hot representation with I in the moth and noth dimensions, respectively. The goal of the symmetry loss function is to
make any two dimensions of the same module symmetric with respect to a certain transformation -(t)
Finally, we take above three losses together as our equivariance constraint:
LEC
Lrecon + byqu + Ag
where A₁ and A2 are the weights of the equivariant loss and symmetry loss, respectively.
Pattern Completion: Based on the modular autoencoder trained in the previous section, we can treat each module general modality or pattern and then perform conditional generation through pattern completion. This process can be formulated as:
== ())
(8)
where f(() is ith module representation of J. The pattern completion is learning completion function to complete the whole representation f. If we have a process of module specialization, then we train completion function in self-supervised manner, similar how humans do.
In practice, we implement the pattern completion process based on the modular autoencoder designed in the previous section, referring to the existing outstanding controllable generation work NontrolNet.

---

Specifically, if we use module f as control condition, then the overall learning objective of self-supervised controllable generation based on ControlNet can be expressed as:
MC = B
- <o(2z, L, Ct, 1(
20 is the latent representation for the input image and 2ς is the noisy image at time step t. C2 is the text prompt, and f is a module of the pre-trained modular autoencoder. Image diffusion algorithms learn network Eş to predict the noise e. The generation process conditioned by parts of the module's information can be regarded as a pattern completion process. Thus, we achieve self-supervised controllable generation through automated modularization and pattern completion process.
4 Experiments and Results
Modular Autoencoder: We trained our modular autoencoder mainly on two typical datasets. One is MNIST, which is a small dataset with gray digits. The other is ImageNet, which is a relatively large dataset with color natural images. On both MNIST and ImageNet, modular autoencoder can achieve obvious function specialization as shown in Figure 3 Many learned kernels have significantly gabor-like orientation selectivities as shown in Figure 3 and b. which is a brain-like characteristic. On datasets consisting of gray images, such as MNIST, modules primarily specialize into selectivity for different spatial frequency orientations, with each module having the same spatial frequency and complete orientation selectivity (Figure (l)) On datasets consisting of natural color images, modules not only specialize into selectivity for different spatial frequency orientations but also specialize into modules that are sensitive to color and brightness (Figure 3p. Unlike imposing translation equivariance constraints on each kernel, Figure[Bp imposes both translation equivariant and trans-rot equivariance constraints on the entire module. This allows it to learn brain-like center-surround receptive fields, and generate center-surround antagonistic and color-antagonistic modules in A.3.3) which are considered to be more robust to noise. Similarly, Figure 3 also forms obvious functional specialization such as color, brightness, and edges, which can also be observed from the reconstruction visualization of each module in Figure 31. See more experiment, training setup, dataset and ablation study details in Appendix A.4.1, A.4.2, A.4.3 and A.2.
Self-supervised Controllable Generation on MS-COCO(30): We implement our SCG based on our trained modular autoencoder in Figure 3p (see more details in A.4.2). Specifically, we divided the 16 modules into 6 groups named from HCO to HC5 (see details in I. and generated images conditioned by them respectively as shown in Figure 4 The original images with text prompts are randomly selected on MS-COCO in the first line. The condition images are in the second line. In Figure some modules provide the color information, some provide the brightness information, and some provide the edge information with different spatial frequencies. The last condition image the edge map extracted by the Canny detector, which is used in ControlNet. The third line is the generated images by different conditions. With incomplete control information input, the diffusion model Icarns to complete the missing information to obtain a complete image. Due to the different control information, the generated images also have their own characteristics. The bottom shows more generation results. The three lines are original images, condition images, and generated images, respectively. HCO mainly extracts color information, and the generated image is also closely similar to the original image. HC1 provides brightmess information but lacks color information. The generated image is recolored with similar brightness structure the original image. HC2 and HC3 provide structural information such as edge and corner but lack color, brightness and detail information. The corresponding generated images can recolor images and regenerate many vivid and reasonable details. A quantitative analysis is performed on the validation set of MS-COCO in Table
I We use two traditional similarity metrics PSNR and SSIM and two semantic-orienter metrics FID and CLIP to measure the similarity between the generated image and the original image. For PSNR and SSIM, we measure it in gray and color images, respectively. As in Table I in terms of the traditional metrics PSNR and SSIM, the proposed SCGs conditioned by the autonomously differentiated modules all achieve higher similarities than ControlNet conditioned by artificial Canny edge and show a decreasing trend from HCO to HC5. In the gray images that does not consider color information, the image generated by HCI that provides brightness information is higher than that generated by HCO that provides color information in terms of PSNR and SSIM similarity. In terms of semantic similarity, HCO is the best in FID and CLIP similarity, and shows a gradual decrease in similarity from HCO to HC3, and all of them are better than the similarity of ControlNet. It also

---

clock that hanging underneath glass arch.
SCG
Condition
Original
Generation
O
SCG(HCO)
SCG(HCI)
SCG(HC2)
SCG(HC3)
Figure 4: Images generated by SCG in MS-COCO. The upper part shows an image randomly selected in MS-COCO with text prompt. On the right show the condition images extracted from
modular autocoder and the corresponding generated images. The last column is a generated image by
ControlNet conditioned by the canny edge. The bottom part shows more generated images. The three
row images are original, condition and generated images, respectively.(See more in Figure S7 and S6
Table 1: Qualitative evaluation MS-COCO. means that higher better, and means the opposite. and means gray images and color images, respectively.
80
ControlNet
SCG
Winiate (%)
60
SCG
40
Metrics
ControlNet
HCO
HC1
HC2
HC3
HC4
HC5
20
PSNR(g)
10.7
19.2
19.9
17.9
14.8
11.5
11.2
PSNR(C)
10.3
19.0
16.8
15.6
13.7
10.8
10.7
0
Fidelity
Aesthelics
SSIM
0.125
0.486
0,519
0.445
0.352
0.191
0.177
SSIM(c)
0.128
0.490
0.443
0.388
0.320
0.179
0.170
Figure 5: Subjective evalua-
FID 20
13.7
8.5
11.0
11.2
12.1
17.1
14.9
tion on zero-shot oil painting
CLIP
0.846
0.958
0.898
0.890
0.878
0.812
0.830
accociation generation.
demonstrates that the information provided by HCO to HC3 can better reflect the semantic structure than HC4 and HCS, the generated images are also more semantically similar to the original images.
Conditional Associative Generation
Associating real-world scenes from different styles of paintings and abstract graffiti is capability that everyone possesses. And this ability does not require supervised training, self-emergent and has zero-shot generalization ability. The proposed SCG can also emerge with conditional and association generation ability. We use SCG trained on COCO dataset to test its zero-shot generalization capabilities conditional and associative generation with sketches, oil paintings, wash and ink paintings and more challenging ancient graffiti on rock.
completely
Sketches: We first test it on manual sketches. We tested both ControlNet conditioned by Canny edges and SCG conditioned by HC3. In Figure ControlNet can generate images from sketches in excellent fidelity and aestheties to the original sketches. Basing the Canny edge detector,

---

Prompt: High-quality, extremely detailed, 4K, HQ, turtle.
Prompt:
High-quality, extremely detailed, 4K, HQ, hot air balloon.
ControiNet (Canny)
SCG
(HC3)
Figure Association generation on manual sketches. The original sketches are from ControlNet (55
Prompt:
Prompt:
Prompt:
High-quality, extremely detailed, 4K, HQ, tree
High-quality, extremely detailed, HQ. blucherry.
High-quality, extremely derailed, 4K, HQ,
Controltiet [Canny)
SCs (HC1)
SCG
Figure 7: Associative generation on oil painting (top) and wash and ink painting (bottom). (See more generation results in Figure S10 and Figure
ControlNet possesses a natural advantage when dealing with sketches, resulting in high-quality generation outcomes. For SCG, sketch is entirely novel domain with significant divergence from the training dataset distribution. Nevertheless, SCG can still generate images with remarkable fidelity and aesthetics by an automatic specialized feature extractor, demonstrating its excellent generalization capabilitics.
We also tested two typical painting styles: Western Western-style oil paintings and Eastern-style wash and ink paintings. Figure/]showw the results of associative generation on oil paintings on the top part. The first row is the original oil painting images with text prompts, and the second row is the condition image Canny edge and generated images of ControlNet. Compared to clean sketches, oil paintings inevitably contain oontent-irelevant textures that can interfere with the feature extraction process the Canny edge detector. Therefore, the generated images will also affected to some extent, resulting some detailed textures not that natural and beautiful. For our SCG in the third row,
Painting:

---

ControlNet
SCG
ControlNet
SCG
Comtoieer 5CG
Controhee 800
Wirraae
60
Whrmmtee (%)
8 & 8
10-
Ficeliy
Assihotic
Fidelity
100- Controltet
100
Controbhrt
900
8. 80
Wertea 150
Ficeity
Fidelty
Aadithrtio
Figure 8: Association generation on ancient graffiti on rock. (See more generations in Figure S8)
we use HC1 as condition, which is sensitive to brightness. Our brain-like HC1 has a better noise suppression effect, and the generated image details more vivid and beautiful. For quantitative evaluation in Figure55 we conduct subjective evaluations of the generated images in terms of fidelity and aesthetics (see details in Appendix A.4.4). As shown in F compared to ControlNet, SCG has significantly higher winning rates in both fidelity and aesthetics. In addition to Western-style oil paintings, we also test Eastern-style wash and ink paintings. In the bottom part of Figure 7 the first column is the original wash and ink paintings, and we generated 2 images each with HCI and HC3 as conditions. respectively. The generated images have similar content and structure the original paintings. At the same time, SCG recolors the painting and generates natural and reasonable textures.
Ancient Graffiti: In contrast to clean sketch without noise and painting, the associative generation of ancient graffiti on rock is a more challenging task due to the inherent high noise levels resulting from their presence natural rock surfaces over thousands of years. Similarly, we also test the results of ControlNet based on Canny and SCG based on HC3 in Figure g respectively. Due to high-level noise, the artificially designed Canny edge detector is seriously disturbed by noise, resulting in relatively cluttered edge maps. Compared to Canny, our learned modules exhibit strong robustness to noise. For quantitative comparison, we conduct a subjective evaluation on fidelity and aesthetics. For the first graffiti of a horse, the images generated by the proposed SCG significantly outperformed those from the Canny-base based ControlNet in terms of both fidelity and aesthetics. For fidelity, the four vertical lines above the horse's ' back are not reflected in the image generated by ControlNet, but four trees corresponding the lines are generated by SCG. The overall clarity of SCG is also better than ControlNet. In the hunting scene on the right, Canny edge captures many noise lines and spots on the rock, which is also reflected in its generated image. Although makes the generated images more similar to the original image, these noisy pieces of information are content-irrelevan and can significantly reduce the aesthetics of the image. Our feature suppresses the noise effectively while extracting overall structures and contents. Therefore, the generated images are more natural and aesthetically pleasing in terms of detail while being structurally similar to the original graffiti. In subjective evaluation, SCG has a similar winning rate on fidelity and significantly higher winning rate on aesthetics. These demonstrate that the human-like like sulf-super pattern completion approach can effectively learn controllable generation capabilities, while exhibiting exceptional generalization ability and robustness in more challenging scenarios.
We also add comparisons to other conditions besides Canny, including depth maps, normal di- rections, and semantic segmentation in Appendix Fig. SIT These methods require supervised pre-trained feature extractors, which are sensitive to data distribution. We also tested more tasks such as super-resolution and dehazing, as shown in Appendix Fig. [S12] and found that SCG, in addition to spontancously emerging conditional generation capabilities for oil paintings, ink paintings,

---

ancient graffiti, sketches, etc., still zero-shot generalized abilities of super-resolution, dehazing, and controllable generation under line conditions.
Discussion and Limitation
The proposed equivariance constraint enables the network to spontaneously specialize into mod- ules with distinct functionalities. Further experiments reveal features within each module exhibit similar spatial frequency selectivity, while different modules possess distinct spatial frequency prefer- ences(see in A.3.1 Within each module, features exhibit varying orientation selectivity, covering the entire orientation space through a population coding scheme, forming a continuous and closed independent manifold space(see in A.3.2). These findings suggest the proposed equivariance con- straint effectively promotes both intra-module correlation and inter-module independence, facilitating functional specialization. The brain- -ain-insspire mechanisms and emergence of brain- brain-like phenomena provide a novel perspective for gaining deeper insights into the learning mechanism of the brain. However, the current equivariance constraint is simplified and preliminary version, resulting in only low-level feature specialization. While more features, such as depth (parallax), motion(e.g. optical flow), and semantic-orienter contours and instance segmentation, remain unlearned, they hold significant potential to further enhance the controllability or more various tasks. Nevertheless, validates the efficacy of using equivariance constraints achieve modularization. Future endeavors will focus on inducing richer functional specialization encompassing semantic hierarchies.
Inspired by the hippocampal pattern completion in the brain, propose self-supervised controllable generative framework based on automatically specialized functional modules. Since our modular autoencoder learns set of complementary and relatively complete modules, most conditional generation or reconstruction tasks can be considered as scenarios where information in one or more modules is missing damaged. This is also the source of SCG's zero-shot generalization capabilities such as sketch-conditioned generation, super-resolution, dehazing, etc.. While fully self-supervised training exhibit impressive emergent capabilities and demonstrate strong generalization across data distributions and task variations, still falls short compared to dedicated supervised methods in specific tasks. Leveraging our self-supervised model as pre-trained model and then fine-tuning labeled data for specific downstream tasks can significantly improve performance on those tasks. We leave these future works.
Conclusion
Inspired by the visual hypercolumn, we propose modular autoencoder framework with equiv- ariance constraints to automatically achieve functional specialization by promoting inter-modulc independence and intra-module correlation. Experimental results on multiple datasets demonstrate the effectiveness of this approach in generating functional specialization. exhibiting characteristics reminiscent of the biological primary visual cortex. Building upon these differentiated modules. we develop a self- supervieee controllable generative framework inspired by hippocampal modal completion. Experiments reveal that this framework can spontaneously zero-shot emerge human- like associative generation, conditional generation by sketch and line art, superresolution, dehaze. And it exhibits strong generalization abilities even in scenarios involving significant distributional differences, such as associating sketches and ancient graffiti with natural visual scenes.
8
Acknowledge
This work was supported by National Key R&D Program of China((0027/001113333 International Partnership Program of Chinese Academy of (117322221111K2200000) Beijing Natural Science Foundation (No. JQ24023) and the Beijing Municipal Science & Technology Commission Project (No. Z 10000663301).)
References
[1] Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniy Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable
10

---

image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18370-18380, 2023.
[2] Dina Bashkirova, José Lezama, Kihyuk Sohn, Kate Saenko, and Irfan Essa. Masksketch: Un- paired seuucture-guider masked image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1879-1889, 2023.
[3] Tobias Bonhoeffer and Amiram Grinvald. Iso-orientation domains in cat visual cortex are arranged in pinwheel-lit like patterns. Nature, 353(6343):429 431, 1991.
[4] Paul C Bressloff and Jack D Cowan. The visual cortex as crystal. Physica D: Nonlinear Phenomena, 173(3-4):22-258, 2002.
[5] Cindy M Bukach, Isabel Gauthier, and Michael J Tamr. Beyond faces and modularity: the power of an expertise framework. Trends in cognitive sciences, 10(4): 2006.
[6] Jiaxuan Chen, Yu Qi, and Gang Pan. Rethinking visual reconstruction: Experience- content completion guided by visual cues. 2023.
[7] Wengling Chen and James Hays. Sketchygan: Towards diverse and realistic sketch to image synthesis. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 9416-9422 9425, 2018.
[8] Zhiqiang Chen, Yang Chen, Xiaolong Zou, and Shan Yu. Continuous rotation group equivariant network inspired by neural population coding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 14622-11470, 2024.
[9] Zhiqiang Chen, Ting-Bing Xu, Jinpeng Li, and Huiguang He. Sharing weights in shallow layers via rotation group equivariant convolutions. Machine Intelligence Research, 19(2):11-1126, 2022.
[10] Alex Chernjavsky and John Moody. Spontaneous development of modularity in simple cortical models. Neural Computation, 2(3):334-344, 1990.
[11] Grigorios G Chrysos, Jean Kossaifi, Zhiding Yu, and Anima Anandkumar. Unsupervised controllable generation with self-training. 2021 International Joint Conference on Neural
Networks (IJCNN), pages 1-8. IEEE, 2021.
[12] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference machine learning, pages 2999. PMLR, 2016.
[13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.044. 810004 2018.
[14] Courtney L Gallen and Mark D D'Esposito. Brain modularity: a biomarker of intervention-related plasticity. Trends in cognitive sciences, 23(4):293-304, 2019.
[15] Ruiqi Gao, Jianwen Xie, Siyuan Huang, Yufan Ren, Song-Chun Zhu, and Ying Nian Wu. Learning v1 simple cells with vector representation of local content and matrix representation of local motion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6674-6684, 2022.
[16) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.
[17] Julie Grollier, Damien Querlioz, KY Camsari, Karin Everschor-Sitte Shunsuke Fukami, and Mark D Stiles. Neuromorphic spintronics. Nature electronics, 3(7):360-370, 2020.
[18] Segundo Jose Guzman, Alois Schlögl, Michael Frotscher, and Peter Jonas. Synaptic mechanisms of pattern completion in the hippocampal ca3 network. Science, 35346304) 2016.
[19] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[20] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[21] Geoffrey E. Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In International conference learning representations, 2018.
11

---

[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[23] Aidan J Horner, James A Bisby, Daniel Bush, Wen-Jing Lin, and Neil Burgess. Evidence for holistic episodic recollection via hippocampal pattern completion. Nature communications, 6(1):7462, 2015.
[24] K-Y Hsu, H-Y Li, and Demetri Psaltis. Holographic implementation of a fully connected neural network. Proceedings of the IEEE, 78(10):1637-164) 1990.
[25] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A compre- hensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36.78723-78747,, 2023.
[26] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10124-10134, 2023.
[27] Adam Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey Hinton. Stacked capsule autoen- coders. Advances in neural information processing systems, 32, 2019.
[28] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi- concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931-1941, 2023.
[29] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. Controllable text-to-image generation. Advances in neural information processing systems, 32, 2019.
[30] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt V 13, pages 740-755. Springer, 2014.
[31] Fayao Liu, Chunhua Shen, and Guosheng Lin. Deep convolutional neural fields for depth estimation from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5162-5170, 2015.
[32] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[33] Ningning Ma. Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pages 116-131, 2018
[34] Yue Ma. Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: Pose-guided text-to-video generation using pose-free videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4117-4125, 2024.
[35] David Meunier, Renaud Lambiotte, and Edward Bullmore. Modular and hierarchically modular organization of brain networks. Frontiers in neuroscience, 4:7572, 2010.
[36] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4296 4304,,304,22 2024.
[37] Leigang Qu, Shengqiong Wu, Hao Fei, Liqiang Nie, and Tat-Seng Chua. Layoutllm-12i: Eliciting layout guidance from I1m for text-to-image generation. In Proceedings of the 31st ACM International Conference on Multimedia, pages 643-654, 2023.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 5748-8763. PMLR, 2021.
[39] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
12

---

[40] Jathushan Rajasegaran, Vinoj Jayasundara, Sandaru Jayasekara, Hirunima Jayasekara, Suranga Seneviratne, and Ranga Rodrigo. Deepcaps: Going deeper with capsule networks. In Pro- ceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 10725-10733, 2019.
[41] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or, Or. Encoding in style: a stylegan encoder for image-to to to-image translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2287-2296 2021.
[42] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8599 2021.
[43] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. Advances in neural information processing systems, 30, 2017.
[44] Zhengyang Shen, Lingshen He, Zhouchen Lin, and Jinwen Ma. Pdo-econvs: Partial differential operator based equivariant convolutions. arXiv preprint Ttv:2007.104006 2020.
[45] Ke Sun, Mingjie Li. Dong Liu, and Jingdong Wang. Igev3: Interleaved low-rank group convolutions for efficient deep neural networks. arXiv preprint arXiv:1806.001 00178, 2018.
[46] Mriganka Sur, Sarah L Pallas, and Anna W Roe. Cross-modal plasticity in cortical development: differentiation and specification of sensory neocortex. Trends in neurosciences, 13(6)2227-333, 1990.
[47] Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1-9, 2015.
[48] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818 288-22826, 2016.
[49] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14453-14463, 2023.
[50] Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimation via deep neural
networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1653-1660, 2014.
[51] Maurice Weiler and Gabriele Cesa. General e(2) equivariant steerable cnns. arXiv preprint arXiv:1911.0822 2019.
[52] Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 849- 858,858,22 2018.
[53] Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015,
[54] Zhongcong Xu, Jianfeng Zhang. Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. arXiv preprint arXiv:2311. I.64993 2023.
[55] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 3836-3847, October 2023.
[56] Ting Zhang, Guo-Jun Qi, Bin Xiao, and Jingdong Wang. Interleaved group convolutions. In Proceedings of the IEEE international conference on computer vision, pages 4373-4382, 2017.
[57] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848-6856. 2018
[58] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
13

---

[59] Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Lay- outdiffusion: Controllable diffusion model for layout-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 22490-22444 22499,, June 2023.
[60] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoe V Le. James Laudon, et al. Mixture-of-experts with expert choice routing, Advances in Neural Information Processing Systems, 35:7103-7114 2022.
[61] Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexci A Efros. Unpaired image-to- to-ii translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference computer vision, pages 2223-2233 2017.
14

---

With
Without equivariant constraint
Feature maps
Reconstruction
equivariant constraint
S
5
5
$
5
N
S
b
Figure S1: Ablation study equivariance constraint.
A / supplementa material
A.1 equivariance constraint
equivariance constraints are crucial for modular autoencoders. this work, we designed simple equivariance constraints to validate the proposed modular autoencoder framework. We primarily em- ployed two equivariance constraints: translational equivariance constraints and equivariance constraints. These constraints were applied to different module partitions. Since transla- tional equivariance encompassed by wewlllaional-ottttinn to translational equivariance should be included within those corresponding to translatiomal-rotttionna equivariance. When multiple translational modules are combined into translational-roottionna module, the learned features exhibit pronounced orientation selectivity, illustrated in Figure SS Each convolutional kernel constitutes a translational equivariant module, and every translational equivariant modules (a row) combine to form wanslational-rotation hypercolumn. Figure SSp demonstrates that using two convolutional kernels to constitute a trans- lational equivariant module, and eight such modules to form vanslationallrotation equivariant module, also results in learned orientation selectivity. When translational and cunslational-rotatioon modules coincide, depicted in Figure [S3), each row represents both translational equivariant module and vaanslatioall-otatiioe symmetry. Regardless of the specific variations in the equivariant module settings, the autoencoder consistently produces clear functional specialization, demonstrating the effectiveness of the proposed cquivariance constraints achieving functional differentiation.
equivariance, modules corresponding
cquivariant module, termed
equivariant module. The learned features exhibit rotational
A.2 Ablation Study on Equivariance Constraint and Modular Autoencoder
To validate the efficacy of equivariance constraints, we trained a modular autoencoder without utilizing these constraints under the same settings. Figure STha depicts the learned features with equivariance constraints, while Figure SIp and visualize the feature maps and reconstruction results of each module or hypercolumm individually. Notably, each module reconstructs images with a distinct focus on different spatial frequencies. When equivariance constraints are not employed, as shown in Figure STd, the learned features fail to exhibit meaningful functional specialization or acquire hypercolumn- like orientation selectivity. This underscores the significance of equivariance constraints in facilitating functional specialization and the emergence of hypercolumn-ling characteristics.
By removing the symmetric loss (see Fig. S2 the autoencoder overall still works, but some modules exhibit unrelated sub-features within them. When we change the number of modules, the autoencoder still reliably forms feature differentiation. When removing the translation transform of the data, the
15

---

16 modulars
PCA
No rotation
No translation
32 modulars
With(upper) and without (bottom) symmetry loss
Figure S2: Ablation study on Modular Autoencoder.
learned features lose their direction selectivity. When removing the rotation transform, each module can only learn features with the same orientation. We also compare our modular features with PCA clear difference compared to approach, namely the
features. As shown in Fig S2 we lack of modular features in PCA. Thcoretically, PCA, due to lack specialized modular features, struggles flexibly manipulate different aspects of features, leading generative model that is almost a reconstruction of the original image and and thus lacks the ability to achieve widespread zero-shot generalization.
A.3 Brain-like Characteristies
A.3.1 Brain-like Tuning Curve
To further analyze the functional specialization and brain-like characteristics the modules, we tested the orientation and frequency tuning curves of each kernel using grating images of varying frequencies and orientations, as illustrated Figure S54 In Figure (S4p, each plot represents module or hypercolumn, with each curve representing the orientation tuning curve of kernel. Notably,
16

---

Visualization Positive waight
Negativee weight
Input Image
DeconvolLtion hypercolumm
2 5T PE
8
5
8
6
EE Z1 TE DE
b
Figure S3: Color antagonism and center-surround receptive fields.
Group 1
Drapp
Grop3
Group
freq wwr.:ttuingg.oom curve
Forplace
argengee
L1
Ora
treq
(b) Freqency tuning curve without equivariant constraint.
Group
Gruu 12
Anglout
Teeg_tuutinggcom curve
frea
(c) Freqency tuning with equivariant constraint.
(a) Orientation tuning curve
Figure $4: Orientation and frequency tuning curves.
17

---

0.2
0.1
8
O.ty
5 C,C,
0.30
035
[-0.15.0.15 1.0.1.155
0.05
0.5
000
-010
-0.3
by = [-0.15,0.15]
8 [-
#= [-2,
2 0,t,
0,0
Selected hypercolumn visualize
Figure S5: Close and complete submanifold for each hypercolumn.
most kernels spontaneously learned orientation selectivity, exhibiting Gaussian-shaped tuning curve. Figure SS4p and depict the overall spatial frequency tuning curves of each module with and without equivariance constraints, respectively. Thin lines and thick lines represent the tuning curves of individual kernels and modules, respectively. In the absence of equivariance constraints, the learned features do exhibit significant spatial frequency differentiation or clear frequency selectivity between modules. However, in the presence of equivariance constraints, the modules demonstrate highly pronounced frequency selectivity, characterized by distinct Gaussian maassiinn-shapdd tuning curves. Collectively, these modules effectively cover relatively complete frequency space.
A.3.2 Population Coding and Submanifold
Each hypercolumn encompasses complete orientation space, forming self-contained and complete spatial representation, as visually demonstrated in Figure Figure S5p depicts the experimental hypercolumn. By activating a single neuron within hypercolumn and subsequently utilizing the prediction matrix M+ (8) to predict and visualize the response pattern of the hypercolumn under various transformations, Figure showcases the visualization of the predicted representation for translational reconstruction, revealing continuous spatial translations in the reconstructed images. Figure 1551 presents the visualization of the predicted representation for rotational reconstruction, demonstrating continuous and complete orientation variations. These reconstructions are obtained through linear combinations of kernels within the same hypercolumn, echoing the phenomenon of population coding in biological systems. This suggests that each hypercolumn constitutes closed and complete submanifold space. Figure (S5e illustrates the orientation representation visualization of another hypercolumn, highlighting that different hypercolumns form distinct submanifold spaces. This formation of independent, closed, and complete submanifold spaces indicates that our equiv- ariance constraints effectively enhance intra-modula correlations and inter-modular independence, thereby facilitating functional specialization among modules.
A.3.3 Color Antagonism and Center-surround Receptive Fields
As shown in Figure S3 when translational equivariant modules and iosolationallrotattionn equivariant modules coincide, rotationally symmetric features can be learned. These features exhibit a prominent center-surround antagonistic receptive field, as observed hypercolumns 10, and 12 Figure S3h. This bears resemblance to the on-center, off-center cells found in the biological primary visual cortex. For hypercolumns 2, 8, and 15, distinct color selectivity emerges, accompanied by prominent center-surround receptive field and the presence of color antagonism, characteristic
18

---

Comfortable, modern living room overlooking a wooded area.
HyperColumn
Canny
zebra standing top grass covered field.
HyperColumn
Canny
Figure S6: More generated images in COCO.
commonly observed biological systems. These brain-like center-surround receptive fields and color antagonism are believed to enhance robustness to noise.
A.4
More results for SCG
A.4.1 Experiment Details
During the self-supervised pattern completion stage, utilize the differentiated modules from Figure S3a as feature extractors. We divide the 16 modules into six parts: modules 1, 2, 4, 8, 9, and 15 form HCO; modules 2 and 3 form HCI; modules 5 and 12 form HC2: module 10 forms HC3; modules and 7 form HC4; and modules 11, 13, and 14 form HCS. Accordingly, HCO comprises all modules sensitive color, HCI comprises modules sensitive to brightness, HC2 and HC3 comprise modules sensitive to edges, and HC4 and HC5 comprise modules sensitive to higher frequency features.
A.4.2 Training Setup
On the pattern completion stage, we employed ControlNet as our backbone network. We utilized SD1.5 as the pretrained diffusion model, which remained frozen during training. For the control conditioning branch, we extracted features from the selected modules and upsampled them to pixel space to serve as control images. We trained our model for 5 epochs with batch size of 4 on the COCO dataset using an NVIDIA A A100 GPU. For the ControlNet baseline reported in the paper, we followed the same training procedure. We used random translation and rotation augmentation within 0.3 times the image length and 360 degrees for training our modular autoencoder. The modular artoencoder is single-layer or be cquivalently considered as single-layer network in our
19

---

BOV
Figure S7: More generated images in COCO.
ControlNet
SCG
ControlNet
SCG
Figure S8: More results for ancient graffiti.
20

---

Original
HC1
Generations
HC3
Generations
Figure S9: More results for wash and ink painting.
implementation. We train it for 40000 steps with AdamW optimizer on cosine anneal strategy with a start learning rate 0.005.
A.4.3 Datasets
The MNIST database of handwritten digits has training set of 60,000 examples, and test set of 10,000 examples, where the digits have been nize-normalized and centered in a fixed-size image.
ImageNe Large Scale Visual Recognition Challenge (ILSVRC) contains over 1.2 million various size images of 1,000 classes for training and 50,000 images for validation. As the time and computing resources limits, we utilize 100 instead of the whole 1000 classes to train our modular autoencoder.
MS-COCO (Common Objects in Context) is large-scale dataset for object detection, segmentation, captioning, and other computer vision tasks. It contains over 200,000 images with more than 1.5 million labeled objects. We use coco2017 train including of 118K images train SCG.
A.4.4 Subjective Evaluation
We primarily conducted subjective evaluations on oil paintings and ancient graffiti. For oil paintings, used six images and generated pairs of images for each image using both ControlNet and SCG, resulting in total of 12 image pairs. Participants were asked to rate which image in each pair had better fidelity and aestheties. Finally, we statistics the mean winning rates for ControlNet and SCG. A total of 40 participants were recruited for this study, and all test images are shown in Figure For the ancient graffiti, a total of 37 participants were recruited for this study. And we statisties
STO
http://yaan.leuuu.com/eddd http://www.inngge..ccrr/hhaaaaap
21

---

Original
Canny
Generations
Generations
Figure $10: More results for oil painting.
Original Image
Our SCG
Depth
Normal
Segment
Canny
Extracted condition
Generated image
horse on the grass.
Original image
Extracted condition
Generated image
Many vallege along river.
Figure $11: Comparisons under more conditions. Depth, Normal, Segment all extracted by pretrained models.
22

---

LincArt
Super-resolution
Dehazing
Sketch
Figure S12: Zero-shot conditional generation of SCG on more tasks. With green box are conditions and with red box are generated images.
the winning rates for each pair generation. We use plattform https://ww.wjx.com to collect subjective evaluations. Participants were asked which image, out of the two, had better fidelity and aesthetics basing on the original image.
A.4.5 More Generations
Figure and Figure S7 showcase additional generation results the COCO dataset using different hypercolumns as conditions. Figure [S9 and [S1O showcase additional generation results on the associative generation from wash and ink painting and oil painting. Figure (S more generation results for graffiti. irrelevant noise present in the ancient graffiti, leading either blurry images abundance of unnatural details. In contrast, our proposed SCG method produces clearer images with more natural and realistic details, demonstrating superior robustness to noise.
evident that ControlNNe-bassed generations are significantly impacted by
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See in section 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See in section d
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]

---

Justification:
We have no theory to proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See in section 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
Code is submit in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli//uuuiu wwwpc//ggue////eeeeceooc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See in section 4 and supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Statistical significance is not applicable for image controllable generation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See in appendix A.4.2
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:://eeuippppeeeee https//////lleettccccooeeeeeeeee
Answer: [Yes]
Justification: We conducted the research in the paper conform, every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See in section i
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: We use exist excellent ControlNet and their safeguards. Our model will not bring more risk than ControlNet.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We use CC BY-NC 4.0 license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.paaersstt paperswi thcode. cow datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release new code based on ControlNet. The code include an anonymized
zip file.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We conducted a brief, unpaid survey on the Internet social App lasting less than 5 minutes for each survey. No crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "dUd6OJWrg2", "reviewer": "Reviewer_G9Tm", "summary": "Presents a self-supervised approach for learning multiple distinct representations from images through a loss leading to specialized modules, each learning different aspects of the data without manual design. Demonstrate that those modules may be useful for the conditional generation of images through a controllable diffusion model.", "questions": "1.\tWhat is the respective contribution of the symmetry loss vs the equivariance loss? What does performance look like with only one of them? What is the problem solved by introducing each of them?\n2.\tWhat are the equivariance properties of HC1 and HC3 (i.e., their functional specialization) and make them useful for controllable generation?", "limitations": "There is only a limited discussion of the approach's limitations and the limits of functional specialization (summarized as “only low-level feature specialization”); it would have been nice to see a discussion of what useful features are NOT learned by this approach and why.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "*\tOriginality: the main contribution of the work is in the novel loss function and the insight (perhaps inspired by neuroscience) that modular networks are favourable for learning in self-supervised settings. The idea that a module may develop functional specialization from imposing equivariance to some property of the stimuli is also a nice extension of previous supervised approaches. I think the other half of the work, where the representation of a single module is used for controllable generation, is a natural extension of previous work. Previous works on both manual designing specialized blocks and on Group equivariance are properly quoted.\n*\tQuality: the presented experimental results are fine (but as always with image generation, visual judgement is difficult), so the quantitative evaluation (Table 1) and subjective evaluation (Figure 4 and some panels in Figure 7) are appreciated.  \n*\tClarity: the submission is clearly written and well organized, using neuroscience inspiration to improve the intuition on how to make progress on self-supervised learning of rich representations. The “pattern completion” jargon, on the other hand, does not contribute to the reader’s understanding; it would have been better to adhere to “conditional, controllable generation,” which actually describes what was done. \n*\tSignificance: the suggested equivariance and symmetry loss components are easily usable for developing a functional specialization in other projects and may impact seemingly unrelated problems.", "weaknesses": "*\tQuality: the evaluation is limited only to ControlNet, which is probably a decent baseline, but there are other approaches for image generation that can be compared with (e.g., it’s fine to show a manual-designed solution outperforming the current self-supervised model or compare with other self-supervised approaches).\n*\tSignificance: it would be great to see if the suggested loss components can be utilized in a setting other than controllable generation where their contribution can be better quantified, e.g., in lowering MSE in a reconstruction or prediction task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your affirmation of our originality!\n\n**Suggestion**：Better use \"controllable generation\" rather than “pattern completion” .\n\n**Response**：Thank you for your suggestion. “Controllable generation” is indeed a more accurate and understandable term in the field of AI, as it clearly reflects the concept of generating content with specific controls. We will clarify this by explaining that “pattern completion” corresponds to “controllable generation” when we introduce the concept in our paper. From that point forward, we will consistently use “controllable generation” throughout the paper.\n\n**Q1**：Quility: the evaluation is limited.\n\n**R1**：We added comparisons to other conditions besides Canny, including depth maps, normal directions, and semantic segmentation  in PDF Fig. R1, with each condition extracted by a pre-trained model. These condition extraction networks are highly sensitive to data distributions. For instance, all models fail to extract condition information for the ancient graffiti, resulting in low-fidelity generated images. When suitable features can be extracted, more abstract control conditions can achieve better generation results, such as ControlNet based on Canny and depth. Here are two possible reasons for this: 1) Our SCG was only trained on COCO, not a larger dataset; 2) SCG training did not use supervised data, such as sketch and image pairs. Addressing these two issues could significantly improve SCG’s ability on specific tasks. Our goal in this work is more focused on demonstrating that SCG can spontaneously emerge conditional generation capabilities on a wide range of tasks. Refer to GR4.1 for additional information.\n\n**Q2**：Significance: suggestions to test on reconstruction task other than generation.\n\n**R2**：Thanks for good suggestions. We quickly conducted simple tests on image super-resolution and dehazing tasks (see PDF Fig. R2) and still see that the proposed SCG has emerged with both super-resolution and dehazing capabilities. As SCG is fully self-supervised, its performance is hard to compare with supervised and dedicated networks. However, it has the potential to be used as a self-supervised pre-trained model and then fine-tuned on specific tasks, thereby enhancing the network’s ability in specific tasks while maintaining high generalization capabilities. Refer to GR4.2 and GR1.1 for additional information.\n\n**Q3**：More analysis on symmetry loss vs equivariance loss.\n\n**R3**：The equivariance loss is central to the modular autoencoder architecture, playing a crucial role in promoting independence between modules while simultaneously strengthening feature correlations within each module. The symmetry loss further enhances pairwise correlations (or symmetry) between features within a module, effectively suppressing the emergence of multiple unrelated sub-features within the same module. Removing the equivariance loss prevents the network from learning brain-like Gabor-shaped features and hinders the emergence of complementary functional specialized modules (see Appendix Fig. S1 a and d in the main text). While removing the symmetry loss allows the autoencoder to still generate complementary functional specializations overall, the correlations (or symmetry) within individual modules are reduced. This can lead to multiple unrelated features appearing within some modules (see PDF Fig. R3). Refer to GR2 for more.\n\n**Q4**：What are the equivariance properties of HC1 and HC3 and make them useful for controllable generation?\n\n**R4**：\n1. It is noteworthy that all modules within the same network are under the same type of equivariant constraint (i.e., translation and rotation). The distinct functions that emerge from these different modules are entirely self-emergent and not a result of imposing distinct constraints on each module. \n2. Modular autoencoder, under equivariance constraints, disentangles and decomposes into modular and complementary features. For example, HC0 represents color features, HC1 represents brightness features, HC2 and HC3 represent edge features, and HC4 and HC5 represent higher-frequency edge features. When we retain the brightness features (HC1) and drop the other features, the generation model will use the brightness features as clues to complete the other missing information, such as color. Therefore, conditional generation based on HC1 can be used for tasks like colorizing ink paintings or other re-color tasks. Similarly, when we retain the edge features (e.g., HC3) and discard other features, the generation module can use the edge information to generate color, brightness, and other information. Therefore, it often performs better in more abstract conditional generation tasks, such as sketch and ancient graffiti. For HC0, which represents color information, it is insensitive to fog due to its white color. Therefore, we can utilize HC0 for dehazing. \n3. Different conditional generation tasks can be viewed as scenarios where certain aspects of an image are missing or corrupted. The goal is to use the remaining information as clues to complete or generate the missing information. The modularity of the autoencoder, resulting in relatively independent (at a local scale) and complementary features, allows for zero-shot generalization across a wide range of tasks, despite not being specifically trained for any particular task. Refer to GR3 for more.\n\n**Q5**：Limitations: what useful features are NOT learned by this approach and why.\n\n**R5**: While more features, such as depth (parallax), motion(e.g., optical flow), and semantic-oriented contours and instance segmentation, remain unlearned, they hold significant potential to further enhance the Controllability. Our current version of the autoencoder is limited to learning static, local features, thus preventing it from disentangling dynamic, volumetric, and semantically related modular features. We will discuss this in more detail in the Limitation section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OjdHQat8cE", "reviewer": "Reviewer_eu8g", "summary": "This paper proposes a self-supervised controllable generation (SCG) framework with two training stages. The first stage exploits equivalence invariance to learn a modular autoencoder, aims to extract different visual pattens from input images, each extracted (learned) visual pattens can be regarded as a kind of image condition. In the second stage, perform as the standard ControlNet training with these self-supervised extracted image conditions. After self-supervised training, SCG surprisingly shows some zero-shot capabilities, which can show excellent generalization capabilities for various tasks such as associative generation of paintings, sketches, and ancient graffiti. The authors compare SCG with ControlNet in terms of SSIM/PSNR/CLIP-score and achieved competitive results. However, the authors lack further exploration of the self-supervised trained ControlNet. Overall, I like the idea and method very much, but I still have some doubts about the validity of the experimental results and the fairness of the comparison.", "questions": "Please refer to the weaknesses.", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. I like the idea that exploring self-supervised training on controllable generation models. Compared with existing methods, the self-supervised training framework proposed in this paper does show better generalization ability to a certain extent.\n2. Using the equivariant constraint to learn a modular autoencoder to extract different visual patterns is a simple, elegant and effective approach.\n3. Although self-supervised training has not seen conditions such as painting, sketches, and ancient graffiti, it still shows good zero-shot generation capabilities, which verifies the rationality of training modular autoencoder with equivariant constraint.\n4. Compared to previous excellent works ControlNet, SCG has a higher or similar winning rate in fidelity and a significantly higher winning rate in aesthetics.", "weaknesses": "1. The MS-COCO dataset is used for both the self-supervised training phase and the evaluation phase, which may lead to a serious unfair comparison because the original ControlNet is not trained on MS-COCO.\n2. Although this paper explores the self-supervised training of ControlNet and its zero-shot generalization ability, no further experiments are conducted, such as whether using it as pre-trained weights for further fine-tuning on labeled tasks such as depth, segmentation, etc. will bring further improvements of controllability? The zero-shot capability alone cannot fully demonstrate the effectiveness and necessity of the proposed self-supervised training.\n3. Based on the above weakness, it is necessary to conduct ablation experiments on the number of modules (from modular vae) and the types of Equivariant Constraints.\n4. In addition to the painting, sketches, and ancient graffiti mentioned in the paper, does SCG also have certain zero-shot capabilities under some other conditions such as Edge/LineArt/Segmentation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your fondness of our idea and are truly encouraged by it.\n\n**Q1**：Evaluation may lead to a serious unfair comparison because the original ControlNet is not trained on MS-COCO.\n\n**R1**：The ControlNet we used as baseline is trained from scratch with the same setting as the proposed SCG. We apologize for the inconvenience and confusion caused to readers, including reviewers, by placing the introduction of some of our training settings, including the comparison baseline ControlNet, in Appendix A.4.2 of the main text. We will move this information to the main text’s experimental section in the future.\n\n**Q2**：Further experiments are conducted, such as whether using it as pre-trained weights for further fine-tuning on labeled tasks such as depth, segmentation, etc. will bring further improvements of controllability? The zero-shot capability alone cannot fully demonstrate the effectiveness and necessity of the proposed self-supervised training.\n\n**R2**：This is an incredibly insightful suggestion, opening up new avenues for our research. Previously, we focused on fully unsupervised methods for modular, independent, and complementary feature disentanglement, exploring the emergent capabilities. Your suggestion has broadened our perspective. While fully self-supervised training can exhibit impressive emergent capabilities and demonstrate strong generalization across data distributions and task variations, it still falls short compared to supervised methods in specific tasks. Leveraging our self-supervised model as a pre-trained model and then fine-tuning it on labeled data for specific downstream tasks, such as sketch-conditioned generation, super-resolution, or dehazing, can significantly improve performance on those tasks. We believe this approach will enable both competitive capabilities on specific tasks and the ability to generalize to out-of-distribution data and tasks. Moreover, this fully unsupervised pre-training process has the potential to replicate the scaling law observed in large language models and other domains, establishing a foundation model for conditional generation. We will incorporate this into our future work section. Refer to GR1.1 for more.\n\n**Q3**：Ablation experiments on the number of modules (from modular vae) and the types of Equivariant Constraints.\n\n**R3**：We apologize for our focus on demonstrating the zero-shot generalization capability of the conditional generation model and neglecting this aspect. We have included an ablation study in PDF Fig. R3.\n1. As visualized in Fig. R3, when the number of modules or the number of convolutional kernels within each module varies, the modular autoencoder network can still reliably produce relatively independent and complementary functional specializations.\n2. When translational motion is removed, modular features are still emerged, but they lose their orientation selectivity, exhibiting a center-surround receptive field similar to a Difference of Gaussians. When rotational motion is removed, features within each module exhibit the same orientation selectivity.\n3. When the symmetry constraint is removed, modular features are still generally produced, but some modules may contain multiple unrelated (or asymmetric) sub-features. As shown in Appendix Fig. S1 a and d of the main text, when the equivariant constraint is removed, the model cannot generate relatively independent and complementary functional modules.\n\nWe will add these ablation experiments to the appendix. Refer to GR2.2 for more.\n\n**Q4**：Does SCG also have certain zero-shot capabilities under some other conditions such as Edge/LineArt/Segmentation?\n\n**R4**：Yes.\n1. We quickly test SCG in other zero-shot tasks including generation conditioned by LineArt (see PDF Fig. R2), as well as super-resolution (see PDF Fig. R2) and dehazing (see PDF Fig. R2)(Refer to GR4.2 for more). \n2. Furthermore, we also can manipulate saturation and contrast of the generated image by changing the coefficient of HC0 and HC1, see in PDF Fig. R5. (Refer to GR4.3 for more). \n\nThis demonstrates the advantage of self-supervision over supervised learning, allowing the emergence of broader capabilities rather than just specific ones. It also indicates the effectiveness of the features learned by our modular autoencoder.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GuJdiiip8T", "reviewer": "Reviewer_gXoh", "summary": "The authors train a modular auto encoder with an auxiliary custom equivariance objective, which enables them to get independent sets of representations of an image. The authors then use some of these representations to condition a ControlNet. \n\nThe authors find that the auto encoder’s submodules learn to encode functional specializations such as edge detection and other low-level image features.", "questions": "- The authors chose to evaluate ControlNet with the Canny image detector. Would a different representation to condition on have worked better? It seems obvious that the Canny image detector is a lossy representation of the original image. \n- The authors could use a tilde (~) before \\cite commands to put a small space before the square brackets, which would make the text more visually pleasing. For example~\\cite{schmidhuber_1994}.\n- I believe that instead of “equivariant constraint”, the correct term is “equivariance constraint”. \n- It would be great if the README.md of the code could state what kind of GPU and how much RAM is required to run their code. \n- It would be great if the authors could include more detail in the caption of Figure 2, especially for part C of the figure. \n- On line 56, it appears that a word is missing between “sparsely” and “in”\n- On line 66, the authors use the term “closed and complete independent manifold”. The terms “manifold”, “closed” and “complete” have specific meanings in the mathematical area of topology, and if I understand correctly the authors are not using the words in the sense of the definitions from topology. It may be useful to use different terms here to avoid confusion. \n- On line 93, the authors mention “simple cells”. It would be helpful for some readers to change this to “simple cells in the visual cortex”. \n- On line 130 it says “where M^(I)(delta) is a learnable prediction matrix determined by delta”. I still do not understand what M^(I)(delta)  is. Is it a neural network that takes delta as input? Or is it computed for every delta? More generally: what is the set or distribution of all deltas? Equation (6) involves a sum over deltas, and I don’t know what the set of all deltas (or its distribution) is. \n- I would be grateful if the authors could describe equation (8) more. Is C able to construct z(I) based on z^(i)(I) and i alone? \n- On line 172, “We” should be lower-cased.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper contributes an exciting and challenging field: controllable image generation. The work presented in the paper is original. \n- The authors make their code available with the submission. This makes the paper easier to review, and will greatly improve the paper’s usefulness for other research groups working on controllable generation.", "weaknesses": "- The paper is titled “Learning from Pattern Completion”. What are we learning from pattern completion? \n- Section 3 should be written more clearly. \n- While the paper promises controllable image generation, it appears that the proposed method is more of an image reconstruction method: the experiments from the paper evaluate the proposed method by giving it features generated using the proposed modular autoencoder as input. My sense is that if I wanted to draw “a monkey climbing on the moon”, then I could give ControlNet a hand-drawn sketch of this, while the proposed SCG method would not be able to generate an image based on my sketch. Is this correct?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your efforts and valuable feedback on our paper.\n\n**Q1**：What are we learning from pattern completion?\n\n**R1**：Pattern completion focuses on the relationships between different module features at a global scale. By learning these relationships, SCG can utilize information from a subset of modules as clues to complete or generate information in other modules, thereby achieving various conditional generation tasks. Refer to GR3.2 for more.\n\n**Q2**：Section 3 should be written more clearly.\n\n**R2**：We sincerely apologize for the lack of clarity in some of our expressions. We will further clarify the captions of Figure 2, the prediction matrix M, the distribution of $\\delta$, equation (6) and equation (8). Since we only presented a very coarse-grained overview of our framework in Figure 1, it may have been difficult to grasp the finer details of the method, especially regarding the modular autoencoder part. To address this, we have added a more detailed structural diagram of the modular autoencoder in PDF Fig. R4. It provides a more intuitive visualization of the modular autoencoder with proposed equivariance constraint, including the prediction matrix M, the equivariant loss, and the symmetry loss. And we will add this figure in section 3. Also can refer to GR2.1.\n\n**Q3**：It appears that the proposed method is more of an image reconstruction method... The proposed SCG method would not be able to generate an image based on my sketch. Is this correct?\n\n**R3**：The proposed SCG can generalize to both reconstruction-oriented tasks and generation-oriented tasks, such as sketch (main text Figure 5), LineArt (PDF Fig. R2), and ancient graffiti (main text Figure 7) in a zero-shot manner. \n1. In the PDF Fig. R2, we use SCG to generate an image of “a monkey climbing on the moon” using a sketch as control. \n2. Different modules have different characteristics. For instance, HC0 extracts color information, while HC1 extracts brightness information. This makes them more suitable for reconstruction-oriented tasks like image super-resolution (PDF Fig. R2), dehazing (PDF Fig. R2), and colorization (main text Figure 6 and Appendix Fig. S8). \n3. For HC2 and HC3, the extracted information is more abstract, focusing on edges. Thus, they are more suitable for generation-oriented tasks such as conditional generation under sketch (main text Fig. 5), line art (PDF Fig. R2), and ancient graffiti (main text Fig. 7). \n\nIt is undeniable that our fully self-supervised controllable generation approach was not specifically trained for tasks like sketch and line drawing, and therefore, its performance and stability fall short compared to dedicated supervised methods. However, SCG’s performance on specific tasks can be significantly improved by supervised fine-tuning for those tasks, such as adding sketch and images pairs for training. We will incorporate this into the discussion of future work. Refer to GR4.\n\n**Q4**: Would a different representation to condition on have worked better than Canny?\n\n**R4**: \n1. We have introduced controllable generation using depth maps, normal direction maps, and semantic segmentation maps as conditions for comparison (see PDF Fig. R1). These conditions are more abstract than Canny edges, offering a larger generation space. However, they require a supervised pre-trained network to extract the conditional information. Due to their abstract nature, the generated images exhibit greater diversity and aesthetic appeal.\n2. However, they are highly sensitive to the distribution of the conditional image. For instance, in tasks involving generating ancient graffiti and ink paintings (PDF Fig. R1), most condition extractors fail to extract the correct conditional information, leading to uncontrollable generation results. Conversely, when accurate conditional information is available, such as high-quality depth information in ink painting, the generation results surpass those of SCG and the Canny operator. For further details, please refer to GR4.1.\n\n**Q5**: What kind of GPU and how much RAM is required to run their code.\n\n**R5**: We use A100 GPU and it requires about 11G RAM to run it per image.\n\n**Q6**: More detail in the caption of Figure 2, especially for part C of the figure.\n\n**R6**: Revised caption of Figure 2: \"Feature Visualization of modular autoencoder. Each panel shows all features learned by an individual model with multiple modules (one module each row). We trained modular autoencoder with a translation-rotation equivariance constraint on a)MNIST and b)ImageNet, respectively. c) On ImageNet, We also train an autoencoder with an additional translation equivariance constraint besides the translation-rotation equivariance constraint. d) We visualize reconstructed images by features of each module in c.\"\n\n**Q7**: More details on $M^{(I)}(\\delta)$,distribution of all $\\delta$, Eq. (6) and Eq. (8).\n\n**R7**: \n1. $M^{(i)}$ is a codebook of learnable prediction matrices that can be indexed by $\\delta$. $M^{(i)}$ is a 3D codebook, with the three dimensions corresponding to the three parameters of $\\delta$: two translation parameters $t_x$, $t_y$, and one rotation parameter $\\theta$ (see PDF Fig. R4).\n2. The range of $t_x$ and $t_y$ is [-0.5s, 0.5s), where s is the convolution stride. The range of $\\theta$ is [0, 2pi). Each parameter or dimension is sampled at n (i.e., n=24) equally spaced values, which is also the range of sum in Equ (6). The prediction matrix $M^{(i)}(\\delta)$ is obtained from the codebook through linear interpolation.\n3. Ideally, C in Equ (8) reconstructs the complete representation from the partial module $z^{(i)}$ and its index i. In practice, the training process of C aims to achieve this goal, for example, using a diffusion model-based conditional generation process. Refer to GR2 for more.\n\n**Q8**: Some typos and imprecise term.\n\n**R8**: Thank you for your careful review and pointing out these issues. We will fix them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SdHGZR4Ntj", "reviewer": "Reviewer_h4wx", "summary": "Conditional generation in the era of diffusion models has been positively impacted by ControlNets, which allows for the fine-tuning of diffusion models using additional image input allowing for fine-grain conditioning. Popular ControlNets rely on pose, edge, or segmentation map conditioning which requires additional, usually supervised, networks to derive the condition of interest. \nIn this paper, the authors propose a fully unsupervised alternative to ControlNets and position their contribution as being proficient by mimicking the brain's modularity. They propose the training of an equivariant auto-encoder (SCG) model which allows for the disentanglement (\"modularisation\") of the input image information thereby allowing one to naturally access an altered version of an image that can serve as a condition for the generation in a ControlNet framework. Authors experimentally show the performance of the approach by 1) showing images conditioned on various representation modules, 2) comparing it against a Cany edge ControlNet, and highlighting how the approach is more robust to noise and produces more realistic images that resemble the input condition more closely.", "questions": "- can authors confirm that the conditioning used is $f$, the latent representation?\n- can you clarify what the symmetry loss terms aim to do? \n- can you explain what translation and rotation augmentations were used for the training of the autoencoder? what is the sensitivity of the representations to these parameters?\n- can you explain the training procedure of the autoencoder? number of epochs, optimizer, ...\n- can you confirm that the encoder and decoder \"architectures\" are single-layer convolutional filters? \n- how is the subject evaluation performed? how many annotators? through which annotation platform? on how many images? which questions are asked of annotators?", "limitations": "The authors discuss the fact that the current equivariant autoencoder disentangles mostly based on spatial frequency rather than following a human-interpretable partitioning of the information (high-level/semantic vs. low-level information). Beyond that point which limits the impact of this work, I believe additional limitations such as the lack of control over the redundancy between the original image and the image condition is a limitations that is tied to the use of fully unsupervised despite clear advantages to choosing fully unsupervised methods.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- SCG is a *fully unsupervised* method, which in contrast to most ControlNets, does not require relying on supervised tools\n- SCG allows for the generation of *various conditioning inputs* by allowing one to select one out of a set of representation modules. This is fundamentally appealing as ideally, it allows the user to select what part of the input information to condition the generation on instead of being tied to a set of (supervised) tools for information extraction in a classic ControlNet setup.", "weaknesses": "I believe this paper is not in a publishable state for the following reasons: \n- Presentation/Clarity/Rigor: the paper should be polished as it contains a lot of typos, confusing sentences/sections (see below), and missing information (see questions). In general, sections 1 and 3 are hard to follow and cause some confusion, and figures - including captions - should be refined.\n- Impact: While the general goal of this paper (unsupervised + modular image conditioning) is appealing, in practice, SCG becomes a trade-off between supervision and control over the information that will condition the image generation. Standard ControlNets require supervised tools but allow for explicit control over the information that is embedded into the conditioning (e.g., body pose, and structural information through Canny Edges). SCG on the other hand, does not require supervision but results in a set of image conditions that mostly disentangles the input image according to spatial frequencies as pointed out by the authors. As a consequence, most images capture similar semantic information and generated images are very similar to one another. To summarize, I believe the current use cases of SCG are cases where one wants to generate novel images that slightly differ from the input image condition, similar to using a ControlNet on Canny Edges (which is indeed the only ControlNet that authors compare against). I think this should be made more clear in the manuscript from the get-go (only discussed in the limitations) and ideally the paper should address this concern to increase the impact of SCG. \n- Experimental validation: I believe the experimental validation does not allow the reader to have an in-depth understanding of the performance of SCG. Mainly the following questions are unanswered: how much variability (in quantitative terms) can one expect from conditioning on various modules? how do images generated from SCG compare with images generated by conditioning on the lower principal components (see [1] for a more detailed example)? \n\n[1] Balestriero, Randall, and Yann LeCun. \"Learning by reconstruction produces uninformative features for perception.\" arXiv preprint arXiv:2402.11337 (2024). \n\nMinor: \n- section 1: overall confusing, I believe the important message here is \"brain performs functional modularity, we aim at mimicking this for associative generation purposes; As a starting point, we believe equivariance helps in achieving functional modularity in neural networks\". The current state of the introduction is convoluted and it takes some thought to actually extract the general message.\n- line 33-36: should be rephrased in my opinion, the sentence is confusing/hard to follow.\n- line 43: are we talking about the brain or neural networks here? this sentence is confusing.\n- line 45: problem in the sentence formulation \n- line 53: I believe \"devide\" should be changed to \"divide\" throughout the paper\n- line 109: loose terms $\\rightarrow$ \"is a kind of change\" \n- line 120: confusing formulation \"train the autoencoder in homologous images\" \n- section 3: what is the fundamental difference between z, the latent representation and, f, the feature maps ? We seems to be switching back and forth between both notations.\n- section 3: why does the latent representation $z$ become a function (i.e., $z(I)$)?\n- Eq 6: sum over $\\delta$, which set of values does $\\delta$ belong to?\n- section 3: symmetry loss, the intuition behind this additional loss term is unclear to me.\n- line 138: typo\n- line 146: confusing, either ControlNets should be described in more detail in section 2 or the phrasing \"based on ControlNets\" should be clarified in section 3 to make it evident that the \"pattern completion\" is entirely mediated by the existing ControlNets framework and is not a contribution.\n- Eq 9: the expectation symbol should be replaced by \\mathbb{E}\n- Figure 1: This figure is supposed to reflect the approach proposed by the authors, however, it is not straightforward which elements in this figure correspond to the SCG building blocks detailed in section 3. Instead of clarifying SCG, figure 1 brought some confusion in my case.\n- Figure 3: formulation problem in the caption \n- Figure 4: typo in the caption and the x-axis label", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thorough review and constructive feedback.\n\n**Q1**: Presentation/Clarity/Rigor\n\n**R1**: We sincerely apologize for the confusion caused in the introduction and methods sections. Our aim was to bridge concepts from neuroscience with controllable generation in AI, hoping to stimulate cross-disciplinary inspiration and collaboration. However, in attempting to reach audiences from both fields, we presented the methods in a coarse-grained manner, leading to misunderstandings among readers, including reviewers. To address this, we will: 1）Rephrase: Clarify terms, sentences, equations, and figure captions in sections 1 and 3 to ensure their accuracy and accessibility. 2）Enhance understanding: Provide a more detailed diagram of the proposed modular autoencoder in Section 3 (see it in PDF Fig. R4), including a clearer representation of the prediction matrix M, equivariant loss, and symmetry loss. Refer to GR2 for more.\n\n**Q2**: Impact\n\n**R2**: \n1. This proposed method’s most significant contribution lies in introducing a novel self-supervised conditional generation training framework. This framework enables the acquisition of broad capabilities through a self-supervised approach, offering the potential to fully leverage the scaling law and achieve a large-scale foundation model for controllable generation. Furthermore, by fine-tuning the foundation model on specific downstream tasks or utilizing adapters, powerful specialized models can be easily derived. \n2. Since there has been extensive research on conditional generation, we focus on designing a modular autoencoder based on proposed equivariant constraints, which is crucial for subsequent self-supervised controllable generation. And the visualization of the autoencoder component and the demonstration of its conditional generation capability on various tasks with zero-shot generalization prove the effectiveness of our method.\n3. On specific tasks, our method still falls short of the generative capabilities of dedicated supervised methods. A major reason for this is the use of labeled data, such as sketch-image pairs, in supervised methods. If we were to fine-tune our model with labeled data, its performance on specific tasks would significantly improve.\n\nOur primary goal at this stage is to demonstrate that the proposed self-supervised framework can emerge broad conditional generative capabilities. Further supervised fine-tuning will be explored in future work. Refer to GR1 and GR4 for more.\n\n**Q3**: Variability and Comparison with principal components.\n\n**R3**: \n1. In PDF Fig. R5, we demonstrate the ability to precisely control the performance of the generated image on specific components by adjusting the intensity of the components (specifically, by multiplying different coefficients). It is evident that our method can easily manipulate saturation and contrast by changing the coefficient of HC0 and HC1. \n2. Regarding the principal components, in PDF Fig. R4, we showcase the principal components obtained through PCA. A fundamental difference between components of PCA and modules of our method is that the different components in PCA are independent. As a result, PCA lacks the ability to decouple highly correlated internal representations into relatively independent and complementary representation modules, which is the core of zero-shot conditional generation capability.\n\n**Q4**：Minors\n\n**R4**: \n\n1-2）33-36 line rephrase: We revise it to \"The brain’s remarkable ability to generate associations emerges spontaneously and likely stems from two key mechanisms. One is the brain’s modularity in terms of structure and function, enabling it to decouple and separate external stimuli into distinct features at various scales. This modularity is essential for a range of subsequent cognitive functions, including associative generation.\" \n\n3-7）We will fix them. \n\n8）We will unify z and f to z. \n\n9）We will revise z(I) to z. \n\n10）$\\delta$'s range: $\\delta$ contains 3 parameters: two translation parameters $t_x$, $t_y$, and one rotation parameter $\\theta$ (see PDF Fig. R4). The range of t_x and t_y is [-0.5s, 0.5s), where s is the convolution stride. The range of theta is [0, 2pi). Each parameter or dimension is sampled at n (i.e., n=24) equally spaced values. \n\n11）Refer to GR2. \n\n12-14）We will revise them. \n\n15）Refer to R1 and GR2 .\n\n16,17）we will fix it.\n\n**Q5**: Questions\n\n**R5**: \n\n1）Yes, we will revise it to \"z\". \n\n2）The symmetry loss aims to further enhance the internal correlation (or symmetry) within modules, building upon the equivariant loss. Without symmetry loss, it will learn unrelated features within module as shown in PDF Fig. R3, referring to GR2. \n\n3）We used random translation and rotation augmentation within 0.3 times the image length and 360 degrees. When removing translation augmentation or rotation augmentation (PDF Fig. R2), modular features are still learned. The difference is that without translation augmentation, the learned features lack orientation selectivity and resemble Gaussian difference, while without rotation augmentation, features within each module have the same orientation. \n\n4）We train it for 40000 steps with AdamW optimizer on a cosine anneal strategy with a start learning rate of 0.005. \n\n5）Yes, it’s a single-layer or can be equivalently considered as a single-layer network. \n\n6）We use plattform https://www.wjx.cn/ to collect subjective evaluations, gathering 40 questionnaires. Each questionnaire contained 12 pairs of comparison images and one original image. Participants were asked which image, out of the two, had better fidelity and aesthetics basing on the original image. （see appendix A.4.4 in main text）\n\n**Q6**: Limitation\n\n**R6**: Refer to R2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their efforts and constructive comments on our manuscript, which is very valuable and enlightening to us. We have revised our manuscript according to reviewers’ concerns. We would like to make an overall response before specific response to reviewers.

# GR1：Significance of SCG:
Our original motivation is to propose and validate a novel self-supervised pipeline rather than a model to achieve broad generalizations.
1. We propose SCG and experimentally demonstrate that it can spontaneously emerge (or 0-shot generalize) various abilities, including super-resolution, dehazing, saturation and contrast manipulation, as well as conditional generation based on diverse styles such as oil paintings, ink paintings, ancient graffiti, sketches, and LineArt. Furthermore, SCG possesses two significant potentials: 1) Leveraging its self-supervision, SCG can further scale up its data and models to benefit from the scaling law, enhancing its basic capabilities; 2) Subsequently, SCG can be fine-tuned for specific tasks, leading to improved performance on particular tasks. These potentials suggest that SCG has the potential to become a foundation model for controllable generation.
2. This framework comprises two components: a modular autoencoder and a conditional generator. Given the extensive research on conditional generation, we leverage the existing, mature ControlNet for this aspect. Our core contribution lies in designing a modular autoencoder based on proposed equivariance constraint, successfully enabling the network to spontaneously develop relatively independent and highly complementary modular features. These features are crucial for subsequent conditional generation.

# GR2：Clarification of Modular Autoencoder: 
1. We agree to all reviewers’ suggestion and will add a more detailed framework diagram of the modular autoencoder in Section 3, including a more intuitive prediction matrix $M^{(i)}(\delta)$, equivariant loss $L_{equ}$, and symmetric loss $L_{sym}$. See PDF Fig. R4 for details. The equivariance loss $L_{equ}$ is the core of the equivariant constraint, primarily serving to increase independence between modules and correlation (or symmetry) within modules. The symmetry loss $L_{sym}$ further enhances the correlation (or symmetry) of features within modules and suppresses the emergence of multiple unrelated features within the same module. 
2. By removing the symmetric loss $L_{sym}$ (see PDF Fig. R3), the autoencoder overall still works, but some modules exhibit unrelated sub-features within them. When we change the number of modules, the autoencoder still reliably forms feature differentiation (see PDF Fig. R3). When removing the translation transform of the data, the learned features lose their direction selectivity. When removing the rotation transform, each module can only learn features with the same orientation (see PDF Fig. R3). We will explain the purpose of each loss more clearly in Section 3 and supplement the above ablation experiments in the appendix.

# GR3：Pattern Completion: 
1. Pattern completion is a classic neuroscience concept.The SCG aims to mimic this process by "mask and predict" at component instead of widely used space level. We agree with the reviewers that the introduction of it is not clear enough. We plan to provide more details about this concept and its relationship with the SCG.
2. The pattern completion process learns the global relationships between different modules, enabling it to complete or generate missing information from clues provided by other modules. For example, HC1 is brightness moduler. Therefore, conditional generation based on HC1 can be used for tasks like colorizing ink paintings or other re-color tasks. Since our modular autoencoder learns a set of complementary and relatively complete modules, most conditional generation or reconstruction tasks can be considered as scenarios where information in one or more modules is missing or damaged. This is also the source of SCG’s zero-shot generalization capabilities.

# GR4：More Comparisons and More Tasks: 
1. We added comparisons to other conditions besides Canny, including depth maps, normal directions, and semantic segmentation in PDF Fig. R1. These methods require supervised pre-trained feature extractors, which are sensitive to data distribution. All condition extractors fail to extract reasonable conditional information on ancient graffiti, resulting in uncontrolled generated images, and the performances are all inferior to our SCG. In the case of ink painting conditional generation, the depth feature extractor and Canny operator extracted relatively good features, resulting in better generation results than our SCG. However, other feature extractors failed to extract reasonable features, and the generated training results were not as good as SCG.
2. We also tested more tasks, as shown in PDF Fig. R2, and found that SCG, in addition to spontaneously emerging conditional generation capabilities for oil paintings, ink paintings, ancient graffiti, sketches, etc., still zero-shot generalized super-resolution, dehazing, and controllable generation capabilities under line art and sketch conditions.
3. As shown in PDF Fig. R5, we also discovered that by changing the coefficient of HC0, we can easily manipulate the saturation of the generated image, and by changing the coefficient of HC1, we can manipulate the contrast of the generated image. 
4. On specific tasks, SCG’s generative ability still falls short of supervised, specialized models. There might be two reasons: 1) SCG is trained only on COCO, while specialized models are often trained on larger and high-quality datasets. 2) No labeled or supervised data (e.g., scribble and image pairs) has been incorporated for training.

In summary, despite being trained solely on COCO through self-supervised learning, SCG has demonstrated a wide range of capabilities. These extended experiments will be included in the appendix.

Meta-review: This study proposed a self-supervised controllable generation (SCG) framework inspired by the cortical modularization and hippocampal pattern completion. Importantly, this model exhibits several brain-like properties, and good performance and generalization abilities. It seems that all reviewers provided positive ratings given sufficient interaction with the authors. I am inclined to accept this paper.
Decision: Accept (poster)