Connectome Mapping: Shape-Memory Network via Interpretation of Contextual Semantic Information
Authors: Kyungsu Lee
Haeyun Lee
Jae Youn Hwang
Venue: ICLR 2025 Poster (2025)
Keywords: neural representation
Primary area: learning theory

Abstract: Contextual semantic information plays a pivotal role in the brain's visual interpretation of the surrounding environment. When processing visual information, electrical signals within synapses facilitate the dynamic activation and deactivation of synaptic connections, guided by the contextual semantic information associated with different objects. In the realm of Artificial Intelligence (AI), neural networks have emerged as powerful tools to emulate complex signaling systems, enabling tasks such as classification and segmentation by understanding visual information. However, conventional neural networks have limitations in simulating the conditional activation and deactivation of synapses, collectively known as the connectome, a comprehensive map of neural connections in the brain. Additionally, the pixel-wise inference mechanism of conventional neural networks failed to account for the explicit utilization of contextual semantic information in the prediction process. To overcome these limitations, we developed a novel neural network, dubbed the Shape Memory Network (SMN), which excels in two key areas: (1) faithfully emulating the intricate mechanism of the brain's connectome, and (2) explicitly incorporating contextual semantic information during the inference process. The SMN memorizes the structure suitable for contextual semantic information and leverages this structure at the inference phase. The structural transformation emulates the conditional activation and deactivation of synaptic connections within the connectome. Rigorous experimentation carried out across a range of semantic segmentation benchmarks demonstrated the outstanding performance of the SMN, highlighting its superiority and effectiveness. Furthermore, our pioneering network on connectome emulation reveals the immense potential of the SMN for next-generation neural networks.

Published as a conference paper at ICLR 2025
CONNECTOME MAPPING: SHAPE-MEMORY NETWORK VIA INTERPRETATION OF CONTEXTUAL SEMANTIC IN- FORMATION
Kyungsu Lee
Department of Computer Science & Artificial Intelligence Jeonbuk National Univesity
ks1e jonu. ac. kr
Haeyun Lee
School of Computer Science and Engineering Korea University of Technology & Education haeyun. lee@koreatech.ac ac.
Jae Youn Hwang"
Department of Electrical Engineering & Computer Science Daegu Gyeongbuk Institute of Science and Technology (DGIST)
syhwang@dgist.dd .c.kr
ABSTRACT
Contextual semantic information plays pivotal role in the brain's 's visual inter- pretation of the surrounding environment. When processing visual information, electrical signals within synapses facilitate the dynamic activation and deactivation synaptic connections, guided by the contextual semantic information associated with different objects. In the realm of Artificial Intelligence (Al), neural networks have emerged as powerful tools to emulate complex signaling systems, enabling tasks such as classification and segmentation by understanding visual informa- tion. However, conventional neural networks have limitations in simulating the conditional activation and deactivation of synapses, collectively known the nectome, a comprehensive map of neural connections in the brain. Additionally, the pixel-wise inference mechanism of conventional neural networks failed to account for the explicit utilization of contextual semantic information in the prediction process. To overcome these limitations, we developed novel neural network, dubbed the Shape Memory Network (SMN), which excels in two key areas: (1) faithfully emulating the intricate mechanism the brain's 's connectome, and (2) ex- plicitly incorporating contextual semantic information during the inference process. The SMN memorizes the structure suitable for contextual semantic information and leverages this structure the inference phase. The structural transformation emulates the conditional activation and deactivation synaptic connections within the connectome. Rigorous experimentation carried out across range of seman- tic segmentation benchmarks demonstrated the outstanding performance of the SMN, highlighting its superiority and effectiveness. Furthermore, our pioneering network on connectome emulation reveals the immense potential of the SMN for next-generation neural networks.
INTRODUCTION
The past few decades have witnessed remarkable progress in deep learning (DL) research, largely driven by the significant advancements in graphics processing units (GPUs). These GPUs, with their exceptional computational powers, have played a pivotal role in accelerating the development of DI. methodologies. Fully Connected Networks (FCN) and Visual Geometry Group (VGG) have

---

Published as a conference paper at ICLR 2025
been introduced early baseline networks (Long al. 2015 Simonyan & Zisserman 2014).
Recently, deep neural networks (DNNs) have been applied to many tasks, such as YoLo for detection tasks (Redmon & Farhadi 2018) and U-Net for segmentation tasks Ronneberger et al. (2015). More recently, efforts to emulate the neuronal and cognitive intricacies of the human brain have continued to prompt the development of advanced DL models. For instance, |Woo et al. (2018) conceptualized the Convolutional Block Attention Module (CBAM), a novel architecture encapsulating the attention mechanism, and Vaswani et al. (2017) introduced the Transformer model processing sequential data. Moreover, an innovative perspective of treating images as sequences by Dosovitskiy et al. (2020) led to the development of Vision Transformer (ViT), which interprets images as sequential data.
Nonetheless, Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) have limi- tations when it comes to effectively leveraging contextual semantic information. In contrast, the human brain excels at visually interpreting objects' morphological attributes by actively incorporating contextual semantic information, thereby facilitating a holistic understanding of the surrounding environment (Trobel 2001 Farah 2000). For example, the human brain intuitively recognizes that the sky appears above the land, vehicles tend to be found on roads, and the road area is typically more extensive than the space occupied by cars. These contextual semantic cues contribute to our comprehensive understanding of the visual environment (Farah) 2000 Grill-Spector & Malach 2004). Similarly, artificial intelligence (AI) relies on the utilization of contextual semantic information to accurately identify objects within its environment (Brézillon) 1999; Chalmers et al. 1992). This
parallelism with human brain functioning implies that neural networks (NNs), like their biological counterparts, also require the ability to incorporate contextual cognition (Chalmers et al. 1992] Goodfellow et al. 2016, Nebauer 1998). However, despite recent advancements in NNs that integrate spatial information of objects (Jaderberg et al. 2015; He et al. 2016), there are still constraints in effectively incorporating contextual semantic information. This limitation stems from pixel-level clas- sification approaches that lack deep comprehension of the morphological attributes of objects (Guo) et al. 2022] Zheng et al., 202T). As a result, although NNs demonstrate proficiency in identifying objects based on spatial information, they have faced challenges in comprehending the intrinsic semantics |Waldrop 2019; Goodfellow et al. 2016 Guo et al. 2022).
In the early era, the pioneering work of Rosenblati (1958), Minsky & Papert (1988) introduced the concept of multi-layer perceptrons as an attempt to emulate the mechanism of human neurons. Furthermore, the concept of the NNs, interconnected perceptrons designed to mimic the complex mechanisms within the brain, has been introduced (Rumelhart et al.) [985] 1986). Additionally, a range of activation functions have been developed alongside NNs to emulate neurotransmission, including baseline activation functions such as sigmoid and ReLU, as well as advanced functions such as ELU (Clevert et al. 2015). GALU (Hendryeks & Gimpel (016). Swish (Ramachandran iet al. 2017). SeLU (Klambauer et al. 2017). and ASH (Lee et al. 2022). The activation functions in NNs have been developed to simulate neurotransmission by emulating the mechanisms of membrane and action potentials. However, it is important to note that the human signal transmission system is significantly more complex, incorporating both electrical signals within neurons and chemical signals in synaptic transmission, as depicted in Figure 1 Hence, the previous methods have been limited in simulating complex neural transmission systems. To address this, a novel Spike Neural Network (SNN) has been proposed in recent years (Tavanaci et al., [2199] Lee al., 2016). However, the lack of an optimal training algorithm for SNNs has been a limiting factor for their practical application in real-world scenarios.
Contribution We present two-fold objective aimed at (1) devising a novel neuro-mechanical neurotransmission model inspired by the signal transmission processes within the human brain and synapses, implementing an electrochemical methodology, and (2) developing a novel network that explicitly employs contextual semantic information during segmentation. Building upon the insights from the ensemble network (see Section 2). we introduce a novel network architecture that aims to optimize predictive performance. This innovative network dynamically adapts its structure based on the contextual semantic information embedded in the input image during the prediction process. Due to its ability to store and recall the optimal architecture in response to contextual semantic information, we have coined the term Shape-Memory Network (SMN) for this network. Furthermore, to facilitate structural modifications, we designed a novel conditional neuron capable of altering the inter-neuronal connections in response to received control signals that regulate inter-neuron signal transmission and neuronal activation (Fig. I In summary, our approach involved the two-fold design of network that explicitly harnesses and capitalizes on contextual semantic information. First,
2

---

Published as a conference paper at ICLR 2025
Conneciinn In Syrapte Beamne on th sightt Berig berigra the
Devctiatoo
Consut
Oggnall
ex6
Desctivate
Figure Schematic illustrations depicting the electrochemical neurotransmission process in the human brain, as well as the mechanism by which control neurons emulate neurotransmission In this mechanism, the architecture of neural connections control neurons dynamically modified in response to control signals, resulting in variable network architecture.
proposed conditional neuron with a novel signal transmission system facilitating structural variants upon the control signal. Second, this conditional neuron was then organically integrated into the SMN.
The main contributions of this paper are summarized below:
We proposed a novel network (i.e., SMN) that explicitly leverages the contextual semantic information for segmentation tasks by adjusting its architecture in a test-time fine-tuning manner and providing an appropriate structure for a particular domain.
To realize the explicit utilization of contextual semantic information, we developed a novel algorithm that reconstructs an entropy-map by integrating CAMs (Zhou et al. 2016).
In SMN, we introduced novel mechanism, the conditional neuron, incorporating a control signal and enabling contextually adaptive activation of inputs, drawing inspiration from synaptic mechanisms while maintaining its distinct computational characteristics.
Through rigorous mathematical justification, we provided a solid theoretical foundation for the concept of conditional neurons within the SMN. Furthermore, validate the practical performance of the SMN, we extensively tested it on multiple segmentation benchmark datasets, demonstrating its superior segmentation capabilities compared with other methods.
PROBLEM STATEMENT
Semantic Domain Gap Many studies have highlighted the effect of domain gaps hampering the predictive efficacy of DL networks (Shul 2015 Wei et al. 20188) The domain gap significantly emerges due to the diversity in sensors, environmental conditions during data acquisition, or variations in pre-processing methodologies, particularly in the field of computer vision (Regmi & Shah 2019)
Nam et al. 202IJ. The datasets, each with characteristic attributes, are typically considered distinct domain groups, yet [Pan et (2020) introduced the existence of intra-group domain gaps within one single domain as well. In this work, we aim to focus contextual semantic information as primary one of the multiple factors contributing to domain gaps. Contextual semantic information incorporates semantic attributes of objects such their size (proportional area occupied in the image, denoted as density in this paper), spatial location, and morphological form.
Suppose three subsets of U1, U2. and U3 C U in the multi-dimensionn information (U C I2" H XWX) Suppose U2 and U2 are similar, and U₁ and U3 are different in terms of contextual semantic information, such that [U1 1 U2l [U1 Us! where - B) indicate the average distance between all samples in the sets of A and B. Additionally, let L(M ((MM;U) U)) be a loss function that leverages the samples in U, using the parameters (0*) of DL. model (M), leading to
space of contextual semantic
3

---

Published as a conference paper at ICLR 2025
Proposition I. [U1 U2| [U1 Us! [(O":U2) <(O"; Us) where O argmings L(0' UU)).
;Us)
The Proposition I implies that the DL model optimized to a specific domain provides imprecise predictions the different domains. Thus, the problem statement that aims to find the DL network providing precise predictions disregarding domain gaps is formulated as below:
θ
argmin 6M
U1) and [U1 U2 UU1
Us!
L( M. U2) 0 and L(e"; (Us) (I)
Ensemble Model The trivial solution to bridge the domain gaps and achieve precise predictive accuracy is to employ an ensemble DL model for the prediction. Suppose different sets of Ug where (i = ...,...... N) indicating different N numbers of domains and DL models (M,), which is optimized to U, respectively. Then the ensemble model can provide precise prediction as below:
Proposition II. To precisely predict sample (u U, Uz). the ensemble model (M) can be derived using models optimized for each domain as M (u) = M.) (if u E U.).
Therefore, by employing appropriate model suitable to each domain, the ensemble model can provide precise predictions and thus realize Equation [ However, note that the ensemble model exhibits limitations, including heavy memory requirements and the inability to provide accurate predictions for domains that have not been trained.
Optimal Solution Many domain adaptation (DA) methodologies have been extensively studied
to mitigate domain gaps, encompassing various approaches such as transfer learning (Patricia & Caputo 2014 Kouw & Loog, 2018), generative DA (Bousmalis al. 2017 Hu al. 2018). and
unsupervised and self-supervised DA (Pan et al.) [020 Xu et al. 2019 Liu et a.. 2021] Bartler et al.) 2022). Among them, test-time adaptation (TTA) has emerged as prominent approach, similar to the ensemble model for addressing Equation]] 1 where the TTA models yield improved prediction accuracy by retraining the network with optimized parameters during the inference phase (Liu et al., 2021; Bartler et al., 2022). In this work, integrate the ensemble model with the TTA method to tackle Equation [ et al. 2021] Bartler et al.] 2022). Consequently, we propose a DL model that fine-tunes its parameters and dynamically adjusts the optimal architecture.
METHOD
3.1
SHAPE-MEMORY NETWORK
Encoder
Skip Connection
Decoder
Segmentation Map
fA
Counel Spead
Gilobal Average
Pooling
(x(x,y)
renonstruction
Actvation Maps Wifz
Dersity Prodiction (df))
Figure 2: Illustration of the architecture of the SMN, consisting of several components, including segmentation, density regression, entropy map reconstruction, and signal control.
Design Principle
The fundamental architecture of Shape-Memory Network (SMN) is designed to process and utilize contextual semantic information in visual data effectively. For instance, consider semantic segmentation task urban scene datasets. The input images typically contain multiple
4

---

Published as a conference paper at ICLR 2025
object classes with consistent spatial and contextual information: transportation infrastructure (roads, sidewalks) occupies the lower regions, architectural structures appear with specific scale constraints, and environmental elements (sky, vegetation) maintain consistent spatial positions.
To implement this, the SMN captures the contextual patterns via two primary computational compo- nents. First, the component implements density mapping. quantifying the proportional distribution of object classes within the input space. Particularly, in urban scene analysis, road surfaces typically con- stitute 30-40% of the pixel space, while vehicular objects occupy 5-10%. The density distributions are represented as statistical priors, leading to the network for validating segmentation predictions against expected contextual pattern.. Significant deviations from the learned distributions (e.g.. vehicles occupying 80% of the pixel space) are automatically flagged as anomalous configurations. The second component facilitates entropy mapping, quantifying information complexity in the spatial regions. The entropy mapping mechanism is particularly important for analyzing regions with high-class intersection probability, such as object boundaries or regions of class ambiguity. Computationally. regions exhibiting higher entropy values indicate areas requiring more sophisticated feature extraction and analysis than regions with uniform class distribution.
Architectural Design Regarding the design principles, formalize SMN structure with several key mathematical components. Particularly, the SMN employs conditional neurons to transform its structure during test-time adaptation (TTA) dynamically. Furthermore, we implement a self- supervised learning- based re-optimization method, utilizing the entropy-map as medium for loss minimization and explicit integration of contextual semantic information. While spatial information is effectively conveyed through skip connections, we focus on optimizing the network's 's contextual understanding by introducing density measurements that quantify the proportional distribution of object classes. Therefore, we focus on optimizing the network's s insight into contextual semantic information of input images by introducing density, representing the proportion of the occupied area in the image.
Definition I. Let 9e(h, w; I) be category (c) recognition function at pixel a in input (7), such that Oe wx T) is iff argmax TllR, = c, otherwise
Definition II. Let df "XXXX R be the density function of the target object (c) in semantic label @ E D C R xxx x3), such that dg) HW Σn Ew Da(h,, w; 1) with the image of height (H), width (W), and the number of categories (C).
Lemma Σ. di (g) since Σ. Σ" ΣW (h, 1) = HW.
The density-regressing pipeline facilitates two functions: (1) it enables the generation of Class Activation Maps (CAM) and entropy-maps for TTA optimization, and (2) it manages control signals for structural transformation based on input characteristics. By leveraging our novel approach, the CAM not only captures the visual attributes of target objects but can also be transformed into an entropy-map. This allows to optimize the SMN, by minimizing the similarity loss between the entropy-map reconstructed using CAM and the entropy-map generated in the segmentation pipeline.
Network Architecture Fig. illustrates the detailed architecture of the SMN based on multi-task and self-supervised learning for TTA. The SMN incorporates the segmentation pipeline and the density-reeresssoon pipeline. Here, the main task of the SMN is the semantic segmentation task to localize objects into segmentation maps, and the pretext task is to predict the density of the target object in a labblel manner. Note that the cogmition of the contextual semantic information could be explicitly realized by incorporating the spatial information conveyed from the skip connections and the mathematical morphology achieved by the recognition of density prediction. Subsequently, the cognition of contextual semantic information leads to the structural transformation of the SMN via the caroopy-map-baate optimization in self-supervised and TTA manner.
Let X C #x and y C I "X be the sets of input RGB images and corresponding segmentation labels, where H and W are the height and width of an input image, and the number of categories of inputs, and E : Hx x3 + II x W' xkk and D : [R xW'' xk + H X/XXWCCC the
encoder and decoder of SMN, respectively, where " xw' xk is the encoded feature space for fx in Fig. 2 such that (x) Subsequently, assuming the predicted segmentation map (M (x)) represents the probabilities obtained from the softmax output, the following constraint is imposed on the subsequent operations.
(Do (x) and fx E(x) where x E X' and is is the number of channels.
5

---

Published as a conference paper at ICLR 2025
Lemma IL. 0 & (a))lss, and thus Σ. (x) Ih,
Therefore, in the predicted segmentation map and density-regression pipeline, the density of the larget object is defined as below:
Definition III. Let d² R #xxxx3 + IR be the density function of the target object (c) on x E X' using the SMN, such that (x) HI ΣS E" 2 w; M(x) and Σ. d=(x) by Lemma II.
Definition IV, Let d' R 2XX x3
- IR be the density function of the target object (c) predicted density on E X using the SMN. such that df (x) Ex WE En,, Ju(h, w).
The SMN trained via three loss functions. Like in the general segmentation task, the predicted segmentation map (j = (D (x) where X') by the SMN is optimized to the segmentation label (y E y) via the cross-entropy loss function (CCE), such that (O (X)))) == Σn LCr(y. Additionally, the predicted entropy-map in the segmentation pipeline is optimized to the entropy-map reconstructed from the CAM, such that (0 X) :=
Optimization and Inference of SMN
Σx fasim [R(x), E, W [x))), where [[x)) -E (x))]lb,.nncc .n. log M M(x)) [l. Lesim is the structural similarity loss (Lu 2019), and EM is the entropy-mape -ww---aaaamt-eneeeee map-reconstruuting algorithm from the explicitl segmentation. Furthermore, the predicted density (d") by the SMN is optimized to the calculated density by the predicted segmentation map based on Definition II and Definition IV, such that (0 (X) == Σx (x) - dy))e. Therefore, the following constraint is imposed on subsequent operations.
CAM (See Sec. p.2p. The L2 implies explicitly utilizing contextual semantic information during
Proposition 111. d' in the such that 0 S > WF (h, 1 and E.E. W Σ,,., fa,h, w) by Lemma II.
pipeline of the optimized SMN satisfies the property in Lemma 1,
Proposition IV. The optimization of d, facilitates the incorporation of contextual semantic information into the parameters of the SMN, enhancing its ability comprehend and utilize such information effectively.
During the inference phase, the SMN re-optimized via a self-supervised manner only using Le. Particularly, C2 (.( X²) is applied, where 8M represents the parameters for the condition signals, thus indicating the manifestation of the structural transformation of the SMN at the inference phase. The TTA employing L2 yields primary outcomes: (1) TTA enables expedited re-optimization and minimizes memory requirements by utilizing a reduced number of training parameters, and (2) TTA enhances prediction accuracy by providing optimal architecture aligned with the contextual semantic information of input x.
3.2 ENTROPY-MAP RECONSTRUCTION VIA CLASS ACTIVATION MAP
In the previous work. Zhou et al. (016) introduced the CAM (Σ fi), directly indicating the importance of the activation spatial grid (h, w). Additionally, d' (x)) indicates the density of target objects in x (e.g., (x) = 0 implies that the absence of e x and (x) = implies that En W En.w (h, w) ΣN E. Wh, w) I Based on this, we expect the important area in density regression to be the contextual semantic information terms of mathematical morphology, and thus Σn Wgfn highlights the regions of the target object (c), and Ex W (h, w) implies the expected density at the spatial grid of (h, w)
the filled with c). Since we designed and trained 0 S d²(x)
Proposition V. The CAM Ce WE.fx) generated by the SMN is figured to highlight the regions of
Moreover, by stacking the CAMs for each category (c) as Σ f (h, w)] E RCC, the concatenated feature-map incorporates the important ratio for density prediction of each category c, and thus the factors could be normalized by using the softmax function to calculate stochastic variables d' (z)]l,,,, for the density of target and thus Σg d((x))) = 1. Therefore, the entropy- -mp is calculated by leveraging the probability as Ee (d' (x)))) log d' dx))),,,,) and the Theorem for the definition of EM (x) is formulated.
Theorem I. (M)l = E.E. (8 (d((x))),, log (x) where d'z))ln,,)
EA (EFN(
E. E.e WEIg(h..

---

Published as a conference paper at ICLR 2025
3.3 CONTROL NEURON
The control neuron functions as fundamental element within the adaptive architecture of the SMN, It processes information through three interconnected pipelines that collectively define its operation: (1) standard neural inputs from linked neurons analogous to those in traditional neural networks, (2) control signal based on predicted density distributions, and (3) self-activation mechanism gating signals. The three pipelines enable the network to dynamically adapt its structure in response to varying input characteristics, resembling how biological neural systems adjust connectivity patterns. During the processing of an input image, control neurons selectively engage or disengage connections based on contextual information, thereby achieving an optimal configuration for the specific input.
As Proposition IV, optimizing de promotes the interpretation of contextual semantic informa- tion. Consequently, employing (x) as control signal enables the structural transformation of the SMN by aligning its structure with the contextual semantic information. For instance, to achieve Proposition II, suppose three inputs of 21, 22. and ø3 E X, and let (21) and (x2) be in similar density distribution (z)), whereas d' dz(xg) be in different distribution, such that D&(d (x1), d (x2)) < DD (d(())), (xs)), where Dy Q) is Bhattacharyya distance. Addition- ally, suppose two structurally distinct networks of M,, which yields optimal performance for æ and 22 and M2 for I3. Then, the SMN predicts T1, 22, and 13 as below:
(a) x or = M2((r) (if = ma)
M(x)
Note that, the control signal affects the selection of optimal M, in Equation signal that interprets the contextual semantic information, we employed the predicted densities (d") as inputs based on Proposition IV, and formulated the control signal as below.
2 To design control
Definition V. The control signal of SMN is obtained by a linear combination of predicted densities (d").
To implement Definition V. the densely connected parameters were employed in the SMN (Fig. 2). Suppose (d' [d=())c==.2... E RIxC and the parameters of Weelemt....lmm...... I,.. E RCx,,, where Nen the number of control neurons. Subsequently, we de-
Signal
Selfactivation
fined the linear combinations of (x) and 1/, as Σe Vn)ede(z) E
Ingux
Ourgut
R as the control signal of 1h control neuron. Note that, the pa- rameter of is trainable, and thus the architecture of the SMN is aligned alongside the contextual semantic information of input (x) upon the predicted density in the train and inference phases.
Control Neurow
Figure 3: control neuron.
Furthermore, to effectuate the structural transformation in response to the control signal, each neuron in the SMN is designed receive three distinct input signals of inter-neuronal inputs, self-activation, and a control signal that incorporates contextual semantic information resampled from the density distribution (Definition V), as depicted in Fig. 3 Similar to the human brain, wherein the inter-neuronal transmission is facilitated via chemical signal trans- mission in synapses, inputs from other neurons the SMN are designed to emulate the chemical neurotransmission. Additionally, the control signal and self-activation are conceived to emulate the electrical transmission mechanism. Consequently, the control signal is intended to simulate the threshold for membrane potential, and self-activation is associated with the sustained stimulus to neurons. Furthermore, the output of the control neuron emulates the action potential.
Definition VI. The output of control neuron is A(Σ Input,) ((Control > (x)) (self-activation)) where A is an activation function, 4,, is a neural threshold of nh control neuron of SMN, is arithmetic multiplication operator, and is bit-wise or operator.
To design output responsive to a control signal and neural threshold, the terms of control signal and self-activation are implemented into the A(Σ Input,). which is the same output of the conventional neuron. The output signal is activated when the amplitude of the control signal surpasses an intrinsic threshold or when self-activation is true. The activation of the control signal indicates that the neuron has been subjected introduced to prevent the loss of informative activation caused by the sparsity problem. The criteria for informative activation derive from the premise that among interconnected neurons (feature maps),
stimulus exceeding the threshold, and self-activation is
7

---

Published as a conference paper at ICLR 2025
specific neuron hold more information than others. Let s. s and sa be imput signals, output signal, and control signal, and xn and A be the intrinsic threshold of neuron and the adaptive activation function filtering informative activation (Appendix), then Definition VI is formulated as: your = (H Xa) + (A( s)"))) E; 88. where (xr) is a Heaviside step function, where H (x) = max (e,,0) 0). However, since the An is not arithmetically connected to input signals, the Xn is not trainable, but the intrinsic threshold of X, should be trainable. To realize, we approximated the (z) into arithmetic form, and thus the output of the control neuron is formulated using sigmoid function (z)) and large value of
Theorem II. The output signal of the control neuron with input (s(")and control (s*) signals is formulated as (o(- (a(-2x1(s* Xm.) a( (-2anA(E,, s!") Σ, s."
Since the An is arithmetically connected the inputs, the À is trainable during the training phase. Therefore, the control neurons retain the intrinsic threshold to store the optimal architecture of the SMN, and the SMN adjusts its own architecture by fine-tuning e in run-time to align the appropriate architecture to contextual semantic information.
MAIN RESULTS
To evaluate the performance SMN, various benchmark datasets, such as acrial imagery datasets of Inria (Maggiori et al. 20176) and LoveDA (Wang et al. (202I) and scene understanding benchmarks of ADE20K (Zhou al. (017). Youtube- VOS (Xu et al. 2018). and BDD100K (Yu et 2020) were utilized. To demonstrate the general feasibility of the SMN for semantic segmentation, the datasets for scene understanding were employed. addition, since the density of the objects should be a crucial feature for aerial imagery, aerial datasets are employed to illustrate the strength of the
SMN effectively. Furthermore, the GTAS dataset (Richter et al. (016)) was utilized to evaluate the scalability of the SMN to the synthesis parsing. The more detailed descriptions of the datasets illustrated in the Appendix.
4.1 COMPARISON ANALYSIS
To compare the segmentation performance of the SMN, we employed compatible DL models, such the baseline models (Ronneberger et al. 2015 Xie et al. 2021). multi-Path models (Zhuang 2018
Bai & Zhou| 2020), state-of-the-at for segmentation models (Seg-SotA) (Wang et al. 2022b and video object segmentation models (VOS) Cheng Schwing (2022) Yang al. (2022). To evaluate the efficacy and superior performance of the SMN in the segmentation task, the baseline models, the SotA models, and the VOS models were employed. Additionally, the performance of SMN was compared using the multi-path models, which stand a similar role to the ensemble model. The detailed descriptions for the datasets and experimental setups are illustrated the Appendix.
Table 1:
Quantitative comparison analysis of SMN to other compatible deep learning models in terms of intersection over union (loU). The best performance values are highlighted in bold, and the second-best values are underlined.
Baseline Model
Multi.F
Sey
U-Net
SegFormer
LADDERNet
InternImage
Xmem
AOST
Inria
62.96%
67.97%
64.77%
64.51%
68.60%
66.69%
64.85%
69.30%
68 60%
72.72%
LoveDA
47.71%
51.33%
49.66%
48.25%
49.81
49.63%
51.29%
50.57%
50.40%
54.28%
ADF20K
42.61%
46.72%
52.66%
44.30%
$1.04%
51.67%
44.88%
52.06%
48.84%
55.76%
Youtube-VO VOS
72.12%
81.07%
86.0 16.04%
83.57%
85.13%
86.67%
83.36%
87. 77.09%
85 66%
88.66%
BDD11OK
36.69%
42.59%
41.13%
40.03%
47.25%
39.85%
42.69%
43.09%
43.81%
48.83%
GTAS
70.94%
70.90%
68.00%
06.6%
69.07%
76.58%
Quantitative Analysis Table illustrates the segmentation performance of the SMN compared to other deep learning models. Here, Ours-SA indicates the SMN without self-activation path in control neurons. iibvollveev two novel findings of (1) the SMN exhibits a superior segmentation performance than other compatible DL networks, and (2) the self-activation incorporates significant role the SMN. The SMN provides the 6.36% improved loU values on average, and also exhibits a powerful improvement with 13.15% improvement in maximum. Additionally, depending on the
8

---

Published as a conference paper at ICLR 2025
Image
Label
Baseline Model
Multi-Path
Seg- Seg-SotA
VOS
Ours
ADE20K
GTAS
LoveDA
Figure 4: Predicted segmentation maps by SMN (ours) and other competitive DL models.
existence of self-activation, performance difference of up to 7.51% is exhibited, indicating that self-activation drives significant contribution to the feature extraction SMN, implicating that self-activation plays compulsory role in improving the sparsity problem.
Quantitative Analysis Fig. exhibits the segmentation results of SMN and other compatible deep learning models using ADE20K, GTA5, and LoveDA datasets. Upon inspecting the semantic segmentation map predicted by SMN and other DL models, it is evident that the segmentation facilitated by the SMN is remarkably predicted precise, contrasting with other comparative networks. The quantitative analysis demonstrated the superior performance of the SMN in the segmentation task. Particularly, it is noteworthy that the SMN provides precise segmentation maps when segmenting small objects and objects of various sizes, distinguishing the SMN from other networks by precisely segmenting the boundaries of the objects. Therefore, the segmentation performance of the SMN, when compared to other competitive DL models, stand out in aerial images wherein the density of the objects significantly impacts the segmentation performance. Moreover, the exceptional adaptability of the SMN, highlighted by its ability adjust its network structure based on contextual semantic information dynamically, further underscores its superiority delivering precise segmentation, even under diverse and challenging conditions.
corroborated to
4.2 EXPLICIT UTILIZATION OF CONTEXTUAL SEMANTIC INFORMATION
Similarity of latriaaic Threshods
Fmnor Rose (9h)
Envor Row (19%)
Deasity Earged Objjects (c)
(f)
Entropy-Map
Figure 5: (Left) Similarity of the intrinsic threshold of control neurons containing the similar density of the target objects and (Right) samples of entropy-man -ma and reconstructed entropy-map.

---

Published as a conference paper at ICLR 2025
Fig. | offers an explicit depiction of contextual semantic information during segmentation by the SMN. Fig. Left) manifests the degree of similarity between the intrinsic threshold values of the control neurons. These neurons are engaged when predicting two disparate images, exhibiting analogous object densities. The premise for determining similarity rests on acknowledging a match when the concurrence level of a specific control neuron, operating in two distinct structures for the prediction of two different images, falls beneath a defined error rate. Therefore, a reduced error rate in similarity computation signifies imposing more stringent conditions. Nonetheless, Fig. []] Left) highlights a strong correlation between the intrinsic threshold values based on density, and thereby indicating that the structural transformations within the SMN are relevant and fluctuate in response to the contextual semantic information. Fig. Rightt illustrates the similarity between the entropy-map generated via the EM algorithm in the optimized SMN, and the entropy-map conveyed within the segmentation pipeline. The experimental results emphasize using contextual semantic information by the SMN during the segmentation tasks.
5 DISCUSSION
hyppe-paramette Tuning Note that the experiment did not aim to find the best-performing model with the fully searched parameters, but aimed to demonstrate the feasibility of the proposed deep neural networks and to show superior performance compared to other -tate-of-the-art (SotA) models. To search for the best parameters for the highest performance remained future works. Furthermore, among the trainable variables, zg. alpha,, and (22 are not significantly treated in this paper since those values cannot significantly affect the segmentation performance of the SMN. For instance, the values of a are optimized to nearly 10.0, which significantly approximates the sigmoid function to the Heaviside step function. Additionally, the zμ a hyperparameter of the ASH activation function, such that its values are significantly different by the convolution operations and the locations. Therefore, it natural that the self-optimization of 2 could lead to the optimal performance of the SMN.
Computational Complexity To implement the SMN for real-world applications, we address the computational complexity of the TTA mechanism. The current implementation requires optimization of matrix M E RCXN during inference, with time complexity O .NN and space complexity O . N), where T represents optimization steps (typically T S 5). C' denotes categories, and N indicates control neurons. While our current implementation achieves 32.8 FPS with 47.5M parameters and 549.8G FLOPs, we propose several optimization strategies to enhance efficiency. These include early stopping criteria (Ct + 1 - Lt < <), parameter pruning (Mpuuedd = M O (IM) > T)), and quantization (uaunt round(M (M 229) Preliminary experiments suggest these optimizations could reduce computational overhead by 30-40% while maintaining performance within 1-2% of current results. Future work will focus on developing lightweight TTA variants and memory-efficient implementations to further improve real-time performance.
6 CONCLUSION
In this paper, we proposed a novel deep learning network that emulates the brain connectome, which incorporates intricate neural connections, and explicitly leverages contextual semantic information during segmentation. To this end, during the training phase, the network is designed to memorize the optimal structural configurations for contextual semantic information, and to transform into the optimal structure suitable for the input's contextual semantic information during the prediction phase, leading to accurate predictions. To depict the explicit utilization of contextual semantic information for segmentation, we designed a novel optimization method based on the class activation map and entropy-map as illustrated Theorem I. Moreover, to implement the structural transformation of the network, we proposed a novel neuronal system called control neuron, illustrated in Theorem II. To evaluate the performance of the proposed network, we employed several semantic segmentation benchmark datasets, and the experimental results demonstrated the superior predictive performance of our method in the segmentation task. Furthermore, our research is foundational for the next- generative networks capable of emulating the human signal transmission system by incorporating test-tine sime-tmm adaptation methods and structural transformation, demonstrating the potential of the SMN for scalability in various applications.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
This work was partially supported by the 2024 innovation base artificial intelligence data convergence project project with the funding of the 2024 government (Ministry of Science and ICT) (S2201- 24-1002), and by the Korea Research Institute for Defense Technology Planning and Advancement (KRIT) through the Korea Government (MND) under Grant (R230206). This work was partially sup- ported by the National Research Foundation of Korea (NRF) grant funded by the Korea government Ministry of Science and ICT (MSIT) (No. RS-2024-003388889)
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In /2th (USENIX) Symposium on Operating Systems Design and Implementation ({OSDI) 16), pp. 265-283, 2016.
Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound images, data brief 28, 104863 (2020), 2019.
Fatemeh Azimi, Sebastian Palacio, Federico Raue, Jörn Hees, Luca Bertinetto, and Andreas Dengel. Self-supervised test-time adaptation on video data. Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 3439-3448, 2022.
Xing Bai and Jun Zhou. Efficient semantic segmentation using multi-path decoder. Applied Sciences, 10(18);6386, 2020.
Alexander Bartler, Andre Bühler, Felix Wiewel, Mario Döbler, and Bin Yang. Mt3: Meta test- time training for self-supervised test-time adaption. In International Conference on Artificial Intelligence and Statistics, pp. 3080-3090. PMLR, 2022.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of 20MPSTAT'2010, pp. 177-186. Springer, 2010.
Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3722-3731. 2017.
Patrick Brézillon. Context in artificial intelligence: a survey the literature. Comput. Artif. Intell., 18(4):321-344 1999.
Mateusz Buda, Ashirbani Saha, and Maciej A Mazurowski. Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by deep learning algorithm. Computers in biology and medicine, 109:218 0092218-225, 2019.
David J Chalmers, Robert M French, and Douglas R Hofstadter. High-level perception, representa- tion, and analogy: critique of artificial intelligence methodology. Journal of Experimental & Theoretical Artificial Intelligence, 4(3):185-22 211, 1992.
Ho Kei Cheng and Alexander Schwing. Xmem: Long-term video object segmentation with an Mikinson-shiffrin memory model. In Computer Axiin-EECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVIII, pp. 640-658. Springer, 2022.
Ajó Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint 15.0022, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11999, 2020.
Martha J Farah. The cognitive neuroscience of vision. Blackwell Publishing, 2000.
11

---

Published as a conference paper at ICLR 2025
Geoffrey French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation.
arXiv preprint
2017.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference machine learning, pp. 1 1180-1189 PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversaan training of neural networks. The journal of machine learning research,
2016.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
Deep learning. MIT press, 2016.
Kalanit Grill-Spector and Rafael Malach. The human visual cortex. Annu. Rev. Neurosci., 27: 649-677, 2004.
Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. arXiv preprint arXiv:2209.08575, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition. pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.00 150.046606. 2016.
Langing Hu, Meina Kan, Shiguang Shan, and Xilin Chen. Duplex generative adversarial network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2018.
Max Jaderberg. Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in neural information processing systems, 28, 2015.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 141.69.4,,.6980, 2014.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017.
Wouter M Kouw and Marco Loog. An introduction to domain adaptation and transfer learning. arXiv preprint arXiv: 2018.
Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in neuroscience, 10:508, 2016.
Kyungsu Lee, Haeyun Lee, and Jae Youn Hwang. Self-mutating network for domain adaptive segmentation in aerial images. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7068-7077, 2021.
Kyungsu Lee, Jaeseung Yang, Haeyun Lee, and Jae Youn Hwang. Stochastic adaptive activation function. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 13787-13799, Curran Asso- ciates, Inc., 2022. URL RtLLtttss///rreeeeinggge www.:gs....urreee /papr files/paper/ BD227111111441111111100555511111
Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Tit++: When does self-supervised test-time training fail thrive? Advances in Neural Information Processing Systems, 34:21808 -28200, 2021.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Yingjing Lu. The level weighted structural similarity loss: A step away from mse. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 9989-9990, 2019.
12

---

Published as a conference paper at ICLR 2025
Emmanuel Maggiori, Yuliya Tarabalka, Guillaume Charpiat, and Pieree Alliez. Can semantic labeling methods generalize to any city? the inria aerial image labeling benchmark. In 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp- 3226-3229. IEEE, 2017a.
Emmanuel Maggiori, Yuliya Tarabalka, Guillaume Charpiat, and Pierre Alliez. Can semantic labeling methods generalize any city? the inria aerial image labeling benchmark. In IEEE International Geoscience and Remote Sensing Symposium (IGARSS). IEEE, 2017b.
Marvin L Minsky and Seymour A Papert. Perceptrons: expanded edition, 1988.
Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 8690-8699, 2021.
Claus Nebauer. Evaluation of convolutional neural networks for visual recognition. IEEE transactions on neural networks, 94))6685-66, 1998.
Fei Pan, Inkyu Shin, Francois Rameau, Seokju Lee, and In So Kweon. Unsupervised intra-domain adaptation for semantic segmentation through self-supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3764-3773, 2020.
Novi Patricia and Barbara Caputo. Learning learn, from transfer learning domain adaptation: A unifying perspective. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1442-1449, 2014.
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXiv:1710.05944, 2017.
Joseph Redmon and Ali Farhadi. Yolov3:
An incremental improvement. arXiv preprint
arXiv:1804.0227 2018.
Krishna Regmi and Mubarak Shah. Bridging the domain gap for ground-to-acrial image matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 470-479, 2019.
Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), European Conference on Computer Vision (ECCV), volume 9906 of LNCS, pp. 102-118. Springer International Publishing, 2016.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Frank Rosenblatt. The perceptron: probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386, 1958.
David E Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323(6088):333-536 1986.
Le Shu. Graph and Subspace Learning for Domain Adaptation. PhD thesis, Temple University. Libraries, 2015.
Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. arXiv preprint 180.iv::802.0000 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409, 1556, 2014.
Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothée Masquelier, and Anthony Maida. Deep learning in spiking neural networks. Neural networks, 11 111:47-63, 2019.
13

---

Published as a conference paper at ICLR 2025
Jonathan D Trobe. The neurology of vision. Oxford university press, 2001.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell, Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 141.3344, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
M Mitchell Waldrop. What are the limits of deep learning? Proceedings of the National Academy of Sciences, (1664))1074-10000 2019.
Junjue Wang, Zhuo Zheng, Ailong Ma, Xiaoyan Lu, and Yanfei Zhong. Loveda: A remote sensing land-cover dataset for domain adaptive semantic segmentation. arXiv preprint arXiv:2110.08733, 2021.
Qin Wang. Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7201-7211, 2022a.
Wenhai Wang. Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. arXiv preprint MxXiv:2211.05770 2022b.
Wenhui Wang, Hangho Bao, Li Dong, Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022c.
Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. Person transfer gan to bridge domain gap for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 79-88, 2018.
Sanghyun Woo, Jongchan Park, Joon- Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 1207- 2021.
Jaaoong Xu, Liang Xiao, and Antonio M López.. Self-supervised domain adaptation for computer vision tasks. IEEE Access, 7:156694-156706, 2019.
Ning Xu, Linjie Yang, Yuchen Fan, Dingeheng Yue, Yuchen Liang, Jianchao Yang, and Thomas S. Huang. Youtube-vos: A large-scale video object segmentation benchmark. CoRR, wbs//809, 0032273 2018. URL/attp://axxii
org/ abs /2//11110 (09..03322
Zongxin Yang, Jiaxu Miao, Xiaohan Wang, Yunchao Wei, and Yi Yang. Scalable multi-object identification for video object segmentation, 2022.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2636-2645 2020.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr. et al. Rethinking semantic segmentation from a cequence-to-sequeree perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6881-6890, 2021.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
14

---

Published as a conference paper at ICLR 2025
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633 633-641, 2017.
Juntang Zhuang. Laddernet: Multi-path networks based on u-net for medical image segmentation. arXiv preprini arXiv: 1810.00800 2018.
15

---

Published as a conference paper at ICLR 2025
A SHAPE-MEMORY NETWORK
Mechanism To illustrate the mechanisms of the Shape-Memory Network (SMN), we first define the mathematical notations and expressions. Let IE and ID be the encoder and decoder of the SMN, respectively. Then, the encoded feature-map (f&) would be represented as E[,, = [ where I, I C "xxx x3 is the input image with its height (H) and width (W). and C is the number of feature-mas map, and the final output, the segmentation map (Y; E Y C R "x**),,, is represented as Y = [D[[+]) = (TD o (T),, where is the number of categories In the pipeline for the class activation map (CAM), the encoded feature-maph are average-pooled, such that Ex., fu y) represents an individual feature. Here, suppose W for the weight for the density-regression for category (c), and then density prediction(d : RHHxxx3 calculated as d (1.) = >x W Jk(r, y) = Ex W& W ZE, (E(Ti), such that mapped to the control signal (s E R
the datasets.
RR is
RC Furthermore, the predicted densities for each category with the number of individual pixels of feature-maps in ID (N) via dense layers with the trainable matrix (M Cx such that ~8 (d M,, where is the matrix multiplication. Subsequently, the 5t is imported into the D. and thus the final prediction of the SMN is implemented in detail as below:
Y; = (T(T(T.); 8*)
= =D (R(::
Σ WE B(TA) M)
Note that all elements of the segmentation map (Yi) are the softmax output, such that O S Vill S Therefore, we can define the trainable parameters of the SMN as (1) parameters of encoder and decoder, such that Oz and Op: (2) parameters (WF) for the CAM pipeline a dense layer, (3) matrix (M) to map the predicted density to control signal. Note that only the matrix M is optimized the inference phase change the structure of the SMN. To summarize, the key outputs by the SMN listed as below:
Segmentation Map: Y, D2=D(B(L.))
Σ WE Σ
E(I) MM)
Density- d'((")
Σ W B(T.)
(4)
2.1
Class Activation Map: C²((') Σ WCfk Σ WEB(1')
Enaropy-map: E(T') -Y; log Y:
In addition, illustrated in Sec.3.2 in the manuscript, remember that the entropy-map function (EM(!') reconstructs the entropy-map from the CAMs (C" for == 1,2, ..., C) as below:
EM(T)) = Σ de (T log log l (T')ll, where d (T = E. WEE(
(5)
Σe el WCE(
The training process of the SMN is illustrated in Sec 3.1 of the manuscript. The predictive procedure of the SMN in the inference phase listed the following:
1. The SMN generates the pscudo-labels for the segmentation map and density-regressions.
2. The entropy-maps are generated via the pipeline of the segmentation (E(I") and the recon- struction algorithm (EM(T')).
3. To fine-tune M, minimize the similarity loss (Lesim) between E(T)) and EM(T'), such that M' = argminnsaim(E(7 ((('))
M
16

---

Published as a conference paper at ICLR 2025
4. The SMN with M' predicts the final output as: [(L): EE W B(1,) M
It's important to note that during the training phase, M is trained, with each individual M being mapped unique domains, representing different characteristics of contextual semantic information. Hence, M, represents sub-domain XG where the intersection of all X; equates to the dataset X, but no intersection exists among individual X, During the inference phase, the similar M, derived by fine-tuning the Shape-Memory Network (SMN) for sample æ; E X. This process showcases how SMN redeploys its saved structure by modifying its architecture, which led to the network being dubbed the Shape-Memory Network, and Fig. in the manuscript verifies the effective utilization of similar M, Additionally, an illustration of the optimization of M. as well as the provision of the final predicted segmentation map by the SMN, presented in Algorithm T
Algorithm 1: Fine-tuning and Inference of the Shape-Memory Network
Input
sample z; in test-set (X' C R x5), such that 2; E X, where H and W are height and width, respectively, and the pre pr-ttrineed SMN (M).
Output
Predicted segmentation map (yı E C RRCR corresponding to input (r²), where C' the number of category.
Assumption
:x'
O X. where N is the number of subset. N N X; 0. indicating that the Xi represents distinct domain.
J' + (B()): W B(1)) M))
Predict pseudo-labes +/
G t yy' logo
/+ Entropy-map by segmentation pipeline +/
C2 ('' (T))lle
/* Entropy map by algorithm */
M' < argmin EM(()))
/a Fine-ture M +/
D
B WE Ex, ((I))-MM))
To summarize, the mechanism of the SMN is (1) to store the appropriate architecture for a certain domain; (2) to restore its structure corresponding
the input domain by fine-tuning a parameter; and (3) to provide precise prediction to the input image. Here, the structural mutation is achieved by fine-tuning the control signal that supervises the connections of neurons. The neural connections in the SMN are activated or deactivated based on the control signal, and thus fine-tuning the parameter that supervises the control signal enables the SMN modify its network structure corresponding to the input image.
Tllment (Neuren)
Ekenent (Neurea)
Desctrvtee
Feature-map
Appendix Figure 1: Schematic illustration of control neurons in the feature-maps.
17

---

Published as a conference paper at ICLR 2025
Appendix Fig. illustrates how the control signal achieves the structural mutation. By activating and deactivating the output of each neuron, which is the individual element in a feature-map. the condition signal changes the structure of the SMN, and thus the control signal supervises the adaptive domain adaptation with respect to the contextual semantic information of inputs. Thereby, the structural adjustment by the control signal fine-tuned with the contextual semantic information bring out superior segmentation performance of the SMN.
Contributions
To summarize, our contributions, this paper, are listed below:
Construction of Shape-Memory Network. We designed the shape-memory network that can explicitly interpret the contextual semantic information by employing the run-time adaptation method via structural modification.
Design of Control Neuron. We newly devised a control neuron that can adaptively change the connections other control neurons, leading to the implementation of structural modi- fication of the SMN. This mechanism represents the close emulation of the human brain connectome and synapse mechanism.
Implementation of Entropy-Map Reconstruction Algorithm. For the explicit interpreta- tion of the contextual semantic information in the SMN, we newly devised the entropy-map reconstruction algorithm to train the SMN using the class activation maps regarding the con- textual semantic information. The devised algorithm incorporates the contextual semantic information in the training of the SMN.
B ENTROPY-MAP RECONSTRUCTION FROM CLASS ACTIVATION MAP
In the previous research (Zhou al. 2016), was revealed that the Class Activation Map (CAM)
identifies the regions of significant relevance to the primary task. As result, when tasked with density regression, the CAM influenced to concentrate on areas specific to the target object (c), leading to the derivation of Proposition V. In this context, density refers to the proportion of pixels in the input image representing the target object compared to the total pixel count. response, we developed the Shape-Memory Network, which integrates multi-label density regression tasks to yield multiple CAMs for each category (c). Parameters specific to the density regression process for each category then multiplied with the encoded feature- map. and then, CAM for each category is generated. Subsequently, we incorporate the CAMs that exhibit attention areas for each category to produce pseudo-entropy-maa
Goid
Clas Astvatoe Map Wilclea)
Resetreed
Appendix Figure 2: Schematic illustrations of reconstructing entropy- map from the class activation map (CAM). Each CMAs for each category are leveraged to generate entropy-map via probability- based normalization method.
18

---

Published as a conference paper at ICLR 2025
Control Signal
Sclf-activation
Input
Output
Control Neuron
Appendix Figure Schematic illustration of control neurons.
C CONTROL NEURON
Note that the control neuron represents element in feature-map (See Appendix Fig. I). Therefore, the control neuron refers to pixel-wise activation rather than convolutional weights. Suppose there are numbers of control neurons the SMN, and each control neuron has individual intrinsic threshold value (A₂)) for nth control neuron. The output of the control neuron is activated when (1) the value of
the control signal (s is above the intrinsic threshold value, such that s 2 X, or (2) self-activation is true. Therefore, the at is activated when the following condition is satisfied:
(eeff-activation)){{ Ax))
(6)
Here, the self-activation indicates that the current control neuron (pixel element) is more informative than other elements in the same feature-map. To avoid the loss of the informative features from the feature extraction process, the self- activatiiooon is designed. Therefore, the logical ()) operator is placed in Eq. 66 Suppose fü for the encoded feature-mat by E. In f& with its height (H₂)) and width (W+),, the number of control neurons W&, and the ni control neuron more informative when the condition below satisfied:
oui is in top k% among all elements in fk.
In the previous study Lee et al. 2022, the sampling elements met the Eq, is formulated as below:
put S1
is informative when sht m 2 V
(8)
where f, and v,, are the mean and the standard deviation values of all elements in fi. and the refers to statistical z-value for the Z-table corresponding to 6%.
To generalize, suppose feature-map (F) in the SMN, and the Th control neuron in F. Therefore, we can define the function (g(s)) out, F)) that determines whether gout is informative or not as below:
(if out N my 2 Vv)) ((else)
g F)
In this case, the is not trainable since the is not arithmetically placed, but in the conditional statement. To make the 2ạ be trainable, the Heaviside step function and its approximation are utilized. Additionally, the Heaviside step function (H()) is approximated to the sigmoid function (r(-2aar) with large value of a. Therefore, we formulate the Eq. as below:
g(s oot. F) H(g F) m. 24 V
(10)
=o ( 20 s.tt (g my 24 *v))
Here, and 2 trainable. Therefore, the self- self-activation that determines the current condition neuron is informative or not is trained during the training phase, and the pre-trained self-activation determines the activation of the condition neuron in the inference phase.
19

---

Published as a conference paper at ICLR 2025
Furthermore, another condition in Eq. related to the control signal is formulated using the approxi- mation of the Heaviside step function as below:
o(
20 - Xn))
(11)
In addition, the logical or operator replaced by the addition operator in arithmetic and analysis, and thus the Eq. is substituted as below:
o( 2a(g(( F)) mF + 21 *V=)) + or 2a(s" An))
(12)
Therefore, let the input signals s and thus the final output (ss"") value of the n control signal is provided as below:
out = s ( 20:1 ; F) mF + 24 vF)) -2az(s a -n))
(13)
Here, in addition to the trainable parameters Appendix A, the 2k, at, and 02 are trainable, and 23 represents the adaptive threshold to discriminate the informative features, and 01 and a2 the conditional values for the approximation. In the empirical analysis and experiments, the value of is trained at nearly 10.0.
D EXPERIMENTAL ENVIRONMENT DESCRIPTION
The experiments were implemented in the Apple Macbook Pro with M1 Max and 64GB unified memories. Besides, we developed our neural network and the state-of ofate--o--he-artt deep learning models using Tensorflow (for ARM processor) version 2.9.0 (Abadi et al.) 2016) for precise implementation. For the training. the batch size (Bottou 2010) of the training was set to 32, and the Adam optimizer was utilized with the default values of all parameters (Kingma & Ba 2014). Every parameter of the neural networks and the optimizer was initialized with the Gaussian distribution, of which the mean and the standard deviation values are 0.0 and 1.0.
Implementations
Comparative Models To demonstrate the segmentation performance of the Shape-Memone Net- work (SMN), four groups of deep learning models were utilized as shown in the followings; (1) Baseline models including the early vanilla models of U-Nee(Roneberrge
al. (2015), and Seg- Former (Xie et al.] 2021); ; (2) Multi-Path models for the segmentation task including LADDER-
NET (Zhuang 2018) and MPDNet (Bai & Zhou 2020); (3) SotA models for the segmentation task, including Interimage (Wang et al. 2022b) and BeiT-3 (Wang et al. 2022c); (4) SotA models for the video object segmentation (VOS), including Xmem (Cheng Schwing 2022 and AOST (Yang
et al. al.l1.2000 (2022). The baseline networks were compared to demonstrate the standard feasibility of the SMN for the segmentation task. While the SoTA models, used here, were utilized exhibit superior segmentation performance of the SMN for the benchmark datasets of scene parsing and autonomous driving, including VOS. Here, the best-performing SotA models were selected by referring to Kaggle benchmark lists. Additionally, the multi-pat models were employed to compare the SMN in terms of the ensemble models for Eq. (1) in the manuscript.
Dataset In the experiments, five categories of distinct datasets were employed to evaluate the segmentation performance of the SMN compared to other baseline and SotA models; (1) Scene parsing benchmark using ADE20K (Zhou et al.) 2017) and Youtube- VOS (Xu al.) 2018): (2) Autonomous driving using BDD100K (Yu et al. 2020): (3) Aerial image datasets of Inria (Magiooi et al. 2017b ab and LoveDA (Wang al. 2021): l): (4) Medical Imaging datasets using MRI for brain tumor (Buda et al., 2019) and ultrasound dataset for breast cancer (Al-Dhabyaní et al.) 2019): (5) Synthetic images of GTAS (Richter et al. 2016). To demonstrate the general feasibility of SMN for semantic segmentation, the datasets for scene understanding and autonomous driving datasets were utilized. In addition, since the density, which is a crucial feature for SMN, of objects most important in the segmentation of aerial images, the benchmarks using aerial images were utilized. Furthermore, to evaluate the scalability of the SMN, the medical imaging datasets and benchmark for the synthetic images were employed. Note that since the density of the target object, especially the disease area,
20

---

Published as a conference paper at ICLR 2025
is significant key feature in medical imaging, the SMN could be expected to provide its superior segmentation performance in the medical imaging field. Furthermore, the precise segmentation performance of the SMN could provide the potential for transfer learning and extensibility to large- scale models. For training models, the images in each dataset are divided into ten-fold for the k-fold cross-validation.
Comparison to Domain Adaptation Models A great deal of DA methods, such as adversarial
training (Ganin al. 2016), maximum mean discrepancy minimization (Tzeng al. 2014), unsuper- vised DA (Ganin & Lempitsky 2015), and self-ensembling (French et al., (017). have demonstrated success reducing discrepancies between distinct source and target domains. However, methods typically assume the availability of labeled source domain data and unlabeled target domain data during the training phase, a condition that may not hold in real-world scenarios. Test-time DA (TTDA) methods, in contrast, aim to refine models at the inference stage, by leveraging the test data distribution without explicit access to the labels. Techniques such as transductive parameter transfer (Shu et al. 2018), and test-time self-supervised learning (Azimi al. 2022; Lee al.) 2021 Wang et al. 2022a) have been proposed to bridge the gap between the training and test data distributions.
Despite the promising results achieved by the aforementioned methods, they are primarily designed for addressing domain discrepancies between two
distinct domains, rather than within single domain. To apply the domain adaptation method, two significantly distinct domains should be identified. However, in our study, the key factor for the domain discrepancy contextual semantic information, and the contextual semantic information could be identified by the deep learning models, not by the human, and thus the labels for the different domains regarding the contextual semantic information could not be provided. Therefore, despite the promising performance of the domain adaptation decreasing the domain gap, the domain adaptation method could not be applied and implemented to resolve the issues addressed in the problem statement (Sec. 2).
E EXPERIMENTS
Verification of M Appendix Fig. iillsstrates the similarity between M, and M2 for samples of x; and zj the same domain X' alongside three error rates. This experiment was conducted to measure the justification that fine-tuning M could represent similar or different architecture for the SMN within different domains.
To measure the similarity, the following function is devised:
[1 (if [w-vl <r) r)
((r)y:))=
(14)
(else)
where 0 < < represents the error rate, and thus S represent if two elements is within the error rate. Here, the Similarity of the Intrinsic Threshold is calculated below:
Appendix Table 1: Detailed description of the datasets. To validate, 10-fold cross cross-validation was used.
Dataset
Samples
Train
Test
Category
ADE20K
27,574
24,817
2,757
150
Youtube- VoS
7,945
7.150
795
65
BDD100K
8,000
7,200
Inria
144,000
129,600
14,400
2
LoveDA
4191
3.772
419
BrainMRI
7.858
7,073
785
BUSI
789
709
80
GTA5
24,966
22,470
2,496
27
21

---

Published as a conference paper at ICLR 2025
Similarity of Intrinsic Thresholds
85
Error Rase (5%)
Error Rase (10%)
Error Rate (15%)
75
10
20
30
40
30
60
80
90
Density of Target Objects
Appendix Figure 4: Similarity of the intrinsic threshold of control neurons containing the similar density of the target objects.
Σ My
(15)
the error rate decreases, the Similarity of the Intrinsic Threshold guarantees a higher similarity, whereas large value of the eπror rate is rough condition. Therefore, Appendix Fig, 4 verifies that the fine-tuned, M exhibits similar values regarding the same domain.
The SMN contains small number of parameters compared other state-of-the of-the-art models, but the SMN significantly provides precise prediction the segmentation task due to its effective fine-tuning mechanism. Additionally, despite the fine-tuning mechanism of the SMN, the SMN exhibits efficient FPS due to only a small number of parameters (M) being optimized in an inference phase.
Segmentation Performance This section illustrates the evaluation results of our model compared to other deep learning models, including baseline models, multi-path aalti-path models, SotA models for segmentation, and the SotA models for VOS.
In addition, the figure below illustrates the samples of the predicted segmentation by the SMN and other comparative models in eight datasets.
AOC20K
ADE20K
ADE30K
22

---

Published as a conference paper at ICLR 2025
ADEBOK
ACE20K
Jee LOMAHES.0
ASS (HUOHER
Youbuse-Vo0 V05
Youlube-VDC VDS
Youhube VD5S
Youtube VB5
Youtube VDS
BO01
L-mte PHote-BL.0
Day
J0n (W00HBR22)
A00 o9comgh.200
Our Phos-atte
Lomee Proterda se0
Sareer SreA Pa0k-55
JWN'S CROOHE-e
Oer watt
Loteer proskint BN
Herar de (eordd22))
AMI ORcOND
Oer gresk
23

---

Published as a conference paper at ICLR 2025
B001000 DOK
481
UPP (4
MP- Desdet 23)
ASST we
Jebeaa jostwer-e
wPP JAL
Xee LOMANBRR
ASI (WIGHERO
Infe
Indo
Info
Infl
Innio
LovellA
1-Me 2100-21100
JMN EWNOHEX-N
N0I OIND-1SAR)
Our SrOL MOC
LovelAk
Lome Prok-33.00
24 Saree TPAA-
AMN'S CHOPHOONN
N001 Oncomdo.20
Oer Shou wot.
Lote pootes
1NN' (90H4520
AX0 Orcars210
Oer gros
24

---

Published as a conference paper at ICLR 2025
LoweM
joddss
UPP SALL
Syfwee 2ML BET
IPP 94L-BU7
Jee LIMAR122
ASI (WU2H122))
Oet POLLIE
BronnN
Brohwa
Broinats
Bronaits
Bronali
BUS
Herat (Pop-123))
(roperalt))
208 (100-)2200
601 (9420120
Out Prou
Lote Prote22000
Sky Semee Paac
soime Prec-h11
Henter (rope11.0)
(roperack)
JAN'S CROPETAN
A01 09com05.00
Ous Phouvetas
Lote porter 90
Selwew you
www Orea
Heree we (10))711))
Mnd (ropertt))
Jwn's reosered
AXI OrcanD
oer SwoLww
25

---

Published as a conference paper at ICLR 2025
BUB
Horde (edlenecs)
MP- (endertity
Xee (40208715
ASI (6:30DAA))
BUB
Jee CHUHET
CTA5
CTA5
GSAS
GIAS
CLAS
Appendix Table 3: Segmentation Results of the SMN and other comparative models.
Network Complexity The SMN contains the trainable parameters of (1) parameters of encoder and decoder, such that 0% and fp; (2) parameters (W() for the CAM pipeline dense layer; (3) matrix (M) to map the predicted density to control signal. To verify the feasibility and scalability for real-world application,
compared the SMN to other deep learning models terms of the number of parameters (# of Param), the FLoating point Operations Per Second (FLOPs), and Frame Per Second for generating predictions (FPS). The predictions were performed using the Apple Macbook Pro with MI Max and 64GB unified memories.
26

---

Published as a conference paper at ICLR 2025
Appendix Table Comparison analysis in terms of mean IoU.
ADE20K
Youtube- VOS
BDD10OK
2*Baseline Model
U-Net
42.61% % (#33755
77.12% (43.38)
36.69% (+1.41)
65.84% (#1.8))
SegFormer
46.72% (4:3.93)
81.07% (+22377)
42.59% (#:3.83)
65.99% (#2.17)
2*Multi-Path
LADDERNet
52.66% (#33444)
86.04% (+:3553)
41.13% (#:3.44)
68.64% (=3.61)
MPDNet
44.3% (+3.11)
83.57% (+2.544
40.03% (+3.6)
70.69% (+3.42)
2* SotA
Internlmage
51.04% (#:2.67)
85.13% (=2.2)
47.25% (#3.5)
70.94% (=3.27)
BETT-3
51.67% (+:2.08)
86.67% (+:3.02)
39.85% (+1.222)
70.9% (+3.23)
2*VOS
Xmem
44.88% (# 1.78)
83.36% (#:3.11)
42.69% (43335)
68% (#2.31)
AOST
52.06% (#33112
87.09% (+2.04)
43.09% (#141)
71.06% (=2.32)
2*Ours
Ours
48.84%
(+:2.42)
85.66% (+11004)
43.81%
(+33599)
69.07% (=2.97)
Ours
55.76% (+:2.9)
88.66% (+:2.44)
48.83% (1:1114)
76.58 % (# 1.14)
Inria
LoveDA
BrainMRI
BUSI
2*Baseline Model
U-Net
62.96% (#3334)
47.71% (43.12)
75.11% (#1.64)
63.69% (=2,55)
SegFormer
67.97% (#3.19)
51.33% (+:3334)
74.28% (#:336)
71.41% (=3.19)
2*Multi-Path
LADDERNet
64.77% (#:3009)
49.66% (43.599
69.75% (43.34)
60.36% (+2.88)
MPDNet
64.51 % (#:1.09)
48.25% (+:3.43)
67.43% (4:3.89)
67.51% (=1.28)
2* SotA
Internlaage
68.6% (#:.27)
49.81 (11.17)
76.08% (13.82)
70.47% (=2.27)
BETT-3
66.69% (=: 1.82)
49.63% (#1.75)
66.08% (#3.78)
67.46% (=1.63
2*VOS
Xmem
64.85% (1:2.78)
51.29% (1:3112)
61.57% (d:3.04)
67.24% (#3,41)
AOST
69.3% (#3.75)
50.57% (+3.43)
75.22% (#:2.33
60.16% (+3.2)
2*Ours
Ours SA
68.6% (+2.57)
50.4% (12.69)
68.86% (+:2.95)
72.99% (=3.42)
Ours
72.72% (=11006)
54.28% (#:1.31)
74.82% (+1.77)
75.22% 1.24)
Appendix Table 4: Comparison analysis in terms networks' complexities.
U-Net
SegFormer
InternImage
BeiT-3
Xmem
AOST
SMN (Ours)
Resolution
512 x 512
512 512
384 x 384
384 381
512 512
512 x 512
512 512
# of Param
31.0M
64. 1M
335M
1843M
65.6M
47.5M
FLOPs
224.6G
95.7G
163.2G
2859.9G
549.8G
FPS
42.5
30.7
42.6
10.2
41.7
35.2
32.8
27

---

Published as a conference paper at ICLR 2025
F DISCUSSION
Extension to Other Tasks Our framework demonstrates significant potential for extension beyond semantic segmentation tasks. As illustrated in Appendix Fig. [ the SMN architecture can be generalized via modular design approach: maintaining the encoder with control neurons while allowing customization of the header and pretext task for specific applications. The adaptability of the SMN is achieved by two key components: (1) the latent features extracted from the encoder and (2) the control signals derived from the features. The latent features, representing high-level semantic information,
processed through task-specific headers to generate appropriate outputs (e.g., class probabilities for classification, bounding box coordinates for detection), while the control signals guide the structural adaptation of the network based pretext task appropriate for the target application. While our segmentation implementation uses density-based pretext tasks to identify spatial information, other applications might employ different self-supervised learning objectives for instance, classification tasks could utilize feature correlation learning based on variational auto-encoder, while detection tasks might benefit from pretext tasks using object localization patterns.
The adaptability of our SMN architecture extends beyond semantic segmentation tasks through its modular design approach, as illustrated in Fig. [ The architecture maintains its main feature extraction mechanism with control neurons while enabling task-specific customization through two key components: the decoder (header) and the pretext task. This design principle allows the network to be adapted for various computer vision tasks while preserving the benefits of our control neuron mechanism.
Encoder
Header
Input
Output
GAP
Control Signal
Latent Features
Control Signals
Pretext task
Appendix Figure 5: Generalized pipeline of SMN for various computer vision tasks.
In our preliminary study, we demonstrate this adaptability in object detection tasks, where we modified only the decoder while maintaining the control neuron mechanism and semantic information based optimization. Our initial experiments on the COCO dataset show competitive results compared to recent state-of-the-art detection models like DiffusionDet, achieving 47.43 AP, 65.64 AP and 52.21 AP7s- Notably, our approach achieves this performance with minimal architectural modifications, demonstrating particularly strong results in AP, (63.24) for large object detection. For detection tasks, we leverage the same semantic information optimization strategy as used in segmentation, demonstrating the transferability of our mechanism across different vision tasks.
COCO
AP
AP30
AP7s
AP,
APm
AP;
DiffusionDet (1 @ 500)
47.18
65.74
51.42
31.18
50.19
62.24
DiffusionDet (4@ 500)
47.36
65.62
52.13
30.72
50.37
63.18
Ours (SMN)
47.43
65.64
52.21
30.8
50.39
63.24
The flexibility of the SMN extends further through our generalized pipeline (Fig.5) where the pretext task can be customized for different applications. While segmentation and detection tasks benefit
28

---

Published as a conference paper at ICLR 2025
from semantic information optimization, other computer vision applications may require different self-supervised learning approaches. For classification tasks, we are exploring various pretext task approaches leveraging auto-encoders and VAE architectures, including feature correlation learning between augmented views, rotation prediction, and solving jigsaw puzzles of image patches. This modular architecture design, separating the core feature extraction mechanism from task-specific components, ensures that the primary strengths of our approach remain effective across different applications. The empirical experiments in both segmentation and detection tasks and additional ongoing exploration in classification demonstrate the broader applicability of our brain- maiin-nsspired approach across various computer vision tasks, with the selection of appropriate pretext tasks being the key consideration for each specific application.
Reproducibility More detailed experimental results, including class-wise IoU values, and the code for the SMN will be available at https//giihbbbcomm/mooooooo//ppp
29


Reviews:
[{"review_id": "s1CtqxXXB4", "reviewer": "Reviewer_tQxH", "summary": "The paper is about improving semantic segmentation by remembering past semantic context of the dataset used in training to enable live adjustment during inference influenced by the semantic context. In simple terms, suppose the training data for semantic segmentation of 3 object categories (A,B,C) reveals that Object A has a density of 0.3, B a density of 0.6, and C a density of 0.1 meaning the predictions for B should occupy nearly 60% of the image. This kind of semantic context can be estimated by a density map derived during training and used to influence the adjustment of parameters during inference in a self-supervised fashion. \n\n\nThe rest of the wrapping and tie-in to neuroscience and brain Connectome is weak at best and add no further value to the paper.", "questions": "It seems to be the datasets have been carefully chosen to illustrate the method. How well does this method work for benchmark datasets.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The strength of the paper is in using a  methodology of adaptation during inference using learned semantic context beyond the learned parameters of the semantic segmentation network during training.  This has both advantages of achieving better domain generalization and self-supervision. \n\nIf this method could be generalized for arbitrary semantic context, it can become interesting since the density map is only one of the cues that could be used for incorporating semantic context. How are others handled, for example, the typical color of an object, the average shape of an object etc.", "weaknesses": "The paper makes weak references to the Connectome and signaling mechanisms which detract from understanding the main idea. \nIf the order of description in some of the sections had changed, it would be an easier read. For example, giving a simple example of what is the information they hope to capture in the density map, and the entropy map early on would make easy ready. It wasn't until line 281-283 that we start following along. The insertion of propositions and definitions is also a distraction until the method has been clearly explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DYXuyeXCTm", "reviewer": "Reviewer_JdG9", "summary": "This paper introduces the Shape Memory Network (SMN), a deep learning architecture designed to improve semantic segmentation by incorporating contextual semantic information. The SMN emulates the brain's connectome by adapting its structure based on the context of the visual data, specifically in tasks where objects' shapes and densities vary significantly. Through a novel \"conditional neuron\" mechanism, the SMN selectively activates connections in response to semantic cues, allowing it to dynamically adjust its architecture at test time. The authors validate SMN's performance across various segmentation benchmarks, showing that it achieves higher accuracy than several state-of-the-art models, particularly in scenarios involving complex object boundaries or high-density variations.", "questions": "Could the authors explain the SMN’s adaptability in tasks beyond segmentation to determine its generalization?\n\nAre there specific strategies to reduce the computational demands of the SMN’s test-time adaptation for real-world deployment?\n\nHow does the SMN perform under scenarios with limited semantic information, and does this impact its segmentation accuracy?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "An innovative method for adaptive neural architectures, grounded in the idea of selective neuron activation inspired by the brain’s connectome.\n\nThe results across multiple benchmark datasets seem robust, highlighting SMN’s ability to outperform conventional segmentation models in handling complex object boundaries and densities.\n\nThe ability to dynamically adjust network structure during test time is a notable contribution, with potential implications for real-time applications where adaptability to context is beneficial.", "weaknesses": "The evaluation is confined to segmentation tasks, and while the biological inspiration is intriguing, its broader application to different tasks is not yet explored. Have authors tried testing on any other task?\n\nThe SMN’s \"conditional neuron\" mechanism is innovative but requires clearer exposition; some of the explanations are complex, making it challenging for readers to understand the \"exact\" workings of the adaptive architecture.\n\nThe SMN's test-time adaptation seems to demand substantial computational resources, which may limit its practicality in real-time or resource-constrained applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dHVaGyELxu", "reviewer": "Reviewer_1ydt", "summary": "Motivation:\n<1>Conventional neural networks have limitations in simulating the connectome.\n<2>The inference mechanism of conventional neural networks failed to account for the explicit utilization of contextual semantic information in the prediction process.\nTo overcome these limitations, this paper developed the Shape Memory Network (SMN) with two benefits: <1> emulating the intricate mechanism of the brain’s connectome, and <2> incorporating contextual semantic information during the inference process.\n\nMethod:\nSMN consists of three key components:\n<1>Segmentation Pipeline: Responsible for generating semantic segmentation maps.\n<2>Density Regression Pipeline: Predicts the density of target objects, i.e., the proportion of the image area occupied by the target objects.\n<3>Entropy Map Reconstruction: Uses Class Activation Maps (CAMs) to reconstruct entropy maps which represent the uncertainty of contextual semantic information in the input data.\n\nThe SMN is optimized during the training phase using three loss functions:\n<1>Cross-Entropy Loss (L1): Minimizes the difference between the predicted segmentation map and the ground truth labels.\n<2>Structural Similarity Loss (L2): Minimizes the similarity between the entropy map generated by the segmentation pipeline and the entropy map reconstructed from CAMs.\n<3>Density Regression Loss (L3): Minimizes the difference between the predicted density and the density calculated based on the segmentation map.\n\nAdditionally, this work introduces the concept of control neurons to simulate the human brain: by calculating the value of the control signals received by each control neuron, the connection strength (weights) between neurons is altered. The value of the control signals is generated by a linear combination of predicted densities. Furthermore, control neurons can also self-activate.\n\nThis work has conducted experiments on multiple benchmark datasets, demonstrating its superior performance in segmentation tasks. It also discusses the adaptability of SMN in different domains, as well as how to improve prediction accuracy by dynamically adjusting the network structure.", "questions": "<1> Can this method of simulating human brain control neurons be applied to other deep learning models? \n<2> Considering the complexity of the human brain's neural system, will training on models with larger parameter sizes yield better results?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "<1> The control neuron approach proposed to simulate the working mechanism of human brain neurons.\n<2> Comprehensive theoretical argumentation and mathematical proof.\n<3> Ample experimental validation.", "weaknesses": "<1> There has not been further verification of the SMN's performance on larger-scale datasets and when facing more complex tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VGXyxFIUWc", "reviewer": "Reviewer_nT8e", "summary": "This paper introduces the Shape-Memory Network (SMN), a neural network model that mimics the connectome structure of the brain by incorporating contextual semantic information to improve segmentation performance. The model distinguishes itself by emulating conditional activation and deactivation within neural pathways, specifically targeting the need for networks that can interpret and utilize contextual clues in a manner like the human brain. The SMN achieves this through \"conditional neurons\" that adjust dynamically based on contextual cues and synaptic-like processes. Experimental results on diverse semantic segmentation datasets show SMN's effectiveness in improving segmentation accuracy, especially in environments where object density and context significantly impact performance.", "questions": "- Are there alternative ways to reduce the SMN’s computational overhead, particularly for real-time scenarios?\n\n- How might SMN be adapted or validated for other tasks beyond segmentation?\n\n- What strategies could increase generalization to tasks with different structural demands, like sequential or non-spatial data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The SMN’s focus on context-sensitive \"conditional neurons\" for segmentation tasks is innovative and relevant in the realm of biologically inspired AI architectures.\n\n- The SMN’s theoretical grounding is solid, and its performance is evaluated across a variety of datasets, underscoring its effectiveness in segmentation.\n\n- This work contributes to the understanding of context-based adaptive mechanisms, potentially influencing the design of future segmentation-focused neural models.", "weaknesses": "- The conditional neuron mechanism introduces significant overhead, which may hinder scalability and real-time application potential.\n\n- Some of the claims about biological emulation could be tempered, as certain parallels drawn between SMN and synaptic transmission in the human brain may be speculative.\n\n- The design is highly tailored to segmentation, and extending the SMN to broader tasks without significant modification may be challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)