Complex Wavelet Mutual Information Loss: A Multi-Scale Loss Function for Semantic Segmentation
Authors: Renhao Lu
Venue: ICML 2025 poster (2025)
Keywords: Semantic segmentation
wavelet transform
steerable pyramid
mutual information
Primary area: applications->computer_vision
TL;DR: We propose a novel loss function, CWMI, that enhances image segmentation by combining multi-scale wavelet analysis with mutual information, achieving better edge precision and robustness without extra computation.

Abstract: Recent advancements in deep neural networks have significantly enhanced the performance of semantic segmentation. However, class imbalance and instance imbalance remain persistent challenges, where smaller instances and thin boundaries are often overshadowed by larger structures. To address the multiscale nature of segmented objects, various models have incorporated mechanisms such as spatial attention and feature pyramid networks. Despite these advancements, most loss functions are still primarily pixel-wise, while regional and boundary-focused loss functions often incur high computational costs or are restricted to small-scale regions. To address this limitation, we propose the complex wavelet mutual information (CWMI) loss, a novel loss function that leverages mutual information from subband images decomposed by a complex steerable pyramid. The complex steerable pyramid captures features across multiple orientations and preserves structural similarity across scales. Meanwhile, mutual information is well-suited to capturing high-dimensional directional features and offers greater noise robustness. Extensive experiments on diverse segmentation datasets demonstrate that CWMI loss achieves significant improvements in both pixel-wise accuracy and topological metrics compared to state-of-the-art methods, while introducing minimal computational overhead. Our code is available at https://github.com/lurenhaothu/CWMI

Complex Wavelet Mutual Information Loss:
A Multi-Scale Loss Function for Semantic Segmentation
Renhao Lu'
Abstract
1. Introduction
Semantic segmentation, the process of partitioning an image into regions associated with semantic labels, plays a crucial role in applications ranging from autonomous driving to biomedical imaging. Despite significant progress driven by deep neural networks such U-Net (Ronneberger et al., 2015) and fully convolutional networks (Long et al., 2015), challenges persist. Class imbalance, where dominant classes overshadow smaller ones, and instance imbalance, where small-scale structures are frequently ignored, remain major obstacles (Jiang et al., 2024; Kofler et al., 2023). Addressing these imbalances requires not only pixel-wise wise accuracy, but also the preservation of structural similarity, a property vital for ensuring spatial coherence and topological integrity in segmented outputs (Wang et al., 2004).
Recent advancements in deep neural networks have significantly enhanced the performance of se- mantic segmentation. However, class imbalance and instance imbalance remain persistent chal- lenges, where smaller instances and thin bound- aries are often overshadowed by larger struc- tures. To address the multiscale nature of seg- mented objects, various models have incorpo- rated mechanisms such as spatial attention and feature pyramid networks. Despite these ad- vancements, most loss functions are still pri- marily pixel-wise, while regional and boundary- focused loss functions often incur high compu- tational costs or are restricted to small-scale re- gions. To address this limitation, we propose the complex wavelet mutual information (CWMI) loss, novel loss funetion that leverages mutual information from subband images decomposed by a complex steerable pyramid. The complex steerable pyramid captures features across multi- ple orientations and preserves structural similarity across scales. Meanwhile, mutual information well-suited to capturing nigh-dimensional direc- tional features and offers greater noise robustness. Extensive experiments on diverse segmentation datasets demonstrate that CWMI loss achieves significant improvements in both pixel-wise ac- curacy and topological metrics compared to state- of-the-art methods, while introducing minimal computational overhead, Our code is available at http:://ithub..cmm/ httpg//htbcoooouuuueeaaouo////MW
Although the advancements in feature extraction, such as feature pyramid networks (Lin et al., 2017) and attention mechanisms (Woo et al., 2018; Chen et al., 2021; Islam et al., 2020), have enabled models to capture multiscale contextual information, most loss functions still focus pixel-wise optimization (Azad et al., 2023). Region-based loss functions, such as Dice loss (Milletari et al., 2016) and Tversky loss (Salehi et al., 2017), are not subject to class imbalance, but smaller instances within the same class are still easy to overshadowed, which can be critical for preserving regional and boundary details. Zhao al. developed Regional Mutual Information (RMI) loss, which captures statistical relationships over regions. However, RMI is constrained within a relatively small region (3 x 3 pixels) to avoid high computational overhead (Zhao et al., 2019). Thus, a loss function that balances computational efficiency with the ability to model structural and regional dependencies at larger scales is highly desirable.
Addressing these challenges from frequency eequuency-domain domain per- spective provides an innovative pathway. Patterns of vary- ing scales in images are inherently tied to their frequency components: large-scale structures correspond to low fre- quencies, while finer details correspond to high frequencies. Wavelet transforms are uniquely suited for this multiscale decomposition, as they preserve both spatial and frequency information (Mallat, 1989). Among these, the steerable pyramid, proposed by (Simoncelli et al., 1992), leverages
'Meinig School of Biomedical Engineering, Comelll Uni- versity, Ithaca, NY, USA. Correspondence to: Renhao cone.e.cormlll.duu>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Complex Wavelet Mutual Information Loss: A Multi-Scalc Loss Function for Semantic Segmentation
steerable filters for redundant wavelet decomposition, en- abling multiscale and multi-orientation feature extraction. Its extension, the complex steerable pyramid, further en- hances this framework by using complex numbers to explic- itly represent local phase information, allowing for robust extraction of structural details across scales and orienta- tions (Portilla & Simoncelli, 2000). These properties make it a powerful tool for segmentation tasks where structural similarity is paramount.
has shown significant performance enhancement, as in At- tention U-Net (Islam et al., 2020), TransUNet (Chen et al., 2021), Vision Mamba UNet (Ruan et al., 2024), etc. In this study, we compare our proposed CWMI loss using U-Net and Attention U-Net to test the generalization and superior- ity of CWMI, while the general idea is adaptable to other architectures as well.
2.1. Loss Functions for Semantic Segmentation
While the architectural advancements in segmentation mod- els have been significant, the performance of these networks is highly influenced by the design of the loss functions. Pixel-wise cross entropy loss (CE) minimizes the log like- lihood of the prediction error but is significantly prone to class imbalance. To address this issue, class balanced cross entropy (BCE), proposed by (Long et al., 2015), employs higher weights for classes with fewer pixels. Focal loss as- signs higher weights to misclassified pixels with high proba- bilities (Ross& & Dollár, 2017). Region- based losses, includ- ing Dice loss (Milletari et al., 2016). Tversky loss (Salehi et al., 2017), and Jaccard loss (Rahman & Wang, 2016), in- herently handle class imbalance but fail to address instance imbalance within the same class, such as thin boundaries and small objects.
In this paper, we introduce Complex Wavelet Mutual Infor- mation (CWMI) loss, a novel loss function that leverages the complex steerable pyramid for efficient multiscale structural information extraction. By combining the robust multi- scale decomposition capabilities of the complex steerable pyramid with the statistical power of mutual information, CWMI loss explicitly incorporates local phase, orientation, and structural features into the loss calculation. This ap- proach ensures structural coherence and boundary preserva- tion while maintaining computational efficiency, making it well-suited for segmentation tasks with significant class and instance imbalances.
Our contributions are summarized follows:
We propose CWMI loss, which can maximize the mu- tual information in the domain of complex steerable pyramid decompositions. Such a strategy can enhance multiscale structural features for semantic segmenta- tion, especially for tasks with significant class and instance imbalances.
To tackle instance imbalance, weighted loss functions have been proposed. In the original U-Net paper (Ronneberger et al., 2015), the weighted cross entropy (WCE) was in- troduced, employing a distance- weight map to em- phasize thin boundaries between objects. However, WCE assigns weights only to boundary pixels, neglecting object pixels. The adaptive boundary weighted (ABW) loss (Liu et al., 2022) extends this approach by applying distance- based weights to both boundary and object pixels, while the Skea-Topo loss further improved the weight map based on boundary and object skeletons (Liu et al., 2024). Despite their contributions, weighted losses suffer from two major limitations: (1) the weight maps are precomputed and fixed, failing to adapt to errors during training, and (2) they often generate thicker boundaries, which preserve topology but compromise metries like Dice score and mloU, as observed in our qualitative results.
We demonstrated the superiority of CWMI with extensive experiments on four public segmentation datasets: SNEMI3D (neurite segmentation in electron microscopy slices), GlaS (gland segmentation in H&E slices), DRIVE (retinal vessel segmentation in fundus images), and MASS ROAD (road segmentation from aerial imagery). Compared with 11 state-of-art (SOTA) loss functions, CWMI showed better performance on both pixel-wise metrics and topological metrics, while introducing minimal computational overhead.
2. Related Work
Several methods address instance imbalance dynamically during training, but at the cost of computational efficiency. Topology-based approaches, such as persistent homology methods (Stucki al.. 2023; Oner et al., 2023), describe image topologies and identify critical pixels but are compu- tationally expensive, with cubic complexity to image size. The eIDice loss (Shit et al., 2021) employs a soft skele- tonization algorithm to detect topological errors, primarily focusing on thin-boundary objects like retinal blood ves- sels. Similarly, Boundary Loss (Kervadec et al., 2019) and Hausdorff Distance Loss (Karimi & Salcudean, 2019) refine boundaries but incur significant computational overhead.
Semantic segmentation has seen tremendous advancements through deep learning architectures, with U-Net and its vari- ants becoming a cornerstone of this field. The U-Net model, proposed by (Ronneberger et al., 2015). utilizes a symmet- ric encoder-decoder architecture with skip connections to preserve spatial details while capturing global context. Its success has inspired numerous adaptations in its convolu- tional blocks (Huang et al., 2019; Diakogiannis et al., 2020) and skip connections (Zhou et al., 2018; Chen et al., 2021). Inspired by the transformer model, the attention mechanism has also been incorporated into the U-Net structure, which
2

---

Complex Wavelet Mutual Information Loss: A Multi-Scale Loss Function for Semantic Segmentation
Complex Stecrable Pyramid Decomposition
Pazz
Pea
Inpat
Prediction
Par)
10 Pas) + (Caa Poo) + 10Y..Pa))
CW
Vaz
Yus
Nas
Label
Figure 1. Illustration of the proposed Complex Wavelet Mutual Information (CWMI) Loss. The prediction and label images are decomposed using a complex steerable pyramid, which generales subbands at different scales and orientations. Mutual information is calculated for each corresponding pair of subbands, and the CWMI is computed the sum of these mutual information values. YB.: PB. complex stcerable decomposition of label and prediction image at level n; 7(Y B,, YB,): mutual information between Yp, D. and PB.
Region Mutual Information (RMI) loss (Zhao et al., 2019) captures pixel interdependencies regions, but struggles with scalability for large-scale regional analysis due to the high computation cost. These losses either prioritize small regions at the expense of global accuracy or require exten- sive computational resources, necessitating more efficient and balanced approaches.
and SSIM, offering superior segmentation performance and robustness.
3. Methods
To compute the proposed Complex Wavelet Mutual Infor- mation (CWMI) loss, the prediction and ground truth label matrices first decomposed into subbands using the com- plex steerable pyramid. Mutual information (MI) is then calculated for each subband. The CWMI loss is defined as the sum of the MI values across all subbands, as illustrated in Figure
2.2. Wavelet-Based Loss Functions
Wavelet-based metrics were first introduced as Complex Wavelet Structural Similarity (CW-SSIM) (Sampat al., 2009), known for their robustness to small translations and rotations. In the deep learning era, wavelet-hased methods have been employed in loss functions, leveraging their abil- ity to analyze multiscale and multiresolution features. These methods have shown promise tasks like sketch-to-image to-image translation (Kim & Cho, 2023), image super-reson super-resolution (Ko- rkmaz& & Tekalp, 2024), image dehazing (Yang al., 2020), and material analysis (Prantl al., 2022). However, to the best of our knowledge, wavelet-basse loss functions have yet to be explored in semantic segmentation.
3.1. Complex Wavelet Decomposition
Wavelet transforms are widely used for multiscale analysis, enabling decomposition of image into frequency subbands while preserving spatial informa- tion. Unlike traditional Fourier transforms, which analyze global frequency components, wavelet transforms provide localized frequency representation, making them well- suited for tasks involving apatially-varying structures such as semantic segmentation. For an image [(x, y). the wavelet transform is defined as:
Wavelet Transform
Existing wavelet-based loss functions typically rely on L (Zhu et al., 2021; Korkmaz & Tekalp, 2024; Prantl et al., 2022) or L2 distances (Kim & Cho. 2023), or structural simi- larity (SSIM) (Yang al., 2020) in the decomposed domain. While effective, these methods are less suited for handling high-dimensional data with complex directional features, as in wavelet transforms, and may be vulnerable to noise. The proposed CWMI loss leverages mutual information be- tween wavelet-based subband images, effectively capturing multiscale dependencies. As demonstrated in later ablation tests, CWMI outperforms traditional metrics like L1, L2,
(s, = / / [(x, ((2,8)),, v) dr dy (1)
where (,,, y) is scaled and translated version of the mother wavelet function (),,,), with controlling the scale and f the translation.
For discrete signals, the Discrete Wavelet Transform (DWT) decomposes an image into progressively lower resolution subbands using filter banks. However, traditional wavelet
3

---

Complex Wavelet Mutual Information Loss: A Multi-Scale Loss Function for Semantic Segmentation
decompositions suffer from limited orientation selectivity, capturing only fixed horizontal, vertical, and diagonal direc- tions.
Steerable Pyramid
Complex Steerable Pyramid
1:22
Steerable Pyramid To overcome these limitations, steer- able pyramid extends the wavelet framework by introducing orientation-sensiin band-pass filters, significantly enhanc- ing orientation selectivity. Unlike DWT, which provides non-redundant representation, the steerable pyramid offers flexible, redundant image representation, facilitating im- proved multiscale analysis. This decomposition is achieved through the iterative application of steerable band-pass fil- ters followed by downsampling. In the frequency domain, the pand-pass filter for the kan orientation expressed in polar coordinates (r, as:
Iod
Ind
hx3
Insk
Inpur
Prese of
(x 0) (O)) E [1, K],
(2)
where (r) and G (0) represent the radial and angular components, respectively:
In3
cos (3 loge (2²),,
<<<
(r)
I a
(3)
0,
r 2
Prese Ineellgg
Phose 4q
K-1
kk
(0) ag cos
0
(4)
K
where K the number of orientations and Cx 2 (K-1)) -K]( Figure 2A provides an illustration of
Figure 2. Steerable pyramid and complex steerable pyramid. (A) Orientation-sellctiv band-pas pass filters of the steerable pyramid. (B) Example decomposition using a stecrable pyramid with N=3, K=4. (C) Band-pass filters
the band filter with =4
With total recursive levels N, an image I can be decom- posed as:
complex steerable pyramid, where negative frequency components are discarded. (D) Phase representation the complex steerable pyramid output, the real part identical to that of the steerable pyramid.
I HaxW,
IB, RxHoo
IB, R RxH
I
(5)
By E R Ino -N
this formulation, the real part remains unchanged, while the imaginary part is obtained via the Hilbert transform of the real component. In the Fourier domain, this transformation is equivalent to discarding negative frequency components, as illustrated in Figure 2C.
where Lt, and LL are high-frequency and low-frequency residues, and Is represents subband images at level with k-th direction concatenated in the first dimension. Figure 2B shows an example of the decomposed output of an input image I.
For the complex steerable pyramid, the angular component Gg (0) is modified as:
Complex Steerable Pyramid Although the steerable pyramid effectively captures amplitude information across multiple orientations, it lacks the ability to extract local phase information, which is crucial for encoding structural features such as edges and corners (Canny, 1986). To ad- dress this limitation, (Portilla & Simoncelli, 2000) intro- duced the complex steerable pyramid, which extends the conventional steerable pyramid by converting its decom- posed images into their analytic signal representation.
(x(()) 2ax [cos (0 0.
7 |
K-1
K
2
otherwise.
(6)
This modification enables image decomposition into com- plex subbands, where phase information encodes critical structural features such as edges and corners (Figure 2D), while amplitude represents feature strength.

---

Complex Wavelet Mutual Information Loss: A Multi-Scalc Loss Function for Semantic Segmentation
3.2. Mutual Information in Complex Wavelet Domain
This selection enables an assessment of CWMI's general- ization ability across both fully convolutional and attention- enhanced models. Additionally, to examine CWMII's com- patibility with recently proposed Mamba-base based architec- tures (Yue & Li, 2024), we incorporated the Vision Mamha U-Net (VMUNet) (Ruan et al., 2024) into our experiments.
According to Equation 5, the ground truth Y and the prediction P are decomposed into subbands Y B. and PB. E R R=x for each level e [1. For each pixel (Ir, v) at level n, the K-directional features are treated as K K-dimensional random variables.
Datasets We tested CWMI on three public segmentation datasets, all characterized by class and instance imbalance: (1) SNEMI3D, neurite segmentation dataset containing 100 1024 x 1024 grayscale images from electron microscopy slices Irgganda-Carreriy et al., 2013); (2) GlaS, gland seg- mentation dataset with 165 RGB images of varying sizes from histological images of colorectal cancer samples (Sir- inukunwattana al., 2017); (3) DRIVE, retinal vessel segmentation dataset comprising 40 584 x 565 RGB images from fundus photographs (Staal et al., 2004); and (4) the Massachusetts Roads dataset (MASS ROAD), a road seg- mentation dataset with 1171 1500 x 1500 RGB images from aerial imagery (Mnih, 2013). We choose a subset of 120 im- ages (ignoring images without a network of roads). For all datasets, at least two repeated fhree-fold cross-validations were used to ensure robust evaluation.
For mutual information approximation, several studies, such as MINE (Belghazi et al., 2018) and Deep InfoMax (Hjelm et al., 2018), employ neural network-based estimators to produce tight lower bounds. While these methods are the- oretically well-founded, they introduce additional training overhead, which increases algorithmic complexity and may hinder the efficiency of the loss function. Therefore, in this work, we adopt the mutual information estimation approach proposed by (Zhao et al., 2019):
1 log det (M,):
T(YB,, B.; PB.)) R
2
Mn- ΣYe, Cov(YB,. B. PB. (Epb.) )" Cov(Y B, P..... B.,.
(8)
where EYe, and Eps, are covariance matrices, and Cov(Yn,, n,, PB.) is the cross-covariance.
Baselines and Implementation Details We compared CWMI against 11 state-of-the (SOTA) loss functions, including pixel-wise loss functions (c.g.. cross entropy, BCE (Long et al., 2015), Focal loss (Ross & Dollár, 2017)). Sensi- tive loss (Tang et al., 2025), region- based loss functions (c.g.. Dice loss (Milletari et al., 2016), Tversky loss (Salehi et al., 2017), Jaccard loss (Rahman & Wang, 2016)), and struc- tural/topological loss functions (e.g., WCE (Ronneberger et al., 2015), ABW loss (Liu et al., 2022). Skea-topo loss (Liu et al., 2024), RMI loss (Zhao al., 2019), elDice loss (Shit et al., 2021)). Hyperparameters for each baseline were tuned via grid search, with Tversky loss (a = .5,3 3 .5) and Focal loss (7 = 2.5) as examples.
When the ground truth and predictions are decomposed into complex subbands, this equation is extended with Hermitian transpose H:
Mn = EYe, -Covoy B,, PE,.(Y,,,PP..)EPP.. " Cov (YCov(YB,,))) B., P... PB.)
(9)
where the covariance and cross-covariance are calculated as:
EYe, F((YY,. B., (IY B,,- ((YY,.)) (10)
EPan
E[(PB, B, EEPP B. 1).) (PB. EEP E(PB..)) 11 (11)
We utilized steerable pyramids with four decomposition lev- cls and four orientations in all experiments. N = 4 and K = 4 of the steerable pyramid decomposition and a reg- ularization parameter of = 0.1 were determined by the ablation experiments, and applied in all CWMI experiments. For the implementation of Equation 9, although Py ToTch supports complex matrix calculations, our experiments indi- cated that its efficiency remains suboptimal. Consequently, we computed using their real representations, which are mathematically equivalent (Golub & Van Loan, 2013). To (13) assess the significance of the complex steerable pyramid, we compared CWMI with real-number-only variant (CWMI- Real), implemented according to Equations 4 and 8. Adam optimizer with StepL.R scheduler (initial learning rate x 10-4, decay rate 0.8, step size 10) was used. U-Net and Attention U-Net models were trained for 50 epochs; VMUNet models were trained for 100 epochs due to their slower convergence. All models were trained with a batch
Cov(Y Y B., PB..)
E( -E(Y,, :..)==E((YB,... - -E(P.. B.,1) "1
(12)
Finally, the CWMI loss computed as the sum of MI across all levels, which combines with cross entropy loss inte- grate pixel-wise loss:
N Σ #=1
CWMI(Y,P) =
-k(Y,, B,ú PB... PB,))Abe +Abce PY,
4. Experiment
4.1. Experimental Setup
To evaluate the effectiveness of the CWMI loss, we employed U-Net (Ronneberger et al., 2015) and At- tention U-Net (Oktay et al., 2018) as baseline architectures.
Base Models
5

---

Complex Wavelet Mutual Information Loss: A Multi-Scalc Loss Function for Semantic Segmentation
SENME3D
UNet
AltenUNet VIL
Methods
mloUf
mDicet
VIJ
ARTf
HDJ
mloUf
mDicet
ART1
HDJ
CE
.753 805
1551 001
2.082 270
.00g..00
1.145g 176
.751 000
8501 007
1.9144.361
.525 056
1.178 261
BCE
.736+.006
8114
1.756+. 156
5611.020
1.4132.08 068
.735 010
8/10.007
1.8354.1100
.553 023
1.177.224
Dice
.767 008
.622 001
1.4062.088 085
.041.017
.919
.768 005
8612..00
1.5074.096
.5834 o16
.024
Focal
.728
.835
1.751
5564.016
1.725 225
.725 000
833
1.9114.190
.50
1.803
Jaccard
.7664.004 004
.86 +.00
1.357+.114 114
.144.019
.9044.057 .08T
.762- 008
858+ .002
1.300+.079
.6004.000 007
1.000+.120 125
Tversky
76G..
1861+.oo
1.328
617+..017
+990
.765 003
860 .002
1.303++.087 0057
.214 009
.969
WCE
.7144.005
.825 003
1.987 398
.525
1.575 250
.713 005
825 .004
1.8181.034
.5431 008
1.613 192
ABW
.605 066
.739 037
3.637 703
.299 2187
4.605 926
.616 072
.749 063
3.077 11 303
.366 142
4.045
Skca-topo
.472+.199
.671+.197
3 3.822+2.660
+00+.269
1,618=. 207
.022 096
.729+.000 0066
3.759
306 243
4.889=6.920
RMI
.764+.008
.8594.006
1.443+.207 207
558++.087
.943+.150
.764- 008
.8594.006
1.404+.124
.585+.00 028
.958+.170
eiDice
.0642.014
.819..011
2.049
5022.002
1. 1,489 21.8
.701 022
.814.016
1.887.1161
.5184 034
1.720 244
Sensitive
7034.009
8084.005
1.54+.009
.574.022
.948 +448+.148
.757 808
854+.004
1.822+.318
62774.049
1.016+.199 139
CWMI Real
776 004
867 008
1.205
2.18.
.775 004
.664..00
1.193
..44.014
.824
CWMI
7781 000"
869 003
1.1621 058
.388.013
7799 0.s
7771 006
8688 005
1.162: 085
.699 019
8871.088
Gla8
UNet
AtlenUNet VIL
Methods
mloUt
mDicet
VIJ
ARI
HD1
mloU/
mDicet
ARM
HD
CE
6:37 201
72441.19 192
9933 o13
4000 323
6.633 655
208
.725 185
9966 107
406 327
7.914 637
BCE
.6142.290
213
1.032 152
3355..35
4.16541.022
.619 222
270
.9862.189
365
4.354
Dice
.632+.201
.719+.189
4887+.080
.366+922
7.004+4.670
.636 .636-197
197
.723+.187
.954+.059
330++.82
7.335++4.84 4.84
Focal
5881 2.3
.672+.241
1.093 103
336
4.447
.581
.6724.240
1.043
.336 .899
5.040
Jaccard
205
.723+.198
.947 076
.397 +.35
6.979+4.611
.638 202
.725+.190
.888..70
.406+ 331
6.758 431
Tversky
634 205
722++199 198
953+ 051
3944
0.705++4 398
.27 195
+717+.186
:92
.384 .514
7.521+44.4 493
WCE
810
9922
832
.721 7.211.008
2.264
.833
8981.013
.83
.755
2.590 802
ABW
.600+.012
.857 857++008
1.468+.079
.19+..22
3.527
.762 772--019
858+.018
1.472+.108
.6144.08T
3. 3.623 887
Skea-topo
788_..08
.05
1.259 042
2.985 .588
r65
.781= 018
873-..010
4.99
.615 022
3.363 809
RMT
8301.011
.907
.20
.7261.024
2.644
.835
10051.010
.822 083
.720
2.701 1:
elDice
816
.894
.900
6698
2.809
.800
8844
1.0044.113
.6844
3.146
Sensitive
..14
8077+0000 008
.941 101
..4
2.916 304
.824- 018
.880 109
.702+ 041
2.757 399
Real
838 014
.907 .907
7981 oG8
727
2.813_ 603
.822 022
9099..11
.788 2003
.724 o3e
2.758 764
CWMI
843+.016
910+.011
761++
7755 028
2.589 .486
844 0007
.91+ 008
.75+ +.0
737+.015
2.569+.44
DRIVE
UNet
AttenUNNet
Methods
mloUt
mDicct
VIJ
ARIf
HD
mloUt
mDicet
ARI
HD
CF
.770..014
.856
1.379
..06
2.344
.757 011
.85
1.436+.126
.372 000
2.932 875
BCE
.742+.004
.85++.00
1.501+.175
.399+.08
2.355+ 434
.746-.022 .012
838+.010
1.443+.106
.427+.0s5 osa
2.169 264
Diee
.7792.018
814
1.293
.4715.045
1.696
.776
8612.012
1.335
.445. 078
2.146 832
Focal
77371.00m
832
1.404 168
1.11
2.147 2. 3331
.745 7745_.0 011
8831.000
1.410
.450 075
2,141 100
Jaccard
761..018
850+.011
1.289+1111 181
.502..066
1.988 641
.766_
854+.015
1.2714.146
.5214 051
1.623 313
Tversky
72521.020
1843
1.309
.517 5174.0022
1.638 360
.768
8561.001
1.265
.521
1.702
WCE
.711
.835 011
1.4172.009 069
.412
2.100
498
.733_ 009
$28
1.441
.444 030
2.326 848
ABW
Skea-topo
.599-.121
.674...9
1.602
.265 250
5.791#4.499
RMI
7.872.012
.8694.01
1.282 127
.4494.064
1.797
.785 008
8671.007
1.2794.086
.448. 047
1.8844 3.0
elDice
.482g.094
.598 085
2.010 119
9.813 222
153
.509 063
.667
1.9124.167
.238 1611
15.704
Sensitive
7663+.008
850+.007
1.440+.000
376+.089
2.821+.660 562
.757- 137--00 018
8/14.016
1.444.0000
.380+.00 .288
3.134+.762
Ceal-Real
788+.018
870+.012
1.104+
582+.076
1,309
.77 028
870+.0u
L072+.118
594+
1, 1,398+.400
CWMI
7984.012
878 000"
1.032
613 082"
1.079 383
.755 015"
885 012
1.007 102
622 0000""
1.219 410"
MASS ROAD
UNet
AttenUNet VIL
Methods
mloUt
mDicet
V1J
ARIf
HDj
mloUt
mDicct
ARI1
HD.
CB
7334-015
826+.013
3.473+.008
194++.08
1.927+3.00
.610 200
.725+.196
906+.107
406+ .27
7.914+0.c8
BCE
.11
.7944 con
1.708 265
.604
12.79213.684
222
208
.986 139
.365 358
4.354 8609
Dice
7.61
..00..00
1.601
.558..44
11.18044.522
.636 197
.723+.187
.954
.306 822
7.335
Focal
.671+.015
780++.013
1.717+,110 150
.556+.04
11.576+1.950
.581 252
.672+.240
L.04
.330 .39
5.040 -2002
Jaccard
.759.. nos
8884
1.332 041
.630 000
11.27442.905
.638 202
.7252.190
.888 076
.006 .351
6.758
Tversky
.755 008
.845 007
1.379 196
.6181.011
11.50114.470
.627 195
17171.186
.924 102
.3844 314
7.521 493
WCF
.6504.016
.7624.014
1.726 103
.5564.004
1.8452.470
.823_ 016
8894.0111
.873 87772097
.7054 028
2.590 602
ABW
.685+.014
.791+.012
1.678+.124
566+..80
11.65444.681
.762
858+.018
1.472+.108
.614+.057
3.623 88T
Skea-topo
620
.731+.05b
2. 2.236
166
11. 11.653
.784=.018
883+
1.2994.006
.645
3, 3.363 8o9
RMI
.6662.00
.8524 608
1.824 263
.4991.065
11..33213..0 13.998
.839_ 013
.908..o08
.819 0711
.7274.032
2.587 435
elDice
6454..81
75544.06
2.122..467
.476+.081
15.46144.900
.800-.013 0018
.844.000 .009
1.004..11
.6844.00 .028
3.1464 260
Sensitive
731:.002
8274
2.933
105
11.292
19.29223..22
.824
8994.011
880
.7022
2.757 3398
CWMI-Real
7654.010
.8534.008
1.231+ 183
.500.060
1.836+4
904
.846
012
.914.on8
766-_067"
.336+.021
2.504+.222 .688
CWMI
7667+.010
8544.008
1.148 108"
.677+047"
1.826
1.0
8394.015 ..15
.98+.010
762+.084"
728+.095
2.747 0.00
Table Quantitative results of different loss functions across the four datasets and two neural network models. The bold numbers indicate the best performance for each metric, while the underlined numbers denote the second-best performance. A hyphen ("-") indicates cases where the model did not converge. For the loss metrics CWMI and CWMI-rcal, the statistical significance is marked with asterisks: tp < .05, #*p .01. and **ap < .001, where p is the maximum p-value of the student's 's tests against all the baseline loss functions.
6

---

Complex Wavelet Mutual Information Loss: A Multi-Scale Loss Function for Semantic Segmentation
size of 10. Early stopping based on mloU was employed select the best model. Training was conducted NVIDIA A100 GPU using the Google Colab runtime.
Image
Ground Truth
CE
BCE
Dice
Data Augmentation and Evaluation Metrics Random flips and rotations were applied to all datasets to improve generalization. For SNEMI3D and MASS ROAD, images were randomly cropped to 512 512, while for GlaS, im- ages were cropped to 448 x 576 to standardize input sizes.
Jaccard
Tversky
No cropping was performed for DRIVE due to its uniform image dimensions.
Siea
R54
dDice
CWMM Real
CWMI
Performance was evaluated using five metrics: mloU and mDice for regional precision, variation of information (VT) (Nunez-Iglesias al., 2013) and adjusted Rand index (ARI) (Vinh et al., 2009) for clustering precision, and Hausdorff distance (HD) for boundary and topological accuracy. These metrics provide a comprehensive assessment of both re- gional overlap and structural fidelity.
Figure 3. Qualitative results of different loss functions on the SNEMI3D dataset. Red: false positive regions: Blue: false nega- tive regions. Green arrow: challenging false positive and Orange arrow: challenging false negative that are successfully addressed by CWMI.
SNEMIED VMUNet
Image
Ground
Dice
Methods
mloUt
mDiiet
VI
ARIT
HDJ
CE
.223 0288
.829 020
2.577+1.211
178
1.709
235
BCE
.739_
.813 002
1.4991.055
.594g.01
1.274:
231
Dice
.725
.831+.021
2.735+1.218
105
1.767+ r20
Focal
_725_.010
.8334.007 007
1.677
.5632.023 025
1.699
372
Jaceard
.725-.019
.831+.014
1.953+.641
.499+.078
1.716+.660
Focal
Jeccard
Twersky
VICE
ABW
Tversky
-737=.018
.8102.012
L.6194.324
.5552.045
1.500
.714_.00s 000
.8261.00s 004
1.743 .26
.5522. 006
1.432 1688
ABW
.678_
.800
2.076
.490
1.572
RMI
-762-.10 0100
.877+.077
4.88
.588+.080 .80
4.07 306
elDice
000
8244.001
1.754+.00 .022
.542+.008
T,990 262
sensitive
CWMI
7833.000
8724.003
.82 100
660 .222"222
.229_.077 073
Shea Inpo
dDic
CWMI
CWNI
Table Quantitative results of different loss functions SNEMI3D dataset and VMUNet model. The bold numbers indi- cate the best performance for each metrie, while the underlined numbers denote the second-best performance. A hyphen ("-") indicates cases where the model did not converge. For the loss metrics CWMI, the statistical significance is marked with asterisks: *p < .05, where p is the maximum p-value of the student's tests against all the baseline loss functions.
Figure 4. Qualitative results of different loss functions the GlaS dataset.
4.2. Quantitative and qualitative results
As shown in Table 1, the proposed CWMI loss outperforms other loss functions across the majority of evaluation met- rics for all datasets using both U-Net and Attention U-Net architectures. Statistically, CWMI achieves significantly superior performance on all metrics for the SNEMI3D and DRIVE datasets, as well as on clustering bussed metrics (VI and ARI) for the GlaS and MASS ROAD datasets. More- over, CWMI consistently outperforms its real-valued variant, underscoring the importance of incorporating phase infor- mation within the steerable pyramid decomposition. For the Mamba-based architecture, VMUNet, CWMI also signifi- cantly outperforms baseline loss functions on the SNEMB3 dataset (Table 2). Qualitative results from SNEMI3D (Fig- ure 3), GlaS (Figure 4), DRIVE (Figure 5), and MASS ROAD (Figure 6) further illustrate CWMI's effectiveness
Roca
accard
Tvenky
WCE
159
<Dkce
CUNI Real
DWVI
outperform
Figure 5. Qualitative results different loss functions on the DRIVE dataset.
in mitigating challenging false positive and false negative segmentation errors that persist with other sfate-of-ther -rt loss functions.

---

Complex
Wavelet Mutual Information Loss: A Multi-Scale Loss Function for Semantic Segmentation
Image
Ground Buth
SCE
Dice
is insufficient for reliable segmentation. The CWM variant, by contrast, achieved the best performance clus- tering metrics (VI and ARD, indicating its strong capacity to capture regional structures. The full CWMI, combin- ing both magnitude and phase via the complex representa- tion, yielded the best results on pixel-wise accuracy metrics (mloU and mDice) and topological metric (HD), while rank- ing second on VI and ARI, demonstrating well-rounded performance across different dimensions of segmentation quality, The CWMI-Real variant performed worse than both CWMI and CWMI- on most metrics, although it outper- formed CWMI-Phase further supporting the necessity of incorporating both magnitude and phase components. These results highlight the critical role magnitude in structural segmentation, while also demonstrating that the full com- plex formulation of CWMI achieves superior generalization across diverse evaluation criteria.
Focol
Ioccard
Tverrkyy
WCE
ABW
<b3cce
CWML Real
CHNNI
Figure 6. Qualitative results of different loss functions the MASS ROAD dataset.
4.3. Ablation studies
Methods
m
mDDiett
VII
ARTT
HD1
In this section, all ablation experiments were performed on the SNEMI3D dataset with U-Net model.
CWMI-Real .7764 .044
.868., .08
1.211
.635..011
.774+.005
CWMI OWg
.775 004
.867 003
.908 _ 051
.600 00e
.735 065
CWMI-Phasc
.0944..om
_800..00
2. 4.4874..34
.188
061
2,000 258
Mutual Information (MJ) L1, L2 Distance and Struc- tural Similarity (SSIM) As previously discussed, various metrics based on wavelet transforms have been developed for evaluating image similarity and guiding loss functions. To assess the advantages of mutual information (MI) over commonly used metries such as L1. Lg. and SSIM, we con- ducted a comparative analysis. The experimental results, summarized in Table 3, demonstrate that MI consistently outperforms L1. L2 distance, and SSIM across multiple evaluation metrics. Unlike L1 and L2, which focus on pixel- wise intensity differences, and SSIM, which emphasizes structural similarity, MI captures joint statistical dependen- cies between features in each direction. This ability pro- vides more robust representation of structural differences between predictions and labels, particularly in complex seg- mentation tasks.
CWMI .7781.00 000
.869 003 1.1644 .638 0r6 7200 o66
Table 4. Quantitative comparison of mutual information loss across di different components the complex steerable pyramid decom- position. CWMI-Real: real component; CWMI-Mag: magnitude component:
phase component; CWMI: full com- plex representation (combining real and imaginary components). Results are based on U-Net model trained on the SNEMI3D dataset. Bold values indicate the best performance, and underlined values denote the second-best for each evaluation metric.
Impact of Regularization Parameter À To evaluate the sensitivity of CWMI to the regularization weight X we ex- perimented with = 0.5, As shown in Table 5, performance across all evaluation metrics remained largely stable, with only marginal variations. This robustness sug- gests that CWMI provides complementary information to the cross-entropy term and is not overly sensitive to the reg- ularization strength. Additionally, we found the inclusion of the cross-entropy term to be crucial for directional learning, as mutual information is inherently symmetric and cannot distinguish between correct and inverted label assignments. Impact of Decomposition Level N and Number of Ori- entations K Theoretically, the steerable pyramid can de- compose an image into very high levels with an infinite number of orientations, provided the input image is suffi- ciently large. However, does deeper decomposition or a higher number of orientations improve feature extraction and loss computation? To address this, we analyzed the impact of N (decomposition level) and K (number of ori- entations) on the performance of CWMI, as shown in Table 3. Interestingly, both N and K achieved optimal perfor- mance at relatively low values, suggesting that the critical information for segmentation is concentrated in relatively high-frequency regions. This observation supports the idea
Methods
mioUt
mDDsef
V1
ARIT
HD]
LI
7734.002
8604..00
1.22+.0m
632+.007
.70.08
L2
.7744.003
.8672.002
1.232.06
6333..01
.76..13
438
.6131.om 020
5.96 n6
00811.030
9.31_1.61
CWMI
.7794.00
.870
1.16
6640-
.69
Table Quantitative results comparing mutual information (MI),
La, Le distance, and structural similarity (SSIM) based on U-Net with SNEMI3D. The bold numbers indicate the best performance for each metric.
Contribution of different components of complex steer- able pyramid To better understand the role of individ- ual components in the CWMI loss, conducted an ab- lation experiment evaluating the performance of variants based on the mutual information loss of magnitude-only phase-only, and real-only representations, as shown in Table The SWMI-Phase variant consistently underperformed across all metries, suggesting that phase information alone
8

---

Complex Wavelet Mutual Information Loss: A Multi-Scalc Loss Function for Semantic Segmentation
that emphasizing high-frequency subbands refines small instances and narrow boundaries, while lower-frequency components from deeper decompositions contribute less informative features compared to higher-frequency ones.
cesses an image of size HW/4, and the third layer pro- cesses HW/16, and so on. Thus, the total computational complexity follows geometric series with a superior bound O (3 HW (K log( (W) + K²))
Knowing that the first four layers of decomposition are crucial, we further performed a layer ablation experiment (Table Layer Ablation), where only selected layers were used to compute mutual information while discarding the remaining layers. Our results show that the third layer outperforms other layers in region-based (mloU, mDice) and cluster-based metries (VI, ARI), while the fourth layer achieves the best performance in topological metrics (HD). These findings suggest that features extracted from multiple layers are essential for achieving high performance across all evaluation metrics, highlighting the importance of incor- porating information from both mid- and freeffryy subbands in the segmentation task.
From our ablation experiments, the optimal number of ori- entations is K 4, which is relatively small. Hence the complexity of CWMI linear to O(HW log(HW)), which remains scalable for high-resolution images. Compared to topology-aware losses such as RMI, clDice, or Hausdorff Distance Loss, CWMI achieves better structural and bound- ary preservation with lower computational overhead, as shown in Table 4.
Epuch Time (s) CE (s)
CE
2.04
.00
BCE
2.02
-0.02
Dice
2.01
-0.03
Focal
2.01
-0:03
Jaccard
2.02
..02
Tversky
2.01
-0.03
WCE
2.15
0.12
ablation
ABW
2.33
0.30
m
mDDict
VIL
ART
HD.
Skea Skea-lopo
2.72
0.69
A=D.1
.778
004
.869_ 008 1.188 oTG
-636
cos
.703_ 034
RMI
2.28
0.25
A-0.5
.7784 .006
.869.00 .04 1.176+.070
.6394.00 014
.7294.078
clDice
2.37
0.34
A=0.9
869+.om 77744.003
1.219+,073
.G34..013 .740
CWMI-Real
2.18
0.15
K ablation
CWMI
2.27
0.23
mloUt
mDieet
VIL
ART
HD4
K=2
.777 006
8688 004
1.18 10
6637 a. 028
.792. 02
Table 6. Training time per epoch and relative change to CE bascline for various loss functions on U-Net model with SNEMI3D dataset.
Km4
.7794.00
.870
0os
1.164.09
.640..00 005
.694.04
K=8
.7764.00s
oos
868+.om
002
1.20+ .s
.335 009
.762..o os
K=12
8655..os .7724.005
1:33 05
6214.001
88 882..8
N ablation
5. Conclusion
mloUt
mDicef
VI
ARI
IIDI
N=2
.7001 o02
.8572.00
1.30 04
.618 009
1.20
In this study, we introduced Complex Wavelet Mutual In- formation (CWMI) loss, a novel loss function for semantic segmentation that leverages the multiscale, multi-orientation decomposition capabilities of the complex steerable pyra- mid. By integrating mutual information across wavelet sub- bands, CWMI effectively captures high-dimensional depen- dencies and local structural features, including critical phase information, which are essential for accurate segmentation. Extensive experiments on four challenging datasets demon- strate that CWMI consistently outperforms state-of-the-art loss functions across most evaluation metrics, particularly in segmenting small instances and narrow boundaries, while introducing minimal computational overhead. These results highlight CWML MI robust and versatile loss function that effectively addresses key challenges in segmentation, such as class and instance imbalance, boundary precision, and topological consistency.
N-4
777++.002
869-
001
1. 1.21
6344....
.79+ 190
N=6
.768 003
862g
002
1.29 o5
618 6182.001
288
Layer ablation
mloLt
mDiice
V1
ARTT
IDD.
1st layer
.788 o08
8813 006
1.61 07
.57 015
1.38 14
2nd layer
.7621 008
8581. 000
1.25 os
6244 016
1.12
3ed layer
7744
000
867 = 002
1 10
638 020
822.12
layer
.766+.000 .044
881+,00s 000 1.39+.00 09
.614+.0 012
.7++.04
Table 5. The impact of regularization parameter X, dccomposition level N, and oricntation number K on CWMI performance on U- Net with SNEMI3D. In the layer ablation test, only the selected layer is computed for mutual information and all other layers are discard. The bold numbers indicate the best performance for each metric.
Computational complexity analysis For input image of size H x W with K orientation decompositions, the computational complexity of the CWMI loss function is analyzed as follows.
Beyond semantic segmentation, the core principles of CWMI-multiscale feature extraction and structural con- sistency-sugges its potential applicability a broader range of computer vision and machine learning tasks, such as image-to-image translation and super-resolution, which we leave for future exploration.
First, the forward Fourier transform has time complexity of O(HW lW)( For the first layer of decomposition, the operations include: Band- pass filtering: O(HW K); Inverse Fourier transform: O log( HW)K); and mu- tual information computation: O(HW K2). Summing these terms, the total complexity for the first decompo- sition layer is: O( (K log( (HW) + K2)) For subse- quent decomposition layers, the image size reduces by a factor of four at each step, meaning the second layer pro-
Although extensively studied on 2D images, extending it to multi-class and 3D segmentation is theoretically feasible but necessitates further validation in future research.

---

Complex Wavelet Mutual Information Loss: A Multi-Scale Loss Function for Semantic Segmentation
Acknowledgements
Chen, J., Lu, Y., Yu, Q. Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L.., and Zhou, Y. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04333
This study was conducted without external funding. I would like thank the reviewers for their constructive feedback, which significantly improved the quality of this work. I am especially grateful to my wife, Di Duan, for her unwavering support of my passion for machine learning. thank Xinzi He for valuable discussions that contributed to the revision of this study.
2021.
Diakogiannis, F. I., Waldner, F., Caccetta, P., and Wu, C. Resunct-a: A deep learning framework for semantic seg- mentation of remotely sensed data. ISPRS Journal of Photogrammetry and Remote Sensing, 162:94-114, 2020.
Golub, G. H. and Van Loan, C. F. Matrix computations. JHU press, 2013.
Impact Statement
The proposed Complex Wavelet Mutual Information (CWMI) loss introduces novel approach structural- aware learning in deep neural networks. By leveraging multiscale decomposition through the complex steerable pyramid and mutual information across frequency subbands, CWMI enables improved segmentation performance, partic- ularly for small-scale structures and thin boundaries. Our empirical results demonstrate significant improvements in both pixel-wise and topological accuracy across multiple datasets. Beyond segmentation, CWMI has the potential to generalize to a wide range of real-world applications, includ- ing medical imaging, autonomous driving, satellite-based environmental monitoring, and industrial defect detection.
Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808,860800 2018.
Huang, M., Huang, C., Yuan, J., and Kong, D. Fixed- point deformable u-net for pancreas ct segmentation. In Proceedings of the Third International Symposium on Image Computing and Digital Medicine, pp. 283-287, 2019.
Islam, M., Vibashan, V., Jose, V. M., Wijethilake, N., Utkarsh, U., and Ren, H. Brain tumor segmentation and survival prediction using 3d attention unet. In Brainle- sion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 5th International Workshop, BrainLes 2019, Held in Conjunction with MICCAI 2019, Shenzhen, China, October 17, 2019, Revised Selected Papers, Part I 5, pp. 262-272. Springer, 2020.
Overall, CWMI provides computationally efficient, structure-aware loss function that enhances segmentation performance while maintaining practical scalability. By extending its application beyond segmentation, we aim to contribute to the broader field of generative modeling, ob- ject detection, and sepe-suue learning in deep neural networks.
Jiang, W., Li, Y., Yi, Z., Chen, M., and Wang. J. Multi- instance imbalance semantic segmentation by instance- dependent attention and adaptive hard instance mining. Knowledge- Based Systems, 304: 12554, 2024.
References
Arganda-Carreras, I., Seung, H. S., Vishwanathan, A., and Berger, D. R. Snemi3d: 3d segmentation of neu- rites in em images (isbi 2013). https: //sp:s////mmiiii 3d.
Karimi, D. and Salcudean, S. E. Reducing the hausdorff dis- tance in medical image segmentation with convolutional neural networks. IEEE Transactions on medical imaging, 39(2):499-513, 2019.
grand-challenge..o org/, 2013. Accessed: 2024- 12-20.
Azad. R., Heidary, M., Yilmaz, K., Hüttemann, M., Karim- jjafarbigloo, S., Wu, Y., Schmeink, A., and Merhof, D. Loss functions in the era of semantic segmentation: A survey and outlook. arXiv preprint arXiv:2312.05391, 2023.
Kervadec, H., Bouchtiba, J., Desrosiers, C., Granger, E., Dolz, J., and Ayed, I. B. Boundary loss for highly un- balanced segmentation. In International conference on medical imaging with deep learning, pp. 285 296.296.722 PMLR, 2019.
Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Ben- gio, Y., Courville, A., and Hjelm, D. Mutual information neural estimation. In International conference on ma- chine learning, pp. 531-540. PMLR, 2018
Kim, M. W. and Cho, N. I. Whfl: Wavelet-domain high fre- quency loss for sketch-to-image translation. In Proceed- ings of the IEEE/CVF Winter Conference on applications of computer vision, pp. 744-754, 2023.
Canny, J. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine in- telligence, (6):679-66) -698, 1986.
Kofler, F., Shit, S., Ezhov, L., Fidon, L., Horvath, I,, Al- Maskari, R., Li, H. B., Bhatia, H., Loehr, T., Piraud, M., et al. Blob loss: Instance imbalance aware loss functions
10

---

Complex Wavelet Mutual Information Loss: A Multi-Scale Loss Function for Semantic Segmentation
for semantic segmentation. In International Conference Information Processing in Medical Imaging, pp. 755- 767. Springer, 2023.
Portilla, J. and Simoncelli, E. P. A parametric texture model based on joint statistics of complex wavelet coefficients. International journal of computer vision, 40:49-70, 2000.
Korkmaz, C. and Tekalp, A. M. Training transformer models by wavelet losses improves quantitative and visual perfor- mance in single image super-resolution. arXiv preprint arXiv:2404. / 1273, 2024.
Prantl, L., Bender, J., Kugelstadt, T., and Thuerey, N. Wavelet-based loss for high-frequency interface dynam- ics, arXiv preprint arXiv:2209.0233 02300, 2022.
Rahman, M. A. and Wang. Y. Optimizing intersection-owee union in deep neural networks for image segmentation. In International symposium on visual computing, pp. 234- 244. Springer, 2016.
Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117-2125, 2017.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-asiisted intervention-MICAAI 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234 241. Springer, 2015.
Liu, C., Ma, B., Ban, X., Xie, Y., Wang, H., Xue, W., Ma, J., and Xu, K. Enhancing boundary segmentation for topologiaal accuracy with skeleton- based methods. arXiv preprint ar arXiv:2404. 185399 2024.
Liu, W., Chen, J., Liu, C., Ban, X., Ma, B., Wang, H., Xue, W., and Guo, Y. Boundary learning by using weighted propagation in convolution network. Journal of Compu- tational Science, 62:101709, 2022.
Ross, T.Y. and Dollár, G. Focal loss for dense object detection. In proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2980-2988, 2017.
Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Ruan, J., Li, J., and Xiang, S. Vm-unet: Vision mamba unet for medical image segmentation. arXiv preprint arXiv:2402.02499 2024.
Mallat, S. G. theory for multiresolution signal decom- position: the wavelet representation. IEEE transactions pattern analysis and machine intelligence, 11(7):674- 693, 1989.
Salehi, S. S. M., Erdogmus, D., and Gholipour, A. Tver- sky loss function for image segmentation using 3d fully convolutional deep networks. In International workshop machine learning in medical imaging, pp. 379-387. Springer, 2017.
Milletari, F., Navab, N., and Ahmadi, S.-A. V-net: Fully con- volutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pp. 565-571. Ieee, 2016.
Sampat, M. P., Wang, Z., Gupta, S., Bovik, A. C., and Markey, M. K. Complex wavelet structural similarity: A new image similarity index. IEEE transactions on image processing, 18(11):2355-2011 2009.
Mnih, V. Machine Learning for Aerial Image Labeling. PhD thesis, University of Toronto, 2013.
Shit, S., Paetzold, J. C., Sekuboyina, A., Ezhov, I., Unger, A., Zhylka, A., Pluim, J. P., Bauer, U., and Menze, H. cidice-a novel
Nunez-Iglesias, J., Kennedy, R., Parag, T., Shi, J., and Chklovskii, D. B. Machine learning of hierarchical clus- tering to segment 2d and 3d images. PloS one, 8(8): 271715,
loss function for tubular structure segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18560-16569, 2021.
Oktay, O., Schlemper, J., Folgoe, I.. I.,, Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N. Y., Kainz, B., et al. Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv: 1r0ivv/804.0::: 009999 2018.
Simoncelli, E. P., Freeman, W. T., Adelson, E. H., and Heeger, D. J. Shiftable multiscale transforms. IEEE trans- actions on Information Theory, 38(2):587-607, 1992.
Oner, D., Garin, A., Koziński, M., Hess, K., and Fua, P. Persistent homology with improved locality information for more effective delineation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45588- 10595, 2023.
Sirinukunwattana, K., Pluim, J. P., Chen, H., Qi, X., Heng, P.-A., Guo, Y. B., Wang, L. Y., Matuszewski, B. J., Bruni, E., Sanchez, U., et al. Gland segmentation in colon histology images: The glas challenge contest. Medical image analysis, 35:489-5000 502, 2017.

---

Complex Wavelet Mutual Information Loss: A Multi-Scale Loss Function for Semantic Segmentation
Staal, J., Abråmoff, M. D., Niemeijer, M., Viergever, M. A., and Van Ginneken, B. Ridge-based vessel segmentation in color images of the retina. IEEE transactions on medi- cal imaging. 23(4):501-509, 2004.
Stucki, N., Paetzold, J. C., Shit, S., Menze, B., and Bauer, U. Topologically faithful image segmentation via induced matching of persistence barcodes. In International Con- ference on Machine Learning, pp. 32698-32727. PMLR, 2023.
Tang, Q., Liu, F., Zhang, D., Jiang. J., Tang, X., and Chen, C. P. Increase the sensitivity of moderate examples for semantic image segmentation. Image and Vision Comput- ing, 154: 105357, 2025.
Vinh, N. X., Epps, J., and Bailey, J. Information theoretic measures for clusterings comparison: is a correction for chance necessary? In Proceedings of the 26th annual international conference on machine learning, pp. 1073-
1080, 2009.
Wang. Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. Image quality assessment: from error visibility to struc- tural similarity. IEEE transactions on image processing, 13(4):600-6) 2004.
Woo, S., Park, J.. Lee, J.-Y., and Kweon, I. Cbam: Convolutional block attention module. In Proceedings of the European conference computer vision (ECCV), pp. 2018.
Yang, H.-H., Yang, C.-H. H., and Tsai, Y.-C. J. Y-net: Multi- scale feature aggregation network with wavelet structure similarity loss function for single image dehazing. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 2628-2632. IEEE, 2020.
Yue. Y. and Li. Z. Medmamba: Vision mamba for medical image classification. arXiv preprint 033449240..0384, 2024.
Zhao, S., Wang, Y., Yang, Z., and Cai, D. Region mutual information loss for semantic segmentation. Advances in Neural Information Processing Systems, 32, 2019.
Zhou, Z., Rahman Siddiquee, M. M., Tajbakhsh, N., and Liang, J. Unet++: A nested u-net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Con- junction with MICCAI 2018. Granada, Spain, September 20, 2018, Proceedings 4, pp. 3-11. Springer, 2018.
Zhu, Q. Wang, H., and Zhang, R. Wavelet loss function for auto-encoder. IEEE Access, 9:27101 2021.
12


Reviews:
[{"review_id": "5tKWzpeBmk", "reviewer": "Reviewer_p6uN", "summary": "This paper introduces Complex Wavelet Mutual Information (CWMI) loss for semantic segmentation tasks. The proposed method employs a complex steerable pyramid to perform multi-scale and multi-orientation wavelet decomposition on both prediction and label images, computing mutual information in each subband to effectively capture local phase and structural features. This design not only alleviates issues of class and instance imbalance but also maintains computational efficiency. Experiments conducted on SNEMI3D, GlaS, DRIVE, and MASS ROAD datasets demonstrate that CWMI loss significantly outperforms traditional pixel-wise and region-based loss functions in terms of pixel accuracy and topological consistency metrics, while incurring minimal computational overhead. This work provides a novel perspective and solution for structure-sensitive segmentation tasks, showcasing strong generalization and promising practical applicability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We especially appreciate the recognition of the novelty and practical value of our proposed CWMI loss. Your comments helped us refine the manuscript and gain deeper insights into both theoretical and empirical aspects of our method.\n\nBelow we address your main concerns. Due to space constraints, full updated tables are omitted here but will be included in the revised manuscript.\n\n### 1. Statistical Significance\nWe appreciate the reviewer’s observation regarding the importance of statistical significance in validating the effectiveness of CWMI. To address this, we followed Bouckaert & Frank (2004) and repeated three-fold cross-validation three times with adjusted t-tests across all metrics.\n\nWe summarize the key findings as follows:\n\n- On SNEMI3D and DRIVE, CWMI significantly outperforms all SOTA loss functions.\n\n- On GlaS and MASS ROAD, where multiscale structural features are less dominant, CWMI significantly outperforms most baselines and is competitive with RMI. Specifically, CWMI shows significantly better performance in VI and ARI metrics compared with RMI, while showing marginal improvement compared with RMI on mDice and mIoU.\n\n### 2. Ablation Study\nWe thank the reviewer for pointing out the need to deepen our ablation study and further dissect the contribution of the complex representation in CWMI. We extended our ablation study to include magnitude-only and phase-only variants:\n\n- CWMI-Phase: consistently underperformed across all metrics, indicating that phase alone lacks sufficient discriminative power for accurate segmentation.\n\n- CWMI-Mag: achieved the best performance on VI, ARI, and Hausdorff Distance (HD), suggesting that the magnitude component is particularly effective at capturing clustering and topological structures.\n\n- CWMI (full representation) achieved the best performance on pixel-wise accuracy metrics (mIoU and mDice) and was the second-best on VI, ARI, and HD, offering a well-balanced advantage across regional, clustering, and structural metrics.\n\n- CWMI-Real performed worse than both CWMI and CWMI-Mag on most metrics, though better than CWMI-Phase, further validating the importance of combining both magnitude and phase information.\n\nThese results indicate that while magnitude contributes strongly to clustering and topological consistency, the full CWMI formulation provides a more comprehensive optimization across all segmentation dimensions. Notably, our findings also suggest that a magnitude-only version of CWMI could be explored as a potential alternative for faster training or lightweight deployment scenarios.\n\n### 3. Hyperparameter $\\lambda$ Sensitivity\nWe thank the reviewer for raising this important point. To assess the sensitivity of the regularization parameter $\\lambda$, we tested $\\lambda = 0.1, 0.5, 0.9$ and observed no significant variation in metrics, indicating that CWMI contains sufficient complementary information to cross-entropy, and is robust to $\\lambda$. Additionally, we found the cross-entropy term essential for directional learning, as mutual information is symmetric and cannot differentiate between correct and inverted predictions.\n\n### 4. MI Estimation and Deep Estimators\nWe thank the reviewer for the insightful questions on MI estimation and the potential use of deep MI estimators. Our current method follows the RMI loss (Zhao et al., 2019), which estimates a lower bound of mutual information under the assumption of approximately Gaussian-distributed feature vectors. While this approach is efficient and non-parametric, its tightness depends on distributional assumptions. Our preliminary analysis shows that the complex wavelet subbands are near-Gaussian but with sharper central peaks, indicating the bound may not be perfectly tight. We will acknowledge this limitation in the revised manuscript.\n\nWe also reviewed learned MI estimators such as MINE and Deep InfoMax, which offer tighter bounds via parameterized networks. While promising, these methods introduce additional training and complexity, potentially undermining CWMI’s computational efficiency. As future work, we plan to integrate MINE into our framework to study the trade-offs between accuracy and overhead.\n\n### 5. Multi-Class and 3D Segmentation\nCWMI is extendable to multi-class and 3D segmentation. Both Fourier transform and MI estimation naturally generalize to 3D. Although steerable pyramids are 2D by design, prior work (e.g., Delle Luche et al., 2004) introduces 3D steerable filters using polyhedral decompositions. We plan to explore these extensions in future work.\n\n### 6. Minor Comments\nWe will (1) clarify grid search was applied to all baselines, (2) add inline citations to comparison tables, (3) expand textual explanations for key equations (e.g., Eq. 10), and (4) correct the typo in Figure 3 (\"Organge\" $\\rightarrow$ \"Orange\"). We appreciate the reviewer’s attention to these details.", "claims_and_evidence": "The main claim presented in the paper—*“CWMI loss improves semantic segmentation performance, particularly for small instances and fine boundaries”*—is clearly stated and supported by evidence that is convincing to a certain extent. The manuscript provides a thorough discussion of various perspectives on the problem and articulates the technical background comprehensively. The reasoning behind choosing a frequency-domain approach, particularly using wavelet transforms for designing the loss function, is clearly explained. Moreover, the overview of semantic segmentation loss functions is appropriately detailed, explicitly linking the challenges of instance imbalance and the need for more efficient and balanced approaches. Quantitative results, qualitative visualizations, and ablation studies are comprehensive, collectively demonstrating the superiority of CWMI across multiple datasets. Tables 1 and 4 effectively illustrate CWMI’s strong performance in modeling large-scale structural dependencies while also demonstrating advantages in parameter efficiency.\n\nGPT previously mentioned that the description of computational overhead was insufficiently specific, weakening the claim regarding efficiency; however, upon review, this seems adequately addressed.\n\nNevertheless, the provided evidence has certain shortcomings:\n\n1. **Limited depth in ablation studies**, which fails to fully elucidate the mechanisms behind CWMI’s advantages.\n\nThe current ablation experiments only compare the complete complex representation (CWMI) against the real-only representation (CWMI-Real), showing that utilizing the complex representation (which simultaneously incorporates both magnitude and phase information) achieves better segmentation performance. However, the paper does not further dissect this complex representation to separately evaluate the contributions of magnitude (reflecting feature intensity) and phase (reflecting local structural information). Consequently, existing experiments cannot definitively determine whether the improvements are due to the entire complex representation or specifically driven by either magnitude or phase components individually.\n\n2. **Lack of statistical significance testing** (e.g., t-tests or confidence interval analysis), making it difficult to confirm whether the reported performance improvements exceed the range of random fluctuations, thereby reducing the reliability and persuasive power of the results.", "methods_and_evaluation_criteria": "(1) The proposed CWMI loss and the associated evaluation metrics are reasonable and appropriate for addressing the semantic segmentation problem tackled in this paper. The method's design leverages frequency-domain analysis and statistical dependencies to handle multiscale challenges, aligning logically with the intended application. The choice of datasets and evaluation metrics effectively captures the complexity of the issues—namely class/instance imbalance and structural preservation—thus ensuring a robust evaluation of the proposed method. Furthermore, the utilization of standard segmentation models grounds the evaluation within realistic application scenarios. Overall, I see no significant mismatch between the proposed method, the evaluation framework, and the goal of improving segmentation performance, particularly for small-scale instances and fine boundaries. In summary, both the method and evaluation criteria are well-suited to the task at hand and provide a solid foundation for validating the paper's contributions.\n\n(2) The CWMI loss and its evaluation framework are generally suitable for semantic segmentation tasks. By employing the complex steerable pyramid and multi-scale, structure-aware mutual information design, the method effectively addresses class and instance imbalance. Combining CWMI with cross-entropy loss further enhances its broad applicability. The evaluation extensively tests the approach across multiple datasets and metrics. However, regarding hyperparameter configuration, the paper predominantly uses a fixed value of λ = 0.1 and performs grid search hyperparameter tuning only for certain baseline loss functions. The limited discussion on hyperparameter sensitivity and broader tuning strategies may impact the method's generalizability and extensibility.", "theoretical_claims": "As a reviewer, I have verified the correctness of the CWMI loss formulation (Equations 6-10)—the central theoretical claim of the paper—and found it mathematically sound, consistent with standard definitions of mutual information and the design of loss functions. Since the paper prioritizes empirical validation and does not provide formal proofs, it is acceptable that I did not need to verify theoretical proofs. However, the theoretical claims could be enhanced by providing theoretical justifications for the effectiveness of mutual information, clearly articulating the MI estimation process and its theoretical implications, and clarifying the rationale behind hyperparameter choices. These points are not correctness issues but rather theoretical elaboration shortcomings, which, if addressed, would make the proposed framework more convincing and robust beyond its demonstrated empirical success.", "experimental_designs_or_analyses": "I have reviewed the experimental design (datasets, baselines, models, metrics, ablations) and analyses (quantitative, qualitative, computational) for reasonableness and validity and found them to be generally robust. The design choices—including diverse datasets, comprehensive metrics, and fair baselines—adequately test the effectiveness of CWMI. The use of cross-validation and multiple models further strengthens validity. The analysis supports the claimed performance improvements, particularly regarding small instances and boundary details.\n\nHowever, there are some issues:  \n(1) **Lack of statistical significance tests**, weakening the reliability of the results.  \n(2) **Shallow ablation studies**, limiting insights into underlying mechanisms.  \n(3) **Incomplete computational analysis**, reducing clarity on practical usability.\n\nThese concerns do not invalidate the experiments but suggest potential improvements—such as including statistical tests, deepening ablation studies (e.g., detailed feature analyses, hyperparameter sensitivity), and providing detailed breakdowns of running time—to further strengthen reasonableness and validity.\n\nAdditionally, it might be beneficial to add citations directly into the comparison tables of baseline experiments to improve clarity and traceability.", "relation_to_broader_scientific_literature": "The Complex Wavelet Mutual Information (CWMI) loss proposed in this paper makes a significant contribution to the field of semantic segmentation and closely aligns with broader scientific literature. Firstly, the CWMI loss addresses long-standing challenges in semantic segmentation, such as class imbalance and instance imbalance, particularly difficulties related to segmenting small objects and fine boundaries. Traditional loss functions like Cross-Entropy (CE) and Dice have limited effectiveness in tackling these issues. By drawing inspiration from recent region-based losses such as Regional Mutual Information (RMI) loss proposed by Zhao et al. and topology-aware losses like clDice loss introduced by Shit et al., CWMI loss incorporates a complex steerable pyramid to substantially enhance the model’s ability to capture multi-scale and multi-orientation features, significantly improving segmentation accuracy and efficiency.\n\nSecondly, CWMI loss integrates classic wavelet transform theories (such as Mallat’s multi-scale wavelet decomposition and Simoncelli’s steerable pyramid) with the concept of mutual information. This innovative fusion of information-theoretic approaches with deep learning strengthens the model’s capability in modeling high-dimensional dependencies and structural consistency.\n\nFurthermore, the core principles of CWMI—multi-scale feature extraction and structural consistency—possess broad applicability across diverse fields such as medical imaging and remote sensing, extending to tasks including image super-resolution and dehazing. Overall, the CWMI loss not only advances the field of semantic segmentation but also introduces novel perspectives and methodologies applicable to a broader range of computer vision tasks.", "essential_references_not_discussed": "Upon review, the current manuscript has already referenced many relevant studies related to regional mutual information and wavelet decomposition. However, it still omits several important recent developments in the fields of high-dimensional mutual information estimation and deep representation learning. For example, Mutual Information Neural Estimation (MINE) proposed by Belghazi et al. (2018) and Deep InfoMax by Hjelm et al. (2019) have demonstrated excellent performance in robust estimation of high-dimensional mutual information. These works could significantly complement the theoretical justification behind the mutual information lower-bound estimation approach adopted by CWMI loss. Furthermore, recent advances in wavelet information theory, such as those described by de Oliveira & de Souza (2015), could provide additional theoretical context to better understand the role of complex wavelet decomposition in capturing multi-scale structural features.\n\nBelow are the links to the aforementioned relevant papers:\n\n1. **Mutual Information Neural Estimation (MINE)** – Belghazi et al. (2018)  \nLink: [https://arxiv.org/abs/1801.04062](https://arxiv.org/abs/1801.04062)\n\n2. **Deep InfoMax** – Hjelm et al. (2019)  \nLink: [https://arxiv.org/abs/1808.06670](https://arxiv.org/abs/1808.06670)\n\n3. **Wavelet Analysis as an Information Processing Technique** – de Oliveira & de Souza (2015)  \nLink: [https://arxiv.org/abs/1502.05879](https://arxiv.org/abs/1502.05879)", "other_strengths_and_weaknesses": "The proposed CWMI loss combines a complex steerable pyramid with mutual information, presenting a novel multi-scale loss function that effectively addresses class imbalance and instance imbalance in semantic segmentation. This innovative combination not only enhances segmentation accuracy but also demonstrates excellent computational efficiency, offering broad potential for applications, particularly in medical imaging and remote sensing. The experimental section is rigorously designed, clearly validating the superiority of the CWMI loss through comparative experiments across multiple publicly available datasets.\n\nHowever, the paper also has certain limitations. First, although CWMI exhibits strong performance on 2D images, the applicability of CWMI to multi-class and 3D segmentation tasks has not been thoroughly investigated, limiting its generalizability. Second, despite supplementary materials containing some ablation experiments, the theoretical analysis of CWMI loss remains relatively shallow. Specifically, the mathematical foundations underlying its capability to capture multi-scale dependencies in the wavelet domain could be explored in greater depth.\n\nOverall, the paper excels in methodological innovation and experimental validation but leaves room for improvement in terms of theoretical depth and applicability extensions.", "other_comments_or_suggestions": "The overall structure of the paper is clear, but several points could be improved. Firstly, some equations (such as the complex covariance calculation in Equation 10) could benefit from additional textual explanations to help readers understand the underlying derivation logic. Additionally, a minor spelling error appears in Figure 3, where \"Organge arrows\" should be corrected to \"Orange arrows.\"", "questions_for_authors": "1. **Regarding the estimation method of the lower bound of Mutual Information (MI):**  \nCould you please elaborate on how you ensure a tight approximation between the MI lower bound used in your method and the true MI value? A significant gap between this lower bound and the actual MI could impact the loss function's ability to effectively capture structural information, thus affecting the robustness and validity of the proposed approach.\n\n2. **Relationship between Complex Wavelet Decomposition and recent MI estimation methods:**  \nHave you considered integrating recent advanced deep mutual information estimation techniques, such as MINE or Deep InfoMax, into your approach? If these methods could further enhance MI estimation accuracy, it might positively influence the performance of the CWMI loss, consequently strengthening the overall competitiveness of your method.\n\n3. **Hyperparameter λ selection and sensitivity:**  \nCould you provide more detailed insights into how the regularization parameter λ was chosen in your experiments, and whether you performed a systematic sensitivity analysis of this parameter? If the final results are highly sensitive to λ, detailed guidance on parameter optimization would be necessary, influencing the perceived stability and general applicability of your method in real-world scenarios.\n\n4. **Applicability to multi-class and 3D segmentation tasks:**  \nAlthough your paper primarily addresses two-dimensional semantic segmentation, do you have any preliminary ideas or experimental results regarding extending your method to multi-class or three-dimensional segmentation tasks? Demonstrating adaptability to these more complex scenarios would significantly enhance the practical value of your approach, potentially broadening the evaluation of your paper's contributions.\n\n5. **Computational efficiency and real-time applications:**  \nThe CWMI loss combines complex wavelet transforms with mutual information calculations, potentially resulting in substantial computational overhead. Have you considered or planned any optimizations to improve computational efficiency, particularly for high-resolution or real-time applications? If computational complexity could be significantly reduced without compromising performance, it would substantially enhance the practicality and usability of your method in real-world contexts.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review."}}, {"review_id": "MuAEpPoS4N", "reviewer": "Reviewer_go7N", "summary": "This paper proposed a complex wavelet-based loss function and proved its effectiveness in several segmentation tasks.\n\n## update after rebuttal”\n\nAfter I review the author's response and other reviewers' comments, I raise my score to 2 to thank the authors for their efforts. No higher score because the contribution doesn't reach the bar of ICML.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and the opportunity to clarify our motivation and contributions. We respectfully disagree with the assessment that the proposed CWMI loss function is incremental and provides only marginal improvements over traditional losses like L1 or L2. Below, we address each concern in detail.\n\n### **1. Empirical Gains Are Statistically Significant, Not Marginal**\nTo support the significance of our results, we conducted additional statistical analyses following Bouckaert and Frank (2004), repeating three-fold cross-validation multiple times and applying adjusted t-tests. These analyses demonstrate that CWMI provides statistically significant improvements over both L1/L2 and several state-of-the-art loss functions across multiple datasets and evaluation metrics (including mIoU, mDice, VI, ARI, and HD). While some improvements may appear modest in absolute terms, they are consistent, reproducible, and statistically robust—highlighting the practical effectiveness of CWMI, particularly in tasks requiring fine-grained boundary and structural precision.\n\n### **2. Novelty of Wavelet Integration in Loss Design**\nWe believe that incorporating wavelet-based multiscale decomposition into the loss function is a novel and underexplored direction in semantic segmentation. While wavelet transforms have been increasingly adopted in network architectures—e.g., WaveletNet (Jing et al., 2018), WavResNet (Kang et al., 2017), and SwinWave-SR (Dharejo et al., 2024)—these methods use wavelets primarily for efficient encoding or downsampling operations in the feature extractor. By contrast, CWMI is, to our knowledge, the first loss function to directly supervise predictions in the complex wavelet domain, thereby leveraging rich frequency and orientation-aware representations at multiple scales. This framing allows us to encode structured priors about image content directly into the loss signal. Moreover, our ablation studies show that even conventional L1/L2 losses applied in the wavelet domain outperform sophisticated spatial-domain losses like RMI, Tversky, or clDice. This highlights that multiscale representations—when integrated into the objective function—offer a powerful and underutilized mechanism for improving segmentation quality. We believe CWMI opens a new line of inquiry into wavelet-domain loss functions.\n\n### **3. Combining Wavelet Transform with Mutual Information Is New**\nMutual information–based losses (e.g., RMI) have emerged as strong alternatives to standard pixel-wise losses by capturing statistical dependencies between prediction and ground truth. Our work extends this promising direction by combining MI estimation with wavelet-domain features—specifically, complex steerable pyramid coefficients. This combination enables CWMI to align multiscale, orientation-sensitive representations in a statistically principled way. To our knowledge, no prior work has explored this integration. Unlike RMI, which operates in the spatial domain, CWMI captures fine-grained structural patterns across scales and directions, aligning them through mutual information. This results in a loss function that is both information-theoretically grounded and structurally aware. Our empirical results—especially the consistent superiority of CWMI over RMI across datasets and model backbones—further validate the effectiveness and novelty of this design.\n\n### **4. Relation to BiconNet and Related Work**\nWe thank the reviewer for pointing out relevant works such as BiconNet and other connectivity-aware segmentation frameworks. These contributions are highly relevant to structure-preserving segmentation, and we will include a detailed discussion in the revised manuscript. Importantly, CWMI is complementary to such methods and could potentially be used as a regularization term to further enhance structural consistency in connectivity-based models. We are particularly interested in exploring this synergy in future work, as integrating CWMI into BiconNet may combine topological priors with frequency-domain mutual information.", "claims_and_evidence": "The reviewer is confused about the motivation and thinks the contribution of this paper is incremental. Please see details in the section on weaknesses.", "methods_and_evaluation_criteria": "Several concerns are raised in the section on weaknesses.", "theoretical_claims": "yes, the theoretical claims are weak in the paper.", "experimental_designs_or_analyses": "see weaknesses", "relation_to_broader_scientific_literature": "BiconNet: An Edge-preserved Connectivity-based Approach for Salient Object Detection\nDirectional Connectivity-based Segmentation of Medical Images\nSpatial coherence loss for salient and camouflaged object detection and beyond", "essential_references_not_discussed": "BiconNet: An Edge-preserved Connectivity-based Approach for Salient Object Detection\nDirectional Connectivity-based Segmentation of Medical Images\nSpatial coherence loss for salient and camouflaged object detection and beyond", "other_strengths_and_weaknesses": "Strength: Introducing complex wavelet-based mutual information loss\n\nWeaknesses:\n1. The contribution of this paper is minimal and experiments (ablation) also verify this modification is incremental (Using this loss can only surpass traditional L1 or L2 loss in margin increase).\n2. The motivation of this paper is also confused and the experiments are not convincing.\n\nGiven the above contribution, I recommend to reject this paper.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "see weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nAwVNDtljC", "reviewer": "Reviewer_ngWU", "summary": "This work presents a loss function for semantic segmentation based on the steerable pyramid decomposition of images, *i.e.*, wavelet transform. The insight is that the steerable pyramid preserves structural similarity while capturing multi-scale features across multiple orientations. The proposed CWMI consists of several mutual information terms that are well-suited for directional features. CWMI is validated on U-Net and AttenUNet using four benchmarks. Qualitative visualizations suggest that CWMI helps reduce false positive and false negative predictions.\n\n**Update after rebuttal**: I am going to maintain my current rating at 2-WEAK REJECT due to insufficient experimental studies.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedback and the helpful list of recent models and loss functions for comparison. In response, we have conducted additional experiments using Vision Mamba UNet (VM-Unet, Ruan et al, 2024), a segmentation architecture based on the recently proposed MedMamba ([r5])—a direction gaining rapid attention in the vision community due to its sequence modeling efficiency and competitive performance.\n\nThe table below summarizes our CWMI results on VM-UNet and the SNEMI3D dataset. CWMI demonstrates consistent improvements over all tested baseline loss functions, including RMI, Dice, and sensitive loss [r7]:\n\n||mIoU↑|mDice↑|VI↓|ARI↑|HD↓|\n|:-------:|:------:|:-------:|:-------:|:-------:|:-------:|\n|CE|0.723±0.021|0.829±0.016|2.58±0.99|0.414±0.141|1.71±0.19|\n|CE|0.723±0.021|0.829±0.016|2.58±0.99|0.414±0.141|1.71±0.19|\n|BCE|0.739±0.003|0.843±0.002|1.50±0.05|0.594±0.008|1.27±0.19|\n|BCE|0.739±0.003|0.843±0.002|1.50±0.05|0.594±0.008|1.27±0.19|\n|Dice|0.725±0.024|0.831±0.017|2.74±0.99|0.402±0.135|1.77±0.59|\n|Focal|0.725±0.008|0.833±0.006|1.68±0.12|0.563±0.020|1.70±0.30|\n|Jaccard|0.725±0.016|0.831±0.011|1.95±0.44|0.499±0.061|1.72±0.46|\n|Tversky|0.737±0.013|0.840±0.009|1.62±0.26|0.555±0.037|1.56±0.33|\n|WCE|0.714±0.004|0.826±0.004|1.74±0.02|0.552±0.005|1.43±0.14|\n|ABW|0.678±0.003|0.800±0.002|2.08±0.02|0.490±0.004|1.57±0.02|\n|RMI|0.762±0.008|0.857±0.006|1.42±0.15|0.588±0.025|1.07±0.25|\n|clDice|0.714±0.001|0.824±0.001|1.75±0.05|0.542±0.003|1.99±0.21|\n|Sensitive|0.539±0.115|0.625±0.143|6.12±0.73|0.032±0.055|2.27±0.76|\n|**CWMI**|**0.783±0.004**|**0.872±0.002**|**0.98±0.09**|**0.660±0.018**|**0.63±0.06**|\n\nWe also conducted a comprehensive comparison between CWMI and the sensitive loss from [r7] across all four datasets (SNEMI3D, GlaS, DRIVE, MASS ROAD) using both U-Net and Attention U-Net. In every case, CWMI outperformed the sensitive loss across pixel-wise, region-based, and topology-aware metrics. An example of the results on the GlaS dataset is shown below:\n\n||||SNEMI3D||||\n|:-------:|:------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n||| mIoU↑|mDice↑|VI↓|ARI↑|HD↓|\n|**U-Net**|Sensitive|0.762±0.002|0.857±0.002|1.60±0.03|0.563±0.010|0.99±0.14|\n||CWMI|0.778±0.003|0.869±0.002|1.15±0.04|0.642±0.006|0.75±0.09|\n|**AttenUNet**|Sensitive|0.752±0.001|0.850±0.001|2.04±0.23|0.495±0.041|1.10±0.10|\n||CWMI|0.777±0.003|0.868±0.002|1.14±0.01|0.642±0.002|0.79±0.02|\n\n||||GlaS||||\n|:-------:|:------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n||| mIoU↑|mDice↑|VI↓|ARI↑|HD↓|\n|**U-Net**|Sensitive|0.816±0.012|0.893±0.008|0.97±0.11|0.675±0.027|3.12±0.17|\n||CWMI|0.845±0.018|0.911±0.012|0.71±0.10|0.752±0.025|2.64±0.64|\n|**AttenUNet**|Sensitive|0.820±0.015|0.897±0.009|0.88±0.12|0.694±0.040|2.70±0.23|\n||CWMI|0.841±0.018|0.909±0.011|0.76±0.09|0.733±0.039|2.76±0.71|\n\n||||DRIVE||||\n|:-------:|:------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n||| mIoU↑|mDice↑|VI↓|ARI↑|HD↓|\n|**U-Net**|Sensitive|0.757±0.006|0.846±0.006|1.49±0.03|0.356±0.032|2.96±0.45|\n||CWMI|0.800±0.003|0.880±0.003|1.04±0.04|0.605±0.026|1.03±0.30|\n|**AttenUNet**|Sensitive|0.754±0.015|0.843±0.013|1.45±0.04|0.384±0.027|3.16±0.56|\n||CWMI|0.799±0.013|0.879±0.010|1.01±0.10|0.618±0.032|1.15±0.30|\n\n||||MASS_ROAD||||\n|:-------:|:------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n||| mIoU↑|mDice↑|VI↓|ARI↑|HD↓|\n|**U-Net**|Sensitive|0.738±0.016|0.830±0.013|2.98±0.69|0.282±0.123|11.52±4.74|\n||CWMI|0.768±0.010|0.855±0.008|1.11±0.24|0.674±0.066|10.07±2.18|\n|**AttenUNet**|Sensitive|0.742±0.008|0.834±0.006|2.66±0.46|0.335±0.096|10.97±2.40|\n||CWMI|0.763±0.002|0.850±0.002|1.13±0.08|0.672±0.025|7.95±2.52|\n\nWe recognize their relevance and will include a discussion of their strengths and the potential role CWMI could play as a structural loss component in these systems. However, we were unable to evaluate CWMI within the full pipelines of recent universal segmentation models ([r1]–[r3]) due to their architectural complexity and multi-objective training design. Similarly, although we could not reproduce [r6] due to unavailable code, we will cite and acknowledge it in our revision.\n\nIn summary, we prioritized experiments on [r5] due to its relevance and recency and performed a thorough evaluation against [r7], showing CWMI’s consistent improvements in both classic and modern architectures. We thank the reviewer again for these valuable suggestions and for encouraging us to broaden our empirical evaluation.", "claims_and_evidence": "Reasonable claims and sufficient analysis.", "methods_and_evaluation_criteria": "The proposed CWMI and evaluation make sense to the studied problem.", "theoretical_claims": "The equations are either based on the strong foundation of steerable pyramid decomposition or are straightforward.", "experimental_designs_or_analyses": "The proposed CWMI is simple and effective on U-Net and AttenUNet. However, U-Net and AttenUNet are classic methods that were published a long time ago. It is necessary to include more recent SOTA frameworks to validate the effectiveness of the proposed loss function, such as [r1]–[r5]. There are also other works (e.g., [r6][r7]) that focus on developing novel loss functions for semantic segmentation. If the proposed loss function could outperform them, the value of your submission would be significantly enhanced.\n\n[r1] Chen et al., Masked-attention mask transformer for universal image segmentation. In CVPR 2022.\n\n[r2] Wang et al., InternImage: Exploring Large-Scale Vision Foundation Models With Deformable Convolutions. In ICCV 2023.\n\n[r3] Kirillov et al., Segment Anything, In ICCV 2023.\n\n[r4] Tang et al., Category feature transformer for semantic segmentation. In arXiv 2023.\n \n[r5] Yue et al., MedMamba: Vision Mamba for Medical Image Classification. In arXiv 2024.\n\n[r6] Wu et al., Conditional Boundary Loss for Semantic Segmentation. In TIP 2023.\n\n[r7] Tang et al., Increase the sensitivity of moderate examples for semantic image segmentation. In IMAVIS 2025.", "relation_to_broader_scientific_literature": "The proposed CWMI is potentially a network-agnostic loss function and could be applied to any existing semantic segmentation framework.", "essential_references_not_discussed": "Refer to **Experimental Designs Or Analyses**.", "other_strengths_and_weaknesses": "N/A.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "N/A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces Complex Wavelet Mutual Information (CWMI) loss for semantic segmentation tasks. This approach is presented as a novel solution that combines a complex steerable pyramid with mutual information.

Strengths:
- CWMI loss seems to be effective for structure-sensitive segmentation tasks and improves semantic segmentation performance, especially for small instances and fine boundaries.
- The method design takes advantage of frequency-domain analysis and statistical dependencies to handle multiscale challenges, which logically corresponds to the intended application.
- Quantitative results, qualitative visualizations and ablation studies are deemed almost complete, and collectively demonstrate CWMI's superiority over several data sets
- Tables 1 and 4 effectively illustrate CWMI's strong performance in modeling large-scale structural dependencies, while demonstrating the benefits in terms of parameter efficiency.

Weaknesses:
- Lack of further theoretical justification for the effectiveness of mutual information and clarification of the MI estimation process and choice of hyperparameters are noted
- Lack of separate evaluation of the magnitude and phase contributions of complex wavelets does not allow a full understanding of the mechanisms at work behind the CWMI loss benefits

Despite the correct presentation (some typo in eq.3), a few points remain, such as the lack of depth in the analysis (ablation, statistical, theoretical, computational), and the need to validate the method on more recent architectures and compare it with other state-of-the-art approaches. But the authors have taken account of the comments made and provided additional validation elements with complementary tests that remain in favor of their proposal. If the authors actually implement the changes they have produced in their responses, the article will be significantly improved.
Decision: Accept (poster)