Deep Complex Spatio-Spectral Networks with Complex Visual Inputs

Authors: Saurabh Yadav, Koteswar Rao Jerripothula
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Deep Complex Newtworks, Complex-valued color transformation

Abstract:
Complex-valued neural networks have attracted growing attention for their ability to handle complex-valued data with enhanced representational capacity. However, their potential in computer vision remains relatively untapped. 
In this paper, we introduce Deep Complex Spatio-Spectral Network (DCSNet), a fully complex-valued token-based, end-to-end neural network designed for binary segmentation tasks. Additionally, our DCSNet encoder can be used for image classification in the complex domain. We also propose an invertible real-to-complex (R2C) transform, which generates two complex-valued input channels, complex intensity and complex hue, while producing complex-valued images with distinct real and imaginary components.
DCSNet operates in both spatial and spectral domains by leveraging complex-valued inputs and complex Fourier transform.
As a result, the complex-valued representation is maintained throughout DCSNet, and we avoid the information loss typically associated with Real$\leftrightarrow$Complex transformations. Extensive experiments show that DCSNet surpasses existing complex-valued methods across various tasks on both real and complex-valued data and achieves competitive performance compared to existing real-valued methods, establishing a robust framework for handling both data types effectively.

Decision: reject
Reviewer scores: 6, 3, 6, 8

Reviews and discussion:
<thread>
creation date:
1730213020086

thread_type:
official_review

<thread_object>
creator: reviewer_b2bv

<content>
summary:
This study investigates a novel complex-valued deep CNN designed for foreground extraction. It proposes a new method for encoding RGB images as complex values, an end-to-end token-based architecture that maintains the complex representation throughout, and an improved training pipeline. The authors demonstrate superior performance on a variety of complex-valued image benchmarks.

strengths:
I am unfamiliar with the literature on complex-valued neural networks, and defer to the opinion of more reviewers. As a neophyte to this field, I found the manuscript overall to be very readable, well-written, interesting, and convincing. The novel encoding, architecture, and training pipeline seem to work well, and produce a very capable model for handling complex-valued inputs.

weaknesses:
This may be my ascribed to my naivety for the field, but I'm unsure how to interpret the benchmark results. While DCSNet certainly seems to outperform other complex-valued neural networks, the margin of victory is often in the range of 1-5 percent. It is difficult to tell whether this represents a fundamental advance, or a marginal improvement. Further, Table 2 seems to indicate that complex-valued neural networks in general frequently fail to outperform their real-valued counterparts. How should these results be interpreted with respect to the broader viability of complex-valued neural networks?

questions:
See weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730292223401

thread_type:
official_review

<thread_object>
creator: reviewer_qqah

<content>
summary:
This paper introduces the Deep Complex Patio-Spectral Network (DCSNet), a fully complex-valued, token-based neural network developed for end-to-end foreground extraction and adaptable for image classification. Extensive experiments show that DCSNet surpasses current complex-valued approaches across various tasks with real and complex-valued data, achieving results on par with leading real-valued models.

strengths:
This paper presents a novel complex-valued neural network, the Deep Complex Patio-Spectral Network (DCSNet), a fully complex-valued, token-based, end-to-end architecture designed for foreground extraction and adaptable to image classification tasks. Extensive experiments demonstrate that DCSNet outperforms existing complex-valued methods across diverse tasks involving both real and complex-valued data, achieving competitive results relative to state-of-the-art real-valued models. The paper is well-written, concise, and easy to follow.

weaknesses:
Novelty: The novelty of this paper appears questionable. The authors claim that they propose the first token-based complex-valued network that maintains complex-valued information throughout. However, the fully Complex-valued Convolutional Network (FCCN) [1] also processes complex-valued data through the entire model. Could the authors clarify any differences between these two models? What advantages does DCSNet offer over FCCN?

[1] Saurabh Yadav; Koteswar Rao Jerripothula, FCCNs: Fully Complex-valued Convolutional Networks using Complex-valued Color Model and Loss Function. ICCV 2023.

Complex-valued Image Generation (R2C Method): The authors propose an R2C method for generating complex-valued images from real-valued images, presenting it as a novel complex-valued color transformation. However, methods such as quaternion representation, complex logarithmic transformation, and the Hilbert transform are well-established for generating complex-valued images. Could the authors specify the advantages of the R2C method over these alternatives?

DCSNet Architecture:
1. Fourier filters replace self-attention in DCSNet to retain information within the complex domain while preserving global context. How do Fourier filters achieve global information retention in this context, and why were they chosen?
2. The paper briefly mentions dense tokens for image embedding but doesn’t fully explain their purpose. Are these tokens meant to capture pixel-level details (dense information) of the image? If so, why not use high-resolution Fourier filters as localized filters to capture this information directly?
3. If large Fourier filters serve as global filters in the frequency domain while dense tokens capture image details, could a bank of wavelet filters offer a more effective solution? Wavelet filters with multiple resolutions could extract both global (large scale) and local (small scale) image features.

Resolution Tokens: The paper mentions multiple resolution tokens \T_{i} for i \in {0,1,2,3}. What was the reasoning for using exactly four resolutions? Would using more or fewer resolutions impact the results?

Table 7 Clarification: In Table 7, there is a term \calL_{isal}. Is this a typo? Please clarify its meaning if not.

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730527140121

thread_type:
official_review

<thread_object>
creator: reviewer_f89z

<content>
summary:
The work proposes a complex-value deep neural network for computing vision tasks. First, the authors propose an inevitable real-to-complex transformation. Then, the work proposes an architecture comprising spectral convolution and a complex T2T module. 
The authors evaluated their model on image classification, smooth object detection, and defocus blur detection.

strengths:
The work proposes a novel invertible real-to-complex conversion for RGB images in complex-valued neural networks. The procedures are clearly stated using pseudocode and figures.

weaknesses:
1. The authors seem to miss the seminal work on complex values networks and didn’t compare/discuss with the techniques discussed in [1]
2. The work is directed at using complex-valued networks for real-valued images. The majority of the paper involves devising an invertible conversion from real to complex representation. However, the paper fails to demonstrate its utility. For example, for classification on ImageNets, the authors did not consider state-of-the-art models, such as Vit, Swin-v2, etc., which achieve above 90% accuracy. 

3. The paper does not discuss the motivation of the specific real to complex conversion. There are many invertible conversions between real and complex.

[1] DEEP COMPLEX NETWORKS

questions:
1. Line 317: Citation link broken for T2T-ViTYuan et al. (2021)

2. Line 269: Do you use complex-valued  “ normalization” as discussed in [1]

3. How well does the model perform if we consider trivial real to complex conversion that considers the real numbers as complex numbers with $0$ imaginary part? This is also a very crucial ablation that the authors should perform.

4. Does using spectral convolution make it challenging to capture local features as it performs global convolution?


[1] DEEP COMPLEX NETWORKS
</content>
</thread_object>

</thread>
<thread>
creation date:
1730543839409

thread_type:
official_review

<thread_object>
creator: reviewer_buvx

<content>
summary:
This paper presents a transformation to map RGB images into complex domain and an associate network comprising a loss function to handle such complex inputs.

Overall, the contributions are significant and may be of interest to the community, but the paper organization could be improved and more focus should be placed on the transformation.

strengths:
1) The proposed transformation is novel and may be an important contribution for the community working in complex and hypercomplex domains.
2) Although not novel, using the fourier filter module is good to handle complex inputs.

weaknesses:
1) While the method sounds, the results are not impressive and surely they are not statistically significant. As of my experience, complex, quaternion, and in general hypercomplex models clearly outperform real-valued counterparts when they are able to catch some underlying physical process intrinsic into data. Maybe, the tasks chosen by the authors do not highlight the effectiveness of their method. Maybe, the authors could stress more the parameters saving of using a complex model with respect to a real-valued one, which can help reducing the computational load while obtaining comparable results.
2) The authors should have focused more on the transformation, which is a novel contribution, and better show its properties (see questions).
3) Some key references to related works are missing, the authors should at least give credit to them, or better try to compare their method with them. Some of them follow, but I encourage the authors to better explore previous literature on complex, quaternion and hypercomplex networks.

[1] C. Trabelsi, O. Bilaniuk, Dmitriy Serdyuk, Sandeep Subramanian, J. F. Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, C. Pal, "Deep Complex Networks", ICLR 2017.

[2] E. Grassucci, A. Zhang, D. Comminiello, "PHNNs: Lightweight Neural Networks via Parameterized Hypercomplex Convolutions", IEEE Transactions on Neural Networks and Learning Systems, (Volume: 35, Issue: 6, June 2024).


Minor comments:

The Saxon Genitive should be avoided in scientific writing, although I know that both ChatGPT and Grammarly insert it. I suggest the authors to remove all the Saxon genitives in the paper.

questions:
1) I am very curious about the invertibility of the proposed transform. Given the Algorithm 2 in Appendix A, would it be possible to have some experiments to prove its effectiveness? I think that this transformation is the real contribution of the paper, as it allows a direct mapping between greyscale and RGB images, which was lacking in complex and quaternion papers that often struggle to do so.
2) Which is the computational load in terms of FLOPs, runtime memory, and time of the proposed model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732530976701

thread_type:
official_comment

<thread_object>
creator: area_chair_nr2g

<content>
comment:
Dear reviewers,

A reminder that **November, 26** is the last day to interact with the authors, before the private discussion with the area chairs. At the very least, please acknowledge having read the rebuttal (if present). If the rebuttal was satisfying, please improve your score accordingly. Finally, if you have concerns that might be solved shortly, please exploit the remaining time.

Thanks,
The AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1733313447168

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary and Common Response

comment:
We sincerely thank the reviewers for their valuable feedback and constructive discussions. We are encouraged by the acknowledgment of the novelty and contributions of our work.

&nbsp;

## Paper Summary:

This paper addresses a long-standing limitation in applying complex-valued neural networks to computer vision tasks while solving binary image segmentation entirely within the complex domain. Our work builds on the foundational principles of the FCCN [2] paper but extends the field significantly by introducing:

- **iRGB Color Model:** A novel, invertible complex-valued color model, enabling complex-valued processing of RGB images. Our experiments show iRGB surpasses the iHSV color model (proposed by FCCN [2]), providing a robust foundation for future research.

- **DCSNet:** A novel token-based complex-valued architecture exploring the frequency domain using complex Fourier transform and a multi-scale encoder-decoder structure. This approach learns rich complex-valued representations and achieves SOTA performance (while using complex-valued neural networks) across multiple binary image segmentation tasks. 

- **Complex Encoding for Segmentation Targets:** A unique (1, i) encoding method for binary segmentation maps, inspired by the physical independence of foreground and background elements, aligning seamlessly with the mathematical properties of complex numbers.

Beyond achieving state-of-the-art results on real-valued and complex-valued datasets, our work broadens the application of complex-valued neural networks to more computer vision tasks. For instance, our invertible R2C transform further opens opportunities in image synthesis and generation tasks, demonstrating the transformative potential of complex-valued representations.


&nbsp;


## Review & Discussion Summary:

**Reviewer BUvX:**
BUvX acknowledged the potential of complex-valued networks but raised concerns about comparisons and additional analyses. We provided detailed comparisons with real-valued counterparts, computational efficiency results for DCSNet, and further insights into our R2C transform. While our responses appeared satisfactory, the reviewer was disappointed these updates were not included in the revised manuscript. We acknowledge this oversight and will include them in the camera-ready version.

**Reviewer F89z:**
F89z recognized the novelty of our R2C transform but sought clarification on its advantages over existing real-to-complex transforms, including simpler alternatives. We provided detailed explanations and quantitative comparisons to highlight its benefits, addressing these points thoroughly. F89z was further interested in the advantages offered over the trivial [R+Gi,G+Bi] transform, and we explained them too. Upon F89z’s suggestion, we revisited one of the advantages and revised it too. 

Additionally, F89z requested insights into the improvements our DCSNet offers over DCN [1]. We clarified that our approach remains fully within the complex domain, unlike DCN [1], which does not solve computer vision tasks entirely in the complex domain. F89z also raised questions about local feature learning and our (1,i) encoding, which we addressed in detail. 

While we have responded to all concerns of F89z, we have not yet received further communication regarding re-evaluation of the score.

**Reviewer QqAh:**
QqAh sought clarity on the differences between our work and FCCN [2] and requested further details on R2C and architectural choices. We addressed all queries, providing detailed explanations and additional experimental results. The reviewer was satisfied but suggested exploring wavelet filters in future work.

**Reviewer B2bV:**
B2bV was highly positive, awarding a score of 8, and requested clarification on result interpretation. We highlighted how our method surpassed 90% performance and achieved a 2.86% improvement over prior complex-valued approaches.


&nbsp;

`FINAL REMARK:` We believe this paper is a significant step toward realizing the full potential of complex-valued neural networks in computer vision. We hope the advancements made, their implications and the review discussions will be carefully considered while making the final decision.

&nbsp;

Warm Regards,

Authors of #1276
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper considers binary segmentation with a complex-valued neural network. It introduces several novel components, including (a) a real-to-complex transformation of the RGB values, (b) a Fourier module, and (c) a segmentation loss in the complex domain. Of the 3 reviews with confidence > 2, the paper had a negative review round with all 3 reviewers recommending rejection. After the rebuttal phase, 1 reviewer is still highly critical of the work, while the other 2 have retained a marginal accept score. 

In general, there are several issues that were not fully clarified in the rebuttal phase. In particular, (a) the novelty of the method is not clear, especially since the authors have decided to focus on a narrow computer vision use case; (c) the results when comparing to a real-valued scenario are not convincing; (c) many experiments were not included in the main paper, and (d) some reviewers claim that a more complete mathematical analysis of the conversion is missing. 

On my side, I believe the paper is interesting for the community working on complex-valued models, but the narrow scope and unconvincing rebuttal makes me lean towards a rejection. I believe the paper can be significantly strengthened by the inclusion of the comments that were discussed in the rebuttal phase, but this would require a significant amount of work that could also change the structure or claims of the paper.