MetaInv: Overcoming Iterative and Direct Method Limitations for Inverse Learning

Authors: Jingyi Yuan, Muhao Guo, Yang Weng
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Invertible neural networks, switchable Architectures, analytical inverse

Abstract:
Invertible neural networks (INNs) have gained significant traction in tasks requiring reliable bidirectional inferences, such as data encryption, scientific computing, and real-time control. However, iterative methods like i-ResNet face notable limitations, including instability on non-contractive mappings and failure in scenarios requiring strict one-to-one mappings. In contrast, analytical approaches like DipDNN guarantee invertibility but at the expense of performance, particularly in tasks demanding rich feature extraction (e.g., convolutional operations in complex image processing). This work presents a detailed analysis of the limitations in current invertible architectures, examining the trade-offs between iterative and analytical approaches. We identify key failure modes, particularly when handling information redundancy or strict bijections, and propose a meta-inverse framework that dynamically combines the advantages of both i-ResNet and DipDNN. Our framework adapts in real-time based on task-specific signals, ensuring both flexibility and guaranteed invertibility. Extensive experiments across diverse domains demonstrate that our hybrid approach outperforms existing methods in forward accuracy, inverse consistency, and computational efficiency. Our results highlight the utility of this meta-inverse strategy for critical applications where precision, stability, and adaptability are crucial.

Decision: reject
Reviewer scores: 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1730414603316

thread_type:
official_review

<thread_object>
creator: reviewer_ajkp

<content>
summary:
This paper analysed the limitations of two specific invertible models, i-ResNet and DipDNN, and proposed a selection mechanism to choose one of the two models for a task. The basis of selection is essentially selecting the best performing model, preferring one model if neither is good.

strengths:
- The authors did a good review of two prior models and proposed an interesting comparison of the two models.
- The writing is easy to follow.

weaknesses:
- The work lacks strong motivation. Why two specific models were chosen to be analysed? Why do we need a single meta-model to work on tasks of a completely different nature? To me, it is quite obvious that we need different models for image data and physics data. The authors should provide a convincing argument to justify their work.
- It is not clear why only i-ResNet and DipDNN are chosen to be analysed. It would be better to aim for generalisable analyses that provide insight into a class of solutions.
- The main proposed algorithm (MetaInv) is not clearly described in the main text. More importantly, there is insufficient justification for the preference rules.

questions:
- "MetaInv" is in the title, but the algorithm is not in the main text. Is there a reason to put it in the appendix?
- In 3.1, Inv error, should x_i be y_i instead?
- Figure 8 is difficult to understand. Maybe provide a better description in the caption?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661227932

thread_type:
official_review

<thread_object>
creator: reviewer_yngd

<content>
summary:
The authors propose a method, termed MetaInv, to dynamically choose between two approaches for inverse modeling depending upon which one is most appropriate for the problem.

strengths:
1) Inverse problems are important, and therefore the general topic is significant

weaknesses:
1) The presentation is often poor.  The introduction is often vague/unclear regarding existing methods and their weaknesses.  There are major mathematical typos:  e.g., the equation in Line 147 does not seem to be valid.  Although I can guess, the authors never describe what l_{2} is, and what exactly are we summing over in the equations from lines 139-155.  

2) The review of the literature on inverse modeling misses large portions of the literature. Some examples include (but may not be limited to) genetic algorithms, conditional Invertible neural networks, mixture density networks, and the so-called Tandem model.   The authors ought to mention some of these approaches and explain why they were not applicable for this study, or at least why they were not included in experiments.   The authors can find examples of these existing methods/publications in Ardizzone et al. "Analyzing inverse problems with invertible neural networks." arXiv preprint arXiv:1808.04730 (2018),    or publications that cite this one. 

3) The authors show that their method is beneficial for just two methods in the literature: DipDNN and iResNet.  This becomes interesting if the result of combining these two approaches with MetaInv yields a method that achieves state-of-the-art capabilities (e.g., inverse accuracy, or computational efficiency).  However, the authors only show that their MetaInv approach leads to better/comparable performance to DipDNN and iResNet.  Therefore it is unclear whether the result of all this work improves state-of-the-art in any way or just improves these two methods.  Although improvement over these two particular methods is still interesting as a proof-of-concept for the MetaInv idea, in my opinion it is not sufficiently significant/interesting for this venue.

questions:
I welcome a response to comment (2) in the "weaknesses" section, although I request that the authors only respond if their response is clearly written, and addresses in a clear/compelling way (i) why so much existing work on inverse modeling was omitted from consideration, and (ii) why this work is still significant despite the exclusion of so much existin work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730895324134

thread_type:
official_review

<thread_object>
creator: reviewer_jq3f

<content>
summary:
This paper first provides a detailed analysis of the limitations in current invertible architectures, examining the trade-offs between iterative and analytical approaches. It then proceeds with proposing a meta-inverse framework that dynamically combines the advantages of both i-ResNet and DipDNN, by dynamically switching between them based on ask-specific signals.

strengths:
A very detailed analysis of the limitations in current invertible architectures,

weaknesses:
A rather short and not very extensive presentation of the switching algorithm and its associated challenges.

questions:
What were the key challenges in developing the switching algorithm, and what were the main contributions here?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper studies invertible neural network models in the context of invertible prediction tasks with image, power system measurement and fluid dynamics data. The authors provide a detailed analysis of tradeoffs, advantages and limitations of two models: i-ResNet [1] and DipDNN [2] representing iteratively and analytically invertible models respectively. In particular, the authors identify three metrics to consider: forward prediction error, inverse reconstruction error and inverse consistency. Then, the authors propose MetaInv, a method for adaptively deciding on which of the two architectures to choose for a given task. The authors show empirically (1) the expressivity limitations of i-ResNet, (2) trade-offs in performance between different invertible methods, (3) MetaInv can correctly choose which invertible model to use.

Strengths:
- Interesting discussion on the tradeoffs between different invertible architectures
- Interesting insights into the limitations of i-ResNet
- MetaInv method that can decide between competing invertible models in a practical setting
- Experiments on diverse domains

Weaknesses:
- The choice of specific models (i-ResNet and DipDNN) seems arbitrary
- Proposed method is practical, but not particularly interesting from a scientific point of view, as it is deciding on which method to use based on a combination of performance metrics
- Reviewers highlighted that the presentation is poor
- The results do not show an improvement to state-of-the-art on an established benchmark

Decision recommendation:
While the paper provides interesting insights, the methodological novelty is limited and improvements need to be made to the presentation. I would recommend that the authors should improve the presentation and better explain the motivation for the work and the design decisions made in the paper. It would also help to include results on established benchmarks with baselines evaluated in prior work. The reviewers unanimously rejected the paper. I thus recommend a rejection.

[1] Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jorn-Henrik Jacobsen. ¨
Invertible residual networks.

[2] Jingyi Yuan, Yang Weng, and Erik Blasch. Dipdnn: Preserving inverse consistency and approximation
efficiency for invertible learning