Unsupervised Homography Estimation on Multimodal Image Pair via Alternating Optimization
Authors: Sanghyeob Song
Jaihyun Lew
Hyemi Jang
Sungroh Yoon
Venue: NeurIPS 2024 poster (2024)
Keywords: Homography Estimation
Unsupervised Learning
Cross Domain
Image Registration
Image Alignment
Multimodal
Alternating Optimization
Primary area: machine_vision
TL;DR: We introduce an unsupervised framework for estimating homography in multimodal images, improving performance by using a two-phase optimization and Barlow Twins loss.

Abstract: Estimating the homography between two images is crucial for mid- or high-level vision tasks, such as image stitching and fusion. However, using supervised learning methods is often challenging or costly due to the difficulty of collecting ground-truth data. In response, unsupervised learning approaches have emerged. Most early methods, though, assume that the given image pairs are from the same camera or have minor lighting differences. Consequently, while these methods perform effectively under such conditions, they generally fail when input image pairs come from different domains, referred to as multimodal image pairs.
To address these limitations, we propose AltO, an unsupervised learning framework for estimating homography in multimodal image pairs. Our method employs a two-phase alternating optimization framework, similar to Expectation-Maximization (EM), where one phase reduces the geometry gap and the other addresses the modality gap. To handle these gaps, we use Barlow Twins loss for the modality gap and propose an extended version, Geometry Barlow Twins, for the geometry gap. As a result, we demonstrate that our method, AltO, can be trained on multimodal datasets without any ground-truth data. It not only outperforms other unsupervised methods but is also compatible with various architectures of homography estimators.
The source code can be found at: https://github.com/songsang7/AltO

Unsupervised Homography Estimation on Multimodal Image Pair via Alternating Optimization
Sanghyeob Song Jaihyun Lew! Hyemi Jang"
Sungroh 1.2
'Interdisciplinary Program in Artificial Intelligence, Seoul National University Department of Electrical and Computer Engineering, Seoul National University Samsung FlearroMMechnies {songsang7, fudojhl, wkda19512, sryoon) @snu. ac ac.kr
Abstract
Estimating the homography between two images is crucial for mid- or high-level vision tasks, such as image stitching and fusion. However, using supervised learning methods often challenging or costly due to the difficulty of collecting ground-truth data. In response, unsupervised learning approaches have emerged. Most early methods, though, assume that the given image pairs are from same camera or have minor lighting differences. Consequently, while these methods perform effectively under such conditions, they generally fail when input image pairs come from different domains, referred as multimodal image pairs.
To address these limitations, we propose AltO, unsupervised learning framework for estimating homography in multimodal image pairs. Our method employs a two- phase alternating optimization framework, similar Expectation-Maxinizaa (EM), where one phase reduces the geometry gap and the other addresses the modality gap. To handle these gaps, we Barlow Twins loss for the modality gap and propose an extended version, Geometry Barlow Twins, for the geometry gap. As a result, we demonstrate that our method, AltO, can be trained on multimodal datasets without any ground-truth data. It not only outperforms other unsupervised methods but is also compatible with various architectures of homography estimators. The source code can be found at: https: //gt thub. com/ songsang7/AIcl
1 Introduction
Homography is defined as the relationship between two planes when a 3D view is projected onto two different 2D planes. Many mid- or high-level vision tasks, such as image stitching []]. multispectral image fusion (2), and 3D reconstruction
require low-level vision tasks such as image registration or alignment preprocessing steps. Image registration is the process of aligning the coordinate systems of given pair of images by estimating their geometric relationship. If the relationship between the image pair is a linear transformation, it is called homography.
Homography estimation for image alignment is known as reducing the geometric gap between a pair of images as depicted in Figure | It has been an active area of research since the pre-deep learning era, with prominent algorithms such as SIFT [5). SURF [6] and ORB 77]. The advent of deep learning ushered in the exploration of end-to-end approaches utilizing supervised learning, beginning with DHN 8 These studies demonstrated the effectiveness of deep learning homography estimation. However, supervised learning assumes the availability of ground-truth data for the relationship between image pairs or pre-aligned image pairs. The assumption often unrealistic in real-world scenarios. Therefore, the rise of unsupervised learning has become essential to overcome these practical challenges.
"Corresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The Sisters Joseph of
The Staw Joseph Bos
the Sisters Joseph of
The Sh Joseph of Bos
Geometry gap
Geometry
Boston Golden
Buston
er Basten ge Golden
Baston Golden
Cokber
Modality gap
Figure I: Examples of types of gaps. This paper will address both geometry and modality gaps simultaneously. Image pairs are introduced by DLKFM T
Driven by this necessity, various unsupervised learning methods for homography estimation have emerged. UDHN [9] and biHomE [10] estimate homography using only unaligned image pairs. These methods achieve comparable performance to existing supervised learning methods when applied to image pairs with the same modality or minor lighting differences. However, if there is large modality gap. supervised learning methods still succeed in estimating homography due to the availability of ground-truth data, whereas unsupervised learning methods struggle to function effectively. Here, 'modality' refers data domain, and 'modality gap' denotes differences in types of representation as shown in Figure [ To address this modality issue, a simple approach that adding encoder for modality mapping can be considered. Yet, this approach leads to the trivial solution problem, which will be detailed in Section
As solution to all these problems, we propose framework, named AltO, that handles each gap separately, without relying on ground-truth labels and avoiding the trivial solution problem. The proposed method leverages metric learning and utilizes two loss functions. The first is for the Geometry loss, which reduces the geometry gap by making images similar at the feature level. The second is the Modality loss, which guides the mapping of image pairs with different modalities into the same feature space. By repeatedly switching between the training of these two losses, our method achieves outstanding performance than other unsupervised learning-bar based methods multimodal image pairs. Furthermore, our framework is independent the registration network, so it can benefit directly from combining with a gigh-performacc registration network.
In summary, the key contributions of this paper are:
We propose AltO, an unsupervised learning framework for homography estimation on multimodal image pairs.
We train the registration network by separately addressing the geometry and modality gaps through alternating optimization. this process, we also introduce a new extended form of the loss function derived from the Barlow Twins loss [12
Our method has achieved superior performance both quantitatively and qualitatively com- pared to existing unsupervised learning-bar based approaches in multimodal conditions. More- over, it has achieved performance nearly comparable to that of supervised learning.
2 Related Work
2.1 Hand-Crafted Homography Estimation
Homography estimation has advanced significantly with the development of the feature-based match- ing pipeline, especially after the introduction of SIFT (5). This pipeline typically consists of feature detection, feature description, feature matching, and homography estimation. Various methods, such as SURF [6] and ORB (7), have been developed as variants for specific steps within this process. However, these methods do not account for the modality gap and are thus unsuitable for multi- modal environments, Recently, feature-basse matching methods that handle multimodality, such as RIFT [3 and POS-GIFT [14, have been introduced. Despite these advances, their overall perfor- mance remains insufficient compared to learning- based methods. One of the main reasons is that most of them are based on HOG [15), which assumes planar cases without considering perspective effects.

---

2.2 Supervised Learning Homography Estimation
The first method to employ an end-to-end approach in the deep learning era DHN 8 trains a VGG blose backbone network from given pairs of images to predict offsets between four corresponding point pairs. The predicted offsets are then converted into homography using the DLT [17] algorithm. Furthermore, the paper proposes method for synthesizing datasets for training, which been continually cited in subsequent research. Another supervised learning-band method is IHN [18. which, compared to DHN, uses correlation volume to iteratively predict offsets, gradually reducing error. In addition, this method also experiments with multimodal dataset and has demonstrated only minor error of alignment. This approach was also adopted in RHWF [19].
This method
which replaces some convolution layers with a transformer (0) blocks.
2.3 Unsupervised Learning Homography Estimation
Supervised learning requires ground-trude data, which is the homography between two images, for training. practice, collecting such datasets. including the ground-truth labels, nearly impossible. To overcome this limitation, unsupervised learning-basse methods have emerged. The first proposed unsupervised learning method is UDHN ( It predicts offsets between a pair of input images, like DHN R), and converts them into homography using the DLT [77. It then warps one of the input images with this homography and compares measure similarity. Another method, biHomE [10. addresses the challenge of lighting variations through unsupervised learning. It utilizes a ResNet-34 [21 encoder that has been pre-trained on ImageNet [22], benefiting from the variety of lighting conditions in the dataset. This feature makes biHomE less susceptible to differences in lighting. However, it struggles when the input distribution significantly diverges from that of ImageNet, such with multimodal image pairs. To address these cases, MU-Net [23] has been introduced and adopts CFOG [2] based loss function. Although this demonstrates that CFOG can be integrated into an unsupervised learning framework and handle modality gap, CFOG still has weaknesses in handling geometric distortions such as rotation and scale.
the pixel level with the other image to
3 Background
3.1 Homography
Homography is a 3x3 transformation matrix that maps plane to another. It has form of projection transformation and maps an arbitrary point p in the image to as follows:
xx'
Thi
h12
h13
xx
p' ~ Hp W
1/
=
h21
hz2
h23
y/
(I)
hai
hg32
hgg.
[
In Equation (T) represents the homography matrix and the symbol denotes homogeneous coordinates, and w indicates the scale factor that allows the use of equality in the equation. In the matrix H, h13 and hız are related to translation, while h11, h12, h22 and h22 are associated with rotation, scaling, and shearing. Additionally, hai and h32 are related to perspective effects and are set to when perspective effects are not considered. Generally, when all effects are considered, it has degrees of freedom (DOF), while in special cases where perspective effects are not considered, it has DOF or fewer.
3.2 Trivial Solution Problem
As mentioned earlier, our goal is to address both the geometry and modality gaps together using unsupervised learning. One straighttorwwadd approach is to map each image into shared space via an encoder and perform registration, similar to UDHN [9]. using reconstruction loss. Another approach, as in biHomE [10], places the encoder after the registration network, where registration is performed first, followed by loss calculation through the encoder. However, when all modules are trained simultaneously. both approaches collapse into trivial solution: the encoder outputs constant value, and the registration network predicts the identity matrix as the homography. It appears as if the encoder and registration network collaborate to minimize the loss trivial way. To avoid this, special techniques are needed to prevent collapse into trivial solutions.

---

3.3 Metric Learning
Metric learning refers to method that trains models to increase the similarity between data points. Specifically, it is well-known within the field of self-supervised learning, which includes methods
such as NPID 255, SimCLR [26), Simsiam [27], Barlow Twins [22] and VIC-Reg [8]. These methods commonly involve multiple input data and Siamese networks, and they focus on how to compare each output so that the model can learn good representations.
Barlow Twins, which we adopt, requires further explanation for the subsequent sections. Fig- ure iillustrates the conceptual diagram of the Barlow Twins method. The encoder extracts the representations rd and r from inputs x and x²" then produces embedding vectors A and v by passing these r and r through the projector. At this point, the dimension of both embedding vectors becomes (N, Da),, where N the batch size and D, is the feature dimension. By calculating the similarity matrix between them, we obtain a matrix of size (D., D.). In this process, the similarity is calculated using the correlation coefficient Cij in Equation ( for the i-th element of C' from A and the j-th element of C from _ The bar over each vector ( and eB) denotes that the vectors have been mean-centered. The objective of the loss function L is to make the computed similarity matrix close to the identity matrix as possible. This involves balancing factor between the diagonal and off-diagonal elements. Especially, the term related with off-diagonal is called as 'Redundancy Reduction'. The formula is expressed as follows:
D
Enc.
Prj.
(D p D
I
8 (N. )
Sim. Matrix
x
Enc.
Prj.
Figure 2: Conceptual Diagram of the Barlow
Twins Method [12
En (0 ((nj))
L= - + Σ
Σ Ci.y where Cij jæ
(2)
V (1 VV (0
Method
4.1 Alternating Optimization Framework
We present a visualized overview of our training framework at Fig. 3 registration network R that takes moving image I and fixed image 1 as input, each from modalities A and B. It predicts homography matrix AAB that could warp (w) image A to 7 wII AAB) which aligns with _
Our objective to train a
Our training framework goes through two phases of optimization per mini-batch of data given, which 'Geometry Learning' (GL) phase and the GrodalityAAgoosiie Representation Learning' (MARL) phase. This framework optimizes alternatively, similarly to the Expectation -ntitattion-Maxiizzaatio (EM) 29 algorithm and it can be expressed follows:
GL Phase 0g + arrmain (eennneeeteeaG//---------------- dr-1)]
(3)
MARL Phase T7,, di + argmin (ModantGGGllllt( 1/-1, d1-1)]
(4)
In expressions 3 and represents the time step, and 17. and denote the parameters of the registration network, encoder, and projector, respectively. The GL phase aims to maximize the similarity of local features of 7- and 1 enabling the registration network to learn how to align the two images. The MARL phase is designed to learn representation space that is
so that the corresponding features of T and B can optimally match. The two phases of optimization will be further detailed in the following sections.
4.2 Geometry Learning
In this phase optimization, we fix v and di the weights of encoder E and projector P, and then train the registration network R. Our ultimate goal is to train the network R that predicts homography

---

Modality
BOR
Modality
Trainable
RS
Geometry loss
Modality loss
Frozen
R Registration Net.
AA48
Encoder
Beatgh
E
P+
Projector
f
Warp
f
2
fd
z
14
En
P
14
Eg
P
R4
Geometry loss
Modality loss
Rg
Geometry loss
Modality
loss
A
A
En
P
Eq
P
fd
Figure 3: Overview of architecture. Upper diagram shows static view and lower diagrams illustrate phase switching between Geometry Learning (GI.) phase Modality-Annostic Representation Learning (MARL) phase.
matrix AB which could warp a moving image A to align with a fixed image _ Explicitly following this objective, we warp (w) image I to acquire I = w(I AA) and maximize the geometric similarity between the warped image i and the fixed image 1 For the Geometry loss to use in training, we propose Geometry Barlow Twins (GBT) loss, a modified version of Barlow Twins [12 objective for 2-dimensional space. The features of two images A and 1 are extracted with shared encoder E, and the output feature maps are used to compute the GBT loss. The GBT loss Lg is different from the original Barlow Twins in that we consider the spatial axis of the features as the batch dimension of the original Barlow Twins formula:
bg En E( ((n,i)) 1+t ΣΣ C Pm.3))
En P
(5)
where C(njü)) =
V En, (ninirr V (S ((.
where h. and f (... w.). feature vectors in RXDDXXXW corresponding to 7A and B respectively, mean-normalizer along the spatial dimensions, by subtracting the spatial mean from each unit. n and h, w each denote the batch and the spatial (horizontal and vertical) index and i, denote the index of channel dimension. Our objective can be considered applying redundancy reduction on the spatial dimension of an image pair, maximizing the similarities of the local features corresponding regions. By minimizing the geometric distance, or in other words, maximizing the geometrical similarity, the network R learns to geometrically align the given image pair. Visualization of our Geometry Learning (GI.) phase can be found at the bottom-left Figure 3.
Modality-Agnostic Representation Learning
Images of different modalities are susceptible to form different distribution in the deep feature space. Maximizing the similarity of images from different modalities such feature space may not lead to our desired result. Hence, for the model to successfully perform homography estimation given two images from different modalities, should extract geometrie information independent of modality. To help the GL phase successfully work, the goal is to train encoder which could
5

---

On Main Dalivery
Jnn Main oUt Doleeeerry
Scho
(a) Google Map
(b) Google Earth
(c) Deep NIR
Figure 4: Examples of image pair for each datasets. Google Map and Google Earth are introduced by Deep NIR is proposed in [31
DLKFM I1.
form modality- representational space of image features, so that our Geometry loss could properly work intended. This process is referred to Modality-Agnostie Representation Learning (MARL). During this phase, the registration network is fixed, and the encoder and a projector P are trained. The standard form of the Barlow Twins loss [ is adopted as the training loss, defined as follows:
En =B (n( (nii) (n.st)
Lm C1- + ΣΣΣ 37
Oi where Cij =
(6)
V En (( )2 En (2 (z(r))))
where (n.i) E NDD and B (n.) E NXD are the projected vectors of f.A. (n.i.h,))) and fB (o) respectively, mean-normalized along the batch dimension, by subtracting each unit with the batch- mean value. n and j denote the batch and the index of channel dimension. The Modality loss aims to constrain 4 s(,,s) and x(n,)) to share similar representational space, agnostic to the image's ' modality. To capture global feature information, we employ global average pooling (GAP) [21, 30).
which removes the spatial dimensions produce a feature vector. The necessity of GAP discussed
in Section 6.2
Inspired by biHomE [T0], the architecture uses the initial layers of ResNet-34 21 the encoder &
and the latter layers as the projector P. ResNet-34 comprises a stem layer, four stages, and a fully connected (FC) layer. Each stage halves the spatial resolution and contains
residual blocks. To retain finer structural details, we remove the max pooling layer after the first convolution in the stem layer. The stem layer and stages I and 2 serve the encoder, while stage 3, along with GAP, functions as the projector. Section 6.4 confirms that this structure is the optimal division within ResNet-34. Stage is optional, and the subsequent FC layer, excluding GAP, omitted. As shown in Figure encoders for I and I share weights, as do the projectors.
Experiments
5.1 Multimodal Datasets
Google Map is multimodal dataset proposed in DLKFM [11]. It consists of pairs of satellite images and corresponding maps, which have different representation styles. There are approximately 9k training pairs and 1 test pairs of size 128x x 128.
Google Earth another DLKFM dataset that provides multimodality by consisting of images of the same area taken in different seasons. The amount of data is about 9k for training and 1k for test. The input image size is also 128x 128.
Deep NIR is dataset proposed in 31 It is an extension of the RGB-NIR scene dataset proposed by M. Brown al. [32 through cropping and maage-to-image translation. The RGB-NIR scene dataset consists of pairs of images, one captured by an RGB camera and the other by an NIR sensor at the same location. In this experiment, we use the oversampled 100 Deep NIR dataset. The dataset consists of approximately 14k training pairs and 2k test pairs. In this experiment, we resize these images to 192x 192 and then apply dynamic deformation as proposed by DHN (8. Finally, we obtain image pairs with size of 128 128.

---

5.2 Implementation Details
Registration networks are selected aim at demonstrating that our method can universally operate with various types of registration networks. To cover unique modules such as CNN, RNN, and Trans-
former [20], we chose DHN 8 (plain CNN), RAFT [3] (CNN+RNN), IHN [T8] (CNN+iteratiie
process), and RHWF [9] (Transformer+iteratfe
process). In the case of DHN, following bi-
HomE [10], the backhone network was switched from VGG [T6] to ResNet-34 (2T] Additionally,
IHN and RHWF employed a 1-scale network, which we refer to as IHN-- and RHWF-1, respectively. In iterative process- based networks like RAFT, IHN, and RHWF, our AltO framework was applied at every time step, replacing the use of ground-truth -ruth labels. Further details are provided in the appendix.
Experimental settings of ours include using the PyTorch 1.13 library and an Nvidia RTX 8000 GPU with 48GB of VRAM for training each model. The models were optimized using the Adam W optimizer 34 a one-cycle learning rate schedule [35], a maximum learning rate of 3e-4, and weight decay of le-5. Additionally, gradient clipping of #1.0 is applied during the backward pass for the Geometry loss. The training protocol repeats for a total of 200 epochs with batch size set to 16. Regarding the loss parameters, À is set to 0.005, consistent with standard settings of Barlow Twins 12.
5.3 Evaluation Metric
The evaluation metric used is Mean Average Corner Error (MACE), Corner error is the Euclidean distance between the positions warped by the correct homography Н and the predicted homogra- phy AB for corner of Л ACE is the average corner error across four corners, and MACE the overall mean of ACE across the dataset.
MACE(H AB (AA
[W[[, (al(,,HAA) - w(c, f A |al
E DEN
LEE
In Equation 77) C the set of corner points of I and N is the set of samples. In this evaluation, lower MACE indicates less error and better performance.
Table 1: Experimental results three benchmarks, Google Map [1I, Google Earth M and Deep
NIR [31. We report the Mean Average Corner Error (MACE) for evaluation. Our approach shows state-of-the-art performance among unsupervised methods, with robust compatibility across diverse
network architectures for homography estimation.
Learning Type Method
Google Map [II]
Google Earth [1]
Deep NIR [1
(No warping)
23.98
23.76
24.75
DHN 8
4.00
7.08
6.91
RAFT 33
2.24
1.9
3.34
Supervised
I 18
0.92
1.60
2.11
RHWF-
19
0.73
1.40
2.06
UDHN 9
28.58
18.71
24.97
CAU B6
24.00
23.77
24.9
biHomB 10
24.08
23.55
26.37
Unsupervised
DHN K
AltO (ours)
6.19
6,52
12.35
RAFT 33 + AltO (ours)
3.10
3.24
3.60
I 18 + AltO (ours)
3.06
1.82
3.11
RHWF-1 19
AltO (ours)
3.49
1.84
3.22
5.4 Evaluation Results
The experimental results are shown in Table When image pairs are unaligned, or if the identity matrix is used as the homography, MACE typically exceeds 23 px. Thus, any method yielding MACE above this threshold can be considered unsuccessful in training. Most conventional unsupervised methods fail train effectively on the benchmark datasets. UDHN ( which directly compares B and A in pixel space, is highly sensitive to modality changes, resulting failed training on most datasets except Google Earth [Τ]. Although the Google Earth dataset is multimodal, its image pairs have similar intensity distributions, allowing only limited training pixel space. Similarly, CAU (6) which addresses only minor lighting variations, suffers from modality gaps and large displacements,

---

Google Map
Google Earth
Deep NIR
MoneySra
oe Exprest
aul Chiropra
ManeyGramm
IHN-1
Express
(Supervised)
Neul Chirpprecte sic
MaaneyGramy
Service esp
(Supervised)
Nau Chiro Ocsic
Expres
Mone Aeramm
UDHN
Chitoprat
Monneyy ARL
BiHomF
IHN-1
ManeyGram
+AliO
(Ours)
Nauh Chiropractic
RHWF-1
ManeyyGamm
Servec Express
+AltO
(Ours)
NEuU Chirop
SIC
Figure 5: Visualization of homography estimation using center box. The first row shows the state before applying homography (green rectangles). Subsequent rows compare the results after applying ground-truth (green) and predicted (red) homography matrices. Our method, AltO, closely matches supervised learning-basee based methods, while other unsupervised approaches underperform.
leading to consistently low performance across all datasets, including Google Earth. biHomE [[0], structurally similar to our method, uses ImageNet [22] pre-trained ResNet-34 21 as the encoder, which struggles with images outside the ImageNet distribution.
In contrast, our proposed method achieves significantly higher performance than existing unsupervised approaches. Additionally, it is compatible with any registration network architecture, R, showing only marginal performance gap compared to supervised training. Although this gap may appear quantitatively large, it would be negligible practical applications. Figure(]jllluse results, showing boxes warped using ground-truth and estimated homography matrices. While conventional unsupervised methods fail align boxes, our method achieves high alignment accuracy, with only minor differences compared to supervised methods using the same R. These results indicate that our method, AltO, can replace supervision from ground-truth homography matrices.
the qualitative
In the appendix, there are additional qualitative results with another way of demonstration. There are comparisons of A B and A for each method.

---

Ablation Study
6.1 The Effectiveness of Alternating
In our AltO, the key technique is alternating, designed to address the trivial solution problem from unintended collab- oration, discussed in Section 3.2 This ablation study demonstrates its ef- fectiveness by comparing cases with and without alternating. The experiment was conducted on the Google Map dataset [ measuring the MACE values described in Section 5.3 Table 2 presents the experiment conducted with and without Modality-Agnon Representation Learning (MARL) when alternating is absent. As expected, the result shows that effective learning occurs only with alternating.
Table MACE for methods with and without alternating and MARL
No Alternating
Alternating with MARL
Method
with MARL
w/o MARL
DHN g + AltO
24.09
24.27
6.19
RAFT 33 AltO
26.21
25.91
3.10
IHN-1 18 Alto
24.37
23.14
3.06
RHWF-1 [9] + AlIo
18.88
24.07
3.49
6.2 The Necessity of Global Average Pooling
In Section 4.3 stated that global average pooling (GAP) 30 is applied at the end of the projector P. This ablation study compares per- formance with and without GAP. The exper- iment was conducted using the Google Map dataset (T]] with the result presented in Ta- 3le As shown by the result, effective learn- ing occurs only with GAP. This is because, without GAP, local features are forced be similar by comparing fine details before the registration network is fully trained, which leads to the trivial solution problem discussed in Section 3.2
Table 3: MACE for methods with and without global average pooling (GAP) [3].
Method
w/o GAP
with GAP
DHN 8
AltO
24.07
6.19
RAFT 33 + Alto
24.07
3.10
IHN-1 18 + AA4OO
24.01
3.06
RHWF-1 19] Alto
24.08
3.49
6.3 Combination of Loss Functions
Our proposed AltO requires both Geometry and Modality losses. While suggest using Bar- low Twins [2] for both, this section examines the performance impact of alternative losses. For the experiments, the registration network R is set [8]. and the dataset used is Google Map [T]. We compare with Info NCE [37] and VIC-Reg 28 widely used contrastive losses in Siamese self-supervised learning, and additionally test Mean Squared Error (MSE) for the Geometry loss. Modality loss is used in its original form, while Geometry loss is expanded to include spa- tial dimensions, as described in Section 4.2 In this case, Info NCE introduced as Patch NCE
Table 4: Ablation study Geometry and Modality losses, exploring all combinations of three popular contrastive losses and MSE for Geometry loss.
IHN-1
Geometry loss
Modality loss
MACE
Barlow Twins
3.06
Barlow Twins 12
Info NCE 337
3.73
VIC-Reg 28
3.36
Barlow Twins
3.56
Patch NCE 38
Info NCE
4.71
VIC-Reg
3.4
Barlow Twins
21.47
VIC-Reg
Info NCE
4.95
CUT 38
VIC-Reg
23.99
Barlow Twins
3.74
Table presents the MACE results for differrnt combinations of loss functions. It is known from SimCLR (26] that using Info NCE or Patch NCE benefits from a larger number of contrasting in- stance pairs. However, in our implementation, the batch size affecting the Modality loss is limited to 16, and the spatial resolution impacting the Geometry loss is only 32 by 32 (1024 pairs), which relatively small. This likely leads to a decline in performance. Additionally, VIC-Reg exhibited instability due to the three separate balance factors required for its components: variance, invariance, and covariance. MSE is another choice that could be used as a distance metric to compute the geometrical gap. This option of loss leads to fair performance, but short in comparison to
MSE
Info NCE
3,97
VIC-Reg 3.18

---

Barlow brins-based objective. We hypothesize the reason for this observation as below: even if image pairs are at corresponding locations, the information at those locations may not have identical feature representations. For example, while one image may include detailed edge descriptions, the other might lack such details due to differences in representation style. This difference in infor- mation quantity results in discrepancies in the embedded features. Essentially, while features at corresponding locations should be similar, they cannot be exactly the same for this reason.
In contrast, our proposed Barlow Twins-based losses do not suffer from the weaknesses present in other losses. First, unlike Info NCE and Patch NCE, they do not require a large number of contrasting pairs, as shown in the original Barlow Twins paper [2]. Additionally, they use only one balancing factor, making them significantly more stable than Vec-Reg. Lastly, unlike MSE, they do not require the embedded features to be exactly identical but instead aim to increase similarity, which is more applicable to real-world scenarios. These advantages allow our proposed combination to achieve superior performance.
6.4 Architecture of Encoder and Projector
As mentioned in Section 5.2, have adapted ResNet- 34 [2I] by dividing it into an encoder and a projector. In doing so, there were several cases regarding which stages to use as the encoder and which to designate as the projector. This ablation study aims measure the performance for each of these configurations. The Google Map dataset [1] is used, with IHN-1 [8] as the registra- tion network R. The indices used in Table represent the sequential order of ResNet stages.
Table 5: Encoder and projector config- urations within ResNet-34 21 Each number represents ResNet stage; all encoders include the stem layer, and all
projectors end with GAP 30
Encoder (Stage No.)
Projector (Stage No.)
MACE
2, 3
3.82
1
2, 3,4 4
3.78
The result indicates that the encoder performs best when at least two stages are utilized. While using three stages can enhance the encoder's mapping capabilities, the resolution halved, leading to decreased precision. Conversely, using only one stage results in better resolution compared to two stages, but the mapping capabilities are inadequate. The experiments reveal that the sweet spot between resolution and mapping ability trade-offs is achieved with two stages. For the projector, using stages beyond the third does not significantly affect performance, except when only the shallowest, the fourth stage, is used.
1,2
3
3.06
1,2
3, 4
3.07
1,2,3
4
9.76
Conclusion
We propose AltO, new learning framework, capable of training on multimodal image pairs through unsupervised learning. By alternating optimization, like Expectation- Expectaton-Maxxiization (29]. AltO handles both geometry and modality gaps separately. This framework employs two types of loss functions, using the Barlow Twins [12] and its extended version, to effectively address both gaps. Experimental results demonstrate that the registration network is stably trained within our framework across various backbones. Furthermore, it outperforms other unsupervised learning-based methods in MACE evaluations and achieves performance close to that of supervised learning-based methods across multiple datasets.
8 Limitation and Future Work
Despite its advantages, our method encounters some limitations. Firstly, applying the same registration network results in slightly reduced performance compared to supervised learning, which directly utilizes ground-truit truth labels. Secondly, our framework requires training additional modules to replace ground-truth labels, which slows down the training process.
To address these limitations, future work could explore new designs, such integrating Transform- ers [20] into the encoder and projector to improve performance. Additionally, reducing training time essential, motivating efforts to develop a single-phase phase framework that avoids the trivial solution problem mentioned in Section 3.2
10

---

9
Acknowledgements
This work was supported by Samsung Electro-Mechanics.
This work supported by Institute of Information & communications Technology Plan- ning & Evaluation (IITP) grant funded by the Korea MoVe)nmen((MSIT) INO.RS-2021- I1211343, Artificial Intelligence Graduate School Program (Seoul National University)]
This work was supported by the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Scoul National University in 2024.
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2022R O022RIA3BION A3B1R17330777220
References
[1] Matthew Brown and David G. Lowe. Automatic panoramic image stitching using invariant features. lnt. Comput. Vis., 74(1):59-73, 2007.
[2] Yuan Zhou, Anand Rangarajan, and Paul D. Gader. An integrated approach to registration and fusion of hyperspectral and multispectral images. IEEE Trans. Geosci. Remote. Sens., 58(5):302-33033 2020.
[3] Zhongfei Zhang and Allen R Hanson. 3d reconstruction based on homography mapping. Proc. ARPA96, pages 1007-1012, 1996.
[4] B. Zhang and Y.F. Li. An efficient method for dynamic calibration and 3d reconstruction using homographic transformation. Sensors and Actuators A: Physical, 119(2)349--377, 2005.
[5] David G. Lowe. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis., 60(2):91-110, 2004.
[6] Herbert Bay. Tinne Tuytelaars, and Luc Van Gool. SURF: speeded up robust features. In Ales Leonardis, Horst Bischof, and Axel Pinz, editors, Computer Vision ECCV 2006, 9th European Conference Computer Vision, Gruz, Austria, May 7-13, 2006, Proceedings, Part I, volume 3951 of Lecture Notes in Computer Science, pages 404-417 Springer, 2006.
[7] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary R. Bradski. ORB: an efficient alternative SIFT or SURF, In Dimitris N. Metaxas, Long Quan, Alberto Sanfeliu, and Luc Van Gool, editors, IEEE International Conference on Computer Vision, ICCV 2011, Barcelona, Spain, November 6-13, 2011, pages 2564-2571. IEEE Computer Society, 2011.
[8] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Deep image homography estimation. CoRR, abs/1660.03777 2016.
[9] Ty Nguyen, Steven W. Chen, Shreyas S. Shivakumar, Camillo Jose Taylor, and Vijay Kumar. Unsupervised deep homography: A fast and robust homography estimation model. IEEE Robotics Autom. Lett. 30)):2366-2555 2018.
[10] Daniel Koguciuk, Elahe Arani, and Bahram Zonooz. Perceptual loss for robust unsupervised homography estimation. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021, virtual, June 19-25, 2021, pages Vision Foundation IEEE, 2021.
Computer
[11] Yiming Zhao, Xinming Huang, and Ziming Zhang. Deep lucas-kanade homography for multimodal image alignment. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 15950-15959, Computer Vision Foundation / IEFE, 2021.
[12] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Leurning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 13310-13300 PMLR, 2021.

---

[13] Jiayuan Li, Qingwu Hu, and Mingyao Ai. Rift: Multi-modal image matching based on radiation- variation insensitive feature transform. IEEE Transactions on Image Processing. 29:3296-3311 2020.
[14] Zhuolu Hou, Yuxuan Liu, and Li Zhang. Pos-gite gift: A geometric and intensity-invarian invariant feature transformation for multimodal images. Information Fusion, 102: 102:102027, 2024.
[15] N. Dalal and B. Triggs, Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR ) volume I, pages 886-893 vol. 1, 2005.
[16] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors. 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[17] Andrew Harltey and Andrew Zisserman. Multiple view geomelry in computer vision (2. ed.).. Cambridge University Press, 2006.
[18] Yuan Cao, Jianxin Hu, Ze-Hua Sheng, and Hui-Liang Shen. Iterative deep homography estimation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 1869-1878. IEEE, 2022.
[19] Yuan Cao, Runmin Zhang, Lun Luo, Beinan Yu, Zehua Sheng, Junwei Li, and Hui-Liang Shen. Recurrent homography estimation using homography-guide image warping and focus transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages
IEEE, 2023.
[20] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017.
[21] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016.
[22] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[23] Yuanxin Ye, Tengfeng Tang, Bai Zhu, Chao Yang, Bo Li, and Siyuan Hao. A multiscale framework with unsupervised learning for remote sensing image registration. IEEE Transactions on Geoscience and Remote Sensing, 60:1-15, 2022.
[24] Yuanxin Ye, Lorenzo Bruzzone, Jie Shan, Francesca Bovolo, and Qing Zhu. Fast and robust matching for multimodal remote sensing image registration. IEEE Transactions on Geoscience and Remote Sensing, 57(1 (1111))9059999001 9070, 2019.
[25) Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-pan parametric instance discrimination. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 3733-3742. Computer Vision Foundation / IEEE Computer Society, 2018
[26) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 1020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 2020.
[27] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 15750-15785 Computer Vision Foundation/ / IEEE, 2021.
12

---

[28] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: pariance-invarianee -ovariiance-covriannnee regu- larization for self-supervised learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[29] Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin. Maximum likelihood from incomplete data via the em algorithm plus discussions on the paper. 1977.
[30] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. CoRR, abs/1312.4400 2013.
[31] Inkyu Sa, Jong Yoon Lim, Ho Seok Ahn, and Bruce A. MacDonald. deepnir: Datasets for generating synthetic NIR images and improved fruit detection system using deep learning techniques. Sensors, 22(13))4721, 2022.
[32] Matthew Brown and Sabine Süsstrunk. Multi-spectral sift for scene category recognition. In CVPR 201 pages 177-184, 2011.
[33] Zachary Teed and Jia Deng. RAFT: recurrent all all-pairs field transforms for optical flow. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michan MichMeli Frahm, editors, Computer Vision ECCV 2020 - 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part II, volume 12347 of Lecture Notes in Computer Science, pages 402-419. Springer, 2020.
[34] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[35] Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi molti-domain operations applications, volume 11006, pages 369-386. SPIE, 2019.
[36] Jirong Zhang, Chuan Wang. Shuaicheng Liu, Lanpeng Jia, Nianjin Ye, Jue Wang, Zhou, and Jian Sun. Content-aware unsupervised deep homography estimation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 16th European Conference. Glasgow, UK, August 23-28, 2020, Proceedings. Part I, volume 12346 of Lecture Notes in Computer Science, pages 653 669. Springer, 2020.
[37] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1 01111807777.... 2018.
[38] Taesung Park, Alexci A. Efros, Richard Zhang, and Yan Zhu. Contrastive learning for unpaired image-to-image translation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- Michael Frahm, editors, Computer Vision ECCV 2020- 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX, volume 12354 of Lecture Notes in Computer Science, pages 319 345. Springer, 2020.
13

---

A Appendix
A.1 Additional Visualization of the Results from the Main Experiment
We provide additional results of method and baseline methods for comparison as below.
Google Map
Google Earth
Deep NIR
Starbucks Takoouth Delwvry
Bloo
Me
I
ard Universs
nndor track
E Blmmaaash
Harva
Statee
vard University
Indoor Track
E Central
Ha
ard Uiiversity
IHN-1
Indoor Track
(Supervised)
E Centrals
Ha
ard Cinversity
RHWF-1
Inddor Track
(Supervised)
E Central
Ha
Broucks Bh Celwery
Blod
ard Universitty
UDHN
Inddor Track
Centralsh
Harva
States
Starbucks
Takeoou Delivary
Bloo
biHomE
ard University
Windoo trakk
Centralls
Harv
United Staten
IIN-1
ard University
Unddor Crack
+ARO
(Ours)
E Centrals
Ha
RHWF-1
ard Uiiversity
Indoor track
+ARt
(Ours)
E Centrals
Ha
Figure 6: Moving, fixed, and warped images. The first row displays the moving image (TA), the second row shows the fixed image (I"), and the remaining present the warped images (Ia) derived from A for each method.
14

---

A.2 Integration of Iterative Registration Networks into Alto
In this section, we provide detailed explanation of integrating registration network with an iterative process into AltO. In the main experiment, we used DHN [8], RAFT [33] IHN-1 [18], and RHWF-1 [9 the registration network R. Among these, RAFT. IHN-1, and RHWF-1 employ an iterative process, predicting intermediate homographies over multiple time steps for each homography estimation. When supervised learning applied, the loss function used for homographies output at each time step. Since AltO is also intended as replacement for supervision, we similarly apply the Geometry loss the homography output at each time step. For RAFT, originally designed for optical flow estimation, we implicitly incorporate DLT 17 to convert optical flow into homography. Figure illustrates this process.
Geometry Learning (GL)
Simplified Representation Phase
E
Pg
14
A
Geometry loss
R
Module (co. En)
Geometry Joss
Modality loss
R
RA4B
74
E
P
Details when R has an iterative framework (enses of RAFT, IHN, RHWF)
(L 1)
Geometry loss
Geometry (Final)
Ht 2)
Geometry loss (t 2)
RS
Module (w, Ex)
Y is borrowed from the paper
Geo. lnss(t
that proposed
(t T)
Geometry loss (t T)
registration network (R)
Figure 7: Visualization of the iterative registration process in AltO. For RAFT implicitly used to transform optical flow into homography across time steps.
33 DLT
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our contributions and claims are appropriately stated in the introduction and
abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Section g
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

We described the assumptions of our method's operation in Section and demonstrated them through experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We explained both the architecture and implementation details in Sections 4.3
and 2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide our codes at htttt:///ggthhit httppp///ggttuub..oo//oooggggge woom/songsangg/Al
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iips.cc/ppll wttp:ccppppcccceeeeeeeeee..c.ett httpa/////.ieeeoeetuuuuiiiissss Coodeooomisssssseeeeeeee www.siiiccciiiiiccoooooooiiscipi for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Experimental details are described in Section 5.2 Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the experimental setting of homography estimation research com- munity.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mentioned the hardware resources Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Question:
Answer: [Yes]
Justification: The research presented in the paper conforms every respect with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper is focused only on estimating the spatial relationship between two provided images. It neither creates new false information nor reveals hidden content, and does not offer any potential for misuse or adverse social impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper is focused only estimating the spatial relationship between two provided images. It neither creates new false information nor reveals hidden content, and does not offer any potential for misuse or adverse social impacts.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited all the sources we used or referred to, such as datasets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide our at htttt:://gtthhhie cotp:/oogoo/oggg///1.. At
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "W69deQd2wr", "reviewer": "Reviewer_PrYW", "summary": "The paper addresses unsupervised homography estimation from multi-modal image pairs. The authors propose to cope with the issue of 1) modality, 2) registration in two distinct networks that are trained in an interleaved fashion. The networks architecture derives from the Barlow Twins framework, with changes in the loss function. Results are illustrated on several public benchmark of small images (128^2) and compares favorably wrt to related unsupervised approach.", "questions": "-  What is the size of the embedding? What is the training time? Are the Barlow Twins trained from scratch?\n\n- Illustration seems to show  strong geometric features (ie lines) in the input images. Is it a strong limitation of the approach?", "limitations": "From a practical point of view, the size of the image and the strong overlap between the pairs show that the work need to be further developped for applications at full scale. \nFrom a methodology point of view, the authors have discussed the limitation of having two networks trained in an interleaved way, with potential collision or collapse.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1- I enjoy reading the paper. I walked through the paper, first with curiosity and skepticism, then with strong interest. The approach is intuitive (adjust the two representations then compute the transformation) and compelling. I am somehow surprised that it works :) The constrastive-like loss used in  Barlow Twins  is contributing much for the network to learn the correct solution. \n\n2- Overall, the authors are tackling an important problem (unsupervised learning) for which an original solution is proposed --while based on previous recent work. The methodology is clearly presented. Results are convincing (thought only on small images 128x128) and illustrated on various modality pairs. Quantitative results show improvement wrt related unsupervised work", "weaknesses": "1- Not a weakness, but a points which could have been discussed: why not simply transforming the inputs into edge maps before learning a matching/homography function (and putting aside the modality discrepancy). It would not be a very fancy approach, but I believe it could be a baseline for comparison. \n\n2- The approach would be more convincing if each of the two modules (GL and MARL) had demonstrated their effectiveness also individually (ie same image pair modality using only GL).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and for taking the time to provide your feedback. Below is our rebuttal.\n\n**Weakness 1.** *Why not use an edge-based approach as a baseline?*\n\nEdge-based approaches have limitations when used as baselines with different modalities. When dealing with two images of different modalities, one image often has rich edges while the other does not. Even within the closed modality, such as day and night photos, edge differences can arise due to shadows. Thus, converting both images to edge maps and comparing them might not achieve high performance. Although paper [1] utilizes edges in its learning process, it is limited to using image pairs of exactly the same modality.\n\nBelow Table R2 shows the results of applying UDHN [2], a simple unsupervised method that assumes the same modality, to edge maps converted from the Google Map dataset. The results indicate poor performance.\n\nTable R2: Evaluation result of applying edge maps to UDHN\n| Method | MACE (Google Map) |\n|--------|:----:|\n| IHN + AltO | **3.06** |\n| UDHN | 28.58 |\n| UDHN + Edge Map + L1 loss | 24.15 |\n| UDHN + Edge Map + cos loss | 24.00 |\n\n[1] Feng, Xiaomei & Jia et al., Edge-Aware Correlation Learning for Unsupervised Progressive Homography Estimation, in IEEE Transactions on Circuits and Systems for Video Technology, 2023.\n\n[2] Ty Nguyen et al., Unsupervised deep homography: A fast and robust homography estimation model. IEEE Robotics Autom. Lett., 2018.\n\n**Weakness 2.** *Demonstrate the effectiveness of each modules, GL and MARL.*\n\nThe GL module enhances geometry alignment by increasing the similarity between local correspondences. It is inspired by biHomE [3], with the only differences being that biHomE uses an ImageNet-pretrained encoder which is frozen, and applies triplet loss as the loss term. The effectiveness of this approach has already been demonstrated in the paper [3] using the S-COCO dataset, which has the same modality.\n\nThe role of the MARL module is to train the encoder to map input images to the same feature space. This properly trained encoder allows our geometry loss to function correctly regardless of modality differences. Table R1 in the Author Rebuttal (above) shows the results of training on the Google Map dataset without the MARL module or with the MARL module but no alternating. The result indicates that proper learning did not occur.\n\nIn summary, for successful training, the GL module, MARL module, and alternating training are all essential components.\n\n[3] Daniel Koguciuk et al., Perceptual loss for robust unsupervised homography estimation. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021.\n\n**Question 1.**\n\n**Question 1.1.** *What is the size of the embedding?*\n\nAltO serves as a learning framework that does not impose restrictions on the internal structure of each component. However, the encoder and projector used as examples in our paper are based on ResNet-34. The encoder uses the first two stages of ResNet-34, resulting in an embedding size of 128. Nevertheless, these can be flexibly replaced if needed.\n\n**Question 1.2.** *What is the training time?*\n\nTable R3 below shows the training time measurements on Google Map dataset. Although the training time increases, the inference time at run-time is the same as that of supervised learning since the AltO module is not needed.\n\nTable R3: Training times for each registration network. (Nvidia RTX 8000)\n|     | Supervised. | Unsupervised (with AltO) |\n|:-----:|:-----------:|:------------------------:|\n|DHN  | 1h 26m      | 4h 31m                   |\n|IHN  | 3h 27m      | 9h 57m                   |\n\n**Question 1.3.** *Are the Barlow Twins trained from scratch?*\n\nThere are no pretrained components. All layers are trained from scratch.\n\n**Question 2.** *Are strong geometric features (i.e., lines) necessary for the proposed method?*\n\nIt is evident that lines, such as edges, are helpful. However, as visualized in Figure R2 of the rebuttal PDF, the output of the encoder shows strong features in the form of blobs rather than edges. This indicates that not only lines but also blobs play a significant role in the proposed method.\n\n**Limitation.** *Small image size and strong overlapped pairs*\n\nThe dataset settings we used (such as size and overlap) are based on the pioneering paper DHN [4] on end-to-end homography estimation. These settings have become the standard and are used by many studies, including [1-7]. For fair comparison, we followed these settings. However, AltO, as a learning framework, has no size or overlap restrictions. Constraints, if any, come from the registration network (e.g., IHN), encoder, or projector.\n\nTable R4 shows experiments where the Google Map dataset size and displacement were doubled, called 'Google Map x2.' We used DHN as the registration network, which is flexible with size constraints. The results show that DHN with AltO can reduce MACE even without any hyperparameter tuning.\n\nTable R4: MACE evaluation results on Google Map x2.\n| Method | MACE (Google Map x2) |\n|--------|:----------------------:|\n| (No warping) | 49.33 |\n| DHN + AltO | **34.24** |\n\n[4] Daniel DeTone et al., Deep image homography estimation. CoRR, abs/1606.03798, 2016.\n\n[5] Yiming Zhao et al., Deep lucas-kanade homography for multimodal image alignment. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021.\n\n[6] Si-Yuan Cao et al., Iterative deep homography estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022.\n\n[7] Si-Yuan Cao et al., Recurrent homography estimation using homography-guided image warping and focus transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jsmFxFHfsr", "reviewer": "Reviewer_sFim", "summary": "This paper proposes a new unsupervised homography estimation approach for multimodal images. This method is designed as a two-phase optimization framework named AltO. The first phase named \"Geometry Learning\" trains a registration network to align the input multimodal images geometrically. The second phase named \"Modality-Agnostic Representation Learning\" trains an encoder and a projector to extract the image-level features invariant to modality changes. Experimental results demonstrate that AltO outperforms several existing unsupervised approaches on the multimodal registration datasets.", "questions": "Please provide more discussions and experimental results to address the above weaknesses.\n\nMoreover, is the 3D reconstruction task related to \"Homography Estimation\" (line 21)? Generally, 3D reconstruction focuses on non-planar scenes, while homography estimation is designed for the planar scenes. Is there some literature that mentions the relationship between 3D reconstruction and homography estimation?", "limitations": "The authors have discussed the limitations.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed framework is intuitive and interesting. This framework trains a registration network to align the input multimodal images geometrically, and trains another encoder to match the image-level features of the warped multimodal images. This framework has the potential to capture the pixel-level and image-level information in an unsupervised manner.\n2. The organization and presentation of this paper are good. I think I can understand the core idea of this paper.", "weaknesses": "**1. Some central claims of this paper lack experimental evidence.**\n\n1.1 The \"alternating\" optimization framework is a central design in this paper. However, why is \"alternating\" optimization necessary? Will optimizing the \"geometry loss\" and \"modality loss\" simultaneously hurt performance?\n\n1.2 The superiority of the proposed Geometry Barlow Twins (GBT) loss was not verified. The original Barlow Twins loss can be straightforwardly applied to the proposed model by considering both the spatial axis (indexed with \"h,w\") and batch axis (indexed with \"n\") as the batch dimension. This straightforward implementation should be compared with the proposed GBT loss.\n\n1.3 The proposed approaches should be compared with some recent unsupervised approaches. Here are some approaches with released codes.\n\n[1] Unsupervised global and local homography estimation with motion basis learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.\n\n[2] A Multiscale Framework with Unsupervised Learning for Remote Sensing Image Registration, IEEE Transactions on Geoscience and Remote Sensing, 2022.\n\n**2. This paper did not discuss the recent hand-crafted approaches for multimodal image registration.**\n\nMany recent hand-crafted methods have been published in the top journals, so this kind of approach should not be ignored. The experiment should also compare the proposed approaches with the recent hand-crafted approaches. Here are some hand-crafted approaches with released code.\n\n[3] Histogram of the orientation of the weighted phase descriptor for multi-modal remote sensing image matching. ISPRS Journal of Photogrammetry and Remote Sensing, 2023.\n\n[4] POS-GIFT: A geometric and intensity-invariant feature transformation for multimodal images. Information Fusion, 2024.\n\n**3. The discussion of the motivation is not sufficient.**\n\nThe Introduction section mentioned some typical unsupervised approaches designed for the images from the same modality (e.g., UDHN and biHomE). However, the unsupervised approaches [2,5] designed for multimodal image registration are not discussed. What is the motivation of the proposed method compared with this kind of approach? \n\n[5] A Novel Coarse-to-Fine Deep Learning Registration Framework for Multi-Modal Remote Sensing Images. IEEE Transactions on Geoscience and Remote Sensing, 2023.\n\n**4. This paper misses some references to hand-crafted and unsupervised approaches.**\n\nI have listed some of them in the above weaknesses. The authors should further survey more papers and carefully revise the \"Related Work\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work in detail. Below are our responses to your comments and concerns.\n\n**Weakness 1.**\n\n**Weakness 1.1.** *Why is alternating necessary?*\n\nTo prevent unintended collaborations and a collapse into trivial solutions, we introduced an alternating training strategy. Training the encoder and registration network together end-to-end without alternating can cause the encoder to output a constant value and the registration network to always output the identity matrix as the homography. Alternating training isolates these modules. This ensures the encoder maps input images to the same feature space, and the registration network estimates the true homography between the two images.\n\nTable R1 in Auther Rebuttal (above) shows the experimental results when alternating training is not applied, demonstrating that proper training does not occur.\n\n**Weakness 1.2.** *What is the superiority of the proposed Geometry Barlow Twins (GBT)?*\n\nGBT is robust in both cases: whether the i.i.d. assumption of dataset holds or not. The method you suggested, using NHW as a batch, matches GBT's performance only when the i.i.d. assumption is satisfied. In the Google Map dataset, where the i.i.d. assumption is met, both loss functions perform similarly (Table R2). However, in the case shown in Figure R3 of the rebuttal PDF, the i.i.d. assumption is not met, and GBT performs better.\n\nThe key point is whether the distribution over NHW dimensions is similar to that over HW dimensions. If the i.i.d. assumption is not met, the distribution over N distorts NHW, leading to a distribution dissimilar compared to HW dimensions.\n\nTable R2: Comparison of two types of Geo. loss on the Google Map dataset.\n| Method | NHW as batch | GBT |\n|-------:|:------------:|:---:|\n| DHN + AltO | **5.35** | 6.19 |\n| RAFT + AltO | 3.55 | **3.10** |\n| IHN + AltO | 3.16 | **3.06** |\n| RHWF + AltO | **3.37** | 3.49 |\n\n**Weakness 1.3.** & **Weakness 2.** *The proposed approaches should be compared with some recent unsupervised approaches and hand-crafted approaches.*\n\nSorry for the insufficient number of baselines. We have added more baselines to Table R3 below. Some of the entries in Table R3 cite SCPNet [1], a very recent paper accepted at ECCV 2024. Nonetheless, **our method demonstrates even superior performance compared to SCPNet.**\n\nAdditionally, the paper [2] you mentioned is designed to address the 6-DOF (degree of freedom) problem, so it cannot be applied to our dataset, which has 8-DOF. Furthermore, the code for that paper is not fully available. Moreover, we also attempted to run the code for paper [3], but encountered errors with the sample images, so we could not include it in our results.\n\nTable R3: MACE evaluation results on the Google Map dataset. * indicates values reported from SCPNet [1].\n| Type | Method | MACE |\n|:----:|:-------|:----:|\n| Hand-crafted | SIFT | 24.53* |\n| | ORB | 24.52* |\n| | DASC | 21.76* |\n| | RIFT | 16.55* |\n| | POS-GIFT | 20.90 |\n| Unsupervised | UDHN | 28.58 |\n| | CA-UDHN | 24.00 |\n| | biHomE | 24.08 |\n| | BasesHomo | 24.49* |\n| | UMF-CMGR | 24.60* |\n| | SCPNet | 4.35* |\n| | IHN + AltO (Ours) | **3.06** |\n\n[1] Runmin Zhang et al., SCPNet: Unsupervised Cross-modal Homography Estimation via Intra-modal Self-supervised Learning, (ECCV 2024 Accepted.)\n\n[2] Yuanxin Ye et al., A Multiscale Framework with Unsupervised Learning for Remote Sensing Image Registration, IEEE Transactions on Geoscience and Remote Sensing, 2022.\n\n[3] ZHANG et al., Histogram of the orientation of the weighted phase descriptor for multi-modal remote sensing image matching. ISPRS Journal of Photogrammetry and Remote Sensing, 2023.\n\n**Weakness 3.** *The discussion of the motivation is not sufficient.*\n\nThe problem we aim to solve is the unsupervised learning of 8-DOF homography estimation for multimodal image pairs. These conditions are very common in many fields, such as industry. For example, matching a real photo to a CAD image. At the time of our research, we found very few papers that directly addressed this specific problem. This scarcity of relevant work highlighted the difficulty and value of addressing this challenge, which motivated us to pursue it.\n\nThe papers you suggested, [2] and [4], do not directly apply: [2] focuses on a 6-DOF problem, and [4] employs a supervised learning approach.\n\n[4] A Novel Coarse-to-Fine Deep Learning Registration Framework for Multi-Modal Remote Sensing Images. IEEE Transactions on Geoscience and Remote Sensing, 2023.\n\n**Weakness 4.** *This paper misses some references to hand-crafted and unsupervised approaches.*\n\nWe apologize for missing them. We will include and refer them during the revision.\n\n**Question 1.** *Is there some literature that mentions the relationship between 3D reconstruction and homography estimation?*\n\nBelow are several documents regarding your question. We will revise the citation section related to 3D reconstruction in our paper.\n\n* Zhang, Zhongfei, and Allen R. Hanson. \"3D reconstruction based on homography mapping.\" Proc. ARPA96 (1996): 1007-1012.\n* Mei, Christopher, et al. \"Efficient homography-based tracking and 3-D reconstruction for single-viewpoint sensors.\" IEEE Transactions on Robotics 24.6 (2008): 1352-1364.\n* https://github.com/ziliHarvey/Homographies-for-Plane-Detection-and-3D-Reconstruction\n* Yong-In, Yoon, and Ohk Hyung-Soo. \"3D reconstruction using the planar homograpy.\" The Journal of Korean Institute of Communications and Information Sciences 31.4C (2006): 381-390.\n* Zhang, Beiwei, and Y. F. Li. \"An efficient method for dynamic calibration and 3D reconstruction using homographic transformation.\" Sensors and Actuators A: Physical 119.2 (2005): 349-357.\n* Dubrofsky, Elan. \"Homography estimation.\" Diplomová práce. Vancouver: Univerzita Britské Kolumbie 5 (2009).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hXC6dl8P6M", "reviewer": "Reviewer_EsVS", "summary": "The paper proposes an unsupervised homography estimation method for multimodal image pairs using an alternating optimization approach. The claimed key innovation is the introduction of the Geometry Barlow Twins loss function for the alternating optimization. The authors show that their approach works on 3 multimodal datasets and different homography estimation architecutres.", "questions": "Please refer to the Weaknesses.", "limitations": "The paper discusses the additional training cost arising from the inclusion of an additional module in the two-phase network, and explores potential solutions for addressing this issue in future research. However, the method’s generalization capabilities are not thoroughly explored, with experimental datasets limited to satellite images, maps, RGB, and NIR images. Future research could involve testing the method on a broader range of datasets to validate its generalization capabilities.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The alternating optimization framework together with Geometry Barlow Twins loss seem to be a fresh perspective on unsupervised multimodal homography estimation.", "weaknesses": "Weaknesses\n1. Discussion on the Feasibility and Rationality of the Proposed Method: First, for unsupervised training of networks based on iterative prediction, such as RAFT, to ensure stability during training, related methods [1-2] typically apply some form of direct supervision to the motion predicted by the network. This is different from the approach proposed in this paper, which only uses the Geometry Barlow Twins loss for brightness supervision. Second, how RAFT can be used for homography estimation should also be explained, because it is designed for optical flow estimation. Moreover, the paper does not explain how the proposed Geometry Barlow Twins loss supervises the intermediate stages of iterative prediction, whereas RAFT, IHN, and RHWF, along with methods leveraging their structures [1-2], generally provide details on their supervision mechanisms on the intermediate stages. This raises concerns about the feasibility of the proposed supervision method in this paper. Additionally, the effectiveness of the Modality-Agnostic Representation Learning (MARL) introduced in section 4.3 is questionable because it lacks spatial information in its supervision. As mentioned in section 3.2, the projector removes spatial information from the feature maps. The authors should provide a convincing and thorough explanation for these issues. \n\n2. Doubt about the Effectiveness of the Proposed Method: For example, the paper proposes the alternating optimization (AltO) method but does not provide sufficient experimental results to demonstrate its superiority over other strategies, such as directly cascading all the modules. Furthermore, the paper lacks a comparative demonstration of the features extracted with and without the MARL phase, making the advantages of introducing this phase less convincing.\n\n3. Insufficient Experimental Validation: The paper conducts experiments on only 3 cross-modal datasets, among which only the GoogleMap dataset exhibits significant modality differences. The GoogleEarth dataset mainly consists of images taken in different seasons [3]. Part of the DeepIR dataset is simulated multispectral data [4], which will significantly reduce the difficulty of homography estimation. It would be beneficial to conduct experiments on more challenging multimodal datasets, such as those involving VIS-SAR modalities.\n\n[1] Stone, A., Maurer, D., Ayvaci, A., Angelova, A., & Jonschkowski, R. (2021). Smurf: Self-teaching multi-frame unsupervised raft with full-image warping. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (pp. 3887-3896).\n[2] Liang, Y., Liu, J., Zhang, D., & Fu, Y. (2023). Mpi-flow: Learning realistic optical flow with multiplane images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 13857-13868).\n[3] Zhao, Y., Huang, X., & Zhang, Z. (2021). Deep lucas-kanade homography for multimodal image alignment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 15950-15959).\n[4] Sa, I., Lim, J. Y., Ahn, H. S., & MacDonald, B. (2022). deepNIR: Datasets for generating synthetic NIR images and improved fruit detection system using deep learning techniques. Sensors, 22(13), 4721.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work. We apologize for the lack of detailed explanations regarding the proposed method. Below are some additional clarifications:\n\n**Weakness 1.**\n\n**Weakness 1.1.** *No other direct supervisions*\n\nIncreasing the similarity of local features between two inputs is sufficient and equivalent to estimating homography. This is because homography estimation assumes a linear transformation between pairs of static scenes. Therefore, smoothness is automatically satisfied, and global motion is equivalent to local motion and homography.\n\n**Weakness 1.2.** *How is AltO applied to iterative frameworks such as RAFT, IHN, or RHWF?*\n\nWe used each prediction from the registration network to individually calculate the GBT loss. Then, all GBT losses at each time step were combined by weighted summation, similar to the supervised learning situation. The weights used are also borrowed from the supervised learning case. The only difference from the supervised case is that AltO is used instead of the ground truth. If we had not effectively utilized the full capability of the registration networks, we would not have been able to achieve such performance.\n\nIn Figure R1 of the attached rebuttal PDF, there is an illustration of the explained details, so please refer to it.\n\n**Weakness 1.3.** *How can RAFT be used for homography estimation?*\n\nA motion flow map from an optical flow network, such as RAFT, can be regarded as HW-correspondences (H: height, W: width). If HW is 4 or larger, the Direct Linear Transform (DLT) algorithm can fit a single homography. This homography best approximates the flow map in terms of least squared error.\n\nGenerally, in homography estimation, which has 8 degrees of freedom, most algorithms predict only 4 correspondences at the corners of images. They then convert these correspondences to a homography using DLT. Therefore, there is no reason HW-correspondences cannot be used for estimating homography.\n\nIf you are concerned about secondary aspects of optical flow, such as smoothness or the correction between global camera motion and local motion, please refer to our response in Weakness 1.1. As we explained there, homography estimation is estimating a linear transformation between pairs of static images. Therefore, these issues do not arise.\n\n**Weakness 1.4.** *What is the effectiveness of Modality-Agnostic Representation Learning (MARL)?*\n\nIt is to train the encoder that maps input images to the same feature space, not to train registration network directly. This goal allows our geometry loss to function properly regardless of modality difference. To achieve this goal, the loss term of MARL should be designed to enhance global similarity, rather than the local similarity of corresponding points, as is done with the GBT loss. This is why the global average pooling (GAP) layer should be included at the end of the projector.\n\nTable R2 shows the training results on Google Map dataset when GAP is not applied and spatial information is preserved to compute the similarity of local correspondences. It can be observed that proper training is not achieved in this case because local similarity is already met by the GBT loss, without considering global similarity.\n\nTable R2: Ablation results for including and excluding GAP.\n| Method | without GAP | with GAP |\n|--------|:-----------:|:--------:|\n| DHN + AltO | 24.07 | **6.19** |\n| RAFT + AltO | 24.07 | **3.10** |\n| IHN + AltO | 24.01 | **3.06** |\n| RHWF + AltO | 24.08 | **3.49** |\n\n**Weakness 2.** *Effectiveness of the Proposed Method. (Why is alternating necessary?)*\n\nTo prevent unintended collaborations and a collapse into trivial solutions, we introduced an alternating training strategy. Training the encoder and registration network together end-to-end without alternating can cause the encoder to output a constant value and the registration network to always output the identity matrix as the homography. Alternating training isolates these modules. This ensures the encoder maps input images to the same feature space, and the registration network estimates the true homography between the two images.\n\nTable R1 in the Auther Rebuttal (above) shows the experimental results when alternating training is not applied, demonstrating that proper training does not occur.\n\n**Weakness 3. & Limitation.** *Insufficient Experimental Validation*\n\nYou pointed out that the difficulty of the DeepNIR dataset is reduced due to the presence of simulated pairs. So we conducted experiments on the 'RGB-NIR' [1] dataset, which consists purely of RGB and NIR pairs without any synthetic images. We selected 'IHN' as the registration network, as it represents our primary model. The experimental results, as shown in Table R3 below, indicate that all methods performed better on this dataset compared to DeepNIR. This suggests that the DeepNIR dataset is harder. We believe this is because the DeepNIR dataset requires the model to learn two distributions (real NIR and synthetic NIR images), whereas the RGB-NIR dataset only requires learning a single distribution (real NIR).\n\nAdditionally, we conducted experiments on the 'OS Dataset' [2], which is a VIS-SAR type dataset, as per your suggestion. The results are also presented in Table R3.\n\nIn conclusion, our method outperformed other unsupervised methods on both of the additional datasets.\n\nTable R3: MACE evaluation on two datasets.\n| Method | RGB-NIR | OS Dataset |\n|:-------|:-------:|:----------:|\n| IHN (+Supervision) | 1.57 | 6.33 |\n| UDHN | 24.30 | 32.93 |\n| CA-UDHN | 24.27 | 24.96 |\n| biHomE | 25.18 | 24.99 |\n| IHN + AltO | **2.66** | **14.12** |\n\n[1] Matthew Brown et al., Multi-spectral sift for scene category recognition. CVPR 2011.\n\n[2] Xiang, Yuming, et al., Automatic registration of optical and sar images via improved phase congruency model, IEEE Journal of Earth Observations and Remote Sensing, 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for taking the time to thoroughly review our paper.

Reviewers highlighted the strengths of our paper.

+ Proposed method is an interesting, intuitive, and fresh approach. (PrYW, sFim, EsVS)
+ The paper tackles an important problem and proposes an original solution. (PrYW)
+ Quantitative results show improvement. (PrYW)
+ The organization and presentation of this paper are clear and good. (PrYW, sFim)

We will incorporate the reviewers' advice to further refine our research and improve the paper.

Additionally, through the review process, we found that our method outperforms the very recent paper, *SCPNet: Unsupervised Cross-modal Homography Estimation via Intra-modal Self-supervised Learning* (ECCV 2024 accepted).

The following experimental result was obtained using the Google Map dataset. * indicates values reported from SCPNet. (Lower is better.)
| Type | Method | MACE |
|:----:|:-------|:----:|
| Hand-crafted | SIFT | 24.53* |
| | ORB | 24.52* |
| | DASC | 21.76* |
| | RIFT | 16.55* |
| | POS-GIFT | 20.90 |
| Unsupervised | UDHN | 28.58 |
| | CA-UDHN | 24.00 |
| | biHomE | 24.08 |
| | BasesHomo | 24.49* |
| | UMF-CMGR | 24.60* |
| | SCPNet | 4.35* |
| | IHN + AltO (Ours) | **3.06** |

--------------------------------------------------------------------------------------------------------------------------------

Note : 
* The attached PDF includes figures addressing each reviewer's comments.

* Table R1 below shows the result of the experiment addressing the common question, 'Why is alternating necessary?'

Table R1: Ablation result on the Google Map dataset for applying the alternating and MARL module. (Alt. is Alternating)
|  **Method** | **No Alt.(with MARL)** | **No Alt.(without MARL)** | **Alt.(with MARL)** |
|:-----------:|:----------------------:|:-------------------------:|:-------------------:|
|  DHN + AltO |          24.09         |           24.27           |       **6.19**      |
| RAFT + AltO |          26.21         |           25.91           |       **3.10**      |
|  IHN + AltO |          24.37         |           23.14           |       **3.06**      |
| RHWF + AltO |          18.88         |           24.07           |       **3.49**      |

Meta-review: The reviewer opinion is split, even after seeing each others reviews and after an extensive post-rebuttal discussion. The main strength of the paper is the problem setting:  the handling of unsupervised training for a geometric problem with a significant domain gap, addressed by a novel loss function  The critical reviewer lists problems that are mainly technical in nature, but none of them is "rejection-implying". The rebuttal covered some of the issues (lack of references, same experiments that provide clarification).  

The paper is close to the accept - reject boundary. Overall, we see a benefit  in presenting the method at the conference, despite some remaining technical issues.
Decision: Accept (poster)