Mixed-View Panorama Synthesis using Geospatially Guided Diffusion
Authors: Zhexiao Xiong
Xin Xing
Scott Workman
Subash Khanal
Nathan Jacobs
Venue: Accepted by TMLR ()

Abstract: We introduce the task of mixed-view panorama synthesis, where the goal is to synthesize a novel panorama given a small set of input panoramas and a satellite image of the area. This contrasts with previous work which only uses input panoramas (same-view synthesis), or an input satellite image (cross-view synthesis). We argue that the mixed-view setting is the most natural to support panorama synthesis for arbitrary locations worldwide. A critical challenge is that the spatial coverage of panoramas is uneven, with few panoramas available in many regions of the world. We introduce an approach that utilizes diffusion-based modeling and an attention-based architecture for extracting information from all available input imagery. Experimental results demonstrate the effectiveness of our proposed method. In particular, our model can handle scenarios when the available panoramas are sparse or far from the location of the panorama we are attempting to synthesize.

Published in Transactions on Machine Learning Research (05/2025)
Mixed-View Panorama Synthesis using Geospatially Guided Diffusion
Zhexiao Xiong
z.zhezioo Swastl. edu
Deportment of Computer Science & Engineering, Woshington University in St. Louis
Xin Xing
saingSuuoomaaa.coo eduu
Department of Computer Science, University of Nebraska Omaha
Scott Workman
scott. www.wwwwwow.oaammaaooooo
Subash Khanal
k.subash edu
Department of Computer Science Engineering, Washington University in St. Lowis
Nathan Jacobs
Macobsn&russte edu
Department of Computer Science Engineering, Washington University in St. Lowis
Reviewed on OpenReview: https: openrevvee. net/ forum? id= ddvylUURss
Abstract
We introduce the task of mixed-view panorama synthesis, where the goal is synthesize novel panorama given small set of input panoramas and satellite image of the arca. This contrasts with previous work which only uses input panoramas (same-view synthesis), or an input satellite image (cross-view synthesis). We argue that the mixed-view setting is the most natural to support panorama synthesis for arbitrary locations worldwide. A critical challenge is that the spatial coverage of panoramas is uneven, with few panoramas available in many regions of the world. We introduce an approach that utilizes diffusion- basod modcling and an attention-basod architecture for extracting information from all available input imagery. Experimental results demonstrate the elfectiveness of proposed model can handle scenarios when the available panoramas
method. In particular, sparse or far from the location of the panorama we are attempting to synthesize. The project page is available at
view. wiw..ddddvie..tuuu........ 10
1
Introduction
The wide availability of street-level panoramas and their integration into mapping applications has dramat- ically improved the navigation experience for users. Access to nearby panoramas reduces the difficulty of navigating from purely overhead map. However, an inherent problem is that panoramas are expensive collect and thus are sparsely available and updated infrequently, with many roads having no panora- mas. This has motivated the task of cross-view view synthesis Zhha al. (017); Regmi & Borji (2018: Shi et al. )022a), where street-level panoramas are synthesized directly from salellite imagery. Unfortunately, existing approaches ignore any available nearby street-level panoramas at inference time.
We propose new task, mixed-view panorama synthesis (MVPS), which also aims to synthesize a street- level panorama. However, available street-level panoramas are also used to control the synthesis process. Figure. [ gives a visual overview the proposed mixed-view setting. In contrast to many recent works on novel view synthesis of outdoor scenes (c.g., NeRF-related works (Mildenhall et al., 2021 Martin-Brunl
al. 2021 Xie et al. 2023 Tancik al. 2022] and 3DGS-related works (Kerbl et al. 2023 Zhou et al.
2024
Liu et al. 20241) which require dense images with accurate camera pose information captured under

---

Published in Transactions on Machine Learning Research (05/2025)
Satellite Image
Model
Panorama of Target Location
Nearby Panoramas
Figure 1: We propose task, mixed-view panorama synthesis, in which satellite image and set of nearby panoramas (blue, yellow, and green) are used to render panorama at novel location (green). Our approach uses diffusion-based modeling and attention to enable flexible, multimodal control.
fairly controlled settings, input data are sparsely distributed panoramas, with only geo-location and orientation information provided. In particular, the distance, orientation, and availability of nearby street- level panoramas can vary dramatically. This task is in many ways more challenging than the text-to-image generation problem. First, previous toxt-to-imace generation methods only focus on semantic accuracy, while our lask, geometric laithfulness the primary factor to be considered. Second, nearby panoramas are often not acquired simultaneously, leading to significant challenges with non-stationary objects, lighting variation, and seasonal changes. Therefore, method needs to be able condition the output on images from different viewpoints, both street-level and overhead, using the geometric relationship between them.
Recently, conditional diffusion models Rombach et al. (2022) have been widely used in the image synthesis task. However, most of them are typically restricted to single-condition inputs, which is unsuitable for our MVPS task that aims to use multiple conditions and their correspondence to synthesize complex outdoor scenes, ControlNet Zhang et al. (2023) extends the pretrained Stable Diffusion model to take multi-modal data as inputs, while ControlNet-basse tially aligned with the target image, such as sketches, depth maps, and segmentation maps, which are essentially different forms of representation of the target image. In addition, recent works on attention- manipulated diffusion models predominantly focus on manipulating the text modality via cross-attention module. For outdoor panorama synthesis, with complex scene layouts, text alone is not sufficient to achieve fine-grained spatial control. Limitations of current diffusion frameworks motivate the need for frameworks that allow multi-conditioned and fine-grained geometry control for outdoor panorama synthesis. To address these challenges, there pressing need to develop a framework capable of handling multiple image input controls from diverse modalities, with the ability to guide the model to focus on the regions of the input images with salient geometric corrospondence the target image.
approaches still focus primarily on conditional inputs that are spa-
In this work, we propose a multi-conditionedd end-to-end diffusion framework for combining information from all input imagery to guide the diffusion- based mixed-viens synthesis process. A key element of our approach is the integration of geospatial attention Workman et al. (2022) to guide the controllable diffusion model for MVPS task. Geospatial attention incorporates the semantic content of the input, geometry, and overhead appearance to identify the geo-informative regions of an input panorama relative the target location. We extend the concept of geospatial attention to include both local-level and global-level attention. The resulting attention maps are fused with the corresponding input images in the latent space of the diffusion model. Finally, the altention-guide! features are inlegrated into the encoders of the corresponding conditional branch in the condiiccndi ControlNet model, achieving coometric-guidedt fine-grained spatial control.
The key contributions of this work can be summarized as:
We propose the mixed-view panorama synthesis task: using a satellite image and a sparse set of nearby panoramas to senthesize target panorama at given location.
2

---

Published in Transactions on Machine Learning Research (05/2025)
We propose unified multi-conditioned controllable diffusion framework, GeoDiffusion, for the mixed-view panorama synthesis task.
We use geospatial attention as the geometric constraint associate nearby views with the target view to achieve geometry-guide fine-grained spatial control.
We validate that our proposed method generates high-fidelity panorama images with geometric accuracy on the Brooklyn and Queens benchmark dataset, achieving state-of-the-ae performance while having more flexibility compared with prior cross-view synthesis works.
2 Related Work
2.1
Cross-view Synthesis
Given a satellite image, the cross-view synthesis task aims to predict the street-level panorama. Prior work includes using a learned linear transformation Zhai al. (2017), applying conditional GANs Regmi & Borji
[2018
2019)
Tang et al. (2019); Li et al. (2021), integrating height estimation explicit supervision Shi
et al.
2022a
using densily field representation Qian et al. (2023 and point-based neural rendering Li
ct al.
Structure-prseeriing panorama. synthosis methods have also shown to be effective in the related task of cross-view image geolocalization Regmi & Shah 2019 Toker et al. (2021); Shi et al. (2022b).
2024
However, such methods typically only consider a satellite image as input and only attempt to synthesize
street-level panoramas the setting where the satellite image is center-aligned with the target. location. Our work integrates near and remote modalities (i.c., mixod-view) and can synthesize panoramas arbitrary locations in the given satellite image region.
2.2
Image-to-lmage Translation
The goal of image-to-image translation to learn a mapping between an input image and a target image Isola
et al. 2017)) Traditional methods are based on generative adversarial networks Park al. (2019): Wang
et al.
2018a); Liu et al. (2017); Zhu et al. (017)) With the development of vision transformers. several
works have successfully applied transformers to this problem Esser et al. (2021) Chang el al. (0022 More
recently, numerous methods have leveraged diffusion model to perform image translation Dhariwal & Nichol
(2021); by Rombach et al. (2022); Xue et al. (0223 Parmar et al. (2023); Wang et al. 2022).. However, these
methods are unable to handle cases in which direct conditions are unknown and only implicit geometric
relationships are available. In contrast, our model is capable of handling multiple indirect image controls
and utilizing the geometric relationships both within and across different input modalities.
2.3 Conditional Diffusion Models
Conditional diffuuson models enable controllable image synthesis and editing. Most recent work in this do- main focuses on the text-to-image synthesis problem, such as GLIDE Nichol et al. (2021). DALL-E2 DALL-E2Ramesh Ramesh et al. (202)) and Stable Diffusion Rombach et al. (0022) These methods require large training datasets containing many image-text pairs to generate high-quality images. However, generating complex scenes is challenging with only text information. Recent works have proposed incorporating local, spatial condi- tioning, such as segmentation maps Mou et al. (2024) or layouts Li et al. (2023); Zheng et al. (2023); Qu ct al. 2023) to overcome these challenges and achieve precise spatial control. ControlNet Zhang et al. (2023) and similar methods Qin et al. (2023) Zhao et al. (2023) use zero-convolution layers to incorporate task-specific conditioning into pretrained image diffusion models, significantly reducing the computational cost and sample complexity while still generating high-quality images. We adopt this approach build a multi-conditioned, end-to-end geospatial attention-guided diffusion framework.
2.4 Attention
Attention mechanisms have shown benefits in a variety of visual tasks. Commonly used attention mechanisms inchude channel-wise attention Hu et al. (2018b); Woo et al. (2018) spatial attention Mnih et. al.) (2014):
3

---

Published in Transactions on Machine Learning Research (05/2025)
Jaderberg et al.
2015): Hu al. 2018a Wang al.
2018b)
spatial-temporal attention Meng et al. (0119
Song et al.
[201] 7), and branch attention Fukui et al.
2001
Self-attention Vaswani et al. 2001 7 and cross-
attention Chen et al. [2021] are widely applied in vision transformers Dosovitskiy et al. (2020) : Carion et al.
2020 : Xic et al. 20021). ). Recently latent diffusion models Rombach et al. 2022 Wu et al. (2023 : Xue et al.
2023); also regard cross-altention as an effective way to allow multi-modal training for class-conditionall text-to-image, and spatially conditioned tasks. Besides cross-attention, geometry has been used to inform the learning of attention, such as using epipolar attention He et al. (2020) in novel view synthesis [seng et al. (0223). Workman et al. WWokmann et al. (2022) proposed the concept of geospatial attention, geometry-aware attention mechanism that explicitly considers the geospatial relationship between the pixels in ground-level image and a geographic location. We extend this concept to our mixed-view panorama hesthesis task to identify geo-informative regions across mixed viewpoints, allowing our diffusion model to "geometry aware.
3 Preliminaries
3.1 Stable Diffusion Architecture
Stable Diffusion Rombach al. (2022) is generative modeling approach via learned diffusion process, by applying the diffusion operation in the latent space. Tt uses UNet-like structure as its denoising model, which consists of an encoder, a middle block, and a decoder. Both the encoder and the decoder are made up of 12 blocks, and the full model contains 25 blocks. The outputs of the encoder are added to the 12 skip-connections and middle block of the UNet. The input for the i-th block in the decoder is represented as:
concat (m, fz) concat (g)-1,fz)
where i i= 1,j - where 2 i 12
where m denotes the output of the middle block and fi and 9i denote the output of the i-th block in the encoder and decoder, respectively.
3.2 Mixed- View Panorama Synthesis
We introduce the task of mixed-vier panorama synthesis (MVPS). Given satellite image S1, and set of sparsely distributed nearby street-level panoramas (P1, ......, Pn) with known geolocations (4,,2, 12,22,... (,)) we aim to synthesize a panorama P: in the region of S at a target location 4. To generate the target panorama Pt with precise layout distribution, the synthesis process should utilize: (1) the semantic informa- tion from P2.... ... Pn and $1, (2) the geometric relationships between street-level P. + P., E 21,2, .n) and across satellite & strect-level S + P.. With only location information provided, how utilize the im- plicit geometric relationships the key challenge of this task.
4 Geospatial Attention-Guider Diffusion Model
We propose novel multi-conditioned, geospatial attention-guided diffusion model, GeoDiffusion, which com- bines information from a satellite image and ncarby strect-level panoramas to synthesize a target panorama. Figure. shows an overview of our framework, which consists of two main modules: (1) a novel geospatial attention adapter that combines information from the input imagery and (2) a multi-conditioned diffusion model, based on ControlNet Zhang et al. (2023), that synthesizes the target panorama.
4.1 Geospatial Attention Adapter
We propose an adapter that uses geospatial atlention Workman et al. (0022) to fuse image features that are extracted using CNN-based encoders. The geospatial attention adapter has two components: (1) local attention for assigning weight to panorama features and (2) global attention for assigning weight to satel- lite features. These attention maps are used to fuse features from the CNN-derived feature maps, which subsequently control the diffusion model.
4

---

Published in Transactions on Machine Learning Research (05/2025)
Prompts
Encoder
Middle Block
Decoder
Noise
Add
Concat
Synthesized Panorama
Zero Conv
Only Used in Training Stage
Copied
Segmentation Map
Noise
Fusion
Near Panoramas
Prompts
Geospatial Altention Extractor
Base Stable Diffusion Model
Geospatial Attention Adapter
Satellite Image
Multi coldiionnd ControlNet Model
The overall framework of our model. For each branch of the near panoramas in the multi- conditioned ControlNet diffusion model, the near panoramas and the satellite image are passed through the geospatial attention adapter and the copied encoder, and the extracted features are injected into the Stable Diffusion encoder.
Figure 2:
Orid
Overhood Image
Oveiheed Fasture
Resnett3 Pretrained model
Orientation
Haversine Distncce
Orientation Festure
CNN Encoder
Distance
Compasx Bearing
Var
1x 100w
trner Product
Pooling
Reshage
Upsample
Enooder F,
Noor Parorama
Feeture msp
Pooling
NKMXO
SCheer Concaterated
Featuure Map
Softmaz
Aus
P
Gecepartal Attention
Local Geospatial Attention Extraction
Figure 3: The local geospatial attention extraction module, which provides attention for near panoramas. The red dot represents the largel. location, and the green dot represents the source panorama location in the overhead image region.
Extracting Local Attention build a local gcospatial attention module, shown visually in Figure. 3 In our setting, we assume images are fully calibrated. Given panorama at location and the target location 4, for MVPS, we apply geospatial attention Workman al. (2022) based on the relative location, the target-relative orientation of each pixel ray, the semantic content of the input, and the overhead appearance, and get a spatial attention map, A. R Based A,, and the feature map of the near panorama F, R#xXXC compute welinteon-weighteed feature descriptor Pi, equation 2
To identify geoinformative regions in a nearby street-level panorama, we
P., reshape((('s (Fs, Ait)
2
where f E XW represents the input feature map of the e-th channel of F,, (.,-) denotes the Frobenius inner product of the two inputs, and Pt, represents the feature output, which represents the information from the input feature map F; that is relevant to the target location 4. We detail the extraction of local-level geospatial attention in the supplementary material.
Extracting Global Attention One of the key challenges MVPS is handling the nonuniform spatial distribution of the input panoramas. To address this, we extract the spatial distributions of the panorama
5

---

Published in Transactions on Machine Learning Research (05/2025)
Pa
Concat
Avg Pooling
+ Upsample
Sigmold
Global Geospatial Attention
Pit
Global Geospatial Attention Extraction
Figure 4: Global geospatial attention extraction module, which provides attention for satellite images.
X
X4
Image
POU
Adention Nank
Feature
Imoope Aiteestion
Donsn
Mask
Dichi
06 21
Oulput Peatures
Overteed
Feeture
dL 20
Olobal Decupatal
Attentio 1
Parkur
Feature DenomaalizationFFO))
Figure 5: The geospatial attention fusion module. We use the same architecture to add attention to near panoramas and satellite image respectively in the latent space, and inject the features into the copied encoder of the diffusion model.
features in the given region and propose global geospatial attention, which is attention map that is used to guide which features should be more heavily weighted in the feature space of satellite image. See Figure. g for overview. Given a target location 4,, this module operates on the attention- features for the input street-level panoramas computed using local-level geospatial attention Pi.., P2.,, ,,.,, where i is the number of panoramas. We aggregate those features using concatenation and average pooling, and get a 32 x 32 weight feature grid. We then upsample this dense grid to 256 x 256 and pass it through a batch normalization layer and sigmoid layer to obtain the final global-level geospatial allention. The resulting attention map the global-level feature distribution in the overhead-view depending on the relative location of the target panorama and the input panoramas.
Fusion After extracting the attention mask from both street-level and satellite views, we fuse them with the corresponding street-level panoramas and the satellite image. The fusion module is shown in the left part of Figure. 5 For the reee-leee panoramas, we use a feature extractor composed of stacked convolutions to extract the attention mask and the near panoramas to the latent space. Both of their shapes are B x 128 x x 256 and we fuse them using Hadamard product. We also add skip connections between the masked near panorama feature map and the original near panorama feature map, which is shown
Ha,(x) (1 Mi,((1) @ Pio(x)
where (x) is the latent space geospatial ion mask ranging from [0, 1], i ranges over all spatial positions, and e e (1. C} is the index of the channel. When (x) approximates O, the output feature Hxx) will approximate original latent features F(x). For the satellite image, based on the extracted satellite attention, we use the same method as nearby street-level panoramas to get the inrention-guided feature map. Finally, the output features passed through multi-scale feature extractor to get the output features.

---

Published in Transactions on Machine Learning Research (05/2025)
We show the detailed structure of the multi-scale feature extractor in the right part of Figure. In the multi-scale feature extractor, for each resolution, we first pass the feature through zero-convolution layer, which progressively starts influencing the generation with the attention-guided features. Then we adopt Feature Denormalization (FDN) Park et al. (2019) as injection module, which uses the condition features for GeoDiffusion to rectily its normalized input noise features as:
FDN, ci) = norm((,, (1 cont,((ero(t,((t))) + coneg(zero(h,(())),
(4)
where Z, is the noise features at resolution T, c, represents the concatenated conditions, h, represents the output of the feature extractor H at resolution r, and comvy and conue refer to the learnable convolutional layers that convert condition features into spatial-sensitive scales and shift modulation coefficients respec- tively. The final extracted features are concatenated and injected into the respective copied encoders. We select the first block of each resolution in the copied encoder for feature injection, corresponding to the 1, 4. 7, 10 layers of the copied encoder respectively. Leveraging the effectiveness of geospatial attention, the multi-conditioned diffusion network can generate more geometrically accurate panoramas.
4.2 Multi-Conditioned Diffusion Model
We introduce our approach based on the conditioning features extracted from the overhead imagery and nearby panoramas. Inspired by ControlNe et al. (2023), we construct a multi- conditioned diffusion model for MVPS (Figure. [ We start. from pretrained Stable Diffusion Rombach al. (2022) model and duplicate the structures and weights of the encoder and middle block for each condition i, which we refer to F and M/ respectively. The conditions the geospatial attention adapter, and then the attention guidded features for each condition are injected into the corresponding copied encoder, with noise added according to the time embedding. The encoded features are passed through a zero-convolution layer and incorporated into the main branch of the Stable Diffusion architecture. During the decoding process, we keep all other elements unchanged while modifying the input of the i-th block of the docoder as:
concatenated in the channel dimension, passed through
concat (m + m', fs + zero (f))) where i 1, 13 concat (gi-1. + zero (f)) where sis 12.j 13
(5)
in which m denotes the output of the middle block, fi and g, denote the output of the i-th block in the encoder and decoder of UNet respectively, and zero represents a zero convolutional layer. Following ControlNet Zhang et al. (2023), the weights of zero convolutional layer is set to increase from zero to gradually add control information into the main Stable Diffusion model. For the text input, as we do not rely on text embedding to achieve content manipulation, we use "A high-resolution street-view panorama" the default prompt.
5 Experiments
We evaluate our approach for mixed-view panorama synthesis quantitatively and qualitatively through var- ious experiments. Results show that our approach, which can take advantage of nearby street-level panora- mas, significantly improves results compared to existing cross-view methods, and shows more flexibility in synthesizing the panorama in arbitrary locations.
Dataset We train and evaluate our methods using the Brooklyn and Queens dataset Workman et al. (2177. This dataset contains non-overlapping satellite images (approx. 30 em resolution) and street-level panoramas from New York City collected from Google Street View. It is composed of two subsets collected from Brooklyn and Queens respectively. The Brooklyn subset contains 43,605 satellite images and 139,327 panoramas. The Queens subset, which we use solely for cross-domain evaluation, contains 10,044 satellite images and 38,630 panoramas. For evaluation on the Brooklyn subset, we use the original train/test split, resulting 38,744 images for training, 500 images for validation, and 4361 images for testing. For cross- domain evaluation on Queens, we randomly select 1000 images from the Queens subset and report the performance on the selected images. During training, when computing geospatial attention, we set the
7

---

Published in Transactions on Machine Learning Research (05/2025)
(a) Sat
(b) Pix2Pix
(c) PanoGAN
(d) Sat2Density
(e) Ours
(I) Ground Truth
Figure 6: Qualitative results versus baselines. The cross-view synthesis methods that we compare with are
trained on our collected center-aligned satellite images. Our approach, which integrates nearby street-level panoramas, not only generates more realistic results when compared baselines, but more accurate results both semantically and geometrically when compared to the ground truth.
Location
Only Satellite Image
Only Panoramas > 50m Only Panoramas < 50m
Full Data
Ground-Truth
Figure 7: Qualitative results for MVPS using different sources of data.
number of nearby street-level panoramas considered, N, to 20 for each satellite image; In the conditional diffusion module, for cach pair of data, we use the 2 closest panoramas to the target location as conditions. During the inference stage, the geospatial attention module is frozen, so only 2 nearby panoramas are in need.
Metrics For evaluating the performance of our approach, consider two classes of metrics. The first class is low-level metrics which evaluate the pixel-wise similarity between two images: peak signal-to-noiso ratio (PSNR), structural similarity index (SSIM), root-mean-square error (RMSE), and sharpness difference (SD). The second class is high-level metrics which evaluate image-level differences between two images. For perceptual similarity (LPIPS) Zhang al. (2018), we use pretrained AlexNet Krizhevsky al. (2012) and denote as Patez- We also adopt Fréchet inception distance (FID) Heusel et al. (011)), a common metric for measuring the realism and diversity of images produced by generative models.
8

---

Published in Transactions on Machine Learning Research (05/2025)
Comparison with cross-view synthesis methods on Brooklyn test set. Center- Aeiter--lliged::: use satellite data that are center-aligned with the target location.
Table 1:
Center-Aligned Aligned
Method
PSNRT
SSIM1
Patert
RMSE]
FID]
SD1
Pix2pix Isola et
(207))
11.93
0.0950
0.6161
64.75
413.29
9.07
PanoGAN Wu Wu et al. (2022
13.10
0.2981
0.5583
56.98
166.30
12.04
Sat2density Qian et al.
2023
13.39
0.4325
0.5407
55.38
153.32
13.22
GeoDiffusion (Ours)
14.14
0.4329
0.4343
51.42
33.68
13.60
Pix2pix Isola et al.
(2017)
12.19
0.3375
0.5503
63.14
263.97
11.71
PanoGAN Wil et al.
2022
13.64
0.4044
0.4856
53.71
130.98
13.28
V
Sat2Density Qian
(023
14.57
0.4465
0.4684
48.54
87.77
13.66
GooDiffusion (Ours)
14.66
0.4498
0.4206
48.31
31.07
13.79
Table 2: Cross dataset evaluation on Queens. Center-Aligned: use satellite data that are center-aligned with the target location.
Center-Aligned
Method
PSNR1
SSIM1
Patex
RMSEJ
FIDI
SDT
Pix2pix Isola et (207))
11.82
0.1719
0.6435
65.74
386.41
9.51
PanoGAN Wu
2022
12.79
0.2968
0.5770
59.24
183.82
12.02
Sat2Dtnsity Qian et al.
2023
12.93
0.4023
0.5524
58.63
181.45
13.31
GeoDiffusion (Ours)
13.54
0.4239
0.4661
55.80
57.66
13.55
Pix2pix Isola ot (2077)
11.77
0.2537
0.5450
66.73
265.53
PanoGAN Wu Wu
al. (2022
13.45
0.4014
0.4933
57.01
99.56
13.08
Sat2Densit Qian al.
(2023
13.87
0.4478
0.4835
53.76
107.14
13.61
GeoDiffusion (Ours)
14.08
0.4488
0.4585
53.46
54.26
13.72
Table 3: Ablation study for geospatial attention. Local: local-level attention. Global: global-level attention.
Local
Global
PSNRT
SSIM1
Pates
RMSEJ
FIDI
SD1
11.69
0.3567
0.5089
67.41
52.56
12.60
12.95
0.3757
0.4686
58.49
37.90
12.79
12.83
0.3898
0.4757
59.45
35.54
12.89
/
14.14
0.4329
0.43443
51.42
33.68
13.60
Implementation Details We use a pretrained Stable Diffusion Rombach et al. (2022) model (v1.5) with the default. parameters. For optimization, images resized to 256 x 1024 as local conditions. Specifically, the satellite image is resized to 256 x 256 and replicated horizontally four times. We adopt ViT-Adapter Chen et al. (2022) to get the segmentation map of the target panorama. We use DDIM [Sng et al. (2020) for sampling with the number of time steps set to 50. The classifier free guidance Ho & Salimans| (2022) is set to 7.5. During training, the segmentation map of the target image, the satellite image, and set of nearby street-level panoramas are passed through the controllable diffusion model synthesize the target panorama. At inference, only the satellite image and available nearby panoramas required to synthesize the target panorama image, following the same assumption made in Regmi & Borji (2018) since sogmentation maps unlikely to be available in the real world.
use AdamW with a learning rate of X = 10-5 The input
Modality Dropout Using multi-conditioned inputs results in the high reliance one or subset of the conditions. To avoid this, we implement a modality dropout strategy in three forms: 1) randomly omit each individual condition 2) randomly keep all conditions and 3) randomly drop all conditions. This allows the model to learn the mixed-view panorama synthesis task based on arbitrary conditions, reducing the model's reliance on individual conditions, and helping learn the relationships between different modality compositions. In experiments, during training, set the rate to keep/drop all conditions 0.3 and

---

Published in Transactions on Machine Learning Research (05/2025)
Table 4: Ablation study on data distribution. Satellite image P: Panoramas.
We set the threshold between 'near" and 'far' to 50m.
s
P 50m
P < 50m
PSNR1
SSIM
Patestk
RMSEL
FID
SD1
11.85
0.3143
0.5416
65.67
48.68
12.26
12.20
0.3229
0.5244
63.10
42.81
12.37
12.44
0.3224
0.5285
63.40
42.94
12.49
12.67
0.3488
0.5174
37.12
39.71
13.42
0.4136
0.5013
55.01
38.38
13.51
14.14
0.4329
51.42
Table 5: Ablation study. We analyze the effect of modality dropout and adding segmentation maps during the training stage.
Ablation Objective
PSNR
SSIM1
Pater
RMSEL
FID.
SD1
w/o modality dropout
11.02
0.3215
0.5764
72.31
185.27
12.84
w/o segmentation maps in training Simple attention w/o geospatial attention adapter
12.54
0.3665
0.5267
64.46
96.87
12.98
11.83
0.3275
0.5783
72.85
98.86
12.75
Concat conditions and noise along spatial dimensions
13.12
0.3589
0.5172
58.23
82.36
13.16
Simple attention w/o geospatial information
13.31
0.3896
0.4765
55.27
56.83
13:35
Full
14.14
0.4329
0.43-43
51.42
33.68
13.60
0.1 respectively, and set the dropout rate of cach condition to 0.1. For the text prompts, we randomly replace 50% of text prompts with empty strings to enhance the model's ability to learn image geometric relationships.
5.1 Quantitative Results
We compare the performance of our method against several baseline methods in Table | [nd cross-domain evaluation results in Table 2 using the Brooklyn and Queens dataset. We also show qualitative results in Figure. G For the baseline methods, Pix2pix Isola et al. (2017) is traditional GAN-based image-to- image translation method. PanoGAN Wu et al. 20222) is a recent CAN-based cross-view synthesis method. Sat2Density Qian et al. (2023) is the statc-of-the-art cross-view synthesis method.
As mentioned in previous works Zhu al. (2021), ,, in practical real-world applications, the panorama that users want to synthesis can oceur at arbitrary locations in the area of interest, in this case perfectly aligned correspondence is not guaranteed. We both report the results on (1) the original Brooklyn and Queens dataset, in which satellite image and target location not center-aligned and (2) our collected satellite images that are center-aligned with the target. location. In both these two circumslances, our method out- performs all the cross-view synthesis methods. Specifically, our method shows more superiority on unaligned satellite data, while other cross-view synthesis methods have a significant performance degradation on un- aligned satellite data. Our reffusion-basel method has advantages especially in high-level metrics LPIPS and FID scores, showing its ability to generate high-fidelity images with global-level accuracy. Overall, our method shows more flexibility as it can achieve competitive performance without requiring the target panorama to be located at the center of the satellite image, improving the efficiency especially in paral- lelly generating multiple panoramas for given region and bridging the gap between current research and practical applications.
5.2 Ablation Study
We conduct an ablation study that evaluates the impact of individual components of our proposed gcospatial attention adapter on the resulting image quality. The results are shown Table 3 Experimental results show that both the local and global geospatial attention can guide the diffusion model to better utilize the
10

---

Published in Transactions on Machine Learning Research (05/2025)
W
150
Figure & Visualization of local geospatial attention. The larget location is represented by a green square in the satellite image. The nearby strect-level panoramas (color-coded borders) represented by same-colored circles in the satellite image.
Figure 9: Visualization of global geospatial attention. The color- attention maps for two target locations are shown, corresponding to the same-colored dots in the satellite image. Darker colors represent more salient regions.
geometric relationships between mixed-view modalitics. Furthermore, combining local and global geospatial allention helps supervise the model towards generaling results with accurate layoul distributions.
We also analyze the effect of the satellite images and near panoramas of different distances on our model. The results are shown in Table D and Figure. 7 In this experiment, we set thresholds to 50 meters based on the haversine distance between the near panorama and the target panorama to define 'near' and 'far'. We train our model on each subset and all available data. Panoramas that are near the target location have larger effect the synthesis quality there more overlapping regions. However, the model does not rely on the 'near' panoramas to capture accurate geometry, since the model trained on 'far' panoramas and satellite images outperforms the model only trained on 'near' panoramas. The model trained on subset of the data cannot synthesize accurate layout and sometimes confuses the sky region with trees. In summary, using all data significantly improves the synthesis quality of the model.
We conduct further ablation study Table We first conduct an experiment on modality dropout strat- egy. Compared with keeping all the input images, adding modality dropout achieves 24.6%% in Patex and 81.8%J in FID. We also remove the segmentation map branch of the multi-conditioned diffusion model in the training stage, which leads to severe performance degregation. Furthermore, conduct experiments about: (1)concatenate conditions along the channel dimension and train a ControlNet to guide generation; (2) concatenate conditions and noise along spatial dimensions, followed by fine-tuning the diffusion model and (3) use simple attention to compute the attention map between the source view image and target view image, without adding location information and orientation information. All of these models show worse per- formance. These experiments demonstrate the effectiveness modality dropout strategy and the supervision of segmentation maps in the training stage.
11

---

Published in Transactions on Machine Learning Research (05/2025)
Table 6: Quantatitive comparison about Seamless.
PanoGAN Wu et 2022
Sat2Dnsstty Qian al. 2023
Ours
LRCE
0.0897
0.0653
0.0436
5.3 Visualization of Geospatial Attention
We demonstrate the concept of geospatial attention visually. Figure. 8 shows an example of local geospatial attention, which uses the relative orientation and distance of nearby street-level panorama in addition to semantic content. The street-level panoramas and paired local-level attention map use an east-north-up coordinate system, i.e,, the center points to the north and the / boundary points to the south as Zhu iet al. 2022I. The target location is indicated by the green square in the satellite image and the location of the panoramas is indicated by the same-colored dot. Larger attention values (red) reflect the region in the panoramas that orient towards the target. Note that when the image and target locations are further apart, the high attention region shrinks, essentially reflecting the narrower field of view.
Figure. shows an example of global geospatial attention for two target synthesis locations. Larger attention values (darker) reflect the regions in the overhead view that contribute the most. Note that, for the Brooklyn and Queens dataset, the street-level panoramas are primarily collected along the stress. This is captured in the global-level attention maps, which tend to show higher attention scores in the street-level regions near the target location (omitting buildings). Additional visualizations are shown in the supplementary material.
5.4 Discussion about Seamless
We adopt the Left-Right Consistency Error (LRCE) metric (Shen et al. 2022 to quantitatively assess the consistency along the left-right boundaries by analyzing the horizontal gradients. For depth evaluation, we integrate Depth Anywhere (Wang & Liu 2024), a state-of-tho-art panorama depth cstimation method, to predict the depth maps of the panoramas. We compute the horizontal gradients of both the ground-truth depth maps and those of the generated panoramas, and then calculate the gradient differences. By randomly sampling 100 generated panoramas from the test set, we determined the mean gradient difference, as detailed in Table 0 Our experimental results indicate that, compared to previous cross-view synthesis methods, approach achieves superior left-right boundary consistency, thereby demonstrating its seamless performance.
6 Conclusion and Future Work
We introduced the task of mixed-view synthesis, which extends the cross-view synthesis task to also in- clude a set of nearby street-level panoramas as input. We proposed novel multi-conditinned, end-to-end geospatial attention-guided diffusion framework for combining information from all input imagery to guide the diffusion- bradssion-basedd syuthesis process, achieving cross-view approaches, our approach does not require segmentation map of the target panorama during the inference stage or, like all previous cross-view approaches, that the target panorama be located at exactly the center of the satellite image. Together, these characteristics dramatically increase flexibility and ease of Experimental results demonstrate the effectiveness of our proposed model, in particular its ability to handle situations when the panoramas far from the target location.
comeetr--guider fine-grained spatial control. Unlike some
For urban scenes, transient objects (e.g., cars, pedestrians) bring challenges to our method. Also due to the inherent randomness of the diffusion models, maintaining the view consistency between adjacent locations is still challenging some detailed regions. Understanding how to reduce the influence of transient objects on guided diffusion models synthesize clean and view-consistent panorama sequences would be future research direction.
12

---

Published in Transactions on Machine Learning Research (05/2025)
References
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Buropean conference on computer vision, pp. 213-229. Springer, 2020.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, 2022.
Chun-Fu (Richard) Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-atlention multi-scale vi- sion transformer for image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXtt:2205.0853) 08534, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa, Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the LEEE/CVE conference on computer vision and pattern recognition, 2021.
Hiroshi Fukui, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Attention branch network:
Learning of attention mechanism for visual explanation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2019.
Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu. Epipolar transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognation, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two timo-scale update rule converge to a local nash equilibrium. Advances in neural information processing syslems, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusson guidance. arxiv preprinl xxXi::2207.12500 2022.
Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Andrea Vedaldi. Gather-excite: Exploiting feature context in convolutional neural networks. In Advonces in neural information processing systems, 2018a.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitttion networks. In Proceedings of the IEEE/CVE confer- ence computer vision and pattern recognition, 2018b.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with condi- tional adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2017.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial (ransformer networks. In Advances in neuraa information processing systems, 2015.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, 2012.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/ IEEE/CVP conference on computer vision and pattern recognition, pp. 22511 22221, 2023.
Zuoyue Li, Zhenqiang Li, Zhaopeng Cui, Rongjun Qin, Mare Pollefeys, and Martin R Oswald. Sat2vid: street-view panoramic video synthesis from a single satellite image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Zuoyue Li, Zhenqiang Li, Zhaopeng Cui, Mare Pollefeys, and Martin R. Oswald. Sat2scene: 3d urban scene generation from satellite images with diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition (CVPR), pp- 7141-7150, June 2024.
Ming-Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. Tn Aduances in neural information processing systems, 2017.
Yang Liu, Chuanchen Luo, Lue Fan, Naiyan Wang, Junran Peng, and Zhaoxiang Zhang. Citygaussian: Real-time high-quality large-scale scene rendering with gaussians. In European Conference Compuler Vision, pp. 265-26652 282. Springer, 2024.
Ricardo Bruall,, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7210-7219, 2021.
Lili Meng, Bo Zhao, Bo Chang, Gao Huang, Wei Sun, Frederick Tung, and Lconid Sigal. Interpretable spatio- temporal attention for video action recognition. In IEEE/CVF International Conference on Compuler Vision Workshops, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communicatiinn of the ACM, 65(1):99-16) 106499900,,1100 2021.
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in neural information processing systems, 2014.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for toct-to-image diffusion models. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 4296-4304, 2024.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. Xr preprint or Xii::11 12. 10741, 2021.
Taesung Park, Ming-Vi Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2337-2346, 2019.
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun- Yan Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH, 2023.
Ming Qian, Jincheng Xiong, Gui-Song Xia, and Nan Xue. Sat2density: Faithful density learning from satellite-gre groutd image pairs. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3683-3692, 2023.
Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang. Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: A unified diffusion model for controllable visual gener- ation in the wild. arXiv preprint ar xxXiv:2305.11147 2023.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Leigang Qu, Shengqiong Wu, Hao Fei, Liqiang Nie, and Tat-Seng Chua. Layoutllm-t2i: Eliciting layout guidance from 11m for text-to-image generation. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 643 654, 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.000 061125 1(2):3, 2022.
Krishna Regmi and Ali Borji. Cross-view image synthesis using conditional gans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2018.
Krishna Regmi and Ali Borji. Cross-view image synthesis using geometry-guides conditional gans. Computer Vision and Image Underslanding, 187:102788, 2019.
Krishua Regmi and Mubarak Shah. Bridging the domain gap for ground-lo-aerial image matching. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022.
Zhijic Shen, Chunyu Lin, Kang Liao, Lang Nie, Zishuo Zheng, and Yao Zhao. Panoformer: panorama transformer for indoor 360 depth estimation. In European Conference Compuler Vision, pp. 195 211. Springer, 2022.
Yujiao Shi, Dylan Campbell, Xin Yu, and Hongdong Li. Geometry-guided street-view panorama synthesis from satellite imagery. IEEE transactions on pattern analysis and machine intelligence, 44(12):10009 10022, 2022a. doi: 10.1109 200///TT...2222.114000.0
Yujiao Shi, Xin Yu, Liu Liu, Dylan Campbell, Piotr Koniusz, and Hongdong Li. Accurate 3-dof camera geo- localization via ground-to-satellle image matching. IEEE transactions on pottern analysis and machine intelligence, (633))66222222
2022b.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ar preprint arXiv:2010.022: 02500,, 2020.
Sijie Song, Cuiling Lan, Junliang Xing, Wenjum Zeng, and Jiaying Liu. An end-to-end spatio-temporal allention model for human action recognition from skeleton data. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 8248 8258, 2022.
Hao Tang, Dan Xu, Nicu Sebe, Yanzhi Wang, Jason Corso, and Yan Yan. Multi-channel attention selection gan with cascaded semantic guidance for cross-view image translation. In Proceedings of the IEEE/CVE /CVF conference on computer vision and pattern recognition, 2019.
Aysim Toker, Qunjic Zhou, Maxim Maximov, and Laura Leal-Taixé Coming down to carth: Satellite-to- street view synthesis for geo-localization. In Proceedings of the IEEE/CVE conference on computer vision and pallern recognilion, 2021.
Hung-Yi Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia-Bin Huang, and Johannes Kopf. Consistent view synthesis with pose guided diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023.
Ashish Vaswani, Noam Shazoer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in neural information processing systems, 2017.
15

---

Published in Transactions on Machine Learning Research (05/2025)
King-Hsu Wang and Yu-Lun Liu. Depth anywhere: Enhancing 360 monocular depth estimation via perspee- tive distillation and unlabeled data augmentation. Advances in Neural Information Processing Systems. 37, 2024.
Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. Xivi preprint ar Xix: 1 12952, 2022.
Ting-Chun Wang, Ming-Yu, Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2018a.
Xiaolong Wang, Ross Girshick, Abhinav Gupla, and Kaiming He. Non-local neural networks. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, 2018b.
Sanghyun Woo, Jongchan Park, Joon-Young Loe, and In So Kwcon. Cbam: Convolutional block attention module. In European conference on computer vision, 2018.
Scott Workman, Menghua Zhai, David Crandall, and Nathan Jacobs. unified model for near and remote sensing. In Proceedings of the IEEE International Conference Computer Vision, pp.
2017.
Scott Workman, M Usman Rafique, Hunter Blanton, and Nathan Jacobs. Revisiting near/remote sensing with geospatial attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022.
Qiucheng Wu, Yujian Liu, Handong Zhao, Trung Bui, Zhe Lin, Yang Zhang, and Shiyu Chang, Harnessing the peatialtemporrd attention of diffusion models for high-fideliyy text-to-ing vext-to-image synthesis. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 7766-77677 7776, 2023.
Songsong Wu, Hao Tang, Xiao Yuan Jing, Haifeng Zhao, Jianjun Qian, Nicu Sebe, and Yan Yan. Cross-view panorama image synthesis. IEEE Transactions on Multimedia, 2022.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In Advances in neural information processing systems, 2021.
Ziyang Xie, Junge Zhang, Wenye Li, Feihu Zhang, and Li Zhang. S-nerf: Neural radiance fields for street views. arXw preprint Xii:23333000749 2023.
Han Xue, Zhiwu Huang, Qianru Sun, Li Song, and Wenjun Zhang. Freestyle layout-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14256 14266, 2023.
Menghua Zhai, Zachary Bessinger, Scott Workman, and Nathan Jacobs. Predicting ground-level scene layout from aerial imagery. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2017.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1836-3841 2023.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectivencss deep features as a perceptual metric. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2018.
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Informa- tion Processing Systems, 36:11127-111 2023.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Guangeong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion Con- trollable diffusion model for layout-to-image generation. In Proceedings of the computer vision and pattern recognition, 2023.
conference
Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang. Drivinggaus- sian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallera Recognition, pp. 21634 21643, 2024.
Jun-Yan Zhu, Tacsung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation us- ing cycle-consistent. adversarial networks. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2017.
Sijie Zhu, Taojiannan Yang, and Chen Chen. Vigor: view image geo-localization beyond one- one-to-one retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021.
17


Reviews:
[{"review_id": "Mq4GkxQJeT", "reviewer": "Reviewer_qnV2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The proposed method effectively combines multiple input conditions to generate consistent target panoramas.\n- The geospatial attention module appears to successfully learn and capture accurate spatial relationships across locations.\n\nWeaknesses:\n- The framework is overly complex, and many modules are not well-explained or justified. There’s no clear evidence showing that these design choices outperform simpler alternatives.\n- Key ablation studies and comparisons with competitive baselines are missing.\n- The paper does not clearly demonstrate the advantages of mixed-view panorama synthesis compared to using only single-condition inputs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles the problem of generating a new panorama using both a satellite image and a few input panoramas. Unlike previous work that uses only one of these inputs, the method combines both sources of information.\nThe authors adopt a ControlNet-style module to incorporate all the input conditions into the generation process.\nThey introduce a complex attention extraction module designed to balance information from different geolocations accurately.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add ablation studies comparing other ways of injecting conditions into the model to validate the proposed method. Suggested baselines include:\n- Concatenating conditions along the channel dimension and training a ControlNet to guide generation.\n- Concatenating conditions and noise along spatial dimensions, followed by fine-tuning the diffusion model.\n- Using simpler architectures, such as pure convolution or pure attention layers, to handle the conditioning.\n2. How is the segmentation map used during training? It is not shown in Figure 3 or Figure 5. Given its importance (as suggested by Table 5), why doesn’t this create a training–inference mismatch?\n\n3. Include comparisons with other diffusion-based models to better demonstrate the advantages of your approach.\n\n4. In Figure 2, “extended prompts” appear in both the Geospatial Attention Extractor and the diffusion model. However, the paper does not explain what they are or how they are used. Please add a detailed explanation.\n\n5. Why does the copied encoder also take noise as input? Please explain the reasoning behind this design.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bqoZwwpL49", "reviewer": "Reviewer_SqTH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The task of mixed-view panorama synthesis that uses a satellite image and a sparse set of nearby panoramas looks interesting. It also shows the potential to apply to diverse scenes.\n+ A geospatial attention as the geometry constraint is proposed to achieve geometry-guided fine-grained spatial control.\n+ The paper is easy to follow, and the organization is well structured.\n\nWeaknesses:\n\n- While the proposed new task looks interesting and practical, the technical contributions presented in this work seem to be somehow weak. For example, the proposed local attention and global attention extractions are similar to normal attention operations, and some of them are derived from previous works. Besides, the proposed multi-conditioned diffusion model also shares similar processes with ControlNet. The authors are suggested to highlight the technical contribution of this work.\n- This work targets panorama synthesis; however, the specific properties of panoramic images have been ignored. For instance, the panorama shows dramatic geometric distortions compared to the perspective images. How did the authors ensure the generated results kept this property? It might be improper to evaluate the generated panoramas only using the metrics designed for perspective images, such as PSNR and SSIM. Could we propose a new metric to specifically evaluate this geometric distortion? I believe some generated panoramas would obey the accurate geometric distortion distribution while looking reasonably at the appearance level.\n- In addition to the above concern, the property of seamless content in generated panoramas has also been omitted. Namely, the panorama can be stitched from left to right boundaries without a noticeable edge effect. Did the authors leverage the cylinder convolutions like previous panoramic vision works? For example, \"Cylin-Painting: Seamless 360° Panoramic Image Outpainting and Beyond\" and \"PanoDiffusion: 360-degree Panorama Outpainting via Diffusion\" use a cylinder-like convolution or circular padding to ensure the seamless content generation of panoramas. The brief reviews and discussions of these works are also expected to be presented in this work.\n- Some previous works propose specific metrics to evaluate the panorama model. For example, \"PanoFormer: Panorama Transformer for Indoor 360 Depth Estimation\" proposes the Left-Right Consistency Error (LRCE) to measure the consistency of the left-right boundaries by calculating the horizontal gradient. The experiments should add this metric for generated panorama evaluations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces the task of mixed-view panorama synthesis, which aims to synthesize a novel panorama given a small set of input panoramas and a satellite image of the area. It supports panorama synthesis for arbitrary locations worldwide. To achieve this, the authors propose a diffusion-based modeling and an attention-based architecture. Experimental results demonstrate the superiority of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the above Weaknesses for more details.", "broader_impact_concerns": "The method may be misused to generate deceptive or misleading geographic content. Biases or inaccuracies in AI-generated panoramas could impact urban planning and environmental assessments. Privacy concerns also arise if sensitive geospatial imagery is used without authorization.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TnHuGVT6Yq", "reviewer": "Reviewer_nsWF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The work has a strong motivation for the proposed problem setting. Using satellite imagery to address the lack of dense panoramas for most regions, and using potential available panoramas for leveraging all the available data. \n+ The experiments are thorough. \n\n? In Fig 6, the Sat2Density results for the center-aligned scenario seem a lot worse that the proposed method. However, in Table 1, the results are almost the same between the proposed method and Sat2Density. Why is this the case?\n\n- The writing could be significantly improved by cutting the long sentences into smaller ones. The paper is full of very long sentences which is hard for me as a reader to comprehend on the first pass. For example, this one: “To generate the target panorama Pt with precise layout distribution, the synthesis process should utilize the semantic information from P1, P2, · · · , Pn and S1, as well as the geometric relationships between street-level Pi → Pt, i ∈ (1, 2, · · · , n) and across satellite & street-level S1 → Pt.”. which spans almost 3 lines. \n- The naming of the global geospatial attention is not the best in my opinion. I struggled to understand that whole section and had to read it back and forth multiple times, as I was expecting some sort of attention mechanism from the transformers literature. It however ended up being a differentiable weighting function. \n- Certain design choices are not well-motivated. For example, the reason behind using the Frobenius inner product, or using the Hadamard product for fusion. These choices which are not necessarily widely used in the diffusion literature need to be backed with intuition or citations or thorough ablations. \n- Some citations are not necessarily to the right papers. For example, consider this sentence: “Recently, conditional diffusion models Rombach et al. (2022) have become the state-of-the-art approach in the image synthesis task”. The paper Robach et al. (2022) is the paper introducing the latent-diffusion models, but wasn’t introducing conditional diffusion models. \n- ControlNet variants with multiple control signals have been widely used in the literature. I believe it is a bit of stretch for the authors to claim they introduce “a novel multi-conditioned end-to-end diffusion framework”. The authors use the same architecture in the context of view synthesis from satellite and panorama imagery. \n- The design of the figures could improve significantly. There is a lot of white space in the figures (e.g., Figure 4). The font family and font size of the figures don’t match the body of the paper. The font and style is not even consistent between different figures (notice Fig. 6 and Fig. 7). These are clear signs that more care could be put in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new task of synthesizing panorama images using mixed input signals. They also propose a diffusion-based approach via attention mechanisms as a potential solution to the proposed problem setting. The inputs are a combination of input panoramas and satellite imagery. The mixed use of inputs is important as not always we have dense panoramas for all points on earth, and only considering satellite imagery as inputs neglects the potential availability of sparse panoramas which is dense in information about textures and geometry of the area.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper has strong motivation and thorough experiments. However, several areas need improvement. First, many sentences are too long, making it hard to follow. Shorter sentences would improve readability. Second, the naming of \"global geospatial attention\" is unclear and misleading, as it resembles transformer-based attention but is actually a differentiable weighting function. Third, some design choices, like using the Frobenius inner product and Hadamard product, lack proper justification or citations. Fourth, some references are inaccurate, such as citing Rombach et al. (2022) for conditional diffusion models. Fifth, the claim of introducing a novel multi-conditioned diffusion framework is exaggerated, as similar approaches exist in ControlNet variants. Lastly, the figures need better design, with less white space, consistent fonts, and uniform styles. Also, the Sat2Density results in Fig. 6 seem much worse than the proposed method, but Table 1 shows similar performance. Clarification on this discrepancy would be helpful.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all agreed that the paper warranted publication.  A number of minor issues around presentation and additional ablations arose during the discussion which should be incorporated into the final paper.
Decision: Accept (poster)