MVPbev: Multi-view Perspective Image Generation from BEV with Test-time Controllability and Generalizability

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 03:38:27

Abstract: This work aims to address the multi-view perspective RGB generation from text prompts given Bird-Eye-View(BEV) semantics. Unlike prior methods that neglect layout consistency, lack the ability to handle detailed text prompts, or incapable of generalizing to unseen view points, MVPbev simultaneously generates cross-view consistent images of different perspective views with a two-stage design, allowing object-level control and novel view generation at test-time. Specifically, MVPbev firstly projects given BEV semantics to perspective view with camera parameters, empowering model to generalize to unseen view points. Then we introduce a multi-view attention module where special initialization and de-nosing processes are introduced to explicitly enforce local consistency among overlapping views w.r.t. cross-view homography. Last but not the least, MVPbev further allows test-time instance-level controllabity by refining a pre-trained text-to-image diffusion model. Our extensive experiments on NuScenes demonstrate that our method is capable of generating high-resolution photorealistic images from text descriptions with thousands of training samples, surpassing the state-of-the-art methods under various evaluation metrics. We further demonstrate the advances of our method in terms of generalizability and controllability with the help of novel evalution metrics and comprehensive human analysis. Our code and model will be made available.

Main paper: 
 related work: 
 Image editing [20] and generation [24] are heated topics in computer vision. Though this can be related to a vast literature, we would focus on two lines of work, conditional image generation and novel view image synthesis, as they are closely relevant.Conditional image generation Generative models, e.g., Gaussian Mixture model [23] and Bayesian network [11] , have been a long-term research problem in machine learning and computer vision as it is able to explain complex data distribution. In particular, generative models of images are not only of great importance for unsupervised feature learning, but also enable applications such as image editing [13, 20] . With the rise of deep learning techniques, such as auto-regressive models [3] , variational autoencoders (VAEs) [14] , and generative adversarial networks (GANs) [10] , as well as the emerging of huge amount of data [8] , we observe photorealistic images with very good quality. Among them, conditionalGANs have been well explored where various constraints, including discrete labels, text, and images, are considered. More recently, stable diffusion models [24] are widely used to generate detailed images conditioned on text descriptions. Compared to the prior art, they not only demonstrate SOTA image generation quality, but also showcase great generalizability with the help of foundation models [15] . Later on, Controlnet [34] Novel view image synthesis There are two broad categories in which the novel view synthesis methods can be divided into geometry-based and learning-based approaches. The former tries to first estimate (or fake) the approximate underlying 3D structures, followed by applying some transformation to the pixels in the input image to produce the output [1, 36] . The latter, on the other hand, argues that novel view synthesis is fundamentally a learning problem, because otherwise it is woefully under-constrained. More recently, neural radiance fields (NeRF) [18] , which belong to the second category, have shown impressive performance on novel view synthesis of a specific scene by implicitly encoding volumetric density and color through a neural network. Starting from small-scales [18] , scene-level NeRFs, such as Block-NeRF [28] , are also proposed such that important use-cases, e.g., autonomous driving [2] and aerial surveying [12] are enabled by reconstructing large-scale environments. In contrast, our method takes the input as BEV semantics and text description and outputs multi-view perspective RGB. 
our method: 
 Our method aims to generate multi-view perspective images from text prompts given pixel-level BEV semantic correspondences. Specifically, we denote the BEV semantics as B ∈ R 𝐻 𝑏 ×𝑊 𝑏 ×𝑐 𝑏 , with the ego car assumed to be located at the center. And 𝐻 𝑏 , 𝑊 𝑏 , and 𝑐 𝑏 are the height, width, and number of semantic classes of B, respectively. Our goal is to generate a set of perspective RGB images with resolution 𝐻 by 𝑊 , or {𝐼 𝑚 } 𝑚 in particular, under 𝑀 virtual camera views. And the 𝑚-th perspective image is referred to as 𝐼 𝑚 ∈ R 𝐻 ×𝑊 ×3 where 𝑚 = {1, . . . , 𝑀 }. In particular, we assume the intrinsics, extrinsic rotation, and translation of the 𝑚-th camera are given and denote them as 𝐾 𝑚 , 𝑅 𝑚 , and 𝑇 𝑚 , respectively.As described above, we obtain visually coherent multi-view images by leveraging both global and local consistency in implicit and explicit manners. Specifically, our method consists of two stages. Our first stage takes BEV semantics B as well as {𝐾 𝑚 , 𝑅 𝑚 ,𝑇 𝑚 } 𝑚 as input and projects BEV semantics to each perspective view w.r.t. its camera parameter set, denoting as 𝑆 𝑚 ∈ R 𝐻 ×𝑊 ×𝑐 𝑏 for the 𝑚-th view. The second stage parses {𝑆 𝑚 } 𝑀 𝑚=1 and text prompts as input. And it produces RGB images from 𝑀 perspective views. 𝐼 𝑚 denotes the generated RGB image from 𝑚-th view. More specifically, our first projection stage enforces global semantic consistency between BEV and perspective view explicitly with the help of geometric transformation. Meanwhile, the generation stage imposes consistency implicitly among overlapping perspective views with a multi-view attention module. Finally, we propose to explicitly enforce the visual cues at overlapping FOVs to be coherent with our novel training initialization and de-noising designs. The overall pipeline of MVPbev can be found in Fig. 2 . We provide more details of the first and second stages in Sec. 3.1 and Sec. 3.2 respectively. And the model training process is described in Sec. 3.3.Assuming that diverse yet plausible BEV semantics B can be obtained effortlessly with existing simulation methods [32] , the first fundamental problem that our method should address is to maintain cross-view semantic consistency from B to perspective images {𝐼 𝑚 } 𝑚 . Secondly, contents at overlapping FOVs should also be coherent. For example, not only the background classes, such as buildings or trees, but also the foreground road participants, should be of similar appearance when they appear in different views. To this end, we first propose to project BEV semantics to 𝑀 perspectives view with camera parameters, which generates {𝑆 𝑚 } 𝑀 𝑚=1 perspective semantics. Compared to existing work [34] , our projection step ensures semantic-wise consistency between BEV and perspective views with the help of geometry constraints, leading to fewer accumulative errors at the generation step. And our projection results can be found in Fig. 3 .Simply working on individual perspective semantics may lead to inconsistent content across different views, especially at overlapping FOVs. For instance, the buildings and the vegetation that appear at the FOVs among multiple views, e.g., the front, front-right, back, and back-left, have different appearances. This is due to the lack of interactions among cross-view cameras. We would like to note that such inconsistency would be reflected by neither BEV layout segmentation nor object detection metrics as it has influences on background classes only.Motivated by this, we propose to focus on these overlapping areas both methodologically and experimentally. As for our method, we apply strong coherency constraints on the background content by estimating the homography of overlapping areas, followed by a multi-view attention module to implicitly enforce the styles at various views to be coherent w.r.t. estimated corresponding points. In this case, appearance consistency can be enforced not only on the background layout areas where the semantics are provided, but also on the other regions where control signals are missing. As for the evaluation purpose, we introduce human analysis to provide reliable evaluations on whether the generated images, especially the overlapping regions, are realistic or not. We demonstrate that Homography estimation. We take the initial step towards enforcing visual consistency at overlapping FOVs by estimating the overlapping regions. To this end, we propose to compute the homography between images with overlapping FOVs. As illustrated in many driving datasets, one view generally overlaps with views on its left and right sides. Therefore, for the 𝑚-th view, we only need to consider 𝑚 𝑙 = mod (𝑚 +𝑀 -2, 𝑀) + 1 and 𝑚 𝑟 = mod (𝑚 + 1, 𝑀), which are the left and right views of the 𝑚-th view respectively. Then we estimate the homography from view 𝑚 𝑟 to view 𝑚 and denote the mapping function as 𝐻 𝑚 . Consequently, the 𝑝 = [𝑥, 𝑦] coordinate in the 𝑚-th view will be mapped to coordinate p = [ x, ŷ] in view 𝑚 𝑟 . Or p = 𝐻 𝑚 (𝑝). Similarly, we define an inverse mapping 𝐻 𝑚 that maps p in 𝐼 𝑚 𝑟 to 𝑝 in 𝐼 𝑚 .Multi-view attention module. What makes a set of views unrealistic? The first and foremost thing is the incoherence among images. In other words, realistic ones must appear consistent, as if they were taken in the same physical location at the same time of a day. More specifically, the visual styling of this set of images needs to be consistent such that all of them appear to be created in the same geographical area (e.g., urban vs. rural), time of day, with the same weather conditions, and so on. To this end, we introduce a multiview attention module such that when generating the RGB from the 𝑚-th view, the views on its left and right sides are considered. For a token located at position 𝑝 in the feature map 𝐹 𝑚 generated As can be found in Eq. 1, our multi-view attention module aims to aggregate information from the target feature pixels 𝐾 ( p) to 𝑝. We provide a simple illustration of our multi-view attention module in Fig. 4 .To train our model, we introduce a multi-view Latent Diffusion Models (LDMs) [24] loss. Basically, the original LDMs consist of a variational autoencoder (VAE) with encoder E and decoder D, a denoising network 𝛿 𝜃 and a condition encoder 𝜏 𝜃 . The input image 𝐼 𝑚 is mapped to a latent space by l 𝑚 = 𝜀 (𝐼 𝑚 ), where l 𝑚 ∈ R ℎ×𝑤×𝑐 . We follow the routine to set 𝐻 ℎ = 𝑊 𝑤 and they both equal to 8. Later on, the latents will be converted back to the image space by Ĩ𝑚 = D (l 𝑚 ). The denoising network 𝛿 𝜃 is a time-conditional UNet, which leverages cross-attention mechanisms to incorporate the condition encoding 𝜏 𝜃 (c). In our case, c consists of text-prompt and semantics in perspective view 𝑆 𝑚 .For each training step, we firstly uniformly sample a shared noise level 𝑡 from 1 to 𝑇 for all the multi-view images {𝐼 𝑚 } 𝑀 𝑚=1 , denoting them as {𝜖 𝑡 𝑚 } 𝑚 . And 𝜖 𝑡 𝑚 ∼ N (0, 1). To leverage the cross-view consistency, we further enforce these noises to be the same if they correspond to the same pixel. Starting from the first view, or 𝑚 = 1, we re-assign value at coordinate 𝑥, 𝑦 of 𝜖 𝑡 𝑚 , or 𝜖 𝑡 𝑚 (𝑥, 𝑦), to 𝜖 𝑡 𝑚 𝑟 ( x, ŷ). And we repeat the process until 𝑚 𝑟 = 1. We provide one example set of our initialized {𝜖 𝑡 𝑚 } 𝑀 𝑚=1 in Fig. 5 . Finally, our model training objective is defined as:EQUATIONwhere 𝛿 𝑚 𝜃 is the estimated noise for the 𝑚-th image. And we use 𝑙 𝑡 𝑚 to denote the noisy latent for the 𝑚-th image.At sampling time, the denoising (reverse) process generates samples in the latent space, and the decoder D produces RGB images with a single forward pass. To incorporate our idea that pixels at overlapping regions should be visually similar even at different views, we again employ the value assignment process. Similar to the noise initialization step, we re-assign value at coordinate 𝑥, 𝑦 of 𝑙 𝑡 𝑚 , or 𝑙 𝑡 𝑚 (𝑥, 𝑦), to 𝑙 𝑡 𝑚 𝑟 ( x, ŷ). This re-assignment starts from 𝑚 = 1 and does not stop until 𝑚 𝑟 equals to 1. In experiments, we observe that our design would improve the visual results if applied to up to 3×𝑇 5 denoising steps. Otherwise, the performance deteriorates. Inference As claimed in above, MVPbev can be extended with instance-level controllability. Specifically, Our MVPbev allows the user to click on the target instance and provide color-specific requirements. To achieve this, we propose a special mechanism for multiple foreground objects control, which manipulates the response of cross-attention layers to accurately guide instance-level synthesis. Assuming that instance-level masks can be obtained at each view with either existing methods [5] or simple retrieval. Specifically, we first obtain the instance-level and scene-level latents individually with its paired prompt. Then they are effectively combined with those binary instance-level mask, leading to more spatial-consistent performance. Please note that such ability on foreground objects of MVPbev are training-free, leading to far better extendability and test-time controllability. We refer the readers to supplementary for more details. 
experiment 4.1 experiment setup: 
 Dataset We validate our ideas on NuScenes dataset [2] where full 360-degree coverage provided by six cameras is available. Specifically, it consists of 1000 examples of street-view scenes in Boston and Singapore, each of which lasts for 20s and is captured at 12Hz. Besides 1.4M camera images, NuScenes also provides multi-modal data, including both global map layers and 3D object bounding boxes annotated on 40k keyframes. We follow the standard split of 700/150/150 for training, validation, and testing. We report our results on the validation set of Nuscenes and follow the split of [2] where 600 sets of images are used. Evaluation metrics We follow the design of [29] to include image quality of generated images as well as their visual consistency in our evaluation metrics. In addition, semantic consistency is also valued in our metrics as it reflects the synthesis quality of different semantic categories.• Image quality is measured by Fréchet Inception Distance (FID) [6] ,Inception Score (IS) [25] , and CLIP Score (CS) [22] . In particular, FID is based on the distribution similarity between the features of the generated and real images. The IS measures the diversity and predictability of generated images. Finally, CS measures the text-image similarity according to pre-trained CLIP models [22] .• Visual consistency provides pixel-wise similarity measurements on overlapping regions. We borrow the idea from Peak Signal-to-Noise Ratio (PSNR) where we first compute this PSNR between all the overlapping regions, and then compare this "overlapping PSNR" for ground truth images and generated images. The higher this value is, the better visual consistency will be. Note that the process of computing "overlapping PSNR" is based on estimated homography matrices, it's possible that generated image yields higher values than ground truth image. • Semantic consistency measures the pixel-wise semantic consistency between generated images and ground truth. In our case, we utilize the Intersection-over-Union (IoU) score. Particularly, we report the semantic IoU both in perspective view and BEV. As for the former, we apply pre-trained segmentation model [5] on generated images, leading to semantic predictions in perspective view. These predictions are then compared with {𝑆 𝑚 } 𝑚 to obtain IoU in perspective view. As for the latter, we apply pre-trained CVT [35] to generated images and the BEV IoU is obtained by comparing predictions from CVT with B. • Object-level controllability measures how accurately the objectinstance is generated w.r.t. test-time descriptions. Here we report the averaged color distance Delta-E in CIELAB color space as well as their standard deviations. Besides these metrics, we also perform human analysis. We request humans to decide which method is more visually realistic and consistent when results from different methods are provided. Please note that method is anonymous to humans and when compared, we ensure that the same input control signals are provided to various methods. Meanwhile, we also conduct experiment with instancelevel controllability. Humans are provided with objects as well as their targeted color, paired with the generated images. And they will vote whether the generated objects meet the requirements. Baselines We select the following two state-of-the-art methods as our baselines for thorough comparisons:• SD+ControlNet [24, 34] Our BEV semantics B reflects an 80m × 80m space with ego car located at the center position. 𝑐 𝑏 represents the drivable area in NuScenes. The resolution of the perspective image is 𝐻 ×𝑊 = 256× 448, leading to ℎ × 𝑤 = 32 × 56. As for the hyper-parameters, we set 𝑀 and 𝐾 to 6 and 3 respectively. We have implemented the system with PyTorch [21] while using publicly available Stable Diffusion codes [31] . Specifically, it consists of a denoising UNet to execute the denoising process within a compressed latent space and a VAE to connect the image and latent spaces. The pre-trained VAE of the Stable Diffusion is maintained with official weights and is used to encode images during the training phase and decode the latent codes into images during the inference phase. In experiments, we use a machine with 1 NVIDIA A40 GPU for training and inference. Batch size is set to 6 and 𝑇 equals to 50. We refer the readers to supplementary for full implementation details.We compare our MVPbev with baselines and report the performance in Table 1 . The first row in this table is obtained on ground truth images. For instance, we split the ground truth validation images into two halves, and then obtain the FID score by taking one split as ground truths and the other half as generated images. As for the IoU scores, we apply the Mask2Former [5] and CVT [35] on validation images and compare their predictions with ground truths We also provide visual comparisons to existing baselines in Fig. 6 . The top row showcases the bev semantics B as well as the projected semantics {𝑆 𝑚 } 𝑀 𝑚=1 in perspective views. We also provide the ground truths as well as multi-view images generated by other methods from the second to the fifth row. Our method MVPbev, compared to other baselines, produces the most consistent perspective images, especially at overlapping FOVs. As highlighted by orange bounding boxes and green lines, our MVPbev not only generates perspective images that are consistent with semantic guidance, but also maintains high visual consistency across multiple views. Such consistency is more visible and valuable for pixels that appear in different views. Qualitative results Besides quantitative results, we also provide qualitative examples in Fig. 7 . As can be found in this figure, our MVPbev is able to generate visually consistent images from diverse bev semantics and text prompts. Compared to ground truth, ours can obtain satisfactory consistency at overlapping FOVs. We refer the readers to supplementary for more visual examples about the controllability over BEV and text prompt.Test-time controllability and generalizability. View-point generalizability As described before, one of the main drawbacks of the existing work is the lack of ability in terms of handling viewpoint changes at test time. To showcase our ability, we revise the camera extrinsic during inference and check whether the results would change accordingly. In practice, we rotate the all 𝑀 camera by {-25 • , -15 • , -5 • , 5 • , 15 • , 25 • } w.r.t. the head direction of the ego car. mimicking potential different setups of camera mounting. This is equivalent to changing the {𝑅 𝑚 } 𝑚 in our input signal. We randomly generate 200 sets of images for each rotation angle and provided the generated results from MagicDrive [9] and ours to humans. Qualitative results are provided in Fig. 9 . We overlay the projected semantics in each view for better visualization. Not surprisingly, prior art merely follows the control signal. While MVPbev gives superior results considering the semantics, demonstrating better test-time generalizability. And this observation is also supported by our detailed human analysis in Table 1 . Object-level controllability A practical generative model should be a controllable one. To this end, we conduct another experiment to showcase the object-level controllability. In this experiment, we include an additional description of the object color in the original text prompt and then check whether such control can be reflected in generated scenes at test time. In experiment, we randomly choose 151 set of images, including 195 object instances, and provide random color requests out of seven popular colors for vehicles. We report our qualitative evaluations in Fig. 10 and qualitative examples in Fig. 8 respectively. Though the Delta-E seems to be noticeable, we argue that this is mainly due to the de-noising process where colors of vehicles are in harmony with the environment, e.g., less bright in rainy days. This is supported by our visual results as well as human analysis.Human analysis. Compared to evaluation metrics, human analysis provides a more reliable tool for image quality measurement. Therefore, we conduct a comprehensive human analysis of our tasks. Specifically, we provide two sets of generated images, which are generated from two different methods with the same input signal, to humans. Then we ask them to decide which set of images is better, considering the image quality and visual consistency. As can be found in Table 1 , our MVPbev outperforms baselines significantly, indicating that we can indeed generate photo-realistic yet consistent images. Meanwhile, we report the test-time viewpoint changes by comparing ours to MagicDrive [9] , showing that MVPbev provides better generalizability quantitatively. Finally, we ask humans to decide whether the generated instance color can be regarded as the requested one. In our experiment, 93.5% of the instances are voted as correctly generated. We refer the readers to supplementary for more details about human analysis. 
conclusion: 
 Our goal is to generate multi-view perspective RGB from text prompts given BEV semantics. To this end, we introduce a two-stage method MVPbev to first project BEV semantics to perspective views and then perform image generation w.r.t. both text prompts and individual perspective semantics. Specifically, we propose a novel initialization and denoising processes to explicitly enforce local consistency at overlapping FOVs. Results showcase the superiority of MVPbev under various metrics and test-time generalizability.  
: 
 provides a tangible interface to the real world, thus is beneficial and practical for higher-level modeling and decision making [16, 17] .Though being of great importance in autonomous driving tasks, reliable BEV representation requires a large amount of data during the training stage, which can be time-consuming to obtain or annotate. One intuitive solution to this data issue is to obtain diverse perspective RGB images as well as their corresponding BEV semantics with generative models. Diverse yet plausible BEV semantics, compared to their corresponding perspective RGB or semantics, are much easier to simulate in a realistic manner with the help of parametric representations [32] . To this end, it is natural and practical to assume that BEV semantics, rather than perspective RGB images, are given. Then the remaining question is to generate cross-view visually and semantically consistent photorealistic RGB images with known BEV semantics.Despite the progress of generative models with given constraints [34] , there are three main drawbacks in existing attempts to address this cross-view image generation problem [9, 27, 29] . Firstly, existing frameworks rely heavily on the training samples, leading to unsatisfactory test-time controllability. For instance, changing camera poses or providing extra control on object instance is beyond prior art. Moreover, cross-view consistency is not well enforced, resulting in inconsistent visual effects in overlapping FOVs. Finally, no thorough human analysis is performed on image generation tasks, resulting in un-interpretable comparison results.To this end, we propose a novel two-stage method MVPbev that aims to generate controllable multi-view perspective RGB images with given BEV semantics and text prompts by explicitly enforcing cross-view consistency (See Fig. 1 .a). Unlike existing work that lacks the test-time generalizability, MVPbev further allows both view-point and detailed text prompt changes at test-time, providing satisfactory performances under human analysis without requiring additional training data. To achieve that, MVPbev consists of two stages, or view projection and scene generation stages. The former stage transforms the given BEV semantics to multiple perspective view w.r.t. camera parameters. On the one hand, it enforces global consistency across views with explicit geometric transformation. On the other hand, such a design decouples the two stages, allowing the second stage to better capture view-point-invariant properties. The second stage of MVPbev starts from a pre-trained stable diffusion (SD) model. By explicitly incorporating a crossview consistent module, together with our noise initialization and de-noising processes design, it can produce multi-view visually consistent and photo-realistic images, especially at overlapping FOVs. To further improve the test-time generalizability on objects, our MVPbev handles foreground instances and background layout individually, leading to better controllability during inference.We validate our ideas on NuScenes [2] and follow the standard split. In contrast to methods that focus on improvements over downstream tasks or semantic consistency, we include additional extensive human analysis, especially on visual consistency over overlapping FOVs, across multiple views, and test-time view point and text prompt changes. We demonstrate that our proposed method not only provides better test-time controllability and generalizability, but also gives high-quality cross-view RGB images.In short, our contribution can be summarized as follows: • A novel multi-view image generation method that is capable of producing semantically and visually consistent perspective RGB images from BEV semantics, with only thousands of images as training data. • A more controllable yet extendable algorithm that gives realistic perspective RGB images. • State-of-the-art performances on large driving datasets, with comprehensive human analysis. 
 
 Appendix:  


Decision: The paper receives two borderline accept and two weak accept. It is a clear acceptance case.

Reviews: review: paper_summary: paper_summary: This paper introduces a method called MVPbev, which generates consistent multi-view perspective images from bird's-eye view (BEV) semantics and text prompts through a two-stage design. This method addresses the shortcomings of existing methods in terms of viewpoint variation and detailed control. Its superior generation performance and control during testing have been validated through multiple evaluation metrics and manual analysis on the NuScenes dataset.
strength_weakness: strength_weakness: 1. The novelty is sufficient, focusing on generating multi-view images for real-time autonomous driving.
2. The writing and diagrams are easy to understand.
3. The main experiments are detailed and compare many existing algorithms for autonomous driving scene generation.
limitations: limitations: Lacks an ablation study for modules such as the Multi-view attention module and Homography estimation.
paper_summary: paper_summary: The paper is about a method named MVPbev, which stands for Multi-view Perspective Image Generation from Bird's-Eye-View (BEV) with Test-time Controllability and Generalizability. The MVPbev method introduces a two-stage design. The first stage projects the given BEV semantics into perspective views using camera parameters. The second stage involves a multi-view attention module with special initialization and denoising processes to enforce local consistency among overlapping views with respect to cross-view homography.
strength_weakness: strength_weakness: 1 The introduction of specific initialization and denoising processes to enforce local consistency at overlapping fields of view (FOVs) is an innovative solution to a common challenge in multi-view image generation. By using a multi-view attention module and enforcing consistency through geometric transformations and homography estimation, the method ensures that overlapping regions in different views are visually coherent.

2 Test-Time Instance-Level Controllability: MVPbev enables changes in viewpoint and instance-level detailed text prompts at test time without the need for additional training data, providing greater flexibility and adaptability.
limitations: limitations: 1 It lacks comparative experiments with Controlnet and MVD in terms of controllability. 
2 Since the title includes the Test-time Controllability, then how to do it should be placed in the main text. Can this mechanism only control colors effectively? It seems that this mechanism can also be applied to ControlNet.
3 It seems that there is no specific mechanism to achieve Test-time generalizability.
paper_summary: paper_summary: The paper proposes a novel multi-view image generation method (MVPbev) to address the multi-view perspective RGB generation from text prompts given Bird-Eye-View(BEV) semantics.
strength_weakness: strength_weakness: MVPbev projects the given BEV semantics to perspective view with camera parameters, empowering the model to generalize to unseen viewpoints.  MVPbev introduces the multi-view attention module where special initialization and denoising processes are introduced to explicitly enforce local consistency among overlapping views w.r.t. cross-view tomography. The method further allows test-time instance-level controllability by refining a pre-trained text-to-image diffusion model. The method obtians state-of-the-art performances on large driving datasets.
limitations: limitations: (1) The analysis of ablation experiments is not exhaustive.
(2) Some parameter symbols are not clearly explained：parameter c of Eq(2).
(3)The experimental results need to explain in detail: The author did not thoroughly analyze the experimental results of table 1, nor did they explain the advantages of their method. In Fig 10, the author also did not explain and describe the content.
(4) Some comparative methods （MVD, Controlnet）have not been tested on this specific dataset; the author should provide the settings for these methods on this dataset, including parameters, experimental environment, etc.
paper_summary: paper_summary: The paper proposes a novel two-stage method MVPbev that aims to generate controllable multi-view perspective RGB images with given BEV semantics and text prompts by explicitly enforcing cross-view consistency. Specifically, it consists of view projection and scene generation stages. For the view projection stage, it projects BEV semantics to perspective view with camera parameters to maintain global semantic consistency. For the scene generation stage, it parses both perspective semantics and text prompts, and generates multi-view images with both visual consistency and test-time instance-level control by explicit enforcing in latent. Through extensive experiments on NuScenes, the MVPbev method demonstrates better test-time controllability and generalizability, surpassing the state-of-the-art methods under various evaluation metrics.
strength_weakness: strength_weakness: 1. Overall: The paper proposes a novel multi-view image generation method MVPbev to produce semantically and visually consistent perspective RGB images from BEV semantics, with only thousands of images as training data. The controllability and generalizability of the MVPbev method are evaluated through extensive experiments through extensive experiments on NuScenes, surpassing the state-of-the-art methods under various evaluation metrics.
2. Better Test-time Generalizability: As for conditional image generation, existing works can’t achieve test-time generalizability, e.g., view-point changes or detailed instance-level text prompts. In contrast, the proposed method exploits both global and local consistency, together with a training-free objects control method to enforce detailed instance-level control. 
3. Robust Evaluation: Existing works of conditional image generation don’t conduct human analysis on image generation quality. In contrast, the paper provides comprehensive human analysis to demonstrate the effectiveness of the proposed method in a more reliable manner.
limitations: limitations: 1. Insufficient Analysis of Experimental Results: Specifically, there is no thorough examination of cases where the proposed method MVPbev failed to achieve state-of-the-art results when compared against methods like BEVGen and MagicDrive. A detailed exploration of the factors leading to these discrepancies is essential to fully understand the limitations and identify potential areas for improvement in the proposed method.
2. Need for Analysis of Imperfect Sample Cases: While the paper includes visual comparisons with existing baselines in Figure 6 and qualitative examples in Figure 7, it only mentions that the MVPbev produces the most consistent perspective images, particularly at overlapping fields of view (FOVs). However, I have observed noticeable inconsistencies between some images generated by the proposed method and the ground truth, which are not adequately analyzed by the authors. Incorporating such an analysis would be beneficial in demonstrating the method's limitations and suggesting areas for improvement.
3. Lack of Ablation Studies: The paper lacks a dedicated section or discussion on ablation studies, which makes it challenging to assess the effectiveness of various components like the multi-view attention module, special initialization, training-free objects control method, and the denoising processes. Without individual validation of these elements, it is difficult to determine the contribution of each to the overall performance of MVPbev.
4. Language and Grammar Errors:
Line 103: “the remaining question is to generate cross-view…” -> “the remaining question is how to generate cross-view…”
Line 331: “lack of interactions” -> “lack of interaction”