Selective LoRA for Domain-Aligned Dataset Generation in Urban-Scene Segmentation

Authors: Minho Park, Sunghyun Park, Jungsoo Lee, Hyojin Park, Kyuwoong Hwang, Fatih Porikli, Jaegul Choo, Sungha Choi
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Dataset Generation, Urban-scene Segmentation

Abstract:
This paper addresses the challenge of data scarcity in semantic segmentation by generating datasets through fine-tuned text-to-image generation models, reducing the costs of image acquisition and labeling. Segmentation dataset generation faces two key challenges: 1) aligning generated samples with the target domain and 2) producing informative samples beyond the training data. Existing methods often overfit and memorize training data, limiting their ability to generate diverse and well-aligned samples. To overcome these issues, we propose Selective LoRA, a novel fine-tuning approach that selectively identifies and updates only the weights associated with necessary concepts (e.g., style or viewpoint) for domain alignment while leveraging the pretrained knowledge of the image generation model to produce more informative samples. Our approach ensures effective domain alignment and enhances sample diversity.
We demonstrate its effectiveness in generating datasets for urban-scene segmentation, outperforming baseline and state-of-the-art methods in in-domain (few-shot and fully-supervised) settings, as well as domain generalization tasks, especially under challenging conditions such as adverse weather and varying illumination, further highlighting its superiority.

Decision: reject
Reviewer scores: 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730558719252

thread_type:
official_review

<thread_object>
creator: reviewer_kfpj

<content>
summary:
This paper proposes a novel approach to address the data scarcity in semantic segmentation by generating datasets (image-mask pairs) using text-to-image models. To solve the issue, it is necessary that the generative images align with the target doman and provide useful information beyond the training dataset. Therefore, this paper intorudces Selective LoRA, a finetuning approach for the pretrained text-to-image model that preserves the distributional diversity of the original pretrained model while aligning with the target domain. The proposed method selectively update weights for key concepts, such as style and viewpoint, which need to be aligned or to maintain the diversity. The authors shows that the proposed method generates datasets with desired distribution via ablation studies and improves in both in-domain and domain generalization settings.

strengths:
The proposed method differentiate the weights to align with target domain while preserving valuable information of the pretarined model. From this, the proposed method selectively fine-tunes the model. This approach effectively addresses challenges when to adapt large pretrained models to different domains, enabling better domain specific performance without losing the benefits of the pretrained features.

The data scarcity problem addressed in this paper is a critical issue not only for semantic segmentation but also for a variety of vision tasks. Therefore, the proposed technique for generating image and ground truth pair datasets can be considered a core technology in the advancement of deep learning.

weaknesses:
I believe that the proposed method tackles an important problem and offers a reasonable approach to addressing the challenges, which I highly commend. However, there are some weaknesses to consider.

First, the method relies on Stable Diffusion, trained on a large dataset. Although leveraging the distributional diversity learned by this pretrained model is the motivation behind the approach, it inherently sets an upper bound on the applicability of the proposed method based on the knowledge of the pretrained model. This is a fundamental limitation.

Defining the desired concepts, identifying the critical parts of the architecture where these concepts are expressed, and retraining the model are all highly manual processes that depend heavily on individual characteristics of target data . To define the desired concepts, the user of this method must analyze the distributions of the pretrained model and the target domain, identify the differing concepts, and guide the process with appropriate text prompts. Additionally, finding the associated weights and determining their importance requires experimental work, which lacks standardized criteria.

Generating images aligned with the desired distribution is crucial, but creating high-quality masks to accompany these images is equally important for semantic segmentation models. This aspect has not been sufficiently addressed. While the current method leverages intermediate features, there could be consideration of various other ways to generate masks from the images. Given that the proposed method utilizes a model trained on a large dataset, it might also be worth exploring the use of models like SAM (Segment Anything Model) for mask generation (of course, there are lots of candidates). I am not requring additional experiments using SAM. It woud be beneficial to analyze the quality of the masks for the generated images. 

From a presentation, the paper is challenging to read. Figures 2 and 3 are difficult to understand, and it is not easy to infer the intended meaning from the related sections in the text. Additionally, the paper mentions L_Concept, but the figures use terms like L_style and L_viewpoint, which are not defined in the main text, causing confusion. The authors should clarify this and revise the figures accordingly. More detailed explanations about the process of creating text prompts are also necessary.

The experimental setup, including the ablation study, is not sufficiently explained. For instance, in experiments like those in Table 3, it is unclear how extensive the generated dataset is and how it is used.

Overall, the paper needs to be written in a way that is easier to understand.

questions:
I believe that the proposed method demonstrates sufficient technical novelty and shows effectiveness through its quantitative experimental results. However, I think it would be beneficial for the authors to revise certain aspects to make the paper easier to understand. Improving the clarity of the text would enhance the overall presentation of the work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730607322581

thread_type:
official_review

<thread_object>
creator: reviewer_kmen

<content>
summary:
This paper addresses the challenge of data scarcity in semantic segmentation by generating datasets through fine-tuned text-to-image generation models. Existing methods often overfit and memorize training data, limiting their ability to generate diverse and well-aligned samples. This paper proposes Selective LoRA that selectively identifies and updates only the weights associated with necessary concepts for domain alignment while leveraging the pretrained knowledge of the image generation model to produce more informative samples.
The authors demonstrate its effectiveness in generating datasets for urban-scene segmentation.

strengths:
1. The paper is well-written and easy to follow.
2. The idea of using concept loss to find important weights is reasonable.
3. The ablation studies and analytical experiments are interesting and inspiring.

weaknesses:
1. The proposed method outlines how to fine-tune LoRA for generating informative target-style images. However, the definition of "informative samples" is not clear. This lack of clarity may hinder the reader's understanding of the intended contributions. For example, Figure 1 would benefit from including examples of informative data to provide a clearer context for what constitutes an informative sample. 

2. In Figure 1(b), the results of the LoRA-finetuned images for both foggy and night-time conditions appear remarkably similar, suggesting that the fine-tuning process may not have effectively generated between these two target styles. It raises concerns about the method's capability compared to the pretrained approach. 

3. The proposed Selective LoRA generates images in a specific style and containing particular content, but it lacks a comparative analysis with existing text-driven diffusion models, such as Instruct-Pix2Pix. A comparison in terms of both the quality of generated images and adaptation performance would significantly enhance the paper's contributions and provide the reader with a clearer understanding of how the proposed method stands in relation to established techniques.

4. I find the results presented in Table 3 somewhat confusing. If I understand correctly, the baseline results are derived from fine-tuning Mask2Former using generated images, while RealFT represents the results from fine-tuning on real data. However, it is unclear how the authors obtained the labels for the generated data. Were these results obtained through an unsupervised training approach, or was an additional decoder trained similarly to the DatasetDM?

questions:
Additional minor questions are listed as follows.
 1.  The method uses selective LoRA to solve the problem of data scarcity in cross-domain segmentation. However, there are some similar methods to select LoRA weights in other fields, like LoRA-SP [3], GS-LoRA [2], Tied-LoRA [1] etc.. The authors should discuss these papers.  

[1] Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying  

[2] Continual Forgetting for Pre-trained Vision Models 

[3] LoRA-SP: Streamlined Partial Parameter Adaptation for Resource-Efficient Fine-Tuning of Large Language Models

2. The authors should compare more text-driven or image-driven generated dataset baselines, such as Instruct-Pix2Pix [4], PTDiffSeg [5], DATUM [6], etc.

[4] InstructPix2Pix: Learning to Follow Image Editing Instructions 

[5] Prompting Diffusion Representations for Cross-Domain Semantic Segmentation 

[6] One-shot Unsupervised Domain Adaptation with Personalized Diffusion Models
</content>
</thread_object>

</thread>
<thread>
creation date:
1730640098055

thread_type:
official_review

<thread_object>
creator: reviewer_hmnt

<content>
summary:
The paper presents a method for fine-tuning pre-trained T2I models to generate datasets specifically for urban-scene segmentation, addressing the challenge of data scarcity. Traditional methods often utilize pre-trained T2I models directly or apply LoRA for fine-tuning, which can lead to generated samples that fail to align with the target domain or lack diversity. To overcome these issues, the paper introduces Selective LoRA, a novel fine-tuning approach that selectively identifies and updates the weights that are most closely associated with specific concepts for domain alignment. This approach reduces the number of parameters that need training, improving training efficiency while ensuring that the original T2I model's generalizability is preserved. Extensive experiments demonstrate that the generated datasets improve the performance of previous segmentation models in urban-scene segmentation.

strengths:
S1: The paper "Selective LoRA" introduces a new training strategy for fine-tuning pretrained T2I models to generate diverse datasets for segmentation tasks, addressing the challenge of data scarcity.
S2: Extensive experiments show that the generated datasets enhance the performance of prior segmentation models in urban-scene segmentation.

weaknesses:
W1: The writing in this paper is somewhat challenging to understand, and the technical descriptions lack clarity, which can lead to confusion during reading. Below are a few examples, though not exhaustive. For instance, in Figures 3 and 4, what do the layer indices represent? Are they the projection layers for all attention in the network? However, according to the description in line 251, it seems that all linear layers in the network are being trained with LoRA. Additionally, Section 3 only covers the first two stages, with the third and fourth stages not being described in detail, making this part less clear. The structure of the experimental section is also somewhat disorganized. 

W2: The design of the tables lacks standardization, leading to confusion for the reader. Here are a few examples, though not exhaustive. For instance, many tables do not clearly explain what the numerical values represent, making interpretation difficult. In Table 3, the baseline names should be listed directly. Additionally, the entries under the "Data Ratio" column correspond to various methods, which creates some confusion. Furthermore, for the methods used to generate datasets that enhance baseline performance in Table 3, it would be clearer to label them as "Baseline + Real FT" rather than just "Real FT."

W3: Additionally, I noticed that the baseline appears to be from a 2022 paper. Are there any more recent baselines available for comparison?
W4: Some modules may not appear particularly novel from a technical perspective. LoRA are also commonly used in various papers.

questions:
Q1: From the design motivation, it seems that training all LoRA parameters may lead to overfitting, resulting in reduced diversity in the generated images. In contrast, Selective LoRA selects a subset of parameters that are most associated with the concepts, effectively training fewer parameters and better preserving the original T2I model's capabilities. The original LoRA setting applies training to all linear layers in the UNet with LoRA. I wonder if training LoRA only on certain layers' cross-attention (few parameters) could achieve a similar effect as Selective LoRA.

Q2: I hope the authors can address the concerns raised in the "Weaknesses" section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646223284

thread_type:
official_review

<thread_object>
creator: reviewer_b65c

<content>
summary:
The paper introduces a new approach to generate training samples with sepecific concepts variants. The proposed method learn specific concepts, such as style or viewpoint, by selectively manipulate the gradients. The method claim that it improve domain alignment and sample diversity.  In experiments, the method are compared with baseline and the DatasetDM, and results show improvements in in-domain, few-shot segmentation. The wide-scope experiments prove its practicability.

strengths:
1. The paper is well-organized, the motivation description is clear and concise. The pipeline figures are easy to understand.
2. The core idea is interesting, use language distinction to learn concept difference, and eliminate the requirement of paired visual data to learn specific concepts. I think the learning procedure is practicable. 
3. The experimental settings are extensive, including both in-domain, few-shot and damain generalization.

weaknesses:
1. The method mainly focus on introduce their Selective LoRA. However, the whole pipline includes training a label generator (stage 3 in Figure 2). The technical details in this part are not well delivered. How the label generator receive the intermediate features from T2I models and generate semantic maps?  In addition, in line 190-197, the authors say their use Mask2Former as label generator as same as DatasetDM. As I know, the DatasetDM use only "perceptual decoder" which only includes a decoder architecture instead of whole Mask2Former segmentaiton. Clarifying this distinction could provide a clearer understanding of the contributions of the current approach.
2. While the method aims for simultaneous sample and segmentation map generation, it requires a two-stage training process for the T2I model and label generator separately, contrasting with DatasetDM’s one-stage training. This additional stage could indeed limit practicality for real-time or large-scale augmentation, and a comparison in training efficiency or practical adaptability would be beneficial.
3. The dataset used for evaluation is Cityscapes and BDD100K, which includes only city streets. Since singlar scene makes learning specific concepts changes easiler, it would be improved if the authors prove their method on more general dataset, e.g. coco, ade20k. Since the main comparison method DatasetDM use more general dataset, I wonder performances of Selective LoRA on other datasets.
4. If the selective learning process affects the reliability of generated segmentation maps? The authors seem not provide relavant discussion.
5. Minor erros: the box of viewpoints and styles in stage 4) of Figure 2 are reversed.

Reference:
[1] DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models, NIPS 2023

questions:
Please see Weaknesses. If the authors address my concerns, I will raise my rate.
Additionally, It would be better if:
1. Provide quantitative metrics on the quality of the generated segmentation maps, comparing them to ground truth or to maps generated by other methods. Discuss any observed differences in segmentation map quality between their method and baseline approaches, particularly in relation to the selective learning process.
2. If possible, include a qualitative analysis (e.g., visual examples) of any artifacts or inconsistencies in the generated segmentation maps that might be attributed to the selective learning process.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732301561211

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
*We sincerely thank the reviewers for their thoughtful and constructive feedback, which has greatly contributed to improving the quality and clarity of our manuscript.*

We deeply appreciate the recognition of the strengths of our work and have refined our manuscript further by thoughtfully incorporating the suggested improvements. The strengths highlighted by the reviewers include:

- The core methodology, Selective LoRA, was acknowledged as both reasonable and novel by all reviewers (**B65c**, **hMNt**, **KMEN**, **kFPj**).
- The extensive experiments and ablation studies effectively demonstrated that the generated dataset enhances segmentation performance across various settings, including both in-domain and domain-generalization tasks (**B65c**, **hMNt**, **KMEN**, **kFPj**).
- Our work addresses the critical challenge of data scarcity, a key issue in semantic segmentation and other vision tasks (**hMNt**, **kFPj**).
- The paper is well-organized and well-written (**B65c**, **KMEN**), with a clear and concise description of the motivation (**B65c**). Additionally, the pipeline figures were noted as easy to understand (**B65c**).

We have carefully addressed all concerns raised by the reviewers and incorporated their valuable suggestions to further enhance the manuscript. Detailed responses to each reviewer’s comments are provided below.

> **Improving the overall presentation of the manuscript**  

- We have added detailed explanations of the segmentation dataset framework in **Section 3.1 Overall Framework** and **Section 3.4. Training Label Generator and Generating Diverse Segmentation Datasets**.  
- Further details are provided in **Appendix A.1. Implementation Details**, along with a newly added **Figure 7 Detailed Label Generator Architecture**.  
- To enhance clarity, we have reorganized **Section 4. Experiments** and updated **Table 1. (previously Table 3)**.  

> **Additional Experiments**: We conducted all additional experiments requested by the reviewers, as follows:  

- Additional image-label alignment experiments are provided in **Figure 6** and **Appendix A.6 Comparison of Image-Label Alignment**, addressing concerns from reviewers **B65c** and **kFPj**.  
- We tested our approach on a general domain dataset (Pascal-VOC), and the results are included in **Appendix A.8. In-domain Experiments for the General Domain Dataset**, as requested by reviewers **B65c** and **kFPj**.  
- We conducted experiments with hand-crafted selection baselines (Selective LoRA fine-tuning on all cross-attention layers) as suggested by **hMNt**. 
- Additionally, we included **InstructPix2Pix** as an additional baseline in the domain generalization setting (**Table 2**), addressing the suggestion by **KMEN**.
- To address concerns from **kFPj**, we added **Appendix A.7. Concept Sensitivity According to Prompt Augmentation**, demonstrating the robustness of our selection approach.  

> **Secondary Additional Experiments**: We conducted all the experiments requested by the reviewers, which were added as part of the second round review.

- We conducted an additional image generation baseline (DATUM) in **Table 2**, as requested by **KMEN**.  
- At the request of **KMEN**, we performed analyses on Image Domain Alignment and Image-Label Alignment in a domain generalization setting, detailed in **Appendix A.12. Additional Analysis of Our Generated Dataset on the Domain Generalization Setting**.  
- In response to **kFPj**'s suggestion, we further analyzed class-specific performance and proposed an additional method to improve specific classes, as outlined in **Appendix A.10. Class-wise Segmentation Performance Analysis**.  
- As requested by **kFPj**, we conducted new experiments on text guidance considerations, presented in **Appendix A.11. Generating Datasets with Diverse Class Names**.  

We appreciate the reviewers' detailed comments and hope that our revisions satisfactorily address their concerns. Detailed responses to each reviewer's comments are provided below.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Dear Authors, 

This draft received 6, 5, 6, 5. Some reviewers updated their scores based on author feedback however, the average rating is still lower than "marginally above the acceptance threshold". After going over the comments, draft, and feedback from authors, we find direction to be interesting but require further work. 

We will encourage authors to update the draft on the basis of comments from the reviewers. 


regards

AC