SeLoRA: Self-Expanding Low-Rank Adaptation of Latent Diffusion Model for Medical Image Synthesis

## Abstract

Abstract The persistent challenge of medical image synthesis posed by the scarcity of annotated data and the need to synthesize ‘missing modalities’ for multi-modal analysis, underscored the imperative development of effective synthesis methods. Recently, the combination of Low-Rank Adaptation (LoRA) with latent diffusion models (LDMs) has emerged as a viable approach for efficiently adapting pre-trained large language models, in the medical field. However, the direct application of LoRA assumes uniform ranking across all linear layers, overlooking the significance of different weight matrices, and leading to sub-optimal outcomes. Prior works on LoRA prioritize the reduction of trainable parameters, and there exists an opportunity to further tailor this adaptation process to the intricate demands of medical image synthesis. In response, we present SeLoRA , a Self-Expanding Low-Rank Adaptation Module, that dynamically expands its ranking across layers during training, strategically placing additional ranks on crucial layers, to allow the model to elevate synthesis quality where it matters most. The proposed method not only enables LDMs to fine-tune on medical data efficiently but also empowers the model to achieve improved image quality with minimal ranking. The code of our SeLoRA method is publicly available on https://anonymous.4open.science/r/SeLoRA-980D .

###### Keywords:

## 1 Introduction

Foundation models [, , ] are increasingly gaining traction in medical imaging, offering a new paradigm for data processing and analysis. While most foundational models are trained with large natural-image datasets [], such as *ImageNet*, the shortage of medical images is increasingly problematic []. Medical image synthesis presents a valid approach to address this issue by generating synthetic images to expand and enhance scarce image datasets.
Challenges in medical image synthesis have been widely explored, leading to various proposed methods. These models are typically trained from scratch for a single modality like brain MRI [], lung CTs[], Cataract Surgery Samples [], and others. However, training models from scratch requires long training times and can lead to performance limitations due to dataset size.

Recognizing the success of pre-trained foundational models on natural images, recent works have shifted towards adapting these models, such as stable diffusion [], to enable more efficient training for medical image synthesis. For example, leveraging text-based radiology reports as a condition to incorporate detailed medical information, and fine-tuning latent diffusion models has achieved significant performance gains [, ]. Further exploration involves the incorporation of parameter-efficient fine-tuning (PEFT) methods, which not only makes fine-tuning more efficient but also demonstrates superior performance compared to full fine-tuning []. In this context, we place a particular emphasis on applying the Low-rank Adaptation (LoRA) [] method, a type of PEFT method, for fine-tuning stable diffusion in medical image synthesis.

Originally designed for adapting large language models, *LoRA* hypothesizes that the weight matrix updated during fine-tuning exhibits a low ‘intrinsic rank’. Therefore, it prompted to use the product of two trainable low-rank decomposition matrices to mimic the weight update for a specific task, expressed as follows:

$$ $W=W_{0}+AB,$ (1) $$

where the weight matrix, $W^{d_{in}\times d_{out}}$, is updated by combining the frozen original weight $W_{0}^{d_{in}\times d_{out}}$, and the product of two trainable low-rank decomposition matrices, $A\in\mathbb{R}^{d_{in}\times r}$ and $B\in\mathbb{R}^{r\times d_{out}}$. The rank, $r$, is deliberately chosen to be significantly smaller than the dimension of the original weight. Consequently, the trainable parameters under this configuration constitute a fraction of $W_{0}$’s parameter count, achieving parameter-efficient fine-tuning.

The design of *LoRA* introduced the challenges of selecting the optimal rank. Small ranks yield suboptimal performance, while large ranks escalate parameter count, and searching for optimal rank for each individual *LoRA* on different layers is computationally expensive. In response, *LoRA* proposes to select a uniform rank across layers, but this intensifies the problem associated with rank when *LoRA* is applied to LDMs. The bottleneck nature of denoising *U-Net* implies a diversity in the shape of weight matrices. Thereby, the underlying rank of the weight update may also vary. Consequently, uniform rank selection on LDMs may introduce exacerbated results, ultimately compromising the quality of synthesized images.

In addressing the challenge of rank selection and aiming to achieve superior synthesized image quality with minimal trainable parameters, we draw inspiration from the concept of self-expanding a neural network []. Departing from approaches of setting a predefined uniform rank, our approach advocates for dynamically expanding the rank of *LoRA* to better align with the unique needs of each layer. As a result, in our work, we present a Self-Expanding Low-rank Adaptor (SeLoRA), akin to *LoRA*’s structure but distinguished by the dynamic growth of ranks guided by Fisher information during training. This enables *SeLoRA* to flexibly adapt to the inherent characteristics of each layer, guaranteeing enhanced medical image synthesis quality while minimizing challenges related to rank adjustments.

## 2 Related Work on LoRA

This section focuses on reviewing low-rank adaptation methods that address the rank selection problem.
Building upon *LoRA*, several lines of research are being developed. While many approaches primarily aim to reduce the number of trainable parameters using techniques like quantization [], shared weights [], and various product operations [], there is limited work on how to enhance the model performance or tackle the rank selection problem as primary objectives.

Figure: Figure 1: Training illustration of a single *SeLoRA*. *SeLoRA* behaves similarly to a basic *LoRA* during training. However, it is tested for the expanded rank every $t$ step and is progressively expanded if the FI-Ratio exceeds the desired threshold.
Refer to caption: extracted/5781792/SeLoRA.png

Among these works, one line of research explores adaptive ranking to address selection issues, wherein each layer learns to have a different rank through pruning. *DyLoRA* utilizes nested dropout loss to sort the rank, allowing *LoRA* to be pruned to the optimal rank after training []. *AdaLoRA* assigns an importance score for each rank, evaluated based on the magnitude of singular values, thereby pruning less important ranks []. *SoRA* introduces a specially designed gating function that zeros out useless rank during training []. While all these pruning-based methods effectively achieve adaptive rank across layers, the fundamental challenge of rank selection remains unsolved, as they introduce a fixed upper bound, which is the initial starting rank of the trainable low-rank matrices.

## 3 SeLoRA

In addressing the challenge of rank selection and maintaining superior performance with minimum rank, we propose *Self-Expanding Low-Rank Adaptation* (SeLoRA). The general idea of *SeLoRA* is to initialize the trainable low-rank decomposition matrices with rank $r=1$ and dynamically expand its rank individually during training to adapt to varying layer needs. Now, to facilitate adaptive growth, we address two key research questions: (1) How can to expand the rank without perturbing the output? (2) At what juncture should the expansion occur?

#### 3.0.1 How to expand?

When *SeLoRA* expands itself by adding a new rank, the model’s final prediction should remain constant. Hence, to prevent perturbations in the model’s output when introducing a new rank, a straightforward approach is to force the product of the expanded rank to be 0 0. However, simple all-zero initialization poses a challenge as it hinders gradient flow through the expanded rank during initial back-propagation. Consequently, utilizing methods e.g., Fisher information to assess the expansion is not applicable. Therefore, we propose to initialize the expanded rank of matrix $B$ to be 0 0, and the expanded sub-matrix, $K$, of matrix $A$ to be initialized with Kaiming uniform initialization. Hence, the expanded form of *SeLoRA* can be expressed as follows:

$$ $f(x)=xW_{0}+x\begin{bmatrix}A&K\end{bmatrix}\begin{bmatrix}B\\ 0\end{bmatrix}+b_{0}$ (2) $$

where $K\in\mathbb{R}^{d_{in}\times 1}$ is a vector initialized with Kaiming uniform initialization. Now, the expanded *SeLoRA* maintains the desired output while allowing the gradient to propagate through $A$.

#### 3.0.2 When to expand?

To determine when an expansion could enhance the model, a crucial criterion is evaluating the potential improvement introduced by the rank addition. In assessing the viability of an expanded *SeLoRA* without excessive training, we employ Fisher information to measure the information conveyed by *SeLoRA* weights from the datasets. The conceptualisation of using Fisher information for expansion decision, stems from prior works on model selection procedures (using Fisher information), such as the Akaike information criterion, the Bayesian information criterion, and later work on low-rank approximation on neural network’s weights [, , ]. Here we utilized the empirically estimated Fisher information, introduced in Equation 3, as deriving an exact value is generally intractable due to the need for marginalization over the entire dataset.

$$ $\hat{I}_{w}=\frac{1}{|B|}\sum^{|B|}_{i=1}\left(\frac{\partial}{\partial w} \mathcal{L}(b_{i};w)\right)^{2},$ (3) $$

where $|B|$ is the batch size, and $b_{i}$ is a sample in the batch. To quantify the information carried by a single *SeLoRA*, we introduce the Fisher information score (FI-Score), which sums over all Fisher information of the weight matrices $A,B$ in *SeLoRA*, as shown in Equation 4:

$$ $\text{FI-Score}=\sum_{i=1}^{d_{in}}\sum_{j=1}^{r}\hat{I}_{A_{i,j}}+\sum_{i=1}^ {r}\sum_{j=1}^{d_{out}}\hat{I}_{B_{i,j}}.$ (4) $$

Here, while the FI-Score is calculated by back-propagating gradients, it is important to note that the optimizer does not update the parameters during this calculation for the expanded *SeLoRA*. To further assess whether the expanded *SeLoRA* is superior to the previous unexpanded version, we introduce the Fisher information Ratio (FI-Ratio). The ratio between the Fisher information score of the expanded and original *SeLoRA* is calculated, as shown in Equation 5:

$$ $\text{FI-Ratio}=\frac{\text{FI-Score}_{orig}}{\text{FI-Score}_{exp}}.$ (5) $$

Equation 5 provides a metric to evaluate the information gained when *SeLoRA* is expanded. This ratio is measured at each *SeLoRA* module, and *SeLoRA* undergoes expansion when the FI-Ratio exceeds a desired threshold $\lambda$. An aggressive threshold, such as $\lambda=1$, accepts any improvement to the model. Alternatively, a conservative threshold, such as $\lambda=1.3$ or a larger value, could result in a more cautious expansion. Additionally, to ensure each expansion is beneficial, a hyperparameter $t$ is introduced. Testing of *SeLoRA*’s expansion is conducted at each $t$ training step, allowing the previous expanded rank to learn and converge before testing the new expansion. The detailed training procedure is shown in Algorithm 1.

Figure: Algorithm 1 *SeLoRA* Training Procedure

### 3.1 Datasets for Evaluation

We compare *SeLoRA* against *LoRA* and other baselines, assessed on two metrics. A rank allocation analysis is then performed to examine the distribution of ranks within the trained *SeLoRA* model, followed by an ablation on different $\lambda$ values.

IU X-RAY Dataset [], collected by Indiana University, consists of 3,955 radiology reports paired with Frontal and Lateral chest X-ray images, each accompanied by text findings. For our experiments, we arbitrarily chose to exclusively focus on frontal projections, considering the substantial dissimilarities between frontal and lateral images. The dataset is partitioned into training, validation, and testing sets, with $80\%/16\%/4\%$ of the datasets, respectively.

Montgomery County CXR Dataset is relatively compact, consisting of 138 frontal chest X-rays []. Each entry in the dataset includes an X-ray image, along with details about the patient’s sex, age, and medical findings. Among the 138 cases, 80 are labeled as ‘normal,’ with the remaining cases describing specific patient illnesses. The patient’s sex, age, and findings are concatenated as prompt. The dataset is then divided into training and testing sets with a split ratio of $80\%/20\%$. Omitting a validation set ensures a sufficient number of samples for both the training and testing phases.

During fine-tuning, to accommodate the text encoder of the stable diffusion, any text finding longer than 76 tokens is truncated, as outlined in the *CLIP* approach []. As for the training images, they are initially resized such that the shortest side is 224 pixels wide, followed by centered cropping to obtain an image size of $224\times 224$.

### 3.2 Implementation Details

The experiments are implemented in Pytorch, utilizing the base stable diffusion model weights obtained from ‘runwayml/stable-diffusion-v1-5’ in huggingface []. *LoRA*, *DyLoRA*, *AdaLoRA*, and *SeLoRA* are used in experiments for comparison. All *LoRA* modules are only injected into the linear layer of the stable diffusion, which consists of the *U-Net* and the text encoder used to condition the denoising process. Prior works have proved that fine-tuning the VAE part of the stable diffusion model leads to no significant differences in performance on medical images, so injecting the *LoRA* module into VAE was omitted []. The models are trained using mean squared error loss and are evaluated after each epoch on the validation set. The best-performing model is then utilized to examine inference on the test set. Model trains for 10 epochs on IU-Xray dataset and 100 epochs on Montgomery County CXR due to its smaller size. For our method, we chose to use a threshold of $\lambda=1.1$, accompanied by $t=40$ training steps. While these parameter choices are arbitrary, a relatively large $t$ step is desired to allow *SeLoRA* to fully converge before the evaluation of expansion. For *LoRA* and *DyLoRA*, the rank is set to $r=4$, and the rank of *AdaLorA* is initialised with rank $r=6$ and reduced to the target rank of $r=4$. The original *LoRA* experiments was conducted with a rank of $r=\{1,2,4,8,64\}$. We selected the median value, $r=4$ across all testing procedures, as it has a comparable number of trainable parameters to our trained *SeLoRA*, ensuring a fair comparison.

### 3.3 Evaluation Metric

We assessed the synthetic images based on both the fidelity and their alignment with the provided prompts. To measure fidelity, we use Fréchet Inception Distance (FID), which calculates the similarity between the distribution of synthetic datasets and the distribution of the original datasets [].
For assessing alignment with the prompt, CLIP score is used, which measures text-to-image similarity []. Results are averaged across three random seeds and summarized in Table 1 for IU X-RAY and Montgomery County CXR Datasets, respectively.

## 4 Results

**Table 1: Quantitative results of IU X-RAY and CXR datasets. The best results are highlighted in bold.**
|  | IU X-RAY | Montgomery County CXR |  |  |
| --- | --- | --- | --- | --- |
| Methods | FID $\downarrow$ | CLIP Score $\uparrow$ | FID $\downarrow$ | CLIP Score $\uparrow$ |
| LoRA | $113.37\pm 7.88$ | $27.11\pm 0.18$ | $461.80\pm 14.10$ | $24.34\pm 0.71$ |
| AdaLoRA | $184.25\pm 53.05$ | $26.90\pm 0.33$ | $489.24\pm 14.28$ | $22.47\pm 0.23$ |
| DyLoRA | $116.03\pm 10.29$ | $26.89\pm 0.05$ | $475.42\pm 7.84$ | $22.93\pm 0.27$ |
| SeLoRA | 113.04 $\pm$ 16.79 | 27.26$\pm$0.22 | 205.54 $\pm$ 9.56 | 26.38 $\pm$ 0.04 |

Our results clearly highlight the superior performance of the proposed method, *SeLoRA*, in terms of fidelity and alignment with the prompt (text condition). In Table 1, we observe that *SeLoRA* consistently outperformed all other methods across all evaluation metrics for these relatively small datasets. These results demonstrate that *SeLoRA* is advantageous and truly promising to be examined and expanded across more medical image synthesis data and problems, given the inherent scarcity of medical datasets [].
Moreover, *SeLoRA* outperforms all other *LoRA* variants on IU-XRay by only using small fractions of trainable parameters, averaging $0.121$ and $0.368$ for the text encoder and U-net, respectively. In contrast, other *LoRA* methods with rank $r=4$ use nearly double the trainable parameters ($0.216$ and $0.803$) while achieving lower scores.

Furthermore, synthetic images generated by a model trained on the IU-Xray dataset are displayed in Figure 2. Notice that, although all synthetic images achieved similar quality, in comparison, *SeLoRA* is the only method that captures the distinct pathology (represented as a black circle) in the lower part of the right lung, closely resembling the original image.

Figure: Figure 2: Qualitative comparison results obtained on the IU-Xray data, generated by fine-tuning stable diffusion models injected with various LoRA variants. Prompt used: ‘‘Heart size and vascularity normal. These contour normal. Lungs clear. No pleural effusions or pneumothoraces.’’ More sample results are presented in the Appendix.
Refer to caption: extracted/5781792/quality_comparison_cropped.png

### 4.1 Rank Allocation Analysis

Figure 3 illustrates the final rank results of *SeLoRA* across stable diffusion model layers after training on the IU-XRay dataset. In the text encoder part, large ranks for *SeLoRA* lie at the q and k parts of the attention weights. For the U-net part, large ranks are allocated to the q and k parts of the second attention layer, namely the cross-attention layer. The rank allocation aligns with the intuition that weight updates would change most dramatically at locations where the latent representations of text and image intersect (where conditioning is more apparent). Hence, it validates our hypothesis that our proposed expansion method allows *SeLoRA* to focus and place more rank on crucial layers.

Figure: (a) Rank of *SeLoRA* in Text Encoder.
Refer to caption: extracted/5781792/text-encoder_rank.png

### 4.2 The Impact of λ 𝜆 \lambda italic_λ

Further results obtained by varying $\lambda$ are displayed in Tab. 1, Fig. 1, and Fig. 2 in the Appendix. These results indicate that a smaller $\lambda$ leads to a more aggressive expansion of all layers in *SeLoRA*, resulting in improved synthesis quality, albeit at the cost of a more expensive training process. However, the critical layers that *SeLoRA* emphasizes remain consistent with our observations in Section  4.1.

## 5 Conclusion

This work introduces a novel parameter-efficient method named *SeLoRA*, designed to effectively fine-tune stable diffusion models for generating X-ray images based on text (radiology) prompts. Our method enables progressive expansion in the rank of *LoRA*, enabling more precise image synthesis with minimal added rank. Through exploratory analysis, we demonstrate that the proposed FI-Ratio is capable of effectively guiding *SeLoRA* to expand its rank and allocate more rank to crucial layers. We believe that *SeLoRA*, when combined with stable diffusion, can be easily employed to adapt to various medical datasets containing text-image pairs, and potentially being applicable for clinical text synthesis. Moreover, given the increasing work on segmentation and detection in 3D (tomographic, e.g., magnetic resonance imaging, computed tomography, and other) medical imaging data and the emergence of prompt-to-3D models, in our future work, we aspire to explore *SeLoRA* adaptations on fine-tuning prompt-to-3D models for 3D medical image synthesis.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: - How is the proposed method compared to other baselines in terms of max training GPU memory, training speed, and
training time costs? An analysis of these criteria would strengthen the paper.

- The paper mentions thresholds (λ and t) for triggering rank expansion. How sensitive is the method to these hyper-parameters?

- In figure 7, why are the other methods not generating similar xray images? As per my understanding, they should at-least make a similar image like figure 8. 

- The proposed method is just compared to methods with rank pruning methods (dylora and adalora), could you mention why there is no comparing with adaptive rank selection papers? as they seem to be a similar approach to your paper.
soundness: 2
strengths: 1. The paper is well-organized and easy to follow.

2. The idea of adaptive computation rank selection is interesting and highly relevant for using pre-trained models for downstream tasks in a memory efficient fashion
summary: The paper introduces SeLoRA (Self-Expanding Low-Rank Adaptation), an extension of the LoRA (Low-Rank Adaptation) technique, designed for the fine-tuning of large diffusion models specifically in medical image synthesis. The core idea is to dynamically expand the rank of low-rank matrices during training, based on a criterion derived from Fisher information. This adaptation is applied selectively across different layers, allowing for a more effective distribution of ranks that aligns with each layer's significance in the model, particularly within the denoising U-Net of the Stable Diffusion framework.
weaknesses: - While the paper mainly focusses on making a more efficient LoRA design as claimed by authors, there is no Analysis of Training Efficiency in the paper.

- Lack of detailed explanation of the procedure of selecting hyper-parameters needed.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please answer the questions mentioned in the above 'weakness' section.
soundness: 2
strengths: 1. 
The papers proposes SeLoRA, a dynamic rank-expanding method using Fisher Information to guide rank expansion during fine-tuning large models with LoRA. This is novel and more applicable to models with diverse weight matrix shapes.
2.
Experimental results demonstrate the effectiveness of the proposed method.
3. 
The paper is well-organized.
summary: This paper proposes a new parameter-efficient fine-tuning method SeLoRA (Self-Expanding Low-Rank Adaptation) for adapting Stable Diffusion to generate chest x-ray images. The main contribution of the paper lies in dynamically expanding the rank of LoRA during the training process, allowing it to adapt the rank according to the importance of different layers and thereby improving the quality of the synthesized images. The novelty of this approach is in using Fisher Information to guide the rank expansion, avoiding the limitations of the traditional LoRA method, which uses a "uniform rank," especially when dealing with models (like Stable Diffusion) that have diverse weight matrix shapes. The paper demonstrates the effectiveness of SeLoRA on two chest x-ray datasets and provides detailed comparative experiments with other LoRA variants.
weaknesses: 1.
The contribution is vague. As a paper focusing on adaptive parameter efficient fine-tuning methods, the paper utilizes LoRA to adapt Stable Diffusion for Chest X-ray synthesis, limiting its technical contribution. For a paper dedicated to adapting foundation model like Stable Diffusion for Chest X-ray (medical image) synthesis, the exploration is also limited and does not compare with previous work (e.g. Chambon et al., 2022a;b). Through visual comparison with image displays in Chambon et al., 2022a;b, the proposed method seems at a disadvantage.
2.
Experiments are conducted on relatively small datasets. Large image-report paired Chest X-ray datasets exist (e.g. MIMIC-CXR). Is this because of the heavy burden of large model like Stable Diffusion, or may be also related to the proposed method? Can the authors provide training time comparisons between SeLoRA and the compared methods? Also, as the test set of IU X-RAY and Montgomery County CXR dataset has only contains 100~200 images, the validation of the effectiveness of the method is weak.
3.
Evaluation and explanation are insufficient. Using a CLIP model trained purely on natural images and a maximum text token length of 76 to compute CLIP-score may not faithfully reflect how good the text-image alignment is for Chest x-ray images.
The training/validation/testing split is strange, could the author explain why the test set only contain 4% of the data? Are Table 1 values computed with valid data or test data? This is unclear. The paper also lacks in-depth discussion of the distribution of final rank (Figure 4,5,6) and why other LoRA methods fail on Montgomery county CXR data (Figure7). How accurate can the model generate an image given the disease or abnormal findings in the text prompt? This may be revealed using pretrained Chest X-ray classification models or manually inspect a small subset of generated results.

Other non-important issues:
4.
the paper title is about "medical image syntheis”; but it only focuses on chest x-ray image.
5.
The formula derivation in section3.3 is unclear.
6.
The Stable Diffusion model is trained to generate a resolution of 512x512, using a resolution of 224x224 may limit the performance.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Computational complexity analysis A comparison of training time, memory usage and FLOPs between SeLoRA, LoRA, and other variants is needed to quantify the computational trade-offs introduced by dynamic rank expansion.
2. Evaluation on a wider range of datasets Evaluating SeLoRA on larger datasets, such as MIMIC-CXR (containing approximately 377,000 images), would provide more insights into its scalability. Future work could also validate SeLoRA on MRI, CT, or ultrasound datasets, as testing on diverse datasets would better demonstrate its robustness and versatility.
3. Incorporating related work. The idea of dynamically adjusting the rank of the LoRA matrix in SeLoRA is conceptually similar to the recently proposed ALoRA (NAACL 2024). However, the two methods differ in implementation: ALoRA utilizes pruning and redistribution strategies, while SeLoRA relies on Fisher information as the adjustment criterion. Insights from ALoRA could provide valuable inspiration for future improvements of SeLoRA.
4. Evaluation with medical doctor may help verify the experiment results.
Other questions
1. Unconventional split of the IU X-Ray dataset. The 80:16:4 split results in a relatively small test set, which could compromise the robustness of the evaluation. A more conventional split (e.g., 80:10:10) might provide more reliable insights.
2. Small sample size in the Montgomery County CXR dataset. With only 138 samples, the Montgomery County dataset is too small for deep learning applications, which may impact the stability and generalizability of the model’s results
soundness: 3
strengths: The paper introduces a parameter-efficient method to e fine-tune stable diffusion models for generating X-ray images based on text (radiology) prompts. And the proposed method can progressive expansion in the rank of LoRA. FI-Ratio is used to guiding SeLoRA to expand its rank. The rank of different layers was given. The experiment shows the result is promisingly.
summary: The paper present SeLoRA, a Self-Expanding Low-Rank Adaptation module, that dynamically expands its ranking across layers during training. The proposed method increases the rank from 1 gradually. FI-Ratio and parameter \lambda were used to determine when to expanding the rank.  The experiment is performed with stable diffusion and X-ray dataset and synthesis the X-ray image with text prompt.
weaknesses: 1. Computational overhead. While SeLoRA reduces the number of trainable parameters, its dynamic rank expansion mechanism introduces additional computational complexity. Computing Fisher information increases the overhead, which may become significant for larger datasets or more complex models.
2. Limited dataset evaluation. Experiments were limited to two 2D X-ray datasets, with no evaluation of SeLoRA’s performance on other modalities, such as MRI or CT scans. Further validation on additional modalities would help confirm the generalizability of the method.
3. Visual result is limited. The visual result is not satisfactory, such as in Figure 8, the contrast and details are not good.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Included in the Weaknesses part.
soundness: 3
strengths: * Originality: good originality, combining self-expanding and LoRA to make LoRA self-adapt to the different needs of layers.

* Quality: simple but effective method. Performance validated in the scope of medical image synthesis.

* Clarity: clear figures and method presentation.

* Significance: Data scarcity is very common in the medical AI field, and a strong LoRA variant is important to the community. IF the proposed method is effective and generalizable in the general scenes, it could be significant in efficient learning.
summary: This paper presents Self-Expanding Low-Rank Adaptation (SeLoRA), akin to LoRA’s structure but distinguished by the dynamic growth of ranks guided by Fisher information during training.   This enables SeLoRA to flexibly adapt to the inherent characteristics of each layer, guaranteeing enhanced medical image synthesis quality while minimizing challenges related to rank adjustments.
weaknesses: * The paper proves the proposed method is significantly outstanding in the medical image synthesis scene (IU X-RAY dataset and CXR dataset). Would it work well in other tasks, like perceptual tasks or other synthesis tasks?

* Time cost is an important metric for the self-expand algorithm. The paper lacks a discussion of the efficiency of the algorithm and the experimental comparison with other LoRA methods.

Meta-review: This paper presents a method to use latent diffusion model for medical image synthesis. While the results show some improvement by FID scores, there is a lack of analysis how the synthesis makes an impact for downstream medical image applications. The authors claimed that the proposed method have an excellent trade-off between efficiency and performance. I am not sure how the training efficiency here helps the clinicians. In addition, I am not convinced in some of he claims and results. For example, the authors claimed that “SeLoRA strikes an excellent balance between efficiency and performance”. However, the training efficiency from different algorithms are very close and differs by a few mins with a total training time around 2hours 40 mins or 1 hour 45 mins in two different datasets, as shown in the table from the rebuttal. This is hard for me to accept the claim that the proposed method benefits much on training efficiency. The ablation study shows that the FID scores change abruptly for lamda from 1.0 to 1.1, a detailed analysis shall be required to evaluate the robustness of the results.

Decision: Reject