LM4LV: A Frozen Large Language Model for Low-level Vision Tasks

Authors: Boyang Zheng, Jinjin Gu, Shijun Li, Chao Dong
Primary area: generative models
Keywords: Low-level vision, Large Language Model, Self-supervised Learning

Abstract:
The success of large language models (LLMs) has fostered a new research trend of multi-modality large language models (MLLMs), which changes the paradigm of various fields in computer vision. Though MLLMs have shown promising results in numerous vision-language tasks such as VQA and text-to-image, no work has demonstrated how low-level vision tasks can benefit from MLLMs. We find that most current MLLMs are blind to low-level features due to their design of vision modules, and thus are inherently incapable of solving low-level vision tasks. In this work, we propose **LM4LV**, a framework that enables a FROZEN LLM to solve a range of low-level vision tasks without any multi-modal data or prior. This showcases the LLM's strong potential in low-level vision and bridges the gap between MLLMs and low-level vision tasks. We hope that this work can inspire new perspectives on LLMs and a deeper understanding of their mechanisms.

Decision: reject
Reviewer scores: 6, 5, 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1729242930230

thread_type:
official_review

<thread_object>
creator: reviewer_ekg6

<content>
summary:
The paper proposes a framework called LM4LV to explore the ability of vanilla Large Language Model in handling with the low-level vision task. LM4LV utilizes the fine-tuned MAE to extract the image features, and take two linear layers as the bridge from vision to language. A paired decoder is used to transform the output tokens to images. It shows competitive quality results in denoising, deblurring, deraining and so on, but the effective in spatial operations like rotation and flipping is ordinary.

strengths:
1. LM4LV explores the potential of LLM in handling with the vision data without any multi-modal data, which can provide some references.

2. Detailed appendices provide the analysis and the process of implementation. It's easy to follow it. 

3. The paper is well written and easy to understand.

weaknesses:
1. Contribution. The contribution is limited. LM4LV relies on the existing auto-regressive ability from the training process in LLM. It only constructs a linear layer as the bridge to align the vision semantic space and language semantic space to predict the next token.

2. Soundness. The soundness of the paper is fair. It only utilizes the LLM as a next token predictor, and not explores another way to process vision information by LLM directly. 

3. The qualitative results are not fair. The comparisons with other methods should cover the other denoising, deblurring, deraining methods like MAXIM [1], Restormer [2]

4. The novelty is limited. The framework is composed with a fine-tune MAE and a LLM.

[1] Maxim: Multi-axis mlp for image processing, CVPR 22
[2] Restormer: Efficient transformer for high-resolution image restoration, CVPR 22

questions:
1. In the line 54-57, paper claims that "Enabling MLLMs to process low-level features can lead to a more fine-grained understanding of images and better control in the process of image generation." It's there any fine-grained qualitative or qualitative results to prove that? 

2. I can't understand the Fig.3, is there lack of labels?

3. For the auto-regressive process, the image tokens a limited in the <img> and </img>, so how to control the number of image tokens to make it same with the original image?

4. What is the order of auto-regressive, as row or column? Is there any difference when use different order to generate the image tokens?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730002284485

thread_type:
official_review

<thread_object>
creator: reviewer_r61j

<content>
summary:
The paper presents LM4LV, a framework that enables a frozen large language model (LLM) to handle low-level vision tasks, such as denoising and deraining, without any multimodal data or prior. By combining a masked autoencoder (MAE) with a frozen LLM and two linear adapters, LM4LV demonstrates non-trivial capabilities in various low-level image processing tasks, thus bridging a gap between large vision-language models (LVLMs) and low-level vision.

strengths:
1. The paper addresses an under-explored area in multimodal research by extending LLMs to low-level vision tasks without requiring multimodal training.

2. Comprehensive experiments cover diverse low-level tasks, including denoising, deblurring, and pepper noise removal.

weaknesses:
1. The paper could benefit from a broader comparison with other non-LLM approaches, particularly in vision-language domains, to better position LM4LV’s relative advantages.

2. The motivation to use LLM for these low-level vision tasks, instead of task-specific models, are still not clear.

questions:
This paper study the use of LLM for low-level vision tasks, which is under-explored. However,  the motivation to use LLM for these tasks are a bit weak, and the model struggles to preserve high-frequency details in degraded images, suggesting room for further exploration in loss functions or architectural enhancements.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730356839435

thread_type:
official_review

<thread_object>
creator: reviewer_kc9o

<content>
summary:
The authors introduce LM4LV, a framework that enables frozen LLMs to handle low-level vision tasks (like image denoising and deraining) without requiring multi-modal data or prior training. Through comprehensive experiments, the authors demonstrate the framework's effectiveness by showing significant improvements over baselines and reveal two crucial design choices: the necessity of auto-regressive generation and the importance of vision module selection.

strengths:
Pro:
1. The paper tackles an interesting research direction by investigating whether frozen LLMs can handle low-level vision tasks without multi-modal training, addressing a significant gap in current MLLM research.
2. The authors propose LM4LV, an efficient framework that achieves impressive results across multiple low-level vision tasks using only two trainable linear layers while keeping the LLM frozen.
3. The paper provides thorough empirical analysis through comprehensive ablation studies, particularly demonstrating the necessity of auto-regressive generation, the importance of choosing appropriate vision modules, and the effectiveness of each module, which helps validate the framework's design choices.

weaknesses:
Cons:
1. The paper said, “Furthermore, we cancel the causal attention mask and the ROPE position embedding in the forward process, as they are not the common practice for vision modules.”. However, the ROPE and it variant 2D-ROPE are widely used in large vision transformers (e.g., EVA). This sentence needs revision, and additional experiments with position embeddings would strengthen the analysis.
2. The paper does not explore different LLM variants and sizes. While testing very large LLMs may be resource-intensive, experimenting with smaller or comparable models (like LLaMA-3.2-3B or Qwen2.5 series) would provide valuable insights into how model architecture and scale affect performance.

Suggestion:
1. The paper would benefit from including comparisons with state-of-the-art (SOTA) results from mainstream methods for each low-level vision task. The readers do not expect the method to outperform SOTA methods, while a clear comparison might be helpful for the readers to understand the position of the paper.
2. It would be valuable to investigate the effectiveness of non-autoregressive language models like BERT for low-level vision tasks. While the paper's experiments show that removing auto-regressive generation from LLMs significantly degrades performance, it alters the pre-trained model. Testing with models that are inherently non-autoregressive (like BERT) could provide clearer insights into whether the auto-regressive nature of LLMs is truly essential for these tasks or if alternative architectures might be viable with proper adaptations.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730478754073

thread_type:
official_review

<thread_object>
creator: reviewer_hf2b

<content>
summary:
This paper proposes a novel framework, named LM4LV, which applies MLLM to low-level vision tasks. Through preliminary experiments, the authors highlight the advantages of MAE in preserving detailed visual features. Therefore, MAE is used as the visual encoder in LM4LV. Keeping the LLM and visual encoder frozen, the training of LM4LV can be conducted without multimodal data. Extensive experiments are conducted on various low-level vision tasks and different vision modules, to verify the low-level vision task capability of LLM.

strengths:
+ The proposed method is simple and easy to understand.
+ This work is the first to use LLM for low-level vision tasks.
+ Some conclusions in the paper are interesting. E.g. MAE visual tokens are robust to rotation.

weaknesses:
- It would be helpful if the authors could specify the number of visual tokens generated by MAE for each image. Moreover, the discussion about computation cost is missing.
- While in sec. 4.1, the authors use MAE-r model as a baseline. Since MAE-r is trained for image reconstruction only, the baseline is not very strong. To establish a stronger baseline, the authors can consider adding the linear adapters into MAE-r, and train the linear adapters for image restoration tasks. Ideally, the linear adapters should be trained separately for each image restoration task, with the same recipe used for training LM4LV.
- Lack of comparison with traditional low-level vision methods. I'm not an expert in low-level vision tasks, so a dedicated part in the related works section would be helpful. Moreover, I don't see any comparison between the proposed LM4LV framework and the previous low-level vision methods. 
- The presentation contains several typos. In line 018 "we purpose LM4LV" and line 039 "purpose to use LLMs", should the word "purpose" be "propose"? In line 471 "(Amid et al., 2022) requires", the citation format can be corrected. In line 914, "ativation" might be a typo.

questions:
1. Why fine-tune MAE to perform image reconstruction, instead of using MAGE or MaskGIT? MAGE and MaskGIT use the MIM training paradigm to perform image reconstruction. Will they achieve better performance than MAE?
2. In sec. 5.2, the authors claim they randomly initialize the LLM. After the random initialization, is the LLM frozen or fine-tuned? It would be interesting if the LLM is fine-tuned but still underperforms the frozen LLM counterpart.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719129858

thread_type:
official_review

<thread_object>
creator: reviewer_ty68

<content>
summary:
This paper analyzes the reasons why most MLLMs cannot handle low-level features and then puts forward a simple solution. It uses MAE fine-tuning with Image reconstruction tasks to save low-level features. After that, it is then interesting to find that frozen LLM can understand low-level features, which taps into the LLM's uncanny ability to understand visual features.

strengths:
(1) It proves frozen LLM can solve low-level vision task
(2) The ablation experiment is sufficient to answer that the processing of low-level information is not due to the trainable linear layer, but the text pre-training plays a role

weaknesses:
vision encoder selection is relatively small, exploring more vision encoders will be more convincing

questions:
None
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduced LM4LV, a new framework for low-level vision, which uses a frozen LLM to solve low-level vision tasks. The authors use MAE as the visual encoder, combine it with LLM and keep both frozen, the training of LM4LV can be conducted without multimodal data. The experiments demonstrate the low-level vision capability of LM4LV.

This paper is interesting to me. It explored a new direction and problem, which shows that frozen LLMs with proper approach can solve low-level vision tasks to som extent.

But the concerns about insufficient ablations, low performance and efficiency, and unclear motivation/advantage do exist.

I recommend rejecting as the current state, but the decision can be bumped up.
I would suggest that the authors explore the potential advantages further. Again, I think this is a interesting exploration.