Lateralization LoRA: Interleaved Instruction Tuning with Modality-Specialized Adaptations
Authors: Zhiyang Xu, Minqian Liu, Ying Shen, Joy Rimchala, Jiaxin Zhang, Qifan Wang, Yu Cheng, Lifu Huang


Reviewer paper summaries: The paper introduces a relatively large instruction following dataset, called LeafInstruct, for text and image generation from text and image inputs. Also, it introduces Lateralization LoRA for modality-specific finetuning VLMs (Vision and Language Models).

**Comments after Author Discussion**: Thanks to the authors for their author response, however I will keep the score: This paper would really benefit from a resubmission and another round of review, as it has potential.
Summary of strengths: 1. The authors finetune the EMU 2 VLM with LoRA and notice bad performance. Therefore, they make the hybrid Lateralization LoRA approach that shows indeed **better results** (at least with the **one model the authors tested**).
Summary of weaknesses: 1. **Problematic framing** (already in the abstract): The analogy to brain lateralisation is overhyped and unhelpful. That the method has two parts, and the brain does too, does not mean anything. The analogy sounds great but is rather not helpful like most of the brain-inspired ML these days. It’s just a hybrid method between linear Lora (for the token components of the model) and Convolutional LoRA (for the image component of the model). A better naming would be: **modality specific**. L126-132 are misleading and unnecessary. To improve, focus on the actual technical contributions and avoid analogies that may mislead readers. 

1. **Novel method tested on one model only**: The experiments with Lateralization LoRA are applied only to EMU2. That it works on just one model is not convincing enough. To strengthen the paper, include experiments on a variety of models to show the generalizability of the approach. 

1. The new dataset is new and potentially useful in the author’s experiments, but the methods used to gather it are arguably not novel (using LLMs to filter caption for quality and to introduce instructions). The paper lacks both deep and shallow assessment of data quality and variance to be a dataset paper, so in this form, the new dataset makes for a much weaker contribution than it could be. To improve the paper, please provide a detailed analysis of the dataset's quality and diversity.
Comments, suggestions and typos: See suggestions to improve the paper in the weaknesses section.
Overall assessment scores: 2