Rethinking Fine-tuning Through Geometric Perspective

Authors: krishna  sri
ipsit  mantri
moshe  eliasof
carola-bibiane  schönlieb
bruno  ribeiro
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-24 11:04:08

Abstract: Fine-tuning pre-trained neural networks has become a cornerstone of transfer learning. However, the practical success of existing methods like low-rank adaptation (LoRA) lacks theoretical explanation. We introduce geometry-guided fine-tuning, a novel paradigm that models the fine-tuning process as the subtle movement of pre-trained weights on a low-dimensional manifold. Our approach formalizes this process through a learnable ordinary differential equation (ODE) - based framework that controls the search space of the weights, bridging existing methods with geometric principles. We empirically evaluate our method in the context of multi-task learning (MTL) fine-tuning of hierarchical vision transformers in computer vision. We propose a parameter-efficient ODE and evaluate it on the PASCAL-Context MTL benchmark.  Our approach, dubbed DeLoRAoffers competitive performance across multiple dense prediction tasks, reducing trainable parameters by up to 4$\times$ compared to the best-performing baseline. This work advances both the theoretical understanding and practical application of fine-tuning, promoting efficient learning in resource-constrained environments.

Main paper: 
 introduction: 
 The success of large pre-trained neural networks across various domains, such as GPT-2 [Radford et al., 2019] and Stable Diffusion [Rombach et al., 2022] , has made fine-tuning an essential technique in transfer learning. As these models grow in size and complexity, efficient and effective fine-tuning methods become increasingly crucial. While approaches like Low-Rank Adaptation (LoRA) [Hu et al., 2022] have demonstrated practical success, they lack a geometric foundation that guides their design philosophy and accounts for their practical effectiveness.Several works study the relationship between differential equations and neural networks, covering areas such as architecture design [Chen et al., 2018] , stablility [Haber and Ruthotto, 2017] , and activation function design [Mantri et al., 2024 , Chelly et al., 2024] . Taking a different approach, this paper proposes using an ordinary differential equation (ODE) to adapt learned neural network weights through a velocity field.We introduce a novel fine-tuning paradigm: geometry-informed fine-tuning of neural networks for multi-task learning (MTL). During training, the weights of over-parameterized neural networks often converge to a low-dimensional hyperplane, with an intrinsic dimension much smaller than the nominal parameter count [Li et al., 2018 , Aghajanyan et al., 2020] . Inspired by this, we hypothesize that the weights of large pre-trained models, such as hierarchical vision transformers [Liu et al., 2021] , lie on a low-dimensional manifold within the high-dimensional parameter space. Fine-tuning can then be modeled as the process of moving these weights on this manifold while optimizing performance on downstream tasks. This geometric perspective on fine-tuning offers several potential advantages:1. It provides a generalized and geometrically grounded approach to fine-tuning, potentially situating existing low-rank adaptation methods, like LoRA [Hu et al., 2022] , within a broader theoretical framework.2. It allows control of inductive biases through the choice of geometric structures and dynamics in the fine-tuning process.3. It enables the use of differential geometry tools to analyze the behavior of fine-tuned models and gain insights into the weight spaces of large pre-trained models.4. It opens new avenues for developing more efficient and interpretable transfer learning techniques, potentially leading to significant reductions in computational resources required for adaptation to new tasks.Our framework introduces a velocity-field-based dynamics approach to formalize the transportation of weights on their manifold during fine-tuning for MTL. This novel perspective bridges the gap between the empirical success of parameter-efficient fine-tuning methods and theoretical understanding, offering both insights into existing techniques and directions for developing new, principled approaches.2 Geometry-Guided Fine-tuningIn this section, we present a conceptual framework for geometry-guided fine-tuning of neural networks, grounded in principles from differential geometry and dynamical systems. Within the context of MTL, we show in Section 2.2 how an ODE can be used to develop a broader theoretical framework. In Section 2.3, we then demonstrate a theoretical connection between our framework and a widely-used parameter-efficient fine-tuning method, LoRA [Hu et al., 2022] . 
background: 
 We begin by formally introducing Multi-Task Learning (MTL). Let T = {T 1 , • • • , T K } represent a set of K tasks. We consider a large model F θ : X → Y with pre-trained parameters θ ∈ R n , where X denotes the input space and Y denotes the output space. For each task T k , we have a task-specific datasetD k = {(x i , y k i )} N k i=1where x i ∈ X and y k i ∈ Y k are the input-output pairs for task k, with a shared input space across all tasks. The MTL objective is formulated asmin θ K k=1 λ k L k (θ; D k ) (1)where L k denotes the task-specific loss function, and λ k are task-specific weighting factors.We hypothesize that the weights of a pre-trained neural network lie on a low-dimensional manifold M within the high-dimensional parameter space. This manifold encapsulates the learned features and structure from the pre-training process [Mao et al., 2024] . Formally, we posit the existence of a smooth manifold M ⊂ R n of dimension d ≪ n such that θ ∈ M. The question then becomes: Can we efficiently fine-tune large pre-trained models by guiding the MTL objective along this manifold M?To achieve this, we constraint the movement of neural network weights θ to M during fine-tuning using the flow of a learned smooth velocity field v ϕ with parameters ϕ. By learning task-specific velocity fields through a neural network, we adapt the pre-trained model to the set of MTL tasks T . Formally, Definition 2.1 (Velocity Field for Fine-Tuning). Let v ϕ : M × R → T M be a smooth vector field on M, where T M is the tangent bundle of M. The evolution of the weights during fine-tuning can be described by the flow of v ϕ given by the ODE:EQUATIONwith the initial condition θ(0) = θ, where θ are the pre-trained weights of the network.The gradient descent step for geometry-guided fine-tuning can be represented using the forward Euler discretization as follows: Definition 2.2 (Geometry-Guided Fine-Tuning). For a smooth velocity field v ϕ , the parameters ϕ and θ are updated during fine-tuning as follows:EQUATIONEQUATION)where p is the iteration of gradient descent, and η is the learning rate.We now show how Low-Rank Adaptation (LoRA) [Hu et al., 2022] can be modeled within our geometric framework, threby providing a theoretical grounding for its empirical success. In LoRA, weight updates are constrained to a fixed low-rank subspace. Using the notation from [Hu et al., 2022] , the weight update of LoRA excluding the gradients of the loss function is given byEQUATIONwhere W 0 ∈ R d×r is the pre-trained weight matrix (frozen), B ∈ R d×r , A ∈ R r×k are learnable small rank matrices with r ≪ min(d, k) and W is the fine-tuned weight matrix. Equation ( 5) is equivalent to one-step discretization of an ODE with a constant (in time) velocity field v ϕ = B A where ϕ are the entries of B and A. That isEQUATIONwith initial condition W(0) = W 0 . This formulation not only provides a geometric interpretation of LoRA but also suggests ways to generalize and improve upon it by considering more complex manifold structures or velocity fields. Moreover, it demonstrates how our framework can recover existing methods under specific choices of velocity fields and discretization techniques, while opening avenues for developing new, theoretically grounded fine-tuning techniques. 
experiments: 
 In this section, we present implementation details of DELORA using the Swin-Tiny backbone [Liu et al., 2021] and conduct preliminary experiments on multi-task learning for dense prediction tasks, including human part segmentation, semantic segmentation, surface normal estimation, and saliency detection.We implement our method using PyTorch and conduct all experiments on a single NVIDIA A100 GPU. Following the approach used in LoRA [Hu et al., 2022] , we restrict our method to only the attention weight matrices W q , W v , which are typically present in the Transformer layer [Vaswani et al., 2017] . For a weight matrix W ∈ R d×k , we compute a parameter-efficient velocity field as follows:v(W, t) = softmax (W 2 W)(W W 1 ) ⊤ √ d (7)where W 1 ∈ R k×r and W 2 ∈ R r×d are learnable projection matrices, r ∈ N is a hyperparameter that determines the projection dimension, typically chosen such that r ≪ min(d, k). Given the combination of an ODE perspective and the low-rank parameterization of the learned velocity field, we refer to our method as DELORA.Setup. We evaluate our proposed geometry-guided fine-tuning approach on the PASCAL-MTL benchmark [Everingham et al., 2010] , following the PASCAL-Context split used in MTLoRA [Agiza et al., 2024] . This dataset, widely used for multi-task learning (MTL) in dense prediction tasks, includes annotations for semantic segmentation, human part segmentation, surface normal estimation, and saliency detection. The PASCAL-Context dataset consists of 4,998 images in the training split and 5,105 images in the validation split. We apply the same data preprocessing and augmentation techniques as described in MTLoRA [Agiza et al., 2024] to ensure a fair comparison.Model Architecture. Following MTLoRA [Agiza et al., 2024] , we implement our approach using a Swin-Tiny transformer backbone [Liu et al., 2021] , pre-trained on the Imagenet22k [Deng et al., 2009] dataset. The model follows an MTL architecture with a shared encoder and task-specific decoder for each of the four tasks. 1 , where we compare DELORA against MT-LoRA [ Agiza et al., 2024] , LoRA [Hu et al., 2022] , single task fine-tuning and MTL full fine-tuning approaches. Our DELORA achieves competitive performance across all tasks:1. Parameter Efficiency: Our DELORA achieves competitive performance while using only 2.42M trainable parameters. Although LoRA has a similar number of parameters, our method achieves better overall performance through joint training and enables single inference for all tasks. 2. Multi-Task Balance: Our approach shows a better balance across tasks, at times even outperforming MTLoRA, while using 4× fewer trainable parameters. This highlights the benefits of an ODE based approach and the choice of the velocity field.These preliminary results provide strong evidence for the potential of geometry-guided fine-tuning. They demonstrate that DELORA can achieve parameter-efficient adaptation while maintaining or enhancing performance across multiple tasks. 
conclusions and discussion: 
 This paper introduces geometry-guided fine-tuning, a paradigm that frames the adaptation of pretrained neural networks within the context of differential geometry and dynamical systems, introducing an early variant called DELORA. Our approach constrains fine-tuning by using the flow of a velocity field on the manifold of neural network parameters, offering several key advantages:1. A broader theoretical framework that encompasses existing methods like LoRA and enables the development of new, principled techniques.2. Insights into neural network behavior during task adaptation, viewed through the lens of manifold structures and dynamical systems.3. Promising empirical results demonstrating competitive performance with significantly fewer parameters.This combination of theoretical grounding and practical efficiency positions geometry-guided finetuning as an promising new direction in transfer learning.Research Directions. Our geometry-guided fine-tuning framework suggests several promising directions for future research:1. Weight Manifold Theory:• Geometric Properties: Examine the curvature, dimensionality, and connectivity of weight manifolds across architectures [Aghajanyan et al., 2020 , Mao et al., 2024] , aiming to better understand model capacity and weight configuration landscapes. • Learning Dynamics: Study how manifold geometry impacts optimization paths, particularly in terms of convergence behavior and efficiency in reaching task-specific weights [Haber and Ruthotto, 2017] .2. Enhanced Fine-Tuning Dynamics:• Tangent Space Linearization: Build on work in tangent space linearization [Ortiz-Jimenez et al., 2024] to enhance weight disentanglement, facilitating task adaptation within our geometry-guided framework. • Advection Techniques: Integrate advection mechanisms to guide weight updates along structured paths [Zakariaei et al., 2024] , potentially improving stability and preserving relevant model properties. • Adaptive Velocity Fields: Develop strategies for learning velocity fields that adaptively direct weights to task-specific configurations, informed by model performance or data.3. Task-Specific Adaptation:• Input-Adaptive Tuning: Enable models to dynamically adjust parameters based on input, using geometry-guided velocity fields for more targeted navigation of the weight manifold. • Geometric Regularization: Apply Riemannian distance [Kirkpatrick et al., 2017] as a regularizer to keep fine-tuned weights close to pre-trained configurations, potentially enhancing generalization.• Convergence Analysis: Investigate conditions for guaranteed convergence and factors influencing convergence rates within geometry-guided tuning. • Generalization Bounds: Establish generalization bounds for geometry-guided finetuning to assess model performance on new data, considering the effects of manifold constraints [Chen et al., 2018] .By pursuing these research directions, we aim to deepen the theoretical understanding of geometryguided fine-tuning and develop practical techniques that enhance the efficiency and effectiveness of transfer learning in neural networks. Incorporating insights from tangent space linearization and task arithmetic offers a promising pathway to more robust and adaptable models. We encourage the research community to engage with these challenges, which hold significant potential for advancing the field. 
 
 Appendix: Preprint. 


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: This paper offers a geometric perspective on LoRA and introduces a new fine-tuning method, DELoRA, by constructing vector fields.
main_review: main_review: ##### Summary of Strengths

- The method is innovative, combining the perspective of ODE to analyze low-rank adaptation.
- The proposed geometry-guided fine-tuning is well-structured and supported by experimental validation in the context of multi-task learning.

##### Summary of Weaknesses

- The paper lacks clarity on the specific implementation of multi-task learning, particularly how DELoRA is utilized for fine-tuning across multiple tasks.
- While the Riemannian metric is introduced, it does not seem to be explicitly discussed or utilized in the experiments.
- Lacks sufficient ablation studies and analysis.
title: title: The paper introduces a novel geometry-guided fine-tuning method for neural networks with promising theoretical backing and parameter efficiency, but its claim of providing a unified framework is less well-substantiated due to limited empirical validation beyond specific cases like LoRA.
main_review: main_review: ## Summary
The paper presents a novel approach to fine-tuning pre-trained neural networks, framed from a geometric perspective. The authors propose geometry-guided fine-tuning, which models the movement of pre-trained weights on a low-dimensional manifold using an ordinary differential equation (ODE)-based framework. This approach is complemented by a Riemannian metric-based regularizer to control the dynamics of weight movement during the fine-tuning process, thereby providing a unified theoretical foundation that connects empirical fine-tuning methods with geometric principles. The authors introduce DELoRA, a method that adapts existing fine-tuning techniques by leveraging differential geometry tools. They demonstrate that DELoRA offers competitive performance across multiple dense prediction tasks—such as semantic segmentation, saliency detection, and surface normal estimation—on the PASCAL-MTL dataset, while achieving reductions in the number of trainable parameters compared to existing approaches.

## Strengths
1. Novel Perspective: The paper introduces a novel geometric perspective on the fine-tuning process. While the theoretical foundation is interesting, it is relatively straightforward and not deeply explored in this work. Nevertheless, the approach offers some insight into the fine-tuning process through the lens of differential geometry.
2. Parameter Efficiency: The proposed method achieves parameter efficiency, using fewer trainable parameters compared to existing methods such as LoRA. This aspect is promising for applications in resource-constrained environments, though the improvement is modest and the evaluation is somewhat limited.
3. Clear Presentation: The paper is clearly written and provides a logical structure, from the introduction of the theoretical framework to the empirical validation. The presentation helps make the core ideas accessible, even if the implementation is not particularly groundbreaking.
4. Experimental Validation: The empirical results, although limited, demonstrate that the proposed method can achieve competitive performance in a multi-task learning setting. The experiments are well-documented and provide a reasonable basis for comparison with existing methods.

## Weaknesses
1. Limited Empirical Demonstration of Generality: The authors only instantiate their geometry-guided fine-tuning (Definition 2.2) in one practical example—its connection to LoRA. This limits the scope of empirical validation and raises questions about the generality and applicability of the proposed framework to other fine-tuning methods. More examples or broader applications would have strengthened the claim of a "unified framework."
2. Clarity of the Unification Claim: The paper claims to provide a "unifying theoretical foundation" for fine-tuning, but this claim is somewhat ambiguous. Although the connection to LoRA is well-explained, it is less clear how broadly applicable the framework is to other fine-tuning techniques or how it generalizes beyond specific cases. Additional discussion on this point, as well as more explicit comparisons to other fine-tuning methods, would be beneficial for clarity.
3. Limited Scope of Evaluation: The experiments are conducted on a single dataset (PASCAL-MTL) with a focus on dense prediction tasks. Evaluating the method on a more diverse set of tasks and datasets could provide a stronger empirical foundation and demonstrate the robustness of the proposed approach.

## Recommendation
While I am not an expert in the fine-tuning of neural networks, I find that the idea is clearly presented, and the geometric perspective on fine-tuning is novel. However, the claim of providing a "unified framework" is less well-substantiated due to limited empirical examples and validation of the approach beyond LoRA. Despite these limitations, given that this paper introduces a promising direction for efficient fine-tuning with reasonable theoretical backing, I believe it passes the bar for a workshop venue. Therefore, I recommend a weak accept.
Suggested Improvements
1. Broader Empirical Validation: To strengthen the claim of generality, it would be helpful if the authors could demonstrate their geometry-guided fine-tuning in more diverse settings beyond the connection to LoRA. Examples could include its application to other fine-tuning strategies or in different model architectures (e.g., CNNs, RNNs).
2. Clarification of Unification Scope: The authors should provide a more detailed discussion of the unification aspect of their framework. Specifically, elaborating on how other fine-tuning methods could be derived or improved using the proposed geometric approach would make the contribution clearer and more compelling.
3. Broader Evaluation: Extending the empirical evaluation to include other datasets or tasks (e.g., text-based tasks or reinforcement learning) could provide additional insights into the method's applicability and limitations. This would strengthen the empirical foundation and support the broader applicability of the approach.
title: title: The paper proposes analyzing the fine-tuning of large pre-trained models using ODEs on a low-dimensional Riemannian manifold.
main_review: main_review: The paper tackles an interesting and timely problem, which is the fine-tuning of the large pre-trained model. They propose to analyze the fine-tuning process through the lens of dynamical systems (via ODEs). In particular, they define the ODE on a Riemmanin manifold representing a low-dimensional space, where the hypothesis is that the parameter lies in this low-dimensional manifold.
A connection with LoRA is given, and experimental results showcase the advantage of framing fine-tuning in terms of ODEs.
The paper is a good trigger for exploring this idea. Furthermore, it is well-written and clear.

I have some comments/questions for the future:

1. It is difficult to see how (1) and (2) are related. Based on the paper, we can frame the weights' evolution as an ODE, but how are they related to the loss function in (2)? The vector field is the gradient of the loss function when it is defined as (2). 

2. In the Section that connects the method with LoRA, how the parameters \theta are related to the low-rank matrices B and A? Is $\theta = BA$? I suggest expanding on this because it is not clear. Looking at the Euler discretization, it seems that BA is the gradient (velocity field), but it isn't easy to follow without clarifying the relationship.

3. How is defined $\mathcal{L}_{task}$ in the experiments? And it says that a regularizer is not used; this means that $ d_g(\theta, \theta_0)$ is 0. Why?

4. How is the practical algorithm of DELoRA compared to LoRA? In DELoRA, the update is given by (3) If this is true, then it is confusing the phrase "Equation (3) is precisely the single-step LoRA update.."

Minor comments:

1. Some important references are missing in the context of fine-tuning pre-trained models; for instance, [1].

2. LoRA citation is the arxiv version instead of the conference paper.
[1] Ortiz-Jimenez, Guillermo, Alessandro Favero, and Pascal Frossard. "Task arithmetic in the tangent space: Improved editing of pre-trained models." Advances in Neural Information Processing Systems36 (2024).


[1] Ortiz-Jimenez, Guillermo, Alessandro Favero, and Pascal Frossard. "Task arithmetic in the tangent space: Improved editing of pre-trained models." Advances in Neural Information Processing Systems 36 (2024).
title: title: Fine-tuning from a geometric perspective
main_review: main_review: The paper proposes to model fine-tuning process as moving weights on a low-dimensional manifold. 

Strength: 1. The paper clearly discusss the advantages of the framework. 2. Empirical evidence demonstrates the effectiveness of the proposed method.

Weakness: 1. Possibly due to constraints on paper length, it is not clear to me how does the concept in Definition 2.2 explains LoRA. 2. Regularization using a Riemannian metric is not new in deep learning; e.g. it appears in continual learning [1]. It might be interesting to discuss the connection.

[1] Overcoming catastrophic forgetting in neural networks, Kirkpatrick et al., https://arxiv.org/abs/1612.00796

Comments: title: Official Response to Reviewers
comment: # General Revisions and Additional Details
In response to all reviewers, we have made the following additional revisions:
- **Broader Theoretical Discussion:** We expanded the discussion on how DELoRA’s geometric framework and its connection to LoRA, setting a foundation for broader application across different fine-tuning methods.
- **Increased Empirical Analysis:** To strengthen our empirical results, we expanded the discussion on multi-task learning and how DeLoRA is used in conjunction with gradient descent.
- **Citation Updates:** We updated citations, including the recommended references on task arithmetic and other relevant works, to better contextualize our contributions within the broader literature.

In summary, we thank all reviewers for their constructive feedback, which has allowed us to enhance the clarity, depth, and theoretical foundations of our extended abstract. We believe the revisions address the key concerns raised and provide a stronger basis for understanding the impact and generalizability of DELoRA within the geometry-guided fine-tuning paradigm.