reviews: 
Comments: The paper explores catastrophic forgetting in continual instruction finetuning across various settings, discovering that model forgetting is influenced by the specific training tasks and the models themselves. It explores correlation between function vectors and catastrophic forgetting, and  propose a function vector-based approach to eliminate catastrophic forgetting. The paper has conducted ablation studies across various tasks, methods and models to demonstrate the generalizability of the method using different llama and mixtral models with classification and generation tasks and evaluating it for supervised finetuning and model generalizability.
Strengths: 1. The idea of using regularization on the function vectors and the function vector guided loss to enable continual instruction fine tuning seems novel and is well motivated in the paper. 
2. The paper presents a strong ablation study for the proposed methods across various settings, demonstrating their generalizability by conducting experiments using multiple llama checkpoints, classification and generation tasks and evaluating across generalizability and supervised finetuning on multiple datasets.
3. The experiment studies which clearly demonstrate the ability of  function vectors to significantly improve the performance on continual instruction fine tuning.
Weaknesses: 1. There is a lack of clarity in the proposed methodology. i.e function vector guided training design. It would help to understand  the proposed methodology by providing the overall  the optimisation objective and end-to-end training algorithm. In the current state, it is difficult to reproduce the results that the authors mentioned in the paper without the details on the proposed methodology. 
2. The  presentation  of the paper and the readability of some sections are difficult to follow.   Notations are unnecessarily complicated. 
3. The experimental results though convincing across most of the experimental setups, its hard to make decisive conclusions especially on the core idea of function vectors and their effectiveness in mitigating catastrophic forgetting.
4. Some experiments and  observations made related to catastrophic forgetting are redundant and may not contribute much. For example, forgetting will be model-dependent since performance on different datasets and tasks is model-dependent.

More details on the above mentioned points are listed below.

1. The authors claim that “Forgetting coincides with changes in FV similarity between tasks”. However, it is difficult to observe the same from the evidence provided because the trend is not constant across all tasks and models; sometimes, a slight change in similarity changes the task performance drastically(i.e. Fig 2(a) on count object and hellaswag datasets), in some cases even though similarity is decreasing, the performance is increasing. It is very difficult to establish a clear relation from the empirical results provided. A suggestion is to  extend the number of tasks or conduct a statistical test using correlation metrics. 
2. The observation that generation tasks lead to greater forgetting cannot be measured directly, even though the same evaluation metric (Rouge) is used for both types of tasks. Rouge is more sensitive to longer sequences because it relies on n-grams overlap. The tasks have different difficulties; one way to measure them is by comparing them against some upper bound baseline and calculating forgetting in terms of percentage loss.
3. The computation of the FV guided loss (eqn 4) is not very clear. If the set S consists attention head with top-10 CE,  is that going to be same before training on the current task and after training. Is this loss recomputed at every iteration of the optimization step? 
4. Although function vectors improve the performance of existing CL methods, it can be helpful to compare against  finetuning models individually, to see how much gap there is between methods with function vector guided fine tuning and the optimal performance that can be achieved by fine tuning.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 2