Venue: NAACL 2025

# Enhancing Language Model Hypernetworks with Restart:

A Study on Optimization

Yihan Zhang\({}^{1}\),  Jie Fu\({}^{2}\),  Rongrong Ji\({}^{3}\),  Jie Chen\({}^{1,4}\)

\({}^{1}\) School of Electronic and Computer Engineering, Peking University, Shenzhen, China

\({}^{2}\) Shanghai AI Lab  \({}^{3}\) Xiamen University

\({}^{4}\) Pengcheng Laboratory, Shenzhen, China

ariszhang@stu.pku.edu.cn, fujie@pjlab.org.cn, rrji@xmu.edu.cn, chenj@pcl.ac.cn

Corresponding Author.Code is publicly available for research purposes: [https://github.com/Aris-z/HyperRestart](https://github.com/Aris-z/HyperRestart)

###### Abstract

Hypernetworks are a class of meta-networks that generate weights for main neural networks. Their unique parameter spaces necessitate exploring suitable optimization strategies to enhance performance, especially for language models. However, a comprehensive investigation into optimization strategies for hypernetworks remains absent. To address this gap, we analyze the loss landscape of hypernetworks and propose that restart optimization strategies can improve their performance for language models. We find that hypernetworks have inherently more complicated loss landscapes compared to conventional networks due to their distinct parameter spaces. Consequently, a restart strategy that periodically resets the learning rate can facilitate better convergence for hypernetworks. Through experiments on instruction tuning and multi-task training, we demonstrate that the restart strategy consistently enhances the performance of hypernetworks for language models, often more effectively than for conventional deep neural networks. Our findings highlight the importance of tailored optimization techniques to unlock the full potential of hypernetworks in natural language processing tasks1.

Footnote 1: Code is publicly available for research purposes: [https://github.com/Aris-z/HyperRestart](https://github.com/Aris-z/HyperRestart)

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 7826–7838

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 7826–7838

## 1 Introduction

In recent research, large language models (LLMs) have been proven to be useful tools for learning complicated representations. However, LLMs have limitations regarding their weights and architecture, typically fixed after training. Hypernetworks, proposed by Ha et al. (2022), use one small neural network to generate the weights for the main network, which attracts great attention for their potential to address this gap. Hypernetworks create a new parameter space distinct from the one created by the main network. Given the significance of model training strategies in the overall training process, it is highly advantageous to investigate optimization techniques within this new parameter space. The existing hypernetworks usually adopt an optimization strategy similar to the standard training of a general LLMs Ye and Ren (2021); Ivison and Peters (2022). This means that they do not specifically conduct comparative research on different optimizers and schedulers during training, which we believe finally prevents hypernetworks from being used to their full potential.

To address the question above, we conduct a study on the optimization of hypernetworks for language models (LMs). When considering the optimization of the neural networks, their loss landscapes depict the direction of optimization. This makes loss landscape important when we try to bridge this gap between conventional LMs and hypernetworks.

The restar

Figure 1: Rugged loss landscapes of Hypernetwork

more rugged compared to the main network, with more local minima and saddle points, as shown in Figure 1. This makes traditional optimization methods prone to getting stuck in these areas and unable to proceed further. The restart strategy introduces a certain perturbation to the model and allows it to escape from the blocked regions and more easily reach the global optimum.

In this work, we conduct a series of experiments to investigate the influence of the restart strategy in hypernetworks for language models. Our investigation also includes an analysis of minima and saddle points, which provides a foundation for the application of the restart strategy in this context. We conduct a series of experiments to assess the robustness and generalizability of the restart strategy when applied to hypernetworks for LMs. As a result, we reveal that the restart strategy enhances the performance of hypernetworks when used for LMs. Notably, this effect is often more obvious in hypernetworks than in conventional LMs.

In summary, our contributions are as follows:

* We conduct an empirical study of the loss landscape of hypernetworks and conventional LLMs, concentrating on their structural differences and point out the similarities and differences between hypernetwork and conventional network.
* We explore the impact of restart phases in the training of hypernetworks, with a special emphasis on their role in instruction tuning for LLMs.
* We provide a comprehensive set of experimental results across diverse datasets and hypernetwork architectures, demonstrating the effectiveness and generalization of the restart strategy in enhancing hypernetwork performance.

## 2 Related Work

### Restart in Optimization

To enhance optimization in deep neural networks, restart techniques are increasingly used to navigate complicated loss landscapes and avoid local minima. Adaptive warm restart O'donoghue and Candes (2015) initiates restarts based on the objective function's behavior or the gradient's orientation. The learning rate's role is pivotal in this context, as it dictates the convergence speed towards the global minimum Mishra and Sarawadekar (2019).

Smith (2017) introduced the cyclical learning rate (CLR) policy, which varies the learning rate within defined boundaries. loshchilov and Hutter (2022) adapted restart methods to stochastic gradient descent (SGD), implementing a cosine annealing schedule to periodically reset the learning rate.

The main obstacle in loss function optimization is often saddle points characterized by small gradients and can decelerate learning Dauphin et al. (2014); Choromanska et al. (2015). Restart strategies have proven effective in traversing these regions more quickly Dauphin et al. (2015). Large language models (LLMs) employ learning rate schedules that include warmup and decay phases during pre-training Zhao et al. (2023), and recent approaches suggest that resetting the learning rate during continual pre-training can further enhance performance Gupta et al. (2023). These strategies are crucial for optimizing the learning process in complicated loss landscapes.

### Hypernetworks

Hypernetworks Ha et al. (2022) are a novel class of neural networks that generate the parameters for a main deep neural network (DNN). Instead of learning parameters of a main DNN directly, the hypernetworks produce these weights Chauhan et al. (2023a). This paradigm has given rise to hyperDNNs, which encompass DNNs utilizing hypernetworks for various applications Mahabadi et al. (2021); Ivison and Peters (2022); Volk et al. (2022); Xiao et al. (2023); Ye and Ren (2021).

Figure 2: Illustration of general hypernetworks. \({\cal F}(X;\Theta)\) is the main network that generates the outputs corresponding to different inputs \(X\). \({\cal H}(C;\Phi)\) is the hypernetwork whose outputs \(\Theta\) are the weights or other parameters of the main network. \(C\) is a learnable task embeddings and \(\Phi\) represents the hypernetworks’ parameters.

 

Hypernetworks are characterized by their ability to facilitate soft weight sharing across multiple tasks, enhancing multi-task learning and transfer learning capabilities Ye and Ren (2021); von Oswald et al. (2020); Chauhan et al. (2023b). They also enable the creation of data-adaptive DNNs, where the hypernetwork tailors the main network's weights to the specific data at hand Ye and Ren (2021). Furthermore, hypernetworks often have fewer weights than traditional DNNs, offering a form of parameter efficiency that is particularly advantageous in resource-constrained scenarios Zhao et al. (2020); Ivison and Peters (2022).

Recent advances in parameter-efficient tuning for NLP models, despite their effectiveness, are often limited to low data scenarios such as zero-shot or few-shot. Ivison et al. (2023) address this by introducing Hypernetworks for Instruction Tuning (HINT). In parallel, He et al. (2022) propose HyperPrompt that leverages a hypernetwork as a global memory for query attention. These hypernetwork-based approaches promise to deliver new task adaptability with controlled computational and parameter overhead. Because of the general application of hypernetworks, investigating how to unlock the potential of hypernetworks is important. Considering that the structure of hypernetworks may be different from that of ordinary DNNs, we hope to find better optimization methods from their differences.

## 3 Preliminaries

In the following part, we refer to the hypernetwork and the corresponding main network whose parameters are not generated by the hypernetwork but are learnable, when we analyze the difference in loss landscape between the hypernetwork and the main network.

### Landscape of Hypernetworks

For a pair of input data \((X,Y)\), we denote the common language model as \({\cal F}\), where the model forward is represented by \({\cal F}(X;\Theta)=Y\), and \(\Theta\) is the set of weights updated via backpropagation. However, in hypernetwork language models (hyperLMs), there is a hypernetwork \({\cal H}(C;\Phi)\) to generate the weights \(\Theta\) of the main network \({\cal F}\), where \(C\) is the input to the hypernetwork. The optimization problems for the hyperLM can be simply written as given below:

\[\min_{\Theta}{\cal L}({\cal F}(X;\Theta),Y)\] \[\rightarrow \min_{\Phi}{\cal L}({\cal F}(X;{\cal H}(C;\Phi)),Y), \tag{1}\]

where \({\cal L}\) is the loss function. Following the formula above, the general gradient update strategy is formulated as: \(\theta^{n}_{m}[t]=\theta^{n}_{m}[t-1]-\eta\frac{\partial{\cal C}}{\partial \theta^{n}_{m}[t-1]}\). In hypernetworks, main network weights \(\Theta\) are the output of hypernetwork \({\cal H}(C;\Phi)\). Then, the gradient of hypernetwork weights is: \(\frac{\partial{\cal L}}{\partial\phi^{i}_{j}}=\sum\limits_{m,n}\frac{\partial {\cal L}}{\partial\theta^{n}_{m}}\frac{\partial\theta^{n}_{m}}{\partial\phi^{ i}_{i}}\), \(\frac{\partial{\cal C}}{\partial\theta^{n}_{m}}\) is the main network's weight gradient, \(\theta^{n}_{m}\) is the \(n\)th parameter in m layer of main network and \(\phi^{j}_{i}\) is the \(j\)th parameter in j layer of hypernetwork. Now we suppose there is a set: \(\mathbb{Y}=\left\{\Theta_{1},\dots,\Theta_{i},\dots\right\}\). The element \(\Theta_{i}\) in \(\mathbb{Y}\) are the parameters such that gradients of the main network are equal to 0. The existence of set \(\mathbb{Y}\) is obvious. It is equivalent to the existence of minima (both local and global) and saddle points in neural networks.

One part of our motivation is based on the fact that the main problem of high-dimensional non-convex optimization is saddle points Dauphin et al. (2014). Given that most neural networks satisfy this condition, we argue that the generalizability of this assumption is valid.

When the parameter \(\theta^{n}_{m}\) is generated by hypernetwork \({\cal H}(C;\Phi)\), the gradient of hypernetwork parameter \(\Phi\) is:

\[\begin{split}&\frac{\partial{\cal L}}{\partial\phi^{i}_{i}}=\sum \limits_{m,n}\frac{\partial{\cal L}}{\partial\theta^{n}_{m}}\frac{\partial \theta^{n}_{m}}{\partial\phi^{j}_{i}}=0.\\ &\mathrm{s.t.}\quad\forall i,j,\phi^{j}_{i}\in\Phi\end{split} \tag{2}\]

This means if there is a set \(\mathbb{X}=\left\{\Phi_{1},\dots,\Phi_{i},\dots\right\}\), whose elements are the parameters such that gradients of hypernetwork equal to 0. We note \({\cal H}^{-1}\) as the inverse transformation of \({\cal H}\), then

\[\begin{split}&{\cal H}^{-1}(C;\mathbb{Y})\subseteq\mathbb{ X}\\ &\mathrm{s.t.}\quad{\cal H}^{-1}(C;\mathbb{Y})=\bigcup{\cal H}^{- 1}(C;\Theta_{i}).\quad\Theta_{i}\in\mathbb{Y}\end{split} \tag{3}\]

In other words, the saddle points or minimum points of the main network are also those of the hypernetwork. Especially \(\frac{\partial{\cal L}}{\partial\phi^{j}_{i}}\) is sum of main network's gradients \(\frac{\partial{\cal L}}{\partial\theta^{n}_{m}}\), which can be close to zero while each part \(\frac{\partial{\cal L}}{\partial\theta^{n}_{m}}\frac{\partial\theta^{n}_{m}}{ \partial\phi^{j}_{i}}\) is not.

### Alignment with Main Networks

In section 3.1, we find hypernetwork has a cumulative number of saddle points and minima that is at 

least as great as the main network's. It is natural for us to figure out if there is a guarantee that the saddle points or minima of the main network align with those of the hypernetwork. Because the geometry of loss landscapes of models is characterized by the Hessian matrix of the loss function, we focus on analyzing the Hessian of hypernetworks. We denote the Hessian matrix of hypernetwork as \(H_{\phi}=\frac{\partial^{2}\mathcal{L}}{\partial\phi_{i}^{T}\partial\phi_{k}^{ T}}\) where \(i,j,k,l\in\{1,\ldots,|\Theta|\}\) and the Hessian matrix of main network as \(H_{\theta}=\frac{\partial^{2}\mathcal{L}}{\partial\theta_{i}^{T}\partial\theta _{k}^{T}}\) where \(i,j,k,l\in\{1,\ldots,|\Theta|\}\). Then we can find that

\[H_{\phi}=J_{\phi}^{T}H_{\theta}J_{\phi}, \tag{4}\]

where \(J_{\phi}\) is the Jacobian matrix of main network. The detailed proof can be found in Appendix A. Note that the Jacobi matrix usually becomes degenerate at the end of training because of instrisic dimensionHu et al. (2018). But at the beginning of training \(J_{\phi}\) is not as degenerate as it is at the end of training. This suggests that the two Hessian matrices likely share the same positive and negative inertia coefficients.

To further validate that, we conduct experiments in Figure 3. It provides a visual representation of the hypernetwork and the corresponding network, both in their trained and untrained states. Our aim is to investigate the alignment of the loss landscape between hypernetworks and main networks. This visualization was generated using the methodology proposed by Li et al. (2018). The figure was plotted with an x-axis and y-axis range of [-10,

Figure 3: (a) and (b) illustrate that after training, the hypernetwork maintains a complicated loss landscape, in contrast to the main network, which exhibits a simplified landscape. And the hypernetwork’s convergence to its minimum coincides with the main network reaching a saddle point. This corresponds to our results in section 3.2, which means the zero gradient points (minimum or saddle point) are corresponding. But it doesn’t mean that the saddle points of the main network are also the saddle points of the hypernetwork. In this figure, it is the saddle point of the main network corresponding to the minimum of the hypernetwork. (c) and (d) demonstrate that, prior to training, both networks exhibit comparably intricate loss landscapes, reflecting the anticipated similarity in their initial optimization topography. This also corresponds to our results in section 3.2, which means the optimization route and difficulty of the two models in the very early stage of training will be similar.

 

10] at a resolution of 256 steps. During the visualization process, specific parameters were fixed, and the Xavier initialization technique [34] was applied. Figure 3 (a) and (b) illustrate that after training, the hypernetwork maintains a complicated loss landscape, in contrast to the main network, which exhibits a simplified landscape. Notably, the hypernetwork's convergence to its minimum coincides with the main network reaching a saddle point, corroborating our prior analysis. Conversely, (c) and (d) demonstrate that, prior to training, both networks exhibit comparably intricate loss landscapes, reflecting the anticipated similarity in their initial optimization topography.

This ensures that, for hypernetworks, using some optimization methods from conventional LMs will not cause any performance loss at the beginning, but the results show that at the end of the optimization, the hypernetwork still maintains a relatively complex loss landscape while general LMs tend to flatten out. This requires special methods such as restart to help the model achieve the similar convergence effect as conventional LMs. The restart strategy's dynamic learning rate adjustments facilitate the model's ability to navigate complicated loss landscapes more effectively. By periodically resetting the learning rate, the strategy helps the model avoid local minima and encourages exploration, which can lead to better generalization on unseen data. It simulates a new warm-restarted run once \(T_{i}\) epochs are performed, where \(i\) is the index of the run. Its learning rate can be formulated as:

\[\eta_{t}=\eta_{min}+\frac{1}{2}(\eta_{max}-\eta_{min})(1+cos\frac{T_{cur}}{T_ {i}}\pi) \tag{5}\]

, where \(\eta_{max}\) is set to the initial learning rate, \(T_{cur}\) is the number of epochs since the last restart and \(T_{i}\) is the number of epochs between two warm restarts.

To enhance model convergence, we train the model for a total of \(\sum_{i}T_{i}\) epochs, where \(T_{i}\) is the number of epochs between two restarts. This approach entails halting the training process when the learning rate reaches its minimum at a given epoch.

## 4 Experiments

Our theoretical derivations and methodologies aim to reveal the problem of hypernetwork optimization and investigate the corresponding solution. In the following part, we mainly use two hypernetworks, hyperdecoders and hyperformer. Hyperdecoders is a type of hypernetwork that generates the parameters of adapters in decoders and the adapters and FFNs are connected in parallel. Hyperformer is another type of hypernetwork that generates the parameters of layer normalization and FFNs in adapters and the adapters and FFNs are connected in series.

### Instruction Tuning for Hypernetwork Restart

Due to the popularity of instruction tuning and the success of hypernetworks on instruction tuning, we first investigate instruction tuning on hypernet

Figure 4: Rouge-L Fmeasure and loss of models on P3 training split. The legend on the left corresponds to the Rouge-L score and loss. The legend on the right is the corresponding learning rate. The solid line corresponds to the left legend, and the dashed line corresponds to the right legend.

 

works. In this section, we will try to transfer hypernetwork to instruction datasets to evaluate the effectiveness of restart for instruction tuning.

**Setup** In this experiment, we have opted to utilize Hyperdecoders based on the FLAN-T5 model and FLAN-T5 (Chung et al., 2022) baseline to compare the convergence of hyperLMs and conventional LMs, both with and without the use of restarts. P3 is a collection of prompted English datasets covering a diverse set of NLP tasks including question answering, dialogue, text generation, text editing, reasoning, etc., providing rich instruction examples for the model. We train these models on P3 (Public Pool of Prompts) T0 split Sanh et al. (catalog=Sanh et al.), which is a multi-task instruction following dataset including 193 tasks. We use the AdamW optimizer in all subsequent experiments due to its proven effectiveness with transformer architectures and hypernetworks. Following the setup of Ivison et al. (2023) that uses RougeL as the evaluation metric, we hope the performance of these models can be properly measured.

Our baseline in this scenario is the Hyperde-coders and FLAN-T5 models without restart. The FLAN-T5 and Hyperdecoders model is simply trained using the P3 dataset above. Because FLAN-T5 has been well-trained on instruction datasets, it is more reasonable to choose it as a baseline if we want to review the performance of hypernetworks on instruction tuning. For cosine annealing restart, we use a learning rate of \(1e-4\), \(\eta_{min}=1e-6\), \(T_{0}=1\) and \(T_{mul}=2\).

**Experiments** The result is shown in Figure 4 and we also show the final validation score in Table 2. In line with our prediction, though each time the result occurs the loss and the accuracy become worse, the results at the end of two restart intervals become better and finally outperform the model without restart. Notably, we set the restart learning rate as the same as without restart. From the perspective of the mean, this means we use a smaller learning rate in restart than in no restart, which results in a slightly higher loss. However, despite training the same number of steps with a smaller learning rate, the restart strategy still achieves better accuracy, demonstrating the value of this method.

Note that the data used for training is less than one epoch, which means the restart strategy effectively helps models (not only including hypernetwork models) converge fast. Moreover, at the end of the final training phase, the performance of models with the restart is much better than models with no restart, but finally, it slows down the convergence speed and reduces the gap between them. This may be because \(\eta_{min}\) of the restart strategy is small, and the learning rate in the end is too small.

To verify this hypothesis, we carried out a series of subsequent experiments, each with a distinct value of \(\eta_{min}\). The impact of varying \(\eta_{min}\) on the outcome is depicted in Figure 5. Our results indicate that an excessively small \(\eta_{min}\) (specifically, when set to 0 in our experiment) leads to suboptimal performance compared to other values. However, it is also noteworthy that a large \(\eta_{min}\) does not necessarily guarantee improved results. This suggests a non-linear relationship between \(\eta_{min}\) and the performance of our model, emphasizing

\begin{table}
\begin{tabular}{l|r r r r r r r r r|r} \hline \multicolumn{2}{c|}{Model} & \multicolumn{1}{c}{CoLA} & \multicolumn{1}{c}{SST-2} & \multicolumn{1}{c}{MRPC} & QQP & STS-B & MNLI & QNLI & RTE & Avg \\ \hline \multirow{8}{*}{Baseline} & Finetuning* & 47.43 & 99.30 & 90.85 & 92.31 & 83.50 & 84.04 & 98.60 & 78.83 & 84.35 \\  & Hyperformer* & 48.18 & 97.48 & 92.14 & 91.51 & 88.93 & 85.63 & 94.59 & 79.56 & 84.75 \\  & Hyperformer+++ & 48.59 & 97.40 & 91.10 & 91.94 & 89.31 & 85.71 & 94.79 & 84.98 & 85.47 \\ \hline \multirow{8}{*}{**+ restart**} & Finetuning & 42.27 & 99.10 & 90.07 & 92.14 & 87.90 & 84.77 & 98.80 & 81.02 & 84.50 \\  & (-5.16) & (-0.20) & (-0.78) & (-0.17) & (+4.40) & (+0.20) & (+2.19) & (+0.15) \\ \cline{1-1}  & \multirow{2}{*}{Hyperformer} & 51.41 & 98.00 & 92.08 & 91.53 & 88.88 & 86.24 & 96.00 & 83.21 & 85.91 \\  & & (+3.23) & (+0.52) & (-0.06) & (+0.02) & (-0.05) & (+0.61) & (+1.41) & (+3.65) & (+1.16) \\ \cline{1-1}  & \multirow{2}{*}{Hyperformer++} & 49.12 & 97.80 & 94.29 & 92.07 & 89.35 & 86.02 & 94.89 & 81.75 & 85.66 \\  & & (+0.53) & (+0.40) & (+3.19) & (+0.13) & (+0.04) & (+0.31) & (+0.10) & (+0.73) & (+0.19) \\ \hline \end{tabular}
\end{table}
Table 1: Result of models on GLUE datasets. For MRPC, QQP and STS-B, we report the F1 score and for other tasks, we report accuracy. The green scores indicate improved performance on these datasets with the restart, while red scores indicate reduced performance. The \(*\) means the result of our reproduction.

 

the importance of carefully selecting this parameter. Overall, we find restart strategy shows better results on the benchmark and outperforms models without restart, and this conclusion still holds for hypernetworks.

### Comparison to Existing SOTA Hypernetworks

SetupTo validate the effectiveness of the state-of-the-art hypernetwork model, we hope to employ the HINT. The authors demonstrated exceptional performance by training the HINT model on the T0 P3 dataset using TPUs and Google Bucket. However, due to the unavailability of these resources, we instead utilize Hyperdecoders as mentioned earlier, and apply them to the MRQA dataset (Fisch et al., 2019) to assess both in-domain and out-of-domain effectiveness. The MRQA dataset comprises 6 datasets for training and evaluation, namely HotpotQA (Yang et al., 2018), Natural Questions (Kwiatkowski et al., 2019), NewsQA Trischler et al. (2017), SQuAD Rajpurkar et al. (2016), SearchQA Dunn et al. (2017), and TriviaQA Joshi et al. (2017), as well as 6 additional datasets for out-of-domain evaluation, which include BioASQ Tsatsaronis et al. (2015), DROP Dua et al. (2019), DuoRC Saha et al. (2018), RACE Lai et al. (2017), RelationExtraction Levy et al. (2017), and TextbookQA Kembhavi et al. (2017). We conduct a full fine-tuning of these datasets using a global batch size of \(256\) and a learning rate of \(3e-4\).

ExperimentsTable 3 and Table 4 show the result on MRQA dataset. In the in-domain MRQA validation set, models employing restart techniques demonstrate a marked improvement over the baseline Hyperdecoders. This suggests that these techniques may facilitate faster convergence and thus enhance performance on tasks within the model's training domain. However, the effectiveness of the restart technique is less consistent in the out-of-domain MRQA validation set. While it outperforms the baseline in some datasets, it does not in others. This indicates that the benefits of the restart approach may not generalize across all tasks, particularly those that lie outside the model's initial training domain.

### Comparison to Other Hypernetworks

SetupWe follow the set and model of Mahabadi et al. (2021) and evaluate the performance on the GLUE benchmark Wang et al. (2018). We select this model due to its unique hypernetwork architecture that generates adapters in both the encoder and decoder. This distinguishes it from our previously discussed model. For simplicity, we only conduct the multi-task training because of its popularity in recent works. This benchmark we use covers paraphrase detection (MRPC, QQP), sentiment classification (SST-2), natural language inference (MNLI, RTE, QNLI), and linguistic acceptability (CoLA). We fine-tune the models with a learning rate of 3e-4 (for restart strategy we make the \(\eta_{max}=3e-4\)). The other hyperparameters are selected similarly as Mahabadi et al. (2021). For better comparison, we choose the finetuning model and the Hyperformer model without restart as a baseline.

ExperimentsTo further study the generalization of this method, we conduct the following experiments to verify this strategy works on most hypernetworks. As evidenced by our experimental results presented in Table 1, substantial enhancements can be realized on standard datasets. With restart, hypernetwork can universally increase performance on nearly every task. The benefits of implementing restarts are especially noticeable in the MRPC, CoLA, and RTE tasks, where the hypernetwork model experiences a 3-point improvement. In turn, this contributes to a significant elevation in the average performance across these tasks. Fascinatingly, the restart strategy does not confer the same degree of improvement on the standard T5 model as it does on the hypernetwork. In fact, in the CoLA task, the application of the restart strategy results in a 5-point performance decrease compared to the model without restarts. This suggests that the

Figure 5: The rouge-L score of different restart \(\eta_{min}\) of models on GLUE dataset. The legend on the left corresponds to the Rouge-L score and the legend on the right is the corresponding learning rate. The solid line corresponds to the left legend, and the dashed line corresponds to the right legend



efficacy of the restart strategy may be different on the hyperLMs and conventional LMs, underscoring the need for careful consideration when applying such strategies. Moreover, our results indicate that the restart strategy generally leads to improved performance across most tasks when applied to hypernetwork models. Notably, the Hyperformer model with restarts outperforms all other models in terms of average performance, demonstrating the potential of this approach.

### Comparison to Other Learning Rate Schedulers

To evaluate the effectiveness of the restart strategy against other learning rate schedulers, we conducted experiments using the Hyperdecoders on the GLUE benchmark. Our comparison includes a variety of learning rate schedules: constant, linear with warmup, cosine with warmup, and polynomial decay with warmup. The results, presented in Table 5, demonstrate that the restart strategy surpasses its counterparts. This strategy consistently achieved the highest scores, indicating its robustness in handling the diverse set of tasks.

The restart strategy's success can be largely attributed to its dynamic learning rate adjustments, which facilitate the model's ability to navigate complicated loss landscapes more effectively. By periodically resetting the learning rate, the strategy helps the model avoid local minima and encourages exploration, which can lead to better generalization on unseen data. Moreover, the periodic nature of the restart strategy acts as a form of implicit regularization. This can potentially reduce the risk of overfitting(Loshchilov and Hutter, 2022).

## 5 Limitations

As previously discussed, the hypernetwork introduces a novel parameter space, making it more suitable for incorporating a new restart strategy. We have demonstrated the effectiveness of this approach and hope to explore these directions further in future work. The hypernetwork architecture is not only applicable to natural language processing but also raises the question of whether it can be effectively applied to other modalities, such as images or multimodal data. Additionally, due to the efficiency challenges associated with hypernetworks, we have not yet extended our experiments to larger-scale language models. Exploring solutions to these challenges remains an important direction for future research.

## 6 Conclusions

To address the problems due to the introduction of novel parameter spaces, we analyze the loss landscape of hypernetworks and main networks. We propose that the restart strategy for the learning rate can improve the performance and conver

\begin{table}
\begin{tabular}{l|c c c c c c|c} \hline Model & BioASQ & DROP & DuoRC & RACE & Relation Ext. & TextbookQA & Avg \\ \hline Hyperdecoders & 54.65/68.82 & 36.264(5.33) & 48.9078.74 & 32.94/46.35 & 74.14/85.44 & 47.04/55.66 & 48.99/60.04 \\ \cline{2-7}  + **restart** & 51.93/66.24 & 37.46/45.83 & 47.17/57.38 & 34.12/47.98 & 72.92/84.60 & 46.17/56.26 & 48.30/59.72 \\  & (-2.272)/(-2.58) & (+1.20)/(+0.50) & (-1.73)/(-1.36) & (+1.16)/(+1.63) & (-1.22)/(-0.84) & (-0.87)/(+0.60) & (-0.69)/(-0.32) \\ \hline \end{tabular}
\end{table}
Table 4: Accuracy and F1 score of models on out-domain MRQA validation split. The green scores indicate improved performance on these datasets with the restart, while red scores indicate reduced performance.

\begin{table}
\begin{tabular}{l|c c c c c|c|c} \hline Model & SQuAD & HotpotQA & TriviaQA & NewsQA & SearchQA & NaturalQs & Avg \\ \hline Hyperdecoders & 84.61/91.22 & 62.63/78.43 & 68.82/73.73 & 53.09/67.63 & 77.31/82.29 & 64.99/77.19 & 68.58/78.41 \\ \cline{2-7}
**+ restart** & 84.38/91.48 & 63.77/79.36 & 70.31/74.83 & 54.30/68.74 & 77.78/82.79 & 66.79/78.96 & 70.00/79.36 \\  & (-0.23)/(+0.26) & (+1.14)/(+0.93) & (+1.49)/(+1.10) & (+1.21)/(+1.11) & (+0.47)/(+0.50) & (+1.80)/(+1.77) & (+1.42)/(+0.95) \\ \hline \end{tabular}
\end{table}
Table 3: Accuracy and F1 score of models on in-domain MRQA validation split. The green scores indicate improved performance on these datasets with the restart, while red scores indicate reduced performance.

\begin{table}
\begin{tabular}{l l} \hline \hline Scheduler & GLUE \\ \hline Constant & \(83.23\pm 0.29\) \\ Cosine & \(83.27\pm 0.33\) \\ Linear & \(84.18\pm 0.20\) \\ Polynomial & \(83.43\pm 0.27\) \\ Restart & **84.88 ± 0.33** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Accuracy of different strategies for Hyperdecoders on GLUE datasets.

 

gence speed of hypernetworks. We conduct our experiments on three different tasks and across two different hypernetwork models: Hyperformer and Hyperdecoders. We anticipate that our work will stimulate further research into hypernetworks optimization, leading to models that are more closely aligned with conventional LMs. Additionally, we hope our findings will inspire more research into improving hypernetworks optimization strategies.

## Acknowledgments

This work was supported in part by the National Key R&D Program of China (No. 2022ZD0118201), the Shenzhen Medical Research Funds in China (No. B2302037), Natural Science Foundation of China (No. 61972217, 32071459, 62176249, 62006133, 62271465), and AI for Science (AI4S)-Preferred Program, Peking University Shenzhen Graduate School, China.

## References

* C. M. Bishop, A. Krizhevsky, S. Sutskever, and J. E. Hinton (2016)A deep learning approach to solve problems in neural networks. In Advances in Neural Information Processing Systems, pp. 1-10. Cited by: SS2.
* V. Kumar Chauhan, J. Zhou, P. Lu, S. Molaei, and D. A. Clifton (Eds.), pp. 2023a. External Links: Link Cited by: SS2.
* V. K. Chauhan, J. Zhou, S. Molaei, G. Ghosheh, and D. A. Clifton (2023b)Dynamic inter-treatment information sharing for heterogeneous treatment effects estimation. arXiv preprint arXiv:2305.15984. Cited by: SS2.
* A. Choromanska, M. Henaff, M. Mathieu, G. Ben Arous, and Y. LeCun (2015)The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pp. 192-204. Cited by: SS2.
* H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al. (2022)Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. Cited by: SS2.
* Y. Dauphin, H. De Vries, and Y. Bengio (2015)Equilibrated adaptive learning rates for non-convex optimization. Advances in neural information processing systems28. Cited by: SS2.
* Y. N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli, and Y. Bengio (2014)Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. Advances in neural information processing systems27. Cited by: SS2.
* D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner (2019)Drop: a reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Vol. 1, pp. 2368-2378. Cited by: SS2.
* M. Dunn, L. Sagun, M. Higgins, V. Ugur Guney, V. Cirik, and K. Cho (2017)Searchqa: a new qca dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179. Cited by: SS2.
* A. Fisch, A. Talmor, R. Jia, M. Seo, E. Choi, and D. Chen (2019)Mraq 2019 shared task: evaluating generalization in reading comprehension. In 2nd Workshop on Machine Reading for Question Answering, MRQA@ EMNLP 2019, pp. 1-13. Cited by: SS2.
* X. Glorot and Y. Bengio (2010)Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, Cited by: SS2.
* K. Gupta, B. Therien, A. Ibrahim, M. I. Richter, Q. Anthony, E. Belilovsky, I. Rish, and T. Lesort (2023)Continual pretraining of large language models: how to (re) warm your model?. arXiv preprint arXiv:2308.04014. Cited by: SS2.
* D. Ha, A. M. Dai, and Quoc V. Le (2022)HyperNetworks. In International Conference on Learning Representations, Cited by: SS2.
* Y. He, S. Zheng, Y. Tay, J. Gupta, Yu Du, V. Aribandi, Z. Zhao, YaGuang Li, Z. Chen, D. Metzler, et al. (2022)Hyperprompt: prompt-based task-conditioning of transformers. In International Conference on Machine Learning, pp. 8678-8690. Cited by: SS2.
* E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2023)Lora: low-rank adaptation of large language models. In International Conference on Learning Representations, Cited by: SS2.
* H. Ivison, A. Bhagia, Y. Wang, H. Hajishirzi, and M. Peters (2023)Hint: hypernetwork instruction tuning for efficient zero-and few-shot generalisation. In The 61st Annual Meeting Of The Association For Computational Linguistics, Cited by: SS2.
* H. Ivison and M. E. Peters (2022)Hyperdecoders: instance-specific decoders for multi-task nlp. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1715-1730. Cited by: SS2.
* M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer (2017)Triviaqa: a large scale distantly 

supervised challenge dataset for reading comprehension. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1601-1611.
* Kembhavi et al. (2017) Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannanoh Hajishirizi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In _Proceedings of the IEEE Conference on Computer Vision and Pattern recognition_, pages 4999-5007.
* Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. _Transactions of the Association for Computational Linguistics_, 7:453-466.
* Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 785-794.
* Levy et al. (2017) Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In _21st Conference on Computational Natural Language Learning, CoNLL 2017_, pages 333-342. Association for Computational Linguistics (ACL).
* Li et al. (2018) Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018. Visualizing the loss landscape of neural nets. _Advances in neural information processing systems_, 31.
* Lohschilov and Hutter (2022) Ilya Lohschilov and Frank Hutter. 2022. Sgdr: Stochastic gradient descent with warm restarts. In _International Conference on Learning Representations_.
* Mahabadi et al. (2021) Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 565-576.
* 2019 IEEE Region 10 Conference (TENCON)_, pages 2087-2092.
* O'donoghue and Candes (2015) Brendan O'donoghue and Emmanuel Candes. 2015. Adaptive restart for accelerated gradient schemes. _Foundations of computational mathematics_, 15:715-732.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. _arXiv preprint arXiv:1606.05250_.
* Saha et al. (2018) Amrita Saha, Rahul Aralikette, Mitesh M Khapra, and Karthik Sankaranarayanan. 2018. Duorc: Towards complex language understanding with paraphrased reading comprehension. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1683-1693.
* Sanh et al. (2017) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutauwika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al. Multitask prompted training enables zero-shot task generalization. In _International Conference on Learning Representations_.
* Smith (2017) Leslie N Smith. 2017. Cyclical learning rates for training neural networks. In _2017 IEEE winter conference on applications of computer vision (WACV)_, pages 464-472. IEEE.
* Trischler et al. (2017) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. _ACL 2017_, page 191.
* Tsatsaronis et al. (2015) George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotiis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. 2015. An overview of the bioasp large-scale biomedical semantic indexing and question answering competition. _BMC bioinformatics_, 16(1):1-28.
* Volk et al. (2022) Tomer Volk, Eyal Ben-David, Ohad Amosy, Gal Chechik, and Roi Reichart. 2022. Example-based hyperentworks for out-of-distribution generalization. _arXiv preprint arXiv:2203.14276_.
* Oswald et al. (2020) Johannes von Oswald, Christian Henning, Benjamin F Grewe, and Joao Sacramento. 2020. Continual learning with hypernetworks. In _8th International Conference on Learning Representations (ICLR 2020)(virtual)_. International Conference on Learning Representations.
* Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. _arXiv preprint arXiv:1804.07461_.
* Xiao et al. (2023) Zedian Xiao, William Held, Yanchen Liu, and Diyi Yang. 2023. Task-agnostic low-rank adapters for unseen english dialects. _arXiv preprint arXiv:2311.00915_.
* Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2369-2380.
* Ye and Ren (2021) Qinyuan Ye and Xiang Ren. 2021. Learning to generate task-specific adaptive from task description. In 

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 646-653. Cited by: SS1.
* D. Zhao, S. Kobayashi, J. Sacramento, and J. von Oswald (2020)Meta-learning via hypernetworks. In 4th Workshop on Meta-Learning at NeurIPS 2020 (MetaLearn 2020), External Links: Link Cited by: SS1.
* W. Xin Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al. (2023)A survey of large language models. arXiv preprint arXiv:2303.18223. Cited by: SS1.

 



## Appendix A Proof of Hessian.

We denote loss function \({\cal L}={\cal F}(\theta_{1},\ldots,\theta_{n})\), \(\theta_{i}={\cal H}(\phi_{1},\ldots,\phi_{n})\). From the chain rule, we can derive that:

\[\begin{split}\frac{\partial{\cal L}}{\partial\phi^{j}_{i}}=& \frac{\partial{\cal L}}{\partial\mathbf{\theta}}\frac{\partial\mathbf{ \theta}}{\partial\phi^{j}_{i}}\\ =&\nabla_{\mathbf{\theta}}{\cal L}\cdot{\bf X}_{ij}, \end{split} \tag{6}\]

where \(\nabla_{\mathbf{\theta}}{\cal L}=\left[\frac{\partial{\cal L}}{\partial\theta_{1} }~{}~{}~{}\frac{\partial{\cal L}}{\partial\theta_{2}}~{}~{}~{}\cdots~{}~{} \frac{\partial{\cal L}}{\partial\theta_{n}}\right]\) and \({\bf X}_{ij}=\left[\frac{\partial{\theta_{1}}}{\partial\phi^{j}_{1}}~{}~{} ~{}\frac{\partial{\theta_{2}}}{\partial\phi^{j}_{i}}~{}~{}~{}\cdots~{}~{} \frac{\partial{\theta_{n}}}{\partial\phi^{j}_{i}}\right]^{T}\). Then we estimate their loss landscape Hessian matrix by calculating the second-order derivative,

\[\begin{split}\frac{\partial^{2}{\cal L}}{\partial\phi^{j}_{i} \partial\phi^{l}_{k}}&={\bf X}^{T}_{ij}{\bf H}_{\theta}{\bf X}_{ kl}+\nabla_{\mathbf{\theta}}{\cal L}\cdot{\bf X}_{ij,kl},\\ {\bf X}_{ij,kl}&=\left[\frac{\partial^{2}\theta_{1} }{\partial\phi^{j}_{i}\partial\phi^{l}_{k}}~{}~{}~{}\frac{\partial^{2}\theta_ {2}}{\partial\phi^{j}_{i}\partial\phi^{l}_{k}}~{}~{}~{}\cdots~{}~{}\frac{ \partial^{2}\theta_{n}}{\partial\phi^{j}_{i}\partial\phi^{l}_{k}}\right]^{ T}.\end{split} \tag{7}\]

Note that \(\frac{\partial^{2}{\cal L}}{\partial\phi^{j}_{i}\partial\phi^{l}_{k}}\) is the items of matrix \(H_{\phi}\). We can further obtain

\[\begin{split}{\bf H}_{\theta}=&\begin{bmatrix}{ \bf X}^{T}_{2}\cdot{\bf H}_{\theta}\cdot{\bf X}_{1}&{\bf X}^{T}_{1}\cdot{\bf H }_{\theta}\cdot{\bf X}_{2}&\dots&{\bf X}^{T}_{2}\cdot{\bf H}_{\theta} \cdot{\bf X}_{m}\\ {\bf X}^{T}_{2}\cdot{\bf H}_{\theta}\cdot{\bf X}_{1}&{\bf X}^{T}_{2}\cdot{\bf H }_{\theta}\cdot{\bf X}_{2}&\dots&{\bf X}^{T}_{2}\cdot{\bf H}_{\theta} \cdot{\bf X}_{m}\\ &\ddots&\ddots&\\ {\bf x}^{T}_{m}\cdot{\bf H}_{\theta}\cdot{\bf X}_{1}&{\bf X}^{T}_{m}\cdot{\bf H }_{\theta}\cdot{\bf X}_{2}&\dots&{\bf X}^{T}_{m}\cdot{\bf H}_{\theta} \cdot{\bf X}_{m}\\ =&{\bf J}^{T}_{\Phi}{\bf H}_{d}J_{\phi}+{\bf R},\end{bmatrix}\\ =&{\bf J}^{T}_{\Phi}{\bf H}_{d}J_{\phi}+{\bf R},\end{split} \tag{8}\]

where \(J_{\phi}\) is the Jacobi matrix of main network and \(R_{ij}=\sum\frac{\partial{\cal L}}{\partial\theta_{i}}\frac{\partial^{2} \theta_{i}}{\partial\phi^{j}_{i}\partial\phi^{l}_{k}}\).

In usual LMs, we can suppose the second derivative in \(R\) is small enough to be ignored. Then we find that

\[H_{\phi}=J^{T}_{\phi}H_{\theta}J_{\phi}. \tag{9}\]

## Appendix B Dataset Statistics

Tables 6, 7 and 8 provide some summary statistics of each dataset used and split sizes.





\begin{table}
\begin{tabular}{l|c c} \hline \hline Dataset & Train Split Size & Validation Split Size \\ \hline SQuAD & 86588 & 10507 \\ HotpotQA & 72928 & 5901 \\ TriviaQA & 61688 & 7785 \\ NewsQA & 74160 & 4212 \\ SearchQA & 117384 & 16980 \\ Natural Qs & 104071 & 12836 \\ BioASQ & - & 1504 \\ DROP & - & 1503 \\ DuoRC & - & 1501 \\ RACE & - & 674 \\ Relation Ext. & - & 2948 \\ TextbookQA & - & 1503 \\ \hline \end{tabular}
\end{table}
Table 7: Summary statistics of splits used when evaluating MRQA

\begin{table}
\begin{tabular}{l|r r l} \hline \hline Dataset & Train Split Size & Validation Split Size & Test Split Size \\ \hline P3-split & 3301000 & 3301 & 3301 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Summary statistics of splits used when evaluating P3 split