Note Background read to understand loss landscapes in deep neural networks and why local minima are not a problem for SGD.

Note It's the classical merge method: a simple weighted average.

Note A new paradigm for modifying the behavior of neural networks using “task vectors.” that represent directions in the weights space of a pre-trained model, instead of magnitudes, and point towards improved performance on a specific task.

Note The TIES-merging approach addresses the problem of interference between parameters from different models: redundant parameters and sign conflicts.

Note Examines the theoretical properties and practical implications of weight averaging applied to larger generative models.

Note DARE is a novel approach to model merging. It uses a similar approach to TIES with 2 main differences: pruning and rescaling.