Package: mergekit
mergekitmergekitis a toolkit for merging pre-trained language models, using a variety of merge methods including TIES, linear, and slerp merging. The toolkit also enables piecewise assembly of a language model from layers.Runpip install -e .to install the package and make the scripts available.The scriptmergekit-yamltakes a YAML configuration file defining the operations to perform.ConfigurationBelow are the primary elements of a configuration file:merge_method: Specifies the method to use for merging models. Can be one of 'ties', 'linear', 'slerp', or 'passthrough'.slices: Defines slices of layers from different models to be used. This field is mutually exclusive withmodels.models: Defines entire models to be used for merging. This field is mutually exclusive withslices.base_model: Specifies the base model used in some merging methods.parameters: Holds various parameters such as weights and densities, which can also be specified at different levels of the configuration.dtype: Specifies the data type for the merging operation.Parameter SpecificationParameters are flexible and can be set with varying precedence. They can be specified conditionally using tensor name filters, which allows finer control such as differentiating between attention heads and fully connected layers.Parameters can be specified as:Scalars: Single floating-point values.Gradients: List of floating-point values, specifying an interpolated gradient.The parameters can be set at different levels, with decreasing precedence as follows:slices.*.sources.parameters- applying to a specific input sliceslices.*.parameters- applying to a specific output sliceinput_model_parameters- applying to any tensors coming from specific input modelsparameters- catchallMerge MethodsResolving Interference When Merging Models("ties")Requires a base model.
Parameters:density- fraction of weights in differences from the base model to retainweight- relative (or absolute ifnormalize=False) weighting of a given tensornormalize- if true, the weights of all models contributing to a tensor will be normalized. Default behavior.LinearDoes not require a base model. Takes parametersweightandnormalize, with same definition as above.SLERPRequires exactly two models, one of which must be the base model. Takes one parameter -t- the interpolation factor from the base model to the secondary model.ExamplesSimple linear merge of multiple models:models:
  - model: psmathur/orca_mini_v3_13b
    parameters:
      weight: 1.0
  - model: WizardLM/WizardLM-13B-V1.2
    parameters:
      weight: 0.3
  - model: garage-bAInd/Platypus2-13B
    parameters:
      weight: 0.5
merge_method: linear
dtype: float16bakllama.pystyle layer recombination:slices:
  - sources:
    - model: psmathur/orca_mini_v3_13b
      layer_range: [0, 24]
  - sources:
    - model: garage-bAInd/Platypus2-13B
      layer_range: [20, 40]
merge_method: passthrough
dtype: float16Gradient SLERP with different weights for mlp/self attention:slices:
  - sources:
      - model: psmathur/orca_mini_v3_13b
        layer_range: [0, 40]
      - model: garage-bAInd/Platypus2-13B
        layer_range: [0, 40]
merge_method: slerp
base_model: psmathur/orca_mini_v3_13b
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5 # fallback for rest of tensors
dtype: float16UsageOnce you have created the YAML configuration file, runmergekit-yamlwith the config file and output path as arguments:mergekit-yamlpath/to/your/config.yml./output-model-directory[--cuda]Legacy WrappersMergekit originally featured two separate scripts with different inputs. The functionality of these is maintained in themergekit-legacyandbakllamawrappers. Example usage:mergekit-legacy./output-model--base-modelTheBloke/Llama-2-13B-fp16--cuda\--mergeWizardLM/WizardLM-13B-V1.2--weight0.3--density0.5\--mergegarage-bAInd/Platypus2-13B--weight0.5--density0.5mergekit-legacycan output a YAML configuration for easy migration with the--print-yamloption.