Hugging Face's logo

alpindale
/
goliath-120b

Text Generation
Transformers
Safetensors
English
llama
Merge
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
20

Great work! Question about the merge specifics.

#4
by UMCU - opened
Discussion
UMCU

Thank you for this awesome contribution! I am interested in exploring the merging strategy for our projects; I see that you have overlapping and non-overlapping layers. I assume that the non-overlapping layers are simply appended, and that the overlapping layers use linear interpolation with 50/50 weights?

Thank you for your time.

HenkPoley

You can probably find your answer near the used 'mergekit' software: https://github.com/cg123/mergekit

UMCU

Mergekit gives several options and has some settable parameters , I cannot derive from the model card what these settings are :).

alpindale
 Owner

I explained it a bit in #3, but the TL;DR is that I'm taking slices from each model and stacking those on top of each other. I've mentioned the slice ranges in the README.

Your need to confirm your account before you can post a new comment.

Sign up or log in to comment