| license: apache-2.0 | |
| tags: | |
| - merge | |
| - moe | |
| Open_Gpt4_v0.2 | |
|  | |
| This model is a TIES merger of Mixtral-8x7B-Instruct-v0.1 and bagel-8x7b-v0.2 with MixtralOrochi8x7B being the Base model. | |
| I was very impressed with MixtralOrochi8x7B performance and multifaceted usecases as it is already a merger of many usefull Mixtral models such as Mixtral instruct, | |
| Noromaid-v0.1-mixtral, openbuddy-mixtral and possibly other models that were not named. My goal was to expand the models capabilities and make it even more useful of a model, maybe even competitive with closed source models like Gpt-4. But for that more testing is required. I hope the community can help me determine if its deserving of its name. 😊 | |
| This is the second iteration of this model, using better models in the merger to improve performance (hopefully). | |
| Base model: | |
| - https://huggingface.co/smelborp/MixtralOrochi8x7B | |
| Merged models: | |
| - https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1 | |
| - https://huggingface.co/jondurbin/bagel-8x7b-v0.2 | |
| Instruct template: Alpaca | |
| Merger config: | |
| ``` | |
| models: | |
| - model: Mixtral-8x7B-Instruct-v0.1 | |
| parameters: | |
| density: .5 | |
| weight: .7 | |
| - model: bagel-8x7b-v0.2 | |
| parameters: | |
| density: .5 | |
| weight: 1 | |
| merge_method: ties | |
| base_model: MixtralOrochi8x7B | |
| parameters: | |
| normalize: true | |
| int8_mask: true | |
| dtype: float16 | |
| ``` |