--- license: apache-2.0 tags: - moe - frankenmoe - merge - mergekit - lazymergekit - mlabonne/NeuralBeagle14-7B - fblgit/UNA-dolphin-2.6-mistral-7b-dpo-laser - mlabonne/Marcoro14-7B-slerp base_model: - mlabonne/NeuralBeagle14-7B - fblgit/UNA-dolphin-2.6-mistral-7b-dpo-laser - mlabonne/Marcoro14-7B-slerp model-index: - name: CultriX-MoE-BF16 results: - task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics: - type: acc_norm value: 68.94 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=CultriX/CultriX-MoE-BF16 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics: - type: acc_norm value: 86.96 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=CultriX/CultriX-MoE-BF16 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics: - type: acc value: 65.2 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=CultriX/CultriX-MoE-BF16 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics: - type: mc2 value: 63.47 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=CultriX/CultriX-MoE-BF16 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics: - type: acc value: 81.06 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=CultriX/CultriX-MoE-BF16 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics: - type: acc value: 69.98 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=CultriX/CultriX-MoE-BF16 name: Open LLM Leaderboard --- # CultriX-MoE-BF16 CultriX-MoE-BF16 is a Mixure of Experts (MoE) made with the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing): * [mlabonne/NeuralBeagle14-7B](https://huggingface.co/mlabonne/NeuralBeagle14-7B) * [fblgit/UNA-dolphin-2.6-mistral-7b-dpo-laser](https://huggingface.co/fblgit/UNA-dolphin-2.6-mistral-7b-dpo-laser) * [mlabonne/Marcoro14-7B-slerp](https://huggingface.co/mlabonne/Marcoro14-7B-slerp) ## 🧩 Configuration ```yaml base_model: "EmbeddedLLM/Mistral-7B-Merge-14-v0.2" gate_mode: hidden dtype: bfloat16 experts: - source_model: "mlabonne/NeuralBeagle14-7B" positive_prompts: - "Create a story based on" - "Debate the topic of" - "Come up with some arguments" - "Provide me with instructions on" - "Interpret the sentiment" - "Interpret and execute these cooking instructions" - "Craft a persuasive argument" - "Analyze the motivations" - "Construct a detailed plan for" - "Narrate an event from multiple perspectives." - "Formulate a response" - "Write a script for a short play" - "Generate a sequence of instructions to teach a skill." - "Solve this riddle" - "Create an engaging story" - "Write a fictional" - "Propose a solution to a social issue" - "Develop a dialogue" - "Create a step-by-step guide" - "Devise a strategy" - "Write a narrative" - "Tell me how to" - "Explain the concept of" - "Give an overview of" - "Compare and contrast between" - "Provide information about" - "Help me understand" - "Summarize" - "Make a recommendation on" - "Answer this question" - "How do you approach" - "Explain the concept of" - "Give an overview of" - "Provide information about" - "Help me understand the principles of" - "Summarize the key components of" - "Make a recommendation on how to" - "Answer this question:" negative_prompts: - "Provide in-depth information about quantum computing." - "Explain the inner workings of an internal combustion engine." - "Give a detailed tutorial on advanced calculus." - "Summarize the latest research in genetic engineering." - "Interpret financial markets and stock trends." - "Analyze the chemical composition of" - "Develop a blueprint for." - "Offer a critique of a modern art piece." - "Provide a technical review of" - "Conduct a linguistic analysis of an ancient language." - "Write a user manual for advanced medical equipment." - "Give a step-by-step guide on piloting an aircraft." - "Conduct an in-depth analysis of this code" - "Explain the physics behind black holes." - "Provide a strategy for managing a cyber attack" - "Develop an algorithm for predictive analytics in finance." - "Provide information about advanced programming algorithms." - "Help me understand the details of this code" - "Summarize the process of cellular respiration." - "Improve the security of" - "What are the latest advancements in artificial intelligence?" - "Provide detailed technical coding solutions." - "Analyze complex scientific data and statistics." - "Offer medical diagnoses based on symptoms." - "Conduct a detailed financial audit of a company." - "Perform real-time translation of multiple languages." - "Create high-resolution graphic designs." - "Develop complex mathematical proofs." - "Offer legal advice on specific cases." - "Write a detailed manual on advanced mechanical engineering." - "Conduct an in-depth psychological assessment." - "Perform a security analysis of a computer network." - "Compose an original piece of music." - "Plan and execute a scientific experiment." - "Provide professional career counseling." - "Develop a complex database management system." - "Write a software program for data analysis." - "Give expert advice on cyber" - "Conduct a pentesting security audit" - source_model: "fblgit/UNA-dolphin-2.6-mistral-7b-dpo-laser" positive_prompts: - "Provide step-by-step coding instructions for..." - "Draft a function with detailed steps in [language]" - "Guide me through coding a simple [type of application or script]" - "Recommend best practices for code implementation in [context]" - "Generate a regex pattern for extracting [specific data]" - "Create a regex for matching [pattern]" - "Explain the purpose of this regex pattern" - "Compose regex for [specific use case]" - "Annotate this code with detailed comments for each line" - "Add explanatory comments to this script" - "Comment on each part of this code for clarity" - "Develop a script to [accomplish task]" - "Design a database schema for [specific use case]" - "Outline secure methods for [specific operation]" - "Guide on optimizing [specific aspect] in this code" - "Refactor this code for better readability and efficiency" - "Compare and contrast these code snippets" - "Identify the programming language of this snippet" - "Demonstrate the usage of [specific tool/library/API]" - "Show implementation steps for this [feature/concept]" - "Teach how to use [specific tool/library/framework]" - "Generate a README file for this project" - "Create a manual page for [specific tool/command]" - "Produce comprehensive documentation for this code" - "Build detailed documentation for [specific module]" - "Explain the underlying concept of this code snippet" - "Propose enhancements for this script" - "Suggest improvements for this API call integration" - "Diagnose and solve this coding issue" - "Demonstrate robust error handling in this code" - "Debug and resolve issues in this script" - "Design a user-friendly GUI for this script's functionality" - "Detail the deployment process for this application" - "Deploy an app designed to [perform function]" - "Set up a web service for [specific purpose]" - "Develop a website with [specific features]" - "Craft a webpage showcasing [specific content]" - "Illustrate data flow in this code architecture" - "Convert this code from [language A] to [language B]" - "Translate this script into [different programming language]" - "Explain resource management techniques in [context]" - "Build a basic API endpoint for [functionality]" - "Strategies to enhance scalability in [context]" - "Conduct a security review for this code" - "Enhance security measures in [application/module]" - "Set up a development environment for [language/framework]" - "Visualize data from [specific dataset]" - "Generate a dataset for [specific use case]" - "Scripting guide for automating [task/process]" - "Utilize this code for [specific purpose]" - "Principles of object-oriented programming in [language]" - "Create a mobile-responsive layout for this web app" - "Explain the debugging process for this code" - "Compose code to accomplish [task]" - "Guidance on writing code for [specific purpose]" - "I need a script for [specific function]" - "Clarify the functionality of this code" - "What is the purpose of this code segment?" - "Enhance this code for [specific improvement]" - "Develop a program that [solves problem]" - "Code needed for [specific task]" - "Program a solution for [problem statement]" - "Enhance this function's performance by..." - "Refactor code for better readability in [context]" - "Craft a custom function for [specific requirement]" - "Reduce computational complexity in this algorithm by..." - "Extend the codebase to include [new feature]" - "Incorporate this API into an existing application" - "Assist in troubleshooting and bug fixing for [issue]" - "Review and prep this code for deployment" - "Analyze error logs for potential issues in [context]" - "Create unit tests for [module/component]" - "Evaluate methodologies for [problem-solving]" - "Research [topic] online" - "Utilize the [plugin/tool] to achieve [result]" - "Design an efficient search algorithm for [data type]" - "Create a web crawler for [specific data extraction]" - "Application of web sockets in [real-time scenario]" - "Guide to integrating a third-party library in [framework]" - "Best practices in API design for [application type]" negative_prompts: - "Provide a detailed analysis of historical events." - "Give medical advice for treating a specific illness." - "Write a comprehensive review of a novel." - "Explain legal implications of a contract." - "Develop a marketing strategy for a new product." - "Offer financial advice for stock investments." - "Create a recipe for a gourmet dish." - "Teach a foreign language lesson." - "Compose a symphony or musical piece." - "Provide workout plans and fitness coaching." - "Conduct a psychological analysis of a character." - "Write a script for a movie or play." - "Design a blueprint for architectural structures." - "Give a tutorial on how to paint a landscape." - "Explain quantum physics theories." - "Offer career counseling and resume writing tips." - "Teach how to repair a car engine." - "Plan a travel itinerary for a world tour." - "Guide on how to grow organic vegetables." - "Discuss political strategies for an election campaign." - source_model: "mlabonne/Marcoro14-7B-slerp" positive_prompts: - "Generate a creative story based on these keywords." - "Explain a complex topic in simple terms" - "Provide a detailed summary of" - "Answer this question with factual accuracy" - "Explain the historical significance of" - "Provide a truthful and detailed account of" - "Develop a strategy for solving a practical problem." - "Explain the reasoning behind" - "Provide an analysis of a moral dilemma with possible solutions." negative_prompts: - "imathematical problem-solving." - "scientific theory explanations." - "high-level abstract reasoning tasks." - "professional advice in specialized fields like law or medicine." - "provide me with a coding solution for" - "Academic research" ``` ## 💻 Usage ```python !pip install -qU transformers bitsandbytes accelerate from transformers import AutoTokenizer import transformers import torch model = "CultriX/CultriX-MoE-BF16" tokenizer = AutoTokenizer.from_pretrained(model) pipeline = transformers.pipeline( "text-generation", model=model, model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True}, ) messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}] prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95) print(outputs[0]["generated_text"]) ``` # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_CultriX__CultriX-MoE-BF16) | Metric |Value| |---------------------------------|----:| |Avg. |72.60| |AI2 Reasoning Challenge (25-Shot)|68.94| |HellaSwag (10-Shot) |86.96| |MMLU (5-Shot) |65.20| |TruthfulQA (0-shot) |63.47| |Winogrande (5-shot) |81.06| |GSM8k (5-shot) |69.98|