--- license: apache-2.0 base_model: leveldevai/TurdusBeagle-7B tags: - generated_from_trainer model-index: - name: Metabird-7B results: [] --- ![Metabird](https://cdn-uploads.huggingface.co/production/uploads/645cfe4603fc86c46b3e46d1/Haraj8I91wCax4JjdYmaN.jpeg) ## Metabird-7B [Built with Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl)
See axolotl config axolotl version: `0.3.0` ```yaml base_model: leveldevai/TurdusBeagle-7B model_type: MistralForCausalLM tokenizer_type: LlamaTokenizer is_mistral_derived_model: true load_in_8bit: false load_in_4bit: false strict: false datasets: - path: shuyuej/metamath_gsm8k type: system_prompt: "" field_instruction: question field_output: answer format: "[INST] {instruction} [/INST]" no_input_format: "[INST] {instruction} [/INST]" dataset_prepared_path: val_set_size: 0.05 output_dir: ./out sequence_len: 8192 sample_packing: true pad_to_sequence_len: true eval_sample_packing: false wandb_project: wandb_entity: wandb_watch: wandb_name: wandb_log_model: gradient_accumulation_steps: 4 micro_batch_size: 2 num_epochs: 1 optimizer: adamw_bnb_8bit lr_scheduler: cosine learning_rate: 0.000005 train_on_inputs: false group_by_length: false bf16: true fp16: false tf32: false gradient_checkpointing: true early_stopping_patience: resume_from_checkpoint: local_rank: logging_steps: 1 xformers_attention: flash_attention: true warmup_steps: 10 evals_per_epoch: 4 eval_table_size: eval_table_max_new_tokens: 128 saves_per_epoch: 1 debug: deepspeed: weight_decay: 0.0 fsdp: fsdp_config: special_tokens: bos_token: "" eos_token: "" unk_token: "" ```

# out This model is a fine-tuned version of [leveldevai/TurdusBeagle-7B](https://huggingface.co/leveldevai/TurdusBeagle-7B) on the shuyuej/metamath_gsm8k dataset. It achieves the following results on the evaluation set: - Loss: 0.4017 ## Model description More information soon ## Intended uses & limitations More information soon ## Training and evaluation data More information soon ## Training procedure More information soon ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 5e-06 - train_batch_size: 2 - eval_batch_size: 2 - seed: 42 - gradient_accumulation_steps: 4 - total_train_batch_size: 8 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 10 - num_epochs: 1 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-----:|:----:|:---------------:| | 0.9074 | 0.05 | 1 | 0.9932 | | 0.5012 | 0.26 | 5 | 0.4849 | | 0.4204 | 0.53 | 10 | 0.4435 | | 0.3748 | 0.79 | 15 | 0.4017 | ### Framework versions - Transformers 4.37.0.dev0 - Pytorch 2.0.1+cu117 - Datasets 2.16.1 - Tokenizers 0.15.0