|
--- |
|
library_name: transformers |
|
tags: |
|
- arcee-ai |
|
--- |
|
|
|
 |
|
|
|
# Model Card for Model ID |
|
|
|
This model is a dummy model created for testing purposes only. It utilizes a custom configuration to explore various training scenarios and should not be used for production. |
|
|
|
## Configuration Highlights |
|
|
|
The configuration of this dummy model is distinct from the original model in several key aspects: |
|
|
|
- **Number of Layers:** Reduced to 2, allowing for quicker tests of layer-specific behaviors. |
|
- **Experts:** Configured with 4 local experts and 2 experts per token, experimenting with the model's capacity to handle multiple expert inputs. |
|
- **Hidden Size:** Set at 512, this smaller size is suitable for testing the impact of network width. |
|
- **Intermediate Size:** Enlarged to 3579, to investigate how an increase in the size affects the model's ability to process information deeply. |
