## 4 Model Architecture Design
### Model Architecture Parameter Configuration  
Quantization of Transformer model parameters refers to converting the weights and activations in the model from floating-point numbers to a lower-precision data type [27]. This process aims to reduce the model's memory usage and computational requirements, thus improving the model's efficiency and deployment feasibility on specific hardware. Quantization of model parameters has several significant implications for hardware requirements:  
* Reduced memory usage: By reducing the bit-width of the model's weights and activations, the required storage space for the model is significantly reduced [28; 29; 30]. This is especially crucial for devices with limited memory resources, such as mobile devices, embedded systems, or edge computing devices, enabling the deployment of large Transformer models on these devices.
* Accelerated inference speed: Many modern hardware (such as GPUs, TPUs, and specific AI accelerators) are optimized for low-precision computations, providing faster computational speed at lower precision. This means that quantized models can achieve higher throughput and lower latency when performing inference on these hardware [31; 32; 33; 34; 35].
* Energy saving: Due to the reduction in computations and memory accesses, quantized models consume less energy during runtime, significantly reducing energy costs and environmental impact for battery-powered devices or large-scale data centers.  
* Widened application scope: Parameter quantization enables the application of complex Transformer models that were previously difficult to deploy due to resource constraints in more scenarios, such as real-time natural language processing tasks on IoT devices or advanced decision support systems in autonomous vehicles.
* Enhanced user experience: Faster response times and lower energy consumption mean that users can enjoy smoother and more enduring application experiences, especially on mobile devices.  
The application of artificial intelligence (AI) in the field of knowledge question answering is becoming increasingly widespread. Generative Pre-trained Transformer (GPT) has attracted particular attention. We have developed a new GPT-based, manually modified method to construct the model, comprehensively considering and adjusting parameters such as hidden dimensions, the number of attention heads, and dropout ratios to reduce the training volume of the model, thereby reducing the usage of video memory during matrix operations. We have also proposed a new theoretical framework by adding context detection and text fusing mechanisms at the OUTPUT end, as shown in Figure 8.  
The specific configuration is as follows:  
* Vocabulary Size: 240,000 Uyghur and Chinese word items, covering a rich range of linguistic expressions.
* Sequence Length: The maximum input length is set to 256 to handle complex sentence structures and long text requirements.
* Hidden Dimension (d_model): Set to 256, serving as the dimension for the internal representation of the model, balancing computational efficiency and expressive power.
* Number of Layers (n_layer): A total of 8 Transformer blocks, increasing the depth of the model, which is conducive to learning more complex linguistic structures.
* Number of Heads (n_head): Each layer uses 8 attention heads, enhancing the model's parallel processing capabilities and ability to capture dependencies between different positions [35; 36].  
Figure 8: Transformer Structure Before and After Modifications Comparison.  
* Bias: Bias terms are used in the linear layers to enhance the model's expressive ability.
* Dropout Rate: Set to 0.1, maintaining the integrity of information transmission and facilitating the understanding of the model's behavior during training.