# TwinStar: A Novel Design for Enhanced Test Question Generation Using Dual-LLM Engine
## 4 Experiments
### Results
#### 4.2.2 Evaluating Quality of Test Question Generation  
Table 2 shows the evaluation results of both performance metrics, knowledge relevance (KR) and [PERSON]'s cognitive-level adherence (BA). Table 2 also shows the average accuracy achieved with equally weighed KR and BA, as well as the standard deviation (SD), indicating the balance of performance improvement in terms of both KR and BA measurements across all the evaluated models. To demonstrate the effectiveness of TwinStar, we present the improvement in performance metrics achieved in several steps using various techniques. As shown in Table 2, the baseline model, ChatGLM2 (GLM2), was the least effective model in aligning the generated questions with the target [PERSON]'s cognitive levels and knowledge points. After fine-tuning with [PERSON]'s instruction-question pairs, GLM2-pt shows a significant rise in BA, while the KR value remains low. A low score for KR reflects the fact that the training dataset used in this step is different from the calculus domain. The value of KR is constantly improved when advanced optimization techniques are applied in multiple steps. The application of multiround prompting (GLM2-pt-RP) noticeably boosts KR while slightly reducing BA. The resultant measurement indicates that multiround prompting effectively guides the model toward the target knowledge domain. Figure 5: Heat maps showing the effectiveness of cognitive-level recognition using various LLMs. After complementing two additional examples in the prompt in GLM2-pt-RP-2 shot, the KR measurement was further improved. The fine-tuned GLM2 model can then effectively recognize the relationship between questions and their respective cognitive levels. On the other hand, large LLMs, inluding GPT-3.5, GPT-4.0, Claude, and Bard, all exhibit relatively low performance in recognizing appropriate Bloom's cognitive levels, resulting in unbalanced performance in terms of cognitive-level adherence and knowledge point relevance. Finally, with the help of the evaluation model, TwinStar achieves significant gains across all metrics: BA, KR, and overall average improvement by factors of 6, 29, and 10, respectively, relative to the baseline (GLM2). A comparison between TwinStar and other large language models reveals significant improvement in the overall performance achieved by the proposed TwinStar design. Notably, TwinStar achieves the highest Bloom adherence (BA) among all models, outperforming GPT-3.5, GPT-4.0, Claude, and Bard (renamed as Gemini) by 19%, 21%, 13%, and 50%, respectively, which underscores the effectiveness of the generation evaluation framework. Figure 5 provides heat maps illustrating how each model generates questions aligning with the six Bloom's cognitive levels, reflecting model-specific performance across Bloom's categories. In terms of knowledge relevance (KR), TwinStar and Claude obtain the same result, whereas GPT-3.5, GPT-4.0, and Bard attain the highest KR values. With respect to overall performance (average), TwinStar exceeds GPT-3.5, GPT-4.0, _Claude_, and _Bard_ by 2%, 2%, 6%, and 12%, respectively. Note that the TwinStar architecture consists of two lightweight ChatGLM2 models containing just 12 billion parameters in total, a fraction of the parameters of large LLMs, such as GPT-3.5. In other words, TwinStar achieves better accuracy with far fewer parameters. Therefore, it can produce the desired test questions with a much smaller memory space requirement. This improvement is achieved through a carefully designed dual-model architecture, as well as with the assistance of fine-tuning and prompting techniques. This finding also indicates that well-designed, small language models can outperform their large counterparts, dramatically reducing the costs of model building and management. The proposed approach could be a valuable direction for exploring the efficient deployment of AI engines on resource-constrained computing devices.