# TwinStar: A Novel Design for Enhanced Test Question Generation Using Dual-LLM Engine
## 3 Design  
The quality of test questions generally relies on domain knowledge, such as a question bank covering a substantial network of knowledge points, and a set of criteria profiling the challenge level, such as [PERSON]'s cognitive levels. Leveraging large language models (LLMs) to generate test questions that align with educational objectives actually poses significant challenges.  
First of all, the size of LLMs has a direct impact on the relevance of knowledge points for a certain subject deeply embedded in the neural network. For example, the baseline model used in this paper, ChatGLM2, has only 6B parameters. Compared with ChatGPT-3.5, a full-fledged model with 175B parameters, a single ChatGLM2 model apparently struggles to fulfill the requirement for rendering a network of domain knowledge. In our experiments, a single ChatGLM2 model only achieves 3% accuracy in knowledge relevance compared with ChatGPT-3.5. However, a lightweight model is actually desired for many domain-specific applications, providing a secure and relatively isolated environment on a local machine or an embedded system. The challenge is how to reveal domain knowledge accurately with lightweight models. In this paper, we are interested in how to accurately capture certain knowledge points in a test question generated by a relatively small model. Our work shows that a set of effective techniques, as well as a careful system design, is critical for achieving the design goal.  
Secondly, a classical question generation problem usually needs to go through a process of multi-object optimization, considering both the accuracy of locating knowledge points and the level of challenge, for example. Multi-object optimization, however, is not inherent for LLMs. College-level calculus questions generated by ChatGPT-3.5, for example, only meet the requirement of [PERSON]'s cognitive level with 78% accuracy, even though it shows an almost perfect accuracy in knowledge relevance. This means that a well-trained huge model, such as ChatGPT-3.5, can still hardly assist in the teaching process in terms of generating test questions that fit the learning curve and educational objectives. Many existing projects exercising question generation for tests still rely on a group of experts with experience in question design for a specific subject to review or evaluate the generated questions, imposing a considerable workload, which may not be affordable for various application scenarios.  
To address these challenges, this paper proposes a design framework centered around a dual-LLM engine, as illustrated in Figure 1. It features two models, a question generation model and a cognitive-level evaluation model, as AI engines that collaboratively generate questions aligned with [PERSON]'s targeted cognitive levels and knowledge points. The cognitive-level evaluation engine built on the same baseline model, ChatGLM2, assesses the alignment with [PERSON]'s cognitive levels and provides iterative feedback to refine the direction of question generation. These two AI engines interact iteratively, akin to twin stars in orbit, which inspired the name 'TwinStar' for the proposed framework.