## 3 System Architecture
### Reinforcement Learning Optimization Module
* \(\sum|\Delta T|\): Penalizes excessive changes in fuzzy membership functions, ensuring stability in adaptation. * H: Introduces a hint usage penalty, which discourages excessive reliance on system-provided hints while ensuring timely and appropriate assistance. The reward function is designed to:
* Encourage actions that improve learning efficiency by minimizing unnecessary UI interventions. * Maximize student engagement by ensuring timely assistance. * Prevent unstable UI adjustments by discouraging large, frequent changes to fuzzy thresholds. * Reduce over-reliance on hints by penalizing unnecessary hint activation, ensuring that students receive guidance only when needed and do not become dependent on system-generated hints. The RL agent updates its policy using Q-learning, which allows it to refine its decisions over multiple student interactions. The Q-value update equation is expressed as follows:  
\[Q(S_{t},\ A_{t})=Q(S_{t},A_{t})+\eta\left[R_{t}+\gamma\underset{A^{\prime}}{ \text{max}}Q\big{(}S_{t+1},A^{\prime}\big{)}-Q(S_{t},A_{t})\right]\]  
where:  
* \(\eta\) is the learning rate, which controls how rapidly the agent updates its knowledge. * \(\gamma\) is the discount factor, which defines the significance of future rewards. * \(\underset{A^{\prime}}{\text{max}}Q\big{(}S_{t+1},A^{\prime}\big{)}\) is the best possible Q-value at the next state. In this equation, \(\eta\) controls the learning rate, while \(\gamma\) governs the importance of future rewards. Over time, the Q-table is iteratively updated, allowing the agent to develop an optimal strategy for adjusting fuzzy weights dynamically. The initial fuzzy membership parameters (\(T_{\text{low}}\), \(T_{\text{mid}}\), \(T_{\text{max}}\)) were empirically set to reflect time-based patterns observed during a pilot phase with 20 students prior to the main experiment. These values were selected to roughly correspond to the 25 th, 50 th, and 75 th percentiles of completion times across typical Java exercises. This ensured that the fuzzy categories (Fast, Moderate, and Slow) represented meaningful distinctions in learner pacing. For the reinforcement learning setup, the learning rate \(\eta\) was set to 0.2 to balance learning speed with stability, while the discount factor \(\gamma\) was fixed at 0.8 to maintain sensitivity to long-term learning outcomes without overemphasizing distant rewards. These values were chosen based on recommendations from prior educational RL studies and validated through initial tuning experiments that monitored convergence and adaptation responsiveness. The RL agent requires training before it can be effectively deployed. The training process involves simulating multiple student interactions and updating the Q-table accordingly. The training loop consists of:  
* Simulating a student's interaction with an exercise (randomly generated time spent x\({}_{1}\)). * Computing initial fuzzy weights based on the current membership functions. * Applying UI modifications based on fuzzy weights. * Observing student outcomes (e.g., Did they complete the next exercise faster? Did they need more assistance?). * Computing the reward R\({}_{t}\) based on engagement and learning improvement. * Updating the Q-table using the Q-learning rule. * Repeating for multiple student interactions until the policy converges. The final trained RL agent learns when to modify fuzzy membership thresholds to improve UI adaptivity and optimize student learning experiences. After training, the Reinforcement Learning Optimization Module directly impacts how fuzzy logic is applied by continuously refining \(T_{\text{low}}\), \(T_{\text{mid}}\), and \(T_{\text{max}}\). Over multiple interactions, the RL agent fine-tunes the thresholds, ensuring that:  
* Fast learners are not unnecessarily challenged too early. * Moderate learners remain engaged without excessive UI changes. * Slow learners receive help at the right moment.