# Evaluating LLMs for Automated Scoring in Formative Assessments
## 2 Materials and Methods  
### Development of the Platform  
The introEduAI platform automates grading processes while evaluating the reliability of LLMs compared to human evaluators. By using LangChain and RAG, the platform delivers precise, personalized, and actionable assessments for diverse question types. Hosted on a Virtual Private Server (VPS), its architecture ensures scalability, cost-efficiency, and seamless interactions among students, educators, and backend systems. For visual demonstrations of the platform's interface and functionalities, see Appendix A (Figures A1-A4).  
The platform employs Flask with Jinja2 templates and a Representational State Transfer Application Programming Interface (REST API) backend, adhering to the Model-View-Controller paradigm for a maintainable and flexible design. Secure access is managed through JavaScript Object Notation (JSON) Web Token (JWT), providing adequate function alities for students, teachers, and administrators while ensuring robust data protection. The platform integrates both premium and open-source LLMs through APIs, eliminating the need for local infrastructure. These models were selected as state-of-the-art representatives of their respective categories: OpenAI GPT-4o, widely regarded as a leading premium LLM due to its advanced capabilities and strong performance across a range of applications [18], and LLaMA 3.2, which has been shown to outperform other open-source models on several public benchmarks [20]. This focused selection allows for a meaningful comparison between premium and open-source approaches while maintaining the study's scope.  
The workflow dynamically adapts based on the question type. More specifically, for text-based questions, LangChain retrieves relevant information through RAG and combines it with the query to provide contextually enriched and accurate evaluations. For code-based questions, the LLM's internal expertise is used to assess submissions. Such was implemented through LangChain by embedding information retrieved from external data sources (in this case, ChromaDB) into LLM prompts, mitigating issues related to outdated knowledge [12; 16]. The employed RAG solution dynamically combines retrieval and generative processes, grounding responses in factual knowledge while enhancing the reliability and scalability of grading systems [12; 14; 15]. Furthermore, the platform employs ChromaDB to manage high-dimensional embeddings created using OpenAI's \"text-embedding-ada-002\" model. This vector database was selected as it was shown to be suitable in semantic similarity searches, aligning user queries with relevant data in the RAG workflow.  
The platform utilized few-shot prompting (in this case, it enabled a small set of input-output examples within the prompt to guide the behavior of LLMs). This approach enables the models to generalize effectively to new tasks, reducing the need for extensive retraining [6; 21]. In this implementation, prompts included key elements such as question text, scoring rubrics, example solutions, evaluation criteria, and sample student responses to ensure outputs aligned with scoring standards. An example of a prompt template used in the study is provided in Figure 1.  
Figure 1: Few-Shot Prompt Template for RAG-Enhanced Short-Answer Assessment.