# Comparative Analysis of Chatbots Using Large Language Models for Web Development Tasks
## 2 Related Work  
Chatbots, especially those powered by large language models (LLMs), are being actively experimented with in various fields [8; 9; 10]. These experiments aim to improve areas like customer service, content generation, and even coding and web development. LLMs have demonstrated the ability to understand and generate human-like text, including programming languages [11]. To ensure LLMs meet the desired performance standards in such applications, a multitude of benchmarking tools have emerged to rigorously assess their capabilities across diverse dimensions. These tools often focus on specific capabilities, such as language understanding, reasoning, code generation, or multilingual abilities. For instance, MMLU [12] evaluates general language understanding, while WinoGrande [13] assesses common sense reasoning. HumanEval [14] benchmarks code generation, and MGSM [15] evaluates multilingual capabilities. These tools typically employ a variety of evaluation methods, including automatic metrics and human evaluation. Automatic metrics assess quantitative aspects like fluency and coherence, while human evaluation provides qualitative insights into factors such as factuality, relevance, and overall quality. A series of recent studies have examined the performance, usability, and limitations of LLMs, particularly in the context of code generation. These studies explore various aspects, including user experiences, benchmarking against different metrics, and the effectiveness of LLMs in real-world coding scenarios. A qualitative study [16] examined early adopters' perceptions of ChatGPT for code generation, focusing on user experiences, prompt engineering, trust, and its impact on software development. The researchers gathered and analyzed user comments from Hacker News to understand the challenges and preferences related to the tool. The findings revealed important areas for improvement, such as enhancing prompt engineering techniques and building greater user trust. These insights suggest directions for future research, particularly in refining code generation tools to better meet user needs. In another study [17], researchers evaluated the usability of Copilot with 24 participants. Despite Copilot's high performance on benchmark tests, the study found that it did not significantly improve participants' task completion time or success rates. Participants favored Copilot for providing a good starting point for coding tasks. The study suggests enhancements for understanding and validating generated code, exploring multiple solutions, and aiding task decomposition, indicating room for improvement in real-world application. In a comparative analysis [18], the correctness, efficiency, and maintainability of AI- and human-generated program code were assessed using metrics such as time and space complexity, runtime, memory usage, and maintainability index. Among the generative AIs evaluated, GitHub Copilot (Codex) achieved the highest success rate, solving 50% of the problems, while ChatGPT (GPT-3.5) was notable for solving the only hard problem, despite its lower overall correctness rate. A pilot study [19] explored the use of ChatGPT for generating secure web application code, focusing on mitigating four common vulnerabilities: SQL Injection, Cross-Site Scripting, CRLF Injection, and Exposure of Sensitive Information. The study found that ChatGPT does not inherently address these vulnerabilities but proposed a prompt pattern that can guide the model to generate more secure code by mitigating potential risks. In a comprehensive evaluation [20] of ChatGPT's capability to generate Java Web application code, using a user-login program as a case study, the results show that the generated code meets expected requirements, exhibiting high readability, quality, and functionality. While ChatGPT offers an efficient solution compared to manual coding, its performance diminishes with limited input information, suggesting its potential as a valuable tool for developers in software development.