# Comparative Analysis of Chatbots Using Large Language Models for Web Development Tasks
## 4 Results
### Qualitative Feedback  
The testers provided qualitative feedback on the strengths and weaknesses of each chatbot:  
* ChatGPT 3.5: Highlighted for its diverse and original content and solid HTML/CSS structure. It faced challenges in maintaining conversational memory.
* Copilot: Praised for its flawless HTML and CSS syntax, excellent appearance, and user-friendliness. However, it received lower scores for original and varied content.
* Gemini: noted for creative content but occasionally faltered in syntax and structural aspects.
* TabNine AI: commended for structural correctness but criticized for issues with appearance and user-friendliness.
* BlackBox: struggled with PHP forms and had notable syntax errors, leading to the lowest overall score.  
Comparison  
The performance of the chatbots varied significantly across different tasks:  
1. HTML and CSS Syntax Rules: Copilot and ChatGPT 3.5 excelled, consistently achieving high scores across all testers. BlackBox had the most difficulty, with frequent errors noted by evaluators.
2. Understanding of Prompts: all chatbots showed strong performance, but ChatGPT 3.5 and Copilot stood out for their ability to understand and execute tasks as expected.
3. Conversational Memory: Copilot was the most successful in maintaining context across tasks, while BlackBox showed significant gaps, leading to lower scores in this category.
4. Original and Diverse Content: ChatGPT 3.5 led in this area, followed closely by Gemini. Copilot and TabNine AI showed room for improvement.
5. Appearance and User-Friendliness: Copilot outperformed others, creating visually appealing and user-friendly outputs. Other chatbots, including TabNine AI, had issues with responsiveness and design.
6. Functioning of PHP Form and Database Linking: Copilot excelled, producing fully functional PHP forms that integrated with databases effectively. BlackBox failed to meet requirements consistently.
7. SQL Query Correctness: all chatbots met the criteria and received full points for creating correct SQL queries.  
The observed difference in performance between simpler and more complex tasks can be attributed to the inherent nature of language models. Simpler tasks typically require fewer steps, lower contextual depth, and involve more straightforward prompts, which makes it easier for the models to generate accurate and effective responses. In contrast, complex tasks demand more nuanced understanding, contextual continuity, and intricate code generation. These requirements increase the likelihood of errors, as the model needs to maintain the contextual flow across multiple steps and generate more comprehensive outputs.  
The distinction between conversational and programming-specific chatbots is key to understanding their practical applications. Conversational chatbots like ChatGPT and Copilot excel in understanding prompts in a broader context and generating diverse responses, which makes them highly effective in situations requiring more open-ended problem-solving or creative tasks. Conversely, programming-specific chatbots like TabNine AI are designed to assist with precise code generation, focusing on efficiency and adherence to syntax rules. They are better suited for integration within development environments and generate code with fewer errors, particularly when the prompt is explicitly detailed. A hybrid approach may serve as the ideal solution, combining the strengths of both model types to improve developer productivity across a wider variety of use cases.  
In summary, Copilot was the best-performing chatbot due to its strong technical execution and user-friendly designs, despite needing improvement in content originality (Figure 3). ChatGPT 3.5 was a close second, notable for its creativity and overall solidperformance. Gemini, while strong in content, had occasional technical issues. TabNine AI showed competence in structural tasks but lacked in design and user experience. BlackBox's performance was hindered by technical errors, particularly in PHP form functionality.