# Comparative Analysis of Chatbots Using Large Language Models for Web Development Tasks
## 3 Materials and Methods
### Chatbots and Evaluation Tools  
In this study, we have selected five chatbots capable of generating code, based on a ranking by [PERSON] [46] of the most popular generative AI web products, determined by monthly visits. This ranking provided insight into how consumers are utilizing these technologies. From this list, we specifically chose chatbots that met the following criteria: they are either general-purpose language models or specialized code-generating models; they have a significant user base as indicated by their ranking, ensuring their relevance and widespread adoption; they are accessible to users without prohibitive costs or restrictions, allowing for practical evaluation; and they collectively represent a variety of approaches to AI-driven code generation.  
We selected five chatbots because this number strikes an optimal balance between breadth and depth in our analysis. Evaluating five distinct models allows us to cover a diverse spectrum of available technologies while keeping the scope manageable. This ensures that we can perform a thorough and detailed evaluation of each chatbot without diluting our focus. We categorized these chatbots into two distinct groups:  
1. General-purpose language models: these chatbots are capable of a wide range of tasks, including answering questions, generating text, programming code, and providing conversational responses.  
_ChatGPT 3.5_: Developed by OpenAI, ChatGPT 3.5 is a versatile AI model designed for various conversational tasks. It leverages the GPT-3.5 architecture, known for its ability to generate coherent and contextually appropriate responses across numerous topics. It is widely used for general-purpose dialogue, including generating code and assisting with programming-related queries [47].
* _Copilot_: Microsoft's Copilot is a conversational AI designed to assist users by providing information, generating content, and performing web searches. It differs from ChatGPT 3.5 in that it can perform real-time web searches to provide up-to-date information. In this study, we used Microsoft's Copilot in creative mode, which employed an adapted GPT-4 model at the time of testing (April 2024) [48].
* _Gemini_: This chatbot is part of Google's AI portfolio, leveraging the latest advancements in their language model technology. Gemini is designed to assist with a broad range of tasks, from answering general questions to generating complex code snippets. It integrates Google's extensive search and information retrieval capabilities to enhance its response accuracy and relevance [49].
2. Code-specific language models: these chatbots are designed specifically for generating code snippets and assisting with programming tasks.
* _TabNine AI_: An AI-powered code completion tool that uses deep learning models to provide intelligent code suggestions. TabNine supports a wide range of programming languages and is integrated into various IDEs. It focuses on enhancing developer productivity by predicting and completing code based on the context provided by the user [50].
* _Blackbox AI_: A specialized code generation tool that utilizes machine learning models to assist developers in writing code. Blackbox AI is designed to understand coding patterns and provide relevant code snippets, debugging assistance, and optimization suggestions. It aims to streamline the coding process and reduce the time required to write functional code [51].  
Evaluation tools are essential for ensuring code accuracy in scientific studies. The W3C Nu HTML Checker was used to verify the syntax correctness of HTML and CSS, ensuring adherence to web standards. Similarly, the SQL Validator from Scaler.com was used to check SQL queries for syntax errors. These tools help maintain code quality and reliability.