## 6 Complex Task: Generating Code for an LSTM Model  
The problem and the procedure adopted are described below. The aim is to generate LSTM models using GPT, OAI1, and OAI3, which simultaneously predict short-term fluctuations in temperature, relative humidity, and wind speed. These variables were selected due to their significance and differing levels of variability, ranging from relatively stable (temperature) to highly dynamic (wind speed). Accurate short-term forecasting of these parameters is essential across multiple domains. For instance, the Canadian Fire Weather Index (FWI), a widely used model for assessing wildfire risk, relies on these parameters [64; 65]. Public health also benefits from forecasting these variables, as temperature and humidity affect the heat index. At the same time, wind plays a role in air quality by dispersing pollutants and airborne pathogens [66; 67]. Initially, the CSV file containing the data was uploaded. The columns labelled \"temp\", \"hum\", and \"windvel\" represent the time series of the variables we wish to predict (i.e., temperature, relative humidity, and wind speed, respectively). We began by asking the same question to all frameworks:  
```
''I have uploaded a file with a continuous time series from a weather station. Retain the first 90\% of the data for training and validation and the last 10\% for testing, in order to train an LSTM model that predicts the variables temp, hum and wind-vel. Write the code in Python that executes this task, including the appropriate error metrics, such as mean squared error, mean absolute error, mean relative error, mean fractional error, mean fractional bias, mean bias, R2 score and Pearson correlation coefficient. Include a graph that compares the last 10\% of the data during the testing phase. Figure 3: The InI index for the three frameworks assessed for the data cleaning and statistical computation task. The LSTM has one layer with 10 units, a batch size 16, the activation function is ReLU, and the optimiser is Adam with a learning rate of 0.001, running for 10 epochs. It alsouses a timestep of 3. If possible, please plot the results of the comparisons'. The description of the responses for each framework follows. ### Gpt

The GPT framework initially took 53 s to respond. It generated a code, which read from the data set the three required columns and created a training-testing data set only with these three columns, i.e., it used temperature, relative humidity, and wind speed only to predict temperature, relative humidity, and wind speed. From the question asked, GPT did not understand that all variables in the file had to be taken as input, so a new question (second attempt) was asked, and this was clarified:
```
''Please doasbefore,buttakeasinputtothemodelallgivenfeaturesfromtheCSVfile''. ```
This time, the model generated a Python code within 3 s, considering all variables to predict temperature, relative humidity, and wind speed. The code was executed, and predictions for the three variables were obtained. GPT did not manage to generate code that predicted all requested error metrics. It predicted MSE (0.5668), MAE (0.4735), MAPE (1.71 \(\times\) 10\({}^{13}\)), \(R^{2}\) Score (0.9405), and Pearson \(r\) (0.9997). MAPE is erroneous due to division by a minimal number (due to the existence of observed wind speeds close to 0). As observed, the GPT-generated LSTM model did not manage to evaluate error metrics for each variable separately; instead, it aggregated results across all selected columns and returned an overall mean value. Hence, we cannot comment on the accuracy of the GPT-produced model based on the calculated error metrics. This issue could be remedied by posing an additional question to GPT to compute error metrics for each variable separately. We comment on the model's accuracy and perform detailed comparisons with ground truth for each variable later in this section. In summary, the GPT framework provided an answer after 2 attempts and 2 queries. No SB responses were recorded.