## 3 Simplistic Sequential Networks  
This section presents an analysis based on very simple NNs, consisting of only a few neurons. The goal was to illustrate how the nonlinear behavior of the NN increases with network complexity. We also wanted to determine how different training realizations of the same network can result in different behaviors of the NN. The NN is basically a function that takes a certain number of input parameters and produces a predefined number of output values. In our case the NN is a scalar function since it always outputs a single value (i.e., either \(T_{\text{max}}\) or \(T_{\text{min}}\)). Since we wanted to illustrate the NN behavior we limited the number of input parameters to two--this enabled us to visually show the behavior of the NN as 2D contour graph. We aimed to try out various setups of simplistic NNs, with different degrees of complexity and see how it affects the resulting behavior. We focused on the same-day forecast (forecast for the same day as the radiosonde measurement was made). We wanted to use two profile-based input parameters that would produce a reasonably good forecast of either \(T_{\text{max}}\) or \(T_{\text{min}}\). We experimented with various parameters derived from the vertical profiles. In the end, we chose the average temperature in the lowest layer between the ground and 1 km and the 90 th percentile of RH in the layer between the ground and 12 km (both parameters were calculated from the data of the original profiles, without interpolation to standard altitudes). The first parameter reflects the general temperature conditions in the boundary layer, which will depend on the season and also the general weather situation (the strong link between \(T_{\text{max}}\) and the temperature in the boundary layer is also clearly visible in Figure 2). The second parameter can be associated with the existence of cloudiness. As already mentioned, the clouds will weaken downward shortwave radiation near the ground during the day, which reduces the temperature near the surface. The radiosonde does not directly measure the existence of clouds. Still, it can be approximately inferred from the RH measurements (an RH value larger than 90% indicates a high likelihood of clouds at that altitude). Besides the possibility of either having none or at least some clouds, the cloud thickness also influences the downward shortwave radiation. If there are no clouds, the 90 th percentile of RH will have a relatively low value (i.e., significantly smaller than 100%), whereas if a sufficiently thick cloud layer is present, the 90 th percentile of RH will be close to 100%. The analyzed NN setups are described in Table 1. We started with the most simple NN with only a single neuron (Setup A). We first tried using the rectified linear activation function (ReLU), which did not work well. The reason was that during training, the two weights and the bias were oftentimes set to negative values, after which the training could not proceed anymore (this problem is referred to as the \"dying ReLU\" in the literature). The same problem also happened for other setups shown in Table 1, although not as frequently. The dying ReLU problem can be avoided using a slightly modified version of ReLU called the Leaky ReLU, which has a small slope for negative values that enables the training to proceed even if the weight and bias have negative values. One typical example of the behavior of Setup A is shown in Figure 4a. Since the setup contains only the output layer with a single computational neuron, and since Leaky ReLU was used as an activation function, the NN is a two-part piecewise linear function. As can be observed, the function visible in the figure is linear (at least in the shown region of parameter values--the transition to the other part of the piecewise-linear function happens outside the displayed region). This property is true for all realizations of Setup A. Table 1 also shows the average values of MAE for all the setups.