## 4 Results
### Results of the LLMs  
Figures 11-13 present a summary of the results for GPT-4 and its predecessor, GPT-3.5, regarding word class determination, sentence type identification, and case determination, respectively. The height of each bar reflects the number of entirely correct responses obtained when the same question is asked multiple times (cf. Section 3.2).  
Figure 11: The figure illustrates the part-of-speech (POS) determination accuracy for GPT-4 (blue) and GPT-3.5 (orange) across ten example sentences. The y-axis represents the percentage of completely correct responses out of 20 attempts for each model. Note that the percentages cannot be directly compared to the percentages of the bubble parser in Figure 7.  
Figure 10: A constituent parsing for the sentence “Es sei praktisch ausgeschlossen, dass der Dezember daran noch etwas andere, teilte die Organisation mit”.  
In the word class identification task (Figure 11), GPT-4 achieves a high success rate, consistently exceeding 85% across all sentences and trials. However, in sentences 3 and 5, GPT-4 occasionally misclassifies the relative pronouns as articles. Similarly, GPT-3.5 exhibits the same difficulties with sentences 3 and 5 but with a lower overall accuracy. Additionally, GPT-3.5 makes errors in sentence 2 and is entirely misled by the rare word class interpretation of the word \"einen\" in sentence 1.  
The results for clause type identification (Figure 12) differ significantly. GPT-4 consistently identifies clause types correctly in 5 out of 10 sentences (1, 2, 5, 8, and 10). However, the accuracy for the remaining sentences varies, ranging from 15% (sentence 9) to 95% (sentence 7). Notably, sentences 3, 6, and 9 frequently pose challenges, with the sentence structure and, consequently, the number and types of clauses being misidentified in many trials.  
In contrast, GPT-3.5 does not fully recognize the clause types in any of the sentences. While it achieves a success rate of over 50% in sentences 1, 2, 4, 5, and 10, it fails to correctly detect the structure of sentences 3, 6, 7, 8, and 9.  
Similarly, the accuracy in detecting the grammatical case of all nouns (Figure 13) varies considerably across sentences. In sentences 4, 5, 8, and 10, GPT-4 frequently misidentifies  
Figure 12: Same as Figure 11 for the clause type determination.  
Figure 13: Same as Figure 11 for the noun case determination.  
the case of certain nouns in multiple trials. The case of these nouns fluctuates between two possible cases, and in some instances, GPT-4 also proposes the third and fourth possible cases. These fluctuations are shown for each noun separately in sentences 5 and 8 as depicted in Figures 14 and 15, respectively. For instance, in the noun phrase \"mit dem Coronavirus\" in sentence 8, GPT-4 correctly identifies the dative case 12 times but incorrectly proposes the accusative case six times and the nominative and genitive cases once each. While all three incorrect cases are not valid for the entire phrase, the accusative case, which typically marks a direct object, is semantically more appropriate in this sentence than the nominative or genitive, which are each chosen once.  
GPT-3.5 performs considerably worse in the case determination compared to GPT-4. In sentences 4, 8, and 10, it misidentififies the cases in all 20 trials. It achieves a success rate of over 50% only in sentences 2 and 3, while in the remaining sentences, GPT-3.5 provides more incorrect than correct responses.  
Figure 14: The detection of the grammar cases by GPT-4 for all noun phrases in sentence 5 (“Der Vater, der heute das neue Auto, welches frisch aus der Fabrik kommt, vom Hof abholt, macht eine Probefahrt”). Shades of green and red colors indicate correct and false detections of the four grammar cases (NOM, AKK, DAT, and GEN). The numbers in the boxes represent how often the regarded case is proposed by GPT-4.  
Figure 15: Same as Figure 14 for the sentence 8 (“Nun richtet ein Professor aus Stanford scharfe Kritik an den deutschen Minister und seinen Umgang mit dem Coronavirus”).