## 2 Background and Literature Review
### Scientific Literature on Maldoc Detection Using LLMs  
In a paper published in May 2024 by [PERSON] et al. [30], the authors present data on how LLMs can be leveraged to identify malicious behaviour at runtime. This is achieved by allowing such models to analyse large amounts of system call data. The pre-trained LLMs, including BERT, GPT-2, Longformer and Mistral, were fine-tuned by the addition of another layer to the artificial neural network and subsequent training on classified data. The results demonstrated that, in general, LLMs with a higher context window exhibited superior performance, achieving an F1-score of 0.86. However, the largest tested LLM  
Figure 10: GPQA Eval Leaderboard, last updated 26 June 2024.  
with a context window of 128k tokens exhibited a lower performance than the one with 8192 tokens. Our approach also employs LLMs to identify malicious aspects, albeit not from system calls, but from a single document. Moreover, it is not a fine-tuned LLM, but a general-purpose LLM with a considerably larger context window of 200,000 tokens. A thesis by [PERSON] about 'Detection of Malicious OOXML Documents Using Domain Specific Features' [31] suggests taking all document aspects into consideration when trying to classify a document as either benign or malicious. For that, they are not extracting a few specific aspects out of the document to create their features, but try to represent every aspect, including metadata and file structure. As this approach also employs machine learning techniques with selected features, it may be insufficiently robust to cope with new, unknown methods of exploiting the OOXML file format for malicious purposes. In [32], published on 9 August 2024, by [PERSON] et al., the authors tested the LLMs GPT-3 and GPT-4 in order to detect malicious packages from the JavaScript package manager npm. The authors demonstrated that the precision and F1-score were superior in the models under examination in comparison to the static analysis tool CodeQL. However, the authors also demonstrated that both models exhibited occasional hallucination and mode collapse. Mode collapse refers to the phenomenon where models generate highly similar outputs, failing to identify the distinctive characteristics of the presented data, which ultimately leads to misclassification. The model employed in this study, Claude 3.5 Sonnet by Anthropic, exhibits similarities to GPT-4 by OpenAI. However, it is specifically oriented towards the detection of malicious indicators within OOXML files, such as.docx or.xlsm. In April 2024, a paper was published by [PERSON] et al. [33] that evaluated the ability of LLMs to successfully deobfuscate malicious code. The authors submitted obfuscated PowerShell scripts to LLMs such as GPT-4, Gemini Pro, Llama and Mixtral, with the task of extracting the URLs. It was found that cloud models performed better than local models, with GPT-4 being the best, with 69.56%, and Mixtral accuracy being the worst out of the four, with 11.59%. Hallucinations were observed during the testing phase, resulting in the generation of non-existent domains, and occasionally the LLMs refused to respond due to ethical concerns. The conclusion is that LLMs show potential in deobfuscating code, but are not yet ready to replace traditional methods, also considering the wide range of accuracy when evaluating different models. The approach presented in this paper also makes use of the deobfuscation capabilities of the LLM. Although not inherently relevant for this proposal, the above-mentioned paper highlights the potential of LLMs to aid in malware analysis. A study conducted by [PERSON] et al. [34] from 2020 tests various exploitation techniques in OOXML and ODF files that are possible by design rather than as an unpatched bug. The authors highlight the abuse of URL invocation, data exfiltration and content masking. The techniques described demonstrate the enormous potential for abuse of the OOXML file format.