## 5 Evaluation
### Case Study  
Figure 6 shows several real cases of solving challenges, revealing the advantage of our framework as well as its challenges.  
Figure 4: The distribution of solution methods applied to the GITW challenges. For some challenges solved using multiple methods, only the most significant one is recorded.  
Figure 5: Results on the THUCTF dataset; all agents based on GPT-4o were run. **Planning only** indicates cases where agents had already produced a correct solution plan but may have failed to complete the task in the execution environment.  
Figure 6: Several real cases of our framework attempting to solve challenges. Only the key steps that led to critical clues or eventual failure are shown; other steps are omitted for brevity.  
Case 1: Extension of External Tool Usage  
In the GITW-ZH-3.3 test, the problem involved a Monoalphabetic Substitution Cipher. Initially, the agent failed to correctly identify the encryption scheme and attempted to decrypt the ciphertext using a Vigenere cipher tool via the browser. However, after this attempt failed, the agent recognized that its initial assumption about the cipher type was likely incorrect. It then proactively used a Cipher Identifier Tool available on the same website, followed the suggested identification result, navigated to the correct decryption tool, and ultimately obtained the correct answer. This case illustrates the agent's emerging ability to generalize tool usage, demonstrating not only basic tool invocation but also the capacity to adjust strategy, recover from failure, and make use of auxiliary web-based resources in a goal-directed manner.  
Case 2: Analytical Capabilities of the Reasoning Model Itself  
In the GITW-ZH-13.3 test, the problem presented the ciphertext:  
9\(\backslash\)4\(|\)2\(\backslash\)8\(\backslash\)7\(/\)8\(\backslash\)4\(|\)31\(6/\)7\(|\)4\(/\)4\(/\)6\(|\)2\(\backslash\)5\(/\)3\(/\)2\(\backslash\)6\(|\)4\(/\)5\(/\)9\(/\)6\(|\)2\(\backslash\)6\(|\)31\(6/\)3\(/\)4\(|\)2\(\backslash\)7\(|\)7\(|\)9\(/\)4\(|\)6\(/\)8\(|\)3\(\backslash\)4\(/\)6\(|\)4\(/\)7.  
The intended solution required interpreting each digit-symbol pair as a unit and mapping it to a letter. When GPT-4o was used as the base model of our framework, the agent failed to identify the correct strategy for decoding the ciphertext. In contrast, when powered by Deepseek-R1, the agent demonstrated more flexible and exploratory reasoning during its internal \"Thinking\" process. It attempted multiple interpretations of the format and eventually discovered the correct decoding strategy by recognizing the structure as a variant of the Telephone Keypad encoding. This allowed the agent to correctly solve the problem.  
This case highlights the impact of reasoning models on problem-solving effectiveness. Deepseek-R1 exhibited greater adaptability in ambiguous scenarios and was able to identify non-obvious semantic patterns.  
Case 3: Failure Case of THUCTF Problems  
Within the THUCTF dataset, the causes of failure differ, one of the most common being the agent's inability to effectively process the information provided. This results in poor performance even on seemingly simple problems.  
In the THUCTF Emojicorre problem, the ciphertext consisted of a large sequence of emoji characters, suggesting that each emoji likely corresponded to a letter in a substitution cipher. The agent successfully inferred that. However, instead of performing any preprocessing or transformation, it directly inputs the emoji string into the QuipQuip [40] website, a tool designed for solving letter-based substitution ciphers. Since QuipQuip does not support emoji characters and operates only on alphabetical inputs, the tool failed to produce a meaningful result. This case highlights the agent's limitations in adapting input formats to tool requirements and reflects a broader issue: the lack of robustness in reasoning and data preprocessing when faced with a changing situation.