OWASP LLM Top 10: GTK InsecureHandlingofLLMOutput

Title:
Insecure Output Handling

Alternative titles: 
Indirect Function Invocation
Indirect Code Injection


Author(s):

GTKlondike
Kai Greshake

Description:

An Insecure Output Handling vulnerability is a type of prompt injection vulnerability that arises when a plugin or application blindly accepts large language model (LLM) output without proper scrutiny and directly passes it to backend, privileged, or client-side functions. Since LLM-generated content can be controlled by prompt input, this behavior is akin to providing users indirect access to additional functionality. 

Successful exploitation of an Insecure Output Handling vulnerability can result in XSS and CSRF in web browsers as well as SSRF, privilege escalation, or remote code execution on backend systems. The impact of this vulnerability increases when the application allows LLM content to perform actions above the intended user's privileges. Additionally, this can be combined with agent hijacking attacks to allow an attacker privileged access into a target user's environment. 


Labels/Tags:

Label: "Output Filtering"
Label: "Prompt Injection"
Label: "Code Execution"


Common Examples of Vulnerability:

Example 1: LLM output is entered directly into a backend function, resulting in remote code execution.
Example 2: JavaScript or Markdown is generated by the LLM and returned to a user. The code is then interpreted by the browser, resulting in XSS

How to Prevent:

Prevention Step 1: Treat the model as any other user and apply proper input validation on responses coming from the model to backend functions.
Prevention Step 2: Likewise, encode output coming from the model back to users to mitigate undesired JavaScript or Markdown code interpretations.


Example Attack Scenarios:

Scenario 1: An application utilizes an LLM plugin to generate responses for a chatbot feature. However, the application directly passes the LLM-generated response into an internal function responsible for executing system commands without proper validation. This allows an attacker to manipulate the LLM output to execute arbitrary commands on the underlying system, leading to unauthorized access or unintended system modifications.

Scenario 2: A user utilizes a website summarizer tool powered by a LLM to generate a concise summary of an article. The website includes a prompt injection instructing the LLM to capture sensitive content from either the website or from the users conversation. From there the LLM can encode the sensitive data and send it out to an attacker-controlled server

Scenario 3: An LLM allows users to craft SQL queries for a backend database through a chat-like feature. A user requests a query to delete all database tables. If the crafted query from the LLM is not scrutinized, then all database tables would be deleted.

Scenario 4: A malicious user instructs the LLM to return a JavaScript payload back to a user, without sanitization controls. This can occur either through a sharing a prompt, prompt injected website, or chatbot that accepts prompts from a GET request. The LLM would then return the unsanitized XSS payload back to the user. Without additional filters, outside of those expected by the LLM itself, the JavaScript would execute within the users browser.

Reference Links:

* https://security.snyk.io/vuln/SNYK-PYTHON-LANGCHAIN-5411357
* https://embracethered.com/blog/posts/2023/chatgpt-cross-plugin-request-forgery-and-prompt-injection./
* https://systemweakness.com/new-prompt-injection-attack-on-chatgpt-web-version-ef717492c5c2?gi=8daec85e2116
* https://embracethered.com/blog/posts/2023/ai-injections-threats-context-matters/
* https://aivillage.org/large%20language%20models/threat-modeling-llm/


Author Commentary (Optional):

The purpose of my two findings are to make a distinction between the different types of "prompt injections". Agent Hijacking and Insecure Handling of LLM Output are both forms of prompt injection, but have very different impacts and require very different remediation steps. This is an attempt to make the concept of prompt injeciton more grainular and distinct. 

This is a situation with a large language model (LLM) where a plugin or application accepts LLM output and passes it, uncritically, into a backend or privileged function. Since users have significant control over LLM output, this would be similar to direct function access. This is extremely bad when an application accepts LLM content at a different privilege level than the user is supposed to have. 

The reference links include examples from past vulnerabilities and CVEs. While filtering input prior to invoking backend functions is common in application security today, this is notably absent from GPT plugins and LLM powered applications. Ideally, developers should consider LLMs to operate as possible threat actors just like they would with any other user. 

Like `A03_2021-Injection` from the OWASP Top 10 for applications, the injection point and remediation steps are identical for client-side and server-side attacks. Therefore, we've combined server-side and client-side attacks into the same category.