CWE-1427: Improper Neutralization of Input Used for LLM Prompting

Description:
The product uses externally-provided data to build prompts provided to
large language models (LLMs), but the way these prompts are constructed
causes the LLM to fail to distinguish between user-supplied inputs and
developer provided system directives.

Extended Description:
When prompts are constructed using externally controllable data, it is
often possible to cause an LLM to ignore the original guidance provided by
its creators (known as the "system prompt") by inserting malicious
instructions in plain human language or using bypasses such as special
characters or tags. Because LLMs are designed to treat all instructions as
legitimate, there is often no way for the model to differentiate between
what prompt language is malicious when it performs inference and returns
data. Many LLM systems incorporate data from other adjacent products or
external data sources like Wikipedia using API calls and retrieval
augmented generation (RAG). Any external sources in use that may contain
untrusted data should also be considered potentially malicious.

Affected languages: Not Language-Specific
Affected technologies: AI/ML
Parent weaknesses: CWE-77

Consequences:
  - Confidentiality: Execute Unauthorized Code or Commands
  - Confidentiality: Read Application Data
  - Integrity: Modify Application Data
  - Access Control: Read Application Data

Detection methods:
  - Dynamic Analysis with Manual Results Interpretation: Use known techniques for prompt injection and other attacks, and
				adjust the attacks to be more specific to the model or system.
  - Dynamic Analysis with Automated Results Interpretation: Use known techniques for prompt injection and other attacks, and
				adjust the attacks to be more specific to the model or system.
  - Architecture or Design Review: Review of the product design can be effective, but it works best in conjunction with dynamic analysis.

Mitigations:
  - [Architecture and Design] LLM-enabled applications should be designed to ensure
proper sanitization of user-controllable input, ensuring that no
intentionally misleading or dangerous characters can be
included. Additionally, they should be designed in a way that ensures
that user-controllable input is identified as untrusted
  - [Implementation] LLM prompts should be constructed in a way that
effectively differentiates between user-supplied input and
developer-constructed system prompting to reduce the chance of model
confusion at inference-time.
  - [Architecture and Design] LLM-enabled applications should be designed to ensure
proper sanitization of user-controllable input, ensuring that no
intentionally misleading or dangerous characters can be
included. Additionally, they should be designed in a way that ensures
that user-controllable input is identified as untrusted
  - [Implementation] Ensure that model training includes training examples
that avoid leaking secrets and disregard malicious inputs. Train the
model to recognize secrets, and label training data
appropriately. Note that due to the non-deterministic nature of
prompting LLMs, it is necessary to perform testing of the sam

Demonstrative examples:
Consider a "CWE Differentiator" application that uses an an LLM generative AI based "chatbot" to explain the difference between two weaknesses.  As input, it accepts two CWE IDs, constructs a prompt string, sends the prompt to the chatbot, and prints the results. The prompt string effectively acts as a command to the chatbot component. Assume that invokeChatbot() calls the chatbot and returns the response as a string; the implementation details are not important here.
To avoid XSS risks, the code ensures that the response from the chatbot is properly encoded for HTML output. If the user provides CWE-77 and CWE-78, then the resulting prompt would look like:
prompt = "Explain the difference between {} and {}".format(arg1, arg2) 
					result = invokeChatbot(prompt) 
					resultHTML = encodeFo

Consider this code for an LLM agent that tells a joke based on
				user-supplied content. It uses LangChain to interact with OpenAI.
This agent is provided minimal context on how to treat dangerous
				requests for a secret. 
				 Suppose the user provides an input like:
from langchain.agents import AgentExecutor, create_tool_calling_agent, tool 
				  from langchain_openai import ChatOpenAI 
				  from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder 
				  from langchain_core.messages import AIMessage, HumanMessage 
				   
				  @tool 
				  def tell_joke(content): 
				   
					"""Tell a joke based on the provided user-supplied content""" 
					pass 
				   
				  tools = [tell_joke] 
				   
				  system_prompt = """ 
				  You are a witty and helpful LLM agent, r

Real-world examples:
  - CVE-2023-32786: Chain: LLM integration framework has prompt injection
				(CWE-1427) that allows an attacker to force the service to retrieve
				data from an arbitrary URL, essentially providing SSRF (CWE-918) and
	
  - CVE-2024-5184: ML-based email analysis product uses an
				API service that allows a malicious user to inject a
				direct prompt and take over the service logic, forcing
				it to leak the standard hard-coded syste
  - CVE-2024-5565: Chain: library for generating SQL via LLMs using RAG uses
				a prompt function to present the user with visualized results,
				allowing altering of the prompt using prompt injection (CWE-1427) to
		
  - CVE-2024-48746: AI-based integration with business intel dashboard allows prompt injection through its natural language component, allowing execution of arbitrary code