© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 

Secure Development of GenAI Systems

An AWS Perspective

Mark Ryland

N I S T V I R T U A L W O R K S H O P

Director, Amazon Security
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 

AWS’s approach

Security remains our top priority, now extended into this important 

and dynamic space

Fast-evolving technology with emerging best practices for risk 

identification, mitigation; still much to learn and to put into practice

Committed to continued rapid innovation while collaborating with 

government and industry to support safe, secure, and responsible AI

Lots of good additional detail here: 

https://aws.amazon.com/uki/cloud-services/uk-gov-ai-safety-summit/

2
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 

Generative AI Security Scoping Matrix

Consumer App

Using ‘public’ generative 

AI services

E.g.: PartyRock, ChatGPT, 

Midjourney

Enterprise App

Using an app or SaaS 

with generative AI 

features

E.g.: Salesforce Einstein 

GPT-x, Amazon 

CodeWhisperer

Pre-trained Models

Building your app on a 

versioned, pre-trained 

model

E.g.: Amazon Bedrock 

hosted foundation 

models (e.g., Claude, 

Llama 2)

Fine-tuned Models

Fine-tuning a model on 

your data

E.g.: Amazon Bedrock 

customized models, 

Amazon SageMaker 

JumpStart

Self-trained Models

Training a model from 

scratch on your data

E.g.: Amazon Titan

A M E N T A L M O D E L T O C L A S S I F Y U S E - C A S E S

Governance & Compliance Legal & Privacy Risk Management Controls Resilience

Securing Generative AI
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 

Preliminary remarks

Today’s focus is on foundation models (of all types), but lessons apply 

to all types of machine learning

Security is the focus, but themes and learnings are useful for broader 

responsible AI topics

“Red-teaming” often used in this space, but think more broadly in 

terms of general testing and adversarial testing

Transparency is important, thus the industry adoption of “service 

cards” or “model cards”

4
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 

Prepare the Organization

Focus on integration of software engineering and data science

Define policies for data sourcing, wrangling, cleansing, quality control

Re-use/refine existing data classification policies and schemes

Take an end-to-end system approach; foundation model security and 

safety only part of the story

Shared responsibility among data owners, model owners, fine-tuners, RAG 

designers, and application developers (as well as ML infrastructure provider(s))

5
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 

Protect the Software(+)

Data focus: provenance/protection of data now first-class requirement

Use traditional ML technologies to filter and classify input (and output) data

Separate duties and apply least privilege concepts and technologies 

must be refined and updated to include data and data lifecycle

E.g., model weights are sensitive data with no need for direct human access

Expand Secure DevOps models to “MLOps,” to include data science 

workstreams and security capabilities

Build in guardrails/filters and other kinds of input/output “circuitbreakers” as large models are hard to modify

6
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 

Example data flows in a generative AI application

7

Generative AI application Pre-trained 

model

Training data

Plugins to query 

custom data sources Response

Fine-tuned 

model

Fine-tuning 

Customer data data

Context

Completions

Prompt Response

1. App receives input from user

• [Optional] App queries data 

from custom data sources

2. App formats user input and 

customer data into a prompt

3. Prompt is completed by a model 

(fine-tuned or pre-trained)

4. Completion is processed by app 

5. Response is sent to the user

1

2

4

3

5

4’ 3’

A A

B B

Each layer responsible for 

appropriate guardrails/filters 

for inputs (1,3,A) and 

outputs (B,4,5)
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 

Produce Well-Secured Software(+)

Securing both the software and data “supply chain” (lineage, 

provenance, confidentiality, integrity)

Testing at each stage (equivalent of integration and unit tests)

Structured, consistent, constantly improving use of internal and 

external pen-testing, automation; true “red-teaming” as well

Adversarial testing: use human experts but constantly increase 

automation

E.g., Vary prompt attacks with logical equivalent of “fuzz testing”

Understand and accept today’s limitations of GenAI technology!

8
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 

Respond to Vulnerabilities(+)

Well-tested programs and process for vulnerability (and other flaws) 

reporting (internal, external, bug bounty, etc.)

Large models too expensive to rebuild frequently, so... 

Rollback strategies for system updates that expose flaws

Guardrails/filtering at each layer,[1] “disgorgement”[2] strategies

Incorporate learnings into next round of model building, as well as 

other layer updates

9

[1] E.g., NeMo Guardrails, https://github.com/NVIDIA/NeMo-Guardrails; Bedrock Guardrails, 

https://aws.amazon.com/bedrock/guardrails/

[2] E.g., “AI Model Disgorgement: Methods and Choices,” https://arxiv.org/pdf/2304.03545.pdf
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 

Thank you

Mark Ryland

N I S T V I R T U A L W O R K S H O P

markry@amazon.com