Image by evertonpestana from Pixabay

INTRODUCTION TO CLOUD ☁ FORENSICS: STORAGE & ACQUISITION
Simson L. Garfinkel*
US Census Bureau
Tuesday, December 10, 2019

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

DISCLAIMER:
The views in this presentation are those of theauthor, and not those of the US Census Bureau.

* Associate Professor Naval Postgraduate School: 2006-2014
Senior Advisor, Information Access Division, NIST: 2015-2016
Senior Computer Scientist, US Census Bureau: 2017-

Disclaimer & Level SeQng
Cloud forensics — DeﬁniTons
Amazon Web Service forensic
targets
Instance-based acquisiTon:RAM
Instance-aZached drives
ElasTc Block Service (EBS)

Backup Slides:
CreaTng an instance
EC2 Command Line Tools
AWS EBSAWS CloudTrailAWS EFS
Running bulk_extractor in AWS

Service-based acquisiTon:
Simple Storage Service (S3)Cloud Watch

Outline of this brieﬁng
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

Disclaimer
This presentation is based on:• Working with Amazon Web Services (AWS) since August 2006 (S3 and EC2)
• Cloud Forensics course at George Mason University—http://bit.ly/Cloud_Forensics_2018• Work at US Census Bureau in AWS GovCloud.
—Elastic Compute Cloud (EC2)—Simple Storage Service (S3)—Elastic Map Reduce (EMR)—Largest Cluster size: 50 r5d.24xlarge nodes = 4,800 CPU cores & 38.4 TiB RAM

This presentation focuses on Amazon Web Services• Microsoft Azure and Google Cloud Platform have many competitive services.
• IBM and Oracle also have significant offerings. Alibaba Cloud has great prices!
• US Census Bureau has adopted AWS GovCloud.
• I use Dreamhost for my personal stuff (limited to compute & storage).
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

3

Disclaimer 2 —I will not cover AWS Security Guides or AWS “digital forensics”
AWS Security Incident Response Guide• https://d1.awsstatic.com/whitepapers/
aws_security_incident_response.pdf

Detecting and Responding to Security issues inyour AWS environment• https://d1.awsstatic.com/Marketplace/scenarios/
security/SEC_11_TSB_Final.pdf“Margaritashotgun”• Claims to be a “Python Remote Memory
Acquisition” for EC2• Actually logs into the EC2 instance and performs amemory acquisition…
• https://github.com/ThreatResponse/
margaritashotgun

AccessData Lab• “Bring your own lab.”
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

4

Level Setting — Show of Hands
Who has worked with AWS EC2 instances?
Collected RAM from an EC2 instance?
Imaged EBS volumes?
Collected data from Amazon S3?
Used Amazon’s internal APIs to perform undocumented functions?

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

5

Cloud Forensics: DeﬁniYons
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

We are all using the cloud☁https://www.pexels.com/photo/interior-of-oﬃce-building-325229/

The Internet

Apps onend-userdevices

https://www.pexels.com/photo/facebook-like-mi-mobile-325053/
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

7

Cloud forensics: It's where the data are!
"Everything is moving to the cloud."That means digital evidence is moving to the cloud.
Digital evidence on end-user-devices is increasingly encrypted.

This works less and less.
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

All of the data are in the cloud.
CLOUD ☁ FORENSICS

8

Cloud forensics: It's where the research is.
The "cloud" is just a bunch of data centers.
Forensics tools for data centers are poorly developed.
• Most “cloud forensics” is really traditional incident response running in AWS.

This is an excellent opportunity!
• Virtualization• Software Defined Networks (SDN)• Big data analysis techniques.

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

9

Cloud computing is more than just “big, remote data centers.”
NIST Special Publication 800-145: The NIST Definition of Cloud Computing
https://csrc.nist.gov/publications/detail/sp/800-145/final

Essential Characteristics:• On-demand self-service• Broad network access• Resource pooling• Rapid elasticity• Measures service

Deployment Models:• Private cloud• Community cloud• Public cloud• Hybrid cloud
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

10

Cloud hosting models

https://www.hostingadvice.com/how-to/iaas-vs-paas-vs-saas/

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

11

Different kinds of data are at each layer
Program Layer — code that runs in the cloud to manipulate the data
• Desktop applications (cloud-based desktops) (e.g. Amazon WorkSpaces & AppStream)• Custom applications written in python, Scala, C++, Go, Java, etc.
• Back-ends for websites and mobile apps.

Software Infrastructure Layer — where the code runs
• YARN, Hadoop, MapReduce, Spark, etc.
• Databases: HBase, RDS

Operating System Layer — what you log into
• Linux (Centos), Windows

Virtualization Layer — the runtime environment• “Bare Iron” or Xen

Hardware Layer — the physical hardware on which the VMs run• Intel or AMD systems; GPUs
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

12

Other services common in cloud computing environments“Functions as a Service” — Serverless Computing• Isolates business logic from the problem of running servers.
• Amazon Lambda; Google Cloud Functions; Azure Functions

Hadoop & Apache Spark — Big Data Computing• Designed for processing data larger than the largest server.
• Amazon Elastic Map Reduce — automatically scales cluster with workload

Other big data services:• Amazon Athena — “serverless interactive query service.”—Runs queries on data stored in Amazon S3.

• Amazon Redshift — Cloud data warehouse for structured and semi-structured
data.

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

13AWS ArchitectureEC2 11/29/2015
Non-EC2 Services

AWS Forensic Targets
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

Amazon Web Services — Brief History
July 5, 1994 — Amazon.com was founded by Jeff Bezos• (Originally named “Cadabra”)• Renamed “Amazon” in 1995 with goal of being the “biggest” store in the world.
• First book ordered in 1995, Fluid Concepts and Creative Analogies.

By 1998, more than 100 computers processed data for every rendered page.
• Authentication; shopping cart; search results; recommendations; feedback; …
• Amazon made organizing thousands of computers an institutional priority.

In 2006, Amazon started making its systems available as a commodity• March: Simple Storage Service (S3) — unlimited storage• July: Simple Queue Service (SQS) — Reliable messages up to 256KB in size.
• August:—Elastic Compute Cloud (EC2) — virtual machines—Elastic Block Store (EBS) — disks for virtual machines
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

15AWS Global Infrastructure:
22 Regions; 69 Availability Zones
Location matters:• Speed of light: 300,000 Km/sec• Distance to Seattle: ≈5,000 Km• Minimum time to Seattle: = 1.6 msec
• Distance to Reston: ≈ 30Km
• Minimum time to Reston: 30 Km ÷ 300,000 Km/sec = 99µsec

https://aws.amazon.com/about-aws/global-infrastructure/

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

16AWS is divided into “regions” and “availability zones”

Code
us-east-1
us-east-2
us-west-1
us-west-2

Name
US East (N. Virginia)
US East (Ohio)
US West (N. California)
US West (Oregon)

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

17

Manage AWS with the Graphical User Interface (GUI) or theCommand Line Interface (CLI)
CLI:$ aws ec2 create-volume --size 10 --region $aws_region --availability-zone us-east-1b{"AvailabilityZone": "us-east-1b","Encrypted": false,"VolumeType": "standard","VolumeId": "vol-46cdb6a5","State": "creating","SnapshotId": "","CreateTime": "2015-12-05T19:01:38.548Z","Size": 10}GUI:

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

18

Amazon EC2 — Elastic Compute Cloud
Virtual machines in the cloudEC2 is based on “Instances”• Horizontal Scaling — Create many VMs.
• Vertical Scaling — Create small and large VMs (cores, RAM, networking)• Geographical Diversity — Create in different locations (“availability zones”)

Each instance has:
• Unique Instance ID (e.g. i-04f679de246dc9c10)• AMI — Amazon Machine Image — the initial “boot volume”• Network interface(s) and firewall• Instance Type (e.g. “t2.micro”) with specific CPU and RAM (1 vCPU; 1.0 GiB)
• Security Groups (what is can and cannot do)• Key pair (used for accessing)

Instances optionally have:• Virtual drives — Elastic Block Store; can survive shut-down.
• Attached physical drives — in the box; lost when VM terminates
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

19

EC2 Instance life cycle:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-lifecycle.html

All instances boot from an AMI (you can upload your own.)
You specify if the EBS volume is kept or lost on termination.
You pay for:• Instances that are running*• EBS-backed storage• Bandwidth from EC2➜Rest of Internet
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

20*NOTE: instances with attached SSDs bill even when stopped

EC2 Instance control panel:

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

21

Accessing an instance: AWS key pairs
Linux instances are accessed via SSH (Secure Shell)• AWS uses SSH “public key authentication.”• Two ways to get your public key to Amazon:—You create a public/private keypair with “ssh keygen -t rsa -f mykey.pem” & import—Amazon will create the pair and you download it.

• You use the private key to authenticate.

Key pairs:• Each key is identified by a “Fingerprint.”• If you lose your private key, you can’t access your server.

Once an instance starts up, you can add additional users.
You can also access using network-based exploits.
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

22

Each instance has at least 1 “virtual” interface,but possibly 2 IP addresses.
Amazon assigns a private IP address and (optionally) a public IP address.
• Private IP address is the “real” address on your private subnet.
• Amazon uses two-way NAT to provide the “public” address.
• NAT implements firewall through “security groups.”

Other options:• You can have only private addresses. (More secure.)• VPN to your organization.

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

23

EC2 Instance forensic targets
Service-based
acquisition:
Acquire fromanywhere within
AWS☁

Public Internet.
Many acquisition
methodologies

Network
Intercept

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

Logs & ACL violations

DynamoS3: Simple Storage Service

Redshift

RDS: Relational Database Service

IAM

EFS: Elastic File System

Cloud Watch

EBS: Elastic Block Service

Instance-based
acquisition:
Acquire fromthe EC2
instance.

EC2
Instance

RAM
Attached SSDs

CLOUD ☁ FORENSICS

24

Instance-Based AcquisiYon
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

Instance-Based Acquisition: RAM
Traditional approaches for RAM acquisition in a virtualized environment:• Suspend VM and access .vmem file
• Acquisition through the hypervisor• Log into VM and run a RAM-dumping tool.

XenServer has the ability to dump memory too!
Acquisition on Amazon:• Log into the VM and run a RAM-dumping tool.
• Amazon might be able to use the hypervisor to dump, but users can’t!
—You may wish to discuss this privately with Amazon under NDA…

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

26

Instance-Based Acquisition: Disk
EC2 has two kinds of storage
Ephemeral storage / Instance Storage• part of the instance (local drives) faster.

EBS — separate devices — slower, but can persist.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

27

Instance-Based Acquisition:
Instance Storage / Ephemeral Storage
Some EC2 instance types include internal storage

vCPUs

MemoryLocal Storage

EBS-Optimized
Bandwidth

Network Bandwidth

r5d.large

2

16 GiB

1 x 75 GB NVMe SSD

Up to 3.5 Gbps

Up to 10 Gbps

r5d.xlarge

432 GiB1 x 150 GB NVMe SSD

Up to 3.5 Gbps

Up to 10 Gbps

r5d.2xlarge

864 GiB1 x 300 GB NVMe SSD

Up to 3.5 Gbps

Up to 10 Gbps

r5d.4xlarge

16

128 GiB2 x 300 GB NVMe SSD

3.5 Gbps

Up to 10 Gbps

r5d.12xlarge

48

384 GiB2 x 900 GB NVMe SSD

7.0 Gbps

10 Gbps

r5d.24xlarge

96768 GiB

4 x 900 GB NVMe SSD

14.0 Gbps

25 Gbps

Instance Name

The only way for you to acquire the NVMe SSD is to log into the instance.
• (Amazon might be able to acquire it using Xen hypervisor)
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

28

Instance storage cannot be used for the root file system.
Most instance storage is now hardware encrypted.

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

29

Instance store is lost when an instance is stopped`

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

30

EBS volumes can be initialized from snapshots...

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

31

Forensic Acquisition: EBS (NETWORK ATTACHED STORAGE)
Option #1: Acquire through EC2 instance:• Log into EC2 instance• Run a traditional disk imaging program (e.g. dd, ewfacquire, etc.)• Write disk image to:—Another device • Amazon S3 • Network socket

Option #2: Acquire through Amazon infrastructure:• Use AWS GUI or CLI to snapshot the EBS volume.
• You can snapshot a running volume• Snapshots are fast: 8GiB in a few seconds• Restore the snapshot on a new volume on another system (CLI or GUI)

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

32

More on EBS snapshots
Snapshots and be Public or Private:

To read an EBS snapshot, restore it onto a new volume:

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/EBSSnapshots.html
CLOUD ☁ FORENSICS

33

AWS EC2 Rescue Tool
Analyzes offline instances to perform reset, restore, and log capture

https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/Windows-Server-EC2Rescue.html
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

34

Many forensic capabilities!
Collect logs!
• https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2rw-cli.html

FixAll — Fix things that tend to break on AWS• System Time• Windows Firewall• Remote Desktop• EC2 Config• DHCP

ResetAccess —Reset the Administrator Password

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

35

Service-based AcquisiYon: S3
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

S3 is an object-based storage system
Every S3 bucket has:
• Name• Owner• Access permissions• Region where located• Optional event notifications• Optional logging• Optional static web hosting• Optional “requester pays”• Policy

Amazon S3
AWS Regions

Buckets

Every S3 object has:
• Size• URL• Access permissions

Objects in the bucket—e.g. world readable• Optional encryption• Optional policy
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

37

Accessing S3 data
Uses of S3:• Storing logs
• Distributing data
• Objects for large-scale web apps (documents, JPEGs, etc.)

Advantages of S3:—permanence; S3 outlasts your EC2/EMR cluster—Pay only for what you need, rather than for virtual drives capacity.

Disadvantage of S3—No data locality. S3 data always moves over the network.
—High-latency to access each object—Bulk data transfer must be done in parallel.

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

38

S3 access protocol: REST
REST is built on top of HTTP.

HTTP PUT
puts object into
bucketU.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

HTTP GET
gets object frombucket

CLOUD ☁ FORENSICS

39

Many ways to access data on Amazon S3AWS GUI• Simple object inspection can be done from a
web browser

AWS CLIaws s3 ls s3://bucketname/prefix/• Allows for parallelized upload and download.

HTTP / REST — Representational State Transfer• Uses HTTP methods (with a bit of JSON)• HTTP GET — Reads a resource without causing
any side effects• HTTP DELETE — Deletes a resources• HTTP PUT (or POST) — Creates a new resources• HTTP POST (or PUT) — Modify a resource’s value

HTTP / SOAP — Simple Object Access Protocol• Structure XML-based protocol• Heavy weight; increasingly not used.

HTTP Hosting• Different from REST
• Must be explicitly enabled
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

BitTorrent• S3 can host a “tracker” and “seeds”• Limited to objects 5GB in size
CLOUD ☁ FORENSICS

40

Amazon Snowball and Snowball Edge
50TB or 80TB of storage in a ruggedized container
e-ink shipping label
Snowball Edge includes:• Amazon S3• Amazon EC2• Amazon Lambda
https://aws.amazon.com/blogs/aws/aws-importexport-snowball-transfer-1-petabyte-per-weekusing-amazon-owned-storage-appliances/

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

41

Backup Slides
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CreaYng an instance
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

Putting it all together…

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

44

Putting it all together…

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

45

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

46

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

47

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

48

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

49

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

50

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

51

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

52

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

53

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

54

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

55

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

56

Instance is running…

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

57

Connect…
simsong@nimi ~ % ssh ec2-user@107.20.122.243
The authenticity of host '107.20.122.243 (107.20.122.243)' can't be established.
ECDSA key fingerprint is SHA256:MKlTMdgi3FvK9rCSe++Q0Bt+/MQfqicf63pkVsD9YDk.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '107.20.122.243' (ECDSA) to the list of known hosts.
__| __|_ )_| (/___|\___|___|

Amazon Linux 2 AMI

https://aws.amazon.com/amazon-linux-2/
5 package(s) needed for security, out of 13 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-172-30-4-236 ~]$U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

58

We have a running instance!
[ec2-user@ip-172-30-4-236 ~]$ df -hFilesystem
Size Used Avail Use% Mounted on
devtmpfs
475M
0 475M
0% /dev
tmpfs
492M
0 492M
0% /dev/shm
tmpfs
492M 400K 492M
1% /runtmpfs
492M
0 492M
0% /sys/fs/cgroup/dev/xvda1
8.0G 1.3G 6.8G 16% /
tmpfs
99M
0
99M
0% /run/user/1000[ec2-user@ip-172-30-4-236 ~]$ top
top - 19:50:44 up 6 min, 1 user, load average: 0.00, 0.05, 0.03
Tasks: 83 total,
1 running, 46 sleeping,
0 stopped,
0 zombie%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 1007276 total,
610048 free,
60176 used,
337052 buff/cache
KiB Swap:
0 total,0 free,0 used.
807136 avail Mem
PID USER
1 root2 root4 root5 root
6 root7 root8 root9 root10 root
11 root
12 root
14 root
15 root
21 root

PR NI
20
0
20
0
0 -20
20
0
0 -20
20
0
20
0
20
0
rt
0
rt
0
20
0
20
0
0 -20
20
0

VIRT
125512
0
0
0
0
0
0
0
0
0
0
0
0
0

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

RES
5500
0
0
0
0
0
0
0
0
0
0
0
0
0

SHR S %CPU %MEM
4064 S 0.0 0.5
0 S 0.0 0.0
0 I 0.0 0.0
0 I 0.0 0.0
0 I 0.0 0.0
0 S 0.0 0.0
0 I 0.0 0.0
0 I 0.0 0.0
0 S 0.0 0.0
0 S 0.0 0.0
0 S 0.0 0.0
0 S 0.0 0.0
0 I 0.0 0.0
0 S 0.0 0.0

TIME+ COMMAND
0:01.59 systemd
0:00.00 kthreadd
0:00.00 kworker/0:0H
0:00.02 kworker/u30:0
0:00.00 mm_percpu_wq
0:00.04 ksoftirqd/0
0:00.15 rcu_sched
0:00.00 rcu_bh
0:00.00 migration/0
0:00.00 watchdog/0
0:00.00 cpuhp/0
0:00.00 kdevtmpfs
0:00.00 netns
0:00.00 xenbus

CLOUD ☁ FORENSICS

59

EC2 Command Line Tools
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

Amazon provides command line tools
Can be run from any Linux, Mac or Windows computer.
• Faster interaction than web interface.
• Can be scripted.

AWS Command Line Interface• Run through “aws” command• Flexible output — JSON, text, tables• List EC2 instance: $ aws ec2 describe-instances
https://aws.amazon.com/cli/
http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html

Elastic Comput Cloud CLI• Run through 176 different ec2-* commands• List EC2 instances: $ ec2-describe-instances
http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DescribeVolumes.html

Credentials:• Credentials kept in $HOME/.aws/ directory• Credentials kept in AWS_USERNAME, AWS_ACCESS_KEY, AWS_SECRET_KEY environment variables.

Both are pre-installed on Amazon’s AMIs. Use the AWS CLI if possible.
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

61

Set up your environment variables and test:AWS CLI command:$ aws ec2 describe-regions
REGIONSec2.eu-north-1.amazonaws.com eu-north-1
REGIONSec2.ap-south-1.amazonaws.com ap-south-1
REGIONSec2.eu-west-3.amazonaws.com eu-west-3
REGIONSec2.eu-west-2.amazonaws.com eu-west-2
REGIONSec2.eu-west-1.amazonaws.com eu-west-1
REGIONSec2.ap-northeast-2.amazonaws.com
ap-northeast-2
REGIONSec2.ap-northeast-1.amazonaws.com
ap-northeast-1
REGIONSec2.sa-east-1.amazonaws.com sa-east-1
REGIONSec2.ca-central-1.amazonaws.com
ca-central-1
REGIONSec2.ap-southeast-1.amazonaws.com
ap-southeast-1
REGIONSec2.ap-southeast-2.amazonaws.com
ap-southeast-2
REGIONSec2.eu-central-1.amazonaws.com
eu-central-1
REGIONSec2.us-east-1.amazonaws.com us-east-1
REGIONSec2.us-east-2.amazonaws.com us-east-2
REGIONSec2.us-west-1.amazonaws.com us-west-1
REGIONSec2.us-west-2.amazonaws.com us-west-2

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

62

EC2 has a command-line interface
Show running instances:[ec2-user@ip-172-30-4-236 ~]$ aws ec2 describe-instances
RESERVATIONS
376778049323
086189789714
r-08090e555597ca8cd
INSTANCES0x86_64
157333535705912490
False
True
xen
ami-02eac2c0129f6376b
i-04f679de246dc9c10
t2.micro
id_rsa2019-11-15T19:18:05.000Z
ip-172-30-1-55.ec2.internal
172.30.1.55
18.212.220.250/dev/sda1 ebs
True
subnet-b1de03c6
hvm
vpc-8e73cfeb
BLOCKDEVICEMAPPINGS/dev/sda1
EBS 2019-11-09T21:36:00.000Z
False
attached
vol-0aab795976166105c
CAPACITYRESERVATIONSPECIFICATION
open
CPUOPTIONS
1
1
HIBERNATIONOPTIONS
False
MONITORING
disabled
NETWORKINTERFACES
Primary network interface
0a:e8:03:05:29:67
eni-094d36080627e1676
376778049323
172.30.1.55 True
in-usesubnetb1de03c6
vpc-8e73cfeb
ASSOCIATION
amazon18.212.220.250
ATTACHMENT
2019-11-09T21:35:59.000Z
eni-attach-092f65b6f1f7f26beTrue0
attached
GROUPS
sg-06ec94cd40903890a
CentOS 7 -x86_64- - with Updates HVM-1901_01AutogenByAWSMP-1
PRIVATEIPADDRESSES
True
172.30.1.55
ASSOCIATION
amazon18.212.220.250
PLACEMENT
us-east-1b
default
PRODUCTCODES
aw0evgkw8e5c1q413zgy5pjce
marketplace
SECURITYGROUPS
sg-06ec94cd40903890a
CentOS 7 -x86_64- - with Updates HVM-1901_01AutogenByAWSMP-1
STATE 16
running
TAGS Name
CentosU.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

63

Use “help” to get help
$ aws ec2 describe-instances help
NAME
describe-instances DESCRIPTION
Describes one or more of your instances.
If you specify one or more instance IDs, Amazon EC2 returns information
for those instances. If you do not specify instance IDs, Amazon EC2
returns information for all relevant instances. If you specify an
instance ID that is not valid, an error is returned. If you specify an
instance that you do not own, it is not included in the returned
results.
Recently terminated instances might appear inThis interval is usually less than one hour.

thereturned

results.

describe-instances is a paginated operation. Multiple API calls may be
issued in order to retrieve the entire data set of results. You candisable pagination by providing the --no-paginate argument. When using--output text and the --query argument on a paginated response, the--query argument must extract data from the results of the following
query expressions: Reservations

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS64

$ ec2-describe-instance-status — see what’s running$ aws ec2 describe-instance-status --output=text
INSTANCESTATUSES
us-east-1b
i-5c306beb
INSTANCESTATE
16
running
INSTANCESTATUS
ok
DETAILS
reachability passed
SYSTEMSTATUS
ok
DETAILS
reachability passed
INSTANCESTATUSES
us-east-1b
i-9a48aa2c
INSTANCESTATE
16
running
INSTANCESTATUS
ok
DETAILS
reachability passed
SYSTEMSTATUS
ok
DETAILS
reachability passed$

Change output format:[ec2-user@ip-172-30-4-236 ~]$ aws ec2 describe-instances --output table---------------------------------------------------------------------------------------|
DescribeInstances|+--------------------------------------------------------------------------------------+||
Reservations|||+-----------------------------------+------------------------------------------------+||| OwnerId| 376778049323|||| RequesterId| 086189789714|||| ReservationId| r-08090e555597ca8cd|||+-----------------------------------+------------------------------------------------+||||
Instances|||||+-----------------------------------+----------------------------------------------+||||| AmiLaunchIndex| 0|||||| Architecture| x86_64|||||| ClientToken| 157333535705912490|||||| EbsOptimized| False|||||| EnaSupport| True|||||| Hypervisor
| xen
|||
||| ImageId| ami-02eac2c0129f6376b|||||| InstanceId| i-04f679de246dc9c10|||||| InstanceType| t2.micro|||||| KeyName| id_rsa|||||| LaunchTime| 2019-11-15T19:18:05.000Z|||||| PrivateDnsName| ip-172-30-1-55.ec2.internal|||||| PrivateIpAddress| 172.30.1.55|||

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

65

JSON output is more useful for scripting$ aws ec2 describe-instance-status --output=json"Reservations": [{"Instances": [{"Monitoring": {"State": "disabled"},"PublicDnsName": "","State": {"Code": 16,"Name": "running"},"EbsOptimized": false,"LaunchTime": "2019-11-15T19:18:05.000Z","PublicIpAddress": "18.212.220.250","PrivateIpAddress": "172.30.1.55","ProductCodes": [{"ProductCodeId": "aw0evgkw8e5c1q413zgy5pjce","ProductCodeType": "marketplace"}],"VpcId": "vpc-8e73cfeb","CpuOptions": {"CoreCount": 1,"ThreadsPerCore": 1},"StateTransitionReason": "","InstanceId": "i-04f679de246dc9c10","EnaSupport": true,"ImageId": "ami-02eac2c0129f6376b","PrivateDnsName": "ip-172-30-1-55.ec2.internal","KeyName": "id_rsa","SecurityGroups": [U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

66

Get console output!
$ aws ec2 get-console-output --instance-id i-0042cc0b3e4175345 --output text
i-0042cc0b3e4175345[
0.000000] Linux version 4.9.76-3.78.amzn1.x86_64 (mockbuild@gobibuild-60009) (gcc version 7.2.1 20170915 (Red Hat 7.2.1-2) (GCC) ) #1 SMP Fri Jan 12 19:51:35 UTC 2018[
0.000000] Command line: root=LABEL=/ console=tty1 console=ttyS0 selinux=0
nvme_core.io_timeout=4294967295[
0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'[
0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'[
0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'[
0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256[
0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[
0.000000] x86/fpu: Using 'eager' FPU context switches.
[
0.000000] e820: BIOS-provided physical RAM map:[
0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable[
0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved[
0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved[
0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000003fffffff] usable[
0.000000] BIOS-e820: [mem 0x00000000fc000000-0x00000000ffffffff] reserved[
0.000000] NX (Execute Disable) protection: active[
0.000000] SMBIOS 2.7 present.
[
0.000000] Hypervisor detected: Xen[
0.000000] Xen version 4.2.
[
0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug
emulated NICs.
[
0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug
emulated disks.
[
0.000000] You might have to change the root device[
0.000000] from /dev/hd[a-d] to /dev/xvd[a-d][0.000000] in your root= kernel command line option[
0.000000] e820: last_pfn = 0x40000 max_arch_pfn = 0x400000000[
0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT[
0.000000] found SMP MP-table at [mem 0x000fbc20-0x000fbc2f] mapped at [ffff8800000fbc20][
0.000000] RAMDISK: [mem 0x371e1000-0x37feffff]
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

67

Per-instance metadata:
Letting the instance know what it is
HTTP API:$ curl http://169.254.169.254/latest/meta-data/instance-id
i-5c306beb$$ aws_instance=$(wget -q -O- http://169.254.169.254/latest/meta-data/instance-id)$ aws_region=$(wget -q -O- http://169.254.169.254/latest/meta-data/hostname)$ echo $aws_instance $aws_region
i-5c306beb ip-172-30-1-33.ec2.internal$

ec2-metadata:$ ec2-metadata -i
instance-id: i-5c306beb$ ec2-metadata -i| awk '{print $2;}'
i-5c306beb

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

68

Instance devices show up in the console
2 block devices:

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

69

Here's what it looks like[ec2-user@ip-172-31-30-242 ~]$ df -hFilesystem
Size Used Avail Use% Mounted on
devtmpfs
1.9G
72K 1.9G
1% /dev
tmpfs
1.9G
0 1.9G
0% /dev/shm/dev/xvda1
7.8G 1.1G 6.7G 14% //dev/xvdb
15G
39M
15G
1% /media/ephemeral0[ec2-user@ip-172-31-30-242 ~]$U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

70

Redo the benchmark$ sudo time dd if=/dev/zero of=/bigfile bs=65536 count=16384 conv=fdatasync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 16.1457 s, 66.5 MB/s
0.00user 0.72system 0:16.17elapsed 4%CPU (0avgtext+0avgdata 2184maxresident)k8inputs+2097152outputs (0major+100minor)pagefaults 0swaps$$ sudo time dd if=/dev/zero of=/media/ephemeral0/bigfile bs=65536 count=16384
conv=fdatasync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 11.6586 s, 92.1 MB/s
0.02user 1.06system 0:11.68elapsed 9%CPU (0avgtext+0avgdata 2192maxresident)k64inputs+2097152outputs (0major+100minor)pagefaults 0swaps$

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

71

Working with EBS volumes
EBS volumes are virtual disks. Each one has:
• Volume ID: vol-0490630760213a246• Size• Volume Type• IOPS• Snapshot it was created from• Created Time
• Availability Zone
• State• Alarm Status• Attachment Information — the EC2 instance it's attached to• Volume Status• Encryption Status

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

72

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

73

Actions for EBS Volumes
Modify Volume —• Migrate to a different storage• Make it bigger or smaller.
• Coordinate with OS!

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

74

Actions and terminology
From a live VM, you can:• Snapshot — A copy of just the blocks• Image — A bootable AMI (Amazon Machine Image)Snapshots can be turned into an AMI by:• Registering it with metadata• Specifying the correct kernel

AMI includes:• Disk image• Metadata — architecture, kernel, AMI name, description, device mappings

EBS volume:• Can start as a blank volume or as a copy of an image.
• Can be mounted on a device.
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

75

All of these commands, and more, can be run from the CLI$ aws ec2 help
336 commands!
accept-reserved-instances-exchangequote
accept-vpc-peering-connection
allocate-address
allocate-hosts
assign-ipv6-addresses
assign-private-ip-addresses
associate-address
associate-dhcp-options
associate-iam-instance-profile
associate-route-tableassociate-subnet-cidr-blockassociate-vpc-cidr-blockattach-classic-link-vpc
attach-internet-gateway
attach-network-interface
attach-volume
attach-vpn-gateway
authorize-security-group-egress
authorize-security-group-ingress
bundle-instance
cancel-bundle-task
cancel-conversion-task
cancel-export-task
cancel-import-task
cancel-reserved-instances-listing
cancel-spot-fleet-requests
cancel-spot-instance-requests
confirm-product-instancecopy-image
copy-snapshot
create-customer-gateway
create-default-vpc
create-dhcp-options
create-egress-only-internet-gateway
create-flow-logs
create-fpga-image
create-image
create-instance-export-task
create-internet-gateway
create-key-paircreate-nat-gateway
create-network-acl
create-network-acl-entry
create-network-interface
create-network-interface-permission
create-placement-group
create-reserved-instances-listing
create-routecreate-route-tablecreate-security-group

create-snapshot
create-spot-datafeed-subscription
create-subnet
create-tags
create-volume
create-vpc
create-vpc-endpoint
create-vpc-peering-connection
create-vpn-connection
create-vpn-connection-routecreate-vpn-gateway
delete-customer-gateway
delete-dhcp-options
delete-egress-only-internet-gateway
delete-flow-logs
delete-internet-gateway
delete-key-pairdelete-nat-gateway
delete-network-acl
delete-network-acl-entry
delete-network-interface
delete-network-interface-permission
delete-placement-groupdelete-routedelete-route-tabledelete-security-groupdelete-snapshot
delete-spot-datafeed-subscription
delete-subnet
delete-tags
delete-volume
delete-vpc
delete-vpc-endpoints
delete-vpc-peering-connectiondelete-vpn-connectiondelete-vpn-connection-routedelete-vpn-gateway
deregister-image
describe-account-attributes
describe-addresses
describe-availability-zones
describe-bundle-tasks
describe-classic-link-instances
describe-conversion-tasks
describe-customer-gateways
describe-dhcp-options
describe-egress-only-internetgateways
describe-elastic-gpus
describe-export-tasks
describe-flow-logs

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

describe-fpga-images
describe-host-reservation-offerings
describe-host-reservations
describe-hosts
describe-iam-instance-profileassociations
describe-id-format
describe-identity-id-format
describe-image-attributedescribe-images
describe-import-image-tasks
describe-import-snapshot-tasks
describe-instance-attributedescribe-instance-status
describe-instances
describe-internet-gateways
describe-key-pairs
describe-moving-addresses
describe-nat-gateways
describe-network-acls
describe-network-interface-attributedescribe-network-interfacepermissions
describe-network-interfaces
describe-placement-groups
describe-prefix-lists
describe-regions
describe-reserved-instances
describe-reserved-instances-listings
describe-reserved-instancesmodifications
describe-reserved-instances-offerings
describe-route-tables
describe-scheduled-instanceavailability
describe-scheduled-instances
describe-security-group-references
describe-security-groups
describe-snapshot-attributedescribe-snapshots
describe-spot-datafeed-subscription
describe-spot-fleet-instances
describe-spot-fleet-request-history
describe-spot-fleet-requests
describe-spot-instance-requests
describe-spot-price-history
describe-stale-security-groups
describe-subnets
describe-tags
describe-volume-attributedescribe-volume-status

describe-volumes
describe-volumes-modifications
describe-vpc-attributedescribe-vpc-classic-link
describe-vpc-classic-link-dns-support
describe-vpc-endpoint-services
describe-vpc-endpoints
describe-vpc-peering-connections
describe-vpcs
describe-vpn-connections
describe-vpn-gateways
detach-classic-link-vpc
detach-internet-gateway
detach-network-interface
detach-volumedetach-vpn-gateway
disable-vgw-route-propagation
disable-vpc-classic-link
disable-vpc-classic-link-dns-support
disassociate-address
disassociate-iam-instance-profile
disassociate-route-tabledisassociate-subnet-cidr-blockdisassociate-vpc-cidr-block
enable-vgw-route-propagation
enable-volume-io
enable-vpc-classic-link
enable-vpc-classic-link-dns-support
get-console-output
get-console-screenshot
get-host-reservation-purchasepreview
get-password-data
get-reserved-instances-exchangequote
help
import-image
import-key-pairimport-snapshot
modify-hosts
modify-id-format
modify-identity-id-format
modify-image-attributemodify-instance-attributemodify-instance-placement
modify-network-interface-attributemodify-reserved-instances
modify-snapshot-attributemodify-spot-fleet-request
modify-subnet-attributemodify-volume

modify-volume-attributemodify-vpc-attributemodify-vpc-endpoint
modify-vpc-peering-connectionoptions
monitor-instances
move-address-to-vpc
purchase-host-reservation
purchase-reserved-instances-offeringpurchase-scheduled-instances
reboot-instances
register-image
reject-vpc-peering-connection
release-address
release-hosts
replace-iam-instance-profileassociation
replace-network-acl-association
replace-network-acl-entry
replace-route
replace-route-table-association
report-instance-status
request-spot-fleet
request-spot-instances
reset-image-attributereset-instance-attributereset-network-interface-attributereset-snapshot-attributerestore-address-to-classic
revoke-security-group-egress
revoke-security-group-ingress
run-instances
run-scheduled-instances
start-instances
stop-instances
terminate-instances
unassign-ipv6-addresses
unassign-private-ip-addresses
unmonitor-instances
wait

CLOUD ☁ FORENSICS

76

AWS EBS
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

EBS: Virtual Disk Volumes
EBS volumes:• Created automatically when EC2 instance starts up.
• Snapshots on the fly.

Options:

T0

T1

T2

• Magnetic or SSD• Destroy or persist on instance termination
T1’• Not Encrypted / Encrypted• Provisioned IOPS

T2’

T3

Uses:
• Boot drives• Read-only drives to share static databases. (Make 1 TB drive and mount)• Database drives for MySQL, etc.. (But you should use Amazon’s managed
service.)U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

78

EBS offers three classes of service.

Characteristic
Use cases

General Purpose (SSD)•

System bootProvisioned IOPS (SSD)•

Critical business

Magnetic•

Cold workloads

volumesapplications that

where data is•

Virtual desktops

require sustained

infrequently•

Small to medium

IOPS performance,

accessed

sized databases

or more thanDevelopment and10,000 IOPS or 160

test environments

MiB/s ofthe lowest storage

throughput per

cost is important••

Scenarios wherevolume•Large database
workloads, such

Volume size

1 GiB - 16 TiB

4 GiB as:- 16 TiB

1 GiB - 1 TiBMaximum throughput

160 MiB/s

320 MiB/s

40-90 MiB/s

IOPS performance

Baseline performance of
3 IOPS/GiB (up to
10,000 IOPS) with theability to burst to 3,000
IOPS
gp2 for volumes underConsistently performs at Averages 100 IOPS, with
provisioned level, up to the ability to burst to
20,000 IOPS maximum
hundreds of IOPS

API and CLI volume
name

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

io1

standard

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html

CLOUD ☁ FORENSICS

79

Pricing — you are probably best off with SSD General Purpose.

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

80

EBS volumes can be created and used for:extra storage, sharing data
Each EBS volume has:
• Size• Name• Region / AvailabilityZone• Attributese.g. 40GB
e.g. vol-65202e2d
e.g. us-east-1 / us-east-1b
e.g. CreateTime, Encrypted, Iops,

Volumes can be mounted:• read/write on a single instance• read-only on multiple instances

Create and share an instance:$ aws ec2 create-volume --size 10 --availability-zone us-east-1a
You must specify a region. You can also configure your region by running "aws configure".
$ aws ec2 create-volume --size 10 --region us-east-1 --availability-zone us-east-1b{"AvailabilityZone": "us-east-1b","Encrypted": false,"VolumeType": "standard","VolumeId": "vol-95cab176","State": "creating","SnapshotId": "","CreateTime": "2015-12-05T18:55:28.052Z","Size": 10}$
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

81

Attach the EBS volume to your VM(Be sure EBS is in same region & availability zone)
First get a volume...
$ aws_zone=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)$ aws_instance=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)$ aws_region=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document|grep
region|awk -F\" '{print $4}')$ aws ec2 create-volume --size 10 --region $aws_region --availability-zone $aws_zone{"AvailabilityZone": "us-east-1b","Encrypted": false,"VolumeType": "standard","VolumeId": "vol-46cdb6a5","State": "creating","SnapshotId": "","CreateTime": "2015-12-05T19:01:38.548Z","Size": 10}$ aws ec2 attach-volume --volume-id=vol-46cdb6a5 --instance-id=$aws_instance \--device=/dev/sdb --region=$aws_region{"AttachTime": "2015-12-05T19:02:11.541Z","InstanceId": "i-5c306beb","VolumeId": "vol-46cdb6a5","State": "attaching","Device": "/dev/sdb"}$

Now we need to make a file system...
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

82

Create a file system on the volume$ sudo mkfs -t ext4 /dev/sdb
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 2621440 4k blocks and 655360 inodes
Filesystem UUID: 681c57f0-1461-4dae-b956-032656ba82a9
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
$ sudo mount /dev/sdb /mnt/extra/[ip-172-30-1-33 ~ 19:04:44]$ df
Filesystem
1K-blocks
Used Available Use% Mounted on/dev/xvda1
41151788 6506728 34544812 16% /devtmpfs
50071260
500652
1% /dev
tmpfs
50972405097240% /dev/shm/dev/xvdb
10190136
23028
9626436
1% /mnt/extra$ lsblk
NAME
MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda
202:0
0 40G 0 disk
└─xvda1 202:1
0 40G 0 part /xvdb
202:16
0 10G 0 disk /mnt/extra$
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

83

AWS CloudTrail
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

85

CloudTrail log validation

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

86

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

87

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

88

Cloud Trails Works!
CloudTrail — the raw files

Note spelling error!

CloudTrail-Digest — metadata about the raw files[hadoop@ip-172-31-49-78 ~]$ aws s3 ls crfs780-trails/AWSLogs/309467262965/
PRE CloudTrail-Digest/
PRE CloudTrail/
2018-02-19 00:56:16
0[hadoop@ip-172-31-49-78 ~]$ aws s3 ls crfs780-trails/AWSLogs/309467262965/CloudTrail/
PRE ap-northeast-1/
PRE ap-northeast-2/
PRE ap-northeast-3/
PRE ap-south-1/
PRE ap-southeast-1/
PRE ap-southeast-2/
PRE ca-central-1/
PRE eu-central-1/
PRE eu-west-1/
PRE eu-west-2/
PRE eu-west-3/
PRE sa-east-1/
PRE us-east-1/
PRE us-east-2/
PRE us-west-1/
PRE us-west-2/

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

89

Cloud Trails Works!
CloudTrail — the raw files
CloudTrail-Digest — metadata about the raw files
aws s3 ls crfs780-trails/AWSLogs/309467262965/CloudTrail-Digest/us-east-1/2018/02/19/[hadoop@ip-172-31-49-78 ~]$ aws s3 ls crfs780-trails/AWSLogs/309467262965/CloudTrail-Digest/us-east-1/2018/02/19/
2018-02-19 01:47:32
342 309467262965_CloudTrail-Digest_us-east-1_CRFS780_Trails_us-east-1_20180219T005615Z.json.gz
2018-02-19 02:47:38
4037 309467262965_CloudTrail-Digest_us-east-1_CRFS780_Trails_us-east-1_20180219T015615Z.json.gz
2018-02-19 03:47:23
4192 309467262965_CloudTrail-Digest_us-east-1_CRFS780_Trails_us-east-1_20180219T025615Z.json.gz
2018-02-19 04:47:26
4124 309467262965_CloudTrail-Digest_us-east-1_CRFS780_Trails_us-east-1_20180219T035615Z.json.gz
2018-02-19 05:47:34
4114 309467262965_CloudTrail-Digest_us-east-1_CRFS780_Trails_us-east-1_20180219T045615Z.json.gz
2018-02-19 06:47:30
4128 309467262965_CloudTrail-Digest_us-east-1_CRFS780_Trails_us-east-1_20180219T055615Z.json.gz
2018-02-19 07:47:27
4186 309467262965_CloudTrail-Digest_us-east-1_CRFS780_Trails_us-east-1_20180219T065615Z.json.gz
2018-02-19 08:47:31
4122 309467262965_CloudTrail-Digest_us-east-1_CRFS780_Trails_us-east-1_20180219T075615Z.json.gz
2018-02-19 09:47:36
4202 309467262965_CloudTrail-Digest_us-east-1_CRFS780_Trails_us-east-1_20180219T085615Z.json.gz...

[hadoop@ip-172-31-49-78 ~]$ aws s3 cp s3://crfs780-trails/AWSLogs/309467262965/CloudTrailDigest/us-east-1/2018/02/19/309467262965_CloudTrail-Digest_us-east-1_CRFS780_Trails_useast-1_20180219T015615Z.json.gz trails.json.gz
download: s3://crfs780-trails/AWSLogs/309467262965/CloudTrail-Digest/useast-1/2018/02/19/309467262965_CloudTrail-Digest_us-east-1_CRFS780_Trails_useast-1_20180219T015615Z.json.gz to ./trails.json.gz

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

90

[hadoop@ip-172-31-49-78 ~]$ ls -l trails.json.gz-rw-rw-r-- 1 hadoop hadoop 4037 Feb 19 02:47 trails.json.gz[hadoop@ip-172-31-49-78 ~]$ gunzip trails.json.gz[hadoop@ip-172-31-49-78 ~]$ ls -l trails*-rw-rw-r-- 1 hadoop hadoop 17833 Feb 19 02:47 trails.json[hadoop@ip-172-31-49-78 ~]$ more trails.json...
{"awsAccountId":"309467262965","digestStartTime":"2018-02-19T00:56:15Z","digestEndTime":"2018-02-19T01:56:15Z","digestS3Bucket":"crfs780-trails","digestS3Object":"AWSLo
gs/309467262965/CloudTrail-Digest/us-east-1/2018/02/19/309467262965_CloudTrail-Digest_useast-1_CRFS780_Trails_us-east-1_20180219T015615Z.json.gz","digestPublicKeyFinge
rprint":"3fc64187ed954b65bd053279acc75d70","digestSignatureAlgorithm":"SHA256withRSA","newestEventTime":"2018-02-19T01:52:22Z","oldestEventTime":"2018-02-19T00:44:47Z","previousDigestS3Bucket":"crfs780-trails","previousDigestS3Object":"AWSLogs/309467262965/
CloudTrail-Digest/us-east-1/2018/02/19/309467262965_CloudTrail-Digest_us-east-1_CRFS780_Trails_useast-1_20180219T005615Z.json.gz","previousDigestHashValue":"cc6026c8656ab9d6f295bfaed9d27
60b969487a8b8148704f27455e344ed27f4","previousDigestHashAlgo
rithm":"SHA-256","previousDigestSignature":"76f4d42d0b12bd9f4fad874c95adaaf6ba05f66c6adc8
f9ed7c1d22fa827b8c8de516b112bca0bad76d9833bcd3e040be64c321a25bb765505ddecafb0b1
32db53edc4e122eed34c4932fb4441b86589dec382d2356157d457e9d3a5114d53761f629bb9e4487e764d312
1dfcdb1bb79d274b497161b9a44143560704249d75dd57150b42e4d757602c92637aa704f822c89
9acf4f47f9e1166ae0692590087eb2ea3aaec97e6e937efde434e36739f37a2eac372e8b83d37f354cb7953d5
4d85780ea5aabd430c6d4f132edf83e9525644fe595f77ca6987f491da7524a8f5a66779bab7e13
c9cf5494e2d064105c592cf2aa332dcabc734ba5b8a8c49a0fd4","logFiles":[{"s3Bucket":"crfs780trails","s3Object":"AWSLogs/309467262965/CloudTrail/us-east-1/2018/02/19/30946726
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

91

If you don’t have a JSON viewer, you an use Python…
[hadoop@ip-172-31-49-78 ~]$ python3.6
Python 3.6.2 (default, Nov 2 2017, 19:34:31)[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import json>>> data = json.loads(open("trails.json","r").read())>>> data.keys()
dict_keys(['awsAccountId', 'digestStartTime', 'digestEndTime', 'digestS3Bucket','digestS3Object', 'digestPublicKeyFingerprint', 'digestSignatureAlgorithm','newestEventTime', 'oldestEventTime', 'previousDigestS3Bucket','previousDigestS3Object', 'previousDigestHashValue','previousDigestHashAlgorithm', 'previousDigestSignature', 'logFiles'])>>> len(data["logFiles"])46>>> data["logFiles"][1]{'s3Bucket': 'crfs780-trails', 's3Object': 'AWSLogs/309467262965/CloudTrail/useast-1/2018/02/19/309467262965_CloudTrail_useast-1_20180219T0135Z_8bl6vp5CfHoaB2Lh.json.gz', 'hashValue':'f4151aa6d39999c32e6a0be3adc9807839861291f75ac30d4a376698caef5230','hashAlgorithm': 'SHA-256', 'newestEventTime': '2018-02-19T01:30:35Z','oldestEventTime': '2018-02-19T01:28:39Z'}>>>—
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS92

We can get that file…
$ aws s3 cp s3://crfs780-trails/AWSLogs/309467262965/CloudTrail/useast-1/2018/02/19/309467262965_CloudTrail_useast-1_20180219T0135Z_8bl6vp5CfHoaB2Lh.json.gz data2.json.gz
download: s3://crfs780-trails/AWSLogs/309467262965/CloudTrail/useast-1/2018/02/19/309467262965_CloudTrail_useast-1_20180219T0135Z_8bl6vp5CfHoaB2Lh.json.gz to ./records.json.gz$ gunzip records.json.gz
hadoop@ip-172-31-49-78 ~]$ python3
Python 3.6.2 (default, Nov 2 2017, 19:34:31)[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import json>>> records = json.loads(open(“records.json").read())>>> len(records['Records'])
11>>>

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

93

Here is one record (keys in bold)>>> records['Records'][0]{'eventVersion': '1.05', 'userIdentity': {'type': 'AWSService', 'invokedBy':'elasticmapreduce.amazonaws.com'}, 'eventTime': '2018-02-19T01:29:36Z', 'eventSource':'sts.amazonaws.com', 'eventName': 'AssumeRole', 'awsRegion': 'us-east-1','sourceIPAddress': 'elasticmapreduce.amazonaws.com', 'userAgent':'elasticmapreduce.amazonaws.com', 'requestParameters': {'roleArn':'arn:aws:iam::309467262965:role/EMR_DefaultRole', 'roleSessionName': 'CCSSession','durationSeconds': 1500}, 'responseElements': {'credentials': {'accessKeyId':'ASIAJPGNR7QOAKKOM5HQ', 'expiration': 'Feb 19, 2018 1:54:36 AM', 'sessionToken':'AgoGb3JpZ2luEA0aCXVzLWVhc3QtMSKAAjI5KkvkNsX7M9SG1nHC/
yT6DFQDLZZ8Ek9dty2WpgQbpgPmn6TE+VLEUz1ftZPJxrHL0OH5RLm27G9xIxQXpRrrwInleeZ50yw62cp9BO8ex
TwNViYCIsVWN3AnqQq1mHwKU+WDd/mVX+c4Uu7EKYY1ijLB6pFUEcW6rHxWCWqQLyJ/
OzIgbZqHpHd7nUyaCMOz1inte450FKew/eSfvI2LIyE0OM/OSh/X3p2/dwa5rIDwuSdxnN/
aTyTHiLTrMF54J+Z7R4zA+qV5KSeHHjS/So4MJoELkWIO1MjpxqMd4GADIkUV/
whzREuTjqvxsQ1l8tVvm90coRc1LL1cd7wqhgMI4///////////
ARAAGgwzMDk0NjcyNjI5NjUiDHfH08I1YiqjkzXwOyraAu/Xisb8QiIZg13/JG81uLwS1tEw/
QFuqtPNQOpJ7s0z6m3DMubsEOOc6E6HEjThNf9VMRjrNsqEH2JuMPG9TzDgjFivoc9X2QEH3SWrK68E7waPTH+35
6nqIueXWiLiatJiWCEUS526URB78kQyvC2KdYaTDnmtqml4/3fM98a+eV3iRBw9Grz1+RGvZ2OZ9Sm+RgFOtMtLc
MYEIHo5qn6odJpid4wIF+qdtkl/Qp/
Uny1qf1FJJ0E80oemhcwJFr23R0IBDNW9c71NK+MEYwZE5XaXSF3J7lg9IH74X1wO5dScPEipiKUJKTTXoxbUk5v
USbPWM9FWeKYtbGHRovOkjaDRqgD5y5ehiF2qF8mDBXLZ2FlW0HaktfC1wnZ6GKznVGYCTzJsa2R3/EG/
dqx6YaZXRo0utlijlMPtAg+azHhhVJBjGiMtL9PKFXwCN3jh7ENwYlyJZvIwgNGo1AU='},'assumedRoleUser': {'assumedRoleId': 'AROAILDJTGTBO4WJMRFDS:CCSSession', 'arn':'arn:aws:sts::309467262965:assumed-role/EMR_DefaultRole/CCSSession'}}, 'requestID':'58b0a175-1514-11e8-9166-8b01e23a16a9', 'eventID': '6bc67694-fef9-4f49-85e3efdbbcb0d855', 'resources': [{'ARN': 'arn:aws:iam::309467262965:role/EMR_DefaultRole','accountId': '309467262965', 'type': 'AWS::IAM::Role'}], 'eventType': 'AwsApiCall','recipientAccountId': '309467262965', 'sharedEventID': '6921e72e-aadc-4336a822-928ea82f2c3a'}U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

94

Working with CloudTrails—If you can avoid it, don’t write your own analysis tools
Amazon partners have tools for processing CloudTrails:• https://aws.amazon.com/cloudtrail/partners/

Amazon Athena is an analysis platform provided by Amazon:• https://aws.amazon.com/blogs/big-data/aws-cloudtrail-and-amazon-athenadive-deep-to-analyze-security-compliance-and-operational-activity/

Amazon Glue and QuickSight also provide tools:• https://aws.amazon.com/blogs/big-data/streamline-aws-cloudtrail-logvisualization-using-aws-glue-and-amazon-quicksight/

Multiple Amazon Blog entries:• https://aws.amazon.com/blogs/mt/category/management-tools/awscloudtrail/
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

95

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

96

Amazon Athena looks like SQL…

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

97

Once you have a the tables created, you can SQL away!
SELECTuseridentity.arn,
eventname,sourceipaddress,
eventtime
FROM cloudtrail_logs
LIMIT 100;

Pricing:• $5 per TB of data scanned• Can scan CloudTrails, text files, Parquet files. and more.

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

98

Other options: AWS Glue & Quicksight(and splunk, too!)

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

99

QuickSight can log to CLoudTrail…

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

100

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

101

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

102

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS103

AWS EFS
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

Options for storing forensic data results:
EBS — Elastic Block Store• Very fast
• Read/write on a single VM at a time
• Snapshot capability• Restricted to an availability zone within a region

S3 — Simple Storage Service• Stores objects, logfiles, etc.
• Integrates with Amazon Athena• Doesn’t have file permissions

EFS — Elastic File System• A consistent read-write file system• Can be mounted on many servers at once
• Works across availability zones• NFS4 (Network File System) interface
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

105

Read the documentation:• https://docs.aws.amazon.com/efs/latest/ug/whatisefs.html
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

106

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

107

You must use security groups to restrict access to your accesspoints.

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS108

I created a SLG Analysis security group
I manually added to it the IP address of each of my VMs.

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

109

Now, create the EFS with my added security group

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS110

My file system info:

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

111

Mount the file system$ sudo yum install -y nfs-utils$ df
Filesystem
1K-blocks
Used Available Use% Mounted on
devtmpfs
72071020
84 72070936
1% /dev
tmpfs
72080004
0 72080004
0% /dev/shm/dev/nvme0n1p1
8123812 1075712
6947852 14% //dev/nvme1n1
82438832 3356568 74871576
5% /mnt$ sudo mkdir /efs
$ sudo mount -t nfs -o
nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2
fs-1a3ac552.efs.us-east-1.amazonaws.com:/ /efs
$ df -hFilesystem
Size Used Avail Use% Mounted on
devtmpfs
69G
88K
69G
1% /dev
tmpfs
69G
0
69G
0% /dev/shm/dev/nvme0n1p1
7.8G 1.1G 6.7G 14% //dev/nvme1n1
79G 3.3G
72G
5% /mnt
fs-1a3ac552.efs.us-east-1.amazonaws.com:/ 8.0E
0 8.0E
0% /efs$

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

112

You can set the file system to mount on reboot.
Be careful — if you damage /etc/fstab, the system won’t boot!
Add the boot instructions to the /etc/fstab$ sudo bash
# cat >> /etc/fstab
fs-1a3ac552.efs.us-east-1.amazonaws.com:/ /efs
nfs4
nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,_netdev 0 0^dand test:# umount /efs
# mount /efs
# df -hFilesystem
devtmpfs
tmpfs/dev/nvme0n1p1/dev/nvme1n1
fs-1a3ac552.efs.us-east-1.amazonaws.com:/#

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

Size
69G
69G
7.8G
79G
8.0E

Used Avail Use% Mounted on
88K
69G
1% /dev
069G
0% /dev/shm
1.1G 6.7G 14% /7.3G
68G 10% /mnt
0 8.0E
0% /efs

CLOUD ☁ FORENSICS

113

Running bulk_extractor in AWS
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

There is no reason to run bulk_extractor on a small machine.

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

115

The spot advisor is a useful tool for finding spot instances.

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

116

Be careful! If you have a standing spot request, it will relaunch
instances when you terminate them.

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

117

Hibernate interrupt behavior

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

118

Give the EC2 instance a sufficient IAM role.

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

119

Here is standing request creates the analysis VM.

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

120

The request is submitted, and a VM has been created.

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

121

lscpu shows that this instance has 72 CPUs!
[ec2-user@ip-172-31-26-133 ~]$ lscpu
Architecture:x86_64
CPU op-mode(s):32-bit, 64-bit
Byte Order:
Little EndianCPU(s):
72
On-line CPU(s) list:
0-71
Thread(s) per core:2
Core(s) per socket:
18
Socket(s):2
NUMA node(s):2Vendor ID:
GenuineIntelCPU family:
6
Model:
85
Model name:Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
Stepping:
3
CPU MHz:
3000.000
BogoMIPS:
6000.00
Hypervisor vendor:
KVM
Virtualization type:full
L1d cache:
32K
L1i cache:
32K
L2 cache:
1024K
L3 cache:25344KNUMA node0 CPU(s):0-17,36-53
NUMA node1 CPU(s):18-35,54-71[ec2-user@ip-172-31-26-133 ~]$U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

122

The IAM Role works:
No configuration required for ‘aws’ command!
[ec2-user@ip-172-31-26-133 ~]$ aws s3 ls
2018-02-24 18:28:03 aws-athena-query-results-309467262965-us-east-1
2018-02-13 02:31:54 aws-logs-309467262965-us-east-1
2018-02-19 00:39:21 cfrs780
2018-02-19 00:55:04 crfs780-trails[ec2-user@ip-172-31-26-133 ~]$ ls -al .aws
ls: cannot access .aws: No such file or directory[ec2-user@ip-172-31-26-133 ~]$ aws s3 cp s3://cfrs780/bin/bulk_extractor
bulk_extractor
download: s3://cfrs780/bin/bulk_extractor to ./bulk_extractor

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

123

Change the hostname to make identifying the hosts easier[ec2-user@ip-172-31-26-133 ~]$ sudo hostname beefy[ec2-user@ip-172-31-26-133 ~]$ logout
Connection to 107.23.131.99 closed.
[Dance ~ 10:19:54]$ ssh -A ec2-user@107.23.131.99
Last login: Sat Mar 3 15:16:22 2018 from 68.33.78.230__| __|_ )_| (/___|\___|___|

Amazon Linux AMI

https://aws.amazon.com/amazon-linux-ami/2017.09-release-notes/
6 package(s) needed for security, out of 12 available
Run "sudo yum update" to apply all updates.
[ec2-user@beefy ~]$U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

124

Performing an experiment with bulk_extractor and EC2
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

Experimental design: are we getting clean EBS boot volumes?
Experimental outline:• Create VMs with 40GB, 100GB and 1000GB boot volumes• Specify that volumes are to be kept after machine termination.
• Terminate each machine after boot• Attach the volumes to a “beefy” VM• Run bulk_extractor on each volume, storing the results in a single file system.
• Compare the results.

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

126

Wrong availability zone…
$ zone=`curl -s http://169.254.169.254/latest/meta-data/placement/availabilityzone`$ region=`echo $zone | sed s/.$//`$ id=`curl -s http://169.254.169.254/latest/meta-data/instance-id`$ aws ec2 attach-volume --device /dev/sdf --instance-id $id --region $region -volume-id vol-08baaf673f8051ab6
An error occurred (InvalidVolume.ZoneMismatch) when calling the AttachVolume
operation: The volume 'vol-08baaf673f8051ab6' is not in the same availability
zone as instance 'i-0d1ab43350cd3e1ee'[ec2-user@beefy ~]$U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

127

To move a volume to a different availability zone:
Create a snapshot, and create a volume from the snapshot.
[ec2-user@beefy ~]$ aws ec2 create-snapshot --description stats_snap --volume-id
vol-08baaf673f8051ab6 --region $region{"Description": "stats_snap","Tags": [],"Encrypted": false,"VolumeId": "vol-08baaf673f8051ab6","State": "pending","VolumeSize": 80,"StartTime": "2018-03-03T15:24:36.000Z","Progress": "","OwnerId": "309467262965","SnapshotId": "snap-0fa3c10efa5cda4ef"}[ec2-user@beefy ~]$ aws ec2 create-volume --availability-zone $zone --region$region --snapshot-id snap-0fa3c10efa5cda4ef{"AvailabilityZone": "us-east-1a","Encrypted": false,"VolumeType": "standard","VolumeId": "vol-07ac3927385281d53","State": "creating","SnapshotId": "snap-0fa3c10efa5cda4ef","CreateTime": "2018-03-03T15:27:30.962Z","Size": 80}[ec2-user@beefy ~]$U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

128

Now we can attach the volume[ec2-user@beefy ~]$ aws ec2 attach-volume --device /dev/sdf --instance-id $id -region $region --volume-id vol-07ac3927385281d53{"AttachTime": "2018-03-03T15:28:08.830Z","InstanceId": "i-0d1ab43350cd3e1ee","VolumeId": "vol-07ac3927385281d53","State": "attaching","Device": "/dev/sdf"}[ec2-user@beefy ~]$ ls -l /mnt/
total 28
-rw-r--r-- 1 root root
0 Mar 3 14:57 0_VOLUME_FOR_STATS
drwx------ 2 root root 16384 Mar 3 02:22 lost+founddrwxr-xr-x 5 root root 4096 Mar 3 02:35 vm_exp1
drwxr-xr-x 5 root root 4096 Mar 3 03:08 vm_exp2
drwxr-xr-x 5 root root 4096 Mar 3 03:21 vm_exp3[ec2-user@beefy ~]$U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

129

Now attach the volumes for the 40GB, 100GB and 1000GB
All needed to be moved to the availability zone of the beefy:$ aws ec2 create-volume --availability-zone $zone --region $region --snapshot-id
snap-0e0bfd05ae61cc002$ aws ec2 create-volume --availability-zone $zone --region $region --snapshot-id
snap-07f5081edb3e0f6c5$ aws ec2 create-volume --availability-zone $zone --region $region --snapshot-id
snap-0e70b2b58d662e10b$ aws ec2 attach-volume --device /dev/sdi --instance-id $id --region $region --volume-id
vol-0a9d462b1f4229dcd$ aws ec2 attach-volume --device /dev/sdj --instance-id $id --region $region --volume-id
vol-0c7e4831ea1e04585$ aws ec2 attach-volume --device /dev/sdk --instance-id $id --region $region --volume-id
vol-04297e43473766836

Don’t mount them!
[ec2-user@beefy ~]$ df -hFilesystem
Size Used Avail Use% Mounted on
devtmpfs
69G
84K
69G
1% /dev
tmpfs
69G
0
69G
0% /dev/shm/dev/nvme0n1p1 7.8G 1.1G 6.7G 14% //dev/nvme1n1
79G 1.7G
73G
3% /mnt[ec2-user@beefy ~]$ ls -l /mnt/
total 28
-rw-r--r-- 1 root root
0 Mar 3 14:57 0_VOLUME_FOR_STATS
drwx------ 2 root root 16384 Mar 3 02:22 lost+found[ec2-user@beefy ~]$U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

130

Get the docs for create-volume
https://docs.aws.amazon.com/cli/latest/search.html

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

131

Now we can run bulk_extractor!
$ aws s3 cp s3://cfrs780/bin/bulk_extractor bulk_extractor[root@beefy ec2-user]# ls -ltotal 21924-rw-rw-r-- 1 ec2-user ec2-user 22450072 Mar 3 02:04 bulk_extractor[root@beefy ec2-user]# chmod +x bulk_extractor[root@beefy ec2-user]#[root@beefy ec2-user]# ./bulk_extractor -o /mnt/vm_40G -e wordlist /dev/sdi
bulk_extractor version: 1.6.0-dev
Hostname: beefy
Input file: /dev/sdi
Output directory: /mnt/vm_40G
Disk Size: 42949672960
Threads: 72
Attempt to open /dev/sdi
15:44:35 Offset 67MB (0.16%) Done in 0:47:28 at 16:32:03
15:44:39 Offset 150MB (0.35%) Done in 0:39:35 at 16:24:14

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

132

Once we have results in EBS, we could copy them to S3 for safe
keeping…
[ec2-user@beefy mnt]$ du -sh .
du: cannot read directory ‘./lost+found’: Permission denied
3.2G .
[ec2-user@beefy mnt]$ time aws s3 cp --recursive . s3://cfrs780/
bulk_extractor_results/
warning: Skipping file /mnt/lost+found. File/Directory is not readable.
upload: ./0_VOLUME_FOR_STATS to s3://cfrs780/bulk_extractor_results/
0_VOLUME_FOR_STATS
upload: vm_1000G/ether.txt to s3://cfrs780/bulk_extractor_results/vm_1000G/
ether.txt
upload: vm_1000G/exif.txt to s3://cfrs780/bulk_extractor_results/vm_1000G/
exif.txt
upload: vm_1000G/find.txt to s3://cfrs780/bulk_extractor_results/vm_1000G/
find.txt…
upload: vm_exp3/wordlist_split_000.txt to s3://cfrs780/bulk_extractor_results/
vm_exp3/wordlist_split_000.txt
upload: vm_exp3/wordlist.txt to s3://cfrs780/bulk_extractor_results/vm_exp3/
wordlist.txt
real 0m31.065s
user 0m13.972s
sys 0m7.476s[ec2-user@beefy mnt]$
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS133

We could also save the result in EFS[ec2-user@beefy mnt]$ sudo mkdir /efs/results[ec2-user@beefy mnt]$ sudo chown $USER /efs/results[ec2-user@beefy mnt]$ time cp -r * /efs/results/
cp: cannot access ‘lost+found’: Permission denied

S3 copy times:

real 0m50.788s
real 0m31.065s
user 0m0.016s
user 0m13.972s
sys 0m3.176s
sys 0m7.476s[ec2-user@beefy mnt]$ ls -l /efs/results/
total 32
-rw-r--r-- 1 ec2-user ec2-user
0 Mar 3 21:33 0_VOLUME_FOR_STATS
drwx------ 2 ec2-user ec2-user 6144 Mar 3 21:33 lost+founddrwxr-xr-x 5 ec2-user ec2-user 6144 Mar 3 21:33 vm_1000G
drwxr-xr-x 5 ec2-user ec2-user 6144 Mar 3 21:33 vm_100G
drwxr-xr-x 5 ec2-user ec2-user 6144 Mar 3 21:34 vm_40G
drwxr-xr-x 5 ec2-user ec2-user 6144 Mar 3 21:34 vm_exp1
drwxr-xr-x 5 ec2-user ec2-user 6144 Mar 3 21:34 vm_exp2
drwxr-xr-x 5 ec2-user ec2-user 6144 Mar 3 21:34 vm_exp3[ec2-user@beefy mnt]$

Compared to copying to S3• Slower to send data
• Easier to use data once sent
U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

134

Speed:Let’s compare running bulk_extractor out of EBS and EFS.
$ sudo mkdir /mnt/work /efs/work$ sudo chown $USER /mnt/work /efs/work$ time wget -O /mnt/work/nps-2009-domexusers.E01 \http://downloads.digitalcorpora.org/corpora/drives/nps-2009-domexusers/
nps-2009-domexusers.E01

I started the download…
• To /mnt/work and got 27MB/sec• After 1GB, I started a simultaneous download to /efs/work in another window• Download rate remained 27MB in both windows…

I ran bulk_extractor sequentially, same conditions:• reboot• mount file system• run bulk_extractor from /efs/work or /mnt/work

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

135

from /mnt (EBS)$ cd /mnt/work$ ls -altotal 4268484
drwxr-xr-x 2 ec2-user root
4096 Mar 3 21:42 .
drwxr-xr-x 10 root
root
4096 Mar 3 21:40 ..
-rw-rw-r-- 1 ec2-user ec2-user 4370913825 May 14 2012 nps-2009-domexusers.E01$ ~/bulk_extractor -o out -e wordlist nps-2009-domexusers.E01
bulk_extractor version: 1.6.0-dev
Hostname: ip-172-31-18-68
Input file: nps-2009-domexusers.E01
Output directory: out
Disk Size: 42949672960
Threads: 72
2:16:06 Offset 67MB (0.16%) Done in 0:38:19 at 02:54:25
2:16:09 Offset 150MB (0.35%) Done in 0:30:14 at 02:46:23
2:16:11 Offset 234MB (0.55%) Done in 0:26:31 at 02:42:42
2:16:13 Offset 318MB (0.74%) Done in 0:23:54 at 02:40:07
2:16:15 Offset 402MB (0.94%) Done in 0:22:47 at 02:39:02
2:16:17 Offset 486MB (1.13%) Done in 0:21:50 at 02:38:07
2:16:20 Offset 570MB (1.33%) Done in 0:21:22 at 02:37:42
2:16:22 Offset 654MB (1.52%) Done in 0:20:44 at 02:37:06…

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

136

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

137

2:21:36 Offset 42849MB (99.77%) Done in 0:00:00 at 02:21:36
2:21:36 Offset 42932MB (99.96%) Done in 0:00:00 at 02:21:36
All data are read; waiting for threads to finish...
Time elapsed waiting for 3 threads to finish:(timeout in 60 min.)
All Threads Finished!
Producer time spent waiting: 0 sec.
Average consumer time spent waiting: 302.728 sec.
********************************************* bulk_extractor is probably I/O bound. ****Run with a faster drive****
to get better performance.
*********************************************
MD5 of Disk Image: 8e7176524a64376631cd7dc9d90339f1
Phase 2. Shutting down scanners
Phase 3. Uniquifying and recombining wordlist
Phase 3. Creating Histograms
Elapsed time: 391.168 sec.
Total MB processed: 42949
Overall performance: 109.799 MBytes/sec (1.52498 MBytes/sec/thread)
Total email features found: 8757$

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

138

Run bulk_extractor from/to EFS
Reboot instance, then:$ sudo mkdir /efs
$ sudo mount -t nfs -o
nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2
fs-1a3ac552.efs.us-east-1.amazonaws.com:/ /efs
[ec2-user@ip-172-31-18-68 work]$ ~/bulk_extractor/src/bulk_extractor -o out -ewordlist nps-2009-domexusers.E01
bulk_extractor version: 1.6.0-dev
Hostname: ip-172-31-18-68
Input file: nps-2009-domexusers.E01
Output directory: out
Disk Size: 42949672960
Threads: 72
2:30:42 Offset 67MB (0.16%) Done in 0:17:58 at 02:48:40
2:30:42 Offset 150MB (0.35%) Done in 0:10:05 at 02:40:47
2:30:43 Offset 234MB (0.55%) Done in 0:07:59 at 02:38:42

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

139

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

140

2:33:52 Offset 42765MB (99.57%) Done in 0:00:00 at 02:33:52
2:33:52 Offset 42849MB (99.77%) Done in 0:00:00 at 02:33:52
2:33:52 Offset 42932MB (99.96%) Done in 0:00:00 at 02:33:52
All data are read; waiting for threads to finish...
Time elapsed waiting for 5 threads to finish:(timeout in 60 min.)
All Threads Finished!
Producer time spent waiting: 0 sec.
Average consumer time spent waiting: 142.452 sec.
********************************************* bulk_extractor is probably I/O bound. ****Run with a faster drive****
to get better performance.
*********************************************
MD5 of Disk Image: 8e7176524a64376631cd7dc9d90339f1
Phase 2. Shutting down scanners
Phase 3. Uniquifying and recombining wordlist
Phase 3. Creating Histograms
Elapsed time: 263.151 sec.
Total MB processed: 42949
Overall performance: 163.213 MBytes/sec (2.26685 MBytes/sec/thread)
Total email features found: 8757
[ec2-user@ip-172-31-18-68 work]$

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

141

Another example:
m5.4xlarge[ec2-user@beefy2 work]$ ./bulk_extractor -e wordlist -S write_feature_sqlite3=NO-o jo_nosql jo-2009-12-11-001.E01
bulk_extractor version: 1.6.0-dev
Hostname: beefy2
Input file: jo-2009-12-11-001.E01
Output directory: jo_nosql
Disk Size: 15382241280
Threads: 16
0:35:39 Offset 67MB (0.44%) Done in 0:05:42 at 00:41:21
0:35:40 Offset 150MB (0.98%) Done in 0:03:45 at 00:39:25
0:35:41 Offset 234MB (1.53%) Done in 0:03:16 at 00:38:57
0:35:44 Offset 318MB (2.07%) Done in 0:04:49 at 00:40:33
0:35:44 Offset 402MB (2.62%) Done in 0:04:09 at 00:39:53…

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

142

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

143

Time elapsed waiting for 1 thread to finish:
18 sec (timeout in 59 min42 sec.)
Thread 14: Processing 11475615744
All Threads Finished!
Producer time spent waiting: 198.711 sec.
Average consumer time spent waiting: 31.1094 sec.
********************************************* bulk_extractor is probably CPU bound. ****Run on a computer with more cores ****to get better performance.
*********************************************
MD5 of Disk Image: a6c44b7387a67333b8566955dcad6f50
Phase 2. Shutting down scanners
Phase 3. Uniquifying and recombining wordlist
Phase 3. Creating Histograms
Elapsed time: 416.436 sec.
Total MB processed: 15382
Overall performance: 36.9378 MBytes/sec (2.30861 MBytes/sec/thread)
Total email features found: 8480

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

144

Try same run with SQL output as well[ec2-user@beefy2 work]$ bulk_extractor -e wordlist -S write_feature_sqlite3=YES-o jo_sql jo-2009-12-11-001.E01
bulk_extractor version: 1.6.0-dev
Hostname: beefy2
Input file: jo-2009-12-11-001.E01
Output directory: jo_sql
Disk Size: 15382241280
Threads: 16
0:50:32 Offset 67MB (0.44%) Done in 0:52:05 at 01:42:37
0:50:33 Offset 150MB (0.98%) Done in 0:24:17 at 01:14:50
0:50:33 Offset 234MB (1.53%) Done in 0:16:23 at 01:06:56
0:50:36 Offset 318MB (2.07%) Done in 0:14:20 at 01:04:56
0:50:37 Offset 402MB (2.62%) Done in 0:11:37 at 01:02:14

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

145

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

146

The SQLite3 output has 25% performance penalty whencombined with with feature file output .
All Threads Finished!
Producer time spent waiting: 260.394 sec.
Average consumer time spent waiting: 28.6439 sec.
********************************************* bulk_extractor is probably CPU bound. ****Run on a computer with more cores ****to get better performance.
*********************************************
MD5 of Disk Image: a6c44b7387a67333b8566955dcad6f50
Phase 2. Shutting down scanners
Phase 3. Uniquifying and recombining wordlist
Phase 3. Creating Histograms
Elapsed time: 567.171 sec.
Total MB processed: 15382
Overall performance: 27.121 MBytes/sec (1.69506 MBytes/sec/thread)
Total email features found: 8480

MD5 of Disk Image: a6c44b7387a67333b8566955dcad6f50
Phase 2. Shutting down scanners
Phase 3. Uniquifying and recombining wordlist
Phase 3. Creating Histograms
Elapsed time: 416.436 sec.
Total MB processed: 15382
Overall performance: 36.9378 MBytes/sec (2.30861 MBytes/sec/thread
Total email features found: 8480

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

147

Class demonstration: show the results of the runs

U.S. Department of Commerce
Economics and Statistics Administration
U.S. CENSUS BUREAU
census.gov

CLOUD ☁ FORENSICS

148