2. Focused Research

Acquiring Forensic Memory Images on Linux

2.1 Linux Memory Dump Toolsn

avoid changing data on the target host as much as possible,

We use the Volatility Framework when performing memorycompilation of the analysis tools, etc., should be done on aanalysis for incident response and forensics. In IIR Vol. 32,

separate host.

“1.4.1 Creating Profiles for the Volatility Framework,” we explained the procedure for creating a Volatility profile for Linux*1.

2.2 What is LiME?
Here, we explain how to acquire Linux memory images for

LiME is short for Linux Memory Extractor, and is the tool

analysis in Volatility. Several tools for acquiring Linux mem-that Volatility recommends*5 for acquiring memory images.

*3

ory images exist, but here we will look at LiME and crash .

Since LiME operates as a Linux kernel module, it must be

We will also describe a memory image acquisition method

compiled against the same kernel version as that running onthat minimizes the impact on disk forensics. Note that thisthe target host.

*2

article assumes your system is running CentOS 7.7-1908.
Another tool for acquiring memory images is Linpmem*4, but

2.3 Compiling LiME

it did not work well in our test environment, so we decided

To compile LiME, you first obtain the LiME source code by

to leave it out.

running the git command in Figure 1 or by downloading thezip file from LiME’s GitHub page and extracting it into a

Note that in this article we refer to the machine being ex-directory. The src subdirectory inside the source code di-amined as the “target host” and the machine on which therectory contains a Makefile, so running the make command

examination is performed as the “examination host”. To

from within that subdirectory will generate the LiME module$ git clone https://github.com/504ensicsLabs/LiME.git

Figure 1: Cloning the LiME Git Repository$ cd LiME/src$ make$ ls
disk.c lime-3.10.0-1062.el7.x86_64.ko
disk.o lime.hhash.c lime.mod.chash.o lime.mod.olime.omain.cmain.oMakefile

Makefile.sample tcp.o
modules.order
Module.symvers
tcp.c$ sudo yum install kernel-devel gcc

Figure 2: Compiling the LiME Module

16

Figure 3: Installing the kernel-devel Package*1Internet Infrastructure Review (IIR) Vol. 32, 1.4.1 Creating Profiles for the Volatility Framework (https://www.iij.ad.jp/en/dev/iir/032.html).

*2

LiME (https://github.com/504ensicsLabs/LiME).

*3

crash (https://people.redhat.com/anderson/).

*4

Velocidex/c-aff4 (https://github.com/Velocidex/c-aff4/releases).

*5

Linux · volatilityfoundation/volatility Wiki (https://github.com/volatilityfoundation/volatility/wiki/Linux#acquiring-memory).

© Internet Initiative Japan Inc.

Feb.2020

Vol.

45

2. Focused Research(see the red outline in Figure 2). Certain packages are re-kernel-devel package was extracted. You also need to installquired to compile LiME (kernel-devel, gcc, etc.), so if the

any other required packages (on our machine, we needed tomake command produces an error, try installing the corre-

install elfutils-libelf-devel). We confirmed that the LiME mod-sponding packages as shown in Figure 3.

ule generated in this manner works on CentOS 8 (1905).

Note that unless you specify a version at install time, the lat-

2.4 Dumping Memory to an External Drive

est version of the kernel-devel package will be installed. If a

When acquiring memory images, you need to minimize disk

different version of the kernel is running on the target host,write operations on the target host. In particular, the mem-run a search using the yum command, install the appropriate

ory image is almost always several GB or more is size, sokernel-devel package version, and then compile the LiME

writing it to the target host’s disk would cause many unused

module, as shown in Figure 4. You need to pass the KVER

areas to be overwritten, which could greatly affect forensic

option to the make command when doing this.

analysis of the disk.

If the OS version differs, you can generate a LiME module

So if you have physical access to the target host, you canfor that version by downloading the corresponding ker-connect a USB stick or mobile SSD containing the LiME mod-nel-devel package separately, extracting it using the cpio

ule to it and load the LiME module into the kernel from therecommand, and then running make with the KVER and KDIR

via the insmod command, as shown in Figure 6. This will

options specified, as shown in Figure 5. KVER specifies thelet you acquire a memory image while barely writing any-kernel version, and KDIR specifies the directory where thething to the target host’s disk. The double quotes in Figure$ yum --showduplicates search kernel-devel$ sudo yum install kernel-devel-3.10.0-1062.1.2.el7.x86_64$ make KVER=3.10.0-1062.1.2.el7.x86_64

Figure 4: Compiling LiME Against a Specific Kernel Version (1)$ curl -OL http://ftp.iij.ad.jp/pub/linux/centos/8.0.1905/BaseOS/x86_64/os/Packages/kernel-devel-4.18.0-80.11.2.el8_0.x86_64.rpm$ rpm2cpio ./kernel-devel-4.18.0-80.11.2.el8_0.x86_64.rpm | cpio -id$ sudo yum install elfutils-libelf-devel$ make KVER=4.18.0-80.11.2.el8_0.x86_64 KDIR=~/src/usr/src/kernels/4.18.0-80.11.2.el8_0.x86_64/

Figure 5: Compiling LiME Against a Specific Kernel Version (2)$ sudo insmod /media/lime-3.10.0-1062.el7.x86_64.ko "path=/media/centos77.mem format=lime"

Figure 6: Loading the LiME Module© Internet Initiative Japan Inc.

17

6 contain the LiME module options. In this example, a lime

other tools to acquire a memory image via the network. Here,format dump file will be saved to the USB drive mounted at

we show how to combine it with Netcat, NFS, and SSH.

/media under the filename /media/centos77.mem.

The examination host’s IP address is 192.168.232.131,
and target host’s is 192.168.232.132.

The memory dump starts when the LiME module is loaded,
but the command prompt does not return until the mem-

■ Netcat (1)ory dump is completed. So don’t worry if you’re unable to

The LiME module has functionality that enables memory

enter commands after loading the module; just wait for thedumps over the network, and we combine this functional-

dump to complete. More recent servers, in particular, can be

ity with the Netcat command to acquire a memory image

expected to take a while because of their large memory ca-

via the network. Once the Netcat command is installed on

pacity. Note that the LiME module remains loaded even after

the examination host, the commands in Figures 8 and 9

the memory dump finishes, so unload it using the rmmod

will connect the examination host to the port on which thecommand, as shown in Figure 7.

LiME module is listening (4444/tcp) to acquire the memory
image data.

2.5 Dumping Memory via the Network
If you do not have physical access to the target host as■ Netcat (2)covered above (for example, if the target host is in a remote

The above procedure will result in data equivalent to thelocation that makes physical access difficult), or if you don’t

capacity of memory being received. If the target host is ahave a large-capacity USB stick, you can combine LiME with

server, that data may take up several dozen GB or more, so$ lsmod | grep lime
$ sudo rmmod lime

$ sudo insmod /media/lime-3.10.0-1062.el7.x86_64.ko "path=tcp:4444 format=lime"
Switch to another login session to execute the following command$ nc localhost 4444 | gzip -c | nc 192.168.232.131 5555

Figure 7: Unloading the LiME Module

Figure 11: Commands Executed on the Target Host
$ sudo insmod /media/lime-3.10.0-1062.el7.x86_64.ko "path=tcp:4444 format=lime"

Figure 8: Listening on 4444/tcp with LiME (Target Host)$ nc 192.168.232.132 4444 > centos77.mem

Figure 9: Acquiring a Memory Image via the Network Using Netcat(Examination Host)

Figure 12: Setting the NFS Server (Examination Host)/mnt/nfsserv/ 192.168.232.132(rw,all_squash)$ nc -l 5555 > memorydump.lime.gz

Figure 10: Command Executed on the Examination Host18

$ sudo yum install nfs-utils$ sudo mkdir /mnt/nfsserv/$ chown -R nfsnobody:nfsnobody /mnt/nfsserv/$ sudo vi /etc/exports$ sudo systemctl start nfs.service$ sudo systemctl status nfs.service

Figure 13: NFS Export Settings on the Examination Host (/etc/exports)© Internet Initiative Japan Inc.

Feb.2020

Vol.

45

2. Focused Research

we want to compress it as much as possible. If Netcat ishost with read/write capability. By copying LiME to this NFS

also installed on the target host, executing the commands

volume and mounting it as an NFS from the target host, you

in Figures 10 and 11 will result in the memory image being

can acquire a memory image without creating any files oncompressed using gzip when it is transferred to the exam-

the target host (Figures 12, 13, 14, and 15).

ination host. As mentioned, the command prompt does not
return when the LiME module is loaded, so the command■ SSH

following the nc command needs to be run from a separate

If you cannot use Netcat or NFS, you can use SSH instead.

login session.

By executing the commands in Figure 16, you can transfer amemory image to the examination host via SSH. However,In this example, we first use Netcat on the examination hostthis method can only be used in bash. As with the Netcat (2)to listen on 5555/tcp. Next, we use Netcat on the target

instructions, the exec command onward must be run from ahost to connect to 4444/tcp, where the LiME module is lis-

separate login session.

tening, acquire a memory image, compress it using gzip, andthen transfer it to the examination host via Netcat.

2.6 What is Crash?
The crash command is a tool for analyzing Linux mem-

■ NFS

ory images. The primary purpose of the tool is to perform

If the target host can mount an NFS volume, set up theanalysis, but a module can also be used to perform mem-examination host on a network that the target host has ac-ory dumps. But as there is no RPM package of the crashcess to. Then export the NFS volume on the examination

memory dump module, we have to compile it from source$ sudo firewall-cmd --permanent --add-service=nfs$ sudo firewall-cmd --reload$ sudo exportfs -v

Figure 14: Configuring the Examination Host’s Firewall
and Checking the NFS Export$ sudo mount -t nfs 192.168.232.131:/mnt/nfsserv/ /mnt/$ sudo insmod /mnt/lime-3.10.0-1062.el7.x86_64.ko "path=/mnt/centos77.mem format=lime"

Figure 15: Mounting the NFS and Acquiring a Memory Image (Target Host)$ sudo insmod /media/lime-3.10.0-1062.el7.x86_64.ko "path=tcp:4444 format=lime"
Switch to another login session to execute the following command$ exec 5<>/dev/tcp/127.0.0.1/4444; cat <&5 | ssh -c user@192.168.232.131 'cat > centos77.mem'

Figure 16: Commands Executed on the Target Host

© Internet Initiative Japan Inc.

19

(Figure 17). The crash command also requires a kernel with

2.7 Analyzing the Memory Image

debug symbols, so we install the kernel debugging package

To analyze a Linux memory image using Volatility*5, you(Figure 18). Next, we copy the three files shown in Figureneed a profile corresponding to the Linux kernel version.

19 to the same directory on a USB stick. On the target

As noted at the beginning, the procedure for creating ahost, we execute crash as shown in Figure 20 to acquire a

Volatility profile is described in IIR Vol. 32*1. Please refermemory image.

there if you are unsure.

If the kernel version of the target and examination hosts

While we were writing this article, Lorenzo Martínez

differ, then search for the appropriate version of the ker-

opened a Bitbucket repository that automatically gener-

nel-debuginfo package as demonstrated in Figure 4 (yumates and publishes LiME modules and Linux profiles for--showduplicates search). Download and extract the pack-

Volatility*7. This repository is updated whenever new Linuxage as shown in FIgure 5 (curl, rpm2cpio, cpio commands),

kernel packages are released. Note, though, that the OSs

and copy the vmlinux file. We also recommend using a crashthat it automatically generates for are CentOS 5, 6, 7,

version that corresponds to your OS version (for example,

8 and Ubuntu 14.04 LTS, 16.04 LTS, 18.04 LTS. You

crash-7.2.3-18 is provided in CentOS 8.0).

can find and download the appropriate LiME module and$ sudo yum install crash crash-devel$ yumdownloader --source crash$ rpm -ivh crash-7.2.3-10.el7.src.rpm$ cd rpmbuild/SPECS$ rpmbuild -bp crash.spec$ cd ../BUILD/crash-7.2.3$ make extensions

・/usr/bin/crash・rpmbuild/BUILD/crash-7.2.3/extensions/snap.so
・/usr/lib/debug/usr/lib/modules/3.10.0-1062.el7.x86_64/vmlinux

Figure 17: Installing the crash Source Package and Compiling the Module

Figure 19: Files Required for a Memory Dump

$ sudo yum install --enablerepo=base-debuginfo kernel-debuginfo-3.10.0-1062.el7.x86_64

Figure 18: Installing a Kernel with Debug Symbols

20*6

The Volatility Foundation - Open Source Memory Forensics (https://www.volatilityfoundation.org/).

*7

Lorenzo Martínez’ tweet (https://twitter.com/lawwait/status/1181469996821700609).

© Internet Initiative Japan Inc.

Feb.2020

Vol.

45

2. Focused Research

Volatility profile by filtering the repository webpage by ker-files one is not concerned about losing. It has been observed,

nel version.

however, that attackers take advantage of these properties
by using tmpfs as an anti-disk forensics haven for files.

Also, it does not support architectures other than x86_64,
so if you are using a Linux distribution or architecture otherFor this reason, Volatility provides the Linux-specific linux_

than those covered by the autogenerated files, you will need

tmpfs command. This command is used to restore files held

to prepare the files yourself. The same applies if you arein a tmpfs. Figure 21 illustrates tmpfs files mounted on theusing a customized Linux kernel. If using crash, you will also

/home/user/tmp directory being restored. The command re-need your own customized kernel with debug symbols.

sults in the file called “tmpfs_example.txt” being restored,
and evidently it contains the string “hello!!”

2.8 Tips for Acquiring Memory Images
In Linux kernel version 2.4 upward, the tmpfs file system is

If available memory is running low, however, the contents

available. Data on a tmpfs file system resides only in memoryof the tmpfs are swapped out, making it impossible to re-and is lost if the host is shut down or restarted, so tmpfs isstore the data in the tmpfs via a memory dump (Figure 22).

usually only used for temporary directories and the like with

A potential countermeasure in this case is to temporarily

Move to the directory where the files were copied and then execute thesecommands.
$ sudo ./crash ./vmlinux(Then from within the crash command prompt)
extend ./snap.so
snap centos77.mem$ hexdump -C ~/vol_output/swapout/tmpfs_example.txt
00000000 00 00 00 00 00 00 00 00
00000008

Figure 20: Acquiring a Memory Image|........|

Figure 22: Example of a Failure to Restore tmpfs Data due to a Swap-Out$ python2 ./vol.py --profile=LinuxCentOS77x64 -f ~/tmpfs_swapoff.mem linux_tmpfs -L
Volatility Foundation Volatility Framework 2.6.1
1 -> /sys/fs/cgroup2 -> /run3 -> /home/user/tmp
4 -> /dev/shm$ python2 ./vol.py --profile=LinuxCentOS77x64 -f ~/tmpfs_swapoff.mem linux_tmpfs -S 3 -D ~/vol_output/$ hexdump -C ~/vol_output/tmpfs_example.txt
00000000 68 65 6c 6c 6f 21 21 0a|hello!!.|00000008

Figure 21: Restoring and Examining the Contents of Files Stored in a tmpfs© Internet Initiative Japan Inc.

21

disable the swap. This means forcing data that has been

it with all types of memory images, but in our setup, weswapped out to be swapped in. Whether or not this can bewere able to analyze Windows RAW memory images anddone, however, depends on memory usage on the target

CentOS memory images acquired using LiME. A big change

host. If data has been swapped out because of a temporaryis that the option to specify a profile, which was required inincrease in memory usage, you may be able to subsequently

Volatility 2, is no longer present. Instead, Volatility 3 infers

disable the swap if memory has since been freed up. ThisOS type and version from the memory image being analyzed

means a situation like that illustrated in Figure 23, for ex-

and refers to the corresponding symbol table. For exam-

ample, where the value of Swap used is lower than freeple, when a Windows memory image is read in, Volatility

Mem. Analysis of the memory image acquired after disabling

3 automatically downloads the PDB file from Microsoft andthe swap confirms that the tmpfs data can be restored, as

analyzes it to construct a symbol table that it can reference.

shown in Figure 24.

However, you still need to prepare symbol tables for macOSand Linux in advance, as in Volatility 2. You can use theThis method, however, does overwrite data in unused areas

symbol tables provided by the Volatility developers, but theof memory. Because unused memory areas may still containsymbol tables for Linux lack information relative to those

useful data, a better way to perform Linux memory foren-

for Windows and macOS, so in most cases, you’ll need tosics is to dump memory once before disabling the swap andprovide the table yourself.

once again after.
Symbol tables are created using a tool called dwarf2json*9.

2.9 Volatility 3

As dwarf2json is written in Go, we first install the golang

Version 2 of Volatility has been in use for a long time, but

package and then build dwarf2json. We also install the ker-

during the writing of this article, a public beta version ofnel-debuginfo package because we need a Linux kernel with

Volatility 3 was released . Of course we did not check

symbols. Executing dwarf2json with a Linux kernel with*8

$ free
total
used
free
Mem:
186324875920
307192
Swap:
209714856776
2040372$ sudo swapoff -a$ free
total
used
free
Mem:
1863248118804247652Swap:000(After dumping memory, reenable the swap)$ sudo swapon -ashared buff/cache768
1480136

available
1591860

shared buff/cache
9800
1496792

available
1539936

Figure 23: Checking Memory Usage, and Disabling and Enabling the Swap$ hexdump -C ~/vol_output/swapoff/tmpfs_example.txt
00000000 68 65 6c 6c 6f 21 21 0a00000008|hello!!.|Figure 24: tmpfs Data Restored from a Memory Image
After Disabling the Swap

$ sudo yum install epel-release$ sudo yum install golang$ git clone https://github.com/volatilityfoundation/dwarf2json.git$ cd dwarf2json$ go build$ sudo yum install --enablerepo=base-debuginfo kernel-debuginfo-3.10.0-1062.1.2.el7.x86_64$ ./dwarf2json linux --elf /usr/lib/debug/usr/lib/modules/3.10.0-1062.1.2.el7.x86_64/vmlinux > centos77-3.10.0-1062.1.2.el7.x86_64.json$ xz -z centos77-3.10.0-1062.1.2.el7.x86_64.json$ wget https://downloads.volatilityfoundation.org/volatility3/symbols/linux.zip$ zip ./linux.zip ./centos77-3.10.0-1062.1.2.el7.x86_64.json.xz

Figure 25: Building dwarf2json and Generating a Symbol Table

22*8

Volatility Labs: Announcing the Volatility 3 Public Beta! (https://volatility-labs.blogspot.com/2019/10/announcing-volatility-3-public-beta.html).

*9dwarf2json (https://github.com/volatilityfoundation/dwarf2json).

© Internet Initiative Japan Inc.

Feb.2020

Vol.

45

2. Focused Research

symbols specified will generate a symbol table, so we add

analysis to fail. And with Windows memory images, some-

this to the file (linux.zip) that contains the symbol table dis-times analysis of the downloaded PDB fails, preventing you

tributed by the developers. See Figure 25 for the specificfrom advancing to the memory image analysis stage of thecommands. Copy the generated symbol table to the speci-

process.

fied Volatility 3 subdirectory (Figure 26).
The Volatility development team has announced an official
Volatility 3 is executed using this command line format: “py-Volatility 3 version will be released in August 2020. Support

thon3 vol.py -f <memory image file> <plugin>”. Figure 27

for Volatility 2 will continue for one year after that throughshows the results of analyzing a CentOS 7.7 memory image

August 2021, but with Volatility 3 set to become main-using the pstree plugin. The format has changed slightly fromstream ahead, it’s probably a good idea to get accustomed

that of Volatility 2, with the * character now used to denoteto the new usage and configuration methods before the of-process nesting. The method for specifying the plugin to beficial release hits.

executed has also changed. For the pstree plugin, you specify“linux_pstree” in Volatility 2, but in Volatility 3, you specify“linux.pstree.PsTree”. A list of available plugins can be foundby running “python3 vol.py -h”.
While we did get it to work properly, we also identified
some bugs. As noted above, OS type and version are inferred from the contents of the memory image specified on
the command line, but it can fail to recognize the correctLinux kernel version with some memory images, causing the$ cd ..
$ sudo yum install python3$ git clone https://github.com/volatilityfoundation/volatility3.git$ pip3 install --user pefile yara-python capstone$ cp ./dwarf2json/linux.zip ./volatility3/volatility/symbols/

Figure 26: Installing Volatility 3 and Copying the Symbol Tables$ cd volatility3$ python3 vol.py -f ~/centos77.mem linux.pstree.PsTree
Volatility 3 Framework 1.0.0-beta.1
Progress:
23.13
Scanning LimeLayer using RegExScanner
PID
PPID
COMM
1
0
systemd
* 833
1
login
** 1695 833
bash
* 841
1
firewalld* 843
1
NetworkManager** 992 843
dhclient* 558
1
systemd-journal* 593
1
systemd-udevd* 818
1
dbus-daemon* 1171 1
sshd** 1720 1171
sshd*** 1724
1720
sshd**** 1725
1724
bash***** 1751
1725
sudo
****** 1753
1751
insmod* 1172 1
tuned
* 821
1
systemd-logind* 822
1
irqbalance* 823
1
polkitd* 1174 1
rsyslogd* 1397 1
master** 1402 1397
pickup** 1405 1397
qmgr
(Subsequent output omitted)

Figure 27: Running the pstree Plugin on a Linux Memory Image

Minoru Kobayashi
Forensic Investigator, Office of Emergency Response and Clearinghouse for Security Information, Advanced Security Division, IIJ
Mr. Kobayashi is a member of IIJ-SECT, mainly dealing with digital forensics. He works to improve incident response capabilities and inhouse technical capabilities.
He gives lectures and training sessions at security events both in Japan and abroad, including Black Hat, FIRST TC, JSAC, and Security
Camp events.

© Internet Initiative Japan Inc.

23