Package: rekall-layout-expert
# Rekall Layout ExpertLive Memory analysis on the Linux platform has traditionally been difficult toperform. Memory analysis requires precise knowledge of struct layout informationin memory, usually obtained through debugging symbols generated at compiletime. The Linux kernel is however, highly configurable, implying that debugginginformation is rarely applicable to systems other than the ones that generatedit. For incident response applications, obtaining the relevant debugginginformation is currently a slow and manual process, limiting its usefulness inrapid triaging.## How do we analyze Linux systems right now?The current process for generating a Rekall profile for a Linux system istedious:1) You must find and install the kernel headers package for the same kernel asthe running kernel (for example `apt-get installlinux-headers-3.16.0-39-generic`).2) Then you need to build a kernel module (`rekall/tools/linux/module.c`) onthat system to generate the debug kernel module `module_dwarf.ko`.3) Finally on a system with Rekall installed, one needs to convert this to aRekall profile (using `rekall convert_profile 4.2.0-generic.zip4.2.0-generic.json` for example).This is hard to do in an incident response situation. Sometimes servers do nothave the required compilers, kernel headers etc. This is especially hard if thekernel was cusom made. In that case it may be difficult to even find therequired kernel headers package (it may not have even been built with the customkernel). In all likelyhood you may need to copy the kernel config and System.mapoff the system you want to analyze to another system (with compiler tool chainsand kernel headers installed) so you can build the profile.This logistical issue make it difficult to do Linux live memory analysis inpractice - so you end up taking a memory image of the system for later analysis(Then you have to deal with transferring huge images around, smear and lot ofother fun problems :-).If you really want to be prepared, you must build a huge library of kernelprofiles in advance. For each released kernel version, you need to have everyvariation released by every distribution. For example in Ubuntu, there aregeneric and low latency variation (e.g. `linux-headers-3.16.0-39-generic`,`linux-headers-3.16.0-39-lowlatency`) for each minor version). You can justforget about having custom kernels in your library because you can not predictin advance what config parameters someone will change!## Is there a better way?Have you ever found yourself uttering: "I will pay someone $1000 to find a wayto do Linux Memory forensics without building a *$@#!% profile for every #@$#%@#kernel?" - I know I have!In a perferct world, we would just run Rekall on any Linux system, point it at`/proc/kcore` or `/dev/pmem` and just go without worrying about buildingprofiles! That would be nice.We are not quite there, but almost :-). The Layout Expert is the small stepforward. The process using the Layout Expect is much simpler:1) On the system you want to analyze, run the Layout Expert which will downloada single Pre-AST file for every kernel version (regardless of kernelconfiguration, distribution flavour etc.).2) Then launch the layout expert, providing it the local system's config file andSystem.map:```$ layout_tool make_profile --config_file_path boot/config-4.2.02.0.smp \--system boot/System.map-4.2.02.0.smp pre_ast_4.2.0-22.json profile.json2016-01-23 09:44:29,416 INFO     LOADING PREPROCESSOR AST FROM: pre_ast_4.2.0-22.json2016-01-23 09:44:34,494 INFO     DONE2016-01-23 09:44:34,495 INFO     LINKING INCLUDES2016-01-23 09:44:34,937 INFO     LINKED2016-01-23 09:44:34,937 INFO     EXTRACTING CONFIG FLAGS2016-01-23 09:44:34,994 INFO     EXTRACTED2016-01-23 09:44:35,108 INFO     PREPROCESSING2016-01-23 09:44:50,856 INFO     PREPROCESSED2016-01-23 09:44:50,856 INFO     Completed preprocessing pre-ast in 16 Seconds2016-01-23 09:44:50,856 INFO     GENERATING PURE C FILE2016-01-23 09:44:53,047 INFO     GENERATED2016-01-23 09:44:53,048 INFO     Completed generating pure C file in 2 Seconds2016-01-23 09:44:53,048 INFO     TRIMMING C FILE2016-01-23 09:45:14,340 INFO     Completed trimming C file in 21 Seconds2016-01-23 09:45:14,341 INFO     TRIMMED C FILE2016-01-23 09:45:14,354 INFO     PARSING STRUCTS2016-01-23 09:45:37,853 INFO     Completed parsing struct layouts in 23 Seconds2016-01-23 09:45:37,853 INFO     PARSED2016-01-23 09:45:37,853 INFO     GENERATING PROFILE2016-01-23 09:45:37,949 INFO     Exporting 627 structs2016-01-23 09:45:38,763 INFO     GENERATED```The Layout Expert is able to calculate the memory layout of critical kernelstructures at runtime on the target system without requiring extra tools, suchas the compiler tool-chain to be pre-installed.## How does it work?The main problem with memory analysis on Linux is that the Linux kernel is soconfigurable and customizable. For example, in order to properly parse thememory layout of `struct task_struct`, we can see the source:```struct task_struct {volatile long state;    /* -1 unrunnable, 0 runnable, >0 stopped */void *stack;atomic_t usage;unsigned int flags;     /* per process flags, defined below */unsigned int ptrace;#ifdef CONFIG_SMPstruct llist_node wake_entry;int on_cpu;struct task_struct *last_wakee;unsigned long wakee_flips;unsigned long wakee_flip_decay_ts;int wake_cpu;#endifint on_rq;int prio, static_prio, normal_prio;unsigned int rt_priority;const struct sched_class *sched_class;struct sched_entity se;struct sched_rt_entity rt;#ifdef CONFIG_CGROUP_SCHEDstruct task_group *sched_task_group;#endif#ifdef CONFIG_PREEMPT_NOTIFIERS/* list of struct preempt_notifier: */struct hlist_head preempt_notifiers;#endif....```Depending on kernel configuration options there will be different membersinserted in the middle of the struct - even for the same kernel version. This isprimarily why you need to compile a debug kernel module for every single kernelconfiguration - even of the same version. Depending on various kernel configoptions the layouts can change dramatically (sometimes if the profiles are veryclose some fields will be parsed correctly by Rekall but others wont - thefamiliar missing data in plugin outputs).The Layout Expert attempts to emulate the GCC compiler chains to the extent ofbeing able to predict the struct layout that the compiler might decideon. Essentially we simulate the compilation of the kernel debug module.The GCC compiler, reads the kernel config file and then preprocesses the kernelheaders to add or remove code depending on these configuration options. In theLayout Expert we wish to have a data structure that can be re-used for differentconfigurations without needing the kernel config.Therefore, the Layout Expert first parses all the kernel headers into aPreprocessor Abstract Syntax Tree (Pre-AST for short). The Pre-AST includes allthe possibilities of each `#ifdef` branch. This is the file which the LayoutExpert operates on.At runtime (i.e. during system analysis), the Layout Expert combines the systemconfiguration with the Pre-AST to produced the Preprocessed C code. In essence,the headers `#ifdef` directives are removed, and the different options arecombined to produce a final C file, free from preprocessing macros. In thisphase, the Layout Expert acts as a C pre-processor. The result is a huge C filewith all the code in all the headers stuck together.Next, the Layout Expert applies trimming to this file. This is essentially aquick once over pass to identify only structs, unions, enums and typedefinstructions. This optimization step means that we have much less code to parsein the next step and that the code that we do need to parse is more consistentand so easier to parse.Finally the Layout Expert parses the structs that Rekall is actually interestedin (i.e. those structs with plugins that look at them). This parsing phaseemulates a C compiler. We then apply the GCC struct layout model to the parsedstructs in order to predict the precise memory layout of all fields in thestructs (considering attributes, e.g. ``__attribute__((packed))`,`__attribute__((aligned(8))`).The last step is to generate a Rekall profile ready for use.## Preparation.Before the profile generation can occur in the field, we need to build the`Pre-AST` file for the specific kernel version. This is easy since it does notrequire any specific configuration file or special tools (Remember that thePre-AST includes all branches of any `#ifdef` directives so we do not need toevaluate any macros at this stage.).You can use the kernel headers package for the specific kernel, or the fullkernel source. There is no need to actually compile the source (i.e. `makedepmod` etc). Note that the kernel header package does not include "private" ornon exported structs, so these will be missing from the profile, but currentRekall does not need those.```$ layout_tool build_pre_ast --source_file_path ~/rekall/tools/linux/module.c \--linux_repository_path /usr/src/linux-headers-4.2.0-22-generic/ pre_ast_4.2.0-22.json2016-01-23 10:38:00,493 INFO     LOADING AND PARSING HEADERS2016-01-23 10:38:58,912 INFO     Completed built pre-ast forest in 58 Seconds2016-01-23 10:38:58,913 INFO     LOADED AND PARSED```## Bugs and supportThe Layout Expert is brought to you by the same people who develop Rekall, butit is considered a separate project. It is available under an Apache license(Check the LICENSE file). However, there is no official support or warranty; noteven for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.If you think you've found a bug, please report it at:https://github.com/google/rekall/issuesYou can also mail to the list rekall-discuss@googlegroups.com