Data Source Analysis and Dynamic Windows RE using WPP and TraceLogging
By: Matt Graeber
Tags: Windows, Security, Etw, Debugging, Software Tracing

WPP Manifest Recovery and Tracing Methodology

Before diving into WPP, I should make clear that WPP was intended solely for the purposes of debugging and is not intended to be consumed for any other purpose (e.g. defensive telemetry, etc.). That said, there may be scenarios where certain target Windows binaries may have extremely valuable information that can be obtained from a WPP trace. For example, from a security research perspective, WPP traces frequently leak pointers that may be of value as an exploitation primitive. How you end up using WPP is ultimately up to you but I feel it is worthwhile to document a methodology for recovering WPP traces from black-box binaries so that you can begin to explore potentially interesting, previously unknown data sources.

Note: application of WPP tracing methodology requires reverse engineering skills. WPP, by design (with a couple exceptions), does not supply an event manifest. There are many hints that a reverser is provided, however, upon loading symbols that allow you to infer and gradually build out a manifest, the steps to which are established in this post.

Step 1: Identify a potentially interesting binary that you suspect might supply interesting information.

Binaries that utilize WPP (specifically, user-mode binaries in this case) will import RegisterTraceGuids and TraceMessage. Upon loading symbols for a binary, the use of WPP will become evident. Throughout this section, amsi.dll will be used as our target binary. Upon loading symbols for amsi.dll, the following WPP-related functions were present:

Internal WPP-related functions you would expect to see in user-mode code that support WPP tracing.

All of the WPP_SF_* functions (where “SF” stands for “staging function”) were generated at compile time by the WPP preprocessor and they offer a convenient contextual clue as to how data is traced based on the data type signature that follows “SF_”. The data type meanings can be found here but I will break down the ones seen above:

q: pointer type (either 32-bit or 64-bit depending on the process architecture)

P: pointer type (either 32-bit or 64-bit depending on the process architecture)

D: an unsigned 32-bit integer

d: a signed 32-bit integer

I: an unsigned 64-bit integer

S: a Unicode string

The data are pushed onto the stack in the order listed in the function name. For example, in the case of the WPP_SF_qqDqq function, the last five arguments pushed onto the stack would be: pointer, pointer, unsigned 32-bit integer, pointer, pointer. Knowledge of the data types as they are pushed onto the stack is helpful but by themselves, it doesn’t offer the full context. Additional reverse engineering will be required to acquire more context. We’ll come back to that later.

Step 2: Identify the WPP provider GUID

WPP relies upon ETW for tracing, meaning that if you want to collect WPP trace data, then you need specify an ETW provider using its unique identifier — a GUID. Within a WPP binary, the provider GUID is supplied via the RegisterTraceGuids ControlGuid parameter which is populated by the “_WPP_REGISTRATION_GUIDS” global variable — 75697175–75e2–4d85–83bc-7278acc12de4 in the case of amsi.dll:

WPP GUID being supplied to RegisterTraceGuids

Step 3: Identify Message GUIDs

When WPP trace data is logged with TraceMessage, a MessageGuid argument is required if the TRACE_MESSAGE_GUID flag is specified. A message GUID serves as a unique identifier for a trace message. When recovering a WPP manifest, it is important to capture all unique message GUIDs present in the binary. This process is relatively straightforward. When symbols are applied, each unique message GUID is defined in a global variable named “_WPP_GUIDVALUE_Traceguids”. A search of “_Traceguids” quickly yields all code references to where message GUIDs are used:

Cross-references to WPP message GUIDs

So within amsi.dll, there are three unique message GUIDs:

589f8473-ee36–3dff-b2d3–87ad72c4e5b3 53928f2d-4ad3–314c-3b3c-3c72f80ad6c1 b81299c2-b875–3726–1d3a-e6232a5e451b

These message GUIDs will be needed when we start to build out an event manifest in the form of a trace message format (TMF) file.

Step 4: Start building out a TMF file

While TMF files were not intended to be written by hand, that’s what we will have to resort to when they are not supplied in symbols, which will most often be the case. The TMF file “schema” is described in default.tmf which you can obtain from the Windows SDK. You can also find it here. Some TMFs are included within PDB files. Here’s an example event extracted from ndis.pdb:

52f7bea8-bd1d-3ca1–4d06–195c90839a85 mp // SRC=Unknown_cxx00 MJ= MN=

#typev Unknown_cxx00 85 “%0<==ndisWMIDispatch: Miniport %10!p!, Status %11!x!” // LEVEL=NDIS_TRACE_LEVEL_INFO FLAGS=DBG_COMP_WMI FUNC=ndisWMIDispatch

{

Arg, ItemPtr — 10

Arg, ItemLong — 11

}

The event description consists of the following:

Message GUID: 52f7bea8-bd1d-3ca1–4d06–195c90839a85

Event source name: mp

Source code file/line: Unknown_cxx00 (The “00” indicates the line number but this is was likely stripped from public symbols)

Message number/event ID: 85

Event message: “%0<==ndisWMIDispatch: Miniport %10!p!, Status %11!x!”

The 10 and 11 refer to the format string index. 10 is the beginning of user-supplied event parameters. See default.tmf for more information.

The 10 and 11 refer to the format string index. 10 is the beginning of user-supplied event parameters. See default.tmf for more information. Trace level: NDIS_TRACE_LEVEL_INFO

An optional field. You can define custom tracing levels beyond the standard logging levels: 1 — Fatal, 2 — Error, 3 — Warning, 4 — Informational, 5 — Verbose

An optional field. You can define custom tracing levels beyond the standard logging levels: 1 — Fatal, 2 — Error, 3 — Warning, 4 — Informational, 5 — Verbose Trace flag: DBG_COMP_WMI

Function name: ndisWMIDispatch

Message parameter #1 name: Arg

Message parameter #1 data type: ItemPtr

Message parameter #1 format string index: 10

Message parameter #2 name: Arg

Message parameter #2 data type: ItemLong

Message parameter #2 format string index: 11

Now, let’s start to build out our own TMF event entries for amsi.dll functions. Let’s look at one example WPP_SF_* function and start building a TMF event entry. Here is a fully annotated disassembly of WPP_SF_DDD:

Annotated WPP staging function

Most of the relevant event information will be passed via arguments to this function. We will analyze a function that calls WPP_SF_DDD shortly. For now, the most important information to notice is the MessageFlags argument, specifically, the TRACE_MESSAGE_GUID and TRACE_MESSAGE_SEQUENCE values. TRACE_MESSAGE_GUID tells us that a message GUID will be supplied (which we extracted in the previous step) and TRACE_MESSAGE_SEQUENCE tells us that when we capture a WPP trace, we will need to tell the trace session to log either local or global sequence numbers (both options in tracelog.exe).

Next, we’ll need to inspect the functions that call WPP_SF_DDD and reason over the arguments. One function that calls WPP_SF_DDD, as an example, is VerifyProtectionLevel:

Annotated VerifyProtectionLevel function in amsi.dll that makes use of WPP tracing

Without needing to fully reverse the VerifyProtectionLevel function, basic WPP message information can be extracted:

Message GUID: b81299c2-b875–3726–1d3a-e6232a5e451b

Message number: 12

Arguments: 3 unsigned integers (which was previously established)

Trace flag value: 4. How did I know 4 corresponds to a trace flag? I will cover this logical leap soon.

So now we have the minimum information necessary to build out a skeleton trace message for a TMF file:

b81299c2-b875–3726–1d3a-e6232a5e451b VerifyProtectionLevel

#typev amsiantimalware_cxx00 12 “[VerifyProtectionLevel] Arg0(%10!u!), Arg1(%11!u!), Arg2(%12!u!)”

{

Arg0, ItemLong //10

Arg1, ItemLong //11

Arg2, ItemLong //12

}

Note that the “amsiantimalware_cxx00” string was derived from the presence of the following string in amsi.dll: “onecore\amcore\antimalware\source\amsi\com\amsiantimalware.cpp”. The “_cxx00” naming simply uses the convention for file name/lines in public symbols.

In order to build out a more detailed trace message, reversing the function in order to gain sufficient context will be necessary. After reversing VerifyProtectionLevel sufficiently, the following trace message content was derived:

b81299c2-b875–3726–1d3a-e6232a5e451b VerifyProtectionLevel

#enumv PS_PROTECTED_TYPE

{

PsProtectedTypeNone,0

PsProtectedTypeProtectedLight,1

PsProtectedTypeProtected,2

} #enumv PS_PROTECTED_SIGNER

{

PsProtectedSignerNone,0

PsProtectedSignerAuthenticode,1

PsProtectedSignerCodeGen,2

PsProtectedSignerAntimalware,3

PsProtectedSignerLsa,4

PsProtectedSignerWindows,5

PsProtectedSignerWinTcb,6

PsProtectedSignerWinSystem,7

PsProtectedSignerApp,8

} #typev amsiantimalware_cxx00 12 “%0[VerifyProtectionLevel] ProcessID(%10!u!), ProtectedType(%11!s!), ProtectedSigner(%12!s!)” // FUNC=VerifyProtectionLevel FLAGS=DBG_4

{

ProcessID, ItemLong //10

ProtectedType, ItemEnum(PS_PROTECTED_TYPE) //11

ProtectedSigner, ItemEnum(PS_PROTECTED_SIGNER) //12

}

As you can see, the level of effort put into reversing will dictate the amount of detail you will be able to include in your trace message. Here are some of the relevant portions added that warrant additional explanation:

“#enumv” — The TMF format allows you to define enums where you can then apply the definition to event parameters using the “ItemEnum” data type. Note that when you apply the ItemEnum data type, the string format specifier (“s”) should be used in the message.

“%0” — An undocumented, convenient macro that tracefmt.exe will interpret and replace with the CPU number, process ID, thread ID, message source name. PDBs that include TMF metadata will often prepend message strings with “%0”.

“FLAGS=DBG_4” — The trace flag level of 4 was used enough times in code that I wanted to name it. I have no idea what it would have been named in source so that was the name I came up with.

You may be wondering how I was able to infer that “test byte ptr [edx+1Ch], 4” instruction referred to a trace flag value. This was easy enough to determine by tracing back to how the “_WPP_GLOBAL_Control” structure was initially populated, some of which occurs in the WppControlCallback which can be seen below:

WppControlCallback function populating the trace flag value

What you can’t see in the screenshot above is that ESI is populated via the RequestContext parameter and that the WppInitUm function passes in the “_WPP_GLOBAL_Control” structure as its argument. So as can be seen, the return value of GetTraceEnableFlags is stored in _WPP_GLOBAL_Control+0x1C (an offset into the WPP_TRACE_CONTROL_BLOCK structure) which is the offset referenced prior to any WPP_SF_* function call. And considering only a single “test” instruction is called prior to each WPP_SF_* call, it is evident that there is a minimal performance impact when WPP tracing is not being performed.

So hopefully, now that you’re armed with sufficient information needed to build out a TMF, given enough time and energy, you can do a pretty good job of building up a TMF from scratch. Here is my mostly complete TMF file for amsi.dll for you to play around with.

Step 5: Initiate a WPP trace

Now, building out an event schema for WPP trace events in the form of a TMF file is great and all but we haven’t actually done anything valuable with it yet. Now that we’ve gone through the painstaking process of creating a TMF file, let’s actually launch a WPP trace. There are several methods to launch a WPP trace. Here is one method:

logman start AMSITrace -p “{75697175–75e2–4d85–83bc-7278acc12de4}” 7 -ets -mode 0x8000 -rt

The components of this command can be broken down as follows:

logman — one of the built-in utilities used to create/start/update/delete ETW trace sessions

start — this indicates that logman should create and start an ETW trace session

AMSITrace — the name of the ETW trace session that will be started. You can name this whatever you like.

-p “{75697175–75e2–4d85–83bc-7278acc12de4}” 7 — this specifies the ETW provider that we want to capture. In this case, this is the AMSI WPP provider we extracted earlier. 7 indicates the desired trace flags. 7 was selected based on reversing amsi.dll where all the possible trace flags supported were 1, 2, and 4. 7 is the result of ORing those values together. If you wanted to always capture everything though, you could just supply 255 as the value.

-ets — starts the trace session immediately.

-mode 0x8000 — indicates that local sequence numbers should be captured in the trace. 0x8000 refers to EVENT_TRACE_USE_LOCAL_SEQUENCE, one of the logging mode constants. This value was specified due to the fact that TRACE_MESSAGE_SEQUENCE was specified as one of the MessageFlags when TraceMessage is called. EVENT_TRACE_USE_GLOBAL_SEQUENCE (0x4000) would be an acceptable value as well.

-rt — captures a real-time trace — i.e. one that is not backed by an ETL file. Instead of saving to an ETL file in this example, tracerpt.exe will be used to dump trace data directly to properly formatted EVTX file.

Here is the equivalent PowerShell code to launch an identical trace:

New-EtwTraceSession -Name AMSITrace -LogFileMode 0x8100

Add-EtwTraceProvider -SessionName AMSITrace -Guid ‘{75697175–75e2–4d85–83bc-7278acc12de4}’ -MatchAnyKeyword 0x7

Note the LogFileMode of 0x8100. 0x8000 refers to the same EVENT_TRACE_USE_LOCAL_SEQUENCE as described above. 0x100 refers to EVENT_TRACE_REAL_TIME_MODE. New-EtwTraceSession doesn’t have the equivalent of a “-rt” swtich so you just have to specify the value in this fashion.

So now that the trace is running, we need something to capture trace data. For this, we will use tracerpt.exe.

tracerpt -rt AMSITrace -tmf AMSI.tmf -of EVTX -o ParsedAMSITrace.evtx

So as ETW flushes its event buffers for the AMSITrace tracing session, tracerpt.exe will interpret each event, parse it using the TMF file we crafted (AMSI.tmf), and then write the events to an EVTX file. At this point, this is when you would perform your AMSI-related activities. When you want to terminate the trace, run the following:

logman stop AMSITrace -ets

Upon stopping the trace, any AMSI-related trace data will be parsed nicely in the EVTX file.

A parsed WPP trace EVTX file

When I’m doing quick and dirty traces, I like to use tracefmt.exe from the Windows SDK to dump events to the console in real time. When I do this, I prefer to not wait around for buffers to eventually flush so I will specify a flush timer of 1 second. The following commands will dump our AMSI trace directly to the console:

New-EtwTraceSession -Name AMSITrace -LogFileMode 0x8100 -FlushTimer 1

Add-EtwTraceProvider -SessionName AMSITrace -Guid ‘{75697175–75e2–4d85–83bc-7278acc12de4}’ -MatchAnyKeyword 0x7

tracefmt -rt AMSITrace -tmf tmf\AMSI_filled.tmf -displayonly

Of course when you’re done with your trace, remember to stop it:

logman stop AMSITrace -ets or Remove-EtwTraceSession -Name ‘AMSITrace’

The following is the result of a sample trace:

As can be seen, these traces can be revealing. For example, we get some insight into UAC scanning being performed upon being prompted with a UAC dialog. Also, as in-memory content is scanned, you can see the internal functions called in addition to some pointers being revealed. The leaking of the amsiContext pointer (a global variable in PowerShell) is sufficient information alone to subvert AMSI since an attacker can corrupt the structure, placing AMSI into a fail open state.

WPP Tracing Conclusion

This only demonstrated a methodology for performing black-box WPP tracing of user mode code. The methodology for kernel drivers is very similar but different APIs and offsets will be present. For some background on kernel WPP methodology, I encourage you to read this post about an information disclosure issue in an HP keyboard driver.

Despite the technical depth of the WPP tracing methodology covered here, we have only scratched the surface. How you choose to wield these skills is now up to you. You may be surprised to find discover how many Windows binaries support WPP. As will be the case with most binaries though, unfortunately, TMF metadata is not included in the majority of PDBs so you will need to become comfortable applying this methodology. Once you become more comfortable applying this methodology though, depending on you goal, you may be able to answer the following questions:

As a defender, for a given binary I’m interested in, could there be relevant defensive telemetry that I might be able to capture as a means of supplementing/enriching other data sources? Please bear in mind though that any reliance upon WPP for defensive telemetry purposes is subject to change at any point. At some point, perhaps a module will cease to support WPP or events could be added or removed (also altering event IDs as a result). In other words, use this in production at your own risk. As a vulnerability researcher, are there any binaries that support WPP tracing that might leak valuable pointers or some other form of sensitive information? For example, there will almost always be a WPP_SF_* function that logs pointers. Are any of those leaking pointers that might serve as a primitive for further exploitation? WPP tracing can also offer insight into protected processes. As a malware analyst, is there an opportunity to trace API calls without needing to attach a debugger, thus subverting malware attempts to break analysis/sandbox environments? As an attacker, are there any WPP sources that might leak valuable information (e.g. credential material)? For example, a natural research target would be kerberos.dll and msv1_0.dll (used for performing NTLM auth), both of which support WPP tracing.

WPP Tracing References

The following resources were invaluable to me while researching this topic: