plusvic/yara — pull request #184 by wxsBSD (2014-10-10 02:15:36+00:00)
Title: Import hashing, rich signatures and other things

This pull request does a bunch of things:

* Expand the pe module to do import hashing, as discussed here: https://www.mandiant.com/blog/tracking-malware-import-hashing/
* This required me to refactor the import processing. I moved the import parsing to parse_imports() and modified the imports function to walk the processed list. I also added the imphash function which uses the structures generated by parse_imports(). The structuresare conducive to generating the import hash easily but are also easily searched (they are very similar to how imports are stored in the table).
* In order to make sure my structures were zero'ed out I changed yr_malloc() to use calloc() internally.
* I added an ordinal lookup function to ensure that we generate the same values as pefile. The values are taken directly from pefile.
* The import hash code required that I bring in support for MD5. I picked an implementation from https://github.com/B-Con/crypto-algorithms and modified it slightly so there were no conflicts in definitions. At the time I wrote this there was no other MD5 code in yara, and I understand one was recently added. If you want to change this pull request over to use that, just let me know and I'll do the leg work.

* Expand the pe module to do rich signature processing and hashing. The rich signature is documented here: http://www.ntcore.com/files/richsign.htm
* The structure used allows for writing rules looking for the rich data as is, or looking for the clear data. You can also look for start value (the encrypted DanS) or an XOR key.
* Generating a hash of the clear text makes writing these signatures much easier, so I implemented that. I used the SHA256 implementation from the same repository mentioned above.

* As I am storing the rich signature block in a string variable inside the PE module it required that I change module string variables to be SIZED_STRING type internally. This ensures that they can contain NULL bytes and behave properly as the length is stored with them. This required me to change every set_string() to pass a length of the string.
* Because you can now write rules which search for NULL bytes in module strings I added two new pieces to grammar.y which are "contains" and "matches". These allow you to search for strings with NULL bytes by using memmem() for "contains" and memcmp() for "matches".
* I was not able to fully test the external variables in this scenario so please be aware of that.

* Minor change where I missed a use of IMAGE_FILE_MACHINE_AMD64 in an earlier patch. ;)

Here's some rules which show how to use this (the hashes have been changed to protect the guilty):

rule imphash {
meta:
foo = "BAR"
condition:
pe.imphash("a695f4e7d9765a5694b850080f32bd28")
}

rule richsig {
meta:
foo = "BAR"
condition:
pe.rich_signature.richhash("48ef506feb3a848a28818d2e8a21eb1b0264500b28fd48acf0235388b1a749fe")
}

rule richsig_clear {
strings: $a = "\x42\x90\x01\x00\x30\x00\x00\x00"
condition:
pe.rich_signature.clear_data contains $a
}

If you have any issues with this please let me know. I'm not very happy with adding the "contains" and "matches" pieces to the grammar. I'm not very big on parsers so if you have a different way of doing it please let me know so I can fix it.

My apologies for putting so much in one PR. If you would like me to break this down into smaller chunks please let me know. I promise future ones won't mix multiple changes. I promise my future stuff will be much smaller. :)