Yelp/osxcollector — pull request #42 by ivanlei (2014-12-30 22:56:29+00:00)
Title: Grequests rework

* **Create `osxcollector.output_files.util`
Move a bunch of utility functions that have been living alongside a semi-random OutputFilter into their own module.

* **Create `osxcollector.output_files.util.http`
A submodule that deals with issuing multiple requests with grequests. This puts all the grequests code into one file and is easy to use in new filters that want to call an API.

* **Add ShadowServer hash lookup filter**
`osxcollector.output_filters.shadowserver.LookupHashesFilter` uses an API to identify hashes of known good files - parts of the OS and stuff. Use it to effectively mark OS files as non-suspicious. This was a great way to test the new http handling code.
VirusTotal hash lookup got 25x faster. This was enough to let `AnalyzeFilter` do lookups on all hashes all the time.

* **More attribution in the blacklist and related filters**
Rather than adding `{'osxcollector_blacklist': ['hashes']}` add `{'osxcollector_blacklist': {'hashes': ['abcdefg1234']}}`
Rather than adding `{'osxcollector_related': ['files']}` add `{'osxcollector_related': {'files': ['evil.dmg']}}`

* **Remove the concept of "suspicious" from `ThreatFeedFilter`
It was confusing and mostly just led to a lot of false positives.

* **More tests**
Including a good test driver for testing OutputFilters.

* **Lots of comment improvements**
Yeah, just improve em.

* **Handle strange encodings in domains**
This resolves #34. Basically nothing magic just tried a bunch of different things with some good test cases in place.

* **Do better at finding related files**
This resolves #40 by not using a regex. There was never a need to use a regex anyhow.
Also, RelatedFilesFilter takes a `when` function like other filters that lets it do a better job of finding more related files. For example, if the `when` function is
```
def when(blob):
return 'osxcollector_vthash' in blob
```
Then the RelatedFilesFilter will take the filename of any known malware and use it to find stuff related to that file - like the download that caused it maybe.