Yelp/osxcollector — pull request #16 by ivanlei (2014-11-30 23:17:29+00:00)
Title: Add osxcollector.output_filters to assist in analysis of output files

**OUTPUT FILTERS**
So analyzing output is time consuming. I've got a bunch of little scripts and tricks and what not that have been working for me for a while. It was time to automate them. Introducing **osxcollector.output_filters**. Yay! Fanfare!

The idea of output filters is that they enhance the original osxcollector output and make it easier to determine what's really going on. Lemme address each filters:

- `osxcollector.output_filters.domains` adds the `osxcollector_domains` key to output lines. This key has a list of domains that were found while parsing the line. These `osxcollector_domains` can then be used by future filters that blacklist, whitelist, or do threat feed lookups. All filters derive from `osxcollector.output_filters.output_filter.OutputFilter`. The base class implements basic I/O and even a nice method to basically implement the `main()` for derived filters. Run it like:
```
$ cat incident32.json | python -m osxcollector.output_filters.domains
```

- `osxcollector.output_filters.firefox_history` does some combines on several of the sqlite tables collected from firefox to make a nice time sorted browser history that basically matches the history shown in firefox.
```
$ cat INCIDENT32.json | python -m osxcollector.output_filters.firefox_history | \
jq 'select(.osxcollector_section=="firefox" and .osxcollector_subsection=="history" and .osxcollector_table_name =="moz_places")'
```

- `osxcollector.output_filters.chrome_history` like the firefox thingy above, but this time for Chrome.
```
$ cat INCIDENT32.json | python -m osxcollector.output_filters.firefox_history | \
jq 'select(.osxcollector_section=="chrome" and .osxcollector_subsection=="history" and .osxcollector_table_name =="visits")'
```

- `osxcollector.output_filters.opendns` lookups domains with OpenDNS. Domains associated with suspicious categories are further enhanced with additional OpenDNS data. Chain it with the domains filter and run it as:
```
$ cat INCIDENT32.json | python -m osxcollector.output_filters.domains | python -m osxcollector.output_filters.opendns | \
jq 'select(has("osxcollector_opendns"))'
```

- `osxcollector.output_filters.virustotal_hashes` lookups md5 hashes with VirusTotal. Run it as:
```
$ cat INCIDENT32.json | python -m osxcollector.output_filters.virustotal_hashes | jq 'select(has("osxcollector_vt_hashes"))'
```

- `osxcollector.output_filters.blacklist` reads a set of blacklists from the `osxcollector.yaml` and marks any lines with values on the blacklist. The BlacklistFilter allows for multiple blacklists to be compared against at once. Each blacklists requires:
- blacklist_name, A name
- blacklist_keys, JSON paths. These can be of the form "a.b" to look at "b" in {"a": {"b": "foo"}}
- value_file, the path to a file containing values considered blacklisted. Any line starting with # is skipped
- blacklist_is_regex, should values in the file be treated as Python regex

Run it as:
```
$ cat INCIDENT32.json | python -m osxcollector.output_filters.blacklist | jq 'select(has("osxcollector_blacklist"))'
```

**CONFIG FILE**
Many filters require configuration, like API keys or details on a blacklist. The configuration for filters is done in a YAML file. The file is named `osxcollector.yaml`. The filter will look for the config file in:
- The current directory.
- The user's home directory
- The path pointed to by the environment variable OSXCOLLECTOR_CONF

**VIRTUAL ENV**
Filters have dependencies that are not already installed on macs. This is fine because filters are run by analysts who can install things.
The requirements file used for testing also installs all the dependencies needed for the filters.

**PRE-COMMIT**
http://pre-commit.com/ is this awesome open source project from Yelp for adding useful pre-commit hooks to your git environment. It can be super helpful in finding bugs. A ton of minor fixes were made so that all the pre-commit hooks run clean.