Package: osxcollector-output-filters
OSXCollector Output FiltersTheosxcollector.output_filterspackage contains filters that process and transform the output ofOSXCollector. The goal of filters is to make it easy to analyze OSXCollector output.Each filter has a single purpose. They do one thing and they do it right.Running Filters in a VirtualEnvUnlikeosxcollector.pyfilters have dependencies that aren't already installed on a new Mac. The best solution for ensure dependencies can be found is to use virtualenv.To setup a virtualenv for the first time use:$sudopipinstalltoxvirtualenv
$makevenv
$sourcevirtualenv_run/bin/activate# Not necessary if you use aactivatorFilter ConfigurationMany filters require configuration, like API keys or details on a blacklist. The configuration for filters is done in a YAML file. The file is namedosxcollector.yaml. The filter will look for the configuration file in:The current directory.The user's home directory.The path pointed to by the environment variableOSXCOLLECTOR_CONF.A sample config is included. Make a copy and then modify if for yourself:$cposxcollector.yaml.exampleosxcollector.yaml
$emacsosxcollector.yamlBasic FiltersUsing combinations of these basic filters, an analyst can figure out a lot of what happened without expensive tools, without threat feeds or fancy APIs.FindDomainsFilterosxcollector.output_filters.find_domains.FindDomainsFilterattempts to find domain names in OSXCollector output. The domains are added to the line with the keyosxcollector_domains.FindDomainsFilter isn't too useful on it's own but it's super powerful when chained with filters likeFindBlacklistedFilterand orosxcollector.output_filters.virustotal.lookup_domains.LookupDomainsFilter.To run and see lines where domains have been added try:$python-mosxcollector.output_filters.find_domains-iRomeoCredible.json|\jq'select(has("osxcollector_domains"))'Usage:$python-mosxcollector.output_filters.find_domains-h
usage:find_domains.py[-h][--input-fileINPUT_FILE]optionalarguments:-h,--helpshowthishelpmessageandexit--input-fileINPUT_FILE[OPTIONAL]PathtoOSXCollectoroutputtoread.Defaultstostdinotherwise.FindBlacklistedFilterosxcollector.output_filters.find_blacklisted.FindBlacklistedFilterreads a set of blacklists from theosxcollector.yamland marks any lines with values on the blacklist. The BlacklistFilter is flexible and allows you to compare the OSXCollector output against multiple blacklists.Youreally shouldcreate blacklists for domains, file hashes, file names, and any known hinky stuff.Configuration Keys:blacklist_name: [REQUIRED] the name of the blacklist.blacklist_keys: [REQUIRED] get the value of these keys and compare against the blacklist. These can be of the forma.bto look atbin{"a": {"b": "foo"}}blacklist_file_path: [REQUIRED] path to a file with the actual values to blacklistblacklist_is_regex: [REQUIRED] should the values in the blacklist file be treated as regexblacklist_is_domains: [OPTIONAL] interpret values as domains and do some smart regex and subdomain stuff with them.If you want to find blacklisted domains, you will have to use the find_domains filter to pull the domains out first. To see lines matching a specific blacklist nameddomainstry:$python-mosxcollector.output_filters.find_domains-iRiddlerBelize.json|\python-mosxcollector.output_filters.find_blacklisted|\jq'select(has("osxcollector_blacklist")) | \select(.osxcollector_blacklist | keys[] | contains("domains"))'Usage:$python-mosxcollector.output_filters.find_blacklisted-h
usage:find_blacklisted.py[-h][--input-fileINPUT_FILE]optionalarguments:-h,--helpshowthishelpmessageandexit--input-fileINPUT_FILE[OPTIONAL]PathtoOSXCollectoroutputtoread.Defaultstostdinotherwise.RelatedFilesFilterosxcollector.output_filters.related_files.RelatedFilesFiltertakes an initial set of file paths, names, or terms. It breaks this input into individual file and directory names and then searches for these terms across the entire OSXCollector output. The filter is smart and ignores common terms likebinorLibraryas well as ignoring user names.This filter is great for figuring out howevil_invoice.pdflanded up on a machine. It'll find browser history, quarantines, email messages, etc. related to a file.To run and see related lines try:$python-mosxcollector.output_filters.related_files-iCanisAsp.json-f'/foo/bar/baz'-f'dingle'|\jq'select(has("osxcollector_related")) | \select(.osxcollector_related | keys[] | contains("files"))'Usage:$python-mosxcollector.output_filters.related_files-h
usage:related_files.py[-h][-fFILE_TERMS][--input-fileINPUT_FILE]optionalarguments:-h,--helpshowthishelpmessageandexit--input-fileINPUT_FILE[OPTIONAL]PathtoOSXCollectoroutputtoread.Defaultstostdinotherwise.

RelatedFilesFilter:-fFILE_TERMS,--file-termFILE_TERMS[OPTIONAL]Suspicioustermstouseinpivotingthroughfilenames.Maybespecifiedmorethanonce.ChromeHistoryFilterosxcollector.output_filters.chrome.sort_history.SortHistoryFilterbuilds a really nice Chrome browser history sorted in descending time order. This output is comparable to looking at the history tab in the browser but actually containsmoreinfo. Thecore_transitionandpage_transitionkeys explain whether the user got to the page by clicking a link, through a redirect, a hidden iframe, etc.To run and see Chrome browser history:$python-mosxcollector.output_filters.chrome.sort_history-iSirCray.json|\jq'select(.osxcollector_browser_history=="chrome")'This is great mixed with a grep in a certain time window, like maybe the 5 minutes before that hinky download happened.$python-mosxcollector.output_filters.chrome.sort_history-iSirCray.json|\jq-c'select(.osxcollector_browser_history=="chrome")'|\egrep'2015-02-02 20:3[2-6]'Usage:$python-mosxcollector.output_filters.chrome.sort_history-h
usage:sort_history.py[-h][--input-fileINPUT_FILE]optionalarguments:-h,--helpshowthishelpmessageandexit--input-fileINPUT_FILE[OPTIONAL]PathtoOSXCollectoroutputtoread.Defaultstostdinotherwise.FirefoxHistoryFilterosxcollector.output_filters.firefox.sort_history.SortHistoryFilterbuilds a really nice Firefox browser history sorted in descending time order. It's a lot like theChromeHistoryFilter.To run and see Firefox browser history:$python-mosxcollector.output_filters.firefox.sort_history-iCousingLobe.json|\jq'select(.osxcollector_browser_history=="firefox")'Usage:$python-mosxcollector.output_filters.firefox.sort_history-h
usage:sort_history.py[-h][--input-fileINPUT_FILE]optionalarguments:-h,--helpshowthishelpmessageandexit--input-fileINPUT_FILE[OPTIONAL]PathtoOSXCollectoroutputtoread.Defaultstostdinotherwise.ChromeExtensionsFilterosxcollector.output_filters.chrome.find_extensions.FindExtensionsFilterlooks for extensions in the Chrome JSON files.To run and see Chrome extensions:$python-mosxcollector.output_filters.chrome.find_extensions-iMotherlyWolf.json|\jq'select(.osxcollector_section=="chrome" and.osxcollector_subsection=="extensions")'Usage:$python-mosxcollector.output_filters.chrome.find_extensions-h
usage:find_extensions.py[-h][--input-fileINPUT_FILE]optionalarguments:-h,--helpshowthishelpmessageandexit--input-fileINPUT_FILE[OPTIONAL]PathtoOSXCollectoroutputtoread.Defaultstostdinotherwise.FirefoxExtensionsFilterosxcollector.output_filters.firefox.find_extensions.FindExtensionsFilterlooks for extensions in the Firefox JSON files.To run and see Firefox extensions:$python-mosxcollector.output_filters.firefox.find_extensions-iFlawlessPelican.json|\jq'select(.osxcollector_section=="firefox" and.osxcollector_subsection=="extensions")'Usage:$python-mosxcollector.output_filters.firefox.find_extensions-h
usage:find_extensions.py[-h][--input-fileINPUT_FILE]optionalarguments:-h,--helpshowthishelpmessageandexit--input-fileINPUT_FILE[OPTIONAL]PathtoOSXCollectoroutputtoread.Defaultstostdinotherwise.Threat API FiltersBy taking the output of OSXCollector and looking up further info with OpenDNS and VirusTotal APIs, Yelp enhances the output with useful info. Some of these APIs aren't free but they are useful.Using these filters as examples, it would be possible to integrate with additional free or premium threat APIs.osxcollector.output_filters.base_filters.threat_feed.ThreatFeedFilterhas most of the plumbing for hooking up to arbitrary APIs.OpenDNS RelatedDomainsFilterosxcollector.output_filters.opendns.related_domains.RelatedDomainsFiltertakes an initial set of domains and IPs and then looks up domains related to them with the OpenDNS Umbrella API.Often an initial alert contains a domain or IP your analysts don't know anything about. However, by gathering the 2nd generation related domains, familiarfriendsmight appear. When you're lucky, those related domains land up being the download source for some downloads you might have overlooked.The filter will ignore domains if they are in the blacklist nameddomain_whitelist. This helps to reduce churn and false positives.Run it as and see what it found:$python-mosxcollector.output_filters.find_domains-iNotchCherry.json|\python-mosxcollector.output_filters.opendns.related_domains\-ddismalhedgehog.com-dfantasticrabbit.org\-i128.128.128.28|\jq'select(has("osxcollector_related")) |select(.osxcollector_related | keys[] | contains("domains"))'The results will look something like:{
   'osxcollector_related': {
       'domains': {
           'domain_in_line.com': ['dismalhedgehog.com'],
           'another.com': ['128.128.128.28']
       }
    }
}Usage:$python-mosxcollector.output_filters.opendns.related_domains-h
usage:related_domains.py[-h][-dINITIAL_DOMAINS][-iINITIAL_IPS][--related-domains-generationsGENERATIONS][--input-fileINPUT_FILE]optionalarguments:-h,--helpshowthishelpmessageandexit--input-fileINPUT_FILE[OPTIONAL]PathtoOSXCollectoroutputtoread.Defaultstostdinotherwise.

opendns.RelatedDomainsFilter:-dINITIAL_DOMAINS,--domainINITIAL_DOMAINS[OPTIONAL]Suspiciousdomainstouseinpivoting.Maybespecifiedmorethanonce.-iINITIAL_IPS,--ipINITIAL_IPS[OPTIONAL]SuspiciousIPtouseinpivoting.Maybespecifiedmorethanonce.--related-domains-generationsGENERATIONS[OPTIONAL]HowmanygenerationsofrelateddomainstolookupwithOpenDNSOpenDNS LookupDomainsFilterosxcollector.output_filters.opendns.lookup_domains.LookupDomainsFilterlookups domain reputation and threat information with the OpenDNS Umbrella API. It adds information aboutsuspiciousdomains to the output lines.The filter uses a heuristic to determine what issuspicious. It can create false positives but usually a download from a domain marked assuspiciousis a good lead.Run it and see what was found:$python-mosxcollector.output_filters.find_domains-iGladElegant.json|\python-mosxcollector.output_filters.opendns.lookup_domains|\jq'select(has("osxcollector_opendns"))'Usage:$python-mosxcollector.output_filters.opendns.lookup_domains-h
usage:lookup_domains.py[-h][--input-fileINPUT_FILE]optionalarguments:-h,--helpshowthishelpmessageandexit--input-fileINPUT_FILE[OPTIONAL]PathtoOSXCollectoroutputtoread.Defaultstostdinotherwise.VirusTotal LookupDomainsFilterosxcollector.output_filters.virustotal.lookup_domains.LookupDomainsFilterlookups domain reputation and threat information with the VirusTotal API. It adds information aboutsuspiciousdomains to the output lines. It's a lot like the OpenDNS filter of the same name.The filter uses a heuristic to determine what issuspicious. It can create a lot of false positives but also provides good leads.Run it and see what was found:$python-mosxcollector.output_filters.find_domains-iPippinNightstar.json|\python-mosxcollector.output_filters.virustotal.lookup_domains|\jq'select(has("osxcollector_vtdomain"))'Usage:$python-mosxcollector.output_filters.virustotal.lookup_domains-h
usage:lookup_domains.py[-h][--input-fileINPUT_FILE]optionalarguments:-h,--helpshowthishelpmessageandexit--input-fileINPUT_FILE[OPTIONAL]PathtoOSXCollectoroutputtoread.Defaultstostdinotherwise.VirusTotal LookupHashesFilterosxcollector.output_filters.virustotal.lookup_hashes.LookupHashesFilterlookups hashes with the VirusTotal API. This basically finds anything VirusTotal knows about which is a huge time saver. There's pretty much no false positives here, but there's also no chance of detecting unknown stuff.Run it and see what was found:$python-mosxcollector.output_filters.virustotal.lookup_hashes-iFungalBuritto.json|\jq'select(has("osxcollector_vthash"))'Usage:$python-mosxcollector.output_filters.virustotal.lookup_hashes-h
usage:lookup_hashes.py[-h][--input-fileINPUT_FILE]optionalarguments:-h,--helpshowthishelpmessageandexit--input-fileINPUT_FILE[OPTIONAL]PathtoOSXCollectoroutputtoread.Defaultstostdinotherwise.VirusTotal LookupURLsFilterosxcollector.output_filters.virustotal.lookup_hashes.LookupURLsFilterlookups URLs with the VirusTotal API. As this only looks up the reports, it may not find the reports for some unknown URLs.Run it and see what was found:$python-mosxcollector.output_filters.virustotal.lookup_urls-iWutheringLows.json|\jq'select(has("osxcollector_vturl"))'Usage$python-mosxcollector.output_filters.virustotal.lookup_urls-h
usage:lookup_urls.py[-h][--input-fileINPUT_FILE]optionalarguments:-h,--helpshowthishelpmessageandexit--input-fileINPUT_FILE[OPTIONAL]PathtoOSXCollectoroutputtoread.Defaultstostdinotherwise.Maximum resources per requestBoth VirusTotal LookupHashesFilter and LookupURLsFilter can save time by including in a single API request the reports for the multiple resources (hashes or URLs).
As the number of the maximum resources in a request depends on whether you are using a Public or Private API key it is configurable inosxcollector.yamlfile invirustotalsection:resources_per_req:4ShadowServer LookupHashesFilterosxcollector.output_filters.shadowserver.lookup_hashes.LookupHashesFilterlookups hashes with the ShadowServer bin-test API. This is sort of the opposite of a VirusTotal lookup and returns results when it sees the hashes of known good files. This helps raise confidence that a file is not malicious.Run it and see what was found:$python-mosxcollector.output_filters.shadowserver.lookup_hashes-iArkashKobiashi.json|\jq'select(has("osxcollector_shadowserver"))'Usage:$python-mosxcollector.output_filters.shadowserver.lookup_hashes-h
usage:lookup_hashes.py[-h][--input-fileINPUT_FILE]optionalarguments:-h,--helpshowthishelpmessageandexit--input-fileINPUT_FILE[OPTIONAL]PathtoOSXCollectoroutputtoread.Defaultstostdinotherwise.AnalyzeFilter - The One Filter to Rule Them Allosxcollector.output_filters.analyze.AnalyzeFilteris Yelp'sone filter to rule them all. It chains all the previous filters into one monster analysis. The results, enhanced with blacklist info, threat APIs, related files and domains, and even pretty browser history is written to a new output file.ThenVery Readable Output Bottakes over and prints out an easy-to-digest, human-readable, nearly-English summary of what it found. It's basically equivalent to running:$python-mosxcollector.output_filters.chrome.find_extensions.FindExtensionsFilter-iSlickApocalypse.json|\python-mosxcollector.output_filters.firefox.find_extensions.FindExtensionsFilter|\python-mosxcollector.output_filters.find_domains|\python-mosxcollector.output_filters.shadowserver.lookup_hashes|\python-mosxcollector.output_filters.virustotal.lookup_hashes|\python-mosxcollector.output_filters.find_blacklisted|\python-mosxcollector.output_filters.related_files|\python-mosxcollector.output_filters.opendns.related_domains|\python-mosxcollector.output_filters.opendns.lookup_domains|\python-mosxcollector.output_filters.virustotal.lookup_domains|\python-mosxcollector.output_filters.chrome_history|\python-mosxcollector.output_filters.firefox_history|\teeanalyze_SlickApocalypse.json|\jq'select(false == has("osxcollector_shadowserver")) |select(has("osxcollector_vthash") orhas("osxcollector_vtdomain") orhas("osxcollector_opendns") orhas("osxcollector_blacklist") orhas("osxcollector_related"))'and then letting a wise-cracking analyst explain the results to you. TheVery Readable Output Boteven suggests hashes and domains to add to blacklists.This thing is the real deal and our analysts don't even look at OSXCollector output until after they've run theAnalyzeFilter.Run it as:$python-mosxcollector.output_filters.analyze-iFullMonty.jsonUsage:$python-mosxcollector.output_filters.analyze-h
usage:analyze.py[-fFILE_TERMS][-dINITIAL_DOMAINS][-iINITIAL_IPS][--related-domains-generationsGENERATIONS][-h][--readout][--no-opendns][--no-virustotal][--no-shadowserver][-M][--show-signature-chain][--show-browser-ext][--input-fileINPUT_FILE]optionalarguments:-h,--helpshowthishelpmessageandexit--input-fileINPUT_FILE[OPTIONAL]PathtoOSXCollectoroutputtoread.Defaultstostdinotherwise.

RelatedFilesFilter:-fFILE_TERMS,--file-termFILE_TERMS[OPTIONAL]Suspicioustermstouseinpivotingthroughfilenames.Maybespecifiedmorethanonce.

opendns.RelatedDomainsFilter:-dINITIAL_DOMAINS,--domainINITIAL_DOMAINS[OPTIONAL]Suspiciousdomainstouseinpivoting.Maybespecifiedmorethanonce.-iINITIAL_IPS,--ipINITIAL_IPS[OPTIONAL]SuspiciousIPtouseinpivoting.Maybespecifiedmorethanonce.--related-domains-generationsGENERATIONS[OPTIONAL]HowmanygenerationsofrelateddomainstolookupwithOpenDNS

AnalyzeFilter:--readout[OPTIONAL]Skiptheanalysisandjustoutputreallyreadableanalysis--no-opendns[OPTIONAL]Don't run OpenDNS filters--no-virustotal       [OPTIONAL] Don'trunVirusTotalfilters--no-shadowserver[OPTIONAL]Don'trunShadowServerfilters-M,--monochrome[OPTIONAL]Outputmonochromeanalysis--show-signature-chain[OPTIONAL]Outputunsignedstartupitemsandkexts.--show-browser-ext[OPTIONAL]Outputthelistofinstalledbrowserextensions.Contributing to OSXCollector Output FiltersWe encourage you to extend the functionality of OSXCollector to suit your needs.Testing OSXCollector Output FiltersA collection of tests for OSXCollector Output Filters is provided under thetestsdirectory. In order to run these tests you must installtox:$sudopipinstalltoxTo run this suit of tests,cdintoosxcollectorand enter:$maketestDevelopment TipsEnsure that all of the OSXCollector Output Filters tests pass before editing the source code. You can run the tests using:make testAfter making changes to the source code, runmake testagain to verify that your changes did not break any of the tests.LicenseThis work is licensed under the GNU General Public License.