Package: machinae
Machinae Security Intelligence CollectorMachinae is a tool for collecting intelligence from public sites/feeds about
various security-related pieces of data: IP addresses, domain names, URLs,
email addresses, file hashes and SSL fingerprints. It was inspired byAutomater, another excellent tool for collecting information. The Machinae
project was born from wishing to improve Automater in 4 areas:Codebase - Bring Automater to python3 compatibility while making the code
more pythonicConfiguration - Use a more human readable configuration format (YAML)Inputs - Support JSON parsing out-of-the-box without the need to write
regular expressions, but still support regex scraping when neededOutputs - Support additional output types, including JSON, while making
extraneous output optionalInstallationMachinae can be installed using pip3:pip3 install machinaeOr, if you're feeling adventurous, can be installed directly from github:pip3 install git+https://github.com/HurricaneLabs/machinae.gitYou will need to have whatever dependencies are required on your system for
compiling Python modules (on Debian based systems,python3-dev), as well as
the libyaml development package (on Debian based systems,libyaml-dev).You'll also want to grab thelatest configuration fileand place it in/etc/machinae.yml.Configuration FileMachinae supports a simple configuration merging system to allow you to make
adjustments to the configuration without modifying the machinae.yml we provide
you, making configuration updates a snap. This is done by finding a system-wide
default configuration (default/etc/machinae.yml), merging into that a
system-wide local configuration (/etc/machinae.local.yml) and finally a
per-user local configuration (~/.machinae.yml). The system-wide configuration
can also be located in the current working directory, can be set using theMACHINAE_CONFIGenvironment variable, or of course by using the-cor--configcommand line options. Configuration merging can be disabled by
passing the--nomergeoption, which will cause Machinae to only load the
default system-wide configuration (or the one passed on the command line).As an example of this, say you'd like to enable the Fortinet Category site,
which is disabled by default. You could modify/etc/machinae.yml, but these
changes would be overwritten by an update. Instead, you can put the following
in either/etc/machinae.local.ymlor~/.machinae.yml:fortinet_classify:
  default: trueOr, conversely, to disable a site, such as Virus Total pDNS:vt_ip:
  default: false
vt_domain:
  default: falseUsageMachinae usage is very similar to Automater:usage: machinae [-h] [-c CONFIG] [--nomerge] [-d DELAY] [-f FILE] [-i INFILE] [-v]
            [-o {D,J,N,S}] [-O {ipv4,ipv6,fqdn,email,sslfp,hash,url}] [-q]
            [-s SITES] [-a AUTH] [-H HTTP_PROXY]
            [--dump-config | --detect-otype]
            ...See above for details on the-c/--configand--nomergeoptions.Machinae supports a-d/--delayoption, like Automater. However, Machinae
uses 0 by default.Machinae output is controlled by two arguments:-ocontrols the output format, and can be followed by a single character
to indicated the desired type of output:Nis the default output ("Normal")Dis the default output, but dot characters are replacedJis JSON output-f/--filespecifies the file where output should be written. The default
is "-" for stdout.Machinae will attempt to auto-detect the type of target passed in (Machinae
refers to targets as "observables" and the type as "otype"). This detection can
be overridden with the-O/--otypeoption. The choices are listed in the
usageBy default, Machinae operates in verbose mode. In this mode, it will output
status information about the services it is querying on the console as they are
queried. This output will always be written to stdout, regardless of the output
setting. To disable verbose mode, use-qBy default, Machinae will run through all services in the configuration that
apply to each target's otypeandare not marked as "default: false". To modify
this behavior, you can:Pass a comma separated list of sites to run (use the top level key from the
configuration).Pass the special keywordallto run through all servicesincludingthose
marked as "default: false"Note that in both cases,otypevalidation is still applied.Machinae supports passing an HTTP proxy on the command line using the-H/--http-proxyargument. If no proxy is specified, machinae will search the
standardHTTP_PROXYandHTTPS_PROXYenvironment variables, as well as the
less standardhttp_proxyandhttps_proxyenvironment variables.Lastly, a list of targets should be passed. All arguments other than the
options listed above will be interpreted as targets.Out-of-the-Box Data SourcesMachinae comes with out-of-the-box support for the following data sources:IPVoidURLVoidURL Unshortener (http://www.toolsvoid.com/unshorten-url)Malc0deSANSFreeGeoIP (freegeoip.io)Fortinet CategoryVirusTotal pDNS (via web scrape - commented out)VirusTotal pDNS (via JSON API)VirusTotal URL Report (via JSON API)VirusTotal File Report (via JSON API)Reputation AuthorityThreatExpertVxVaultProjectHoneypotMcAfee Threat IntelligenceStopForumSpamCymru MHRICSI Certificate NotaryTotalHash (disabled by default)DomainTools Parsed Whois (Requires API key)DomainTools Reverse Whois (Requires API key)DomainTools ReputationIP WHOIS (Using RIR REST interfaces)Hacked IPMetadefender Cloud (Requires API key)GreyNoise (Requires API key)IBM XForce (Required API key)With additional data sources on the way.HTTP Basic Authentication and ConfigurationMachinae supports HTTP Basic Auth for sites that require it through the--auth/-aflag. You will need to create a YAML file with your credentials, which will include
a key to the site that requires the credentials and a list of two items, username
and password or API key. For example, for the included PassiveTotal site this might
look like:passivetotal: ['myemail@example.com', 'my_api_key']Inside the site configuration underrequestyou will see a key such as:json:
  request:
    url: '...'
    auth: passivetotalTheauth: passivetotalpoints to the key inside the authentication config passed
via the command line.Disabled by defaultThe following sites are disabled by defaultFortinet Category (fortinet_classify)Telize Geo IP (telize)TotalHash (totalhash_ip)DomainTools Parsed Whois (domaintools_parsed_whois)DomainTools Reverse Whois (domaintools_reverse_whois)DomainTools Reputation (domaintools_reputation)PassiveTotal Passive DNS (passivetotal_pdns)PassiveTotal Whois (passivetotal_whois)PassiveTotal SSL Certificate History (passivetotal_sslcert)PassiveTotal Host Attribute Components (passivetotal_components)PassiveTotal Host Attribute Trackers (passivetotal_trackers)MaxMind GeoIP2 Passive Insight (maxmind)FraudGuard (fraudguard)Shodan (shodan)Hacked IPMetadefender Cloud (Requires API key)GreyNoise (Requires API key)IBM XForce (Requires API key)Output FormatsMachinae comes with a limited set of output formats: normal, normal with dot
escaping, and JSON. We plan to add additional output formats in the future.Adding additional sites*** COMING SOON ***Known IssuesSome ISP's on IPvoid contain double-encoded HTML entities, which are not
double-decodedUpcoming FeaturesAdd IDS rule search functionality (VRT/ET)Add "More info" link for sitesAdd "dedup" option to parser settingsAdd option for per-otype request settingsAdd custom per-site output for error codesVersion HistoryVersion 1.4.9 (2020-11-25)Fix bug in JSON as_time processing when time is an epoch time, but str typeVersion 1.4.1 (2018-08-31)New FeaturesAutomatically Defangs outputMISP Support (example added to machinae.yml)Version 1.4.0 (2016-04-20)New features"-a"/"--auth" option for passing an auth config fileThanks johannestaas for the submission"-H"/"--http-proxy" option, and environment support, for HTTP proxiesNew sitesPassivetotal (various forms, thanks johannestaas)MaxMindFraudGuardShodanUpdated sitesFreeGeoIP (replaced freegeoip.net with freegeoip.io)Version 1.3.4 (2016-04-01)Bug fixesConvert exceptions to str when outputting to JSONShould actually close #14Version 1.3.3 (2016-03-28)Bug fixesCorrectly handle error results when outputting to JSONCloses #14Thanks Den1al for the bug reportVersion 1.3.2 (2016-03-10)New features"Short" output mode - simply output yes/no/error for each site"-i"/"--infile" option for passing a file with list of targetsVersion 1.3.1 (2016-03-08)New featuresPrepend "http://" to URL targets when not starting with http:// or https://Version 1.3.0 (2016-03-07)New sitesCymon.io - Threat intel aggregator/tracker by eSentireNew featuresSupport simple paginated responsesSupport url encoding 'target' in request URLSupport url decoding values in resultsVersion 1.2.0 (2016-02-16)New featuresSupport for sites returning multiple JSON documentsAbility to specify time format for relative time parametersAbility to parse Unix timestamps in results and display in ISO-8601 formatAbility to specify status codes to ignore per-APINew sitesDNSDB - FarSight Security Passive DNS Data base (premium)Version 1.1.2 (2015-11-26)New sitesTelize (premium) - GeoIP site (premium)Freegeoip - GeoIP site (free)CIF - CIFv2 API support, from csirtgadgets.orgNew featuresAbility to specify labels for single-line multimatch JSON outputsAbility to specify relative time parameters using relatime libraryVersion 1.0.1 (2015-10-13)Fixed a false-positive bug with Spamhaus (Github#10)Version 1.0.0 (2015-07-02)Initial releaseLicense InfoThe MIT License (MIT)Copyright (c) 2015 Hurricane Labs LLCPermission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.