Package: yager
Yager = Yet Another GEneric Reporteryageris Yet Another GEneric Reporter tool for parsing of XML data into an SQLite database and subsequent universal reporting based on SQL queries and Jinja2 templates.Getting StartedInstallingyageris distributed through thePython Package Indexasyager. Run the following command to:install a specific versionpipinstall"yager==0.1"install the latest versionpipinstall"yager"upgrade to the latest versionpipinstall--upgrade"yager"install optional DEV dependencies likepytestframework and plugins necessary for performance and functional testingpipinstall"yager[test]"Configuringyagerlooks for aYAMLconfiguration file in the following locations:/etc/yager/yager.yaml~/.config/yager/yager.yaml~/.yager/config/yager.yaml~/.yager.yamlBelow is theexample configuration filethat parses XML data fromQualys Cloud Agent APIabout Azure VMs and generates reports for each Azure Subscription.### Yager Configuration Settings---yager:# Toggle application level debug (does not toggle Cement framework debugging)# debug: true# Data location and layoutdata:# URI to the SQLite3 database with Qualys datadb_uri:"file:./instance/data/results.db"# Tables exclude from deletion during `refresh-db`exclude_from_refresh:["QualysKB",]# List of table definitionslayout:# Table name-name:AzureSubscriptions# Table columns described as an SQL statement;# Used in `CREATE TABLE IF NOT EXISTS {name} ({columns})`columns:|id TEXT PRIMARY KEY,name TEXT,contactName TEXT,contactEmail TEXT# Source of data for the table in a form of `{type}:{locator}`#   {type} could be `csv` or `xml`#   {locator} is a file path for `csv` type or XPath for `xml`#   XML files are provided with `--file path/to/file.xml` optiondata_source:csv:./instance/subscriptions.csv-name:HostAssetscolumns:|id INTEGER PRIMARY KEY,name TEXT,fqdn TEXT,os TEXT,firstSeen DATETIME,lastUpdated DATETIME,lastVulnScan DATETIME,azureVmId TEXT,azureRgName TEXT,azurePublicIp TEXT,azurePrivateIp TEXT,azureSubscriptionId TEXT,FOREIGN KEY (azureSubscriptionId)REFERENCES AzureSubscriptions (id)data_source:xml:.//HostAsset# Mapping between table columns and XPath to text values# of sub-elements for each element returned by `data_source` XPathinput_map:id:idname:namefqdn:fqdnos:osfirstSeen:createdlastUpdated:modifiedlastVulnScan:lastVulnScanazureVmId:./sourceInfo/list/AzureAssetSourceSimple/vmIdazureRgName:./sourceInfo/list/AzureAssetSourceSimple/resourceGroupNameazurePublicIp:./sourceInfo/list/AzureAssetSourceSimple/publicIpAddressazurePrivateIp:./sourceInfo/list/AzureAssetSourceSimple/privateIpAddressazureSubscriptionId:./sourceInfo/list/AzureAssetSourceSimple/subscriptionId-name:Vulnscolumns:|id INTEGER PRIMARY KEY,qid INTEGER,firstFound DATETIME,lastFound DATETIME,hostAssetsId INTEGER,FOREIGN KEY (hostAssetsId)REFERENCES HostAssets (id)data_source:xml:.//HostAssetVulninput_map:id:hostInstanceVulnIdqid:qidfirstFound:firstFoundlastFound:lastFound# Similar to `input_map` but mapping is parametrized with any {table_column}# already defined in the `input_map` section. Prior to running XPath match,# each {table_column} is expanded with a value already acquired for `input_map`.input_map_parametrized:hostAssetsId:".//HostAssetVuln/[hostInstanceVulnId='{id}']/.../.../.../id"# Path to directory with Jinja2 templates for reportstemplate_dir:"./config/templates/"# Templated reportsreports:-name:"single_sub_md"description:|Report provides detailed Qualys vulnerability data for an individual Azure subscription.It expects the following parameters to be provided in CLI:* `sub_id` - Subscription ID from `AzureSubscriptions` tabletemplate_file:"report_single_sub_md.j2"template_params:-query:|-- Subscription detailsSELECTid,name,contactName,contactEmailFROMAzureSubscriptionsWHEREid == '{sub_id}'var_mapping:subscription_id:idsubscription_name:namesubscription_poc_name:contactNamesubscription_poc_email:contactEmail-query:|-- Vuln details for all hosts in subscriptionSELECTHostAssets.name as host_name,HostAssets.azureVmId as host_vm_id,Vulns.firstFound as vuln_seen_first,Vulns.lastFound as vuln_seen_last,Vulns.qid as vuln_qid,QualysKB.severity_level as vuln_levelFROMHostAssetsINNER JOIN Vulns ON Vulns.hostAssetsId = HostAssets.idINNER JOIN QualysKB ON QualysKB.qid = Vulns.qidWHEREHostAssets.azureSubscriptionId == '{sub_id}'AND QualysKB.vuln_type != "Information Gathered";var_mapping:all_vuln_details:"*"-query:|-- Additional host detailsSELECTHostAssets.name as host_name,HostAssets.fqdn as host_fqdn,HostAssets.os as host_os,HostAssets.lastVulnScan as host_last_scan,HostAssets.azureRgName as host_rg,HostAssets.azureVmId as host_vm_id,HostAssets.azurePrivateIp as host_priv_ip,HostAssets.azurePublicIp as host_pub_ipFROMHostAssetsWHEREHostAssets.azureSubscriptionId == '{sub_id}'var_mapping:all_host_details:"*"-query:|-- Reference information from Qualys Knowledge BaseSELECTQualysKB.qid as vuln_qid,QualysKB.severity_level as vuln_level,QualysKB.title as vuln_title,QualysKB.diagnosis as vuln_details,QualysKB.consequence as vuln_risk,QualysKB.solution as vuln_solutionFROMQualysKBvar_mapping:kb_details:"*"# Logging configurationlog.colorlog:# Whether or not to colorize the log file.# colorize_file_log: false# Whether or not to colorize the console log.# colorize_console_log: true# Where the log file lives (no log file by default)# file: null# The level for which to log.  One of: info, warning, error, fatal, debug# level: INFO# Whether or not to log to console# to_console: true# Whether or not to rotate the log file when it reaches `max_bytes`# rotate: false# Max size in bytes that a log file can grow until it is rotated.# max_bytes: 512000# The maximum number of log files to maintain when rotating# max_files: 4UsageGenericusage: yager [-h] [-d] [-q] [-v] {query,refresh-db,report} ...

Yet Another GEneric Reporter tool for parsing of XML data into an SQLite database and subsequent universal reporting based on SQL queries and Jinja2 templates.

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           full application debug mode
  -q, --quiet           suppress all console output
  -v, --version         show program's version number and exit

sub-commands:
  {query,refresh-db,report}
    query               execute a query against database
    refresh-db          generate database from configured data sources
    report              execute a pre-configured report

Usage: yager {sub-command} {options}Queryusage: yager query [-h] [--output FORMAT] QUERY

positional arguments:
  QUERY                 query to be executed

optional arguments:
  -h, --help            show this help message and exit
  --output FORMAT, -o FORMAT
                        defines how to format the output (choose from 'csv' or
                        'table'; default: 'table')Refresh-dbusage: yager refresh-db [-h] [--file PATH]

optional arguments:
  -h, --help            show this help message and exit
  --file PATH, -f PATH  path to XML file with data (could be repetated)Reportusage: yager report [-h] [--param PARAM] NAME

positional arguments:
  NAME                  report to be executed

optional arguments:
  -h, --help            show this help message and exit
  --param PARAM, -p PARAM
                        NAME=VALUE pair defining a global query parameter to
                        be used for each query in the template (could be
                        repetated)ExamplesRefresh the database using the layout and data sources described in the YAML config.$yagerrefresh-db-fdata.xmlGenerate report based on SQL queries and Jinja2 templates defined in the YAML config.$yagerreportTEMPLATE--paramKEY1=VALUE1--paramKEY2=VALUE2RequirementsPython >= 3.7Built usingCement Framework- CLI application frameworkVersioningWe useSemantic Versioning Specificationas a version numbering convention.Release HistoryFor the available versions, see thetags on this repository. Specific changes for each version are documented inCHANGELOG.md.Also, conventions forgit commitmessages are documented inCONTRIBUTING.md.AuthorsOleksiy Kuzmenko-OK-UNDP@GitHub-Initial design and implementationAcknowledgmentsHat tip to all individuals shaping design of this project by sharing their knowledge in articles, blogs and forums.LicenseUnless otherwise stated, all authors (see commit logs) release their work under theMIT License. SeeLICENSE.mdfor details.ContributingThere are plenty of ways you could contribute to this project. Feel free to:submit bug reports and feature requestsoutline, fix and expand documentationpeer-review bug reports and pull requestsimplement new features or fix bugsSeeCONTRIBUTING.mdfor details on code formatting, linting and testing frameworks used by this project.