Package: ace-metrics
Metrics for the ACE EcosystemThis library is intended to help with the calculation and management of metrics as they pertain to any data produced by the ACE ecosystem. As of now, the library meets a base set of use cases for answering questions that arose during the initial organic operational development that took place around ACE's development. So it's certainly not the end-all-be-all but hopefully, it can serve as a base for further statistical analysis, metricization, analytics, and real-time presentations.Updates12/11/2020: Graphing functions were added, as well as, some new graphs for alerts, users, alert type, and events.Alert MetricsThe following statistics are available for alert based metrics and are used over and over again.>>>frommetrics.alertsimportVALID_ALERT_STATS>>>frommetrics.alertsimportFRIENDLY_STAT_NAME_MAP>>>>>>forstatinVALID_ALERT_STATS:...print(FRIENDLY_STAT_NAME_MAP[stat])...TotalOpenTimeAverageTimetoDispositionQuickestDispositionSlowestDispositionStandardDeviationforTimetoDispositionAlertQuantitiesbyDispositionAs of now, all of the above statistics are calculated by month and disposition. These stats can be calculated on any group of ACE alerts passed in a pandas dataframe. Themetrics.alerts.get_alerts_between_datesfunction can be used to get a group of alerts you're interested in.As of now, functionality exists to view any or all of these alert based stats from the viewpoint of users and alert types. All this means is that alerts are filtered down to only the alerts that apply during a given date range for respective alert types and/or users and then the VALID_ALERT_STATS are calculated for that set of alerts. Functionality related to viewing these stats as they relate to users and alert types are store in respective files in themetrics/alertsdirectory.The following independent alert based tables are also available:Operating Hours Summary:Essentially, a high-level view of how an ACE based team is performing during the three different categories of operational time. From the function definition:Summarize the overall cycle-time averages, the standard deviation in cycle times,
and the alert quantities observed over the alerts passed when organized by month 
and placed in the respective operating hour category. Categories are:
  business hours, weekends, weeknightsOverall Alert Cycle-Time Summary:Generate an overall statistical summary for alerts by month. Similar to the hours of operation summary but not broken up into time categories. From the function definition:Organize alerts by month and then summarize the business hour and real hour
cycle-time averages, the standard deviation in the business hour and real
hour cycle-times, and the alert quantities observed over the alerts passed.Total Alert Type Quantities:A straight up count of alerts by type, between two dates.Business HoursFor time-based statistics, you can also calculate based on business hours. When business hours are applied, only the time inside of business hours is counted when calculating time-based statistics.
You can define business hours as needed withmetrics.alerts.define_business_time().The default business start hour is 0600 hours, the default business end hour is 1800 hours. The default time zone is US/Eastern. Default holidays are defined atmetrics.alerts.SiteHolidays. Holidays are the only defaults that currently can't be overriden. The GUI uses theSLAconfig section to override start hour, end hour, and timezone.Expectmetric generation to take about ten times as long when applying business hours. Every alert time field has to be modified before the stats are calculated.Event MetricsCurrently, an event and incident table summary is available. For the GUI and CLI, a count of emails per event is appended to each event, but this is a separate function call that's not necessary if you don't care about emails per event/incident.CLIAccess to this library is made available on the CLI atace metrics.Command Line Instructionsusage: ace metrics [-h] [-so {json,csv,ascii_table,print}] [-fo {json,xlsx}]
                   [-f FILENAME] [-c COMPANIES] [-bh BUSINESS_HOURS]
                   [-s START_DATETIME] [-e END_DATETIME]
                   {alerts,events} ...

positional arguments:
  {alerts,events}
    alerts              alert based metrics
    events              event based metrics. With no arguments will return all
                        events

optional arguments:
  -h, --help            show this help message and exit
  -so {json,csv,ascii_table,print}, --stdout-format {json,csv,ascii_table,print}
                        desired standard output format. ~~~ NOTE: 'print' (the
                        default) will also summarize large tables. Use
                        'ascii_table' to avoide that.
  -fo {json,xlsx}, --fileout-format {json,xlsx}
                        desired file output format. Default is xls.
  -f FILENAME, --filename FILENAME
                        The name of a file to write results to.
  -c COMPANIES, --company COMPANIES
                        A list of company names to gather metrics for. Default
                        is all defined companies.
  -bh BUSINESS_HOURS, --business-hours BUSINESS_HOURS
                        Use business hours for all time based stats. Set like
                        start_hour,end_hour,time_zone. Example:
                        6,18,US/Eastern
  -s START_DATETIME, --start_datetime START_DATETIME
                        The start datetime data is in scope. Format: YYYY-MM-
                        DD HH:MM:SS. Default: 7 days ago.
  -e END_DATETIME, --end_datetime END_DATETIME
                        The end datetime data is in scope. Format: YYYY-MM-DD
                        HH:MM:SS. Default: now.ExamplesGet the overall alert counts, by disposition, from '2020-06-01 00:00:00' to now. Print the results to an ASCII table.ace metrics -so ascii_table -s '2020-06-01 00:00:00' alerts --alert_countSame thing but return the json representation.ace metrics -so json -s '2020-06-01 00:00:00' alerts --alert_count | jq '.'Generate all alert based statistic tables for the user 'jdoe', and print the results as ASCII tables.ace metrics -so ascii_table -s '2020-06-01 00:00:00' alerts users -u jdoe --all-statsThe following command will output the calculated metrics in json. Business hours will be applied to time-based calculations and all statistics for alerts by disposition and month between '2020-06-01 00:00:00' and now will be calculated. Additionally, all alert statistics for the alert type 'mailbox' will be calculated during the same date range with business hours applied.ace metrics -fo json  -bh 6,18,US/Eastern -s '2020-06-01 00:00:00' alerts --all-stats types -t mailbox --all-statsGUIAlmost all of the metrics that are available on the CLI are available through the GUI with a few exceptions.User MetricsBy default, users only have access to their metrics. Users needing access to other users statistics, through the GUI, can be added to the following configuration item:$aceconfiggui.full_metric_access[gui]full_metric_access=1,3Thefull_metric_accessconfig item expects a comma separated list of user IDs that can get all stats through the GUI.Exporting MetricsFrom the GUI, you can export metrics to an XLSX spreadsheet or to JSON documents.XLSX exportBecause of limitations with a popular XLSX application, the names of data tables are heavily sanitized before being written to tabs on the resulting XLSX sheet.
As a result of this, the first tab on every XLSX spreadsheet will be a table that shows the mapping from tab names to original pivot table names.JSONWhen JSON export is selected, all tables are converted to JSON, and added to a tar.gz archive. Names are mostly preserved but special characters that can cause problems when used in filenames are replaced with '-'.CompaniesThe ability to select ACE data, where the data belongs to a specific company, is only made available if more than one company is defined.Library StructureCurrently, the project is structured so that every directory in the root$SAQ_HOME\metricsdirectory is an ACE database table, data source, or data target. As of now, metrics are calculated around theace.alertsandace.eventsdatabase tables and their relevant mapping tables. Any additional functionality should follow this structure. For instance, there is a desire to add metrics aroundace.observables, which will happen and will likely be placed in anobservablesdirectory. Another example would be writing functionality that injects and tracks$SAQ_HOME\data\stats\modulesstatistics to be made available in this metrics lib. Such functionality should go into a new directory, as well.Goals, Enhancement Ideas, To-doCreate a central location, in the ACE repo, for metrics to be managed in a modular and flexible way.Make metrics a library, on the side of, and not directly interconnected with the ACE codebase.Take metric calculations out of the GUI and have the GUI use the metrics library.Make all metrics available via the CLI.Make all or a subset of metrics persistableAutomatically update the metric pivot tables as the ACE database tables are updated. MySQL database replication looks like a viable solution. This library appears promising:python-mysql-replication.One way to do this would be to create a daemon service that can run and continuously update the pivot tables. This will allow for near real-time access to the data and also lay a groundwork for real-time graphical metrics to eventually populate a dashboard.The pivot table metrics could run in memory in something like REDIS, as long as the metrics service is running. AND/OR.. a time-series database.. or explore more options.TO-DOImplement pytest based testing.Make Holidays configurable.Should config items be seperate from ACE or configurable in ACE?Add observable statistics.Distinguish detection points? yara, indicator, etc.