Package: openwisp-monitoring
OpenWISP Monitoring is a network monitoring system written in Python and Django,
designed to beextensible,programmable,scalableand easy to use by end users:
once the system is configured, monitoring checks, alerts and metric collection
happens automatically.See theavailable features.OpenWISPis not only an application designed for end users,
but can also be used as a framework on which custom network automation solutions can be
built on top of its building blocks.Other popular building blocks that are part of the OpenWISP ecosystem are:openwisp-controller:
network and WiFi controller: provisioning, configuration management,
x509 PKI management and more; works on OpenWRT, but designed to work also on other systems.openwisp-network-topology:
provides way to collect and visualize network topology data from
dynamic mesh routing daemons or other network software (eg: OpenVPN);
it can be used in conjunction with openwisp-monitoring to get a better idea
of the state of the networkopenwisp-firmware-upgrader:
automated firmware upgrades (single device or mass network upgrades)openwisp-radius:
based on FreeRADIUS, allows to implement network access authentication systems like
802.1x WPA2 Enterprise, captive portal authentication, Hotspot 2.0 (802.11u)openwisp-ipam:
it allows to manage the IP address space of networksFor a more complete overview of the OpenWISP modules and architecture,
see theOpenWISP Architecture Overview.Available FeaturesCollection of monitoring information in a timeseries database (currently only influxdb is supported)Allows to browse alerts easily from the user interface with one clickCollects and displaysdevice statusinformation like
uptime, RAM status, CPU load averages,
Interface properties and addresses, WiFi interface status and associated clients,
Neighbors information, DHCP Leases, Disk/Flash statusMonitoring charts foruptime,packet loss,round trip time (latency),associated wifi clients,interface traffic,RAM usage,CPU load,flash/disk usage,
mobile signal (LTE/UMTS/GSMsignal strength,signal quality,access technology in use)Charts can be viewed at resolutions of 1 day, 3 days, a week, a month and a yearConfigurable alertsCSV Export of monitoring dataAn overview of the status of the network is shown in the admin dashboard,
a chart shows the percentages of devices which are online, offline or having issues;
a geographic map is also available for those who use the geographic features of OpenWISPPossibility to configure additionalMetricsandChartsExtensible active check system: it’s possible to write additional checks that
are run periodically using python classesExtensible metrics and charts: it’s possible to define new metrics and new chartsAPI to retrieve the chart metrics and status information of each device
based onNetJSON DeviceMonitoringTable of Contents:Available FeaturesInstallation instructionsDeploy it in productionInstall system dependenciesInstall stable version from PyPIInstall development versionInstalling for developmentInstall and run on dockerSetup (integrate in an existing Django project)Quickstart GuideInstall OpenWISP MonitoringInstall openwisp-config on the deviceInstall monitoring packages on the deviceMake sure OpenWISP can reach your devices1. Public internet deployment2. LAN deploymentCreating checks for a devicePassive vs Active Metric CollectionDevice Health StatusUNKNOWNOKPROBLEMCRITICALDefault MetricsDevice StatusPingTrafficWiFi ClientsMemory UsageCPU LoadDisk UsageMobile Signal StrengthMobile Signal QualityMobile Access Technology in useDefault Alerts / NotificationsAvailable ChecksPingConfiguration appliedSettingsOPENWISP_MONITORING_SHORT_RETENTION_POLICYOPENWISP_MONITORING_AUTO_PINGOPENWISP_MONITORING_PING_CHECK_CONFIGOPENWISP_MONITORING_AUTO_DEVICE_CONFIG_CHECKOPENWISP_MONITORING_CONFIG_CHECK_INTERVALOPENWISP_MONITORING_AUTO_CHARTSOPENWISP_MONITORING_CRITICAL_DEVICE_METRICSOPENWISP_MONITORING_HEALTH_STATUS_LABELSOPENWISP_MONITORING_MANAGEMENT_IP_ONLYOPENWISP_MONITORING_DEVICE_RECOVERY_DETECTIONOPENWISP_MONITORING_MAC_VENDOR_DETECTIONOPENWISP_MONITORING_WRITE_RETRY_OPTIONSOPENWISP_MONITORING_TIMESERIES_RETRY_OPTIONSOPENWISP_MONITORING_TIMESERIES_RETRY_DELAYOPENWISP_MONITORING_DASHBOARD_MAPOPENWISP_MONITORING_METRICSOPENWISP_MONITORING_CHARTSOPENWISP_MONITORING_AUTO_CLEAR_MANAGEMENT_IPOPENWISP_MONITORING_API_URLCONFOPENWISP_MONITORING_API_BASEURLRegistering / Unregistering Metric Configurationregister_metricunregister_metricRegistering / Unregistering Chart Configurationregister_chartunregister_chartRegistering new notification typesExceptionsTimeseriesWriteExceptionInvalidMetricConfigExceptionInvalidChartConfigExceptionRest APILive documentationBrowsable web interfaceList of endpointsRetrieve device charts and device status dataCollect device metrics and statusSignalsdevice_metrics_receivedhealth_status_changedthreshold_crossedpre_metric_writepost_metric_writeManagement commandsrun_checksmigrate_timeseriesMonitoring scriptsMigrating from monitoring scripts to monitoring packagesExtending openwisp-monitoring1. Initialize your custom module2. Installopenwisp-monitoring3. AddEXTENDED_APPS4. Addopenwisp_utils.staticfiles.DependencyFinder5. Addopenwisp_utils.loaders.DependencyLoader6. Inherit the AppConfig class7. Create your custom models8. Add swapper configurations9. Create database migrations10. Create your custom admin1. Monkey patching2. Inheriting admin classes11. Create root URL configuration12. Create celery.py13. Import Celery Tasks14. Create the custom commandrun_checks15. Import the automated testsOther base classes that can be inherited and extendedDeviceMetricViewContributingInstallation instructionsDeploy it in productionSee:ansible-openwisp2docker-openwispInstall system dependenciesopenwisp-monitoringuses InfluxDB to store metrics. Follow theinstallation instructions from InfluxDB’s official documentation.Note:OnlyInfluxDB 1.8.xis supported inopenwisp-monitoring.Install system packages:sudoaptinstall-yopenssllibssl-dev\gdal-binlibproj-devlibgeos-dev\fpingInstall stable version from PyPIInstall from PyPI:pipinstallopenwisp-monitoringInstall development versionInstall tarball:pipinstallhttps://github.com/openwisp/openwisp-monitoring/tarball/masterAlternatively, you can install via pip using git:pipinstall-egit+git://github.com/openwisp/openwisp-monitoring#egg=openwisp_monitoringIf you want to contribute, follow the instructions in“Installing for development”section.Installing for developmentInstall the system dependencies as mentioned in the“Install system dependencies”section.
Install these additional packages that are required for development:sudoaptinstall-ysqlite3libsqlite3-dev\libspatialite-devlibsqlite3-mod-spatialite\chromiumFork and clone the forked repository:gitclonegit://github.com/<your_fork>/openwisp-monitoringNavigate into the cloned repository:cdopenwisp-monitoring/Start Redis and InfluxDB using Docker:docker-composeup-dredisinfluxdbSetup and activate a virtual-environment. (we’ll be usingvirtualenv)python-mvirtualenvenvsourceenv/bin/activateMake sure that you are using pip version 20.2.4 before moving to the next step:pipinstall-UpipwheelsetuptoolsInstall development dependencies:pipinstall-e.pipinstall-rrequirements-test.txtnpminstall-gjshintstylelintInstall WebDriver for Chromium for your browser version fromhttps://chromedriver.chromium.org/homeand extractchromedriverto one of directories from your$PATH(example:~/.local/bin/).Create database:cdtests/./manage.pymigrate./manage.pycreatesuperuserRun celery and celery-beat with the following commands (separate terminal windows are needed):cdtests/celery-Aopenwisp2worker-linfocelery-Aopenwisp2beat-linfoLaunch development server:./manage.pyrunserver0.0.0.0:8000You can access the admin interface athttp://127.0.0.1:8000/admin/.Run tests with:./runtests.py--parallelRun quality assurance tests with:./run-qa-checksInstall and run on dockerNote: This Docker image is for development purposes only.
For the official OpenWISP Docker images, see:docker-openwisp.Build from the Dockerfile:docker-composebuildRun the docker container:docker-composeupSetup (integrate in an existing Django project)Follow the setup instructions ofopenwisp-controller, then add the settings described below.INSTALLED_APPS=[# django apps# all-auth'django.contrib.sites','allauth','allauth.account','allauth.socialaccount','django_extensions','django_filters',# openwisp2 modules'openwisp_users','openwisp_controller.pki','openwisp_controller.config','openwisp_controller.connection','openwisp_controller.geo',# monitoring'openwisp_monitoring.monitoring','openwisp_monitoring.device','openwisp_monitoring.check','nested_admin',# notifications'openwisp_notifications',# openwisp2 admin theme (must be loaded here)'openwisp_utils.admin_theme',# admin'django.contrib.admin','django.forms',# other dependencies ...]# Make sure you change them in production# You can select one of the backends located in openwisp_monitoring.db.backendsTIMESERIES_DATABASE={'BACKEND':'openwisp_monitoring.db.backends.influxdb','USER':'openwisp','PASSWORD':'openwisp','NAME':'openwisp2','HOST':'localhost','PORT':'8086',}urls.py:fromdjango.confimportsettingsfromdjango.conf.urlsimportinclude,urlfromdjango.contrib.staticfiles.urlsimportstaticfiles_urlpatternsfromopenwisp_utils.admin_theme.adminimportadmin,openwisp_adminopenwisp_admin()urlpatterns=[url(r'^admin/',include(admin.site.urls)),url(r'',include('openwisp_controller.urls')),url(r'',include('openwisp_monitoring.urls')),]urlpatterns+=staticfiles_urlpatterns()Configure caching (you may use a different cache storage if you want):CACHES={'default':{'BACKEND':'django_redis.cache.RedisCache','LOCATION':'redis://localhost/0','OPTIONS':{'CLIENT_CLASS':'django_redis.client.DefaultClient',}}}SESSION_ENGINE='django.contrib.sessions.backends.cache'SESSION_CACHE_ALIAS='default'Configure celery (you may use a different broker if you want):# here we show how to configure celery with redis but you can# use other brokers if you want, consult the celery docsCELERY_BROKER_URL='redis://localhost/1'CELERY_BEAT_SCHEDULE={'run_checks':{'task':'openwisp_monitoring.check.tasks.run_checks','schedule':timedelta(minutes=5),},}INSTALLED_APPS.append('djcelery_email')EMAIL_BACKEND='djcelery_email.backends.CeleryEmailBackend'If you decide to use Redis (as shown in these examples),
install the following python packages.pipinstallredisdjango-redisQuickstart GuideInstall OpenWISP MonitoringInstallOpenWISP Monitoringusing one of the methods mentioned in the“Installation instructions”.Install openwisp-config on the deviceInstall the openwisp-config agent for OpenWrton your device.Install monitoring packages on the deviceInstall the openwrt-openwisp-monitoring packageson your device.These packages collect and send the
monitoring data from the device to OpenWISP Monitoring and
are required to collectmetricslike interface traffic, WiFi clients, CPU load, memory usage, etc.Note: if you are an existing user ofopenwisp-monitoringand are using
the legacymonitoring templatefor collecting metrics, we highly recommendMigrating from monitoring scripts to monitoring packages.Make sure OpenWISP can reach your devicesIn order to performactive checksand other actions liketriggering the push of configuration changes,executing shell commandsorperforming firmware upgrades,the OpenWISP server needs to be able to reach the network devices.There are mainly two deployment scenarios for OpenWISP:the OpenWISP server is deployed on the public internet and the devices are
geographically distributed across different locations:in this case a management tunnel is neededthe OpenWISP server is deployed on a computer/server which is located in
the same Layer 2 network (that is, in the same LAN) where the devices
are located.in this case a management tunnel is NOT needed1. Public internet deploymentThis is the most common scenario:the OpenWISP server is deployed to the public internet, hence the
server has a public IPv4 (and IPv6) address and usually a valid
SSL certificate provided by Mozilla Letsencrypt or another SSL providerthe network devices are geographically distributed across different
locations (different cities, different regions, different countries)In this scenario, the OpenWISP application will not be able to reach the
devicesunless a management tunnelis used, for that reason having
a management VPN like OpenVPN, Wireguard or any other tunneling solution
is paramount, not only to allow OpenWISP to work properly, but also to
be able to perform debugging and troubleshooting when needed.In this scenario, the following requirements are needed:a VPN server must be installed in a way that the OpenWISP
server can reach the VPN peers, for more information on how to do this
via OpenWISP please refer to the following sections:OpenVPN tunnel automationWireguard tunnel automationIf you prefer to use other tunneling solutions (L2TP, Softether, etc.)
and know how to configure those solutions on your own,
that’s totally fine as well.If the OpenWISP server is connected to a network infrastructure
which allows it to reach the devices via pre-existing tunneling or
Intranet solutions (eg: MPLS, SD-WAN), then setting up a VPN server
is not needed, as long as there’s a dedicated interface on OpenWrt
which gets an IP address assigned to it and which is reachable from
the OpenWISP server.The devices must be configured to join the management tunnel automatically,
either via a pre-existing configuration in the firmware or via anOpenWISP Template.Theopenwisp-configagent on the devices must be configured to specify
themanagement_interfaceoption, the agent will communicate the
IP of the management interface to the OpenWISP Server and OpenWISP will
use the management IP for reaching the device.For example, if themanagement interfaceis namedtun0,
the openwisp-config configuration should look like the following example:# In /etc/config/openwisp on the device

config controller 'http'
    # ... other configuration directives ...
    option management_interface 'tun0'2. LAN deploymentWhen the OpenWISP server and the network devices are deployed in the same
L2 network (eg: an office LAN) and the OpenWISP server is reachable
on the LAN address, OpenWISP can then use theLast IPfield of the
devices to reach them.In this scenario it’s necessary to set the“OPENWISP_MONITORING_MANAGEMENT_IP_ONLY”setting toFalse.Creating checks for a deviceBy default, theactive checksare created
automatically for all devices, unless the automatic creation of some
specific checks has been disabled, for more information on how to do this,
refer to theactive checkssection.These checks are created and executed in the background by celery workers.Passive vs Active Metric CollectionThethe different device metriccollected by OpenWISP Monitoring can be divided in two categories:metrics collected actively by OpenWISP:
these metrics are collected by the celery workers running on the
OpenWISP server, which continuously sends network requests to the
devices and store the results;metrics collected passively by OpenWISP:
these metrics are sent by theopenwrt-openwisp-monitoring agentinstalled on the network devices and are collected by OpenWISP via
its REST API.The“Available Checks”section of this document
lists the currently implementedactive checks.Device Health StatusThe possible values for the health status field (DeviceMonitoring.status)
are explained below.UNKNOWNWhenever a new device is created it will haveUNKNOWNas it’s default Heath Status.It implies that the system doesn’t know whether the device is reachable yet.OKEverything is working normally.PROBLEMOne of the metrics has a value which is not in the expected range
(the threshold value set in the alert settings has been crossed).Example: CPU usage should be less than 90% but current value is at 95%.CRITICALOne of the metrics defined inOPENWISP_MONITORING_CRITICAL_DEVICE_METRICShas a value which is not in the expected range
(the threshold value set in the alert settings has been crossed).Example: ping is by default a critical metric which is expected to be always 1
(reachable).Default MetricsDevice StatusThis metric stores the status of the device for viewing purposes.Pingmeasurement:pingtypes:int(reachable and loss),float(rtt)fields:reachable,loss,rtt_min,rtt_max,rtt_avgconfiguration:pingcharts:uptime,packet_loss,rttUptime:Packet loss:Round Trip Time:Trafficmeasurement:traffictype:intfields:rx_bytes,tx_bytestags:{'organization_id':'<organization-id-of-the-related-device>','ifname':'<interface-name>',# optional'location_id':'<location-id-of-the-related-device-if-present>','floorplan_id':'<floorplan-id-of-the-related-device-if-present>',}configuration:trafficcharts:trafficWiFi Clientsmeasurement:wifi_clientstype:intfields:clientstags:{'organization_id':'<organization-id-of-the-related-device>','ifname':'<interface-name>',# optional'location_id':'<location-id-of-the-related-device-if-present>','floorplan_id':'<floorplan-id-of-the-related-device-if-present>',}configuration:clientscharts:wifi_clientsMemory Usagemeasurement:<memory>type:floatfields:percent_used,free_memory,total_memory,buffered_memory,shared_memory,cached_memory,available_memoryconfiguration:memorycharts:memoryCPU Loadmeasurement:loadtype:floatfields:cpu_usage,load_1,load_5,load_15configuration:loadcharts:loadDisk Usagemeasurement:disktype:floatfields:used_diskconfiguration:diskcharts:diskMobile Signal Strengthmeasurement:signal_strengthtype:floatfields:signal_strength,signal_powerconfiguration:signal_strengthcharts:signal_strengthMobile Signal Qualitymeasurement:signal_qualitytype:floatfields:signal_quality,signal_qualityconfiguration:signal_qualitycharts:signal_qualityMobile Access Technology in usemeasurement:access_techtype:intfields:access_techconfiguration:access_techcharts:access_techDefault Alerts / NotificationsNotification TypeUsethreshold_crossedFires when a metric crosses the boundary defined in the
threshold value of the alert settings.threshold_recoveryFires when a metric goes back within the expected range.connection_is_workingFires when the connection to a device is working.connection_is_not_workingFires when the connection (eg: SSH) to a device stops working
(eg: credentials are outdated, management IP address is
outdated, or device is not reachable).Available ChecksPingThis check returns information on deviceuptimeandRTT (Round trip time).
The Chartsuptime,packet lossandrttare created. Thefpingcommand is used to collect these metrics.
You may choose to disable auto creation of this check by settingOPENWISP_MONITORING_AUTO_PINGtoFalse.You can change the default values used for ping checks usingOPENWISP_MONITORING_PING_CHECK_CONFIGsetting.Configuration appliedThis check ensures that theopenwisp-config agentis running and applying configuration changes in a timely manner.
You may choose to disable auto creation of this check by using the
settingOPENWISP_MONITORING_AUTO_DEVICE_CONFIG_CHECK.This check runs periodically, but it is also triggered whenever the
configuration status of a device changes, this ensures the check reacts
quickly to events happening in the network and informs the user promptly
if there’s anything that is not working as intended.SettingsOPENWISP_MONITORING_SHORT_RETENTION_POLICYtype:strdefault:24h0m0sThe default retention policy used to store raw device data.This data is only used to assess the recent status of devices, keeping
it for a long time would not add much benefit and would cost a lot more
in terms of disk space.OPENWISP_MONITORING_AUTO_PINGtype:booldefault:TrueWhether ping checks are created automatically for devices.OPENWISP_MONITORING_PING_CHECK_CONFIGtype:dictdefault:{}This setting allows to override the default ping check configuration defined inopenwisp_monitoring.check.classes.ping.DEFAULT_PING_CHECK_CONFIG.For example, if you want to change only thetimeoutofpingyou can use:OPENWISP_MONITORING_PING_CHECK_CONFIG={'timeout':{'default':1000,},}If you are overriding the default value for any parameter
beyond the maximum or minimum value defined inopenwisp_monitoring.check.classes.ping.DEFAULT_PING_CHECK_CONFIG,
you will also need to override themaximumorminimumfields
as following:OPENWISP_MONITORING_PING_CHECK_CONFIG={'timeout':{'default':2000,'minimum':1500,'maximum':2500,},}Note:Abovemaximumandminimumvalues are only used for
validating custom parameters of aCheckobject.OPENWISP_MONITORING_AUTO_DEVICE_CONFIG_CHECKtype:booldefault:TrueThis setting allows you to choose whetherconfig_appliedchecks should be
created automatically for newly registered devices. It’s enabled by default.OPENWISP_MONITORING_CONFIG_CHECK_INTERVALtype:intdefault:5This setting allows you to configure the config check interval used byconfig_applied. By default it is set to 5 minutes.OPENWISP_MONITORING_AUTO_CHARTStype:listdefault:('traffic', 'wifi_clients', 'uptime', 'packet_loss', 'rtt')Automatically created charts.OPENWISP_MONITORING_CRITICAL_DEVICE_METRICStype:listofdictobjectsdefault:[{'key':'ping', 'field_name':'reachable'}]Device metrics that are considered critical:when a value crosses the boundary defined in the “threshold value” field
of the alert settings related to one of these metric types, the health status
of the device related to the metric moves intoCRITICAL.By default, if devices are not reachable by pings they are flagged asCRITICAL.OPENWISP_MONITORING_HEALTH_STATUS_LABELStype:dictdefault:{'unknown': 'unknown', 'ok': 'ok', 'problem': 'problem', 'critical': 'critical'}This setting allows to change the health status labels, for example, if we
want to useonlineinstead ofokandofflineinstead ofcritical,
you can use the following configuration:OPENWISP_MONITORING_HEALTH_STATUS_LABELS={'ok':'online','problem':'problem','critical':'offline'}OPENWISP_MONITORING_MANAGEMENT_IP_ONLYtype:booldefault:TrueBy default, only the management IP will be used to perform active checks to
the devices.If the devices are connecting to your OpenWISP instance using a shared layer2
network, hence the OpenWSP server can reach the devices using thelast_ipfield, you can set this toFalse.OPENWISP_MONITORING_DEVICE_RECOVERY_DETECTIONtype:booldefault:TrueWhen device recovery detection is enabled, recoveries are discovered as soon as
a device contacts the openwisp system again (eg: to get the configuration checksum
or to send monitoring metrics).This feature is enabled by default.If you use OpenVPN as the management VPN, you may want to check out a similar
integration built inopenwisp-network-topology: when the status of an OpenVPN link
changes (detected by monitoring the status information of OpenVPN), the
network topology module will trigger the monitoring checks.
For more information see:Network Topology Device IntegrationOPENWISP_MONITORING_MAC_VENDOR_DETECTIONtype:booldefault:TrueIndicates whether mac addresses will be complemented with hardware vendor
information by performing lookups on the OUI
(Organization Unique Identifier) table.This feature is enabled by default.OPENWISP_MONITORING_WRITE_RETRY_OPTIONStype:dictdefault:see below# default value of OPENWISP_MONITORING_RETRY_OPTIONS:dict(max_retries=None,retry_backoff=True,retry_backoff_max=600,retry_jitter=True,)Retry settings for recoverable failures during metric writes.By default if a metric write fails (eg: due to excessive load on timeseries database at that moment)
then the operation will be retried indefinitely with an exponential random backoff and a maximum delay of 10 minutes.This feature makes the monitoring system resilient to temporary outages and helps to prevent data loss.For more information regarding these settings, consult thecelery documentation
regarding automatic retries for known errors.OPENWISP_MONITORING_TIMESERIES_RETRY_OPTIONStype:dictdefault:see below# default value of OPENWISP_MONITORING_RETRY_OPTIONS:dict(max_retries=6,delay=2)On busy systems, communication with the timeseries DB can occasionally fail.
The timeseries DB backend will retry on any exception according to these settings.
The delay kicks in only after the third consecutive attempt.This setting shall not be confused withOPENWISP_MONITORING_WRITE_RETRY_OPTIONS,
which is used to configure the infinite retrying of the celery task which writes
metric data to the timeseries DB, whileOPENWISP_MONITORING_TIMESERIES_RETRY_OPTIONSdeals with any other read/write operation on the timeseries DB which may fail.However these retries are not handled by celery but are simple python loops,
which will eventually give up if a problem persists.OPENWISP_MONITORING_TIMESERIES_RETRY_DELAYtype:intdefault:2This settings allow you to configure the retry delay time (in seconds) after 3 failed attempt in timeseries database.This retry setting is used in retry mechanism to make the requests to the timeseries database resilient.This setting is independent of celery retry settings.OPENWISP_MONITORING_DASHBOARD_MAPtype:booldefault:TrueWhether the geographic map in the dashboard is enabled or not.
This feature provides a geographic map which shows the locations
which have devices installed in and provides a visual representation
of the monitoring status of the devices, this allows to get
an overview of the network at glance.This feature is enabled by default and depends on the settingOPENWISP_ADMIN_DASHBOARD_ENABLEDfromopenwisp-utilsbeing set toTrue(which is the default).You can turn this off if you do not use the geographic features
of OpenWISP.OPENWISP_MONITORING_METRICStype:dictdefault:{}This setting allows to define additional metric configuration or to override
the default metric configuration defined inopenwisp_monitoring.monitoring.configuration.DEFAULT_METRICS.For example, if you want to change only thefield_nameofclientsmetric towifi_clientsyou can use:fromdjango.utils.translationimportgettext_lazyas_OPENWISP_MONITORING_METRICS={'clients':{'label':_('WiFi clients'),'field_name':'wifi_clients',},}For example, if you want to change only the default alert settings ofmemorymetric you can use:OPENWISP_MONITORING_METRICS={'memory':{'alert_settings':{'threshold':75,'tolerance':10}},}For example, if you want to change only the notification ofconfig_appliedmetric you can use:fromdjango.utils.translationimportgettext_lazyas_OPENWISP_MONITORING_METRICS={'config_applied':{'notification':{'problem':{'verbose_name':'Configuration PROBLEM','verb':_('has not been applied'),'email_subject':_('[{site.name}] PROBLEM:{notification.target}configuration ''status issue'),'message':_('The configuration for device [{notification.target}]''({notification.target_link}){notification.verb}in a timely manner.'),},'recovery':{'verbose_name':'Configuration RECOVERY','verb':_('configuration has been applied again'),'email_subject':_('[{site.name}] RECOVERY:{notification.target}{notification.verb}''successfully'),'message':_('The device [{notification.target}]({notification.target_link}) ''{notification.verb}successfully.'),},},},}Or if you want to define a new metric configuration, which you can then
call in your custom code (eg: a custom check class), you can do so as follows:fromdjango.utils.translationimportgettext_lazyas_OPENWISP_MONITORING_METRICS={'top_fields_mean':{'name':'Top Fields Mean','key':'{key}','field_name':'{field_name}','label':'_(Top fields mean)','related_fields':['field1','field2','field3'],},}OPENWISP_MONITORING_CHARTStype:dictdefault:{}This setting allows to define additional charts or to override
the default chart configuration defined inopenwisp_monitoring.monitoring.configuration.DEFAULT_CHARTS.For example, if you want to change the traffic chart to show
MB (megabytes) instead of GB (Gigabytes) you can use:OPENWISP_MONITORING_CHARTS={'traffic':{'unit':' MB','description':('Network traffic, download and upload, measured on ''the interface "{metric.key}", measured in MB.'),'query':{'influxdb':("SELECT SUM(tx_bytes) / 1000000 AS upload, ""SUM(rx_bytes) / 1000000 AS download FROM{key}""WHERE time >= '{time}' AND content_type = '{content_type}' ""AND object_id = '{object_id}' GROUP BY time(1d)")},}}Or if you want to define a new chart configuration, which you can then
call in your custom code (eg: a custom check class), you can do so as follows:fromdjango.utils.translationimportgettext_lazyas_OPENWISP_MONITORING_CHARTS={'ram':{'type':'line','title':'RAM usage','description':'RAM usage','unit':'bytes','order':100,'query':{'influxdb':("SELECT MEAN(total) AS total, MEAN(free) AS free, ""MEAN(buffered) AS buffered FROM{key}WHERE time >= '{time}' AND ""content_type = '{content_type}' AND object_id = '{object_id}' ""GROUP BY time(1d)")},}}In case you just want to change the colors used in a chart here’s how to do it:OPENWISP_MONITORING_CHARTS={'traffic':{'colors':['#000000','#cccccc']}}OPENWISP_MONITORING_AUTO_CLEAR_MANAGEMENT_IPtype:booldefault:TrueThis setting allows you to automatically clear management_ip of a device
when it goes offline. It is enabled by default.OPENWISP_MONITORING_API_URLCONFtype:stringdefault:NoneChanges the urlconf option of django urls to point the monitoring API
urls to another installed module, example,myapp.urls.
(Useful when you have a seperate API instance.)OPENWISP_MONITORING_API_BASEURLtype:stringdefault:NoneIf you have a seperate server for API of openwisp-monitoring on a different
domain, you can use this option to change the base of the url, this will
enable you to point all the API urls to your openwisp-monitoring API server’s
domain, example:https://mymonitoring.myapp.com.Registering / Unregistering Metric ConfigurationOpenWISP Monitoringprovides registering and unregistering metric configuration through utility functionsopenwisp_monitoring.monitoring.configuration.register_metricandopenwisp_monitoring.monitoring.configuration.unregister_metric.
Using these functions you can register or unregister metric configurations from anywhere in your code.register_metricThis function is used to register a new metric configuration from anywhere in your code.ParameterDescriptionmetric_name:Astrdefining name of the metric configuration.metric_configuration:Adictdefining configuration of the metric.An example usage has been shown below.fromdjango.utils.translationimportgettext_lazyas_fromopenwisp_monitoring.monitoring.configurationimportregister_metric# Define configuration of your metricmetric_config={'label':_('Ping'),'name':'Ping','key':'ping','field_name':'reachable','related_fields':['loss','rtt_min','rtt_max','rtt_avg'],'charts':{'uptime':{'type':'bar','title':_('Uptime'),'description':_('A value of 100% means reachable, 0% means unreachable, values in ''between 0% and 100% indicate the average reachability in the ''period observed. Obtained with the fping linux program.'),'summary_labels':[_('Average uptime')],'unit':'%','order':200,'colorscale':{'max':100,'min':0,'label':_('Reachable'),'scale':[[[0,'#c13000'],[0.1,'cb7222'],[0.5,'#deed0e'],[0.9,'#7db201'],[1,'#498b26']],],'map':[[100,'#498b26',_('Reachable')],[90,'#7db201',_('Mostly Reachable')],[50,'#deed0e',_('Partly Reachable')],[10,'#cb7222',_('Mostly Unreachable')],[None,'#c13000',_('Unreachable')],],'fixed_value':100,},'query':chart_query['uptime'],},'packet_loss':{'type':'bar','title':_('Packet loss'),'description':_('Indicates the percentage of lost packets observed in ICMP probes. ''Obtained with the fping linux program.'),'summary_labels':[_('Average packet loss')],'unit':'%','colors':'#d62728','order':210,'query':chart_query['packet_loss'],},'rtt':{'type':'scatter','title':_('Round Trip Time'),'description':_('Round trip time observed in ICMP probes, measuered in milliseconds.'),'summary_labels':[_('Average RTT'),_('Average Max RTT'),_('Average Min RTT'),],'unit':_(' ms'),'order':220,'query':chart_query['rtt'],},},'alert_settings':{'operator':'<','threshold':1,'tolerance':0},'notification':{'problem':{'verbose_name':'Ping PROBLEM','verb':'cannot be reached anymore','level':'warning','email_subject':_('[{site.name}]{notification.target}is not reachable'),'message':_('The device [{notification.target}]{notification.verb}anymore by our ping ''messages.'),},'recovery':{'verbose_name':'Ping RECOVERY','verb':'has become reachable','level':'info','email_subject':_('[{site.name}]{notification.target}is reachable again'),'message':_('The device [{notification.target}]{notification.verb}again by our ping ''messages.'),},},}# Register your custom metric configurationregister_metric('ping',metric_config)The above example will register one metric configuration (namedping), three chart
configurations (namedrtt,packet_loss,uptime) as defined in thechartskey,
two notification types (namedping_recovery,ping_problem) as defined innotificationkey.TheAlertSettingsofpingmetric will by default usethresholdandtolerancedefined in thealert_settingskey.
You can always override them and define your own custom values via theadmin.Note: It will raiseImproperlyConfiguredexception if a metric configuration
is already registered with same name (not to be confused with verbose_name).If you don’t need to register a new metric but need to change a specific key of an
existing metric configuration, you can useOPENWISP_MONITORING_METRICS.unregister_metricThis function is used to unregister a metric configuration from anywhere in your code.ParameterDescriptionmetric_name:Astrdefining name of the metric configuration.An example usage is shown below.fromopenwisp_monitoring.monitoring.configurationimportunregister_metric# Unregister previously registered metric configurationunregister_metric('metric_name')Note: It will raiseImproperlyConfiguredexception if the concerned metric
configuration is not registered.Registering / Unregistering Chart ConfigurationOpenWISP Monitoringprovides registering and unregistering chart configuration through utility functionsopenwisp_monitoring.monitoring.configuration.register_chartandopenwisp_monitoring.monitoring.configuration.unregister_chart.
Using these functions you can register or unregister chart configurations from anywhere in your code.register_chartThis function is used to register a new chart configuration from anywhere in your code.ParameterDescriptionchart_name:Astrdefining name of the chart configuration.chart_configuration:Adictdefining configuration of the chart.An example usage has been shown below.fromopenwisp_monitoring.monitoring.configurationimportregister_chart# Define configuration of your chartchart_config={'type':'histogram','title':'Histogram','description':'Histogram','top_fields':2,'order':999,'query':{'influxdb':("SELECT {fields|SUM|/ 1} FROM{key}""WHERE time >= '{time}' AND content_type = ""'{content_type}' AND object_id = '{object_id}'")},}# Register your custom chart configurationregister_chart('chart_name',chart_config)Note: It will raiseImproperlyConfiguredexception if a chart configuration
is already registered with same name (not to be confused with verbose_name).If you don’t need to register a new chart but need to change a specific key of an
existing chart configuration, you can useOPENWISP_MONITORING_CHARTS.unregister_chartThis function is used to unregister a chart configuration from anywhere in your code.ParameterDescriptionchart_name:Astrdefining name of the chart configuration.An example usage is shown below.fromopenwisp_monitoring.monitoring.configurationimportunregister_chart# Unregister previously registered chart configurationunregister_chart('chart_name')Note: It will raiseImproperlyConfiguredexception if the concerned chart
configuration is not registered.Registering new notification typesYou can define your own notification types usingregister_notification_typefunction from OpenWISP
Notifications. For more information, see the relevantopenwisp-notifications section about registering notification types.Once a new notification type is registered, you have to use the“notify” signal provided in
openwisp-notificationsto send notifications for this type.ExceptionsTimeseriesWriteExceptionPath:openwisp_monitoring.db.exceptions.TimeseriesWriteExceptionIf there is any failure due while writing data in timeseries database, this exception shall
be raised with a helpful error message explaining the cause of the failure.
This exception will normally be caught and the failed write task will be retried in the background
so that there is no loss of data if failures occur due to overload of Timeseries server.
You can read more about this retry mechanism atOPENWISP_MONITORING_WRITE_RETRY_OPTIONS.InvalidMetricConfigExceptionPath:openwisp_monitoring.monitoring.exceptions.InvalidMetricConfigExceptionThis exception shall be raised if the metric configuration is broken.InvalidChartConfigExceptionPath:openwisp_monitoring.monitoring.exceptions.InvalidChartConfigExceptionThis exception shall be raised if the chart configuration is broken.Rest APILive documentationA general live API documentation (following the OpenAPI specification) at/api/v1/docs/.Browsable web interfaceAdditionally, opening any of the endpointslisted belowdirectly in the browser will show thebrowsable API interface of Django-REST-Framework,
which makes it even easier to find out the details of each endpoint.List of endpointsSince the detailed explanation is contained in theLive documentationand in theBrowsable web pageof each point,
here we’ll provide just a list of the available endpoints,
for further information please open the URL of the endpoint in your browser.Retrieve device charts and device status dataGET /v1/monitoring/device/{pk}/?key={key}&status=trueThe format used for Device Status is inspired byNetJSON DeviceMonitoring.Note: If the request is made without?status=truethen only device charts
data would be returned.Collect device metrics and statusPOST /v1/monitoring/device/{pk}/?key={key}&time={time}If data is latest then an additional parameter current can also be passed. For e.g.:POST /v1/monitoring/device/{pk}/?key={key}&time={time}&current=trueThe format used for Device Status is inspired byNetJSON DeviceMonitoring.Note: Device data will be saved with in timeseries database with the specifiedtime,
this should be in the format%d-%m-%Y_%H:%M:%S.%f, otherwise 400 Bad Response will be returned.If the request is made without passing thetimeargument, the server local time will be used.Thetimeparameter was added to supportresilient collection and sending of data by the OpenWISP Monitoring Agent.Signalsdevice_metrics_receivedPath:openwisp_monitoring.device.signals.device_metrics_receivedArguments:instance: instance ofDevicewhose metrics have been receivedrequest: the HTTP request objecttime: time with which metrics will be saved. If none, then server time will be usedcurrent: whether the data has just been collected or was collected previously and sent now due to network connectivity issuesThis signal is emitted when device metrics are received to theDeviceMetricview (only when using HTTP POST).The signal is emitted just before a successful response is returned,
it is not sent if the response was not successful.health_status_changedPath:openwisp_monitoring.device.signals.health_status_changedArguments:instance: instance ofDeviceMonitoringwhose status has been changedstatus: the status by which DeviceMonitoring’s existing status has been updated withThis signal is emitted only if the health status of DeviceMonitoring object gets updated.threshold_crossedPath:openwisp_monitoring.monitoring.signals.threshold_crossedArguments:metric:Metricobject whose threshold defined in related alert settings was crossedalert_settings:AlertSettingsrelated to theMetrictarget: relatedDeviceobjectfirst_time: it will be set to true when the metric is written for the first time. It shall be set to false afterwards.tolerance_crossed: it will be set to true if the metric has crossed the threshold for tolerance configured in alert settings.
Otherwise, it will be set to false.first_timeparameter can be used to avoid initiating unneeded actions.
For example, sending recovery notifications.This signal is emitted when the threshold value of aMetricdefined in
alert settings is crossed.pre_metric_writePath:openwisp_monitoring.monitoring.signals.pre_metric_writeArguments:metric:Metricobject whose data shall be stored in timeseries databasevalues: metric data that shall be stored in the timeseries databasetime: time with which metrics will be savedcurrent: whether the data has just been collected or was collected previously and sent now due to network connectivity issuesThis signal is emitted for every metric before the write operation is sent to
the timeseries database.post_metric_writePath:openwisp_monitoring.monitoring.signals.post_metric_writeArguments:metric:Metricobject whose data is being stored in timeseries databasevalues: metric data that is being stored in the timeseries databasetime: time with which metrics will be savedcurrent: whether the data has just been collected or was collected previously and sent now due to network connectivity issuesThis signal is emitted for every metric after the write operation is successfully
executed in the background.Management commandsrun_checksThis command will execute all theavailable checksfor all the devices.
By default checks are run periodically bycelery beat. You can learn more
about this inSetup.Example usage:cdtests/./manage.pyrun_checksmigrate_timeseriesThis command triggers asynchronous migration of the time-series database.Example usage:cdtests/./manage.pymigrate_timeseriesMonitoring scriptsMonitoring scripts are now deprecated in favour ofmonitoring packages.
Follow the migration guide inMigrating from monitoring scripts to monitoring packagessection of this documentation.Migrating from monitoring scripts to monitoring packagesThis section is intended for existing users ofopenwisp-monitoring.
The older version ofopenwisp-monitoringusedmonitoring scriptsthat
are now deprecated in favour ofmonitoring packages.If you already had amonitoring templatecreated on your installation,
then the migrations ofopenwisp-monitoringwill update that template
by making the following changes:The file name of all scripts will be appended withlegacy-keyword
in order to differentiate them from the scripts bundled with the new packages.The/usr/sbin/legacy-openwisp-monitoring(previously/usr/sbin/openwisp-monitoring)
script will be updated to exit ifopenwisp-monitoring packageis installed on the device.Install themonitoring packagesas mentioned in theInstall monitoring packages on devicesection of this documentation.After the proper configuration of theopenwisp-monitoring packageon your device, you can remove the monitoring template from your devices.We suggest removing the monitoring template from the devices one at a time instead
of deleting the template. This ensures the correctness ofopenwisp monitoring packageconfiguration and you’ll not miss out on
any monitoring data.Note:If you have made changes to the default monitoring template created
byopenwisp-monitoringor you are using custom monitoring templates, then you should
remove such templates from the device before installing themonitoring packages.Extending openwisp-monitoringOne of the core values of the OpenWISP project isSoftware Reusability,
for this reasonopenwisp-monitoringprovides a set of base classes
which can be imported, extended and reused to create derivative apps.In order to implement your custom version ofopenwisp-monitoring,
you need to perform the steps described in the rest of this section.When in doubt, the code in thetest projectand thesample appsnamelysample_check,sample_monitoring,sample_device_monitoringwill guide you in the correct direction:
just replicate and adapt that code to get a basic derivative ofopenwisp-monitoringworking.Premise: if you plan on using a customized version of this module,
we suggest to start with it since the beginning, because migrating your data
from the default module to your extended version may be time consuming.1. Initialize your custom moduleThe first thing you need to do in order to extend anyopenwisp-monitoringapp is create
a new django app which will contain your custom version of thatopenwisp-monitoringapp.A django app is nothing more than apython package(a directory of python scripts), in the following examples we’ll call these django apps asmycheck,mydevicemonitoring,mymonitoringbut you can name it how you want:django-admin startapp mycheck
django-admin startapp mydevicemonitoring
django-admin startapp mymonitoringKeep in mind that the command mentioned above must be called from a directory
which is available in yourPYTHON_PATHso that you can then import the result into your project.Now you need to addmychecktoINSTALLED_APPSin yoursettings.py,
ensuring also thatopenwisp_monitoring.checkhas been removed:INSTALLED_APPS=[# ... other apps ...# 'openwisp_monitoring.check',        <-- comment out or delete this line# 'openwisp_monitoring.device',       <-- comment out or delete this line# 'openwisp_monitoring.monitoring'    <-- comment out or delete this line'mycheck','mydevicemonitoring','mymonitoring','nested_admin',]For more information about how to work with django projects and django apps,
please refer to the“Tutorial: Writing your first Django app” in the django docunmentation.2. Installopenwisp-monitoringInstall (and add to the requirement of your project)openwisp-monitoring:pip install --U https://github.com/openwisp/openwisp-monitoring/tarball/master3. AddEXTENDED_APPSAdd the following to yoursettings.py:EXTENDED_APPS=['device_monitoring','monitoring','check']4. Addopenwisp_utils.staticfiles.DependencyFinderAddopenwisp_utils.staticfiles.DependencyFindertoSTATICFILES_FINDERSin yoursettings.py:STATICFILES_FINDERS=['django.contrib.staticfiles.finders.FileSystemFinder','django.contrib.staticfiles.finders.AppDirectoriesFinder','openwisp_utils.staticfiles.DependencyFinder',]5. Addopenwisp_utils.loaders.DependencyLoaderAddopenwisp_utils.loaders.DependencyLoadertoTEMPLATESin yoursettings.py:TEMPLATES=[{'BACKEND':'django.template.backends.django.DjangoTemplates','OPTIONS':{'loaders':['django.template.loaders.filesystem.Loader','django.template.loaders.app_directories.Loader','openwisp_utils.loaders.DependencyLoader',],'context_processors':['django.template.context_processors.debug','django.template.context_processors.request','django.contrib.auth.context_processors.auth','django.contrib.messages.context_processors.messages',],},}]6. Inherit the AppConfig classPlease refer to the following files in the sample app of the test project:sample_check/__init__.py.sample_check/apps.py.sample_monitoring/__init__.py.sample_monitoring/apps.py.sample_device_monitoring/__init__.py.sample_device_monitoring/apps.py.For more information regarding the concept ofAppConfigplease refer to
the“Applications” section in the django documentation.7. Create your custom modelsTo extendcheckapp, refer tosample_check models.py file.To extendmonitoringapp, refer tosample_monitoring models.py file.To extenddevice_monitoringapp, refer tosample_device_monitoring models.py file.Note:For doubts regarding how to use, extend or develop models please refer to
the“Models” section in the django documentation.For doubts regarding proxy models please refer toproxy models.8. Add swapper configurationsAdd the following to yoursettings.py:# Setting models for swapper module# For extending check appCHECK_CHECK_MODEL='YOUR_MODULE_NAME.Check'# For extending monitoring appMONITORING_CHART_MODEL='YOUR_MODULE_NAME.Chart'MONITORING_METRIC_MODEL='YOUR_MODULE_NAME.Metric'MONITORING_ALERTSETTINGS_MODEL='YOUR_MODULE_NAME.AlertSettings'# For extending device_monitoring appDEVICE_MONITORING_DEVICEDATA_MODEL='YOUR_MODULE_NAME.DeviceData'DEVICE_MONITORING_DEVICEMONITORING_MODEL='YOUR_MODULE_NAME.DeviceMonitoring'Substitute<YOUR_MODULE_NAME>with your actual django app name
(also known asapp_label).9. Create database migrationsCreate and apply database migrations:./manage.py makemigrations
./manage.py migrateFor more information, refer to the“Migrations” section in the django documentation.10. Create your custom adminTo extendcheckapp, refer tosample_check admin.py file.To extendmonitoringapp, refer tosample_monitoring admin.py file.To extenddevice_monitoringapp, refer tosample_device_monitoring admin.py file.To introduce changes to the admin, you can do it in the two ways described below.Note: for doubts regarding how the django admin works, or how it can be customized,
please refer to“The django admin site” section in the django documentation.1. Monkey patchingIf the changes you need to add are relatively small, you can resort to monkey patching.For example, forcheckapp you can do it as:fromopenwisp_monitoring.check.adminimportCheckAdminCheckAdmin.list_display.insert(1,'my_custom_field')CheckAdmin.ordering=['-my_custom_field']Similarly fordevice_monitoringapp, you can do it as:fromopenwisp_monitoring.device.adminimportDeviceAdminDeviceAdmin.list_display.insert(1,'my_custom_field')DeviceAdmin.ordering=['-my_custom_field']Similarly formonitoringapp, you can do it as:fromopenwisp_monitoring.monitoring.adminimportMetricAdmin,AlertSettingsAdminMetricAdmin.list_display.insert(1,'my_custom_field')MetricAdmin.ordering=['-my_custom_field']AlertSettingsAdmin.list_display.insert(1,'my_custom_field')AlertSettingsAdmin.ordering=['-my_custom_field']2. Inheriting admin classesIf you need to introduce significant changes and/or you don’t want to resort to
monkey patching, you can proceed as follows:Forcheckapp,fromdjango.contribimportadminfromopenwisp_monitoring.check.adminimportCheckAdminasBaseCheckAdminfromswapperimportload_modelCheck=load_model('check','Check')admin.site.unregister(Check)@admin.register(Check)classCheckAdmin(BaseCheckAdmin):# add your changes hereFordevice_monitoringapp,fromdjango.contribimportadminfromopenwisp_monitoring.device_monitoring.adminimportDeviceAdminasBaseDeviceAdminfromswapperimportload_modelDevice=load_model('config','Device')admin.site.unregister(Device)@admin.register(Device)classDeviceAdmin(BaseDeviceAdmin):# add your changes hereFormonitoringapp,fromdjango.contribimportadminfromopenwisp_monitoring.monitoring.adminimport(AlertSettingsAdminasBaseAlertSettingsAdmin,MetricAdminasBaseMetricAdmin)fromswapperimportload_modelMetric=load_model('Metric')AlertSettings=load_model('AlertSettings')admin.site.unregister(Metric)admin.site.unregister(AlertSettings)@admin.register(Metric)classMetricAdmin(BaseMetricAdmin):# add your changes here@admin.register(AlertSettings)classAlertSettingsAdmin(BaseAlertSettingsAdmin):# add your changes here11. Create root URL configurationPlease refer to theurls.pyfile in the test project.For more information about URL configuration in django, please refer to the“URL dispatcher” section in the django documentation.12. Create celery.pyPlease refer to thecelery.pyfile in the test project.For more information about the usage of celery in django, please refer to the“First steps with Django” section in the celery documentation.13. Import Celery TasksAdd the following in your settings.py to import celery tasks fromdevice_monitoringapp.CELERY_IMPORTS=('openwisp_monitoring.device.tasks',)14. Create the custom commandrun_checksPlease refer to therun_checks.pyfile in the test project.For more information about the usage of custom management commands in django, please refer to the“Writing custom django-admin commands” section in the django documentation.15. Import the automated testsWhen developing a custom application based on this module, it’s a good idea
to import and run the base tests too, so that you can be sure the changes you’re introducing
are not breaking some of the existing features of openwisp-monitoring.In case you need to add breaking changes, you can overwrite the tests defined
in the base classes to test your own behavior.For, extendingcheckapp see thetests of sample_check appto find out how to do this.For, extendingdevice_monitoringapp see thetests of sample_device_monitoring appto find out how to do this.For, extendingmonitoringapp see thetests of sample_monitoring appto find out how to do this.Other base classes that can be inherited and extendedThe following steps are not required and are intended for more advanced customization.DeviceMetricViewThis view is responsible for displayingChartsandStatusprimarily.The full python path is:openwisp_monitoring.device.api.views.DeviceMetricView.If you want to extend this view, you will have to perform the additional steps below.Step 1. Import and extend view:# mydevice/api/views.pyfromopenwisp_monitoring.device.api.viewsimport(DeviceMetricViewasBaseDeviceMetricView)classDeviceMetricView(BaseDeviceMetricView):# add your customizations here ...passStep 2: remove the following line from your rooturls.pyfile:re_path('api/v1/monitoring/device/(?P<pk>[^/]+)/$',views.device_metric,name='api_device_metric',),Step 3: add an URL route pointing to your custom view inurls.pyfile:# urls.pyfrommydevice.api.viewsimportDeviceMetricViewurlpatterns=[# ... other URLsre_path(r'^(?P<path>.*)$',DeviceMetricView.as_view(),name='api_device_metric',),]ContributingPlease refer to theOpenWISP contributing guidelines.