Package: aidockermon
aidockermonMonitor system load of the server running the nvidia/cuda docker containers.Featuresysinfo: system static infosysload: system cpu/memory loadgpu: nvidia gpu loaddisk: disk loadcontainers: containers' load that based on the nvidia/cuda imagePrerequisitePython >= 3InstallationpipinstallaidockermonOr usesetuptoolspythonsetup.pyinstallUsage$ aidockermon -h
usage: aidockermon [-h] [-v] {query,create-esindex,delete-esindex} ...

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit

command:
  {query,create-esindex,delete-esindex}
    query               Query system info, log them via syslog protocol
    create-esindex      Create elasticsearch index
    delete-esindex      Delete elasticsearch index$ aidockermon query -h
usage: aidockermon query [-h] [-l] [-r REPEAT] [-f FILTERS [FILTERS ...]] type

positional arguments:
  type                  info type: sysinfo, sysload, gpu, disk, containers

optional arguments:
  -h, --help            show this help message and exit
  -l, --stdout          Print pretty json to console instead of send a log
  -r REPEAT, --repeat REPEAT
                        n/i repeat n times every i seconds
  -f FILTERS [FILTERS ...], --filters FILTERS [FILTERS ...]
                        Filter the disk paths for disk type; filter the
                        container names for containers typeFor example:Show sysinfo$aidockermonquery-lsysinfo{"gpu":{"gpu_num":2,"driver_version":"410.104","cuda_version":"10.0"},"mem_tot":67405533184,"kernel":"4.4.0-142-generic","cpu_num":12,"docker":{"version":"18.09.3"},"system":"Linux"}Show sys load$aidockermonquery-lsysload{"mem_free":11866185728,"mem_used":8023793664,"cpu_perc":57.1,"mem_perc":12.8,"mem_avail":58803163136,"mem_tot":67405533184}Show gpu load$aidockermonquery-lgpu{"mem_tot":11177,"gpu_temperature":76.0,"mem_free":1047,"mem_used":10130,"gpu_perc":98.0,"gpu_id":0,"mem_perc":46.0}{"mem_tot":11178,"gpu_temperature":66.0,"mem_free":3737,"mem_used":7441,"gpu_perc":95.0,"gpu_id":1,"mem_perc":44.0}Show disk usage$aidockermonquerydisk-l-f/{"path":"/","device":"/dev/nvme0n1p3","total":250702176256,"used":21078355968,"free":216865271808,"percent":8.9}$aidockermonquerydisk-l-f//disk{"path":"/","device":"/dev/nvme0n1p3","total":250702176256,"used":21078355968,"free":216865271808,"percent":8.9}{"path":"/disk","device":"/dev/sda1","total":1968874311680,"used":1551374692352,"free":317462949888,"percent":83.0}Show containers' loadNote that theapp_namewould be read from environment variableAPP_NAME, which
is a short description for this training program.$aidockermonquerycontainers-l-fDianAI{"proc_name":"python3 test_run.py","app_name":"测试程序","pid":13540,"container":"DianAI","started_time":1554698236,"running_time":9343,"mem_used":9757}{"proc_name":"python train.py","app_name":"","pid":15721,"container":"DianAI","started_time":1554698236,"running_time":19343,"mem_used":1497}{"mem_limit":67481047040,"net_output":47863240948,"block_read":1327175626752,"net_input":18802869033,"mem_perc":14.637655604461704,"block_write":132278439936,"name":"DianAI","cpu_perc":0.0,"mem_used":9877643264}Configloggingdebug:falselog:version:1# This is the default level, which could be ignored.# CRITICAL = 50# FATAL = CRITICAL# ERROR = 40# WARNING = 30# WARN = WARNING# INFO = 20# DEBUG = 10# NOTSET = 0#level: 20disable_existing_loggers:falseformatters:simple:format:'%(levelname)s%(message)s'monitor:format:'%(message)s'filters:require_debug_true:():'aidockermon.handlers.RequireDebugTrue'handlers:console:level:DEBUGclass:logging.StreamHandlerformatter:simplefilters:[require_debug_true]monitor:level:INFOclass:rfc5424logging.handler.Rfc5424SysLogHandleraddress:[127.0.0.1,1514]enterprise_id:1loggers:runtime:handlers:[console]level:DEBUGpropagate:falsemonitor:handlers:[monitor,console]level:INFOpropagate:falseThis is the default config, which should be located at/etc/aidockermon/config.yml.You can modify theaddressvalue to specify the logging target.address: [127.0.0.1, 1514]: UDP to 127.0.0.1:1514address: /var/log/aidockermon: unix domain datagram socketIf you add ansocktypeargument, you can specify whether to use UDP or TCP as transport protocol.socktype: 1: TCPsocktype: 2: UDPEnable TLS/SSL:tls_enable:truetls_verify:truetls_ca_bundle:/path/to/ca-bundle.pemSetdebugastrue, you can see message output in the console.Cronjobsudocpetc/cron.d/aidockermon/etc/cron.d
sudosystemctlrestartcronsyslog-ngUsing syslog-ng to collect logs and send them to elasticsearch
for future use such as visualization with kibana.cpetc/syslog-ng/syslog-ng.conf/etc/syslog-ng/
sudosystemctlrestartsyslog-ngSample config:@version:3.20

destinationd_elastic{elasticsearch2(index("syslog-ng")type("${.SDATA.meta.type}")flush-limit("0")cluster("es-syslog-ng")cluster-url("http://localhost:9200")client-mode("http")client-lib-dir(/usr/share/elasticsearch/lib)template("${MESSAGE}\n"));};sources_python{#unix-dgram("/var/log/aidockermon");syslog(ip(127.0.0.1)port(1514)transport("udp")flags(no-parse));};log{source(s_python);parser{syslog-parser(flags(syslog-protocol));};destination(d_elastic);};Modify it to specify the elasticsearch server and the log source's port and protocol.