Package: cluster-logger
Cluster Logging library for python 3.6This is a library for python 3.6 that will help fulfil a
requirement for cluster services and logging to elastic search.RequirementsPython 3.6+pipenvElasticsearchFluentdKibanaPipenv & PipfilePipenv combines package management and virtualenv into one tool.pipenv:https://docs.pipenv.org/basicsTo install:$pipinstallpipenvPipfile replaces requirements.txt and it will specify both dependencies and dev dependencies in one file.Development Installation# install dependenciespipenvinstall--dev# activate virtual environmentpipenvshell# alternativly you can start your script in a virtual environment contextpipenvrunpythonmain.pymessagethatyouwanttosendULM ConfigurationThe default configuration uses Environment variables to configure the logger.These values were grabbed from version 2.0 of ULM-logging-common Java implementation. Not every value is implemented yet and changes are upto date in table.RepoVariableDefault ValueDescriptionImplementedULM_FLUENTD_HOSTlocalhosthost of the FluentD service✔️ULM_FLUENTD_PORT24224port of FluentD tcp "in_forward"✔️ULM_FLUENTD_LABEL_ENVScomma separated list(with or without space) of other ENV vars to add as fields to FluentD logs✔️ULM_FLUENTD_TAG(project).(app)tag to use when sending logs to fluentd, e.g. the MLA_IDENT string such as "Myapp.logs"✔️ULM_FLUENTD_BUFFER1048576an integer of the number of bytes to allow in the buffer.  Excess will be dropped (default is 1MiB)❌ULM_FLUENTD_TIMEOUT3.0an float of the number of ms to allow for communication with FluentD.✔️ULM_FLUENTD_CLIENTasyncWhether to send async messages or not. This will send async as long as the value has async somewhere in the string.✔️ULM_FLUENTD_ENABLE_TIMEMS_FIELDFalseEnables an additional field to contain the millisecond timestamp❌ULM_FLUENTD_TIMEMS_FIELDtimemsthe name of the field to contain the millisecond timestamp❌Standalone Usage (Development)It may be required to run the following command to get elastic search to run.sudosysctl-wvm.max_map_count=262144Local$pipenvrunpythonmain.py-h
usage:main.py[-h][-nHOST][-pPORT][-e[ENV[ENV...]]][-oPROJECT][-aAPP][-cCOUNT]ConsoleApplicationtotestClusterLoggingusingfluentd.

positionalarguments:messageMessagetobelogged.

optionalarguments:-h,--helpshowthishelpmessageandexit-nHOST,--hostHOSTHostnametolistenon.(default=localhost)-pPORT,--portPORTPortnumbertobindto.(default=24224)-e[ENV[ENV...]],--env[ENV[ENV...]]EnvironmentKeystoaddtomessage.-oPROJECT,--projectPROJECTTagfortheproject-aAPP,--appAPPTagfortheapplication-cCOUNT,--countCOUNTThenumberoftimestosendmessage.Docker$dockerbuild-tTAG_FOR_THE_BUILD:VERSION.# all env values can be set with -e$dockerrun-ehost=fluentd\-eport=24224-emsg="message you want to send"TAG_FOR_THE_BUILD:VERSION# If you want to run the whole environment you can use docker-compose up$docker-composeupApiFollows the python logger implementation with logging levels.python.orgLevelNumeric valueCRITICAL50ERROR40WARNING30INFO20DEBUG10NOTSET0Logging level can be set for logging using:importlogginglogging.basicConfig(level=logging.WARNING)Setting the logging level will filter levels below the set logging level. For logging levels error and above the stack trace will be added to the log message.Standard ConfigurationThese environment variables can be set either system-wide or added to the.envfile:There are 3 Environment variables that will be extracted by default:HOST- host that the container is running on.APP_VERSION- Version of the app.LOGGER_VERBOSE- Whether the fluent logger library is verbose or not.MARATHON_APP_ID- The ID that is assigned by DCOS.MARATHON_APP_DOCKER_IMAGE- The image that the container was built from.There are three ways to get application properties into log messages.pass a dictionary of key value pairspass a list of keys that are on the container environmentor pass a JSON file with key value pairs.NOTE:The dictionary will overwrite values in environment and JSON file.The Environment will overwrite values in the JSON file.UsageActivate your venv# Linux / MacOSsource{NAME_OF_VENV}/bin/activateREM Windows{NAME_OF_VENV}\Scripts\activateInstall cluster logger into your venv. Cluster Logger has been installed into pypi.$pipinstallcluster_loggerTo Useimportcluster_loggerimportlogging# Then init the ClusterLogger classenv_keys=['MARATHON_APP_LABELS','MARATHON_APP_RESOURCE_CPUS']props={'HOST','192.168.0.12'}logging.basicConfig(level=logging.DEBUG)# set logging levelcluster_logger.initLogger('Rasters',# Project name'intent_client',# Application name'fluentd',# Fluentd Host name (optional) will default to fluentd if nothing is passed in and HOST isn't set on the environment24224,# Fluentd Port Number(optional)'/path/to/settings.json',# JSON settings file (optional)env_keys,# Environment Keys (optional)props)# Specific Properties (optional)# in Module where you will logimportcluster_logger# General Loggerlogger=cluster_logger.getLogger(__name__)# Metric Loggermetrics=cluster_logger.getLogger(__name__+'.metrics',is_metric=True)logger.exception('Custom Message to send typically an error')logger.log(logging.INFO,{'Duration',58})Contributing# bump the version that is stored in setup.py and cluster_logger/__init__.py# assuming pipenv install --dev was ran first$pipenvrunbumpversionminor# possible: major / minor / patch$gitpush