Package: alco
What’s the problemThere is a widely used stack of technologies for parsing, collecting and
analysing logs -ELK Stack. It has
very functional web interface, search cluster and a log transformation
tool. Very cool, but:It’s Java with well-known requirements for memory and CPUsIt’s ElasticSearch with it’s requirements for disk spaceIt’s Logstash which suddenly stops processing logs in some
conditions.It’s Kibana with very cool RICH interface which looses on all counts
togrepandlessin a task of log reading and searching.Introducing ALCOALCO is a simple ELK analog which primary aim is to provide a online
replacement forgrepandless. Main features are:Django application for incident analysis in distributed systemsschemeless full-text index with filtering and searchingconfigurable log collection and rotation from RabbitMQ messaging
servernot a all-purpose monsterTechnology stackLet’s trace log message path from some distributed system to ALCO web
interface.Python-based project callslogger.debug()method with text ‘hello
world’At startup timeLogcollectlibrary
automatically configures python logging (or evenDjangoandCeleryone’s) to send log
messages to RabbitMQ server in JSON format readable both with ELK and
ALCO projects.ALCO log collector binds a queue to RabbitMQ exchange and processes
messages in a batch.It uses Redis to collect unique values for filterable fields and
SphinxSearch to store messages in a realtime index.When a message is inserted to sphinxsearch, it contains indexedmessagefield, timestamp information and schemeless JSON field
namedjswith all log record attributes sent by python log.Django-based web interface provides API and client-side app for
searching collected logs online.RequirementsPython 2.7 or 3.3+Logcollectfor python
projects which logs are collectedRabbitMQserver for distributed log
collectionSphinxSearchserver 2.3 or later for
log storageRedisfor SphinxSearch docid management and
field values storagedjango-sphinxsearchas a database backend forDjango>=1.8(will be available from
PyPI)SetupYou need to configure logcollect in analyzed projects (seeREADME).
If RabbitMQ admin interface shows non-zero message flow inlogstashexchange - “It works” :-)Install alco and it’s requirements from PyPipipinstallalcoNext, create django project, addsphinxsearchdatabase
connection and configuresettings.pyto enable alco applications# For SphinxRouterSPHINX_DATABASE_NAME='sphinx'DATABASES[SPHINX_DATABASE_NAME]={'ENGINE':'sphinxsearch.backend.sphinx','HOST':'127.0.0.1','PORT':9306,}}# Auto routing log models to SphinxSearch databaseDATABASE_ROUTERS=('sphinxsearch.routers.SphinxRouter',)INSTALLED_APPS+=['rest_framework',# for API to work'alco.collector','alco.grep']ROOT_URLCONF='alco.urls'Configure ALCO resources insettings.py:ALCO_SETTINGS={# log messaging server'RABBITMQ':{'host':'127.0.0.1','userid':'guest','password':'guest','virtual_host':'/'},# redis server'REDIS':{'host':'127.0.0.1','db':0},# url for fetching sphinx.conf dynamically'SPHINX_CONF_URL':'http://127.0.0.1:8000/collector/sphinx.conf',# name of django.db.connection for SphinxSearch'SPHINX_DATABASE_NAME':'sphinx',# number of results in log view API'LOG_PAGE_SIZE':100}# override defaults for sphinx.conf templateALCO_SPHINX_CONF={# local index definition defaults override'index':{'min_word_len':8},# searchd section defaults override'searchd':{'dist_threads':8}}Runsyncdbor bettermigratemanagement command to create
database tables.Run webserver and create a LoggerIndex fromdjango
admin.Created directories for sphinxsearch:/var/log/sphinx/
/var/run/sphinx/
/data/sphinx/Next, configure sphinxsearch to use generated config:searchd-csphinx_conf.pysphinx_conf.pyis a simple script that importsalco.sphinx_confmodule which fetches generatedsphinx.conffrom alco http api and created directories for SphinxSearch indices:#!/data/alco/virtualenv/bin/python# coding: utf-8importosos.environ.setdefault('DJANGO_SETTINGS_MODULE','settings')fromalcoimportsphinx_confRun log collectors:pythonmanage.pystart_collectors--no-daemonIf it shows number of collected messages periodically - then log
collecting is set up correctly.Configure system services to start subsystems automatically:nginx or apache http serverdjango uwsgi backendalco collectors (start_collectorsmanagement command)sphinxsearch, redis, default database for DjangoOpenhttp://127.0.0.1:8000/grep/<logger_name>/to read and
search logs online.VirtualenvWe successfully configured SphinxSearch to use python fromvirtualenv, adding some environment variables to start script (i.e.
FreeBSD rc.d script):sphinxsearch_prestart(){# nobody user has no HOMEexportPYTHON_EGG_CACHE=/tmp/.python-eggs# python path for virtualenv interpreter should be redeclaredexportPYTHONPATH=${venv_path}/lib/python3.4/:${venv_path}/lib/python3.4/site-packages/."${virtualenv_path}/bin/activate"||err1"Virtualenv is not found"echo"Virtualenv${virtualenv_path}activated: `which python`"}In this caseshebangforsphinx_conf.pymust point virtualenv’s
python interpreter.Production usageFor now ALCO stack is tested in preproduction environment in our company
and is actively developed. There are no reasons to say that it’s not
ready for production usage.