Package: kekmonitors
KEK MonitorsThis is a ready-to-use codebase onto which you can develop your custom sneakers monitors. It tries to handle everything for you: databases, discord webhooks, network connections etc, but you are encouraged to customize the source code to fit your needs.Here scrapers and actual monitors are separated and working asynchronously, communicating through Unix sockets, leading to improved performance compared to having a single script doing everything synchronously. I also lied the basis for an api, so that you don't necessarily need to ssh into the server to activate monitors, but you can instead just use a rest api.NOTE: THERE ARE NO IMPORTANT ENDPOINTS IN THE CODE! THERE IS ONLY ONE SEMI-WORKING MONITOR PROVIDED AS AN EXAMPLE!If you have any questions please join the Discord server:https://discord.gg/76r8GJyeZZPre-requisitesPython 3> 3.6linux: the monitors have been tested on Arch Linux and Ubuntu, but they should work on any other linux distro/WSL without any problem.libcurlcompiled with async and possibly brotli support (look forbrotliandAsynchDNSincurl --versionfeatures). Brotli support is recommended but often not shipped with packaged versions of curl; if you want to add support to it you can compile and install curl yourself with brotli, making sure withcurl --versionthat you are getting the output from your compiled version, and reinstallpycurlwithpip install pycurl --no-binary :all: --force-reinstallMongoDBinstalled and running (get it from the link or from your package manager)Setup# recommended: setup a virtualenvironment before actually installing to the systempython3-mvenvvenvsource./venv/bin/activate# to install the package from the PyPI:python3-mpipinstallkekmonitors# to install the package from source:python3-mpipinstall.# if you want to try the examples:# please make sure that `~/.local/bin/` is in your `$PATH`cddemo
python3-mpipinstall-rrequirements.txt# you're ready to go!# remember to start MongoDB and perhaps setup a webhook in the configs so that you can see the notifications!UsageIf you want to quickly look at how monitors look like, take a look at the sample codefootdistrict_scraper.pyandfootdistrict_monitor.pyBefore using thekekmonitors.monitor_managermake sure you started the monitors/scraper at least once manually (this is needed to register it in the database):# in a SSH screen session:python3<filename>[--delayn]--[[no-]output]The recommended way to start and control monitors is viakekmonitors.monitor_manager.
Assuming you are remotely working on a server via SSH and you want to start both a scraper and a monitor:# in a SSH screen session:python3-mkekmonitors.monitor_manager# in another SSH screen session:python3-mkekmonitors.monitor_manager_cliMM_ADD_MONITOR_SCRAPER--name<name>[--delayn][otherkeywordargumentsrequiredbythemonitor/scraper]The monitor manager will automatically keep track of which monitors/scrapers are available and can notify if and when they crash; it also manages the config updates (as soon as you change a file in the configs folder (~/.kekmonitors/configby default) it notifies the interested monitors/scrapers).There is also anapp.pythat "bridges" between http and the monitor manager, which allows you to use a REST api to control the monitor manager:# in a SSH screen session:python3-mkekmonitors.appYou can see the available endpoints by navigating to the root endpoint (by default:http://localhost:8888/).app.pyis only used as an example, and you should not use it in "production" since it doesn't use any sort of authentication, so anyone who finds your server's ip address can very easily control your monitors.ConfigurationStatic configuration, like commands and global variables (socket_path), is contained in~/.config/kekmonitors/config.cfgby default (the default file is hardcoded inconfig.py); the "dynamic" configuration files instead, by default, are stored in~/.config/kekmonitors/monitorsand~/.config/kekmonitors/scrapers; every scraper and monitor looks for its corresponding entry inblacklist.json,whitelist.json, and a general not-yet-usedconfigs.json. Here's an example blacklists.json:{"Footdistrict":["some term","another, term"],"AnotherWebsite":["one more, term","so many terms"]}More information on the syntax can be found inkekmonitors.utils.tools.is_whitelist().Thewebhooks.jsonfile can be used to add webhooks configuration, with support to optional customization:{"Footdistrict":{"https://discordapp.com/api/webhooks/your-webhook-here":{"name":"Human readable name, not used at all in the code","custom":{"provider":"your-provider-name","icon_url":"your-icon-url"}},"https://discordapp.com/api/webhooks/another-webhook":{"name":"A webhook with default customization"}}}The default embed generation is found indiscord_embeds.py.How does it all work?The project can be thought of as being divided into several big parts: scrapers, monitors, database manager, webhook manager, discord embeds, monitor manager+api. Obviously you can, and should, customize everything to suite your needs, but you probably want to start by writing the first scraper/monitor combo.I've tried to write everything so that you can easily customize your own monitor/scraper without modifying the source code too much: if for example you want to add custom commands to your monitor, adding statistics for instance, you can just extend theCOMMANDSclass, then write a callback function which will handle the received command and that's it!Important: about NetworkUtils.fetch()By default fetch has the optionuse_cacheset to True. The cache in question is not the typical CDN cache (the hit/miss cache from cloudflare for example), which you typically try to avoid (you look for the miss) to have the most up to date page possible; it's HTTP cache, which is "activated" by theif-modified-sinceandetagheaders: whenthiscache is hit, the return code is 304 and the body is empty, which saves a huge amount of bandwidth; from what I tested this seems like a really good option since the response is almost entirely empty, saving up on proxy bandwidth and general costs, and it doesn't seem to impact performance (remember it's not CDN related, but purely HTTP related). NetworkUtils automatically manages the internal pages cache.Anyway you can turn off this behavior withuse_cache=Falseon each request, retrieving a full response each time.List of executables/useful scripts:monitor_manager.py: manages monitors and scrapers, can be used to talk to them viakekmonitors.monitor_manager_climonitor_manager_cli.py: allows you to issue commands to the monitor managerutils/list_db.py: lists available items in thekekmonitorsdatabaseutils/reset_db.py: resets thekekmonitorsdatabaseutils/stop_moman.py: stopskekmonitors.monitor_manager